E14: The Reasoning Revolution with Ought's Jungwon Byun and Andreas Stuhlmüller

We've looked forward to today's episode since we launched the show! Andreas Stuhlmuller and Jungwon Byun are the co-founders of Ought, a product-driven research lab that develops mechanisms for delegating open-ended thinking to advanced machine learning systems. Their flagship product, Elicit (elicit.org), is an AI research assistant that helps researchers accelerate time-consuming workflows, starting with literature review. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. LINKS REFERENCED IN EPISODE: Ought: https://ought.org/ Elicit: https://elicit.org/ TIMESTAMPS: (0:00) Preview (2:00) Nathan introduces the founders of Ought (4:20) Why doesn't AI serve better reasoning? (6:55) What limits the current paradigm (8:40) Reflections on the last six years of Ought’s research experiments of "composable thinking" (13:10) Error elimination mechanism is a shared challenge between human and AI systems (16:45) Sponsor: Omneky (18:00) Interpretability by construction product philosophy (25:00) Nathan’s personal experience using Elicit as a research assistant (30:00) Explicit concerns about model reasonings, and the importance of going a step further (36:00) What customers of OpenAI Foundry should consider (43:15) Evaluation challenges (48:15) Embeddings challenges (51:00) Vision for a knowledge work assembly line corporate paradigm (56:00) Ought's short-term approach to building: Understanding human ways of teaching the model to be more helpful (59:00) Wishful thinking versus real helpfulness (1:03:00) What's next for Elicit: expansion and new workflows (1:17:00) Zapier for reasoning (1:23:00) What are the most fundamental "magic questions" for all domains? (1:31:43) Significant impact of GPT4 (1:36:00) How people are using Elicit (1:44:00) AI Uncertainty and reason for hope (1:48:00) 3 lightning-round questions TWITTER: @CogRev_Podcast @jungofthewon @stuhlmueller @labenz (Nathan) @eriktorenberg (Erik) SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected]. More show notes and reading material released in our Substack: https://cognitiverevolution.substack.com/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jungwon Byun?

  There's like kind of a sense of wonder that has been lost. And I think it's easy to lose sight of how much about ourselves and our world still remains to be discovered. So I really want tools like illicit and just AI more broadly to be able to help us, you know, not just kind of really amplify existing research efforts, but help us to discover entirely new ways of doing research, discover entirely new research methodologies, entirely new research domains that weren't possible before about like, you know, now using kind of this human artifact of text as a data source, analyze us more intelligently so that we can better understand who we are and how we relate to each other and what kind of systems we can, we can build to help each person and every group of person flourish more.

• 0:42Jungwon Byun?

  And for like AI systems to be able to generate like very decision relevant, but really, really, really simultaneously rigorous kind of one pagers policy memos and answers to people and very high stakes decisions so that we can just become way more intelligent about how we can govern this world.

• 0:58Nathan Labenz?

  Hello and welcome to the cognitive revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host Erik Torenberg.

• 1:21Speaker 02

  Before we dive into the cognitive revolution, I want to tell you about my new interview show, Upstream. Upstream is where I go deeper with some of the world's most interesting thinkers to map the constellation of ideas that matter. On the first season of Upstream, you'll hear from Mark Andreessen, David Sacks, Balaji, Ezra Klein, Joe Lonsdale and more. Make sure to subscribe and check out the first episode with A16Z's Mark Andreessen. The link is in the description.

• 1:49Nathan Labenz?

  Hi, everyone. I'm recording this in the hospital maternity ward where my wife Amy and I just welcomed our third child this week. Thankfully, both are doing well and we should be home by the time you're listening to this. Today's episode is one I have looked forward to since we launched the show. Andreas Stuhlmuller and Jungwon Byun are the founders of AUT, a product-driven research lab that develops mechanisms for delegating open-ended thinking to advanced machine learning systems. Their flagship product, Elicit, online at illicit.org, is an AI research assistant that helps researchers accelerate time-consuming workflows, starting with literature review.

• 2:31Nathan Labenz?

  I was particularly excited to have Andreas and Jungwon on the show for two main reasons. First, in my experience, Elicit works well, notably better than any comparable product I tried in early 2023, and so much so that it adds real value to research workflows. No small accomplishment, given the inherent difficulty of research tasks. Second, and likely more importantly in the long run, AUT has helped lead the AI industry toward a process-focused approach, which emphasizes task decomposition and the training of high-quality reasoning processes, rather than focusing solely on correct or incorrect outcomes.

• 3:16Nathan Labenz?

  We discuss how this process-centric approach works in practice today, why it's likely critical to building AI systems that can help with novel and important questions where little or no training data exists, how it might help us avoid catastrophic AI outcomes, and more. I hope you enjoy this conversation with Andreas Stuhlmuller and Jungwon Byun. Jungwon Byun, Andreas Stuhlmuller, founders of AUT, makers of Elicit, welcome to the cognitive revolution. Thank you. Hey, it's good to be here. Yeah, I'm really excited for this conversation. I have been a fan of what you guys are building at Elicit for some time and had some really

• 3:58Nathan Labenz?

  great product experiences using it in a couple of research questions that have come up in my own everyday life. And so I'm really excited to learn more about the product and also the philosophy that is behind it. Wanted to start with just a question drawn from your website. I went and read the mission statement and there's a really interesting claim, which I think is true, but maybe underappreciated by many. And you say, "It's especially unclear that ML will help in matters that require substantial thought." I think that's really apt. You know, in any sort of the writing or work that I'm doing,

• 4:39Nathan Labenz?

  I find that the best work, you know, the AI is currently the least helpful. So maybe you guys could just start by unpacking that. Like, why do you see that as the case? And then, you know, obviously we'll transition into what you're doing about it.

• 4:51Andreas Stuhlmüller?

  Yeah, it's actually interesting. We wrote that six years ago and I think it's still true. Like roughly speaking, I think there are two things that machine learning can do. One is it can imitate some behavior, like for example, it can write text like a human can write text. Or you can use reinforcement learning and can optimize some reward like win games of Go. I think you get reward when you win. I think essentially all cases where AI has substantially exceeded human capability are cases of reinforcement learning because that's what you need to get to unforeseen creative moves. And so then the question is, well, when does reinforcement learning work? And I think it works best in domains where you have like a pretty well-defined task, like play a game of Go and you have a pretty clear objective, like win the game.

• 5:33Andreas Stuhlmüller?

  But for helping us think more clearly or arrive at truer beliefs or better decisions, there isn't an obvious such objective. So there's a risk and the risk is that we'll use like a proxy objective or something that is like kind of similar to the thing we want, but not exactly the thing we want. And the most kind of natural such proxy objective is like, does it look good? Like, is it producing stuff that looks persuasive or impressive or helps helpful seeming? But that's not the same as actually being well-reasoned, robustly helpful, arriving at really good decisions. So I think actually right now, that's not a big problem. I think in terms of behavior, these models are still mostly in the imitation regime. Like they cannot just write stuff or brainstorm stuff or write code.

• 6:17Andreas Stuhlmüller?

  And I think that's all great and actually is helpful. But if you ultimately want them to be as helpful for kind of better thinking as they are at like playing Go, which is like better than the best human, then I think there's like some work to be done to make that work.

• 6:33Nathan Labenz?

  So there's kind of two, at least two issues there, right? There's the sparse reward issue.

• 6:39Nathan Labenz?

  And there's also the like human unreliability in rating or valuation. Are those kind of the core two issues that you see limiting the current paradigm? Or are there others too that you would want to highlight?

• 6:54Andreas Stuhlmüller?

  Those are real things, but I think it's more that there's not really a clear task definition. So I think the most natural task is I ask the model, hey, what decision should I make in this like setting? And then the model is like, here are some considerations for these considerations. You make that decision and then I can be like, well, does that sound good? Does it not sound good? I think like I have like some kind of resolution in judging that. But I think if I want the model to eventually exceed how good I am at making these decisions, then something else needs to happen like either. I mean, I think there are various ideas for how to address that. But I think the most natural paradigm doesn't solve the problem of kind of exceeding

• 7:34Andreas Stuhlmüller?

  human capability at reasoning and decision making in a kind of like really helpful way.

• 7:41Nathan Labenz?

  So that kind of feeds into then the next part of the mission statement, which is really your take on solving it. So you say, our mission is to automate and scale open ended reasoning so that future improvements in ML help as much with thinking and reflection as they do with tasks that have clear, short term outcomes. And it's interesting that you say you wrote that six years ago. I hadn't realized, you know, that the mission had been established that early. I guess I'd love to hear a little bit of like what you thought you were going to be doing when you started off and how that has kind of compared to reality. Cause I'm sure, you know, if you're anything like me, you've had a few surprises and updates over the course of the last six years.

• 8:22Nathan Labenz?

  So what was kind of, you know, how was that expected to cash out versus how it in fact has cashed out over time?

• 8:28Andreas Stuhlmüller?

  I think the surprising thing is actually how in line with expectations is everything. Everything is, I think, I mean, I think maybe timelines are somewhat shorter than we expected. But roughly speaking, when we started out, we were like, well, machine learning can't actually help with, and we can't like do even the simplest thinking tasks yet. That this was like before GPT two, there were no language models. I mean, I guess there were like a thing called language models, but it's very different from what you would call language model today. And so we were like, well, how can we still? Get ahead of the situation and study how to use future AI systems to help with that. Like, well, I guess humans are our closest standard for AI systems. So we mostly ran experiments with humans, thinking about, well, how can we decompose

• 9:12Andreas Stuhlmüller?

  complex tasks into tasks that kind of humans who don't know the big picture can help with, like break it down and all that. And I think it is kind of crazy how much AI systems now look like these kind of context-free humans that we ran simple toy experiments with back in the day. Yeah, I think I've been surprised by how much things played out in line with expectations, actually.

• 9:35Jungwon Byun?

  Yeah, it's crazy when we started the aggressive AGI timelines were like 2100. That's what the crazy people believe. They're like, oh, maybe we'll have AGI in 2100. And then over the span of a few years, that's gone to like, you know, condensed by so much. So yeah, when we started, we were doing more research experiments. But even then, I don't know, I think this is you, Andres, but like we were, we were trying to break down tasks to be very context-free. So basically giving our human participants a context window where they could only like read so much text to do a specific task. And like even, even details of our experiments like that have actually generalized quite a bit to, to what language models have actually ended up evolving into.

• 10:16Jungwon Byun?

  But yeah, when we started, because we had no line of sight to when ML would be actually good for, for helping with reasoning, we ran experiments, it was way more researchy. And then as language model progress happened, we realized like building a product around that and directly trying to help people with reasoning would be way more impactful.

• 10:34Nathan Labenz?

  Yeah, that's fascinating. So can you describe those experiments in a little bit more detail? The idea of, you know, giving a human a context window, I think is, is pretty fascinating. Like procedurally, what did that look like in the early days?

• 10:47Andreas Stuhlmüller?

  Yeah, there, there were a few different versions of those experiments. So I'll describe one simple one, what we called a relay programming. Where basically there's a programming task, like, like a programming puzzle you might, might get at one of these programming competitions. And every participant only has, let's say, a minute to kind of make progress on the task and then pass on their notes to the next participant. Where the thought was, if you could make that work such that, like, A person who doesn't like know the big picture can make small contributions that move the state of thinking on that problem forward such that you eventually solve it. Then you could get to mechanisms where you can solve like much more complex problems.

• 11:31Andreas Stuhlmüller?

  Like there's a fundamental kind of question of like, how composable is thinking? How much do you need to know like the big picture? Need to have like some mental model where you load everything into your head? And to what extent can you like get away with not doing that? And so one of the settings where we studied that was in the setting of programming.

• 11:49Jungwon Byun?

  I think the high level research question was in a world where we have super intelligent AI systems who are doing work that are, you know, that are smarter than us and so can do work that we can't evaluate directly. Can we break down complex tasks into smaller tasks that are easier for people to evaluate? So part of the reason why in some, in some of our experiments, we gave, you know, people context windows was because if like, you know, the AI system can read an entire book, but you can't, then that's really hard to evaluate. And so can we then break down work into smaller pieces such that if a human want to spot check parts of it, the amount of work that they would have to spot check is like manageable for them to do. That's kind of what we were studying.

• 12:29Nathan Labenz?

  So how, how productive were people able to be in this kind of one minute context window environment? Were, were teams or groups able to solve like substantial problems? I would expect that honestly to be very hard. And I think this is a really interesting comparison now as well, because we do have, you know, there's obviously a lot of debates around like, what do the latest systems mean? Like just how capable are they? How should we understand them? But I've not previously considered this notion of like testing the human on the sort of AI's turf. So I'm guessing we didn't do nearly as well as like GPT four would currently do. Certainly if you, you know, give it a minute, you know. As like your hard limit, you know, to make progress. So.

• 13:13Andreas Stuhlmüller?

  I think it is very challenging and I think the challenge is actually somewhat similar between humans and AI systems where one challenge is you need to get rid of errors more quickly than you introduce them. So like with every like you do your minute task, you probably are like, uh, like there's like some chance you get it wrong. And so the more such tasks you compose together, the more likely it is that like at least one of them goes wrong. Even if you only have like a 10% probability of getting like a subtask wrong. That's like, if you do 20 tasks, it's like probably gonna happen. And so you need some error correction mechanism and that error correction mechanism itself is probably needs to be composed out of these like one minute pieces.

• 13:53Andreas Stuhlmüller?

  And so if you are not getting rid of errors more quickly than you introduce them, then you're doomed. And I think that is, that is, I think, a thing that we're currently facing with language models as well. If you compose together many tasks, I think their models still are like somewhat flaky. And so that I think limits scalability of both the humans we work with and also current models to like very complex compositions of tasks.

• 14:20Jungwon Byun?

  In addition to the programming task, we also did text based tasks. And again, this is like, you know, before a lot of the language models, but we would have tasks where, you know, one person would make, would make, generate some summary of like a movie review. And then other people would have to decide like, was the summary accurate or inaccurate? And everyone only can only see little parts. No one, no one except, you know, one, maybe like the original generator can like read the whole, the whole summary. And yeah, and I think we, we ran into a lot of similar issues surprisingly there as we are with language models, which is like, you know, one maybe specific flavor of this error propagation issue is that this interaction can kind of converge in this like suboptimal place

• 15:02Jungwon Byun?

  where the people might like get derailed by a particular argument, line of reasoning, and then get stuck there and kind of like miss the overall point of the summary or like not find the most efficient way to decide if the summary was accurate or not. And then I think the other thing that we struggled with there is that like the context windows were really too limited, I think. And there needed to be some better mechanism for passing along state across different participants. And I think we're still not quite, we haven't figured that problem out with language models either. It's like, what is an intelligent way to pass state so that you're not kind of making this, starting from scratch with every call, call to the language model, basically.

• 15:38Andreas Stuhlmüller?

  Yeah, maybe here it's worth like zooming out and like briefly asking, yeah, well, why does this sort of context freeness matter? I think Deng one mentioned earlier a little bit, but I think the key thing is really, how can you make it so that humans can evaluate what's going on? I think one world is the world where in the future, I mean, I think these context windows are obviously going to grow, are going to go away eventually. And so one world is the world where AI systems are building complexes, like internal mental models, we can see those models, they make decisions. We're like, well, why did it make that decision? It's really hard to tell. Contrast that with the world where all the state that is being passed along is explicit. Then you can like, okay, well, this is what the model knows at this point. This is the local decision it made.

• 16:22Andreas Stuhlmüller?

  It said like, oh yeah, here's like the next thing we need to check out based on the things we know so far. And that's the thing you can evaluate. And so I think we just, we care a lot about getting AI systems to a setting where what they do is transparent and where humans can evaluate what's going on. And so that, that has driven like those experiments with humans in the past. And also drives a lot of our work with language models now.

• 16:46Speaker 02

  Omniki uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omniki so much that I invested in it and I recommend you use it too. Use Cogrev to get a 10% discount.

• 17:03Nathan Labenz?

  Yeah, it's funny, I would say that's pretty prescient really. Cause as you're describing the challenges in performing tasks that way, you know, and then you said like the zoom out, why do we care about this? My immediate thought was like, well, having used language models a lot, it's pretty obvious why I would care about it now. It may not have been nearly so obvious five, six years ago. But that is a really interesting, you know, just kind of history as to how you were exploring the. The cognitive space, I guess I'll say. And that's also like a pretty natural, I guess, way to bridge toward how you're building today. And it seems like this product philosophy of interpretability by construction reads almost like a mantra for what you guys are doing now.

• 17:48Nathan Labenz?

  And again, so fascinating that you started with that even before, you know, the technology kind of shaped up the way that it has. So tell us about this, this product philosophy of interpretability by construction, how it feeds into illicit and, you know, what illicit is today. And then I also want to hear what, you know, your vision is for illicit, you know, two and, you know, maybe end of the decade type of timeframes.

• 18:10Andreas Stuhlmüller?

  We mostly think about it as how can we make language models follow processes that humans can understand as opposed to these kind of blatant, unobserved, like only in the language models head kind of things that have been optimized end to end to accomplish some goal, but we don't actually know what's going on. So like we, I guess, the most common way that we and also others refer to this as, is this distinction between processes and outcomes where I think the most kind of natural framework for most machine learning is to think about it as like outcome based. You have some metric, like predict the next token or try to optimize reward in the gameplay setting.

• 18:54Andreas Stuhlmüller?

  And you're trying to literally set all the floating point numbers in your matrices to make it so that like that final number goes up as much as possible. And you don't really care about the, the meaning of the internal components. And so they might get like inscrutable. Contrast that with kind of more like programming in, I don't know, programming, like just natural program. Like if you're usually, if you're programming, you're like, well, I need to write functions. These functions need to have independent meanings that I can like kind of evaluate, like even outside of like a particular context. And if that's not the case, then it's like terrible spaghetti code that no one will understand. And it's hard to debug, hard to kind of like improve.

• 19:36Andreas Stuhlmüller?

  And so we're, I guess, on the spectrum from more outcome based systems to more process based systems. Our philosophy is like always try to push as much as possible for being in the process based setting where you build on human understandable task decompositions. You directly supervise the individual reasoning steps. And so try to get more to this sort of transparent setting. And then I think the obvious objection is, well, how's that gonna scale? That seems like involves a lot of manual labor. Isn't the whole bitter lesson of machine learning? That you just wanna like scale up your like matrices and optimize the end to end. And I think there is a lot of truth to that.

• 20:15Andreas Stuhlmüller?

  But the one maybe the idea that pushes against that is actually we can use language models and like AI more generally to help with the task decomposition itself. So it doesn't actually have to be a manual thing. The process of decomposing tasks into understandable pieces is itself a thing that AI can help with. Even in the world where end to end optimization is in principle more powerful. We think that the benefits from transparency and understandability are enough that I think it's worth like pushing really hard in that direction.

• 20:48Jungwon Byun?

  Yeah, the way I think about the interpretability by construction point is, um, I think the current world we're in is like, as kind of Andreas alluded to, do whatever it takes to score well on this feedback metric. And where like we often don't understand why, um, why a particular method works really well. Or we can't kind of like, yeah, it's not, it's not human legible, whatever it takes is. And scaling this up to superhuman levels and like scaling this up with a lot of compute is very concerning. We don't really want to do whatever it takes to score well on this metric. Um, and then there's kind of like an interpretability, you know, an approach to interpretability, which is like, okay, cool. I get this result from this answer. Now let me figure out how it arrived at that.

• 21:31Jungwon Byun?

  But I think that is also still concerning at kind of like superhuman levels. And even if you ask the model to justify why it did what it did, it can rationalize, you know, it can kind of like make up reasons that were not its actual reasons. So interpretability by construction is, you know, it was trained on kind of human legible processes upfront. Like you have this guarantee that this is what it did because that was, that was, it was programmed to do that. And so, you know, you can always go back and just see like, what were the steps you followed to get, get this answer? So that's kind of where the, that's like what, um, I think there's a distinction like interpretability by, by construction versus like just trying to understand why the model did what it did after the fact and using the model to generate explanations that way. And so like an example of this that we, uh, we have in illicit is, uh, this part.

• 22:16Jungwon Byun?

  This feature that runs through a trustworthiness checklist for papers. So currently, Alistair broadly is a research assistant. We've been focused on making it useful for literature reviews. And so right now, when you run Alistair, you get back a table of papers. We automate the process of finding relevant papers and extracting details and summarizing the papers so that you can very efficiently figure out if a paper is relevant to your research question or not. But good information is not just relevant, it's also trustworthy and something you should actually update in light of. And so, once we set up some infrastructure for finding relevant information better with language models, we then wanted to tackle, okay, how can we evaluate,

• 22:56Jungwon Byun?

  of all of these relevant papers, which one should most update the users' question? And we found in talking to a lot of researchers that there are very good standardized processes that exist for this already in academia. So we learned about things like strength of evidence checklist, risk of bias checklist. There are just these checklists that researchers already make to evaluate lots of different papers on. And so we started to just automatically run that checklist over every single paper. So I think a tempting thing here would have been to try and come up with some score for trustworthiness and probably other tools or whatever. Other attempts have been made like this. How do we think about citation count and what journal it was published in?

• 23:40Jungwon Byun?

  And then give it a rating. But the ratings can be easily hacked. And so what we're doing is still giving the researcher all the pieces they would need to evaluate trustworthiness without distilling a lot of that richness and losing a lot of context by putting it into a score. And then, that checklist currently has things like, what was this straightforward thing? Like, what was the sample size? Are there any conflicts of interest with the funding sources? Did they control for a bunch of different things? Stuff like that, like what, what is the actual like number that they reported on and who's it based off of? And so in the future, we could like take this and like condense it into a score if we wanted to.

• 24:22Jungwon Byun?

  But because we, we did this in this kind of like compositional way. If we decide to summarize that checklist into a score, it'll be very easy for someone to then expand the score and see the checklist again. And that's again, kind of like interpretability by construction. Do it in pieces first, then you can collapse it. But then, once you collapse it, it's also easy to expand it again and like see what went into the score. Whereas if we had just kind of collected a large data set and trained on, I don't know, what people rate papers as. Like no user could then go and really get a detailed understanding of why this score in this specific instance.

• 24:57Nathan Labenz?

  It really does work amazingly well, at least for many of the use cases that I've tried. So I would definitely recommend that our audience go check it out. I've used it for kind of random medical queries that have come up in my, you know, personal life. I would not, you know, my standard disclaimer is I wouldn't recommend making it your doctor at this point, but I would recommend getting a second opinion. I say the same thing about, you know, GPT-4, and I would definitely say here, like it can, it can help you surface valuable information. So that's been a really good experience. I've also had some, some really nice experiences, honestly, just even at the highest level of like,

• 25:41Nathan Labenz?

  it's often hard to search for a particular paper that I kind of recall what the main conceptual takeaway was. But maybe I can't remember the name or I, you know, I'm terrible with the authors' names in many cases. So now I'm like, you know, it's, my options at this point are often like, well, I usually found it on Twitter. So I can go like, search for things that I liked on Twitter and like put one word in at a time and hope that I actually did like it as I, you know, as I should have and that, you know, it pops up. Or I can go to a list and type like.

• 26:17Nathan Labenz?

  Sentence length, you know, it could still be a sentence fragment in many cases, but it's like, you know, what is that kind of conceptual thing that I'm recalling that, you know, is drawing me back to this in the first place? I've had a couple of good examples where I could not find the damn paper. And then, you know, with that kind of 10, 12 word, uh, you know, conceptual memory that I was able to, to finally, you know, spit out, then boom, like I was able to get exactly to the, the paper that I wanted. So that's obviously not the most advanced use case, but honestly, I would say that does put, uh, Alyssa ahead of almost all the competition at this point. You know, I've tried some other kind of research focused tools and frankly, in many cases,

• 27:01Nathan Labenz?

  just gotten nothing of real, you know, value out of it. And even with Bing, it's like very hit or miss, you know, exactly what you're going to get back and, and how good it is. So, you know, this is not a paid endorsement, although it's starting to sound like one. It actually just has been really useful to me. So go try it out. I always, we're big on telling people to go try out the actual tools. So, okay, back to questions, supervising reasoning processes, not just outcomes.

• 27:35Nathan Labenz?

  I'm feeling a little bit like the line there is kind of blurring in some ways. Like I'm, I'm feeling a little bit confused because on the one hand, I'm like, okay, it makes a ton of sense to try to create this, like, what was the thing thinking at sort of every step of the way? And certainly if I like rewind the clock a little bit, you know, and you're looking at, you know, very finite context window, language models is kind of all you have available. Then it's like, yeah, you can only do so much. So you kind of have to chop it up into these micro tasks. It's like the only, only way. But how do you see things like, for example, with like a 32,000 token context window now coming online? And with techniques that are kind of like, you know, whatever, least to most or step by step.

• 28:20Nathan Labenz?

  Is there a way in which we're kind of headed toward some sort of? Equivalence where you could like spell out all the steps that you want, you know, a GPT four thirty two K to take and then like have kind of watch it take all those steps, you know, one by one and then sort of say, here's your score on that, you know, with kind of all of that out loud. Are these paradigms converging in some sense with that with those kind of language model best practices? Or do you still feel like there are like fundamental differences that I'm not rocking?

• 28:53Andreas Stuhlmüller?

  I think there's like partial convergence. So I think the step by step is definitely a kind of move in the right direction. I think it's good to see like the reasoning spelled out. I think it's good for many reasons. I think it's still the case that I think there are some results where you replace the steps just with dots, like literally you make the model type out dot, dot, dot, dot, dot, as opposed to writing out steps. And it still leads to its answer improving because there's like some latent computation that can do during those steps that it wouldn't otherwise do. And I think that is maybe a little bit concerning. I think when you write out the reasoning, I think you definitely want it to be the case that the reasoning is causally responsible for the answer you

• 29:37Andreas Stuhlmüller?

  get in the end. Because if you're like, well, yeah, that reasoning all makes sense. And then you get an answer and you're like, well, did the does that really depend on the reasoning? I think you'd prefer not to be in that situation. Maybe it's like not really a big deal for basically all of the use cases we have of language models right now because they're so low stakes. But I think if you think a little bit ahead and in the future, like suppose you're running a company, you're like, well, should we like, I don't know, invest in this thing or hire this person or whatever decisions people will try to make in the future. I think there will matter more that the you can actually trust that the answer in fact, like faithfully depends on the reasoning. And so I think even though like writing out the reasoning steps in the prompt, I think is like, is a good step.

• 30:21Andreas Stuhlmüller?

  I'm like very excited for techniques like that. I think you ideally would go a step further and have guarantees that the answer causally depends on the steps, which you don't have for most of those approaches.

• 30:34Jungwon Byun?

  Yeah. So one, one example that I give to kind of contrast these two methods is, you know, maybe you ask a language model.

• 30:43Jungwon Byun?

  A research question, you're like, I just kind of like 101 pager on this topic. Like, you know, what should the government do about long COVID or something? And the language model produces an answer. And if you're actually going to make a high stakes decision about that question, you want to know, why should I trust this answer? And at that point, like, you know, the more trustworthy, like if the language model is able to say to you, well, I went out, I searched over these databases, I read these papers, I ranked these papers by this, by these dimensions, I checked to see if, you know, they were all trustworthy, I summarized them, and then I compared it against this data, da, da, da. If you can like list out all of those processes to you and to like actually follow that process, that is like that you can, you can have way more trust in that answer.

• 31:26Jungwon Byun?

  And you can then go and you can look at the process itself and say like, do I believe, do I trust this process or not? And you can also kind of inspect like each step and did it actually follow that step? But like, I think with just chain of thought, like, I don't know, the alternative would be like, it kind of says and inherently rambles about a bunch of things. And like, as I said, may not be like actually what it did, even though having more insight into into the, some more insight into the model state is helpful. I think you, that kind of interpretability by construction piece comes up again, where you want, you want some guarantee that this is what the actual model did.

• 31:57Nathan Labenz?

  Seems like a big part of the, the advantage there is also the kind of grounding, right? I mean, in a pure chain of thought, you're just relying on whatever the language model thinks it knows, right? And it's, it's then another thing to sort of say, okay, we're going to go to sources and evaluate those sources, try to find takeaways and then kind of synthesize that. So if I was going to try to, you know, continue to close, close the gap between the two paradigms, where you guys have taken things and kind of made the complex, you know, discrete and comparatively simple. And ultimately that like involves a lot of calls and each one is kind of, you know, a small input output.

• 32:41Nathan Labenz?

  And then you can, you know, view that whole history.

• 32:45Nathan Labenz?

  It seems like maybe much closer to that if I was like a, you know, let's say I'm an OpenAI Foundry customer and I have, you know, I just dropped a 1.5 on my, you know, 32,000 token robustly fine-tunable thing. The main thing that I would really be wise to do, it sounds like from your, your counsel would be like, ground in sources. That's like, you know, pause at some point, bring in information. Maybe it could all still be in like one context window, but you're going to want to interrupt at certain points and like bring in information. And then, you know, maybe at the end, you could sort of have like a summary that kind of goes back through all of

• 33:29Nathan Labenz?

  that history and is like, okay, here's what I did. And then, you know, I hit some stop token and that, you know, executed a search. And then I hit some other stop token and that, you know, fetched the abstracts, you know, for all those papers or what have you. And you could, in theory, kind of layer that on, but you'd be having kind of like layered calls as opposed to like compositional structure, I guess, to the, you know, to the overall process. Like linear with these sort of outbound calls might be at least much more, you know, kind of comparable to the composable structure that you guys have built. Does that sound right? I'm really just trying to understand it. So feel free to tell me if I'm, if I'm off base here.

• 34:11Andreas Stuhlmüller?

  I think, I think the thing you describe where you make like you halt, make calls to sources, I think that is a pretty common thing that maybe the simplest instantiation is like the thing that I think Bing does, which is just do a search, like don't do it even like in multiple steps, just do a search, take the results and then do a generation based on that. That's maybe like V zero of that. And then I think the, what you described is like one step more complex, which is you can kind of do some generation, halt, do a search, pull in the results, continue. And then I think there's like versions after that, which are like, well, one of the things you could be doing is you generate halt, decide to farm out some

• 34:54Andreas Stuhlmüller?

  Sub computation to another model that like is roughly speaking the same situation the current model is in. It does its thing, it comes back, then you continue generating. So I think there is a kind of incremental path from just like do everything in one prompt to a more compositional settings. And I think the world where everyone pursues those more compositional paths, I think is a pretty good world.

• 35:19Nathan Labenz?

  I guess another big opportunity for difference, and this is probably something that that does exist in illicit. I know it does from, I don't know exactly to what degree, but I know from your Twitter presence that it does exist to some degree is you can have your, if you do truly break down into a compositional approach, then you can have specialized models for all the different subtasks, potentially if you really want to, right? I guess you could also, you know, kind of, I'm really contrasting this because I just get the sense that like, you know, if I had to guess, I would say like half of the Fortune 1000 is probably going to try to buy OpenAI's new thing. And they're really going to be striving for reliability in their,

• 36:03Nathan Labenz?

  you know, in their language model output, right? Like they're, they're not going to be generally wanting it for brainstorming. They're going to want to be completing tasks and they're going to want to be completing them reliably. And we don't, we don't know like what, you know, kind of fine tuning they're going to offer, but you could sort of fine tune a bunch of different models for discrete subtasks. So you could maybe try to do like kind of 10 tasks in one, you know, giant fine tuning of a giant model as well. And then you're kind of back to maybe a question of like,

• 36:36Nathan Labenz?

  you know, does, does the sort of super generalization like help you? It might in some ways, you know, on performance, like certainly there seems to be this kind of, you know, bigger is better. Maybe you could help me unpack a little bit more of the worry of when I'm looking at kind of chain of thought reasoning, you know, you throw in those halts or whatever, and I get to a final answer, you know, you, you kind of said like, You'd really like to know that the answer was actually the result of the reasoning. And like on some level it is, right? Because like it is incremental token prediction. So like all the, you know, one by one prediction, right? So all those previously generated tokens in some sense are causal for the tokens that follow through obviously a still fairly alien mechanism.

• 37:20Nathan Labenz?

  So I definitely, you know, appreciate the, uh, the danger of, you know, just we have really no idea what's going on in very profound ways. But how is that? So how is that different? Like i- i- if you then like pull things apart into a bunch of different language models and you're like, okay, this one did this task. And then that task like feeds, you know, that output feeds into the next one. How do you see the like, where is that extra causality coming in versus just like the one token at a time prediction?

• 37:46Andreas Stuhlmüller?

  Yeah, so I guess it's important to clarify that it doesn't, it doesn't matter whether it's the same model or different models. So it like, it like, you definitely don't need to find, you know, different model. Like it's fine if all the subtasks are done by the same model. Um, and so the causality mostly comes through, um, removing information. So in the case where someone is doing a task and you know, the only thing they know is they are supposed to figure out, does this website answer this question? They, they don't know you're trying to make a decision about whether to take some medicine or not. They're literally just trying to answer, does this website like answer this question or not?

• 38:26Andreas Stuhlmüller?

  You know, they, they're not influenced in some way by like making it seem more or less relevant or otherwise taking into account, um, like context that it shouldn't take into account. So whereas if it, if it's all happening in a single prompt, then you're like, well, it could be, there could be kind of causal influence that you're not tracking.

• 38:45Nathan Labenz?

  Right, everything's just a lot more tangled.

• 38:46Andreas Stuhlmüller?

  Yeah, everything is more tangled, which for, for better or worse, right? Like sometimes that's really useful because sometimes to do the subtasks, you want to know, oh, which are, which are the things that actually matter for the bigger, bigger question. But also it has the drawback that it's harder, it gets harder for you to analyze like what actually happened. Um, and the in pieces lose their individual meanings.

• 39:07Nathan Labenz?

  Yeah, okay, that's really helpful because it does, especially with these really long, you know, super high coherence now with some of the latest models, like the, you know, the ability to attend effectively to these super long windows. Yeah, it does come. I've thought primarily, honestly, about the pros.

• 39:26Nathan Labenz?

  But you're highlighting a pretty good con, which is the fact that now everything is feeding into this one decision. And that just makes, you know, for every incremental decision you have, or every incremental, you know, task, whatever, you have a just fundamentally noisier situation where you're like, well, geez, you know, would it have done the same? You now have the question of like, okay, for that paper where we wanted to get this one bit, like, did that result depend on the original query? Or did it depend on like, the two papers before it, you know, that got analyzed in this kind of linear stream?

• 40:06Nathan Labenz?

  And I do see how that now would, I now see how that would make things a lot harder to tease apart. So, yeah, that's really interesting. Thank you for bearing with me as I

• 40:22Nathan Labenz?

  try to sort out my own confusion. I took a note also from your, I think it was from your big Twitter thread, which I've recommended previously about how you break down, you know, the task and approach the composability. So your guideline is, once language models do well at small tasks, roll up to bigger tasks. That reminds me of the Drexler Comprehensive AI Services a little bit, like, he's very, you know, let's deploy narrow, superhuman AI everywhere before we even attempt to do very general superhuman AI. Have you guys taken inspiration from that? Or how would you kind of, you know, contrast what you're building to the vision that he's outlined?

• 41:08Andreas Stuhlmüller?

  Yeah, it's definitely related. The maybe dimension along which to contrast it is like, what is the task size? Where I think Drexler mostly thinks about fairly large tasks. So like, there's a, let's say there's a service for evaluating like, whether an investment is good or something. And then, within that service, as I understand it, it's still fine if like, everything is more or less engine optimized. And I think we'd probably just try to push harder for even smaller, even more understandable components. And I think Eric also has more recent work. I don't know if you've seen like called the open agency model, which I think is also a bit more in that direction, like trying to get language models to do even smaller tasks like evaluating, like suggesting different

• 41:53Andreas Stuhlmüller?

  proposals for policies and then different language models evaluate those proposals and such. So I think there's definitely some convergence here.

• 42:01Nathan Labenz?

  Yeah, that's interesting. So how about the question of task size? Because like I've noticed this in my own work, right? At Waymark, we use kind of an ensemble of different AI systems, some of which are the core ones being language models, but also like some visual type stuff as well, because we're ultimately trying to create, you know, visual content. And I've definitely noticed over the last year and a half that it is getting easier and easier to just kind of throw more into a single call. You know, we used to have like a fine tuned model that would do this. And then we had a different fine tuned model to write the voiceover script. And now we're kind of like, okay, well, you could just write the main script and the voiceover script in the same call. Like, you know, that got easy enough that we can just go with it, right? Similarly, with like what image, you

• 42:45Nathan Labenz?

  know, content would complement this text, you know, we used to fine tune on that separately. Now we can kind of bundle it into one thing. That's all still a pretty narrow task, you know, in the certainly in the grand scheme of human endeavor. But how are you guys shifting the size of task, if at all? Like, are you sort of saying, okay, these are getting more capable, like we can trust them on, you know, somewhat bigger bytes out of the overall task? Is there kind of a pull in that direction?

• 43:15Andreas Stuhlmüller?

  I can, I can give like the general answer and then John can maybe address it in the context of the product if that seems helpful, but. In general, I think we must think about what are tasks that humans can fairly robustly evaluate? And you kind of want tasks to be as large as possible under that constraint. I think it's like, it's great if models do more work at once because for all the usual reasons for why it's useful for models to have more context. But it's bad if you get to a stage where a human can't look at it and be like, okay, this is like good or bad. I think there's like some caveats to this where you're like, well, maybe humans can use AI assistance to do that evaluation better.

• 43:54Andreas Stuhlmüller?

  But then you still have like that constraint of like, can a human with AI assistance robustly evaluate whether a subtask was done well or not? And you never want to push beyond like the threshold where that's true.

• 44:09Nathan Labenz?

  Where is that kind of settling? Like, what do you find that people can in fact evaluate effectively? I mean, for context too, for the audience, like, boy, is evaluation hard. I mean, I think most people probably know that. I was really amazed to see from the GPT-4 technical report that the GPT-4 versus 3.5 win rate is only like 2 to 1 ratio. It's a little bit higher than that, and it's like maybe like 70, 30. But, you know, it's not like people are preferring this like 99 times out of 100. It's nowhere close to that. It's like 2 out of 3. So that in and of itself is hard. And there's just a ton of noise.

• 44:50Nathan Labenz?

  Again, at Waymark, we've done a lot of experimentation with like, which are the best looking images? Because our users will have these image libraries. And like, one of the biggest things we want to do for them is just for convenience, like, pull the most beautiful images forward. And like, that is so easy for people. You know, it takes, it's like quarter of a second, you know, that they can kind of differentiate between which one they like. But the noise and the inter-rater reliability just gives us insane headaches. It's like the R squared on two humans sitting next to each other, looking at the same images is like really tough. And so the models struggle as well, right? Because they're trained on that signal and the signal is just insanely noisy.

• 45:33Nathan Labenz?

  So in your context, where are you kind of finding that it settles in terms of what people can reliably evaluate?

• 45:41Jungwon Byun?

  Yeah, I feel like this is another place where breaking tasks into smaller tasks really helps with the evaluation. So we have a bunch of kind of, in Elisa, there are a bunch of abilities to kind of ask questions of the paper. And the paper is like, you know, this has lots of text. It is kind of this like long context situation that we were talking about earlier. And Elisa, using language models can say, oh, this paper found this result. This paper studied these types of people. This paper had these limitations. And like, if that's all you got and you wanted to evaluate that, you would have to read the entire paper to figure out if that was true or not. Like, maybe you could, for certain things, you could control F it.

• 46:23Jungwon Byun?

  But in a lot of cases, you'd have to read the entire paper, which is very expensive and time consuming to do. But because the way we build out these tasks in Elisa is compositional, what we do is we first find relevant parts of the paper. Then using that part, generate the answer to that question. So when we show you the answer, we can directly link it back to where in the paper we got that. And then you, all you have to evaluate is kind of the answer and like a small chunk of text, like a few sentences, which is way more manageable. And it becomes very obvious what, you know, sometimes a model still make mistakes, right? So, but those mistakes become very obvious because you can check it. And then you can always also like, you know, scroll around the paper around that area to see like, was that section taken out of context?

• 47:05Jungwon Byun?

  So yeah, I think that's just like a really, another really important feature of like doing things compositionally and making that easier to, to evaluate really quickly. And like that size is definitely very manageable, but having to read an entire paper to come up with the answer is like, is way, is way trickier.

• 47:21Nathan Labenz?

  For that type of like source linking, do, like, I can imagine a couple of ways of doing it. One would be, you know, you have a question like, does this part of the, is this part of the paper suited to answer this question? And you could kind of page through, you know, to the degree that your context window allows to try to extract the part of interest. Obviously, you could also like pre chunk it and use embeddings and then kind of do like a embedding comparison at runtime. How are you guys approaching that problem of like grounding? And then, you know, I'm thinking with the embeddings too, it could be kind of hard to determine like. Well, the retrieval might have failed, right? Like, you don't have the answer, but the answer was there.

• 48:06Nathan Labenz?

  But like, you know, you pulled the wrong section. The embeddings feel like not super easy to evaluate. So how are you thinking about that part of the challenge?

• 48:15Andreas Stuhlmüller?

  Basically, you can think about embeddings as roughly speaking, cache language model computation. So I think ideally, the ideal thing is probably more like the former thing you described, which is you go through the paper, like maybe paragraph by paragraph. For each paragraph, you ask like, the most expensive model, think step by step and, you know, make sure you're correct and whatnot, and think really hard, is this relevant? So that's like maybe the most expensive solution. And then ideally, you would just do that over all papers ever written by anyone and all their progress. And so that's like, and then maybe, and then you rank by like the probability that it says it's relevant or something. So that's, that maybe defines like one kind of ideal. And then you're like, well, we can't do that.

• 49:00Andreas Stuhlmüller?

  So how, and then I think you're trying to approximate it. And usually, the way we go approximating it is, is like in a multi-stage process where you're like, well, first we want to find like the thousand papers or paragraphs that might potentially be relevant. And for that, we need to use like something very cheap, like some embedding based search. And then you're like, okay, now we have a thousand papers. We can use something slightly more expensive, like maybe like some cross encoder or something. And then you have like maybe one or a couple of papers. And then you're like, okay, at this point, we can use the expensive model and actually pick out the things. But I think this is, this is a setting where a lot of it is about like, well, what are kind of tractable approximations to the expensive ideal?

• 49:43Andreas Stuhlmüller?

  And hopefully over time, like people will find better ways to both improve what the ideal is and also better ways to distill kind of the ideal into kind of faster cache computations. Yeah, I don't think there's like something super clean. I think this is something that needs a lot of empirical evaluation and like trade offs related to how much you're willing to spend on any one query.

• 50:04Jungwon Byun?

  From a product perspective, we've thought about showing multiple segments. This at least kind of helps bridge some of the gap. So yeah, if you can show maybe like right now, we're just showing the best, the most relevant segment and the answer from that. But if you can kind of let the user tap through a couple of the highly ranked segments, you know, even if you didn't get it right in the top slot, you can kind of compensate for where the model might have missed some things.

• 50:27Nathan Labenz?

  Cool. That's really fascinating. Again, I think you guys are on, the more I learn about what you're doing, the more I think it honestly really resembles what a lot of corporations are going to be trying to tackle over the next like year to two. Because they're just now getting to the point where, first of all, they're just aware of all this stuff. And second, you know, the like frontline demo is good enough that they're like, well, they could do that. I could probably figure out my stuff. But now they've got these questions of like, okay, I've got whatever internal knowledge base that's in whatever form. And then I've got, you know, whatever tasks and processes and how do I, you know, it's kind of like the assembly lining of knowledge work is what I sort of expect.

• 51:13Nathan Labenz?

  A lot of corporations are going to try to figure out how to do, you know, for at least a lot of the processes that they have, you know, the, anything that's not like driving alpha is, you know, is like a good candidate for, you know, can you kind of pull it apart and turn it into some sort of knowledge work assembly line? I suspect that, you know, you guys will be in high demand as it becomes clear that this paradigm is like really what the, the corporate customers are going to need, you know, they, they, they just, they don't want like, they may want some bigger tasks. I think they probably will want bigger tasks than, than what you guys are doing. But still like taking a five minute customer service phone call and like resolving an issue, you know, is

• 51:57Nathan Labenz?

  one or two levels up from where you are, but it's still like pretty small, you know, in the grand scheme of things. And they're going to be, I think, working really hard to figure out how can we structure all this activity that goes on in our business in ways that we can like feed it into, you know, an AI assembly line and kind of reliably get. You know, the desired behavior out the other end.

• 52:22Jungwon Byun?

  As we start to see like more high-impact, high-precision, high-reliability applications, we'll really need that infrastructure. So for for kind of like, you know, approximate use cases, I think something like ChatGPT will work just fine. Like, you know, like you said, maybe just a quick second opinion or a quick like, what do I need to know? What should I then go and ask more questions about? It'll be very useful. But, you know, we're seeing right now, obviously, people know that these models have issues with hallucination. I'm sure a lot of those will get trained out, or, you know, they'll be better at citing their sources. But I think it's still, even in that world, won't be clear that what the model is telling you is the best possible answer. So not just not incorrect, but like

• 53:07Jungwon Byun?

  the best possible answer. And I think with these more serious use cases, we'll need systems that can kind of give us the guarantee that this was the best possible answer. So we'll want to apply these models much more systematically, and not just kind of like take the first thing that they tell us, but like, have them run through every single option and do really detailed checks and double checks and think about everything that could go wrong. And then be like, having all considered all of that, this is the best possible answer. So I do think that with more serious use cases, the way we relate to these models, the way they're deployed will change.

• 53:37Andreas Stuhlmüller?

  Yeah, I also agree, Nathan, with your kind of analogy of kind of assembly lining of cognitive tasks. We've been thinking about that a lot as we built the new version of illicit. So I think, let's say right now, still looks a lot like kind of quick searches. But we've been thinking more about, I guess, we've been calling to kind of large-scale industrial cognition, you know, where you're like, I'm not just trying to get like some quick answer to a thing. I'm trying to actually define some process where I go through like maybe all papers that I have internally at my organization. And then for each paper, I extract the relevant insights. And then I want to do like some systematic ranking at the end of the insights, maybe could connect it to other database. Anyway, there's like

• 54:21Andreas Stuhlmüller?

  some fairly large-scale, probably pretty expensive process that I want to run, where at the end, I want to have some guarantees about like how good the result is. And so I do think there will be a lot more of that in the future.

• 54:33Nathan Labenz?

  That kind of also gets a little bit to this question. I think it's ultimately both, right? So I'm not looking for like an either or answer, because I think that would be silly. But there's this debate right now between like, The co-pilot model on the one hand, and then kind of the assistant model on the other hand, right? The co-pilot would be characterized by you are doing your work and, you know, the thing is kind of helping you out along the way, answering your questions, you know, auto-completing for you. But I think that like the vision that people have in their minds there is largely like you are doing work in the same way that you do it today and you're getting help, you know, as if you had somebody, you know, sitting next to you, helping you.

• 55:18Nathan Labenz?

  Then the assistant model is like, I want to tell the system what to do and its job is to do it, come back when it's done and hopefully have done a good job. And it sounds like you guys are maybe more trying to build like a, an assistant model type of thing where you want the user to be able to like delegate real tasks and, you know, not just be kind of getting the like weaving in and out of like AI commentary on the research process, but like actually get a quality literature review out, you know, and like be able to advance to that square on the, on the board in kind of a discreet way.

• 56:00Jungwon Byun?

  I think it's like, there are maybe like two dimensions to this. One is what is the most helpful way to build this kind of co-pilot helpful like voice next to you and like the, I think our bet is to figure out what the hu- human is actually doing and then to break it down and then, and then train it on those tasks. So instead of just like, you know, letting the language model talk and hoping it's helpful, being more opinionated about what is helpful in this workflow, teaching the model to do that and then the model, like we know that the thing the model does will be helpful. And then I think like, and so that's maybe like about how we kind of approach building in the short term. In the long term, I think it's it kind of like remains to be seen.

• 56:43Jungwon Byun?

  What is, how do we, how exactly do we want to integrate these models into more workflows? So in the beginning, maybe they'll do, yeah, like just how much of that process can be fully automated? And I think, I think our approach will be like, do the small tasks, work up to bigger and bigger tasks. It's very unclear if we'll ever be able to automate the whole thing. We'll just like keep, you know, keep automating more and more as it makes sense along the way. Some people talk about how, um, yeah, maybe there's just like a fuzzy component of reasoning that can't be made explicit and easy to automate. So who knows, maybe we'll, we'll encounter that at some point.

• 57:16Andreas Stuhlmüller?

  There is a fundamental constraint with the co-pilot model, which is everything needs to be really fast. Right? Like you want the, you want the model to respond immediately to you. You want your code completions to be there in the moment. And I think, I think that's just a fundamental limit on how much cognition it can do. And so if there are tasks where you're like, well, I really would like the model to think harder about this thing, then either it's going to be an extremely slow co-pilot, which I think, I guess, maybe that's just starts being more equivalent to the assistant. But I, I think I would be like very surprised if all use cases that we are interested in just look like the co-pilot model, because I think that would mean accepting that we never do more thinking than some like very small amount for any given task.

• 57:58Jungwon Byun?

  Yeah, I think plausibly, there will just be some types of work that machines are better suited for than people. And like, it makes sense for machines to like to divide and conquer here. Like there will just, it will just, you know, language models will just be able to read way more text and extract information from them way faster and be like way more comprehensive. And like, they should do that type of work. So it might not just, it might, we might not exactly be on a spectrum of like minimal automation of what the person does to maximal automation of what the person does. But like, it's more about specific tasks and how to split them up.

• 58:30Nathan Labenz?

  Yeah, that was a big point in the economic analysis report, the GPTs are GPTs paper that kind of accompanied or you know, fast followed on with the GPT four release. An interesting study there of like, okay, if we break down a, you know, we know it's been commonly, you know, said, and I think it is true that like, jobs as they exist today are not really amenable to like AIs showing up and doing the jobs. There's too much context needed, you know, there's physical demands, there's, you know, whatever, there's, there's a number of, it's pretty easy in any given job and the report says this, like, you look at any given job, it's very rare to find one where you could feel like the AI could do the whole thing. But then they break it down into tasks and say, okay, well now what are the tasks that make up

• 59:15Nathan Labenz?

  this job? Kind of, you know, a composability paradigm, right? For what is a job? And then they find like, well, a lot of those tasks for a lot of jobs, you know, are in fact tasks that language models are now capable of doing. So, yeah, I think I come down, I mean, I think we will certainly see both, but it feels to me right now, like, there's honestly, I think, a lot of wishful thinking kind of feeding into the copilot model that, or there's like, there's a, it seems like a bias, you know, that is kind of a wishful thinking bias toward the copilot model. And one thing that's been really interesting in doing these interviews for this show has been,

• 59:56Nathan Labenz?

  I always ask, I'll ask you guys too, but I always ask like, what AI products are you using regularly? And strikingly, the answers are very few. Like, there's not that many, you know, products that are cracking into folks' daily lives at this point. And I don't think it's really because in most cases, like, there aren't tasks that they could delegate, but rather like, it's often just not quite helpful enough to go do it totally on the fly in these kind of, you know, oh, let me pop over to here and, you know, get my social media content written or whatever, like, eh, you know, I think what people really want is to like, Be able to delegate whole tasks, you know, and have them done.

• 1:00:42Nathan Labenz?

  But then there's like some people who are, you know, either afraid of that, or it just feels like uncomfortable to think of kind of building technology, or it may just be bad in marketing, right? I mean, that's the other thing, like, people may, they may know on some level that things are headed one way, but they may sort of still be telling a co-pilot story because, you know, like, what else is GitHub going to do? I don't know how, they can't really, it's going to be tough for them to reposition around like the AI developer for, you know, lots of reasons.

• 1:01:11Jungwon Byun?

  For me, the barrier is when I'm using some of these language models to actually do a job that I care about, I have very low tolerance and patience. Like, I don't want to, I don't want to read this like paragraph, I don't want to hear about how you're this like helpful assistant, I don't want to like give you feedback on this answer that you didn't quite get right. I want just, I just want the best answer right away. And so I wonder, I wondered how many other people have, like, some people I think are enjoying this flow with the language model where it suggests something and they give it feedback, but I'm sure there are more people like me who are like, I just want to get this task done. And there's just like a little bit too much hullabaloo right now around interacting with this like assistant. And I think that's, yeah, I think we'll probably start to see more products that kind of apply

• 1:01:56Jungwon Byun?

  these language models in these higher reliability, well-defined, discrete tasks. Because when you're trying to get something done, like, you don't, I don't know that you want to have a long conversation about it, you just want the thing done.

• 1:02:08Andreas Stuhlmüller?

  I had one other thought on the marketing point you made,

• 1:02:12Andreas Stuhlmüller?

  which is, I think, even if you ultimately want to automate whole tasks, maybe it's like pretty useful to get feedback from people. So the models are still like relatively weak in many ways, and so being able to have millions of users use your product and give feedback on the places where they break down, I think is pretty useful, even if your long game is to cut the people out of the loop. Maybe that's a cynical view, but that, I think, that's also one view that's compatible with what's going on.

• 1:02:41Nathan Labenz?

  Yeah, there definitely is an element of that. Like, we had Omen named Kirtina from Google Robotics Research on, and she was talking about a paradigm where as they envision deploying robots in the future, You know, there's obviously going to be so many edge cases and long tail scenarios. As I understand it, their plan is to train the model to know when to stop and call for help. And then have like a remote operator in a call center somewhere that will literally like use joysticks, you know, to do the task for the robot. And then there's your training data, you know, patching. So yeah, there's definitely going to be a lot of of that.

• 1:03:27Nathan Labenz?

  But yeah, I do wonder, I mean, Copilot is very good, right? Because if you accept, they have a very natural loop there. If you accept, it must have been at least decent. You know, they're getting a strong signal like from a lot of a lot more suggestions probably than almost like any other product. Anyway, enough Copilot. Let's talk about illicit and the evolution. Right now I go to the site. It asks me or it invites me to ask a research question. And what I understand is you're kind of expanding the workflows that it can do. So tell us how you are expanding what new workflows and what we can expect next.

• 1:04:05Jungwon Byun?

  Yes, I think the way to think about this is last this like kind of initial phase of illicit was about taking unstructured data and information is kind of like trapped in PDFs. And structuring it into a much more organized table where you can kind of efficiently skim across a lot of information at once. And automate the process of going into each PDF and figuring out what is it that they did and what should I know? And then the next phase of illicit will be, okay, cool. Now we have this like structured data model, like this this table. How can we run interesting and decision relevant queries on top of that? How do we take this information and then make it even more useful and easy to get good answers out of that?

• 1:04:50Jungwon Byun?

  So that, you know, researchers don't have to then just walk out with this long list of PDFs that they kind of still have to read. Even if they they understand better what they are about better than they did before. So specifically, the new workflow that we're launching, we're calling List of Entities. And the idea is, can we extract concepts that are discussed across multiple papers? And then show you the concepts with links to the papers and the parts of the papers where they're discussed. So this could be like a list of data sets, you know, you know, that were used in lots of papers. A list of like machine learning benchmarks that a bunch of models were evaluated on. A list of effects of a certain medication, a list of interventions tried. I'm trying to get it to work for like a list of like outstanding research directions, for example.

• 1:05:35Jungwon Byun?

  So this list form is like, is really, it's quite general. I think the core thing here is querying this table in like a much more rich way. And like organizing the information in that in a way that's like way more decision relevant. So yeah, that's, that's like the big next workflow. And then more generally, trying to think about moving from one workflow to supporting many workflows. So some of the other ones we have on, on the docket are running the kind of current version of illicit. But over like a much larger set of papers. So we have a ton of organizations who have their own databases of like thousands or tens of thousands of paper. And want to basically like get an illicit table for their own paper. So thinking about supporting use cases like that. Yeah, those are some of the things that are on the roadmap.

• 1:06:21Nathan Labenz?

  You guys are a research nonprofit, but you describe yourselves as a product driven research or product led research nonprofit. What like, how do you envision that evolving in the future? It sounds like you're thinking of offering this on a commercial basis to large companies. Is that right?

• 1:06:40Jungwon Byun?

  In terms of our long-term roadmap, we see this as having three phases. We have a three-phase plan. Phase one is discover what is known. Phase two is discover what is unknown. And phase three is decide in the face of the ennoble.

• 1:06:55Jungwon Byun?

  So we're currently in phase one, discover what is known. And the core problem we're trying to solve here is the fact that there's already a tremendous amount of information that exists in the world. It's just trapped in unusable PDFs and books, right? There's just too much information. It's not structured to be very decision relevant. It's very hard to query. It's really hard to read an entire book to get, you know, a very specific answer to a question you're looking for. So a lot of the capabilities we want to build out in this phase are around information retrieval and synthesis. It'll probably mostly focus on text. And we'll be trying to take kind of this unstructured text that exists in the world in papers and start structuring it into a tidier data format. That's kind of what you see in illicit today.

• 1:07:40Jungwon Byun?

  We're taking a bunch of these papers. We're converting it into nice tables. And then this next list of entities workflow is a way to query that table in a richer way to reorganize it at a higher level of abstraction where you're kind of looking more at concepts discussed across papers rather than the papers themselves. So I think, I think phase one will look a lot like query and kind of converting text into richer structured data formats. And then flexibly querying that that underlying data into more decision relevant insights.

• 1:08:14Jungwon Byun?

  And then phase two is discover what is unknown. In this phase, we want to help researchers start to contribute to that body of knowledge. So having kind of parse through all of the texts that already exist in the world in phase one. We should be able to start finding gaps in that literature and then helping researchers resolve those gaps or identify cruxes that they can resolve. Where I expect that we'll relate to language models totally differently in phase two than we did in phase one. And here, I think the core, core kind of like technical capabilities we need to build are around causal modeling. So using these language models to, to develop good, good models of the world and how it works. I think you can think of all the work in phase one as kind of collecting a bunch of observations, things that people have,

• 1:08:59Jungwon Byun?

  yeah, observations that people have gathered from running experiments. Then we want to collect those observations to build towards a richer causal model that we can generalize into new contexts. And then researchers can kind of take that causal model and come up with new hypotheses that they can design experiments around. So it'll be very different. Where in phase one, we might want to apply it. Language models kind of at scale in batch and go broad. In phase two, I think we want to go much deeper. And then in phase three, decide in the face of the ennoble. We want to, we will probably like switch user groups a little bit and move from research producers to research consumers.

• 1:09:38Jungwon Byun?

  People who want to incorporate research into their workflows are making very high stakes decisions based off of research, but are not researchers themselves and who are kind of doing much more than just understanding the state of science.

• 1:09:55Jungwon Byun?

  And, and again, here, I suspect that we will relate to language models very differently. I think this here, the kind of problem we want to address is that for some really high stakes decisions or some parts of those decisions, there is no right answer. It, it really comes down to your values or your preferences or just certain choices you want to make. And so at this stage, I think we want to use language models totally differently and use them to elicit uh, the values and maybe kind of unresolvable assumptions or beliefs about the world that the, that the individual has or the decision maker has. So for example, if you want to make a decision about your career,

• 1:10:38Jungwon Byun?

  part of that might be informed by research, but a lot of the rest of that might just be around your own values or other kind of hard to resolve beliefs about the world that you have. So anyways, those are our three phases. I think the important thing to emphasize is that literature reviews is very much a small part of our long-term vision. We don't see ourselves being a lit review tool forever. We don't see ourselves just being kind of a search engine forever. But understanding the state of literature and being able to work with that is just a really important building block to everything else we want to do. So that's why we're starting here.

• 1:11:12Jungwon Byun?

  And if that sounds exciting to you, we're hiring for a product manager. So this is one of my biggest priorities right now. So if you are interested in really helping us convert this grand master three-phase plan into a concrete roadmap and are excited about all the different things Alyssa will become over time, please reach out or recommend people you know that seem like good fits.

• 1:11:35Nathan Labenz?

  How like sciency is the target market for that versus just like general use corporate, you know, because I could see you like, well, I was thinking, who would be your number one most natural customer? And I don't have a great answer, but the first thing that came to mind was the Broad Institute. And I was like, they probably have a ton of stuff, you know, that they might like to be able to search over. I don't know how much of it would be public versus private, whatever. But okay, there's, that's kind of one. Certainly they have like the science, you know, pedigree. But then I think like, okay, you know, Coca-Cola, like, would that be a target customer? You know, they're doing stuff with AI, I guess. Is that in scope or is that just kind of too sort of commercial, too consumery?

• 1:12:17Nathan Labenz?

  You know, not sciency enough to be within what you guys would try to do.

• 1:12:21Jungwon Byun?

  I think broadly, we, um, we want to prioritize researchers in a broad sense. And that kind of, that will get broader over time. So up to now, we've, you know, most of our users are kind of academic researchers, PhD students or professors. But the, there are kind of researchers like that in so many different places at many different companies. There are, you know, researchers who are trying to study the effects of sugar in Coca-Cola or like what happens if you reduce the sugar content, probably lots of researchers in industry. And I think the activities that they're doing are very similar, even if the domains are slightly different. So we do want to support all of these people who are trying to have good models of how things work in the world and have like good, kind of like science-backed beliefs

• 1:13:04Jungwon Byun?

  and are thinking a lot about this information. And that's really a big part of why we started with literature review. I mean, thinking about scaling up reasoning and research. This is like, is a workflow that's general to many different domains and generalizes even beyond research. So, so like, yeah, probably in the short term, we'll keep focusing on more academic researchers, but over time want to scale to everyone who's, who's thinking really hard about evidence and trying to make good decisions in light of that.

• 1:13:34Andreas Stuhlmüller?

  I think it depends less on like, What like, what is the flavor of the company? Like, do they sell sodas or not? It's more like, what is the, what is the task they're doing? Like, are they trying to do serious reasoning? I think we probably do not want to just build like an internal search engine for someone's documents. I think there are going to be plenty of others who are going to be doing that. And but we do want to help. If there, if there are people who are like, well, we're trying to figure out like, how to use AI to make better decisions or figure out what's true in some setting. I think then it becomes more interesting to us.

• 1:14:08Nathan Labenz?

  Okay, that's really interesting. When you think about going to customers, like, you know, any sort of big company, would you imagine allowing them to kind of create their own

• 1:14:23Nathan Labenz?

  composable workflows that are like riffs on yours? And then I guess the follow up question would be like, as soon as I imagine doing something like that, then I imagine, well, these people are probably going to struggle with this. And then I go to, well, maybe I could get a language model to give them some good suggestions, which might also be in your plans like separately. So at some point, you guys can't do all the, all the decomposing, right? Like, what's the future of the decomposing process?

• 1:14:52Jungwon Byun?

  Yes, this is the dream. We tried actually earlier this year to use language models to decompose tasks and they just weren't quite ready for that, somewhat surprisingly. And so that's why with this list of entities workflow, we're still hard coding the workflow. Like, we're saying first, you should search over these papers and you should extract these statements and then you should cluster them in this way, et cetera. I think the evolution here will be, and I, and I do think it's like, for a lot of users, they don't want to look at a blank screen and think about, how do I need to make this tool do what I want it to do? So I think for most people, they'll want kind of like a predefined workflow to start with. And then from our perspective, the evolution will be we'll let users kind of edit

• 1:15:34Jungwon Byun?

  parameters of the workflow to start. So now I have this list of entities workflow. Maybe I can change the search query in the first step. Maybe I can get a few more papers. Maybe I can delete some of the papers. Maybe I can like change the way the clustering is done. That'll be like the next phase. The phase on top of that is I can make more meaningful changes to the steps of the process. Maybe I'll search over a different corpus. Uh, maybe I'll add another search step at the end to check if like this entity came up in Google somewhere or whatever. Um, and then I think once you're there, then you can get to a place, um, where, you know, people are just kind of creating their own workflows, right? Like editing each steps, you know, could lead you to edit the workflow to an unrecognizable state. And then from there, you can just kind of start from scratch, um, and create the workflow from that thing. So, um, and then once, you know, we have

• 1:16:17Jungwon Byun?

  many more examples of that, presumably we could have language models also automating those steps or kind of, you know, co-pilot suggesting, um, the steps that someone might want to take. Uh, and, and interestingly enough, we already have some researchers who are basically doing this. So in it, like the main workflow in illicit is lit review, but there's this like secret page called tasks, which is like our graveyard of all the things we experimented with before we landed on lit review. And it's a ton of things like brainstorm research questions, suggest search terms, like generate counter arguments. All these are very kind of well-defined, um, little, little tasks. And we have some researchers who have like basically manually like stitched together the results of those tasks and like created their own workflow in this janky way because we don't currently

• 1:17:00Jungwon Byun?

  support it. Where they like, you know, have a question, they like generated a bunch of research questions, looked, uh, asked each of those questions into lit review, got a bunch of papers, took the abstracts of those papers, ran the abstract summarization task, and then like, you know, generate a bunch of other ideas from that. So they're like currently manually like stitching together these tasks to do more complex workflows. And that's something we want to like automate and make way more easy to do natively in the platform going forward.

• 1:17:25Nathan Labenz?

  It's like Zapier for reasoning.

• 1:17:28Jungwon Byun?

  Yeah, exactly.

• 1:17:29Nathan Labenz?

  Does the code base get hard to we hard to like corral? I mean, when you describe all those different tasks, and especially if they're all kind of explicitly defined. It sounds like a lot of code under the hood. Does that become a challenge?

• 1:17:45Andreas Stuhlmüller?

  I mean, it's, I think, as with any fast-moving field, I think it's an ongoing thing where we're trying to figure out the right abstractions. But I think ultimately there aren't too many of them. So there are a few core reusable components like search, answer, given results, summarize, rank, cluster, filter. Yeah, I often kind of think, and so you have those components, you can compose them. And I often think a little bit about it as kind of functional programming with natural language. In functional programming, you have like some core higher order abstractions. You have like map, fold, filter, and then like most of the, like those are, there are like maybe five of them. And then there are, you can apply those to more atomic tasks, which are either deterministic computation or language model calls.

• 1:18:30Andreas Stuhlmüller?

  But there isn't, it's not like there is a huge number of like invalid components and you need to add. In some sense, that's the beauty of language models is that you don't have like that many components. It's about a bunch of pieces and you need to be kind of reasonably smart about how you compose them. And I think that gives you a lot of power. So I don't, I don't think we'll be in the business of like building out like resilience of components. I think we are more in the business of, as you, I think, I guess, I don't know how this relates to the SAPIR analogy. But I think we are more in the kind of infrastructure for composing stuff business than in the make lots of small like bits, bits business.

• 1:19:10Nathan Labenz?

  So what about when you get to like low level bits where, you know, like if it's medicine, then, you know, as you're validating the trustworthiness of the result, you might ask like, is it double-blind study or not? But then if I'm, you know, doing some different question in AI, then it's like there's, you know, generally that's not really relevant. So it seems like there's got to be some, it's almost like, to me, it seems like there'd be this like fractal complexity at the low level. And like, maybe those aren't different components, but it seems like we're still, I'm struggling to imagine how you could handle the breadth that you handle without some sort of like dynamic decision-making, you know.

• 1:19:54Nathan Labenz?

  To figure out where you want to be on those low-level tasks.

• 1:19:57Andreas Stuhlmüller?

  Yeah, yeah, I think there are different sources for that information. I think one source is, well, language models probably know some of those things. So I think you could just have dynamic task decomposition where you ask the language model, what are criteria that computer science papers are generally evaluated on when reviewers make decisions or something like that, and then they probably say a bunch of things. And then I think the other, the other source that we talked about like just moments ago is people are using the platform and people have views on what they consider important or less important. So I think that's, I think in some settings, we might also be the users of our platform and might also like have views and help people kind of come up with things like with illicit right now, the risk of bias analysis, I think is a thing that we put in.

• 1:20:42Andreas Stuhlmüller?

  But I think the trajectory over time is probably more of that stuff comes from some combination of users and our language models.

• 1:20:50Jungwon Byun?

  Yeah, I think in the, I actually think when you look at it, it is, there is a lot more generalization than, than the example you gave might seem, Nathan. So in both that kind of computer science question and biomedical question, the activity that the model has to do is the same. It's usually given this segment from a text. What is it? It's a question answering over a piece of text, basically, like according to this part of the paper, what is the answer to that question? And so that, the source of that, you know, that, that text can be a computer science paper, or it can be a biomedical paper, but like the activity of use this text to answer this question actually generalizes to both of them. And so I, I think so far in our experience, it hasn't, they, those two don't have, need to be treated differently.

• 1:21:34Jungwon Byun?

  And then maybe in some of these kind of higher level workflows, they might be a little bit different. So, you know, like Andreas was saying, maybe this risk of bias checklist, where we ask about sample size, whatever, like is that, that checklist looks a little bit different for medicine than it does for machine learning, but we might just want checklist in both cases. And then the way we create those checklist might still be pretty similar, even if the content of them is different.

• 1:21:58Nathan Labenz?

  So today, it sounds like there's a challenge that you guys probably have made a real art out of. Again, we had something kind of like this with Waymark at one point in time where, especially before language models, you know, we're trying to generate content for people in this like automatic way. And so, but you know, our customer base is like all small businesses. So the diversity is insane. And we got pretty good, definitely not as good as like the language model experience is now. We got pretty good for a while at like classifying businesses into one of 20 categories. And then having like, you know, kind of uncanny valley copy for a lot of those categories where we'd be like, all right, well, if this is a law firm, then we're going to classify it as like professional services.

• 1:22:42Nathan Labenz?

  And then, you know, the kind of like marketing messages that they would want to talk about would be like that they're trustworthy and trusted and, you know, versus like if you're a restaurant, you know, it's not, you're not going to go with messages like that, but you'll go with, you know, what a great taste and what a great, you know, experience and how much fun it was, whatever. So I imagine you have to work pretty hard to kind of think about like, what are the right types of questions, you know, because even in that scenario of like, it's just question answering over a text. So the question of like, what question, right? Is it, is it answering? And I imagine you have thought pretty long and hard about like, what are the questions that are kind of most generally applicable?

• 1:23:27Nathan Labenz?

  What if there are any questions that you could, you know, that you could share that are like, yeah, this is where, you know, yeah, we don't ask, is it double blind? Cause that, you know, doesn't, you know, generalize super well. But instead we ask like, you know, we found this is the magic question to ask for, you know, evaluating a random piece of text from a random paper in a random field.

• 1:23:47Jungwon Byun?

  Yeah, I think we, I think generally the things people want to know across all domains or some like, so this is maybe for empirical domains. And then I do think for theoretical domains, it might be a bit different. But within empirical domains, I think the fundamental questions are like, what did they do? What did they find? What did they not find?

• 1:24:09Jungwon Byun?

  Something like that, like at a, at a very, very high level. Yeah. And so right now we've been, last year we were focused on kind of helping biomedical researchers. And then this year we started trying to help more like people who are working on AI safety or like empirical kind of machine learning research. And I think a lot of the structure generalized, even if the specific questions were a little bit different. So now, and when we ask, what did they do? In, in medicine, we would ask, what was the population? Where were they located? What was the intervention? And now we ask like, oh, what was the data set? What was the model used? What was, what were the techniques used or something like that? But yeah, we just, I don't think we really like run into significant problems with generalization there.

• 1:24:52Andreas Stuhlmüller?

  Yeah, I think the, what did they not do? I feel like is maybe the one magical question to ask about everything. So I always find it most useful to ask, what limitations do the authors say their work has? Like, that's like extremely true for machine learning papers, because if you just look at the abstract often, you're like, wow, that's amazing. This solves everything. And I think it's also true of other work, because knowing what they didn't do really helps you understand the boundaries around the thing they do often better than the literal explanation of what they did. So yeah, if I had to pick one question, maybe that would be it.

• 1:25:26Nathan Labenz?

  I know you guys are like kind of sewers of language models. And I've seen from some, you know, some of your published stuff on Twitter that you're using a mix. So I'd love to get a sense for kind of what the ensemble of language models are that you're using. I know you've done some in-house training and, you know, also using some commercially available stuff. So kind of sketch that out, you know, to the degree that you're, that you're comfortable doing so. I think people would learn a lot from that.

• 1:25:55Andreas Stuhlmüller?

  And so I think we have tried basically everything that exists. So including like all APIs, like topic, OpenAI, Cohere. I guess we haven't tried the Google API. Anyone from Google is listening, please give us access.

• 1:26:10Andreas Stuhlmüller?

  Yeah, and then I think we've tried most of the open source models to, I guess, Galactica and GPTJ, Flanty 5. The one that we've found most useful for deployment and practice right now is like the Flanty 5 XXL model, the 11 billion one. I think it's a good kind of trade off between like reasonably small and not too hard to deploy and like reasonably powerful. We did, I guess, there's a little bit of a switch from mostly using the OpenAI's API before and the reason mostly being that for the switch mostly being that it does get very, I guess, we have something like 250,000 users now.

• 1:26:51Andreas Stuhlmüller?

  It does get very expensive and it's just a lot cheaper to deploy your own thing. I think this space is changing very quickly, so I wouldn't be surprised if we talk to you again in three months and we're like, actually, we switched everything to some other model, so it wouldn't make too much of it.

• 1:27:07Jungwon Byun?

  I think this is generally something that we see, we think we have to be really good at and probably a lot of like language model applications will want to be really good at, is the ability to very efficiently test and deploy and swap in different models, actually because I think there will continue to be so many of them from commercial providers as well as open source providers, but also because the rate at which these model capabilities are changing is moving so quickly that I really want us as an organization to be very good at taking a new model and then figuring out, okay, what new workflows are enabled by this and what is not, what is still missing. And this is something that I really hope like we get really good at as an organization. I think it's like somewhat researchy in nature as well.

• 1:27:47Andreas Stuhlmüller?

  One of the things we haven't been doing too much of, but that we're very excited about is kind of combining, ensembling multiple models. I think especially if you have models that are trained on different data, you get a task and they all agree on the answer. I think you can be a lot more confident that the answer is correct than if you just use one model. So I think often people are like, well, you know, aren't the big labs gonna do everything? And I think that's like maybe one situation where that's not the case and there is room for other organizations to be like, okay, you're trying to kind of combine the results of different models in ways that lead to overall better performance than you could have gotten from any one model.

• 1:28:27Nathan Labenz?

  Yeah, okay, that's really interesting as well. That's another good nugget. So I mean, one thing that I would just highlight here is you have an 11 billion parameter model. Working roughly as well, I guess, as like top of the line models once fine tuned for your particular tasks. How narrowly are you fine tuning those models? Like, do you have like a whole set of different blonde XXL 11Ds or whatever exactly it was?

• 1:28:59Andreas Stuhlmüller?

  No, so it's like fairly broad, although the task still has relatively short answers generally. So the tasks that we've used this for are one illicit has abstract dependent summaries. So if you ask a question in illicit, it tells you not just here's like here are the papers and here's the abstract of each paper. And it also doesn't just say here's like a TLDR for each abstract. It says here's like a TLDR that was computed for your question specifically. So that's one task that we find you the model for. And then another task is you're just question answering from papers like the task we talked about a bunch earlier where given these paragraphs from the paper and this question, tell me concisely what the answer is.

• 1:29:43Andreas Stuhlmüller?

  And I think in both cases, yeah, first of all, I think those are both like not that specific there. I think they apply to lots of papers, but they do share that you kind of want to use a common rubric where the rubric includes things like, is your answer kind of concise and truthful and useful to the reader? And there are a few other entries in the rubric. So that, yeah, that's the rough shape of the training we've been doing.

• 1:30:11Nathan Labenz?

  What is the role of, I mean, it started to sound like a constitutional AI approach there for a second. How are you thinking about incorporating AI into the evaluation and feedback cycle? That definitely seems to be one of the biggest trends in the space right now.

• 1:30:30Andreas Stuhlmüller?

  Yeah, yeah. So, yeah, that's exactly right. That's the constitutional AI approach where I guess for context, so usually in. RLHF, Reinforcement Learning from Human Feedback. You have human annotators. They look at like two options and ask which of these options is better. And so in Constitutional AI, basically what you do is you replace the human annotator with a language model, ideally like as large a language model as you can find, because the evaluation task is pretty tricky. And then you tell the language model, here's the rubric, here are the options, judge the options that are according to the rubric and select the one which is better. This gives you pairwise judgments. Then you can train a reward model on those judgments. And then, once you have the reward model, you can then train like a generation model, in our case, as mentioned,

• 1:31:13Andreas Stuhlmüller?

  95, on kind of getting higher rewards from that reward model. And yeah, that's a process that like our engineer Charlie has been leading and that has been like working very well for us.

• 1:31:27Nathan Labenz?

  You also mentioned like you've used the biggest language model you can find in the evaluation. Obviously, new biggest available language model just dropped in the last week and a half. What are you seeing right now in terms of like the absolute frontier of language models? In other words, like, is GPT-4 changing the game for you at all? Where is it? What are the tasks that, you know, are kind of the ones you wish it could do, but it can't yet do?

• 1:31:57Andreas Stuhlmüller?

  My very brief answer is, I think, yeah, it is a significant step up. I think it still is not great at avoiding hallucination. So I think we keep saying this every time we talk to someone from OpenAI or any of these arcs, it's like, stop making bigger models, just make them hallucinate less. I think that would really make, I think, tying it back to our earlier discussion about kind of error rates and such, I think, if you could just get high robustness, I think actually that would often allow you to do more than if you had a bigger model. So that's a, that's a thing, like if you ask the model, for example, who is Andreas Stuhlmuller? It would, it would like probably give you a description that kind of sounds like me, but not quite, and would have like some made up facts about what universities I went to.

• 1:32:42Andreas Stuhlmüller?

  That's a thing we wish for, we don't have yet. Just the, I guess, T P four nominally also supports image inputs, that sort of thing that is really deployed yet. But I think that will actually be quite transformative, because a lot of Papers are like communicate substantial information in like their figures and their tables. And I think being able to access those will be very useful. So excited to see how that goes.

• 1:33:08Jungwon Byun?

  I think there are just like so many workflows left that we would like to be able to use the models for that we can't. I think one really exciting, I think a lot of it comes down to like having good causal reasoning and like causal reasoning and logical reasoning abilities. One thing that I would really love them to be able to use the models for are like kind of efficient clustering and cutting up of a space. So if you have like a bunch of research ideas, how do you organize them? Or like a bunch of ways you can kind of set your product roadmap? How do you cluster the space? What are actually the like the cruxes and the constraints that kind of segment the space up most efficiently?

• 1:33:45Jungwon Byun?

  And like another one is, I, I think this came up earlier in this conversation, but like the ability to know when to stop and like trace it's when to realize that it's hit a dead end. Part of why our earlier experiments this year with trying to use the models to choose the reason themselves didn't work is because the model was just like really bad at being like, wait, I'm confused. This answer is wrong. I need to go somewhere else. And it just like once stuck on a path, just keeps going down the path. I think that that makes sense as like a next token predictor. It's just going to keep predicting tokens and not be like, stop, stop and go do something else. So I think that, that is like, uh, yeah, another capability that I'm, that I really hope can like, you know, help unlock a lot of good reasoning.

• 1:34:27Nathan Labenz?

  Yeah, I've observed that too. I was also thinking about that recent, uh, like jailbreak, I guess they call it token smuggling where, you know, you can get GPT four to generate some like flagrantly, you know, toxic or whatever kind of content that it ordinarily wouldn't.

• 1:34:47Nathan Labenz?

  If you sort of embed that generation in like a function and you know, it sort of reads the whole thing as code and attempts to like execute the function and it only needs like a few tokens. I almost imagine it as like pulling on a string, you know, pulling a thread out of a sweater or something, you know, and then like, once you've got, once you've like just pulled a little bit of toxic content out of it, next thing you know, you know, the rest of it just kind of unspools. That's like almost, you know, like mechanistic insight there somewhere, I think.

• 1:35:20Jungwon Byun?

  Yeah. The other thing that I think people have already started to notice is that a lot of the reinforcement learning is kind of causing the models to be incredibly uncalibrated and overconfident in their answers. And so that is preventing us from being able to use these models in particular ways. Like, we would really love to be able to use GPT-4 for ranking, but it's just like, it's like, really not helpful for things like that. So I don't know how we're going to kind of like resolve this, this trade-off.

• 1:35:49Nathan Labenz?

  Yeah, they're also not giving the log probes anymore either. So you've got kind of a couple barriers on some of those things. You mentioned earlier, like a headline number of users, but one thing I'd love to hear, if you have any kind of interesting anecdotes or observations on, is how people are using your product. Like, you said a little bit about who they are. Is this something that they come to like from time to time? Do they sort of sit there all day, like rifling through lit reviews? Anything that's like interesting or kind of, you know, surprising about your users' behavior? I'd be really interested to hear about.

• 1:36:25Jungwon Byun?

  Yeah, I think there's a pretty wide spectrum of use cases. So maybe something like 60% of our users are academic researchers, but the rest of them work in, you know, I think tanks and government and finance and consulting. Some of them are just like small business owners, or they work in like kind of, a lot of people actually from like medicine, a lot of clinicians and medical people. There's definitely a core group of people who are using it very regularly. At least 10,000 people use it more than once a week. And some, we took, when we built Alyssa, we took a lot of inspiration from the existing systematic review process. It's a very common process in, mostly in biomedicine. And

• 1:37:04Jungwon Byun?

  Uh, it's yeah, it's like a pretty structured process for synthesizing a lot of research and trying to understand across many different papers, what is the answer and how do we kind of triangulate the results across different papers. So different teams who are kind of working on that are actually using illicit as part of their process to find papers and extract information that they need to kind of work on those types of projects. And then, yeah, I think the other cool thing is like finding researchers like the one that I mentioned earlier, who are stitching together many different tasks and kind of like hacking together his own like workflow automation and all of that.

• 1:37:39Nathan Labenz?

  Uh, when people are hacking your product to get access to things that it can do that you haven't bundled up for them, that's usually a pretty good sign of something. So we mentioned a little bit earlier, just briefly, that you you guys are a nonprofit research company, but you do have some business plans. It seems like, you know, nonprofit AI orgs, developing business plans is also kind of a trend. How are you guys thinking about your future as, you know, like a revenue generating, self sustaining organization, perhaps?

• 1:38:13Jungwon Byun?

  We really want to be able to scale list its impact up quite a lot and be able to have it guide like and and really help with a lot of very high stakes decisions. It's grown a ton over the last year. So we are kind of hitting the limits of what is feasible with philanthropic funding and also want to, you know, just kind of want to be self sustaining as an organization. So, so, yeah, that's like a question that's kind of like actively on our minds. And we're thinking about what is the right vehicle to house illicit in going forward?

• 1:38:46Nathan Labenz?

  I wonder if there's a, you know, of all the like language model first companies I've talked to, you guys remind me most of Wolfram.

• 1:38:54Jungwon Byun?

  Mm hmm.

• 1:38:56Nathan Labenz?

  There's, there's definitely some like shared DNA there of kind of. Highly explicit reasoning.

• 1:39:02Andreas Stuhlmüller?

  Yeah, there's definitely some like shared shared aesthetic in terms of like having like very explicit processes that are like robust and trustworthy. I think there's also important differences in aesthetic. I think Wolfram is like very much open to just building a gajillion widgets and then putting them together. And I think we're probably a bit less excited about that. Yeah, I don't know. I like Wolfram. It's a cool company.

• 1:39:29Nathan Labenz?

  Obviously, you guys are motivated, you know, to a significant degree by AI safety considerations. And you made you made one kind of offhand comment earlier about when you talk to the, you know, the biggest leading labs in the space right now, you say, you don't need to make bigger models, just make them more reliable. How would you characterize the overall state of the AI safety issue landscape, you know, and kind of our prospects? Like, right now, it seems like things are changing really quick. We're getting signals from like OpenAI that they're gonna, you know, try to do something to kind of keep things under control, hopefully. It would be very interesting to see what that turns out to be.

• 1:40:13Nathan Labenz?

  But what's your kind of outlook on the whole scene?

• 1:40:16Jungwon Byun?

  I think my honest answer is that I have just have like a tremendously wide range of what could happen.

• 1:40:23Jungwon Byun?

  That includes like kind of all of the possibilities.

• 1:40:27Jungwon Byun?

  And I just have tremendous uncertainty. I feel like it's just an impossible thing to predict. The things that I do feel more confident about is that I really would prefer for this not to just be a technical solution. At least in the community that we grew up, I think there's a lot of focus on on the technical challenges and there are real technical challenges. I think they would be very high impact. And so it's really important. But I also kind of want to think a little bit about like. How, you know, especially with things like process supervision, one of the things that I think we want to see happen is consumers demanding this type of process supervision and transparency from their AI products. And we don't want to see safety just happening at the large AI labs, but also at these end user applications, I think they will care

• 1:41:09Jungwon Byun?

  internally as they're building their systems about being able to supervise the process and control and debug. And also for their users, enabling a lot of those features. So I think there can be a lot of helpful pressure coming from that side. I think there, yeah, at Upstart, we were trying to use machine learning. My last company, we were using machine learning for credit within finance, which is an incredibly heavily regulated industry. And I think there I got to see both how regulation could really protect consumers and be very good for the world. I was on the marketing side and I was just like not allowed to make any false statements or any statements that were not very heavily supported by by like evidence and data. And so that that seemed like really good for consumers and like people lying about their products and their financial products seems really bad.

• 1:41:53Jungwon Byun?

  But also kind of saw how, just how much regulation lags relative to the pace of innovation. So I really hope there, you know, that's like another kind of mobilizing force on the safety side. That I don't, I don't, I don't, I'm not the expert here. So I don't know how exactly we kind of bridge that gap, but I think it's so important that the regulatory side moves at pace with the technology and also that people have examples of where regulation can be really good for the world, like especially the tech industry. And like be trying to kind of set, be setting a lot of those things up. So yeah, I think that the technical challenges are definitely pretty tough, but I think there, there are a lot of other ways we can be coordinating in different parts of society to make this go as well as possible.

• 1:42:39Andreas Stuhlmüller?

  Yeah, I think I roughly agree. I think also things are very uncertain and up in the air, I think. Partially because in the big scheme of things, I think still not that many people are actually working on these issues, both on alignment and also on kind of making AI go well more broadly construed. I think in our space specifically, which is like using AI to kind of increase wisdom, improve decision making, etc. I think actually basically no one is working on this, which is maybe kind of surprising. I think one of our best hopes to make things go well is probably figure out better ways to use AI to support us as it becomes more capable, help us coordinate better and figure out what kind of good plans look like.

• 1:43:22Andreas Stuhlmüller?

  So I think things are up in the air because it's really a high lever situation. I think individuals can probably still make a less surprising amount of difference by pushing in the right directions. I think there are many signs for for hope. I think big AI labs like all have expressed that they do consider process supervision among like their core alignment bets, which I think is a great sign. I think we haven't really seen that cash out in real investments yet. I think if we, for example, like I don't think we've seen any kind of scaling up of investment in process supervision that looks at all comparable to how much has been invested in scaling up models.

• 1:44:04Andreas Stuhlmüller?

  I think if you did see that, like better tools for writing compositional language model programs, better debuggers, visualizers, compilers, I think there's a lot you could do in principle. And I think people having expressed that they're excited about this should make us somewhat more optimistic. But I think not having seen the kind of investments on the ground yet should still make us feel like, ah, these things could go. We don't quite know yet how things will go. But on more optimistic days like today, I feel like, yeah, probably AI will like help us kind of better see like what the risks are, will help us better coordinate. Probably eventually we'll figure out that we'll have to limit large scale training runs to some extent.

• 1:44:47Andreas Stuhlmüller?

  And AI will help us see more to the extent that that is the right decision. It will help us see that that's the right decision to the extent it's not. It will help us help us see that. So I think I think there is reason for hope.

• 1:44:59Nathan Labenz?

  What Jungwon said at the beginning really resonates with me around just very extreme uncertainty. Your comment too about like some sort of high end, I mean, that seems to be obviously a big. Idea that's kind of circulating in the space is like, is, could we put some sort of, and this, this seems like it could be a really good idea, put some sort of like high-end cap on super large models. Beneath that, there doesn't really seem to be any reason to worry too much. Now, that could change, obviously, with a new architecture, or, you know, some sort of conceptual breakthrough. But at least for now, it seems like that limit could be

• 1:45:38Nathan Labenz?

  pretty high and not really the kind of thing that like inconveniences, you know, PhD students or startups or whatever, but still would like create some decent visibility into, you know, what is going on that might actually have some tail risk associated with it. I'm thinking a lot about that right now as well. Obviously, I'm not, you know, any sort of ultimate decision maker, but to the degree that I can shape the discourse or whatever, that does seem to be a pretty attractive scenario right now that like, again, doesn't hurt people, it doesn't, doesn't constrain most people too badly, which is nice. Like, you guys can, you know, continue to fine tune your blonde XXLs all you want, but, you

• 1:46:21Nathan Labenz?

  know, if somebody's going to go 100x past GPT-4, you know, it seems like society has an interest in getting a tip on that before, you know, before the whole thing kicks off.

• 1:46:32Jungwon Byun?

  Yeah. I, I am encouraged by the fact that I think probably compared to other technological revolutions of the past, a lot of the decision makers in this context have been, have cared about safety from the very beginning. So obviously, there's like some disagreement about what safety should look like and how much, you know, some individual opinions vary, but I think everyone, yeah, I think, I think everyone is trying really hard to be as safe as they can and like think a lot about their, their, the consequences of their actions and trying to be altruistic in that way. So that's, I think that's very good. And if we think about like, I don't know. Tobacco or something like that. I just like, don't think that was the case in the beginning. So.

• 1:47:12Nathan Labenz?

  Yeah, or even just, you know, you go back like the industrial revolution. I don't think, I've often said this, you know, I don't think James Watt had any idea what was gonna happen. There does seem to be just a lot more emphasis on the kind of trying to get ahead of it, you know, than there were in previous things too, right? Or like Gutenberg, I don't think he had any great sense of like, you know, what he was unleashing in that moment. And I know we don't have a great sense of necessarily what we're unleashing either, but at least we have a sense that we're unleashing something. I guess that's, you know, that's one step ahead of, you know, Gutenberg. So yeah, that's a, that's a reason for some optimism anyway.

• 1:47:50Nathan Labenz?

  So guys, this has been fantastic. I really appreciate your time. Thank you for going so long with me and educating me so much along the way. I've got three kind of fun questions that I always end with. You can, you know, go on as long as you want, or you can just give me quick hitters. But number one is, what are the AI products, services, tools, whatever? That you guys are using, you know, obviously aside from your own, that you would recommend to others that they should check out?

• 1:48:23Andreas Stuhlmüller?

  So this might only apply to a small subset of your audience, but I use Emacs a lot and I wrote my own kind of GPT Emacs extension, which is open source. You can search for it, gpt dot E L. I actually use it a lot. I think both for coding and for other tasks. I think it's just really nice to have

• 1:48:42Andreas Stuhlmüller?

  kind of language models accessible in the environment where you do basically all your cognitive work. Yeah, maybe John Wan has some more scalable suggestions, but that's mine.

• 1:48:51Jungwon Byun?

  I've, I've honestly been like really amazed and delighted by the chat GPT deep note combination. Cause I'm not, I don't have much of a programming background, but I've just been able to like do way more cool things with Python with that combination. So yeah, I don't know that it's not an AI, it's, I don't know, I guess it's chat GPT, but I think that's the, that's the thing that I use most regularly. Yeah.

• 1:49:13Nathan Labenz?

  Those are both good answers. One of the things I'm going to read about later today is the new plugins architecture that OpenAI just launched today. And there may be some, you know, really cool new stuff there as well. I'm sure there is. But it is funny, actually. Most people don't have a lot of answers right now. I do keep coming back to that. It's like the, I think the application layer is, you know, maybe kind of a mirage right now. There've been literally like thousands of things launched and, you know, sent out in newsletters or whatever. But the number of things that I actually hear back, like you guys, you know, gave more original, you know, kind of novel, interesting answers than most. Honestly,

• 1:49:58Nathan Labenz?

  it's been just a lot of like chat GPT. Okay, number two. So let's imagine a future scenario where 1 million people have a Neuralink implant. If you get one, you are enabled with thought to text. In other words, you can control your computer purely with thought. Would you be interested in getting one yourselves?

• 1:50:21Jungwon Byun?

  100%. Absolutely. I've already thought about this.

• 1:50:27Jungwon Byun?

  I just have this, I just like have this experience all the time where I'm like, why do I have, I just spend so much time converting my thoughts and typing them into my like note taking system and I don't, I just want it to be automated and then like transferring them to messages to other people. So yeah, I would definitely want this.

• 1:50:44Andreas Stuhlmüller?

  I mean, I would have a few questions about the details of the implant. But I don't know. It seems like if you're paying for it, Nathan, sure, why not?

• 1:51:00Nathan Labenz?

  Yeah, maybe that'll be part of our universal basic intelligence stipend at some point.

• 1:51:07Nathan Labenz?

  So, okay, last one, just zooming out, you know, as far as, as you can, and you've touched on this throughout the conversation, but, you know, zooming out, big picture, rest of the decade, what are your biggest hopes for or positive vision for an AI-enabled future? And then what are your biggest fears of what might come as AI permeates society?

• 1:51:32Jungwon Byun?

  I think we're like culturally, I don't know how much this is just me being in my bubble, but I have the sense that like,

• 1:51:41Jungwon Byun?

  A lot of people have a lot of pessimism, or there's like kind of a sense of wonder that has been lost. And I think it's easy to lose sight of how much about ourselves and our world still remains to be discovered. So I really want tools like illicit and just AI more broadly to be able to help us, you know, not just kind of really amplify existing research efforts, but help us to discover entirely new ways of doing research, discover entirely new research methodologies, entirely new research domains that weren't possible before about like, you know, now using kind of this human artifact of text as a data source, analyze us more intelligently so that we can better understand who we are and how we relate to each other

• 1:52:22Jungwon Byun?

  and what kind of systems we can, we can build to help each person and every group of person flourish more. I'm really excited, like, you know, I have, I've had friends who kind of worked in different layers of government throughout COVID and they tell us about all of the kind of very impactful decisions that were made on complicated email threads. I would really like that to stop. And for like AI systems to be able to generate like very decision relevant, but really, really, really simultaneously rigorous kind of one pagers policy memos and answers to people and very high stakes decisions so that we can just become way more intelligent about how we can govern this world. And then I think on a kind of a somewhat like not illicit related dimension, I feel like there's so much that AI could do for us.

• 1:53:06Jungwon Byun?

  Like, you know, things, things like just being able to translate so much more easily, not, not literally in terms of language, but in terms of like style or, or tone, or trans, like helping us kind of communicate with like, you know, different neuro like atypicals, people or styles or, or something like that. I feel like there's just a lot it might be able to do that makes us more human, you know, and not just kind of like, even though it's, it's, it's a machine, I think there are a lot of ways that it can help us be better humans as well.

• 1:53:37Andreas Stuhlmüller?

  My perhaps idealistic view is that like most disagreements are about beliefs, not values. That like, I think people disagree about like, how big a deal is AI? There's like, is it an existential risk? If so, is it like 90% doom or 10% doom? But I like, I think that's true much more broadly. Is universal basic income good? Is it bad? Probably a lot of these disagreements in terms of like policy and like what should be done are more about people having seen different kinds of evidence and less about people having fundamentally incompatible values. And I think on the optimistic view, AI helps us better see the totality of the evidence. And so be able to better coordinate and be less in the situation where I'm like, well,

• 1:54:21Andreas Stuhlmüller?

  it's like my faction against your faction. I think this is the right thing because I have those values. And more like, well, actually, this relating to the translation comment that Jungwon made also, maybe actually AI can help me understand like, why do you think the thing that you think, what evidence have you seen that I haven't seen? And vice versa. And so I think, I think there is a like very optimistic vision that could happen like this decade or could happen later, where it just becomes much more clear what the right things to do are for a lot of big picture decisions and that could lead to things going quite well.

• 1:54:57Jungwon Byun?

  On the, on the negative side, I think, yeah, I'm quite worried about chaos. Like, I think there, things can get chaotic for a lot of reasons, like perhaps even before and kind of unaligned AI takes off, just the fears around this, like the imagined problems can cause a lot of chaos. I think, you know, like the kind of generative models, the classic fear of like generative models creating disinformation or just like creating a lot of stuff. I feel like we already have so much noise that we need to sort through already in our current society. And I worry that the noise will increase without the signal or the ability to synthesize that information, those capabilities increasing.

• 1:55:41Jungwon Byun?

  So I think, I think we really need to keep up with, like, keep up the ability to make sense of the world as it gets more and more complicated.

• 1:55:51Andreas Stuhlmüller?

  Thanks, Nathan.

• 1:55:52Jungwon Byun?

  Great to chat, Nathan.

• 1:55:54Speaker 02

  Omnikie uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omnikie so much that I invested in it, and I recommend you use it too. Use Cogrev to get a 10% discount.