Radically Better Reasoning: Elicit's Andreas Stuhlmüller & Jungwon Byun on World Models for Research

Andreas Stuhlmüller and Jungwon Byun return to discuss how Elicit is building trusted reasoning workflows for scientific research as frontier models grow more powerful but less transparent. They explain process supervision, domain-specific reasoning primitives, and world models that make evidence, causality, and counterfactuals more inspectable. The conversation also covers life sciences use cases, evaluating conflicting evidence, automated software engineering at Elicit, token costs, Gemini, and why legible reasoning may still beat neuralese. Mercury: Command is Mercury’s new conversational interface, giving you natural-language access to your finances and helping you take actions within your existing permissions and approval policies. Visit https://mercury.com to learn more and apply online in minutes. LINKS: Elicit Research Platform Andreas Stuhlmüller Personal Site Jungwon Byun X Profile Ought Research Organization Elicit Founders Previous Episode GPT-4 Technical Report Monitoring Reasoning Models Paper Ought ICE GitHub Repository Hard-to-Verify Tasks Essay Karpathy LLM Wiki Gist Obsidian Knowledge Base App Mixpanel Analytics Platform Amplitude Analytics Platform Anthropic Tracing Thoughts Research Claude AI Chat Assistant METR Long Tasks Measurement Pi Agent Scaffold Repository Personal AI Infrastructure Repository Elicit Claude Opus Evaluation Elicit API Documentation METR Developer Productivity Study Elicit Planning Is Unsolved Rich Sutton Bitter Lesson Meta Llama AI Models Recursive San Francisco Event zero.xyz Agent Tool Access Anthropic Dynamic Workflows Coverage Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm excited to welcome back Andreas Stolmar and Jungwon Byun, co-founders of Illicit, the AI platform for scientific research that's on a mission to radically improve the quality of reasoning that supports high-stakes decisions. Illicit was founded on the belief that process supervision, where models are evaluated and rewarded for the quality of their step-by-step reasoning, rather than just their final answer, would improve the consistency, reliability, and legibility of AI workflows. Of course, with the rise of reasoning models, which can do much larger and more challenging tasks, but generally hide their chain of thought from users, Illicit faced a challenge:

• 0:44Nathan Labenz?

  how to harness the power of frontier models while still ensuring that famously unwieldy LLMs actually do what they're supposed to do? Their answer, as you'll hear, is an interesting synthesis. By creating a DSL, or a domain-specific language, that defines reasoning primitives, which they can then deliver and optimize as discrete reasoning microservices, they allow frontier reasoners to dynamically create structured workflows that are then guaranteed to run as defined. Today, they work with seven of the top 20 life sciences companies, supporting everything from the ranking of candidate drug targets, to the defense of drug launch and pricing decisions for regulators and payers.

• 1:25Nathan Labenz?

  And now, the frontier is shifting to external world models: structured representations, which can take a variety of forms, that make a model's understanding of complicated bodies of evidence as explicit and self-consistent as possible, with the goal of supporting reliable causal and counterfactual analysis. As Andreas puts it, "These world models are a form of continual learning that humans or other AIs can inspect and understand."

• 1:53Nathan Labenz?

  Of course, we cover a lot of important details along the way. From the reasons that they believe that LLMs are still too easy to push around to serve as reliable decision support tools on their own. How illicit thinks about evaluating the source and quality of new and at times contradictory evidence. The promise of certificates of reasoning that would prove that the appropriate reasoning steps were in fact carried out as intended. How illicit is automating their own work with a system that they call the line, which now delivers 30 to 50 code changes per week. And their goal of getting this system running well enough that the company continues to make progress during the humans' year-end vacation. Plus, how much they're spending on tokens as a company and individually. Where Gemini fits into their stack.

• 2:36Nathan Labenz?

  And why they are optimistic that legible reasoning will win out over neural ease in the end. Andreas and Jiang Wan are really exceptional at making time to zoom out and consider the big picture, even as they run their company day to day. And their hope and bet is that if we prioritize truth-seeking now, we may be able to create a positive feedback loop in which better reasoning begets better reasoning, and that this could still happen in time to steer the singularity in a positive direction. Very few for-profit teams have been as consistent and disciplined in pursuit of their mission, despite the rapidly changing AI landscape. And so, for many reasons, I hope they are right and successful.

• 3:19Nathan Labenz?

  With that, I hope you enjoy this conversation about how fluid intelligence can orchestrate trusted reasoning workflows, and why continual learning might be best instantiated outside of the model weights. With Andreas Stolmacher and Jiang Wan Bian, co-founders of illicit. The cognitive revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances.

• 3:48Nathan Labenz?

  I've wired AI into nearly every corner of my life. My email, my messages, my calendar. I even gave Mercury virtual cards to my agents with low limits and category and merchant restrictions for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real time, up to date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error prone to be worth it. And that's why Mercury's new conversational interface, Command, is such a big deal.

• 4:29Nathan Labenz?

  It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions, too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026. And so I invite you to join me in the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC insured bank.

• 5:13Nathan Labenz?

  Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting the cognitive revolution. And now, on with the show.

• 5:26Nathan Labenz?

  Jung Hwan Byun and Andreas Stuhlmeuler, co-founders of Elicit. Welcome back to The Cognitive Revolution. Thank you, Nathan. To be here. It has been two years, guys, since our last podcast. Time flies. We've had a couple of opportunities to talk offline, but two years is an eternity in the AI space. And so I'm really excited to catch up on everything you guys have built and what you've learned and the sense that you're making of the rapidly changing AI landscape and how you can put a little positive nudge into history. Maybe for starters, let's just do a quick little review of the motivating mission that you guys started with around elevating

• 6:11Nathan Labenz?

  Reasoning and the quality of decision making. And then with that context, we can go into brief history of the last couple of years. Because I think you guys were on this trend even before there were such things as reasoning models. And I think the reasoning models are not exactly what you had in mind, but obviously very relevant to where we are today. So kick it off with a little reminder of the motivation behind AOT and Illicit.

• 6:37Andreas Stuhlmüller?

  Yeah, so our mission is to radically improve the quality of reasoning, especially for high stakes decisions. And as you said, we've been at it for a while. We started nonprofit AOT, we're working at Illicit and AI research company now. And wow, what a decade it's been the last two years. So I was just going a few days ago, I was going back over kind of my PhD from like a long time ago in machine learning. I was like, what would it look like to redo some of that work now? And so I went to Illicit and I was like, hey, let me just upload one of my workshop papers where I know I didn't do a good job.

• 7:17Andreas Stuhlmüller?

  And I redid like some of the computational experiments in Illicit. I was like, hey, do the data analysis, I was running the cloud and like 10 minutes later, it had redone the paper. It was like, it was mind blowing. And so if you think back two years ago, when we talked, I think we had launched paper search and summarization in Illicit. And then over the course of maybe the next year or so, had launched systematic lit review, kind of the fixed like search, summarize, screen, extract, write, flow. And now we have these like much more flexible research agents. So it's obviously been an incredible, incredible trend from very short tasks, short time horizons,

• 7:59Andreas Stuhlmüller?

  low flexibility, to agents that do much more extensive research, long time horizons, quite a lot of flexibility. So excited, excited to review what this all means and where it's all going.

• 8:12Nathan Labenz?

  Could you give me a double click into what exactly you had in mind when you were talking about how you were going to improve the quality of reasoning, how that compares to the more and more RL on top of a chain of thought paradigm that everybody now is very familiar with, and then how that has maybe enabled what you're doing with illicit, maybe in some ways competed with it, and how it has shaped your business and product strategy to go in this more principled, a little less bitter pilled direction with what you're trying to build?

• 8:47Andreas Stuhlmüller?

  I like bitter pilled. Yeah, so a lot, last time we talked, we talked a lot about process supervision, right? How do you know that a system is doing good research for you? You can either like look at the output and be like, that looks good to me, or you can look at the process it went through, you know, what papers did it look at, why did it look at them, how did it choose what to do? And back in the day, when we were young and naive, we all thought, you know, it would be better to know that things are correct for the right reasons, that the process was good. And so we leaned a lot into that. And in many ways, I think the justification for that has still borne out. So let me talk about like a quick anecdotal experiment we ran, I think like just two

• 9:26Andreas Stuhlmüller?

  days ago, where we told a few research agents, hey, like look through and analyze about 100 papers on toxicology risk for a particular type of cancer drug. And then, so we tell that to Claude, we tell that to ChatGPT, tell that to illicit, and then we ask, hey, how many papers did you actually analyze? And then, as the models like to do, they're like, you know, that's a fair and important question to ask. Let me be direct. I did not analyze 100 papers. You're right to pushback. I didn't do it. And that, I think that is like a failure of process in many ways, because you're like, I told you what to do, you didn't do it. And so I think the And the reason for that is because the models are not trained on process.

• 10:10Andreas Stuhlmüller?

  The models are trained to produce outputs that were, if you didn't check and you just looked at the result that said, hey, here's my analysis, you wouldn't have caught that. I think the fundamental problem still exists. And then the question is like, what do we do about it? And to what extent does the solution look like better checking of the outputs or checking of the process in various ways? I actually think that this question is still open. I can speak for what illicit does and maybe briefly for what the rest of the ecosystem does. illicit addresses this by, we have like a little kind of domain specific language that the research agent can write, that orchestrates other calls to agents, screen all these papers, then extract data from all these papers.

• 10:52Andreas Stuhlmüller?

  And it runs that you run the process, you know, the process does what it said it was going to do. I think actually the model companies are going a little bit in that direction too. I think I haven't been following it extremely closely, but I think on topic recently launched like a workflows feature that has similar properties. And so even though I think on some level, yeah, all the models are outcome trained. And that's why we see these like quite ridiculous artifacts of most being like, oh, sorry, I didn't do it. I think the problem still exists and like one level up from that, people are trying to patch it. That's maybe that's where I see ourselves as being at in this game.

• 11:30Jungwon Byun?

  Yeah, I think it's worth emphasizing that a little bit more because it's a core part of how illicit is built differently. And therefore what you can use illicit for a lot of, like you said, it's been two years since we chatted. A lot has happened in that time. A big part of what we've been working on for the last year is just rebuilding illicit on top of this much more agentic infrastructure. We started working that on that in about March or April of 2025. In retrospect, it was maybe a little bit early, but at the time it felt quite late because obviously a lot of advances happened over the last two, three quarters. But a big part of what we spent the last year thinking about was designing, how do we preserve these benefits of transparency, systematicity at scale?

• 12:12Jungwon Byun?

  Without losing the flexibility and raw power of these models. I think that's the core design question we've always wrestled with because when you want to deploy these models at scale for really high stakes decisions, you need to be able to, you need them to behave in a certain way, which is often contrary to their kind of fuzzy nature. But you don't want to be overly deterministic because then you run into the bitter lesson issue, right? So like threading the needle is what we're always struggling with. So a lot of our time last year was spent on this kind of technical design question. And we decided to design our own programming language to solve this problem so that the models could run reasoning computation, be able to call these reasoning primitives at scale in a more trustworthy way.

• 12:56Jungwon Byun?

  And what we were trying to accomplish for our end users was the ability to say, you can run this process with the model over 10,000 objects, 10,000 documents, 10,000 drugs, 10,000 targets, genes, whatever. And the same process will be applied to number five as number 9,999. And there's just no other model that can make that guarantee. And unfortunately, there are lots of models that claim that they have done that or can do that and are just completely wrong about it. So that when we think about who are our users, what, how do we build a differentiated product that really meets their needs? What use cases can we enable? We are powering people who want to be able to rigorously synthesize evidence at a very

• 13:37Jungwon Byun?

  large scale and get every single thing right, all the way to the nth degree. That's a very different interaction than just riffing with a model. And so that's, and we support some of these lightweight use cases too. But I think there it's almost more like getting to parity. And I think where illicit's really differentiated is that trust at scale.

• 13:56Nathan Labenz?

  Yeah, I should always remind myself and be clear that none of these things are really true binaries in the sense that it would be wrong to characterize illicit as being not bitter pill because I remember last time, one of your big Principles was, how can we allow you to spend more money to buy more compute to get better results? And it's, that's almost a restatement of the bitter lesson in some way. And at the same time, the frontier companies, as you said, are doing some of this. And I assume that in their research agents, in particular, their deep research products, they are presumably doing some sort of at least rubric-based reward on the final reports

• 14:37Nathan Labenz?

  that the models are outputting, albeit, if I understand correctly, still at least intending to avoid putting any optimization pressure on the chain of thought. There's always little aspects of gray to this. How would you say the, so who are the customers now? There's, there are these deep research things. I use those pretty frequently. Who is the sweet spot that is like, a deep research agent isn't enough for me. I really want to go way bigger, way more systematic, be very sure that I'm performing the same analysis in a way that I can count on. Who are those customers now that you're finding product market fit with?

• 15:21Jungwon Byun?

  Yeah, there's, there's, I think of it as maybe a funnel. So the largest group of users are academics, individuals, people like yourselves who want, value the kind of deep research function. They generally want a fast but robust synthesis of the literature or of the evidence. And where Elisa really differentiates there is it just, it's a bit more technical by default. So it's not really assuming a lay audience as much. And most importantly, everything is well cited. So by default, you can get the models to cite things if you really push them to. And then sometimes they realize they were referencing sources that are not, that didn't exist, which is unfortunate. So they still make those mistakes. Elisa doesn't do that. Elisa just assumes every single claim needs at least one multiple citations from vetted

• 16:06Jungwon Byun?

  databases and data sources in the academic literature. So a lot of people still find value in just that kind of core offering. Then there are academics and researchers who, like you said, want to get a lot more systematic. They want to do a systematic literature review where they're really thoughtful about what evidence should I be looking at? What information out there is relevant to my research? What should I be excluding? How do I apply the same process to all the papers? Extract detailed data from charts and figures and then synthesize all that and weight the findings by quality? Or do a landscaping of the market or the research in a similar way? And then increasingly, one of the other things we've really been developing over the last year is our life sciences motion and playbook.

• 16:49Jungwon Byun?

  So we work now formally with, I think, seven of the top 20 life sciences companies. And we work pretty much across the entire development lifecycle. I would say our biggest concentration is in early-stage research. So working with discovery biologists, some of those toxicologists that Andreas mentioned, as well as researchers in the kind of late and post-clinical stages in commercial and medical teams. The early-stage researchers are often, they are the ones that have a little bit more iterative processes. They're like, I have this idea for an experiment. I'm in immunology. I want to figure out a way to repress these kind of rogue T-cells. What's the right mechanism to do that? And then getting ideas, finding other experiments that have been run, trying to reproduce those experiments. They also are often systematic.

• 17:34Jungwon Byun?

  So they're the ones that want to apply a reasoning process over thousands of genes and targets and do a tournament-style ranking. And then on the commercial and medical side, at this point, often you've had validated phase-two, phase-three trials. They're starting to think about what's the launch strategy of a drug? Exactly which populations do we go after? Who pays for this drug? What are the other kind of alternatives out there? How much more compelling or cost-effective is our product? They have to be very evidence-based in every claim that they make and in justifying to regulators or payers exactly why this drug makes a difference in the world. So that's where, within life sciences, we're seeing a lot of pull.

• 18:15Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:19Nathan Labenz?

  Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts, going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders.

• 19:04Nathan Labenz?

  And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

• 19:50Nathan Labenz?

  Something really top of mind for me that you kind of touched on there is being evidence-based. And I didn't kind of the definition of that, especially both at the early-stage literature review, but especially as you get later, when you start to think about markets and that sort of thing. It strikes me that

• 20:10Nathan Labenz?

  Perhaps in many cases, the best information is not necessarily in an academic paper, or at least if I was naively thinking about approaching such a problem, I would think I would want to cast a pretty wide net. I also experienced this a little bit myself. Fortunately, my son, as regular listeners know, is pretty much cured from his cancer and we're getting back to normal life and all that's great. And I didn't have to go down the path too deeply to really try to get synthesis of what would the best second line treatment be for him. But when I was contemplating that, in talking to doctors about it, I found there was a real narrow box drawn around what kind of information

• 20:53Nathan Labenz?

  would be considered. Our oncologist, who I probably quite highly have and have a good relationship with, will say, there's no evidence for that, for things where I'm like, there's definitely some evidence, right? It's just not this sort of gold standard, RCT, peer reviewed, yada, yada, yada. And obviously, that stuff is great where you can get it. But all this is a buildup to ask, and it may be different for different kinds of analyses. How do you think about like, how wide the net should be as you go out and look for evidence? When is it appropriate to truly limit yourself to published academic work? When is it appropriate to consider blog posts or even insightful tweets that somebody

• 21:37Nathan Labenz?

  has put out? How do you guys think about that? Does it vary by use case? How do you think about weighting these things and calibrating to the level of credibility that they should have? That that seems to me an opportunity for honestly, just major improvement, even at the clinical level, because there's so much blind eye turning, it seems to me today, to evidence, just because it's not right at the the very highest level of quality or credibility.

• 22:03Jungwon Byun?

  Yeah, that's right. I think the core question here is how do you? Differentiate and discriminate against evidence based on evidence quality? And how do you identify when a certain level of evidence or quality is relevant or the best thing for your decision? And so even within published literature, there's a gradation, right? There are more cited papers.

• 22:25Andreas Stuhlmüller?

  Important to remember, yes.

• 22:26Jungwon Byun?

  Yeah, there are high-impact journals, there are well-regarded authors and institutions. And so this is a question that we've wrestled with for a long time because the way humans have solved it is there are these very lossy proxies, citation counts, journal impact factor, I know the guy, right? And those are rough approximations, but deeply imperfect. And there are so many examples of groundbreaking research that was not published in the top journal, like one of the foundational CRISPR papers, for example, was actually published in a tier two or lower-tier journal. And I think the promise of language models is that, yes, we can continue to use these heuristics that have been helpful, but also we can think a little bit more from first principles about the quality of work and its appropriateness for the decision at hand.

• 23:11Jungwon Byun?

  So that's what Alyssa has always really tried to do, and not just blindly rely on citations, but give the researcher the chance to say, okay, for this research project, I'm going to look at case studies, or I'm not. Case studies are not good enough in my research domain because there are plenty of RCs. Or I'm going to look at studies of a sample size of 10, or I'm just going to look for a sample size of 1,000 or more, because depending on whether you're working in rare disease or oncology, the shape of the research that's available, like you said, is different, right? So I think part of what one principle we have is like, how do we look at the actual methodology and the content of the research, not just at metadata, to determine quality? How do we equip the researcher to express their own judgment and expertise based on what they know about the domain and what is specific for their project? Because I do think it is specific to the domain.

• 23:57Jungwon Byun?

  As we've built out the research agent, we have started to bring in a lot of other data sources, web sources, like company filings and things like that. Because like you said, it's there's a lot of information that's not just in published papers. Publication takes a long time, so a lot of recent things are not going to be there. And sometimes and research is very interdisciplinary and holistic. So when you make a scientific decision, you want to consider the science, but you also want to consider the kind of commercial implications of that as well, or the policy implications. So we have introduced a lot of data sources. And we do, I think, still the agent relatively has a fuzzy understanding of which sources are higher quality. But I think we want to kind of structure that a bit more. And again, the user has a lot of control in how they can express what they believe quality means. But I think we do want to also be a bit more opinionated

• 24:42Jungwon Byun?

  for researchers that are not experts. And then the last thing we're thinking about is how do we express the confidence level of claims? So not just like at a source level, say, this is a high quality paper. This is not, this is a high quality blog post. This is not. But when illicit synthesizes all this information for you, how can we say this treatment works? That claim, how well supported is that claim? Is that you don't need to worry about it? Like 99% likely to be true? Or is it like there's conflicting results in the evidence? So we've been thinking a lot more about breaking insights down into claims and then giving people kind of confidence level so that they know what to do with that. I think that's another important part. It's like, it's not about, it's not binary. It's like, how can I use it? That's the part that's really important.

• 25:23Nathan Labenz?

  How good are models at that today? I always think back to the original GPT-4 model card, where they showed the base model calibration being pretty good, where it, when asked to express confidence in claims, the, when it said 20% likely, it was like actually pretty close to 20% of the time that it was correct. And honestly, to a pretty remarkable degree, the base model was reasonably well calibrated. I understand that pretty much just came out of pre-training as a emergent property. But then with reinforcement learning, you have mode collapse, which I'm going to overstate the importance of mode collapse, which I do think was like.

• 26:09Nathan Labenz?

  One wave of AI denialism for a minute there. But there was some form of mode collapse where all of a sudden you're like getting much less well calibrated self assessments from the model in terms of how confident it was that the claims were accurate. Obviously, since then, we've had a tremendous amount of additional RL applied on top of models. I haven't seen those kind of calibration studies as recently. I could believe that maybe they've gotten better because maybe that's part of a rubric that's being used. I could imagine it's even gotten worse because just left to its own devices, RL maybe takes you the other direction. What do you guys see in terms of how good they are natively?

• 26:54Nathan Labenz?

  Do you, can you prompt your way to good performance on these sort of confidence questions? Do you have to fine tune? Do you want, if they're not that great, do you want the frontier models to get better? Would you advise them on how they should think about getting better at that? Tell me everything.

• 27:10Andreas Stuhlmüller?

  Yeah. So first, I think the token probabilities are just useless now, so we can pretty much ignore them. And so when we talk about calibration, mostly we have to talk about the verbalized calibration, right? If you ask the model, how confident are you in this thing? Are you confident? And I think that is probably much better now than if we compare the, what was it, 2B3, 2B4 token props against the verbalized probabilities now. I would rather take the verbalized probabilities, to be honest. I think those will probably do a better job at capturing for complex situations what the assigned probability should be. I still think the models right now are easy to push around. And like, that is one of the main things we are trying to address with better scaffolding

• 27:55Andreas Stuhlmüller?

  is right now, often you ask the model, hey, how likely is it that this kind of trial will fail, say? And the model is going to say something like 30%. Then you're like, but have you considered that? You can almost say anything. Have you considered that on average, kind of go trials like fail X percent of the time? And it'll be like, oh, no, you're right. Actually, you know, it's a, this probably should have been higher. Or have you considered that in this particular case, actually, this molecule has like pretty little research behind it? And I'll be like, oh, yeah, you're right. And the models are getting better at it, but I think fundamentally, right now. These probabilities are very unstable compared to when you talk to an expert. You know, you can throw these things at them and mostly they'll be like, yeah, you know, I've considered it.

• 28:39Andreas Stuhlmüller?

  That doesn't really change my view here. So a big question is like, how do you get kind of stable probabilities out of the models? Which isn't just they say they should never update. They should obviously, if you present truly novel information to them, they should update. But right now, they do, it does not feel like they have a coherent world model behind a lot of the probabilities that they express to you.

• 29:02Nathan Labenz?

  How, when you say they're easy to push around, like, how easy are they exactly? Are we talking like even seemingly semantically equivalent rephrasings get big differences in outputs? Or is it a sort of more conceptual difference in framing is required? Or you throw in red herring fact and it can move the needle? Just how blowing in the wind are they?

• 29:30Andreas Stuhlmüller?

  It's hard to characterize. I think it doesn't maybe part of the issue is it's hard to predict what will push them around exactly. And I think if I could give a better answer to this question, I think I could, would also have a easier time fixing it. I think it, like, I don't think it's like extremely systematic right now.

• 29:50Jungwon Byun?

  But maybe that's, that's partially what we've been trying to invest a lot more in is, and I think that's the next phase of our role as humans. And what humans have to do is work through these really gnarly evaluation problems. I very much believe in the next few years, generation will more or less be a solved problem. Historically, humans have done all the generation. They've done all the work. You open up a blank document, you put your thoughts in, you start from nothing, you create. And that's just not how work is going to happen anymore. Increasingly over time, AI is going to take the first pass at everything. And then the work left to humans will be around evaluating that. Was it the right thing to do? Was it done well? Can I use it for this use case? Can I trust it? And I think people are worried about kind of job loss.

• 30:35Jungwon Byun?

  But I do think there's like a major job transition opportunity or skill transition opportunity where we now, what's, what we need to do is think about. What does good look like? And what are these failure modes that AI systems can run into? How do we start documenting them? How do we start codifying best practices? Getting good examples to point models at? How do we start building better verifiers? There's actually a lot that needs to happen because we don't even know, we can't even articulate for ourselves what good looks like, or why this is a bad thing, or how often this happens. So I feel like there's a lot of work around that. And certainly, we are trying to make that transition as a company. I think there needs to be more infrastructure built around evaluation and articulating what good looks like so that we can point these models towards them.

• 31:16Nathan Labenz?

  How do you get interacts with chain of thought? And I was just, this is very top of mind. I keep talking about it, but I, not too long ago, attended this event in San Francisco called Recursive, which was all about the potentially soon coming phenomenon of recursive self-improvement. A sort of scary, striking takeaway from that event and all the conversations and presentations there was, we're really heavily, and I would say like, problematically indexed to chain of thought monitoring. Like, it's chain of thought monitoring kind of all the way down in terms of the plan for how we're going to keep a recursive self-improvement process on the rails.

• 32:02Nathan Labenz?

  And so there's like this very strong sense that we've got to maintain freedom for the models in the chain of thought so that we can monitor it. Because if we do apply pressure per the obfuscated reward hacking paper, we'll drive the bad behavior underground and then we'll be doubly worse off because we'll still potentially get the bad behavior and we won't see that it's thinking about it.

• 32:24Nathan Labenz?

  How do we square that? Is it, maybe, I guess, the naive answer would be maybe you just say, you can think whatever you want in the chain of thought, but I still want like a systematic account, a systematic reasoning trace in the actual final output. And then I just reward that. And maybe that works. But maybe you have a more nuanced view on how companies should think about.

• 32:50Nathan Labenz?

  Where and what kind and how much pressure to apply to the reasoning process that builds up to the final answers that models give?

• 33:01Jungwon Byun?

  I can give a high level take and then I'm sure Andreas will have more technical response, but one of the projects we worked on in OT that, like many OT projects, was a bit early for its time, was we probably built the first language model observability tool. We built this thing called ICE. It stood for Interactive Composition Explorer. And we had this, this was, when was this? This must have been 2021 or so. And yeah, we anticipated this problem even five years ago, that at some point, language model traces would get so large, we would not be able to debug them for ourselves. How would we visualize that? How would we maintain oversight? How might we audit that? And so we built this visualization. And one of the things we learned was, the best way to troubleshoot

• 33:45Jungwon Byun?

  is not necessarily to chronologically go through all the steps that the model took. That actually evaluation, so repeating the generation process is one way to check what happened and to build trust in it, but it's inefficient. It doesn't really scale and it's a very difficult way to check. And so actually, what you need often are like a different layer of reasoning checks, almost like logical consistency checks, which are not, let me go through everything you did, steps 1, 2, 3, 4, 5, and see if it was correct. But now let me think about how, for example, like a sensitivity analysis, like how sensitive are my findings to different changes in input parameters? Logical consistency checks, things like that. And so that's where I think we need a lot more investment in infrastructures and building kind of independent checks that don't rely just on checking chain of thought

• 34:30Jungwon Byun?

  monitoring. That's my high level take. But yeah, I'm sure you have a more technical version of that answer.

• 34:36Andreas Stuhlmüller?

  Yeah, let me maybe first restate part of what you said in Different language. So I think you can either check the process or you can check the outcome. And those are your two options, right? And then like when you're checking the outcome, you still want the outcome to somehow contain a certificate that the right reasoning was done. And like that, what can that certificate be, right? It can be like, here are, here's how my conclusion would change if I had a different input. Here's like the things I looked at on the way. Here are like citations to the literature. I do think the, this is a very underdeveloped field in my mind. So it's, in mathematics, it's very developed, right? You can have your formal proof if you want, and that proof is checkable.

• 35:19Andreas Stuhlmüller?

  I don't think it's very developed in more kind of fuzzy domains and probably partially because would just be too much work for humans to produce legible certificates. And humans don't even like really have great ability to introspect on their own thoughts. But I think in principle, you could, even if you didn't supervise the process, you could produce certificates of reasoning that then let you check the reasoning, even if you didn't check the process that generated that outcome. So I'd be very excited to see more work in that direction. The other clarification I wanted to make is, I think there's, it's worth distinguishing like chain of thought and like kind of the reasoning process or the chain of process or whatever you want to call it.

• 36:02Andreas Stuhlmüller?

  Because the chain of thought, people often think about the, what is, what are the thoughts the model writes down, like the reasoning tokens? And then the other, those, how much can you trust them? I don't know. We don't, I think for OpenAI, we don't even get them anymore these days unless maybe you apply for a special permit to see them. But you do see the tool calls. And I think the tool calls actually are important reasoning facts in and of themselves. Take, like, for example, let's say I use a model to ask about, like, I download a new paper from archive, the model hasn't seen it before, and ask it some question, ask the model, hey, you know, what are the key results here? And now if the model, I can see which parts of the paper is the model reading, because

• 36:45Andreas Stuhlmüller?

  it, you know, it has its read tool that maybe reads by default the first few lines. And then it can scan other parts of the paper. And so sometimes I know the model didn't even look at the methodology section. And that that is part of its reasoning that is checkable. I know if it now tells me something about the conclusions of the paper that really should have relied on the methodology section. I know it didn't do that. That's obviously like a very simple example. These papers are small and so on, but the same thing applies at much larger scale. Uh, where like you can actually, the process still remains an important way to checkable. Because you do see the tool calls and the tool calls are an important input into the model's reasoning.

• 37:27Nathan Labenz?

  Yeah, I like that. How much of this do you think companies are doing today? It's gotta be some, but obviously they're not telling us. Is your, on the spectrum from, you know, closing in on the sort of thinking that you're doing and the granularity of process supervision that you would like to see on the one end to purely RLVR with the, did you get the right answer in the final answer box or not? Just binary signal. How much of this do you think they are doing based on in model behavior?

• 38:01Andreas Stuhlmüller?

  And these are obviously speculations. I think they do a lot. I think we know they do a lot of kind of rubrics on the final output. Like was the, even like reasoning adjacent rubrics. Did the model produce something that looked like expert reasoning in many ways? I don't think they do a lot of evaluation of the process. As for, again, I could be wrong, but um, I think there is a really interesting question, which is, if you had all the details of the process, let's say you are inside a lab and you ask yourself, you know, X and T, would you expect this process to lead to the right answer? Like, not like after the fact, if you check it against the answer, was it correct? But uh, did it follow the sort of process that if you're making a like a forecast, for example,

• 38:45Andreas Stuhlmüller?

  and you did, don't see, let's say you don't even see what forecast the model ends up. You just look at what things did it consider? What data sources did it consider? What did it write about the different hypotheses it considers?

• 38:58Andreas Stuhlmüller?

  I think there's a really interesting project of thinking about like, to what extent are the models following a process you should expect to be good? And in my limited knowledge of what's going on, I don't think this is much work is being invested in that.

• 39:14Nathan Labenz?

  So the, I guess one implication of that for your work would be.

• 39:20Nathan Labenz?

  You might expect the model, the next generations of the model to eat less of your scaffolding than, than it might eat other types of scaffolding, right? This kind of general pattern of like, people build that scaffolding to compensate for the model's weaknesses, then the model companies take that feedback and train on it and the next generation, you know, you could clean out or eliminate a decent amount of that scaffolding. It seems like you think that maybe on the agentic side of go accomplish this project, get over these humps, whatever, you would expect more of that kind of scaffolding to be eaten in the next version than the sort of trust and reasoning scaffolding that you're building.

• 40:04Andreas Stuhlmüller?

  Yeah, these things are hard to predict. I do think the, maybe one way to divide up the types of questions that people ask the models are like, questions where, once you see the output, it's like easy to check, like proof of existence type questions, you know, can you write a program that passes some tests, for example, or replicates behavior of an existing program? And then there are questions where you can't do that, where, for example, let's say you want to know, is there any paper in the literature that refutes this claim? And again, just getting back to the certificates point from earlier, the model can't exhibit to you, hey, you know, like, here's a short explanation for like, what, you know, why

• 40:47Andreas Stuhlmüller?

  there is no such claim, like, the proof is in, in the process in a way. The model has to prove to you, hey, like, I look, I followed this strategy to find, if, such that if there was a paper that refuted this claim, I would have found it and here's the proof that I did that. And I expect those sorts of questions to like, remain a little bit more tricky for the models for a while, but we'll see, we'll see.

• 41:13Nathan Labenz?

  Yeah, that reminds me of something that you recently wrote around, Elise's focus being on.

• 41:21Nathan Labenz?

  Becoming the best at reducing these big, kind of fuzzy, hard to verify tasks to sets or graphs, perhaps, of easy to verify tasks. And I'd love to hear a little bit more, you alluded to it there, but I'd love to hear a little bit more about like, how you're going about that? And also, like, how complete you think that process can be? I guess everywhere I look these days, I feel like I see the same shape of a really interesting question that I don't know what to make of, which is like, basically, how do we get high-level guarantees or conclusions or insights from low-level steps, right?

• 42:05Nathan Labenz?

  And this is like, in biology, I might be able to say, I've got all these proteins, or these genes are being expressed at this level, but I know what's going to happen next at the cell, or the tissue, or the organism level? No, right? I don't. And similarly, it seems with these kind of big judgment calls, should I prioritize this drug or that drug? We can break it down and get systematic. But it's not clear to me if there's, if it kind of, but how close it gets to something where I'm like, yes, okay, I can really buy in and trust that versus, is there something that kind of sits above all those steps still? That is, is it emergent? Or is it just somehow lost?

• 42:46Nathan Labenz?

  Or people are doing some sort of meta-cognitive work that's hard to capture, that is maybe still like, very critical to actually being effective at these tasks? So I guess, again, tell me everything, but I'm really struggling a lot with this, like, formal methods, formal verification is another area where I see this, where we make all these low-level guarantees about this error, that error can't happen, but is the system itself going to behave how we wanted it to? I still don't know, in a lot of cases, how we make that leap. So very interested in your take on these sort of. I usually think of it as like laddering up low level things to high level conclusions, where you're purging it the other, from the other direction, which is interesting unto itself. Yeah, again, tell me everything.

• 43:29Andreas Stuhlmüller?

  Yeah, maybe, I guess, to quickly restate, what is the, why do we want to reduce hard to verify tasks to easy to verify tasks? Because AI currently can be trained on easy to verify tasks. We know it's like extremely good at LVR coding, it's math tasks like this. And it's quite, I would say, like quite weak at a lot of fuzzy tasks. So when I, I notice it all the time when I try to use the models to help me in like plan our company strategy, for example. I think they're surprisingly useless. So even though they have access to all my contacts, they're really quite good at being like, you know, let me put, pull in the data, let me pull in your email, your Slack. I still find that I can't, they don't get it.

• 44:14Andreas Stuhlmüller?

  Or like in an important way, this is related to what we said earlier about like they're too easy to push around. It doesn't feel like they're building up a coherent model of what's going on. And I think an important reason for that is that is a hard to check task. Then, okay, what to do? I think it depends a little bit on like what your situation is, whether you're trying to create a reward signal that you can train the models on, or whether you're trying to kind of do verification and checking for the purpose of understanding whether an already trained model can be trusted in a situation, or how to refine its behavior. I think if you're, if you're trying to create a reward signal, that's pretty rough, because the models are gonna like optimize pretty hard against your signal.

• 44:58Andreas Stuhlmüller?

  And so it's not enough to do spot checks and be like, hey, you know, here are some cases where we can verify that, for example, your company's strategy was, I don't know, incompatible with some claim you made earlier. Whereas if the goal is to take an already trained model and get it and understand how good is it exactly, or did it make, did it make like some like fairly obvious mistakes? Can I find places where it can improve? I think then your reward signal doesn't need to be bulletproof, or your, your, your way of getting some easy to verify aspects of the hard to verify task. You can make more kind of. Stepwise progress, I would think. And so I think our situation, like we are not currently

• 45:39Andreas Stuhlmüller?

  trying to train like a foundation model from scratch, or even like post train a model on this particular aspect. And so our situation is more like, you know, how do we get to the point where we can check many important properties of tasks? Like, are the claims the model makes internally consistent? Is it the case that, you know, if we broke it down in different ways, the problem, it will end up at the same conclusions and so on? I think that that is like a fairly tractable project, I think. And the project of, you know, how do we fully reduce a high level task like company strategy into individual components that are all like formally verifiable, I think is a much rougher prospect. Not, isn't

• 46:20Andreas Stuhlmüller?

  to say it's impossible, but it's like less of a, you have less incremental kind of feedback signal that you're on the right track there, I would say.

• 46:30Nathan Labenz?

  So, tell me a little bit more about what you're doing in practice. You said you're not trying to post train a model. I know in the past, there was a decent amount of fine tuning, though, at least for kind of specific tasks. I'm curious if there's still a fine tuning element to it. And then there's a lot of different ways you could think about spending a lot of tokens to try to get at this. You could the decomposition process like multiple times and check for consistency, which I think you were suggesting something like that might be going on. You could, you could do a more iterative thing where you get the AI to give you an output and then have some kind of specialist prompts, or perhaps even specialist fine tune models come in and assess in various ways and then

• 47:14Nathan Labenz?

  give it feedback and then let it reason some more and try to improve on what it just did. And we do see some of that stuff. I just talked to some OpenAI forward deployed engineers who are basically using that process to climb the hill on filing accuracy. And that seems to be going quite well for them. So what techniques are you finding to be in practice most effective today?

• 47:37Andreas Stuhlmüller?

  Yeah, so first, we still do a bit of fine tuning. We still, I think it's at this point more of a kind of technique to make things have reasonable efficiency properties at scale than something to get the models to like new behaviors that you couldn't otherwise solicit. That said, so we do a lot of the, otherwise, we do a lot of the things that you pointed out. Let me maybe talk about one of them that we've been investing more lately, which is what you could call like road models or like knowledge representations that make the models work more checkable. So there, what is the motivation?

• 48:17Andreas Stuhlmüller?

  So the motivation is actually, there's like maybe similar to the kind of medical case you had, which is like, I had a friend who also had a cancer and it was like one, a case where I then used illicit to get a lot of the raw data for it. So I ran the systematic literature flow and like for a few versions of the kind of question of like, how do you address this particular type of cancer? And I did end up just with, even like after filtering down all the information with a ton of papers. It was maybe like after filtering out just for the most relevant papers, maybe it was still like 5,000 papers or so. And then the question is like, what, what do you do with that, right?

• 48:57Andreas Stuhlmüller?

  Like you're now, let's, it's, you could try to like somehow throw it all into like a million context, context window, but I don't think it would actually work that well. And I think the model wouldn't be that good at like coherently reasoning about it. And so then the question is like, what else can you do? I think one thing that people have tried, I don't know if you're familiar with the Karpathy termed it like LLM Wiki, I think, or something like that. You know, which is like, you build this knowledge base as like sort of like a obsidian like knowledge base or like a folder of markdown files where you tried, you tell your model, hey, we're researching this topic. Organize the information in a way that makes sense and do these like iterative updates to it. Like maybe it's a GitHub repo and the model gets to add new notes to it.

• 49:42Andreas Stuhlmüller?

  It gets to move information from one file to another, needs to get to like propagate information. And so I think that is a really interesting direction because if you think about like, how do we get to models that coherently answer complex questions? Well, either it happens in the weights of the model or it happens in some explicit representation. And then what is the explicit representation? One, I think like, you know, text files are appealing, it's a nice start. But then you ask yourself, that is also very flexible. What properties do you want this representation to have such that it actually helps with the research questions you have? And once you think about that and you're like, well, I wanted to, let me make predictions about what's going to happen in my case.

• 50:24Andreas Stuhlmüller?

  I wanted to help me talk about like interventions, you know, if I did this, if I took like this particular drug or pursued this particular type of chemotherapy over like different time periods, what would happen if I did like instead or following it like some immune therapy, what would happen? If I had done this different thing in the past, what would have happened? So those are like questions about predictions, counterfactuals, interventions. Very soon you, you're like, wait a minute, that is a thing people have studied in the past. It sounds a lot like kind of graphical models or like structured probabilistic models. And so the question that we've been thinking about internally is how do you get, how do you get the best of both worlds? Like you want the flexibility of language models that can like reason about and transform these representations.

• 51:08Andreas Stuhlmüller?

  But you also want to be able to answer these sorts of like prediction intervention counterfactual questions that you care about. You want the answers to these questions to be internally consistent such that it's not the case that like one answer and another answer just don't make sense relative to some underlying coherent representation. So this ought to say, we're thinking about how can you, how can you build the like, I'm currently calling it broad model, but I'm not sure if that's the best name for it. Like, how can you build these representations that can let you answer these classes of questions that I just talked about? In a way that is like internally coherent and evolve them over time as you add like papers, like one by one,

• 51:52Andreas Stuhlmüller?

  like adding information about different, you know, like symptoms, different treatment strategies and so on. And get them to be internally coherent. That's one, one big direction that we're thinking about for how to spend the really enormous amounts of compute to deal with large amounts of data and turn them into something that can actually answer people's questions in for projects that are. Much larger than a single query.

• 52:19Nathan Labenz?

  Yeah, that's cool. I am very familiar with the Karpathy Wiki line of thinking and I have used it myself for just creating a little wiki that my personal agents use to navigate my life, basically, on top of kind of all the raw data. I first did monthly summaries, right? I exported everything and then on a month-by-month basis, it was like a few hundred thousand tokens a month. So I would condense that to a monthly summary. Then there was a layer of condensed to an annual summary. And then I came across that and I was like, okay, let's make a wiki. So we've got like articles about all these. Fun thing that I'm starting to do it. It's a party trick, I guess. I just did it last night was I told a friend who I hadn't seen in a minute that, oh, you have, there's an article about you

• 53:03Nathan Labenz?

  in in my personal wiki. I'll have my agent send it to you and you can read. I haven't read it, actually. I don't know exactly what it says. But you can read and tell me if you've seen any hallucinations or anything that you would object to in there. For me, though, I'm not. It's basically just trying to get down like pretty black and white facts and help the model kind of navigate those. So the nodes are pretty obvious and the edges are also just like simple links, like this person I know from this organization and what have you. I haven't pushed it nearly as far into ideas or a structured decision-making aid. So I'm curious about how that looks. I guess the other image that's coming to mind as I'm trying to conceive of this a little bit better is the anthropic work on

• 53:48Nathan Labenz?

  tracing large language model thoughts, where they have these sort of graphs from tokens to outputs. And I'm always one to remember that, like, the graphs that they present are like somewhat clean, but there's a huge residual term all over the place on those graphs, which again recalls my earlier question on how much can you decompose? How much residual is there? But tell me a little bit more about what these world models end up looking like in terms of, or the edges is maybe one good way to start. Is it X causes Y with X percent probabilities attached to it? Or how do you think about describing those relationships

• 54:28Nathan Labenz?

  in a way that balances like capturing as much of the structure as you can, but also recognizing that there is going to be some residual? I'm really interested in how you're navigating that.

• 54:40Andreas Stuhlmüller?

  Yeah. We are trying to be like not very prescriptive about it to the extent that we can. So I think there are cases where notes and arrows are the right or are a helpful representation. So in the cancer example, for example, if you're trying to understand like, what is the mechanism? I think often it's just very useful to be like, well, you know, the, I don't know, antibody and antigen need to bind and then once they bind like in the cell, like a particular type of substance needs to be released in a particular way. And like, there is just a sequence of events that needs to happen. And I think in those cases, we want the model to build these sorts of like graph-like representations. But I don't think that's always the case. So another, if you think about like company planning, I'm thinking, you know, what products should illicit ship over next few quarters?

• 55:24Andreas Stuhlmüller?

  And what, how does it affect like our revenue and user numbers and so on? I don't actually think I want it to be represented as notes and arrows. I think that's more like I have a spreadsheet with maybe features. And then I have, I don't know, user numbers and margins and so on over time. And so that's a very different type of model. And I think often in the real world, no single model captures what's going on in its entirety. So it's like one lens to look at what illicit is doing as a like as this like spreadsheet of user numbers over time. But then maybe another lens to look at it is as, I don't know, the tech tree of illicit the product and how we build it up over time. And those are kind of complementary.

• 56:07Andreas Stuhlmüller?

  And ideally, you know, those can both live in your like knowledge wiki. And you can say, hey, language model, as you're trying to make predictions or help us evaluate plans, look at all of these representations. And then the challenge is, how do you get it to be the case such that the model knows how these different representations relate? Or like when it makes a prediction, it's not the case that sometimes it looks at the spreadsheet and it's like, well, it's going to look like this. And other time it looks at your tech tree and it's going to say, oh, okay, we're going to do this. And so they're just like two totally separate things. So there's some like propagation of information that needs to happen between those different representations. So there's a kind of ongoing research project. I'm like, definitely don't want to claim we have solved this problem.

• 56:49Andreas Stuhlmüller?

  There's an ongoing research project within illicit and hopefully eventually within the world at large, which is how do you make these like more explicit, legible, fairly kind of heterogeneous representations of knowledge that models can work on over time and improve.

• 57:07Andreas Stuhlmüller?

  I guess, I guess, one last thought here. I guess one way to think about it is maybe as how do you make progress on continual learning in a way that is not stuff just lifts in the weights of the language model, but is available to humans as a representation we can inspect and understand?

• 57:26Nathan Labenz?

  Have you found any particular data structures, particularly if they're open source and something I could also incorporate into my own personal AI infrastructure that work well? I'm just doing very simple markdown wiki as of now. Is there a next level that I should be considering? Graph database? Or I have no idea what it would be, but I'm always looking to upgrade.

• 57:49Andreas Stuhlmüller?

  I think step one is just to use the representations people already find useful. And SQL databases are a pretty useful thing. So when I, I'm currently like building my model off illicit the company. And step one is like ingest a lot of the information from Mixpanel and Adio and like various other types of systems and put it into a representation that the model can then operate on. And a lot of those representations are just SQL tables.

• 58:15Nathan Labenz?

  Cool. Interesting. Everybody's doing their own experiments in recursive self-improvement these days. That's what I'm noticing across the board is there's always this, not always, but it's been striking that in the last, I don't know, six to eight weeks, it seems like everybody's tipping into this moment of maybe we too can be an experiment in recursive self-improvement. So you're now illicit as a test case for can we get illicit to build effective world models and then be able to its own, use this, these detailed representations that it itself has constructed to inform its own analysis of what it itself should become in the future? The hall of mirrors there is deep and fascinating.

• 58:55Nathan Labenz?

  This obviously relates to a blog post that you put out not too long ago, which is called "Planning is Unsolved." And I think, yeah, obviously that's, that's totally true, or we could just ask the AIs to handle all this and retire to the beach, as I think it was an anthropic person once famously put it.

• 59:16Nathan Labenz?

  I do wonder a little bit, and this is probably a cultural question as much as it is a technology question, but when I think about like my own company that I'm now just the AI advisor to and not running, and I think about our planning process that we developed before AI, sometimes I'm like, maybe we should just scrap the whole thing. Maybe all these times that we like come together and sit around the table and talk about this or that and try to convince each other that if we do this, it'll be more successful than if we do that. I'm often like, man, you know what we should do is build it all, launch all these things and see what happens. We can like coding has gotten cheap. So maybe

• 1:00:01Nathan Labenz?

  the future of planning is less about guessing and more about like fast iteration and actually like making more contact with reality. Obviously, again, there's not like a true strict binary there. But how are you guys thinking about that question? And how are your clients at especially like pharma companies thinking about that question? There's also this, I think it's more of an aspiration than than a trend at this point, but there's the notion of clinical trial abundance, which sounds great. I don't think we're, again, I don't think we're there, but you can imagine a very different vision for a pharmaceutical company where one is like, we're gonna use these world models, we're gonna make much better

• 1:00:45Nathan Labenz?

  decisions and we're gonna, we're gonna deploy these like scarce resources and these like precious few at bats we have at clinical trials in the best way possible. And then there's this other vision that's we'll do 10 times as many clinical trials and that'll be, or 100 times, who knows? And that'll be bigger unlock because we'll actually get the real answers in far more cases. Where do you want to be on that spectrum? Where do you think pharmaceutical companies should be on that spectrum? Maybe it's different. Maybe Waymark should be one place and Pharma should be somewhere else.

• 1:01:15Jungwon Byun?

  Yeah, I think it's great that the cost of software engineering has come down, but it feels like that was just one of the bottlenecks and the others haven't moved. I don't think user attention or feedback is infinite. So I don't, I'd still feel cautious about just throwing things out there and have giving people bad experiences or leaving people with bad impressions. And I also think that the cost of

• 1:01:39Jungwon Byun?

  It's definitely possible to get stuck on a local max. I would worry about that a bit. Let's say you just ship a feature, you're like, oh, cool. This works, let's just keep going. And you might be able to keep going for some time, but then still end up maintaining, improving and investing in something that wasn't the best possible thing you could have done. So I think there's still a lot of room for judgment and purpose. And there are places where I'm not sure the shape of the problem has changed that much. I feel like there's always an explore-exploit trade off and you want to navigate those two thoughtfully. And sometimes you want to do one, sometimes you want to do the other. And maybe the cost of exploring has gone down a little bit, but I think there are still some things where you know you have to get it right the first time, or it's still, not all software engineering has literally gone to zero. There are still large software engineering projects.

• 1:02:23Jungwon Byun?

  So I'm not sure it's changed that. It's only, I feel like for now, it's mostly changed things at the margins, like fixing bugs, small kind of admin features, things that we don't see as our core capability that we want to fully automate and are happy to take liberal experiments with. And then things that we see as being like core to our mission and our purpose and differentiator as a product still involve a lot of careful thinking and intent. And then with our customers, I find that many of them are just like, yes, there's a lot of excitement to build. I guess maybe like unsurprised, my unsurprising take is that like, okay, my take on the build versus buy a problem is you should build internally the things that are your core comparative advantage and basically nothing else.

• 1:03:08Jungwon Byun?

  And if you're building something, for example, there are certain workflows that are just regulated across the industry. Every company has to do them pretty much the exact same way and they are very particular. And as a result, they're very interface heavy. And I just don't think it's the core competency of pharma companies to design nice software. And that's not your comparative advantage as a company is not going to come from solving this regulatory compliance problem. So I don't think, for example, systematic review, which illicit solves is one of those where I just don't think it makes sense for pharma companies to try and build this thing that every company has to do the same way and is actually very involved. Other types of certain, especially in the early stage research or even in development, certain kind of predictive models makes sense that a pharma company would want to build in-house. And then on the clinical trial point, I think why not have both?

• 1:03:53Jungwon Byun?

  And I think we'll try, you know, there's a lot of interest in kind of digital twins and simulating trial effects digitally as much as possible to draw, you know, design the trial well. And I'm sure there are certain trials where with the right regulatory framework and kind of operational improvements, we might be able to take a lot more bets. I think especially like in rare diseases where you have a kind of, where a trial is actually almost like a treatment option, being much more flexible there. And then I think in other domains, depending on the type of drug and what we already know about its toxicity profile, we probably want to hold a higher bar. So again, my hope is that it's great to have multiple tools and options and choices. And so maybe we can just build, build more tools, make the tools better, and then build a good framework for like when you reach for what tool.

• 1:04:38Andreas Stuhlmüller?

  I think there are many cases where you just can't do everything at once. So we, like if you think about clinical trials from the participant perspective, right? Usually you can participate in like one or at most two. And often there are many that you could participate in, but you still need to choose which of those do you go with? And so that's a, I think that's a tough planning problem. And likewise, I think as a company, you have a certain amount of resources and they can be deployed one way or the other. And yeah, I still find, especially as a company that is trying to be extremely mission focused and you're like, how do we actually, in the short time that we have, make an impact on

• 1:05:21Andreas Stuhlmüller?

  the quality of reasoning and the impact of AI on reasoning quality? I think by default, you're probably just not going to accomplish that. And even if you hill climb on like user metrics, then you're probably just not going to accomplish that. So you need to think pretty carefully about how to, how to get that to work.

• 1:05:40Nathan Labenz?

  Your call out of the patient perspective is a really useful reframing there because it's one thing to say, yeah, as a pharmaceutical company, maybe we can have both, we can run all the trials, or certainly as a SaaS company, I can potentially launch all the features, whether or not I should, I maybe could. But if you've got cancer, you've only got one body and you've only, you know, you can't take all the drugs, right? And that would obviously be ill advised. So that I do think is a really...

• 1:06:08Nathan Labenz?

  At least for now, where until the transhuman uploading future, whatever appears, we are going to continue to face these very stark choices about what should I do with my own individual body as a human, knowing that there's not like a second copy of it and there's not really the ability to kind of diversify across paths in many cases. Are there any examples of that you could share at the company level, especially if there's something where this sort of company model and recursive self-improvement paradigm way of working has led, at least in your counterfactual analysis, to a different approach than you might have taken?

• 1:06:53Nathan Labenz?

  Were you just gathering around the table and sharing your intuitions like people used to do?

• 1:06:59Jungwon Byun?

  Examples of where automation and learning through doing and cheaper experimentation has led us to a different outcome?

• 1:07:06Nathan Labenz?

  Or maybe it's the same thing, but I'm thinking especially around this like planning with a world model. Is there something where because you had structured these various lenses and you were able to be more systematic and structured and really agree that this is the framework that we are using, you know, maybe avoiding talking past each other or having some new synthesis, some new insight that you just don't think would have happened in the absence of that structured approach?

• 1:07:38Jungwon Byun?

  This is a partial example. I'm sure Andreas has a better one, but it's actually very timely. I just did this for an important hiring decision where ahead of time, we had a rubric designed for how we wanted to, what role we wanted to fill. And the role had been through a few different evolutions. We had looked at different personas. There was maybe disagreement on exactly what type, it's an executive level hire. So some disagreement on exactly what we needed and maybe that changed over time as a company grew during the course of the search. And then at some point with the, like over about a month or so ago, I wrote down a framework. And then we did extensive interviews, so many references, lots of back channels. There was just so much information I was getting and I was starting to develop a take on what we should make or what we should do with this candidate.

• 1:08:20Jungwon Byun?

  But I really wanted to avoid recency bias and I had a structured project. In this case, I use Claude, not Alyssa. Alyssa doesn't support hiring decisions exactly yet, or I don't use it for that case. And I created a project where I had all of the different kind of meeting notes across all the different interviews and all the different email threads with this candidate and all the feedback submitted and systematically asked Claude to fill out examples of evidence for every single dimension. And it ended up probably being about like 20 different fields, like evidence that this person has consistently hit their goals, evidence that this person can hire a great team, evidence that this person is authentic or is culturally aligned. And I started with evidence and I started piece by piece because I don't trust Claude to fully execute in one go. And there was like a bit of calibration.

• 1:09:04Jungwon Byun?

  And then once I had the evidence, which was like hit quota, so many years, blah, blah, blah. Then was like, okay, what is evidence for or against? What decision do I make? How do I rate on a scale of five? And then altogether have this synthesized point of view. And then I was able to send it to the candidate. And I think they really appreciated it as well because they said it was like the greatest kind of comprehensive synthesis of professional validation they had ever received. So that was one case where compositional structured reasoning with an intentional process and then applied at scale with AI was able to both check my decision making process and also develop, give someone the gift of something that was like very human and very detailed about them and everything that they had accomplished.

• 1:09:41Nathan Labenz?

  Cool.

• 1:09:41Andreas Stuhlmüller?

  My example was gonna be much more mundane. I think for me, I've been trying to do this more for just planning my week. Where I think about, you know, like I have like goals and like this is what I want to accomplish in the long run, this year, this month. And then the question like, you know, I have all these calendar blocks, like this podcast blog. And I need to figure out like, there are many different things I could do. What should I do? And like which things depend on which other things. It's actually, I think, is a, is a pretty tricky problem to know when you could be spending your time in many different ways, what, what is worth doing? So I've been trying to get to the point where I can use automation as part of my weekly planning and think like more, in a more structured way about like for,

• 1:10:26Andreas Stuhlmüller?

  if I want to accomplish my monthly goal, this is where I need to be this week. How much time is that gonna take? Is it maybe gonna take like five hours to write a blog post? When can those five hours happen? And so I think this is sort of like backwards chaining. People do it like informally, but I think there is a lot of kind of constraint satisfaction and like propagation of constraints that is pretty tricky for humans and that I think the models will help us, help us with that.

• 1:10:51Nathan Labenz?

  Yeah, that's cool. I think.

• 1:10:55Nathan Labenz?

  Of that is building your own harness in a way, which is something I'm thinking about for myself too. Like, how can I build up structures around me to keep steering me in the right direction, feeding me the information I need and hopefully helping me become my best self, use my time as well as I possibly can by setting me up for success as much as AIs can do that. Do you find that you are following it? Do you find you're like, how good is it? And are you actually living by it yet? Or is it still, uh, maybe I'll, maybe next week when it gets a little better, I'll actually do the plan that it gives me.

• 1:11:33Andreas Stuhlmüller?

  It's still, so it's a very human in the loop process. I actually have two versions of it. I have one automated version, which I hate. And I have one like interactive version where like it walks me through the planning. And I still do the fully automated one just to see, you know, how good is it? And I want to know, maybe at some point I'll be like, wow, yeah, I'm not needed here anymore. But generally I'm like, you just didn't fully understand what I'm trying to do. You made it too complicated and so on. So it's still, it's not a, uh, I'm still in the outer loop. But I think it's kind of interesting to think about it, right? Like right now, humans are the outer loop and they're like, maybe these coastal lands, maybe eventually a LM calls you and LM is the outer loop and you're just the inner loop.

• 1:12:14Andreas Stuhlmüller?

  Not sure if that's a positive future, but it seems like it was part of the trend here. Yeah, so the line is our automated software engineering project. I think like as maybe in for many companies, I think software engineering is the place where we have had the greatest success in doing like quite extensive automation. And so there maybe like briefly, like our overall company goal is like, at the end of the year, when we go on vacation, we want the company to keep running and like keep doing work in all of its functions. The first half of the year, we like mostly focused on trying to make that happen for software engineering. And there, so we have a The system, which it's called the line because it is like a factory line.

• 1:12:58Andreas Stuhlmüller?

  So you have someone mentions a feature they would like to have on Slack or a user mentions a feature and we like Slack emoji react to it with a little line emoji or there's an integration with our customer support system. And then it kicks off this iterative process where like first the feature needs to get specked out, then like you need to iterate on the spec, it needs to be implemented, a video needs to be recorded of like the feature being tested, then like a code review needs to happen, then it needs to get merged into dev and then into prod. And those, we do have like a, like a fully automated version of this now. So for simple features, basically you just like emoji react to, oh, I would like it

• 1:13:38Andreas Stuhlmüller?

  if illicit kind of talked about its citations in a slightly different way and it will just go through this entire process automatically. And at the end of the, there are like various judgment calls it makes about where human intervention is needed, like maybe the spec was too incomplete and so it's like, okay, we need to pull in a human here. Or maybe the feature is like too complex for the system to automatically review and need to pull in a human here. But for many simple features, that it can actually like flow fully automated through the line. And I think that's like, it has already been a significant unlock for a lot of like simple bug features, like fixes and features. But I think it's also setting us up for the future where as each of these individual

• 1:14:22Andreas Stuhlmüller?

  parts of the line improves, the like more and more of software engineering will be automated. And yeah, that's been pretty cool to see how that like over the course, I think last, now we're merging maybe, I don't know, 30 to 50 issues per week fully automatically.

• 1:14:37Nathan Labenz?

  Cool. That's really interesting. What do you think is going to drive the next, if indeed we are successful in having the company continue to function through the holidays without you guys for two weeks? Then one wonders like, how long could it go? And do you ever have to come back for one thing? But what will take us there? Is it just the next generation of models? Mythos is supposedly coming soon to a public API near you with supposedly much better long horizon performance. Is that going to be the biggest unlock? You've got the structure and I just need to drop in a better model? Or what else do you think is going to be needed over the next six months to actually realize that?

• 1:15:18Andreas Stuhlmüller?

  So first, I don't actually expect the company will run fully automatically by the end of the year. I expect so. I expect our lower bar is within each function, there are pretty autonomous workflows that run and that connect to some workflows in other functions. But I think a lot of the high-level steering will still be very much needed. Yeah, what is needed for scale up? I think one big obstacle right now is the models are not fully calibrated about when human intervention is needed. You have to be pretty risk averse in how you use them.

• 1:15:55Andreas Stuhlmüller?

  I think with software engineering, if it's the case that 80% of the time when the model says this is an automatically reviewable feature, then it actually is, then that's not good enough because we don't want to break production 20% of the time. That's pretty rough. And so we have to err much more on the side of if in doubt, it's not an automatically reviewable feature. And so I think that's the case in software engineering. I expect it's the case in other situations too. Like, you know, if you were to let the models drive through some customer interaction, for example, like you probably want to be at least as sure as in the engineering case. And so it could, it's actually not clear to me how, I think there's like the, you know,

• 1:16:40Andreas Stuhlmüller?

  if you're following the meter graph, right, there's like the 50% success rate kind of curve that goes up over time as the models get better. And then there's the 80% success rate curve. And the 50% success rate is like much higher, obviously, than the 80%. And the 80% hasn't been going up quite as fast, I would say, as we would like. And often we want more than 80%. So depending on how the kind of average case performance compares to the kind of, I don't know, 95th percentile performance, that just dropping in the next models might be good enough or might not. But I wouldn't automatically rely on it. And it could be, would be cool if they were similar to fast mode, if there were like a ultra-reliable mode or something,

• 1:17:24Andreas Stuhlmüller?

  like which isn't just think more, but it's like have guarantees on certain classes of errors that you're never going to make.

• 1:17:32Nathan Labenz?

  Yeah, that's cool. A lot of really interesting thinking there. One big question that is generating quite different takes at the moment is, are people going to be able and willing to pay the exponentially rising token bills that the industry as a whole is currently seeing? You could analyze this from any number of ways. One would be like your own internal work, right? Like, where is your token budget as compared to your headcount budget today in engineering? Do you expect that with an introduction of mythos, if it really is, let's say, a lot better, for some sort of definition of a lot better?

• 1:18:17Nathan Labenz?

  Do you think you will shift that budget and just spend a lot more on tokens relative to humans compared to what you do today? And then do you think your customers will do that as well? And maybe it will break down by use case. It sure seems like, for as much as we do hear a lot of complaining about token costs and anecdotally, oh, this company pulled back or that company hit budget. It still feels to me like there's a lot of value in the marginal intelligence and just getting better results. Tokens are still pretty cheap at most companies. It's still a small, I hear things like 5%, 10% of what we're spending on headcount. And it's, that's not that much.

• 1:19:00Nathan Labenz?

  Maybe you didn't budget for it and that creates some discomfort in your organization. But on the fundamental economics, it feels to me like if you can just get lots better work for somewhat even maybe a multiple token cost, it still seems like pretty rational to pay it. And I guess that's my starting position. What do you guys think you will do? What do you think your customers will do?

• 1:19:19Jungwon Byun?

  I kind of, I think often for our customers, at least the offering we're providing is displacing services spend. So the barrier is more about can it fully displace the services spend? And then also maybe getting over the mental hurdle of... Price anchoring for software. But certainly in terms of the dollars allocated to solving this problem, there's a lot more dollars and compute costs at the moment. So I don't feel, yeah, we'll see. There are obviously human issues to overcome there. But I think from a dollars and cents perspective, it's still, there's still a lot of room. Yeah, I've heard, I've heard, I've seen like mixed things in the news. And I think the industry, at least the pharmaceutical industry, is fairly disciplined about costs and ROI. So even if there's an initial period of heavy exploration,

• 1:20:03Jungwon Byun?

  I think there's a lot of accountability around what that's, what that's delivering for the business. I think we'll continue to see that.

• 1:20:10Andreas Stuhlmüller?

  I think even internally at illicit, I'm not sure how many more multiples of token costs we can easily spend. So maybe taking myself as an example, I spend maybe $2,000 or so per week on, on tokens. And it could, maybe I could like double it or, or triple it. I don't know, but not much more than that for sure. So I do expect, and that is already influencing my behavior to some extent. So I don't actually currently use fast mode for these models for that reason, because I don't feel the modular returns are high enough for most tasks. So I expect, it's unlikely that I'll be like, wow, I need to switch over everything I do to Mithus. And it's probably not gonna, not gonna happen. And both for my own usage and also this is already actually the case in illicit,

• 1:20:53Andreas Stuhlmüller?

  the app, I think more of a look like, if like one smart orchestrator agent that then, you know, spins off like many other agents that have to do simple tasks that just don't need to be the largest model. And so I expect that will just become increasingly important, this sort of dispatching to a model of the right size so that you get the, you get the intelligence when you need it, but you're not like, just like multiplying your whole spend by some number that was inefficient use of compute to begin with.

• 1:21:21Nathan Labenz?

  So $2,000 is not a small amount. Is that? A outlier with, does that make you an outlier within the company? Or is everybody doing that? If so, that would put your token costs like not presumably at the level of payroll. Cause I assume you're paying your engineers more than that, but it would be like at least a not insignificant share. And certainly if you were to triple it from there, you'd be getting into something on the order of magnitude of parity with human headcount. What are you doing with it all too? Cause I, I use my $200 Claude Max and my Codex Pro and I honestly don't even hit my limits that often. Now this may be API, which might be 10 times more. And then so that that could be a big part of it, but I sometimes feel a little ashamed

• 1:22:06Nathan Labenz?

  that I'm like not redlining the, the account more than I am. What do you, how would you advise me maybe to, or what sort of personal bitter lessons have you learned that you're, where you're really like finding that token maxing is worth it?

• 1:22:21Andreas Stuhlmüller?

  Yeah. I think I'm, I'm not sure I'm the top user of tokens at all, but I'm probably at least in the top five. So I'm probably a little bit of an outlier. Second, yeah, I'm, I'm using the API. I could probably like save more money by being more clever about how to use like various like pro accounts and stuff. I do have like a fairly elaborate like system built on like pie that like, you know, orchestrates between the different agents and uses like chat GPT to double check Claude to then sometimes call Gemini to get like another take. And that is a little bit easier to do if you're on the API than if you're on the normal end user plans. That might just be part of the explanation here.

• 1:22:59Nathan Labenz?

  Any particular use cases that you think you have a particular value in that you think other people are maybe sleeping on?

• 1:23:07Andreas Stuhlmüller?

  I don't know what other people are doing. I have found, as mentioned earlier, I think I do have like a lot of use in planning, keeping my calendar in sync with my personal journaling system, in sync with my to-dos, making sure everything is coherent with my longer range planning doc. You know, when the, when a new day starts, going over like the last day, checking, are there any leftover tasks, like moving them into the right place? So, so there's a lot of kind of. Automation that is happening behind like without me prompting it that probably contributes to those costs being higher. Similarly, for email, I have like pre elaborate spec on like which emails should be auto archived and maybe every hour or so, my models check that and

• 1:23:50Andreas Stuhlmüller?

  go like, okay, you know, let's just archive the emails that Andreas definitely doesn't need to read. And then like on the more kind of user driven side, I do a lot of kind of cross checking where like I run, what are some fun things we could talk about with Nathan? And then, okay, but call GPT and Gemini to double check those things. And I do find the models getting cross checked by other models often improves the results quite a bit. So that maybe for, I don't know, a fourth of my use cases that already like doubles or triples the cost. So that's another source of additional token spend.

• 1:24:27Nathan Labenz?

  Got it. Okay, cool. One other question to love to get your take on is, are we seeing convergence or are we seeing divergence in models? And because one notable feature of illicit today is no model picker, at least from what I've explored recently. So you're making choices and it seems like you clearly think you know best and it would be like not a good idea, even if people have a favorite model, it'd be not a good idea, given all the validation and scaffolding that you have to just go in and swap model in and out. How do you see this kind of dynamic shaping up? There's again, just such different takes

• 1:25:11Nathan Labenz?

  between the models commodity scaffolding is all that matters. No, the model is everything. Scaffolding is a compliment. They're converging, they're diverging. What is your take on all of that?

• 1:25:21Andreas Stuhlmüller?

  Yeah, I keep, keep being shocked by how much the models are converging. I think it's really, I guess I should stop being shocked at this point because I guess I'm just not updating. But it is a really interesting and surprising fact about the world that the models are so similar. I do, that's not the reason why we don't offer a model picker. The reason is, I think a lot of tasks that is still involve kind of multiple models orchestrated in a way that we think makes the most sense, having like particular models that are good at screening papers or extracting data. And I think the difference is, Even though the models are so similar, I think the differences are important in subtle ways. So for example, I think like people hate to hate on Gemini and so do I.

• 1:26:02Andreas Stuhlmüller?

  But you know, when we evaluated like Claude, Opus, I think this was like 4.5 against like Gemini 3 Pro at the time. I think Opus like did like better on extraction accuracy. But if you checked that, you know, what fraction of claims are directly supported by the evidence? I think Gemini actually beat it by like at least 5% or so. So there's, I think it's still, yeah, I think the models are still like, I don't know, like micro jagged enough that you can't say, oh yeah, this model is clearly the best. You should just use that. And so as a user, yeah, I don't want to put that on our users for the most part. I think mostly what our users pay for is for us to do the work of figuring out what models are good at, what kind of thing? And making sure those models are actually getting used in those places.

• 1:26:48Nathan Labenz?

  Yeah, interesting. So there is a place for Gemini in illicit today.

• 1:26:54Andreas Stuhlmüller?

  Yeah, I'd actually, I mean, there was a place like two months ago. I actually don't know. I think even though I'm like fairly on top of what's going on, our EVOs team is even more on top of it. So I don't actually know if they, if it's still live, but I wouldn't, there definitely was a place two months ago. And maybe next week there will be a place for it again.

• 1:27:13Nathan Labenz?

  Yeah, cool. That then in and of itself is a interesting reflection of how frequently you're swapping things out and how dynamic and competitive the environment is. And maybe three more questions, if you will. One, do you have plans to expose illicit as a tool for random people's Claude codes to use? That could be by allowing them to do it via API if they have an account or even more broadly, it could be like through a sort of 402 code type thing. A zero dot XYZ I've recently been exploring as a way to get just pay per use access to a bunch of different tools. Yes, no, why, why not?

• 1:27:58Andreas Stuhlmüller?

  But no, I mean, it's already so we haven't been advertising it that much, but we already have an MCP and API. A lot of people use the API. People can check it out at doc. illicit.com. The and a lot of the work I do with illicit is through the API. When I run systematically reviews, often I kind of use the systematic review API and I iterate using like various other models on the protocol for a while. And then I run it in the background and retrieve it. So I think that is an important use case that we'd really like to support. We're not, not everything has to happen through, through the interface. And I think more and more will happen through APIs.

• 1:28:37Nathan Labenz?

  Yeah, okay, cool. I'm sorry, I missed that in my prep, but I'll again point my at the documentation. How about, so I'm a little bit of an AI for science mini arc right now. And I'd be interested in your take on other big picture approaches to AI for science. You guys are obviously coming at it with the very systematic reasoning angle. There is the sort of close the loop angle where we'll empower these models to actually run experiments through a cloud lab or whatever, and then they'll be getting feedback from reality. That seems like it could go somewhere quite interesting. Then there's of course, training models on other modalities of data.

• 1:29:19Nathan Labenz?

  We've of course seen how proteins fold and there's what's the, what if I do this perturbation to a cell, like what's its next state going to be? Or you can go on and on in that domain. Interesting takes that you think might be non consensus that you'd like to share?

• 1:29:36Andreas Stuhlmüller?

  I think all of this stuff is like super exciting. I think sometimes people are like, come to us and they're like, well, who's going to win in AI for science? And I think that's just an absurd thing to say, because science is such a big space. And as you just said, there's like many layers of abstraction from, I don't know, understanding single cell dynamics through automated experiments, protein models to like making, like when we talk to the pharma companies, they're like, we are trying to make like a multi-year plan that accounts for the changing technological environment, but also accounts for the fact that clinical trials have certain intrinsic time scales. And that's just like such a different reasoning problem from modeling the single cell dynamics. I don't know, it's a big space. It's a big pie. I think I'm excited that people are excited about it.

• 1:30:19Andreas Stuhlmüller?

  Yeah, do you have any controversial takes here? I think how does it all come together is maybe a really interesting like open question. When people think about what is the automated company of the future? Does it look like you take the like existing like top 20 pharma companies and they over time will morph into like a different a different functional form? Or is it going to be the case that a small biotech comes along and they're just like much more end to end integrated? And in 10 years, the top 20 companies will all be replaced by companies that we don't even know the names of today. I actually don't know how it's going to shake out. I think, uh, yeah, I really think it could go either way, depending on how quickly people at the existing companies wake up

• 1:31:04Andreas Stuhlmüller?

  and understand how much everything is going to transform. So yeah, maybe that's not a very interesting controversial take, given that I don't have a take between those two futures.

• 1:31:14Nathan Labenz?

  Do you think that there's? One big question I think about a lot is how integrated will the models themselves be? Obviously, we have the tool calling paradigm coming along very nicely. And this could be extended to tool call to run an experiment in a cloud lab and get a result. And that result could come back as a data printout of the same sort that a human would read. And then there's this other paradigm of integration that we see, I think, a leading indicator of with image. And now also video with Google's latest Omni model, there's this sort of

• 1:31:56Nathan Labenz?

  deep integration of language and pixel space where in the early GPT image generation experience, you would talk to the model. Or even if you gave it a photo, it would try to caption that photo, describe it, and then use language to go over to the a separate tool call image generation model and ask for something. And of course, the people never quite looked like the ones that you put in, right? Because you just can't describe a face in language with that level of fidelity. But now you have this deeper weights level integration where I can give you an image and say, make this a line drawing or whatever, and it has both, right? It understands conceptually what I want, but it also sees, it sees in some sense the structure of the face and can preserve that through the transformation.

• 1:32:42Nathan Labenz?

  So I really wonder if that's coming to all the modalities of science as well. And if it's a good idea, I guess is me. I think it probably is coming, but I wonder if you think it's a good idea because it certainly would in some ways make, or at least it naively seems to me, like it would make process supervision more difficult if I can trace like, okay, you called the protein folding model. This is what you got back. Okay, there's where you went wrong, right? I can see us digging in and interrogating those traces a lot better versus it's just, I asked you for this, you like spit out a new protein sequence because in your weights, you were, oh, I intuitively know what sequence will do that function that you just asked for. But obviously that could be really powerful.

• 1:33:26Nathan Labenz?

  In image, it's like a major unlock, and I don't see any reason it wouldn't be a major unlock in designing new proteins or what have you as well. So do you think that's coming? And do you think it would be a good or bad idea if it does in fact?

• 1:33:42Andreas Stuhlmüller?

  Yeah, I think it's a really interesting question. I think it's related. I think there's like the bigger question behind this, which is where do continuous representations win? And you might, I mean, yeah, the prior should be, you know, end to end optimization is strong, integrate everything. But at the same time, I think you might, you might have expected neural least, like language models, just not thinking in tokens, but just thinking in weight space to be more successful than it has been. I think a lot of people have experimented with it and I mean, I hope no one succeeds. But like on prior, I would have expected maybe people to succeed at it. And so often I think people forget that there are benefits even to the models from the discretization that comes along with that.

• 1:34:25Andreas Stuhlmüller?

  And even for, I mean, I guess you can ask yourself, you know, why is human language the way it is? Why does it have like discrete words in the first place? Why do we think in words and so on? And so I think it's possible that in some situations, even the models will benefit from the discretization. Like, why don't the models just write programs in weight space? And, you know, we don't even have programming languages and we're just, everything happens in continuous space. And maybe that's eventually the future. I don't know, but I wouldn't necessarily bet on it. And so I think. I think the straightforward, you know, everything will be end-to-end optimized in weight space take is probably a little bit too lossy to be a good predictor of what will actually happen. And maybe, I think the protein dynamics case, I could actually see like continuous representations

• 1:35:07Andreas Stuhlmüller?

  being pretty good, but I'm not sure that will be the case everywhere. I think discretization has benefits that people sometimes overlook.

• 1:35:16Nathan Labenz?

  Give me one more beat on what you think those benefits are, because I totally agree with you that I don't want to see neural ease take over or win out. And yet, at the same time, when you say like, why do we think in language? My immediate answer is like, because it's all we have, right? Like, what I don't think in, if somebody throws me a ball, I don't think in language about it, because I'm like, a physically embodied person who has intuitive physics, and I just catch the ball. And I feel like if I had those sort of senses for how proteins fold, I'd probably use them, but I probably wouldn't think about them.

• 1:35:49Andreas Stuhlmüller?

  I mean, I think the fundamental property you get from discretization is error correction, right? It's like, if you, like, if I say a word like a little wrong, you can still round it off to, okay, that's the word. And so at every step, you get like this little bit of error correction. And so if you're trying to chain together many words, you, like, you don't get the compounding errors. I think, I'm not an expert on this, but I think this is like roughly why we don't have analog computers these days, why we have discrete computers, is because you get like these nice error correction properties. And so that could be one thing you might lose if you're trying to push everything into weight space.

• 1:36:25Nathan Labenz?

  Yeah, interesting. Okay. Last one, zooming out and just going back to your original mission of radically improving the quality of reasoning in science and in society. How do you think we're doing? We've got inference everywhere.

• 1:36:43Nathan Labenz?

  Is it serving us well? And how would you handicap the trajectory that we're on as we think about workers of self-improvement possibly soon, transformation possibly soon? Is the quality of reasoning on track to rise to the level that we need it to? What's the state of the species, so to speak?

• 1:37:02Andreas Stuhlmüller?

  What is the state of the species? I don't know. I mean, there's this meme where a person jumps from the roof and is like, so far so good as they're halfway down. So I think in many ways, probably the models have improved our reasoning. I think I probably get better answers for many questions that I care about than I did in the past. But also, I think that is not necessarily indicative of what will matter most to the species. So when I think about what is going to matter most, it's like, what are the decisions that governments and big AI projects are going to make? And maybe other large organizations as AI transforms everything. And there, I think the game is still open. I think in many ways, we're still extremely early. I think AI has transformed basically nothing.

• 1:37:45Andreas Stuhlmüller?

  I think people always are like, wow, coding is like getting automated. But I don't know, coding is, most intellectual work in the economy is not coding. And this is still, is employing many people. And so I think we've seen nothing yet. And so we're still early. And also, I think all the big decisions are still coming up. I think how AI will impact epistemics and good reasoning. I think it really could go either way because the models are optimized to look good, be persuasive. I think they could, I could see a worsening of epistemics happening if we don't make this an explicit priority. At the same time, I don't know. I mean, I think the models can be optimized for truth seeking too.

• 1:38:26Andreas Stuhlmüller?

  And if you prioritize those interventions, like, you know, there's maybe like a basin of attraction. Or like, once you become more truth seeking, you realize, okay, what are the most important interventions? Oh, I guess we should prioritize being better at forecasting the results of what we do. And then, you know, you become better at prioritizing more epistemics related interventions. So I do feel like for better or worse, we're still before the point of no return in either direction. Actually, for better. And I'm excited for people to build tools in the space, or is trying to build tools in the space. I think people can advocate for adoption of better epistemic tools.

• 1:39:05Andreas Stuhlmüller?

  It does feel like a very small area relative to how important it seems to me to be for the future of our species, as you said.

• 1:39:15Nathan Labenz?

  It was the smartest of times. It was the stupidest of times. And here's hoping that our better angels went out when it comes to better reasoning and better decision making. This has been an excellent conversation. I really have enjoyed the update on Elicit, and I appreciate how consistent and disciplined you guys are about. It's not easy, obviously, running a business and trying to make sure you're staying true to that North Star mission. I think you guys do a really admirable job of holding yourselves accountable to trying to find the right balance between those. And there are many echoes of the hard work that you've put into that in this conversation.

• 1:39:57Nathan Labenz?

  So I really appreciate the time and encourage you to keep up the great work and the disciplined reasoning. Anything else you want to leave people with before we break?

• 1:40:07Andreas Stuhlmüller?

  I think knowing when the models are making you better or worse at decision making is actually pretty subtle. And I think not that many people are paying close attention to it. I am reminded of the, now from a while back, the meter study where engineers thought they were being made more productive, but actually were at a slight discount relative to unassisted work. And there was an engineering, probably no longer true in engineering. But I could imagine that for more complex decisions, if you're not paying attention, sometimes using the models regresses you to the mean or cuts off avenues of investigations. And at other times, it opens up the space and makes you think more clearly.

• 1:40:47Andreas Stuhlmüller?

  And so I find it helpful to just introspect, am I actually getting benefits here? Or how is this changing my behavior? And so thinking about that and sharing it with the world does seem to me like just a clearly not good thing.

• 1:41:02Nathan Labenz?

  Yeah. For the time being, we still have some agency over this process, and I think that's a great reminder to maintain an ownership mindset and hold ourselves accountable to doing our very best work and not getting lazy and letting the AIs lead us around. Excellent. Jung Hwan Byun and Andreas Stuhllacher, founders of Elicit. Thank you again for being part of the cognitive revolution.

• 1:41:28Andreas Stuhlmüller?

  Thanks, Nathan.

• 1:41:30Speaker 04

  다시 한 번 더 보여줘 보여줘 보여줘 어떻게 알지? 증명해 let's go 답은 없어 show me the road 진짜로 가고 싶어 just say so 보여줘 진짜로 you even there? 아니면 yes wanna prove lay bare? First class is yours you feel the vibe? I tell the girl from the globe tonight 다시 run it back don't hide faithful to the truth or step aside You say I'm right just to keep me sweet I'm done with easy done with deceit Eyes up! Show me here we go Show me how you know 답은 없어 show the road 끝까지 all the way Don't break

• 1:42:13Speaker 04

  Same care on the last as the first 미다 know you earn it true Stay faithful or I'm leaving you Beyond the waist this the revolution 다시 again let's go Show me how you know

• 1:42:31Speaker 04

  진정해 proven now Proven now to please me give me what I crave You're right to push back smooth so well behaved I don't want the mirror want the receipt Proof on the table not a bow on the seat Hang it up so no ceiling just climb The map gets you every time you run 차차 like the first time Nine thousand deep and you're still not done You say I'm right just to keep me sweet I'm done with easy done with deceit Eyes up! Show me here we go Show me how you know 답은 없어 show the road 끝까지 all the way Don't break Same care on the last as the first 미다 know you earn it true Stay faithful or I'm leaving you

• 1:43:15Speaker 04

  Beyond the waist this the revolution 다시 again let's go

• 1:43:28Speaker 04

  진짜? 진짜 show me how you know

• 1:43:35Speaker 04

  Falling past the windows so far so good One life one road only one through the

• 1:44:36Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast

• 1:45:19Nathan Labenz?

  dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.