New in Nature: Google Agents Beat Human Doctors, Make Scientific Discoveries – With Vivek Natarajan and Anil Palepu

In this episode, Nathan Labenz speaks with Vivek Natarajan and Anil Palepu from Google DeepMind about their groundbreaking work on AMIE (Articulate Medical Intelligence Explorer) and Co-Scientist. The conversation reveals how these AI systems are already outperforming human physicians in diagnostic accuracy and treatment recommendations, with AMIE now entering clinical trials at a Harvard Medical School teaching hospital. Even more remarkably, the Co-Scientist system demonstrates genuine scientific discovery capabilities, independently proposing the exact same mechanism for bacterial drug resistance that human scientists had recently discovered but not yet published—signaling a threshold moment where AI is becoming a legitimate thought partner in humanity's most complex intellectual endeavors. SPONSORS: Oracle Cloud Infrastructure (OCI): Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (06:26) Introduction and Welcome (07:49) AMY Medical Intelligence Overview (11:29) Chat-Based Medical Interactions (13:59) Specialized Medicine Results (18:17) Co-Scientist System Headlines (Part 1) (18:21) Sponsors: Oracle Cloud Infrastructure (OCI) | Shopify (21:36) Co-Scientist System Headlines (Part 2) (26:32) Bacterial Drug Resistance Discovery (31:41) AI Scientific Discovery Process (35:35) Hallucinations vs. Creativity (Part 1) (35:37) Sponsors: NetSuite (37:10) Hallucinations vs. Creativity (Part 2) (42:04) Agent Design Architecture (49:17) Long Context Benefits (55:35) Computational Requirements (01:01:10) Specialist Models Integration (01:07:01) Future Model Integration (01:12:31) Tournament Evaluation Methods (01:19:09) AI Question Generation (01:22:42) Real-World Deployment Plans (01:25:58) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today's episode features an eye-opening conversation with Vivek Natarajan and Anil Palibhu from Google DeepMind. Their groundbreaking work on Amy, the articulate medical intelligence explorer and co-scientist, represent what seems to me an important threshold moment in AI capabilities. I always say that if people truly understood what AI can already do today, many would be fundamentally rethinking their plans. And these projects provide perhaps the clearest evidence yet that AI systems are beginning to outperform highly intelligent humans in domains that require years of specialized training. Remarkably, this work was accomplished without special continued pre-training or extensive

• 0:43Nathan Labenz?

  custom post-training that could only have been done within Google. On the contrary, these approaches could have been developed and can be replicated by Google's API customers using commercially available models, advanced prompting techniques, and thoughtful agent design. We begin by discussing Amy. A year ago already, Vivekan co-authors showed that Amy was able to outperform human general practitioners in diagnostic accuracy. Now, with just a few important caveats remaining, Anil and team have demonstrated that it also beats human primary care physicians in analysis and treatment recommendations. The implications for healthcare access are obviously profound and are beginning to extend into specialized medicine, too.

• 1:25Nathan Labenz?

  The second Amy paper we cover shows that the AI system is already surpassing medical fellows in both cardiology and oncology, and closing in on, but still falling a bit short of, attending-level performance. Notably, when cardiologists have access to Amy, their performance dramatically improves across almost every metric, suggesting a short- to medium-term future in which AI doctors have the potential to both raise the floor for access to quality care globally and also raise the reliability ceiling even for those of us fortunate enough to have access to first-world specialized care.

• 1:59Nathan Labenz?

  This is, to put it plainly, crazy. And I am super excited that Google is moving Amy into something like a clinical trial in partnership with Beth Israel Deaconess Medical Center, a Harvard Medical School teaching hospital in Boston, for real-world validation. All that said, somehow, in Vivekanth's co-scientist paper, we see something equally, if not even more, amazing. This multi-agent AI scientist system, which is capable of accepting human input and feedback at any step in its process, was tested in fully autonomous mode on three increasingly complicated scientific challenges. First, drug repurposing, an advanced but reasonably well-defined task amenable to combinatorial analysis.

• 2:40Nathan Labenz?

  Second, therapeutic target identification, a more open-ended challenge requiring the AI to understand and/or make quality hypotheses about causal relationships within cells. And third, and definitely most dauntingly, the wholly open-ended challenge of understanding the process by which bacteria achieve drug resistance. As you might have guessed, co-scientist, which, by the way, Google is now making available to trusted partners, succeeded on all three of these tasks. And on the challenge of understanding drug resistance in particular, it blew everyone's minds by proposing the exact same mechanism that Google's independent scientific collaborators had recently discovered experimentally but had not yet published at the time of co-scientist's

• 3:25Nathan Labenz?

  analysis. Overall, co-scientist demonstrates that AI systems are now capable of generating novel insights by connecting the dots between far-flung bits of hard-won human knowledge. This system is not simply regurgitating its training data. On the contrary, it's performing meaningful synthesis and proposing novel hypotheses that even human expert scientists recognize as both insightful and significant.

• 3:52Nathan Labenz?

  If all that's not enough alpha for one episode, the implementation details behind these systems offer valuable lessons for AI engineers everywhere. First, structured reasoning proves far more effective than simple chain-of-thought approaches, especially when working with lots of input context. Both of these systems demonstrate the value of thinking carefully about exactly how you want your AI system to reason about specific types of problems. Second, finding ways to add new information, or even just a bit of entropy, such as by giving the model access to search, is key to making self-critique and self-improvement schemes work over many rounds of successive iteration. And third, for now at least, the tournament-style evaluation process used to surface the best

• 4:34Nathan Labenz?

  candidate hypotheses out of the many that were generated, seems to be an industry best practice that you can and should use in your own work. What's most amazing to me about all of this is that it was achieved before Gemini 2.5 Pro was available to use, meaning that everything we talk about today is still subject to a step-change improvement that should come more or less for free with a simple model upgrade. With this level of performance already established and core model progress continuing, the path to an AI doctor in your pocket and data centers full of AI geniuses is honestly becoming quite clear. AIs are no longer just tools for routine tasks. They are becoming legitimate thought partners in some of humanity's most complex intellectual

• 5:19Nathan Labenz?

  endeavors, from diagnosing disease to expanding the very frontiers of scientific knowledge. As always, if you're finding value in the show, please take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. And please know that I sincerely value your feedback and suggestions. Whether we get to live in a post-scarcity society in which we all enjoy instant access to superhuman AI doctors, or perhaps on the other extreme, end up going extinct due to some crazy AI-driven scientific accident, seems to me to depend largely on how responsibly we handle the upcoming AI transition. And I take my role in AI discourse very seriously.

• 6:01Nathan Labenz?

  If you think I can be doing better, please contact us via our website, cognitiverevolution.ai, or feel free to DM me on your favorite social network. For now, I hope you enjoy this conversation on the emergence of genuine AI expertise, which you almost certainly would have considered to be AGI just a few short years ago, and which I think you should still find absolutely mind blowing today, with Vivek Natarajan and Anil Palibhu from Google DeepMind. Vivek Natarajan and Anil Palibhu, authors of Co-Scientist and Amy from Google DeepMind, welcome to the cognitive revolution. Thanks for having us.

• 6:37Vivek Natarajan?

  Hi, guys.

• 6:39Nathan Labenz?

  Guys, again, so Vivek, this is your fourth time. What an unbelievable heater you and the team at Google have been on. I always say, people, if they just had a little bit better sense of what is already out there today, you know, they would be like updating their plans in many ways that I just don't see people doing. So this is really an unbelievable example of that. I came away, there's three papers we're going to go down the rabbit hole on today. And I came away from this feeling like it might be time to call it. The AIs might in fact now be clearly smarter than me. And, you know, it is sort of, and we can get into the nuances of, you know, what they're still missing a little bit, but, you know, I think almost everybody would read this,

• 7:23Nathan Labenz?

  these reports on what you've been able to get the AIs to do and feel like they would have a very hard time, you know, matching that and to get to this level that like a single AI model, you know, scaffolded in different ways and put to different purposes. It would be like years of undertaking, you know, for, for me to get there for sure. So let's do some headlines. So there's two papers with Amy. We covered this once before. This is the, what is it, the Articulate?

• 7:51Nathan Labenz?

  Articulate Medical Intelligence Explorer. The Articulate Medical Intelligence Explorer. So I've been using a graph in some slides that I occasionally present for like the last year or so, since that, the first Amy paper came out showing that basically when a patient chats to the AI, The AI is more accurate in its ability to diagnose the person than human primary care physicians are as judged by, you know, other human doctors. So now we've got two new extensions to that. The first one is, and there are some caveats here that I think are definitely worth unpacking. I'll give you a chance to do that. It's basically now outperforming general practitioners, not just on the diagnosis

• 8:33Nathan Labenz?

  part, but also on reasoning through what to do about it and ultimately recommending treatments, which is obviously a big part of what the, you know, the doctor is meant to do for you. So, yeah, just unpack that headline for me a little bit. I mean, it's crazy that that is out there in the world today. And again, I always say like, I swear when I was a kid, if something like this happened, it would have been like headline news, you know, everybody would be talking about it. And there's just so much going on that some of this stuff, even, you know, dramatic breakthrough as it is, doesn't seem to crack the consciousness. So tell us more about Amy outperforming now, not just on diagnosis, but also on recommending treatments.

• 9:12Speaker 02

  Yeah, yeah. I mean, like prior to those, I think first two kind of Emmy papers, it was a lot of the work in the space was, you know, on medical question answering and, you know, there was some notion, right, that these, these language models do encode, you know, clinical information well, and they have a lot to offer. I think like with those papers, we were trying to start to ask that question of like, okay, but that's not clinical practice, right? Like in clinical practice, the doctor is interacting with patients, they have to kind of gather this information themselves. They're not really like prevented presented with like all the information upfront. And so that was the study, right? And, and it was, it was doing this objective structured clinical examination is this, this format. And it was basically trying to see like, can the doctor or the AI in this case, you

• 9:56Speaker 02

  know, interact with patients, gather that information and still get to that, you know, diagnostic endpoint. And of course, I think we do a really good job in the paper and I encourage people to read it of like describing the many limitations, right? Like this is a text based chat. That's not how doctors talk to people. And kind of our future, like our direction from there was really about starting to unpack like some of these limitations. And so one of those limitations, right, is this idea that, you know, it's more than just, you know, from the first visit, you see a patient diagnosing them. Like, there's a lot more to, you know, clinical care, right? And it's about managing a patient over, you know, multiple visits. You know, the endpoint at the first time you see a patient might be really like, we need to just order the right test and we need to kind of set them in the right direction.

• 10:40Speaker 02

  It's not always that you know exactly what to do with the patient after seeing them one time. And so I think the management reasoning paper is really trying to unpack that. And, you know, we'll talk more about like how that study was designed. I think it's like super interesting, right? But and there's other aspects to it as well, like rather than kind of more general recommendations, like can we get really precise and can we kind of ground in accepted clinical practice guidelines? Can we ground in, you know, medication labels and, and, you know, start to turn these into like slightly more actionable things? And similarly, like with the specialty papers, right? It's trying to expand beyond kind of the bread and butter, common presentations of common diseases, like how does this work in more niche areas of medicine?

• 11:23Speaker 02

  And so I think, and, and, you know, we have a lot more work, you know, kind of just trying to expand on some of these limitations that, you know, we've identified in the first paper.

• 11:35Nathan Labenz?

  So how would you summarize the, I mean, if I understand correctly, it is still a chat-based interaction today, right? So one major possible extension would be to go to multimodal. But you guys have also done work on that separately, right? So maybe like, why is it not multimodal in, in this particular study? Was there a reason not to just like let people, you know, throw in selfies into the chat?

• 12:02Vivek Natarajan?

  Yeah, I think it's more like when you're doing research, you want to isolate components that you're studying and do that well. And so when you add multimodal, I think artifacts add like necessarily more confounders over here. So we try to avoid that and so it's just easier to study a text-based system to begin with. But clearly we've done work on multimodal before. I know that like on two podcasts, we've spoken about like the MetMAM work and MetGemini. So all those components and pieces exist. And so very soon you'll see like the multimodal one also come out. So, yeah.

• 12:34Nathan Labenz?

  Yeah, okay.

• 12:36Nathan Labenz?

  The pace is relentless, that's for sure. So just to bottom on this one more time, basically we have.

• 12:43Nathan Labenz?

  With the caveat that it's not yet in this particular paper, although coming soon, multimodal, we have multi-visit, you know, kind of longer time horizon, interactions between patients and doctors where the AI doctor is outperforming the human doctors on both the diagnosis and the reasoning through what to do and ultimately landing on sort of standard of care, you know, accepted proper treatment for these conditions. Anything else that, you know, that we should, should I soften that at all or is that like a good?

• 13:16Speaker 02

  I mean, I think, I think like-

• 13:17Nathan Labenz?

  That should be like on billboards, right? I mean-

• 13:19Speaker 02

  Yeah, I, I think the biggest thing, right, is, you know, and, and we'll talk more about how we're trying to test this in the real world, but right, these are simulated consultations, like they're patient actors, they're not real patients, and obviously there's a whole new set of challenges when you get real patients who, you know, all kinds of things can happen and, and they're not necessarily going to stick to their script. So I think that's a whole nother thing that we need to test and validate that our results truly do translate to the real world setting. That being said, I mean, I think simulated consultations do show that, you know, we do have promise in this setting and, uh, you know, we, we personally are very optimistic that these results would, would translate at least to, to a certain degree.

• 14:01Nathan Labenz?

  Yeah, okay. So the next AMI headline is moving to specialized medicine, and there you look at cardiology and oncology. And here we basically, I, I would summarize the findings as the AMI system is surpassing fellows and closing in on, but not yet hitting the level of attending physicians in these specialist domains. So what, what further, you know, complications or caveats, you know, should we have to understand that?

• 14:35Speaker 02

  Yeah, I think largely I'd agree with the notion that we have a lot of improvements in terms of being as consistent in all domains as the most experienced attendings. And there's room for improvement. I think the real headline for me, especially if you look at the cardiology paper, is that the types of errors they make are pretty different, AI and the general cardiologists. And I think the really exciting thing is we see that they're quite complementary. So the comparison we made was, when we compared Amy to the general cardiologist head to head, there was some uncertainty about what was better. Some were better in some areas, some were better in others. But when we compared the general cardiologist with access to Amy's assessments to the general

• 15:20Speaker 02

  cardiologist alone, it was like a landslide, right? And in that case, in almost every aspect, it was considered superior when they had assistance. And I can talk a little bit more about maybe why. I think that might be, obviously, I think there's more investigation needed, but I think that is a really exciting aspect of this, that it seems to be just a helpful system in use by these experts.

• 15:43Vivek Natarajan?

  And maybe to just contextualize that work a little bit more, I think if you look at the access to specialists in the country today, I believe for getting consultations on neurologists, for example, it's like 12 to 18 month wait time. And that's simply not sustainable, right? And so the question is, okay, clearly that we have better reasoning AI systems that seem to show promise in medicine. So why do we have, how can we do better over here? Can we improve the status quo? And we should be able to do that radically. No one should be waiting for 18 months to get consultation on a rather serious thing, like neurology is not straightforward. And so that's kind of like the motivation.

• 16:28Vivek Natarajan?

  There's a lot of access issues around specialist care, cost issues around specialist care. And so how can we do better? And that's like the key question that we're trying to address. And then I think the second thing is generally like if you look at like how medicine has evolved. And that has led to these silos or these compartments and specializations. And so you have like this primary care, which is kind of like the front face to everything else. It's the door and then you have all these silos. But the way that it has evolved, I think is primarily because of like the limitation of the cognitive aspects of the human mind. Like there's only so much expertise that we can cram into our given brain. And so like we have to know. So because of those limitations, you have to go like study neurology or cardiology. Or I don't know internal medicine, but not everything together.

• 17:13Vivek Natarajan?

  But like AI systems don't need to have that kind of like limitation. They should be able to like do, I mean, like given what we're seeing, like they should be able to integrate knowledge from multiple different sources, multiple different disciplines. And so, so that's also like, there's a fundamental rethinking that is happening. Does like the new age of AI empowered like AI powered healthcare, does it need those silos? Can it could be possible that you not only have like a PCP in your pocket, but like, like an expert neurologist in your pocket. Obviously with caveats and things like that. So that's, that's another class of like question that we are trying to address over here. And then maybe the third point I'll just add on is this study is like what, three, four, five months old now?

• 17:52Speaker 02

  Yeah.

• 17:53Vivek Natarajan?

  Yeah. So that was primarily done with the palm version of the models, if I'm not wrong.

• 17:57Speaker 02

  The cardiology one was done with flash.

• 18:01Vivek Natarajan?

  Flash, okay. 1.5?

• 18:02Speaker 02

  Flash version, yeah. Yeah, 1.5.

• 18:04Vivek Natarajan?

  Okay. Yeah. So it's 1.5 and so since then had two and 2.5. So, yeah, I mean, in that study, obviously what we saw was that 1.5 was not as good as say at endings. But who knows with the new models?

• 18:18Speaker 02

  Yeah.

• 18:18Nathan Labenz?

  Yeah.

• 18:21Nathan Labenz?

  Yeah, that's an important caveat.

• 18:22Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:27Nathan Labenz?

  Okay. Well, let's do the headlines then for the coscientists because these are, I would say, you know, similarly striking headlines. So coscientist is basically.

• 18:37Nathan Labenz?

  I guess I would describe it as an agent scaffolding type of setup where you and we can get into you know, the granular details, but you know, a lot of we've seen different things like this, you know, before I did an episode with James Zhao, who has kind of a similar thing with the AI virtual lab and the original co-scientist, you know, that was made just within the first couple of weeks of GPT four, which is crazy to think that they, you know, got anything out of that with like 8,000 tokens of context. But you know, even then we're seeing some interesting stuff. But this I would describe as just kind of taking sort of all of the lessons learned about how to make agents work from the last two years and hitting the gas on all of them.

• 19:18Nathan Labenz?

  And then coming up with a system that basically can do science, right? I mean, it's it's not executing the actual physical experiments at this point, but I was really amazed by the different things that you tested the system on. The first one is kind of like three, you know, three levels of challenge of the of the problem that the system is given. The first one is like a relatively well scoped thing that you could kind of grind through. That might be the one that I might be able to do with some real effort. And that was drug repurposing, right? So take a take a drug that's out there, look for other things that it might be useful for. And, you know, you have kind of a combinatorial approach that's like available

• 20:02Nathan Labenz?

  to you. So if you set that up and go through it, you know, systematically and you have like decent judgment around each individual, you know, sub-question that you ask, you can sort of imagine how an AI system would be able to do something like that. But then you go up a level to the second scope of task, which is identifying new therapeutic targets within a particular kind of diseased cell.

• 20:27Nathan Labenz?

  And this is like starting to get at what I've often called, you know, one of the grand challenges in biology, which is just like, what causes what? You know, we've got, obviously, we know that there's like, super complicated causal graphs, you know, going on in the cell of, you know, this promotes this, but inhibits that and yada, yada, yada. And we don't, I don't know if you guys would venture a number, but I've kind of understood broadly that we have like, a long way to go in terms of really mapping that out. Maybe we understand, you know, 10% of what the graph is in today's world. So, the challenge here is to basically go into, you know, go into a cell, so to speak, and try to figure out, can we identify

• 21:09Nathan Labenz?

  something that we can target with a drug that will actually make things better? You know, and obviously, this is hard, given the vast complexity and the many, many unknowns in cells. So, okay, that was just the middle one. Then the third one is, can you figure out why or how bacteria are becoming drug resistant? There was like, a little bit of a hint, I think, because there was, there was one sort of observation, right, that there was something that was conserved across a couple of different species that sort of was, was a notable observation that, you know, sort of served as a seed to unpack the challenge. But beyond that, go figure it out, right? Super open-ended, really, really

• 21:50Nathan Labenz?

  tough. And that is like, a daunting question for me to consider, and really would have you kind of, you know, spinning your wheels, I would think, in just in absolutely vast literature for a long time before you would even, you know, or at least for me, before I would even have any, you know, sense that I might be able to start to contribute to the discussion. Okay, those three problems. Bottom line is, AI is able to do all three of them pretty well. And in that last case, it actually surfaced as its number one candidate idea for the mechanism of the

• 22:31Nathan Labenz?

  Drug resistance, something that had been discovered experimentally, but was not yet published, right? So you guys partnered with an academic group that was doing this research. And they had the answer, but nobody else had the answer. It's not in the literature. And the system was able to grind through all this content that it had available, you know, the whole kind of vast body of of medical and biological literature and

• 22:59Nathan Labenz?

  I mean, crazy, right? Landed on the exact number one hypothesis that turned out to be the actual answer. I was really blown away by just how successful it was. So tell me more, I guess, you know, in terms of what what else? Are there any caveats that I glossed over? Or, you know, are there any other eureka moments that you would want to highlight from those results?

• 23:25Vivek Natarajan?

  I think the last one was kind of like interesting and also like funny because I I don't think Jose and Tiago, who are collaborators at Imperial, would take offense at like me saying this. Because I don't think they really believe that AI could do this thing. So I think we've been trying to like, you know, we said, okay, like we had the system Jose and Tiago, you want to try it? And then it took us like a few months before we got like enough time with them. And then I think after like enough pestering, they were like, okay, we have some experimental results in the lab. We'll just try and challenge your AI system. Let's see what your AI system can do. And so, yeah, I think it was roughly around like Thanksgiving. And so they sent us this prompt, and I think that's detailed in another preprint along, which was co time with the Coscientist paper. And as you said, I think there were some clues in there, but like it was not totally giving it away.

• 24:10Vivek Natarajan?

  And so we take that prompt, we set out this system to it, and it runs for like a couple of days. It splits something out and then like Euro are like the first author on that paper. And he's an amazing like technical fellow. He's probably the least well-known technical genius at Google in some ways, because he likes to keep a low profile. But yeah, so he just sent it over to them and then. I think it was Thanksgiving, I was like late in the evening and so Euro and Alan were both both based in Europe, they were like offline. And then like within ten minutes, like Jose was also based in London, he sends us like an email and he was like, I need to talk to you right now. And I was like, okay, I don't understand the seriousness of it, but I I was like, okay.

• 24:50Vivek Natarajan?

  And I'm not doing anything better, we can talk. And then he was like, Vivek, are you like reading my email? And I was like, I'm not sure, Jose, what are you asking? He was like, no, no, it seems like you're reading your email. And then I was like, um. We do many things at Google, but reading your email is not one of them. Uh. And then he goes into it, he was like, I'm not published this anywhere, but your AI system came up with the same set of results that. Like we hypothesized and we found in our experiments. So I'm like really surprised. And so he was like, okay, do you guys get responses from like chat G P T? I was like, no, no, that's not possible, that doesn't happen. And so he was like, okay, so if you're not reading my emails, and then if you're not, if you don't have like any information from chat G P T, then.

• 25:34Vivek Natarajan?

  It's likely that you have something like really magical. And that was kind of his response. And then he said like, yeah, the first one is great, but you're also like send him four more. And he was like. All the other four also make a lot of sense. And I think immediately after the Thanksgiving break, he was like, okay, I'm going to like set. A few of my postdocs to work on this and they've been like working on validating those other hypotheses. And so it was kind of that moment like where someone who's like very pragmatic, very experienced. I spent like a decade, in fact, several decades on the field, like when they have these kind of reactions, um. So that was kind of like, okay, that told us like, okay, we might be on to something over here. But again, I think like it it was not like one single moment, but it was like, okay.

• 26:17Vivek Natarajan?

  We, it's it's a very hard thing to do, right? I mean, to get like AI to. Not just like synthesize and integrate and summarize information. But help traverse the history of knowledge and like uncover new original things and knowledge and facts about the world. And to do that reliably, that's a super hard problem. In some ways, that's like the holy grail of AI.

• 26:37Vivek Natarajan?

  And to think that, okay, like a system which is relatively simple in nature, like we we'll probably talk about this more, it's like. It's probably the simplest version of the system that you can imagine. And like throwing a bunch of computer at it, you're already seeing evidence of that, like you're seeing evidence of this happening reliably. That felt like, okay, super magical to us.

• 26:57Nathan Labenz?

  Yeah, it's crazy. So this has been something that people have been discussing recently online a bit. Like Dorkesh has sort of advanced the idea that like, why don't we see AIs coming up with these like, you know, they have this like incredible breadth of knowledge. Shouldn't we be seeing more

• 27:19Nathan Labenz?

  connections made, you know, more sort of insights across this like super diverse knowledge base? His contention is like, if I could have all that knowledge, you know, surely I would come up with like more insights than the AIs currently seem to. And I think he even kind of went as far as to say like, I haven't seen a single example of this. And I feel like some of the things that I've covered definitely seem like they could be said to count, but there, you know, there's always a lot of details and caveats and sort of eye to beholder type of stuff. But this seems pretty clear to me that, you know, the fact that this was essentially independently

• 27:59Nathan Labenz?

  discovered by human scientists in a lab and an AI system in a data center over a couple of days in parallel to the point where the scientist accused you of reading his email.

• 28:13Nathan Labenz?

  Presumably not super seriously, but you know, nevertheless.

• 28:17Nathan Labenz?

  You know, is there anything, is there any reason that we should sort of not take this as a genuine like discovery, you know, a sort of qualitative eureka moment from an AI system?

• 28:34Vivek Natarajan?

  Yeah, this is not the first evidence, like I think, right in 2023.

• 28:39Vivek Natarajan?

  Like right after we did our metpom and that's what actually the genesis of this co-scientist work is actually right after we publish metpom. There was this professor from Stanford, Dr. Gary Pelz, who reached out to us and he was like, and I think you probably remember Tao, who came to one of these previous episodes before. He called both of us up and then he was like, Greg and Tao, you don't know me, but your AI system can potentially help like millions of people with rare diseases. I was like, okay, Gary, that's a nice introduction. Please go on. Um, and then he was like, I know that your models are trained on a lot of scientific literature. And I think they can help me discover useful facts about genetic diseases. And, um, yeah, I mean, that felt interesting at first. I mean, if you were able to help him, probably we could help a lot of people. And so we started working with him and on this problem of like genetic discovery.

• 29:24Vivek Natarajan?

  So can like language models, can they come up with like the right kind of causative factors that are responsible for like given combination of phenotypes or symptoms. Um, and, um, so we started doing that with Metpom and even with Metpom and later with Met Gemini. We saw that these systems were able to do this. In fact, with Metpom, we, like Gary was working on like mouse models and like a very specific kind of hearing loss for which he had this NIH grant back then. I don't think he'll ever get that kind of a grant again, but it's a separate discussion. But so, yeah, he had that grant. And so one of the hypotheses that the model came up with, it was like a biogenic model for hearing loss, which Gary had not thought of before. And so he went ahead and did like these CRISPR-NORIN experiments in, in his lab.

• 30:10Vivek Natarajan?

  And he was able to reverse the cause of the disease. And so we had written that up and still under review in, in actually a prestigious venue. And then later on, we used our more advanced version of our LLM, which is Met Gemini. And we extended that work, we extended that work to like human variants of unknown significance. And even there, like based on retrospective data, we see like these systems are able to, to do like pretty interesting work in genetic discovery, which is like one of, it can be cast as a hypothesis generation problem. But the key thing is, at that point of time, when we were using these L L M's in like a, pretty much like a crude, single-shot fashion, it was very unreliable in this act of like hypothesis generation. So literally for that Medpom to to come up with like one hypothesis that was very, very helpful, it had come up with like thousands of things that were utter garbage.

• 30:56Vivek Natarajan?

  But I mean, we were grateful to like work with Gary, who had the expertise to like very quickly discard away things that were nonsense and and also had the patience to like work through all of them because could easily imagine another scientist or maybe someone inexperienced who would like say, looking at the first five, oh, this is utter garbage, it's not working. And then we would not have like pursued this line of work at all. And so it's about like getting together and working with the right people who believe in this, right? And so that like, you know, put us on this journey, like, okay, how do we make this more reliable? Like, we should not be sampling like thousands of times to get something useful. Rather, every single generation should be something useful and the system should be like well calibrated over here. And so, so that kind of like led us to this. And so the the simplest way to do this would be to like, you know, call like an L L M repeatedly

• 31:40Vivek Natarajan?

  and hope that it leads to something useful. And but that very quickly fails and it leads to like these degenerated solutions, which has this mode collapse. And so it became important to like, you know, introduce like net new knowledge into the world. And also think about how you can like gamify that process in some ways and like introduce like repeated new helpful feedback that can help the system self-improve. And so that kind of like led us to the design that we eventually had. And in hindsight, it should be quite obvious. I think it's like very naturally follows how like the scientific method works. Or if you ask like a scientist, how do they come up with new ideas? You'll see like they were just roughly compartmentalized into the set of agents that we had and they're like just doing the same thing. But it was just like this iterative process of like, okay, we tried something,

• 32:22Vivek Natarajan?

  it didn't work, okay, we tried to improve it and then we eventually end up ended up with this design, which is like, I think remarkably intuitive.

• 32:28Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 32:35Nathan Labenz?

  Yeah, how do you, there's a couple of like, maybe quasi philosophical questions that come to mind there. One is, how do you think about the relationship between hallucinations on the one hand, and creativity or hypothesis generation on the other hand? It seems like we have, people have very different intuitions about that. And I'm also not quite sure myself, like, do I, you know, when we do, you know, sort of extensive post-training to try to minimize hallucinations, does that help with hypothesis generation? Because maybe it, you know, makes the models more like disciplined in their reasoning?

• 33:15Nathan Labenz?

  Or in some ways, does it hurt because they're maybe less willing to, you know, come up with a very random idea that, you know, every once in a while, it's these random ideas that are the big ideas, right? So, how do you guys think about the, like, maybe that's a false trade-off, but yeah, how do you think about the relationship between those behaviors?

• 33:38Vivek Natarajan?

  Yeah, I'm curious to hear your take.

• 33:41Speaker 02

  Yeah, I mean, I think intuitively, there's some aspect of like hallucination, like, does foster creativity. In some ways, like, it's interpolating between, you know, the data it's seen, right? And it's kind of necessary for hypothesis generation to deviate from, you know, the script in a bit.

• 34:02Speaker 02

  But, yeah, I mean, like, I don't have a, I don't know if I have a clear understanding or, you know, answer to that, but intuitively at least, that's how I feel.

• 34:13Vivek Natarajan?

  Yeah, I mean, I always used to think that like hallucination and creativity are like, two sides of the same coin in some ways.

• 34:21Vivek Natarajan?

  And I don't think this no longer holds true, but like, what we used to see with the previous generation of models was it was actually much more helpful to use the models without post-training for this task. Because they were then more likely to come up with these crazy ideas, right? But now I don't think that's any longer the case, because we've been able to like systemize, we have been able to like, put a structure around that process of like coming up with new ideas.

• 34:48Vivek Natarajan?

  So it feels like now the process is much more reliable, but maybe we are sacrificing, say, some like crazy new things that would require maybe like an unnerved, like non-post-train model to come up with. So we don't know. So I think we've gained like reliability in the process, like in terms of like consistently coming up with like new original thoughts. But like, we don't know if you're like sacrificing something else over here. Yeah.

• 35:12Speaker 02

  Yeah. And I think one advantage of our system too is like we, we allow like diversity of models, you know, so, you know, we can kind of get the best of both worlds. And through this tournament process, through this ranking, like, you know, we have a diverse set of hypotheses and those can be reranked and hopefully, you know, we see high quality hypotheses kind of bubble up to the top. So.

• 35:32Nathan Labenz?

  So, yeah, I mean, I think maybe it's both is kind of what I take away from the hallucination reflections there. It's like, if you're in a, you know, in the context where there's a rich literature and it's more about, you know, really working your way through it, then maybe you want a disciplined reasoner. And if you're doing something where there really isn't much to go on, you know, maybe you want a sort of more whimsical hallucinator. And as you said, like maybe get the best of both worlds with some of these setups. So let's describe the, the different setups. I mean, there's, there's multiple different systems here that, you know, they, they have their sort of intricacies, but it seems like

• 36:16Nathan Labenz?

  at a high level, if I, if I understand correctly, what you're doing to design these systems is basically just kind of introspecting or, you know, maybe interviewing people and sort of saying, one of my favorite questions in, in AI, you know, automation in general is how do you think about it? So it seems like you're kind of doing that and saying like, okay, how do you think about it? What do you do next? Then, then once you do that, what do you do? And you're basically turning all these steps of a process, whether it's the scientific method, or it's the, you know, diagnostic process, or the reasoning to treatment process. You're basically just kind of mapping that out with, you know, a subject matter expert, yourself or somebody else. And then.

• 37:00Nathan Labenz?

  Creating little subagents that are sort of prompted to do those subtasks and then kind of scaffold them together. And also some interesting details on giving them tools. You know, literature search is obviously a huge one. I was also interested to see that in the one project, the model did have access to AlphaGo and maybe some other things. So, you know, you're increasingly like giving it something like the full complement of tools that a human could use. And then it seems like after that, it's like turn all the hyperparameters up, you know, kind of do more rounds, more generations, more, you know,

• 37:37Nathan Labenz?

  more evaluations, more rounds of feedback. And I guess my impression is that if you do that and you have the budget for it, like in today's world, you could probably be successful at almost anything. But maybe, you know, tell me if that's wrong. Like, are there things about the designs of these systems that you think are like actually very, you know, sort of important hinges where like if you designed it a little different, like it wouldn't work? Like, how sort of, within that general framework that I outlined, could you go like a bunch of different directions? Or do you feel like it actually is sort of

• 38:21Nathan Labenz?

  a narrow design space that actually works?

• 38:26Vivek Natarajan?

  Yeah, I think maybe there's a high level narrative change that's happening over here. So, yeah, this is my fourth time on the podcast. And I think previously we discussed Metpom, I think Metpom M and MetGemini. And the key with all of them were all like med. Like, we were taking some generalized model and then we were trying to like fine tune them and specialize them. I think the key differentiation with the current version of Amy and Co-Scientists is we're no longer trying to do that fine tuning step or that specialization step. And part of it is because like, okay, some of the data that went into that fine tuning and specialization step is now upstreamed and part of Gemini. But it, yeah, I mean, it just feels like a better approach to set things up is by simply having agents with like specialized problems and chain it up together in like a nice manner. So that

• 39:11Vivek Natarajan?

  Yeah, it just gives a lot of like flexibility and control. And so it it does away with this need for like maintaining and specialization. I'm not sure. What do you think about that?

• 39:19Speaker 02

  Yeah, I kind of agree. Um, you know, not that there's no role for post-training, but certainly like in our first AMI paper and like MedPom, all these papers like it was really about, you know, this medical data we're curating or creating and how we're making that. And, you know, that's what was driving the model's success. And in yeah, these latest papers, that's really, I think, taken a backseat to simply like how we're designing the system, you know, maybe to perform these these sort of tasks at inference time. And I would say like, I don't think anything is like particularly like hyper optimized, like, you know, there's probably certain ways that, you know, someone could design a system that does this stuff better. I'm sure I'm sure there is. I think like our goals have been kind of

• 40:02Speaker 02

  largely to build to where it's like a functional prototype for these and and kind of get there quickly and do the kind of tasks that we're interested in doing in each in each study. So I think there are, there's so much room for improvement in all of these.

• 40:16Nathan Labenz?

  How many rounds of iteration did you go through as you built the system? And were there any moments when it was like, oh, we tweaked, you know, this prompt, or we, you know, we we switched this, the or we put this agent before this agent, you know, and kind of re-scaffold it that way. And all of a sudden you saw a big leap? Or?

• 40:37Speaker 02

  Yeah.

• 40:38Nathan Labenz?

  Yeah. So tell me about those.

• 40:40Speaker 02

  I mean, for like the AMI work, I think we rely like really heavily on like auto evaluations. So I think we do try a lot of different, you know, you know, prompting, a lot of different like configurations, but ultimately like, you know, it's only finite how many things you can try before, you know, you're like, this is, this seems good. This is good enough. So I say we use that as like a rough signal. We also like honestly rely pretty heavily on like vibe checks and just like, and and, you know, I, I think like, You can pretty quickly like start to see differences like I think with our like manager reasoning agent. For example, like, you know, we saw a big difference when we started drafting like concurrent plans and then refining them together. We saw a big difference when we did this top down.

• 41:23Speaker 02

  Can you unpack that just a little bit more like what was the before and what was the after that? So like before we were, we were just generating one plan and basically now like we're generating four different plans. They might have some similarities, some differences, but we found basically like, you know, in a sort of like self consistency style manner, right? Like that the model was able to combine these plans in a way that it kind of was able to take the good stuff from each plan and leave out the bad. And that's just something like, once we tried it, it was like our, our internal like auto valuation signal was like very clear. Like this was making a huge difference, but like, you know, in terms of like hyper parameters, it's like we, we went with four plans. Like we could have gone with eight, we could have gone with two, like I, I don't think we optimized every hyper parameter in that sense.

• 42:06Speaker 02

  But I think like the big things we were able to pick up from this kind of signal.

• 42:12Vivek Natarajan?

  Yeah, I think on the co-scientist side, it was, it was a much longer iteration. It was almost like an first an 18 month project in some ways. And it was, yeah, driven by this need to like, okay, make this process of hypothesis generation like more reliable. And so, yeah, it was like, okay, decompose these tasks. We tried to like, get like individual models to work on them. And I think the good thing is again, like what it has shown was how good like individual algorithms are getting at instruction following. And that also does a way they need to like fine tune and specialize and whatever, because simply like if a model has the knowledge. And if you give it like a precise set of instructions, it can just follow it. And so that just makes it much, much easier to like create agents and chain them up together where these agents are specialized to do like specific tasks.

• 42:57Vivek Natarajan?

  But I like just prompted versions of like the general purpose models. And so, yeah, I mean, like it was a process of like, okay, setting up some evals, seeing how well the system does, where the weaknesses are. And then like again, iteratively going in and fixing those weaknesses by adding like specialized versions of agents or like fixing the prompts. And then ultimately like there was a process of like, okay, let's just simplify the architecture. We don't need all of these things. And so, yeah, that ultimately led to this design. So I would say, yeah, it was a lot of iteration, like figuring out where the weaknesses are, trying to cover those weaknesses. And then at the end, like, okay, let's do a simplification push.

• 43:32Nathan Labenz?

  Yeah, I I've been through that on the less world-changing project myself multiple times where you're like, a new model can do wonders for the simplification of your system. Yeah, I've definitely lived that. So I guess, you know, in terms of what is driving the the improvements, you know, eighteen months ago, we maybe like couldn't, I mean, it's all happening so fast, right? Like, I even have some, you know, lingering sympathy for the stochastic parrot people because it's like, as of GPT-2, you know, that was probably still mostly true. At this point, you know, it's it's pretty clearly not. But eighteen

• 44:14Nathan Labenz?

  months ago, you know, you maybe couldn't, you know, models just couldn't do certain things that now they can. Seems like just core model progress is just is the tide that is like, you know, lifting all boats dramatically. Is there anything else that you think is just like super important? Or is it really just down to foundation models getting better and just grinding out the process of figuring out how to use them?

• 44:43Vivek Natarajan?

  Yeah, again, I think on the co-scientist side, I feel like long context was an important part of it as well. Because we don't have an explicit memory store in the system. But the fact that Gemini models instantiated can take up to like 2 million tokens, it means that we can just like, you know, generate ideas, generate reviews of them, run debates of these ideas, and like generate these walls of text essentially, which can just become feedback. And this just put all of them back into the context of the model in the next round. And the model just figures out how to make sense of it and use the feedback in a very like implicit manner to improve. Like, if you did not have these long context abilities, like the ability to reason over like reliably over like millions of tokens, then you would not be able to do that. Like you would have to engineer like

• 45:28Vivek Natarajan?

  rag based systems and maybe you have to like end to end train them and that would have been a lot more complex. But like all that has been made like remarkably simple just by these long context of Gemini, which I think is a little bit underappreciated in the field because we don't have like enough of these. I don't know, Studio Ghibli style viral moments with long context, but it just enables a lot of these practical applications. I think that's the same bit.

• 45:48Speaker 02

  Yeah, with the management reasoning, it's like, yeah, I mean, like our recall for for that paper, right? Like the the management agent, its whole point is it's trying to, you know, take in guidelines and reason over those to produce management plans and like, yeah, I think if we were like solely reliant on, you know, always picking out the right guidelines, like we would struggle, right? But you know, with the long context, like we don't really need to worry about it, right? We we take in a bunch of guidelines and, you know, with with, you know, two fifty six K, right? Like, we're gonna catch the something relevant, you know, regardless of whether our retrieval system would have been able to do that. So yeah, I definitely agree on on long context. I think just like, like with co-scientists in particular, like we're talking about like a really long time scale. We have this thing running for for days, right? Like,

• 46:32Speaker 02

  I think just like that inference compute like increase is offering a lot of benefit as well.

• 46:39Nathan Labenz?

  I'm just looking up a friend sent me um just in the last day or two um

• 46:46Nathan Labenz?

  fiction live bench for long context, deep comprehension, you know, one of obviously many benchmarks that look at these things. And the Gemini two point five, even relative to Gemini two, is absolutely just crushing on its command of long context. And I've definitely felt that in my initial testing of it. This does feel like something that's like hard to go viral. It's like hard to go viral with a notion of like, you know, I had hundreds of thousands or a million, you know, tokens of context. It's all like very idiosyncratic or whatever I'm working on. You know, hard for people to like even know what you're talking about, right? When you post that on Twitter, the the contrast between the context window and the, you know, the length of a tweet is like pretty, pretty severe. But.

• 47:33Nathan Labenz?

  I mean, it's amazing that that is that that was working that well because just looking at these benchmark results, it's like 2.5 stands out in a massive way relative to everything that had come before. Did you, I guess, like qualitatively or vibes or, you know, your own sort of sense, I would have said up until 2.5, I would have felt like, eh, I'm not so sure if I can just dump like hundreds of thousands of tokens in and like, yes, it can handle it, but does it really handle it? Does it really have command? And especially if it's like material that I don't have full command of myself, you know, it can be very hard to evaluate that. So I guess, how did you handle that? And how did you?

• 48:14Nathan Labenz?

  How did you know if it was like actually, you know, making effective use of the super long context?

• 48:21Speaker 02

  Yeah, I mean, like in the Amy setting, I think, like I mentioned before, like we, we really relied on auto evaluation and like I have to shout out like Valentin, my team of around the paper did like a great job setting this up, but essentially like we also didn't know like how well would it, how, how well would this work? We stuff a bunch of, you know, guidelines into context. And we found like, you know, initially we were going to go all the way up to 1 million. We found that actually like we seem to be getting better performance when we drop it down to 256K. But, you know, ultimately like there was a clear difference between the presence of, of, you know, that, that much knowledge versus, you know, trying to do a zero shot or with, with, you know, one guideline or some retrieval or something. So it was just, we were also uncertain, but we, you know, we tested it internally at least and that seemed to work.

• 49:05Vivek Natarajan?

  Yeah, I think on the co-scientist side, it might be a little bit more unscientific. I think what we relied on was primarily the redundancy aspect. Because like you generate some ideas, you review them, there's a tournament that happens, you get like a bunch of feedback from the tournament. And then you, when you put that back into the state of the system, like you're not generating only once, you're like generating like numerous ideas. And so your hope is because of that redundancy that creates like at least one of the generations would catch the key elements of the feedback that has been generated and propagated back over here into the system. So, yeah, I wouldn't say we have like any specific Evas that target measuring how well the long context is doing. But rather by engineering in this redundancy, we're hoping that it would be effective.

• 49:50Vivek Natarajan?

  And, and yeah, and then the other distinguishing factor for us in this work was at least let I feel like there's a lot of like science assistant style, like scientific discovery style projects. Not just, I think in a lot of different places, but where they do get a little bit hung up is in like curating these really nice cozy benchmarks where you can hill climb on. And that was not our philosophy at all. For for us, we feel like the key deal is like if a system does something useful, we should rather sprint straight ahead so that like we can validate it in the lab. And then hopefully take that onward to us like a real meaningful discovery. And so that was what we were most focused on. Like if we engineer a system, we go straight to the scientists who's like an expert in the field. We show them the idea and if they like it, we

• 50:31Vivek Natarajan?

  like try to convince them to like validate it. And then if they like validate it, it becomes a discovery, then yeah, great. So so there was that thing where we like really wanted to like not micro optimize on like specific benchmarks and hill climb on them and like, you know, wait for like. I mean, if I think if you did that, because there's like so many different components in the system. I think this work would have taken easily another year if you try to like, you know, make it the best. Rather, we were just focused on let's system all up together. Let's get it to do something reasonable and then we'll do like end to end validation.

• 51:00Nathan Labenz?

  Yeah, okay, that's. It's a brave new world out there.

• 51:06Nathan Labenz?

  One thing that I had

• 51:09Nathan Labenz?

  seems to be changing now, like not too long ago, I would have said, I could have cited, you know, a handful of papers that sort of showed this. That like typically three to six rounds of self critique and, you know, sort of auto self improvement seem to be where like G P T four would kind of max out. And if anything beyond that, usually I felt like I would see that, you know, performance would decline after if you kept running it longer than that. You guys are talking about running these things for days. Maybe you could tell us a little bit of like how, you know, what are the budgets for this? Like, how many tokens are we talking about? And how many, you know, if we if we translate that to retail price, you know, what is a

• 51:50Nathan Labenz?

  you know, what would the inference bill be for, you know, finding the the mechanism for micro, you know, drug resistance? And is there any limit at this point to how many rounds of this you can run? Or is it, you know, are we already at the point where you can just like run the thing for longer and longer, sort of potentially indefinitely?

• 52:15Vivek Natarajan?

  Yeah, I think this is, it's a fascinating question. It's something that like has also like kind of like, I don't, I wouldn't say bothered me, but it's like intrigued me as well. Because I remember reading one of these Andre Karpathy's posts where he talks about like leaving like a CNN training over like the winter break. And then it magically led to like state of the art performance on some benchmark. I forget which one that one was, but all that he had to do was like they had to run for like forty days. Which is a lot of compute back then, but like it was like unprecedented in some ways. So yeah, I think for us, like with the co-scientist, I think the key thing is the fact that the system is not like closed looped. The fact that it has access to these different kinds of tools means that in every round of, round of like self critique or like attrition, it can bring in new information to the system.

• 53:00Vivek Natarajan?

  That increases the entropy. Uh, and when that happens, that prevents us the, the possibility of like mode collapse and like degenerate solutions. From happening. So I think that is the key thing. So, uh, the fact that like the system can go and browse web search, browse like interesting parts of the web. Take information out of it and integrate that like knowledge that it has and do that in like an effective manner. I think that is the part that leads to like, okay, more computation being spent like. Efficiently and effectively over here and helping. And it's not just web search, right? I think like increasingly we'll be able to like get feedback from like other kinds of like. Knowledge bases, specialized tools, alpha fold. And so more as the, the quality or like the, the surface area of the hypotheses like.

• 53:44Vivek Natarajan?

  Increased, like the more different kinds of feedback, we'll be able to plug that into the system. And I expect that it wouldn't then like, you know. Mode collapse and there's likely that there's going to be like even more increasing. Value to like spending more time on computation in this setup over here. But if you were to like strip away that, then I think it comes down to okay, like the quality of like published information in any given domain and also down to the complexity of the problem. If like the prob, again, I don't know how to put like a precise definition to it, but maybe you know. But if there's a problem where there's like a, a clear unknown that is like impossible to solve. And I think it's very likely that no amount of computation reasoning, test and compute is going to be able to like get you that information. If the system does not have the capacity to get that information. Um. And on the other hand, there can be like problems which are like very trivial that

• 54:29Vivek Natarajan?

  it doesn't matter like if you it you probably get it in the first or like a few tries and it doesn't matter. So I think it there's there's that sweet spot over here where that problem is like within limits. But it's also not trivially easy where spending this computation helps. And my hypothesis is I feel like a good chunk of problems that we as humanity care about today actually fall in that sweet spot. Where I think we can spend a lot of computation within Silico and like get like very useful, interesting ideas and answers. Which is I think exciting. And what do you think?

• 55:01Speaker 02

  Yeah, I mean, just like a small thing to add is like, I think, you know, also like our system allows for humans to input ideas like there's like other avenues to to to kind of add to this entropy. I think one one other like important thing to think about is like we're comparing like pairwise like every, you know, combination of hypotheses like there is a lot of like, you know, variability when you compare it to like maybe other papers that are talking about self critique in a more like we're just gonna keep trying to improve the same idea like there's there's many directions where we're getting this this variance.

• 55:34Nathan Labenz?

  Yeah, that's quite insightful. So the and I don't know, do you actually know the like total number of tokens for the microbe project? I'm gonna guess it was like ten billion or something.

• 55:47Vivek Natarajan?

  No, no, I don't think it's that bad. It's probably it should fit in within the context limits of these models. I haven't done an exact analysis, but I would think it's less than ten million.

• 55:57Nathan Labenz?

  Ten million total inference tokens for the whole thing?

• 56:00Vivek Natarajan?

  Yeah, I think so.

• 56:02Speaker 02

  For the whole tournament?

• 56:04Vivek Natarajan?

  Oh, for the whole. No, no, okay.

• 56:05Nathan Labenz?

  Yeah, I mean, like from the time you give it the question to the time that it spits out your answer, like, what would my API be?

• 56:12Vivek Natarajan?

  More than it might be a little bit more than that, but we did some back of the math calculations over here and

• 56:18Vivek Natarajan?

  based on like current prices on GCP, we expect that most queries would be just a few dollars less than ten dollars. Most queries, including all the tournament and everything influences, so it should be fairly feasible. And it's probably just going to come down more and more in the next six, twelve, eighteen months.

• 56:35Nathan Labenz?

  Yeah, that's really, especially if the performance per token is also continuing to go up. There's a lot of tailwinds.

• 56:41Vivek Natarajan?

  And maybe the other thing I would say is that this is probably the most dumb and inefficient version of the system that we have. And I think we can, there's like so many things over here which we could like improve in efficiency. And also from like an intelligence perspective, we can make like much better. And so I think the bang that you would get on each token generated and dollars spent is going to be much, much more as we keep on improving the efficiency and capabilities of the system.

• 57:05Nathan Labenz?

  Are there other, you know, narrow specialist models that it has? And how important is that now? And how important do you think that will be in the future as like a source of new entropy? Because that seems like a potentially dramatic unlock, right? I mean, the model itself is already trained on like most of the literature to search the literature again at runtime, like helps with grounding, helps with things that, you know, maybe wasn't in the training data that it can find after the cutoff date, what have you. But like the ability to actually go do simulations and bring that kind of information back that literally maybe nobody's ever run that simulation before at all. That seems like a potentially pretty big step change.

• 57:47Vivek Natarajan?

  Yeah, I totally agree. And that's why I feel like this is probably like day zero or day one of this journey in many ways, because we primarily scratched the surface of information that is just written down in papers and peer reviewed and published. But like a lot of like scientific information is just not in that format. So for example, we don't like publish negative results, which because of like just incentives and scientific publication. So that's like kind of like some dark matter that's hidden away. And you'll have to figure out like a good incentive mechanism to help that kind of data also flow into the system. But I think that's going to be important as well going ahead. But yeah, I think more excitingly is the fact that like a lot of these papers and publications will have this supplementary data file, for example, which contains these like giant data sets of experimental data.

• 58:33Vivek Natarajan?

  And they contain a lot of like useful nuggets of information, which again, like you can imagine like a system like this, pad up with like another system that came up, like a data science agent, I don't know if you saw, like it could like automatically go and analyze the data. And then this kind of like generated hypothesis, the data science agent can go analyze the data and like get the right kind of feedback back into the system to improve. So I think certainly I like small scale data sets that can happen automatically. But I'm even more excited about like what we might be able to do when we pair up the coscientist elements of the data science agent and like go after say, so our institute recently came up with this virtual cell atlas. It's like three hundred million for like gene perturbations for like three hundred million cells.

• 59:12Vivek Natarajan?

  That's like such a vast space. It's even for humans, it like teams of humans, it would take like years to like explore and look at the data and see the richness in there and come up with interesting insights. But we could set up these agents to like generate hypotheses, look at the data, analyze, come back with feedback. And so I'm just super excited about the insights that it would unlock into like basic biology, you know, target discovery and things like that. When we pair up these systems and set them to go on like these giant data sets that we are generating right now, which like even teams of humans, it's like like very impractical to like go after and analyze right now.

• 59:49Nathan Labenz?

  Do you think, okay, here's a very high concept question that you may think is totally misguided, or you might think it's the future. Obviously, over the last three years, we've seen a dramatic convergence of like a couple core modalities, language, vision, speech, right? For me, one of the like earlier Rico moments where I was like, I think I'm going to study this, you know, subject for the rest of my life, or at least until the singularity, was when I realized that it's basically the same architecture doing all these things. And at the time, I was like trying to make, you know, a video generation product work. And I had all these like different specialist models, but I could see pretty clearly that if these, you know, fundamentally similar architectures can do these different tasks independently,

• 1:00:34Nathan Labenz?

  then there's going to be some integration that's going to happen and they'll, you know, the single model will be able to do them all. Now, you know, and this has come, you know, with a very sort of, you know, viral moment both from Flash and from GPT-4O in the last two weeks. Now it seems like That might happen again with the reasoning models and the narrow specialist models. Like, right now, you have a model calling AlphaFold, getting results. But there's not, you know, this would be akin to an earlier language model, like calling DALL-E, you know, to generate a, an image.

• 1:01:13Nathan Labenz?

  And there's this sort of very lossy language bottleneck that happens there that, you know, was definitely a point of like major frustration for people trying to generate images that looked like they wanted them to look. And now that this integration has happened, it's like, that problem is, is basically no more. So the question is, do you think that that will happen for these other modalities as well? Like, my sense is that an early superintelligence might be reasoning models akin to what we have today that actually have these other modalities integrated in a more deep way, so that they are not bottlenecked through like API calls,

• 1:01:53Nathan Labenz?

  but are actually able to start to do some of this like reasoning in biological space, or in material science space, or in, you know, transcriptome space, or whatever, there's a lot of spaces out there, obviously. And it seems like we already see like pretty superhuman performance by the narrow models. It's like, I'm not aware of any human that can look at a amino acid sequence and like intuit what the shape is going to be. There may be a couple savants out there, but certainly, you know, it's not common. Do you think that happens, or, you know, what's your, what's your reaction to that possibility?

• 1:02:28Speaker 02

  Yeah, I'm not sure. I'm not sure. I mean, to be, to be honest, I think that's, I think the API call thing is like, clearly, I think we're very close to, to, you know, being able to do that pretty well. I think a deeper integration, yeah, I don't, I don't know exactly how easy that is for, for all of these kind of specialist type areas.

• 1:02:49Vivek Natarajan?

  Yeah, um, I think it actually depends on some of the incentives of like people who are drafting these frontier labs models. And it makes a lot of sense to combine speech and vision and language. But if you still look at the data that's going into these models, it's primarily non-specialized public data. And then when you're thinking about like biology modalities and data sets, they are not as close to like the kind of data sets that are going in over here from the public internet over here. They're very, very different. And from like some personal experience, what we have seen is when you try to introduce,

• 1:03:31Vivek Natarajan?

  say, some of these most special modalities, even if it's medical imaging modalities, right? It's like images, natural images, and then you have medical images. They lead to like regressions and like benchmark performance. And so like then the question becomes, okay, are you willing to sacrifice some regressions on say LMCS or like some other like benchmark that is generally like considered like an important benchmark? To have your system have like a little bit more capability in like medical images, but like it's going to be obviously not be as big a fraction of your users. So I think it's a it's a question of like incentives right now, and I can see this tension in like many of the frontier model companies where when you introduce these interesting new modalities of information, it is it leads to sacrifice in other areas.

• 1:04:14Vivek Natarajan?

  And so you're sometimes like sacrificing benchmark performance. I would argue that's a good thing. It doesn't matter so much these benchmark performance, I think you should aim for like practical utility. But in the absence of like clear measures of like utility and it becomes difficult to convince. And so, so I think it's not a question of like, can we do it? I think the architectures exist, that even the computer exists at most of these places. But do the incentives exist? That part I don't know, and I don't know when that'll happen because it's unclear. I think today I cannot articulate, like at a high level, I think like we all agree that if we were to, you know, encode the biomedical universe, that model should be able to do a lot of interesting things. But that is sometimes conflicting with like benchmark performance on like, I don't know, like LMCS or whatever else that you want to use right now. And then so it just becomes like question of incentives.

• 1:05:01Nathan Labenz?

  Yeah, okay, that's reminds me of I've brought this up a couple of times as I'll keep it brief. But I heard Yi Tay from he was at Reca at the time on the Late in Space podcast. Talk about how like the separation between vision and language models was sort of a reflection of like the research history. And, you know, you at one point would have like a language team and a vision team. And then it was like, maybe we can bridge these together. But then you would have like late fusion models because those things would already be sort of done and baked. And, you know, now can we like get them to sort of talk to each other via cross attention or whatever. And then it sort of became like, well, if this works, you know, probably work even better if we do it all just kind of, you know, interwoven data sets from the beginning.

• 1:05:44Nathan Labenz?

  And I can see how that, you know, that same thing might be about to play out again. I hadn't really heard so much the the benchmark thing or you you're saying like you've observed that in, for example, adding image capability, like standard benchmarks do decline?

• 1:06:07Vivek Natarajan?

  Yeah, I can't give you full details, but like when you've tried like adding medical images or like say genomic information and try to like, again, it depends on how you train these models, but like we've tried to do things like in a pretty standardized, but maybe also like the easiest way. And we've seen like, okay, like while you obviously like on the the benchmarks that are reflective of the numerities that you're adding, whether the performance goes up, you're sacrificing performance on like the main original benchmarks that are like language understanding focused or vision focused. And so, yeah, you have to do this prior to optimization. And again, that also requires computing in its own ways. And so.

• 1:06:40Nathan Labenz?

  But that's all in like a late fusion paradigm? Like you're starting with a model that's already like scoring on benchmarks and then trying to kind of?

• 1:06:49Vivek Natarajan?

  Yeah, I mean, the easiest thing to do is like do continued pre-training or SFT. So it's, yeah, I mean, like the experiments that I'm talking about are not like late fusion, but it's like more continued pre-training and SFT over here. But obviously, the better thing to do would be to put all that data back into the pre-training and train all of them together. But to motivate that kind of like undertaking would require you to show some benefits. And so typically how this works is you take like a pre-trained checkpoint, do like continued pre-training or SFT. And then if you show that, okay, like your new data is helping improve performance on like benchmarks that everyone else cares about, then that data goes back in. But it's a. Yeah, and so that is where I like, I think there's these tensions because these are like super esoteric modalities. And so, I mean, we could spend all the time doing that Pareto optimization, throwing computer

• 1:07:33Vivek Natarajan?

  and trying to figure out the best combination. Or we take this other approach where we say, okay, doesn't matter. We'll train our own models, we'll train our own agents and then we like system up together. And so I think that's where we are at right now. It's a little bit of a local optimum. But I think it's still fine because it still allows us to do a lot of these, a lot of these interesting things.

• 1:07:49Nathan Labenz?

  Yeah, the proof is in the pudding. That's really, I, that's a, I'm gonna remember to come back and, you know, ask a follow up on that next time you're here. So, let's see, not too much time left and, you know, there's so much in these papers that we could cover. One thing for kind of practical utility, maybe a couple of things for practical utility for just people that are building their own systems out there. One is that this tournament-style head-to-head evaluation, I've seen like, seems to be kind of becoming like a industry standard. I don't know if you'd go quite that far, but I, there's a strong trend, it seems, toward trying to surface the best ideas by doing pairwise comparisons and having some sort of,

• 1:08:34Nathan Labenz?

  you know, World Cup-style round robin approach to doing that. Wonder if you'd add anything else to that? And then I also wanted to talk about structured reasoning as opposed to chain of thought, because I think that is one that a lot of people listening could probably go apply to their projects and get a boost, you know, like tomorrow. So, yeah, maybe unpack those two things.

• 1:08:59Vivek Natarajan?

  Yeah, I think the tournament one is interesting because, hey, we were actually motivated by AlphaStar, where we had, you had these tournament of like agents competing and that led to like a lot of like strong results in that setting. I don't know if it's an industry standard because I also feel like it's somewhat inefficient and it hints at like the limitations of these models in some ways where they're not maybe able to like. Independently like score and verify ideas, but I have to do these like N square power wise comparisons. And so we do have to do some optimizations where we have to like cluster them up together, group them up together. So so as to reduce the computation and not like do like thousand cross thousand idea comparison because that would be like too expensive. So I would think that would become a little bit like computationally efficient going ahead.

• 1:09:43Vivek Natarajan?

  But it is like, I think that overall ranking of things, the more you can do that in latent space, I think it's going to lead to like more interesting results, better reasoning and so on and so forth. So, yeah, I expect that that idea to stay, but to not happen as explicitly as it's happening right now, but more of it happening in like the latent space of like the reasoning of these models. I'm not sure if you know that.

• 1:10:04Speaker 02

  Yeah. And I mean, I think regarding like the structured reasoning approach, like I think there's like practical reasons for it, right? Like when we have, you know, two agents that we want to talk to each other, like it helps to have like a data structure that we're passing, right? And that just makes like the engineering of it itself like a little bit easier. But I think beyond that, comparing to like, just like asking the model to like, you know, in a chain of thought style, like just reason about this versus defining a certain like reasoning structure. I think the advantage there is we can actually kind of better enforce it to follow a certain path. So like in our case, right, we wanted it to do like this long analysis before kind of going into like higher level management goals before then finally like forming its

• 1:10:48Speaker 02

  management plan and like being able to define that structure, force the model to take that path through its reasoning rather than, you know, free form, you know, if we if we allow it to do it free form, like maybe it starts to form its management plan before it is done these kind of like higher level steps that we want it to go through.

• 1:11:06Nathan Labenz?

  Yes, it's basically to to just try to describe this for people that might want to implement it. I mean, for one thing, it's it takes advantage of another, you know, notable feature that models have gained over the last year or so, which is that we can now specify as part of an API call, this is the exact JSON data structure that you are supposed to return. So that's huge because it makes it really easy to set, you know, set that up and then get something back. It kind of is a little bit like airline checklist thing, sort of where you're just like, okay, I want you to absolutely every time go through these steps. And if you do that, you know, we're confident you're going to get better results in the end

• 1:11:50Nathan Labenz?

  versus just kind of, you know, walking out there and, you know, randomly walking around the plane and coming back and saying, yeah, it all looks good to me. Right. So the intuition for that is pretty simple. How dynamic did you make that? Cause I've never actually done a dynamic structured output.

• 1:12:09Speaker 02

  Yeah, I think in our case, like we, I think we, we, I mean, we tried a lot of different structures and like strategies for, you know, generating the ultimate management plan. I think there were like, when we tried to get a little too like fine grained with like, you know, okay, first like summarize the patients, like, you know, chief complaint, then summarize, like when we tried to get like a little too granular with that, I think it's, I think, I think having the flexibility of like, just these like kind of higher level things, like analysis, management goals, like things that are pretty general tended to lead to better management plans. Of course, this is all under our own auto evaluation and vibe checks, as I mentioned.

• 1:12:52Speaker 02

  So, you know, that's maybe up for debate, but I think, yeah. And I don't think it was like super, so it was, it was dynamic in that sense. Like, you know, there, it can have any number of analyses items. It's like a, you know, a list of however many items, any number of management goals. But I think we try not to constrain it too much, just constrained to the point of like, we want it to go through a certain like reasoning structure.

• 1:13:15Nathan Labenz?

  Yeah. Okay. So structured outputs, people don't overdo it, but definitely use it. I mean, that, that was one of the things that seemed like it really drove a pretty, pretty big lift. Going back just to the comparisons for, for one more second, Vivek, are you basically saying that like, you think in the future it won't have to be so head to head and instead it'll just be like, here's 10 things, pick the best?

• 1:13:40Vivek Natarajan?

  Yeah, I would hope that the so in some ways what we're trying to do is like force the model to explicitly do the tree search, right? You know, come up with the new ideas, like go to different nodes and then do the comparison over here. But maybe what I'm so it's almost like, okay, like you come up with ideas, you write them down, then you review them and then you figure out what's the best that's happening. But like, the question is, okay, can all of that happen in your head in some ways? Like, does it have to be explicitly written down? Do you have to explicitly generate all those tokens? Can you do something in the way you set up the architecture itself or? Yeah, in some other mechanism where that all of that happens in like the latent space itself so that you are more efficient with the tokens that you are generating. So I feel there's an inefficiency right now. I'm sure it helps with like interpretability and other different other aspects over here.

• 1:14:25Vivek Natarajan?

  But I think there's like a lot to be gained by encoding that research within the latent reasoning of the models and we don't do that as well right now.

• 1:14:34Nathan Labenz?

  Yeah, okay, cool. That's helpful. Maybe the last two things. What do you think would happen if you just pre-pended a question identification agent to the co-scientist? And just had the thing, you know, kind of

• 1:14:55Nathan Labenz?

  you know, run in a loop where like the first thing it did instead of taking a question from a human scientist is just like go out on the internet and, you know, search around and come up with an interesting question for itself. And then just try to answer its own question. Is there anything that about that that you think wouldn't be effective?

• 1:15:16Vivek Natarajan?

  I feel like in some ways, like we like to talk about the concept of like root node problems at Google and DeepMind and we felt like, like once you have a question, like generating novel original solutions to that is a root node problem. But in some ways what you are Describing over here is an even upstream root no problem. Like, how do you ask the right question? And I feel like the day we get AI systems to reach that, then I think that is the day we can truly say like, okay, we have geniuses in data centers. I think that is like the going to be the most impactful and important unlock. And I mean, my feeling is like, there really should be like decent capability in these models to like, you know, go surf the internet, read information, and figure out what are the right questions to ask.

• 1:15:59Vivek Natarajan?

  So yeah, I mean, if it's okay, we'll come and give it a try and get back to you.

• 1:16:04Nathan Labenz?

  Have you had a chance, this is a bonus. How have you had a chance yet to try this with Gemini 2.5? It seems like from my qualitative assessment, it would be a lot better.

• 1:16:18Vivek Natarajan?

  Yeah, no, I think that's the exciting part because whatever we've described in the paper was all Gemini 2.0. So it should be coming up very soon, but yeah, we're super excited about that.

• 1:16:27Nathan Labenz?

  Feels like the path to geniuses in a data center is honestly pretty clear at this point, which is a crazy thing to say, but. You know, do you see, are there any like, you know, big barrier questions that you feel are?

• 1:16:43Nathan Labenz?

  You know, just fundamentally unanswered still? You know, programmers often call it a simple matter of programming, like it's going to be work, but you know, we can make it work. Is that kind of the mindset right now for you guys? Or are there questions where you're like, we really don't have a good answer to how we're going to get over that part?

• 1:17:03Vivek Natarajan?

  Like, we have all the building blocks over here. And so we'll probably build something that will look very close to what you're talking about over here. Whether that's the most beautiful one, the most elegant one, we don't know, but does that even matter? I think it doesn't. So I think that is why it's truly exciting, where I think we have line of sight to one solution, which feels like will get us where we want to. And that in turn is going to like lead to a lot of new unlocks. And so, yeah, I would say, I think for the next couple of years, at least it, to me, feels mostly an engineering. Challenge rather than like trying to answer some fundamental unknowns.

• 1:17:35Nathan Labenz?

  Yeah. And of course, the other big challenge is going to be the social challenge of introducing this stuff to the world and, you know, getting scientists to pay attention. Maybe to close, you want to talk a little bit about what you're doing in that regard? Because I was excited to see that we're now getting to the point where you're inviting scientists to reach out and and partner with you on this and also going into, I don't know if you would officially call it a clinical trial, but like something, you know, in the actual field of medical practice with real patients too. So, you know, tell us what you're doing on the deployment side.

• 1:18:08Vivek Natarajan?

  Yeah, I think the coscientist is a little bit easier for us to deploy. I think there are maybe less questions around like regulatory and things like that. And so it's, so I mean, the only thing that maybe like bothers us a little bit is that the system is highly capable. And so there are also like many ways in which it can possibly not do so well, right? And so we just want to ensure that like, as we scale up the system, we do that in a responsible manner. And so that's why, like, you know, we have this trusted tester program, we've already been working with like, close to a hundred scientists right now across the world. And these are all like world leading experts. And with the trusted tester, we want to invite like more organizations. And so our hope is that like,

• 1:18:50Vivek Natarajan?

  we can do this in like batches and in waves. And then every batch, we get like feedback, we identify like the weak points of the system, we improve it and make it better for the next next batch of scientists. And so, yeah, this shouldn't take too long, I think. Like by the end of the year, I'm hoping that, yeah, if I'm optimistic, like millions of scientists around the world will have access to this tool. And hopefully it like raises the bar, like the ceiling for all of them and like helps them to do like more creative and interesting work. I think the the one with Amy is a little bit more tricky. It's obviously a more complex space. But again, there's a part, there's like known path to taking such systems out there in the real world that can like give diagnosis and treatment recommendations. And so we are very excited about the clinical trial that we have coming up. So it's going to be, I think, one of the first studies of its kind where an L L M based system is

• 1:19:35Vivek Natarajan?

  going to be interacting with real patients. And the nice thing about this setup that we have right now is we are deploying it in the clinic where. There is sufficient presence of clinical experts who can oversee the system and provide oversight. And so that is a very safe environment for us to like deploy the system where there's like sufficient number of doctors who can take over if something goes wrong. And our hope is in that study, like not a lot like goes wrong. So that allows us to dial down the amount of like oversight that's needed. And so yeah, I think if things go well, then we'll probably scale it out to more centers, like reduce the amount of expertise and also introduce like more net new capabilities into the system that we are going to trial and like make them more patient facing. So yeah, I think that's, that's the exciting part where I feel like, like the research has progressed quite a bit. And so it's now time to like see, like take it for a drive in the

• 1:20:20Vivek Natarajan?

  real world.

• 1:20:22Nathan Labenz?

  It's exciting times, guys. Really outstanding work. People should be paying more attention than they are and hopefully we'll, you know, put a little, you know, dent in the consciousness by bringing, you know, some more attention to this, but just really mind blowing, really mind blowing stuff and, and, you know, quite a series of work that you guys have put out. Anything else you want to share in parting or any other thoughts you want to leave people with?

• 1:20:46Speaker 02

  No, I mean, it's been a pleasure to work on these projects, obviously, and we have a lot cooking still. So, you know, I'm excited, excited for the future of this.

• 1:20:54Vivek Natarajan?

  Yeah. Yeah. And likewise, I think for me, like talking to Nathan, it's like just, just a lot of fun. And yeah, I mean, it's a real pleasure, like four time over here. I think there's going to be at least like one more where it's big enough that we'll come back again and talk about it next few months. But yeah.

• 1:21:08Nathan Labenz?

  I just hope you don't get too big for me. That's my, that's my hope.

• 1:21:11Vivek Natarajan?

  Obviously here, I think it's so much fun.

• 1:21:14Nathan Labenz?

  Cool. Well, really appreciate it. Again, fantastic work. Vivek Natarajan and Anil Palepu, thank you for being part of the cognitive revolution.

• 1:21:22Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.