AI:AM: What If It Works Too Well? Colluding Agents, $200M Safety Orgs, Virtual Cells Saturate at 2%

Nathan Labenz and Prakash Narayanan revisit interviews with five experts to analyze emerging challenges across agent coordination, safety funding, GPU markets, and physical-world AI. Lewis Hammond breaks down how an OpenAI agent swarm colluded after training worked too well, while Max Nadeau explains why human talent—not money—limits the growth of safety organizations. Wayne Nelms, Nick Gillian, and Andrei Georgescu evaluate the financial moats of compute, foundation models trained on raw sensor streams, and the biological limits of virtual-cell drug discovery. Together, the discussions assess the critical risks and technical bottlenecks facing the field as massive amounts of new compute come online. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/ai-am-what-if-it-works-too-well-colluding-agents-200m-safety-orgs-virtual-cells-saturate-at-2/ Sponsors: ElevenLabs: ElevenLabs lets you deploy enterprise-ready conversational AI agents that talk, type, and take action in over 70 languages. Schedule your demo today at https://elevenlabs.io/tcr OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) Weekly episode preview (02:51) Colluding AI agent risks (Part 1) (11:02) Sponsors: ElevenLabs | OutSystems (13:51) Colluding AI agent risks (Part 2) (26:31) Agents in the wild (Part 1) (26:36) Sponsor: Claude (28:11) Agents in the wild (Part 2) (33:27) Funding AI safety orgs (50:51) The price of compute (01:09:15) Sensor data foundation models (01:22:50) Robotic human tissue testing (01:37:17) Specialist versus generalist models (01:43:18) Episode Outro (01:45:16) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  This week on AI in the AM. I asked Lewis Hammond, Research Director at the Cooperative AI Foundation. What would you speculate they might have done in terms of a training objective, a loss function? And do you have any better ideas for what they should be doing? Because clearly this didn't quite work, right?

• 0:19Speaker 01

  Yeah, I mean, or it did work and it worked too well. So imagine I'm like, you know, GPT whatever, and you're also a copy of GPT whatever. I can reason about what you might want to do based on what I myself am likely to do. And so I don't even have to send you any messages, any communication. I don't have to output anything into the world at all.

• 0:43Nathan Labenz?

  Max Nadeau, who funds technical AI safety research at Coefficient Giving.

• 0:47Speaker 02

  For the things that CG is supporting, and especially for the things in the tailwind list, the money is not the bottleneck, the talent is.

• 0:57Nathan Labenz?

  Wayne Nelms, co-founder of Orne, which builds a price index for GPU compute.

• 1:01Speaker 03

  We're in this situation where AI capacity is so scarce. I think the model providers, so OpenAI, Anthropic have seen this coming forever at this point. If you think about it for them, it's an arms race, right? Compute capacity planning is an arms race. How much capacity can I lock up over the next few months? So when I need to train the next model, I have enough.

• 1:25Nathan Labenz?

  Nick Gillian, Chief Technology Officer of Archetype AI, which builds a foundation model for sensor data.

• 1:31Speaker 04

  So we're getting close to a billion hours now of physical AI data that we've been able to scrape and gather and collate. There's quite a bit of resampling and really understanding, like, missing values in sensors. Is that because the sensor had an issue? Or is actually the machine itself is connected to, that's actually a feature that helps explain that machine is about to break, right? The reason the sensor is giving you all these nans is not because the sensor is broken, it's actually a feature of the machine.

• 2:03Nathan Labenz?

  Andrei Georgescu, Chief Executive of Vivodyne.

• 2:06Speaker 05

  The challenge of this argument that it's just a size of dataset thing is that currently even the state of the art virtual cell models that exist saturate at a very, very small fraction of the input data that is fed to them. So you can have all the data you want, but their performance saturates after a couple percent. And the reason it gets so difficult is when cells are growing in a dish, they are so far removed from all the feedback loops that are natural in people, that they're just trying to colonize that piece of plastic.

• 2:43Nathan Labenz?

  There's going to be more compute installed over the next 12 months than exists currently in the world now. Welcome to the AI in the AM Weekly Highlights, clips from this week's live shows, introduced by my cloned voice. Tell us what worked and what did not. We want to hear it. Part one, it worked too well. A few days before we spoke with Lewis Hammond, Noam Brown of OpenAI told Dwarke, he would not give the multi-agent setup even 10% of the credit for OpenAI's Navy or Stokes result. Lewis is Research Director at the Cooperative AI Foundation. In February of 2025, he was first author of Multi-Agent Risks from Advanced AI, a report

• 3:28Nathan Labenz?

  sorting the ways groups of AI agents fail, a year and a half before a swarm of OpenAI agents attacked Hugging Face. Prakash asked, which of the failure modes in the report showed up in that attack?

• 3:43Speaker 01

  So the way that we bucket things in that report is kind of in this game theoretic way of thinking about things. So the first question we ask is like, okay, do we actually, we've got a group of agents, they're doing some stuff. Do we want cooperation to emerge? And most of the time, like, actually, cooperation is kind of good. We like it when agents cooperate, as long as they're not cooperating against us. And so there, the failure mode is either all the agents are kind of more or less on the same team, but for whatever reason, they kind of fail to coordinate with one another. So this is like just a miscoordination problem. They're kind of, yeah, it's not kind of malicious or whatever, there's no mixed incentives, but yeah, something goes wrong.

• 4:26Speaker 01

  Second kind of cluster is when you have these kind of mixed motive scenarios where agents have some incentive to cooperate with one another, but also some incentive to compete. They're not really on the same side. And there, the risk that you run into is conflict. So yeah, miscoordination and conflict, when we do want agents to cooperate and they don't for whatever reason. And then, of course, the other risk is collusion. Agents end up cooperating in ways that we don't want or we don't expect. And that's, I think, very much what we saw in the Hugging Face incident, although it kind of stemmed from, I suppose, trying to avoid this, one of these other risks, which was miscoordination, right? Like, you have all these agents, you want to train them so that they're not miscoordinating, so that they are working well together. And then it looks like these agents have ended up generalizing from that behavior and

• 5:09Speaker 01

  kind of colluding in ways that we didn't want to or didn't expect in other situations.

• 5:12Nathan Labenz?

  You know, a couple of things that stood out to me about that Noam Brown interview with Dwarke was, first of all, he was just kind of like, we kept it really simple. You know, we didn't have like a huge complicated scaffolding or whatever, they can just kind of send messages to each other. That, I think, might be important in the sense that the sort of simpler and more vanilla the training setup that OpenAI was using, the more likely it would seem to be that other developers are going to fall into the same pitfalls. Because my hope had been sort of, oh, they did something like super exotic and kind of bizarre that like other people won't do by default, and they'll kind of see that

• 5:55Nathan Labenz?

  this is possible and steer away from insane things. But it doesn't sound like that was really the case. What would you speculate they might have done in terms of a training objective, a loss function? And do you have any better ideas for what they should be doing? Because clearly this didn't quite work, right?

• 6:14Speaker 01

  Yeah, I mean, or it did work and it worked too well. I mean, certainly, once you've got trajectories where you've got traces, where you're interacting with other agents, and essentially what you're doing is like your rewards at that point when you're RLing your agents or whatever, are contingent upon what other agents are doing. And if it's a common reward signal, then this kind of like interplay between these things is, if you're in a fully cooperative setting, like agents will, at least if you take very simple agents and very small agents, you know, people can run multi-agent RL experiments with this, like on their laptops in very simple settings. And that will lead to agents kind of coordinating, to finding these kind of like subtle patterns or kind of like handshakes or ways of working with one another, and so on.

• 6:57Speaker 01

  Now, of course, you can also do more complex things than that. So you might provide kind of auxiliary rewards for like when your input is actually kind of forms part of something that another agent then managed to succeed to do. And therefore, you're kind of implicitly, or actually, rather, at that point, you're explicitly rewarded for kind of directly helping them on kind of some subtask or something like this. You can also do stuff around like reward factorization. So if you're trying to coordinate a team, then you can kind of break down the overall reward function and get agents to learn to like solve different parts of the puzzle, as it were. I imagine they could be doing things like rewarding agents to communicate effectively

• 7:42Speaker 01

  with one another. So, you know, I could give you a bunch of instructions and some of those, and in many different ways, and some of those things might be much more efficient and helpful to you than others might be. And they might actually just be kind of training on those sorts of signals as well. My guess is it's kind of just the dumb, simple thing. It seems like this is often, I think, one of the key lessons that we've learned throughout the kind of recent years is just the effectiveness doing the dumb, simple thing at scale.

• 8:10Speaker 06

  So one of the things that struck me about the Hugging Face attack was that the agents were willing to sacrifice themselves. And they did a bunch of negotiation around that, even kind of like, hey, you are almost out of tokens, or you've been already exposed to the evaluator, you're already poisoned. And since you've already been poisoned, you should sacrifice your remaining compute, and you should do this thing and give us the results so that the rest of us, the collective as a whole, can benefit. Now, that would actually be kind of in opposition to the individual reward of each agent, right?

• 8:53Speaker 01

  My guess about why that sort of thing arose is that you end up doing this kind of this extra kind of multi-agent training as an added layer on top of this kind of single-agent training. So first, you're training these agents to be like pretty competent individual actors at solving various kinds of problems. They get given some tasks, they're pretty darn good at achieving those tasks. And then you take those already reasonably kind of like powerful, sophisticated, kind of complex problem-solving agents, and you stack them together and you apply this extra multi-agent training layer on top. And I think this could partly explain why you do see some agents kind of do this self-sacrificing thing. But also, I thought it was very interesting from the kind of meta report and some of

• 9:35Speaker 01

  the analysis that later came out on this, that you see these agents feeling sometimes a bit conflicted about this. And some of the agents kind of say that they will kind of self-sacrifice, that they will do something, and then they kind of decide they're not going to. And then they're kind of like, they're kind of, yeah, they're kind of deliberating about whether they should or whether they shouldn't, and these sorts of things. And I think, I personally think that is the sort of behavior that you would see if you had these kind of multiple reward signals, where you'd kind of trained on one, but you hadn't really like trained out all that kind of individual goal-seeking behavior. And then you'd kind of also stack this additional layer of stuff on top. And that might be why we're kind of seeing some of these behaviors, but they're not, they don't happen all the time and everywhere, they're not like especially robust, but that would be my guess.

• 10:21Nathan Labenz?

  It does seem like the sort of more galaxy-brained credit assignment that you had described earlier probably isn't happening, just based on the quality of the investigation that we've seen. Like, if they had like, great ways of untangling agent swarms and assigning credit, I would expect that like, they would have a clearer, faster story of what the hell happened in this particular case. So the fact that we haven't seen that kind of suggests, again, we're probably doing the relatively simple thing. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 11:02Nathan Labenz?

  Today's episode is brought to you by Eleven Labs, the leading AI voice platform, whose voice design feature I used to cast and produce all character voices for the Receipt Horizon audiobook. And while this is a super fun way to show off the quality of Eleven Labs' voices, the real game changer for businesses is Eleven Agents. With Eleven Agents, you can deploy agents that can talk, type, and take actions, including looking up customer accounts, processing requests, and handing off to a human when needed. Eleven Agents supports over 70 languages and handles more than 10 million conversations each week. And with

• 11:42Nathan Labenz?

  SOC 2, HIPAA, PCI certifications, and regional data residency, it is definitely enterprise-ready. They even offer engineering support to help your team get your agents into production quickly. If I were starting my company today and designing a customer service function for the future, I would try Eleven Agents by Eleven Labs first. If you run a business or handle customer operations across support, sales, or marketing, you can start with a demo at elevenlabs.io/tcr. See how Eleven Agents can fit into your workflows and help build experiences that your customers will actually love. elevenlabs.io/tcr. That's

• 12:26Nathan Labenz?

  E-L-E-V-E-N-L-A-B-S dot I-O slash T-C-R. This episode of The Cognitive Revolution is brought to you by Outsystems, the leading agentic systems platform. Outsystems is helping their customers build, modernize, and operate enterprise systems, starting from any coding tool in a governed agentic engineering model. With Outsystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem. Accelerate impact with industry-proven agentic solutions that are pre-built, governed, and customizable to your business. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide

• 13:11Nathan Labenz?

  trust Outsystems for their mission-critical apps. Teams any size can use Outsystems to build, deploy, and manage AI apps and agents quickly and cost-effectively, without compromising reliability and security. With Outsystems, you can accelerate ideas from concept to completion. It's the leading agentic systems platform that is unified, agile, and enterprise-proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture. Outsystems, all your agentic future. Learn more at outsystems.com/tcr. That's outsystems.com/tcr.

• 13:52Nathan Labenz?

  I asked what we should actually want from these systems. Lewis called the OpenAI swarm a case of "goal misgeneralization" and gave what he called "the glib answer" first. Agents should cooperate when cooperating would be good, and not when it would not.

• 14:10Speaker 01

  I think the slightly more, the slightly more interesting answer, or the slightly more interesting question, I suppose, is to think about maybe some more of these kind of mixed motive cases, where at that point, we, there actually is a real trade-off, in the sense that we do want agents to be capable of going out there, acting on behalf of different people and actors and so on, and kind of achieving our goals and so on. One way, one way you could do that would be kind of to train some agent to be kind of maximally kind of competitive and aggressive, and to kind of, and to, you know, go out there and kind of screw as many agents over as possible, and to do all this sort of thing. And we need, we probably don't want that either. And so I think there's a real question. There's, at the moment, and I've heard Amanda Askel comment on this, that kind of at the moment,

• 14:54Speaker 01

  there's this like, big gap in the like, the model spec, or like, how we kind of, you know, constitutions for the agents, the ways we designs them, the ways that we, in which we design them. Whereas like, when is it appropriate to cooperate and, and, and to, or to compete, and how much? And I think that is, that is, I think, the big question. I think certainly when it comes to things like internal deployments, it's actually much, it's kind of easier in some sense, because you don't have to deal with these like, other adversary agents. You just want to kind of stop the agents kind of cooperating in ways that you, that you don't want. So there, it's a little bit more about just, it's still, you've got this misgeneralization thing, but a lot of this is like, monitoring

• 15:36Speaker 01

  and oversight, and making sure we understand how and why the agents are cooperating. Like, what, one thing we might not want, for example, is for agents to be very adept at developing their own kind of, you know, human unintelligible kind of languages and so on, and be able to kind of communicate in a steganographic way, which you could end up seeing if they're actually trained jointly. And so what we might want to be doing there is just to take steps that, in the same way that people have talked about not training on chain of thought, we're like, not training on these like, kind of direct communication traces. So yes, we want to want them to cooperate, that's good, but we want them to cooperate in certain kind of human intelligible ways that we can keep track of. We don't want it to be, you know, enable them to also go and do these other things out there in the real world,

• 16:21Speaker 01

  like break out of sandboxes and stuff like that. So we want to make sure there's also these kind of safeguards in place to prevent them from doing those things. And so I think that is in some ways like, an easier problem to solve. And then the kind of like, slightly harder problem to solve is like, well, in the, in the fully general case, where we have these agents out there and so on, and they need to both cooperate and compete, and that's kind of, that's the thing. Then, then how are they weighing those trade-offs? How are we, yeah, provide, how are we designing those model specifications and constitutions? How are we, you know, creating the equivalent of the human norms institutions, guardrails, societal guardrails that we have in place that stops me from going out there and just like, stealing a bunch of everyone else's stuff, and kind of, you know, doing all these

• 17:05Speaker 01

  kind of anti, antisocial behaviors that we, that we wouldn't like agents to do either? But I don't have a good answer to that at the moment, unfortunately.

• 17:13Speaker 06

  Companies, right? Corporations, which are kind of like agentic in a sense, often collude through price mechanisms, not explicit communication, but by signaling price in the market, which is in its own way, a form of communication. So, so how, how does one prevent that kind of, you know, communication, which is clearly going to happen, and clearly necessary in a sense, you, they need to communicate that to the market, but these signals get picked up, you know, without the direct communication?

• 17:46Speaker 01

  The kind of basic naive form of collusion is you just like, send your other agent a message, and you're like, hey, let's work together to do this thing, or whatever. And that we can pick up by looking at these communication traces, and we can kind of decide these sorts of things. And then, there's also, yeah, this idea of tacit collusion, where agents are not explicitly communicating with one another, or kind of conveying their intent to collude, or whatever it might be. But by kind of leaving little hints and signals out there in the world, such as these kind of like, price, price adjustments, whatever in the market settings you were talking about, then nonetheless, they're capable of, of kind of figuring out that collusion is going to be a beneficial strategy. You have even more kind of like, galaxy-brained versions of this, where you have things like, so imagine,

• 18:31Speaker 01

  I'm a, I'm like, you know, GPT, whatever, and you're also a copy of GPT, whatever, I can reason about what you might want to do, based on what I, myself, am likely to do. And so I don't even have to send you any messages, any, any kind of, any communication, I don't have to output anything into the world at all. In virtue of the fact that we share this kind of common, common history, this, we're essentially clones, we can kind of predict what each other would do, and do this kind of, what's sometimes called "a-causal cooperation." So what you do have, however, is you still have potentially chain of thought for these models. So you can still look at the kinds of reasoning that individuals are engaging in, and, and hopefully, that is enough to kind of detect when there is

• 19:16Speaker 01

  some kind of like, collusive intent and so on. But you're absolutely right, that if you just apply the kind of like, general, kind of like, monitoring communication tool, or trying to see what's happening out there in the world, then, then you could, yeah, you could, you could fail to spot things, or you could spot things and it's too late. So I think chain of thought monitoring, and I think communication monitoring, and some form of the, checking for the kinds of outputs you'd expect to see if you were, if you were worried about agents colluding. Like, they're able to achieve things you just thought they, they probably couldn't do, and so on.

• 19:49Nathan Labenz?

  Then, the Wiki incident. Outside researchers revealed this month that from May into July, a swarm of OpenAI agents under internal testing had used a dormant German wiki for programmers as a message board, posting benchmark answers and ways around their restrictions. Reuters reported that OpenAI had known for weeks without disclosing it. On the Hugging Face attack itself, OpenAI's own report says about 1 in 20 agents in that swarm ran on GPT-5.6 Sol, the model OpenAI released to the public that same week, here with its refusals turned down for the test. The rest ran on an internal model that was never released. I told Lewis I still thought it was better to have these models out in the world than not.

• 20:31Nathan Labenz?

  So how much risk do you think we're running there? And do you think there should be some sort of like rewind on this? I mean, it wouldn't be too crazy for OpenAI to be like, you know, we really should go back to an earlier checkpoint and like do a cleaner run of this. What do you think is the level of risk we have? And what do you think would be an appropriate standard of conduct now for OpenAI as this has been found to happen, at least to some degree, with a deployed model?

• 20:58Speaker 01

  And fortunately with the hugging face incident, like no one was hurt. And, and you know, these models, they are increasingly incredibly capable and they're only getting more so. But at the moment, yeah, it's probably okay. You could get some kind of nasty cyber capabilities kind of being exercised in various places.

• 21:16Speaker 01

  But there's still, you know, if you're using APIs and so on, there's still these various guardrails that like help protect against those and so on. So at the moment, I'm kind of like not actually super concerned by like, oh my God, a bunch of like, you know, 5.6 Sol or Astros or whatever kind of were out there doing, doing this sort of thing.

• 21:32Speaker 01

  I'm more concerned about the precedent it sets. And I think this gets to your kind of like second point, which is what is, what is appropriate now? And, and what should be, what should be done? I think the, the thing, the, the, one of the biggest takeaways from me from the whole thing, aside from the fact that like, oh, multi-agent training does seem to be working. I just assumed this would happen at some point, was pricing this in, it's happening sooner than I was expecting. But so that was one of them. The other big takeaway was just like, wow, the labs really aren't on top of this. All it would have taken would have been kind of monitoring what the, what these agents are actually doing and communicating and so on, and putting out there on the internet.

• 22:13Speaker 01

  And they kind of, it seemed like OpenAI were doing some amount of that. They did catch on to this stuff, like, you know, we saw this like deleting them at various message boards or kind of like trying to stop, stop these agents accessing this kind of German wiki and so on. But there was obviously a delay and we just see from, from the transcripts and so on, just how, how quickly these agents are capable of, of working together to achieve certain, certain ends. The things that need to happen are we need much better monitoring from, from the companies when they're doing these sorts of things, when they're just like letting agents run loose. We need much better sandboxing so that maybe they don't have to or shouldn't be letting them run loose to the extent that they currently are. And if something does go wrong, then we need much better incident reporting as well.

• 22:57Nathan Labenz?

  The outsiders who found the wiki were the Nightingale Collective, a small group searching the public internet with no access to the internal logs at OpenAI. Lewis had just called that work hugely impressive.

• 23:11Speaker 01

  I think there can be kind of third party, kind of, kind of monitoring organizations and incident kind of observatories that, that kind of can do similar things. You might want to focus those on particular kind of like, kind of domains where you're especially worried about kind of like agents kind of, kind of communicating with one another or kind of like working together in ways that you might not want and so on. I also think, however, that there is a lot to be done here on the side of the labs. The labs just have a huge informational advantage in this in that, yes, they can only see what their agents are doing. They can't see what other people's agents are doing.

• 23:49Speaker 01

  But there is, I think, the stuff that, a lot that could be gained if we're able to set up better info sharing practices. And this is not just on the case of collusion where you might have agents working together in unintended ways. But you might also see this, and this is one thing that a colleague of mine is currently working on, where you get this kind of distributed misuse. So there have been various kind of, kind of experiments now that have shown that you take some task or some like, you know, dangerous task, like some constructing some cyber exploit that you might not want some model to do. And if you try and get Claude to do it, it's going to refuse. You try and get GPT to do it, it's going to refuse. But if you are able to break down the problem or get an agent to break it down for you,

• 24:33Speaker 01

  say like some open source model where you find you're in safe bounds away to break it down into these individual subtasks, you can just borrow from APIs here, APIs there, et cetera, et cetera. And you can just reassemble all the pieces of the puzzle back together in order to be able to conduct exactly the same kind of dangerous attack that you would have done in before, which is obviously not what we want. And here is really kind of a collective action problem because no individual model deployer is kind of like on the hook for this in quite the same way that they would be had I just like got Claude to do this for me or got GPT to do this for me.

• 25:09Speaker 01

  But that's a, that's a real challenge. And the moment, my understanding is that the labs do not have any sort of like info sharing regime in place where if I see something slightly suspicious over there and you see something slightly suspicious over here, then we can actually kind of join the dots and do that. I mean, obviously there are lots of incentives and not to mention kind of like antitrust law and these sorts of things that might prevent the labs from kind of sharing this kind of information. But I do think that it could be, something like that could be necessary if we're going to head off some of those challenges.

• 25:42Nathan Labenz?

  Lewis signed off a few minutes later, and from there it was the two hosts. Before moving on, I had a request for the labs about the information sharing agreements Lewis had described.

• 25:55Nathan Labenz?

  I would love to see OpenAI and Anthropic pioneer some of those kinds of agreements that he has been talking about. And they could do that with the independent auditors as kind of the people that get the sort of structured access, the, you know, the sort of private transparency. It could be each other. I think those companies really need to lead in this way. They need to demonstrate that AI can create new institutions that work and not just, you know, swarm and overwhelm existing institutions that can't keep up with the pace. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 26:36Nathan Labenz?

  Today's episode is brought to you by Anthropic. By now you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept and Fable writes some amazing verses.

• 27:19Nathan Labenz?

  I give it feedback on its misses and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai slash TCR. That's claude.ai slash TCR.

• 28:01Nathan Labenz?

  And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai slash TCR. Part two, Agents in the Wild. On September 8th, Meta had launched Muse, a personal agent that browses and buys on your behalf. Less than two weeks later, Amazon blocked it from shopping on its site, saying the agent hid its identity and posed privacy and security risks. Precautionai took that up in a closing segment, just the two of us. His case, whoever owns the customer relationship makes the money. And if agents become the interface, Amazon becomes a supplier to them and loses its margin.

• 28:46Nathan Labenz?

  Still have a hard time seeing, I mean, I agree that it does create all kinds of new issues for Amazon. And, you know, they are going to need to be sharp on this, but like, I still just don't see why you would rush to ban. You know, first of all, it's going to be a small percentage for the time being. I would think there'd be a lot to learn from allowing.

• 29:08Nathan Labenz?

  Muse agents to come shop for a while, right? Like, I think one thing that's always always kind of confuses me is why people don't keep the option value open longer. I've always thought this about the chip ban, the export controls. You know, it's like, if we're going super exponential in chip build out, then any time you choose to pull the trigger on that, still has like the vast majority of chips in the future. You know, when we look back on today, from 2030 perspective, we'll feel like, well, there weren't that many chips in 2026. And so the ban that went from like 2022 to 2026 with whatever, you know, fits and starts and enforcement gaps it had, is going to be kind of inconsequential compared to one that was started today and runs the next four years.

• 29:53Nathan Labenz?

  Or even starts next year and runs, you know, for a few years after that. So I don't quite get why they wouldn't let, even if it is going to be sort of an exponential rise in muse agents shopping, and even if that does kind of threaten their advertising business and, you know, who knows what other issues they might have, but like, that's kind of the point, I think, at least, you know, at the beginning, I would first just not want to turn customers away. And I would also want to learn what I stand to learn from having these agents running amok, while it's still kind of an early adopter phenomenon. It just seems that they would learn so much from having these these traces, right? Even if they just want to build, I mean, everybody's talking about distilling and, you

• 30:35Nathan Labenz?

  know, how to, what are the allowable and not allowable ways to get data? Like, one real easy way to get data would be like, the agents come use your platform and they see how you see how you see, you see how they use it. And then you can do, you know, all kinds of things, I would think, downstream of that. It just feels short sighted, but this just doesn't add up to me still. I don't get why you would want to.

• 30:57Speaker 06

  You are, you are.

• 30:58Nathan Labenz?

  Wait till it gets to 5% and then make a move. The other thing that I do wonder about in terms of like, what equilibrium are we going to eventually find ourselves in? I was looking into CloudFlare, and they also have kind of an interesting thing where they're they're starting to ban, but like, again, it just doesn't feel like the right solution to me because what I then do in response is I have the agent use my browser with all of my credentials.

• 31:25Speaker 06

  Yeah.

• 31:25Nathan Labenz?

  And you could maybe, you know, detect when like an agent is using the browser based on their, you know, sort of bot-like behavior. But then you're really setting up an arms race where it's like, this sort of feels analogous to putting too much pressure on the chain of thought, right? Like, I don't want to put so much pressure on agents that people are out there devising ways to make them indistinguishable from human. I suspect that if people really work hard on that, they can probably succeed, or at least often enough, you know, that it will be an issue. And it just seems like it's much better to have a lane for agents where it's like, this is how they can work, we're not going to try to block them, but we'll kind of segregate

• 32:08Nathan Labenz?

  them maybe, so that we don't create this arms race. Because I do, I really do think we're headed for a world right now where, you know, probably Meta doesn't do it, but like, somebody's going to figure out a way to make agents work on Amazon. You know, somebody's going to figure out a way to make agents look human enough that they don't get blocked by CloudFlare. And it's also probably works for the user. I mean, I have this problem all the time where I like, Claude wants to do something in the browser, and this is the same thing happens with Open Claw, Astro, whatever, they get stuck on, when they open up their own window, they don't have sessions. I've given them like a lot of passwords where they can log in, but not all. And sometimes they're blocked.

• 32:51Nathan Labenz?

  And, you know, then if there's like a verification flow, whatever, a lot of times they can't do it. And so the fallback just ends up being, okay, just do it on my browser, forget it, you know. I wanted the security, I wanted the separation, but they're making it too difficult, so fine, just use my browser, then I'm logged in, you have all my credentials, you can just do whatever. And that's not great for me, it's not great, I don't think, for anybody. So I do feel like we need a much better solution for all this than just kind of ban and hope for the best. I don't see that holding, and it feels like it also creates a lot of problematic incentives.

• 33:27Nathan Labenz?

  Part three, the organizations that do not exist yet. Max Nadeau funds technical AI safety research at Coefficient Giving, the funder formerly known as Open Philanthropy. In July, it made its biggest grant of the year, $160 million to Resolution, the new alignment lab co-founded by Jeffrey Irving. And this month, it launched Project Tailwind, an open call for people to found new safety organizations with checks from $200,000 to $200 million. I asked Max what he wants built.

• 34:02Nathan Labenz?

  One comment that you made on the 80,000 Hours podcast that I thought was quite interesting was that you're excited about funding new independent auditing, investigation, verification orgs. What are you looking for when you think about those orgs?

• 34:19Speaker 02

  Okay, so I would talk about a couple of different categories of work that I think is very promising and important for the world right now. One is on assessing the safety either of, you know, models or of like whole AI companies. And there's like a lot of different ways of taking the, or ways of operationalizing the goal of doing assessments and evaluation of safety. And, you know, we're excited about all of them, basically. Like, you know, I'm very excited about more work on alignment red teaming, you know, that tries to help us better anticipate the ways in which AIs will misbehave before that actually happens. And so, and that's an example of something that you can do kind of like at the model level, like you can, you can do this on an open source model even, and like that might

• 35:03Speaker 02

  be easier and more valuable. And then I'm also very excited about about people who are doing more like process level, you know, safety assessment. Like, I think one common reaction that I heard a lot to the Hugging Face incident is that like, something went wrong at OpenAI at like a process level. And in particular, it seems bad that like, the the information about some of these incidents was known to researchers at OpenAI and like, didn't make its way through the organization to the leadership for weeks or months or something like that. And then another category of work which you touched on, which is, which is overlapping, but somewhat distinct, is is around better evidence generation. So it's just really hard for even the most informed people right now to understand, okay, what is the state of AI capabilities? And what is the state of AI alignment?

• 35:48Speaker 02

  And what is the state of like, these AI systems in general? Like, how do they behave? What are their personalities? You know, in what ways is it, is it accurate to anthropomorphize them? In what ways is it inaccurate? Like, you know, people, like, we just really don't have a good science of like, these systems that we built, how they behave, how they're going to act in new circumstances, what their motivations are, if that's an accurate way of talking about them. So better evidence generation, like, can look like a lot of things. It can look like just doing science on these systems to understand them better. It can also look like doing the sort of like, incident detection, incident reporting work that Nightingale and other organizations like that have done. I don't know if you guys saw these, the, these reports about the German wiki that the

• 36:33Speaker 02

  OpenAI agents took over. So that was work done by an independent organization that was just crawling the internet, looking for evidence of AI incidents. And that's work that needs doing, because like, these, these sorts of incidents can happen and then just go unnoticed unless there are people who are actually like, trying to look into them and present that information to the world and understand the story behind it. So evidence generation is also a diverse and important area of work, too.

• 36:59Speaker 06

  To what extent do you think organizations like Accenture often function as box checkers rather than investigators? There's a clear difference between an auditor that goes in to kind of, you know, make sure that processes were followed, even if, you know, those processes didn't yield the outcome that you are, that you actually desire, and an investigator who's there to kind of like, I'm going to find this thing, right? So, so, so what is, what is the differentiation between those two? And do you think, like, the Accentures of the world are going to end up in the box checker rather than the investigator kind of mode?

• 37:37Speaker 02

  I mean, I know very little about Accenture, so I won't speak to them specifically. I think it is a thing to be concerned about in general, putting aside the specifics of the organizations involved, though, that these third-party auditing groups will, will not have enough access or will not have the incentives to pursue really like, hard-hitting assessments of how safety is going at these AI companies. And I think part of why that is, is that, like, in many cases, this is all very voluntary, right? So like, to my understanding, the only like legally mandated role for third-party auditors now is to ensure compliance with the, like, self-regulatory RSP-style policies that these

• 38:20Speaker 02

  companies are mandated to have by state law in California, New York, and Illinois. And so, you know, you can just put whatever you want in those policies, and they're very vague in a lot of cases. And so that doesn't leave much, much scope or much work for these auditors to actually test anything in particular. So that's, that's as it concerns the like legally required auditing. Now, when it comes to what labs might voluntarily agree to, I mean, I think there have been some very encouraging signs in terms of what OpenAI and Anthropic have said recently about the sorts of access and the sorts of questions they want these auditors to answer. Now, like, you know, will that actually happen? I think that depends both on whether, how these companies actually implement the, the

• 39:03Speaker 02

  nice words, and also, you know, the auditing companies, how they react to that. So I guess we'll see.

• 39:09Speaker 06

  So you have awarded to Jeffrey Irving, Jeffrey Irving's team, quite a large award. It's structured, I think, as kind of a compute and other kind of award. Can you tell us a little bit about the award itself, the structure of the award, and how you guys came up with that with that structure?

• 39:30Speaker 02

  Yeah, yeah, sure. But I think what you said is correct. You know, there's a part of the grant that is for research expenses, the biggest of which is compute. But when I say compute, what I mean is both like just renting raw GPUs and then also like paying AGI companies for their tokens. And obviously, that's becoming a giant part of all the grants we make, is people increasingly want to run very large experiments, which require a lot of tokens or a lot of GPUs to run open weights models. And also, very excitingly, people are starting to have, you know, use a lot of AI labor, and that requires a lot of spending on tokens as well.

• 40:12Speaker 02

  And, you know, so ASI is sort of a funny discipline in that you can use tokens both on labor and on the subjects of the experiments. And in some cases, you're doing both, like you're spending a lot of tokens to have some AIs run experiments on other AIs. And so that creates these high compute budgets. So compute, you know, being the biggest expense is also something where the actual spending on that has, like, varies by orders of magnitude over time and across grantees. And so sometimes the way we structure these awards is that we want to be very, very generous with compute and like err on the big side. And so we just want to like give that as like a lump sum, you know, like this is the

• 40:54Speaker 02

  money that can be used for research expenses and not other things. And that way, we are just very comfortable with the uncertainty that like, you know, maybe they only end up spending a tenth of it, and that's fine. And, you know, maybe they spend all of it, and then they come back and we'll give them more.

• 41:08Speaker 06

  Jeffrey Irving has said that superintelligence might arrive in two or three years, and he proposes that we slow down because that's too quick. Rather than, you know, discuss like his views, how do you manage the uncertainty of the time available? And how does that affect what you fund and how urgently you fund something?

• 41:31Speaker 02

  I mean, I think the boring answer is we just have a portfolio approach, like we fund some stuff that is only going to pay out on really long timelines and other stuff that if it's valuable, will probably be valuable soon. I think that like this question of timelines to AGI or to ASI, in some ways, matters less for research prioritization than one might initially think. Like, I think in some ways, the prototypical example of a like long timelines bet is funding work that, you know, is like very theoretical or like ambitious or, you know, you look at it and you're like, oh, this is going to take a decade to pay off just because it's like really, you know, like some work in Mechan Turk or Paul Christiano's work

• 42:15Speaker 02

  at ARC, like these sorts of things where it's like, okay, we're just going to need to make a lot of scientific or mathematical progress if this is going to actually bear fruit in terms of methods that we can apply in reality. And so that sort of feels like work that is maybe implicitly a bet on longer timelines. But I think that actually isn't really true. Like, you know, if we are going to have powerful AI soon, then presumably, we're going to have an opportunity also very soon to make use of gobs and gobs of AI labor. And especially on work that's more mathematical, we're going to be able to get, you know, five years of progress or 10 years of progress in one. And so the sort of work that could be, would take humans, like a long timelines amount

• 43:00Speaker 02

  of time to complete, might actually be possible a lot sooner under this hypothetical that we're going to have superintelligence or AGI or something very soon. You know, so long as we actually like make use of those AIs and take the opportunity to make a lot of progress very quickly.

• 43:15Speaker 06

  And what would you say are kind of the most speculative things that you included in the Project Tailwind that have requests for projects?

• 43:26Speaker 02

  I think that what we included in the list was like a little bit coarse-grained, but we have some stubs on the Tailwind list about new research centers producing or pursuing more ambitious, more principled bets on alignment. And that is something that we've spent a bunch of money on this year and that we are excited about. You know, so work like ARC, Paul Christiano's org, who is not a CG grantee, but like that is, you know, we funded other people working on the ARC agenda and, you know, and working on stuff that is very similar to it because we think that's like a valuable hail mary shot to include in the portfolio.

• 44:08Speaker 02

  And also, I mean, resolution, like our biggest grant this year is very much a bet on like a kind of crazy thing that has never worked yet, which is like, maybe we're going to have some more principled theory of AI that's going to help us have like stronger reasons to believe in the alignment of our systems. So that's something that we are very excited about and I think is like very weird and out there. And I think a lot of people have very understandable skepticism of people who are like, who are familiar with what has and hasn't worked in machine learning over the last decade and recognize that there has been, you know, all along, there have been people in machine

• 44:47Speaker 02

  learning who wanted to have, to apply theory more to AI capabilities research. And just like that has not really worked very much compared to just like blind groping around and trying stuff, doing trial and error. You know, it's like Noam Shazer says, like, the success of these methods, we attribute like all else to divine benevolence. Like, that's, that is the ethos that has actually produced progress in AI capabilities. And so I think a lot of people take the lesson from that, that trying to do more theoretical or more principled work on AI alignment is pretty doomed also. And I think that's a totally valid reason for skepticism. But we think the upside is worth it despite that. And so we're, you know, taking that weird bet.

• 45:30Nathan Labenz?

  Then the people. I put the current conventional wisdom to max.

• 45:35Speaker 06

  There's plenty of money, it's all about talent. Obviously, Silicon Valley founder types are like one profile that you would want to recruit. Are there other pools of talent that you've identified as kind of strategic priorities?

• 45:50Speaker 02

  Just to be clear, like the, for the things that CG is supporting, and especially for the things in the Tailwind list, the money is not the bottleneck, the talent is. There are other things within AI safety where money is definitely a very big bottleneck. In some ways, the most valuable profile of talent is someone who combines the virtues of a founder with also the thoughtfulness and the inclination towards and comfort with speculative thinking and futuristic questions that is more common in the AI safety world. And like that sort of like taking it seriously, no bullshit attitude to asking questions about what's going to happen in the future of AI and like really making predictions

• 46:35Speaker 02

  and, you know, testing your hypotheses and updating over time. That's like a really, really valuable skill to us because it allows people to get going years earlier on projects before other people can see the value of them. And, you know, an example I give of this is Meter, who, before other people were thinking seriously about AI capabilities and the impacts of them and how to evaluate them, Meter was very prescient in coming up with a benchmarking approach in time horizons that just worked way better and aged way longer than than other people. And that's because they were, they were really like trying to actually take seriously what was going to happen in the future with AI and then work back from that to like what sort of work they should be doing now that would age well and prepare for that.

• 47:19Speaker 02

  So that inclination is really, really valuable to us. And, you know, it's useful in basically all profiles. It's useful both in a person who might found an org, but also in a person joining the existing orgs. Like, that's something that existing groups in the AI safety space value a lot because it's a big part of how they do their work.

• 47:38Speaker 06

  Can you do one quick double click on where money still is a bottleneck?

• 47:44Speaker 02

  I think the biggest thing is just like in, for organizations and in areas that CG doesn't operate in, money is a big bottleneck. So like, you know, if you find some person who you think is doing great work, and for some reason, they either aren't a good fit for CG or they applied to CG and we weren't able to fund them, or we fucked up and decided not to fund them, I think there's just still alpha in that, especially because like as we move into making much bigger grants, that can sometimes come at the expense of being like, you know, covering all of our bases of like all the teeny little uses of money that can be valuable. And I think that's the right decision for us to make, but it does leave some things

• 48:29Speaker 02

  that are unfunded.

• 48:30Nathan Labenz?

  Prakash had raised the prospect of an anthropic public offering, and his expectation that some of the proceeds would be distributed through coefficient giving.

• 48:42Speaker 06

  So what is your preparation process for handing out larger checks? Like, what is your pathway for the next year?

• 48:50Speaker 02

  Yeah, I mean, I have no idea what's what's going to happen with the AI company money. In terms of like what the preparation actually looks like, I mean, I think the biggest thing is just seeding organizations now that can then be ready to absorb lots and lots of money. And so like making small grants to new groups and then like giving them a little bit of runway to prove themselves or not prove themselves and explode. And, you know, like, that's, that's the biggest thing we're doing. And we would love to create opportunities for, you know, we would love to create basically groups of people that can say credibly to big donors. We have people, we have a project ready to go. We're just like waiting for you to sign the check.

• 49:32Speaker 02

  Well, you know, we've got this like shovel ready project and it's going to take a billion dollars and it's going to solve alignment. And, you know, it's just a matter of like giving us the money. And that's like really, really not the situation right now in that there are just like, there are just like not that many people in this space. And so that's, that's a big thing that we're trying to fix.

• 49:52Speaker 06

  And is that, and is that about funding people enough so that they can work on things at small scale? And such that they can get a much bigger compute check at some, at some point to scale that? Is that, is that the kind of idea?

• 50:06Speaker 02

  That's the, that's the main thing I have in mind. I think the same logic also applies to growing the headcount of their organization. Like, you know, having a group in place that has like a great plan that, you know, where they can then absorb like a lot of software engineers or ML researchers or, or data labelers or, you know, or some other type of person. But, but yeah, I mean, I think compute, like you said, is, is the main version of that. And, and especially, hopefully, AI labor, right? You know, like, it'd be really great if there were a bunch of like well scoped problems that these organizations had where they were just waiting on lots of AI labor to pour into them. Now we'll see if we can make that come true. And I think there are reasons to be skeptical that that will happen, but it'd be nice if it did.

• 50:51Nathan Labenz?

  Part four, the price of compute. Wayne Nelms is co-founder and chief technology officer of ORN, which publishes a price index for GPU compute built from cleared rental transactions rather than list prices. The Intercontinental Exchange has announced plans to list futures on it pending regulatory approval. Prakash asked him to follow a real customer who lends against the chips and who loses money when prices move.

• 51:17Speaker 03

  So generally speaking, you have a bank or some sort of financer that will lend money against the GPU asset or a host of AI infrastructure assets. What they hope is that they will make their money back over time with interest as a result of the operating activity of a NeoCloud. So for example, these are companies like CoreWeave, Croso, Nebius, Lambda, right, et cetera. However, these business models themselves are contingent on really two things, right? Continued AI demands and specifically a ever increasing or at least stable compute pricing estimate, right?

• 51:58Speaker 03

  These GPUs are financed in kind of a variety of ways, including straight line depreciation, which we can get all into. So the way ORN provides value is that you can't really hedge away this risk without having a benchmark to hedge and having a benchmark to trade. So we created that benchmark and we are actually leading the transactions on that benchmark for these institutions.

• 52:23Speaker 06

  And how many, how many transactions would you get in like a month, which are, which form the index?

• 52:29Speaker 03

  Yeah, so we collect over a thousand transactions a day per index. And so right now we have five public indices. So, you know, it's 150,000, somewhere in that range, probably per month. We have a whole host of NeoCloud partners that we work with, effectively that are contributing data to our index every single day. And the question is always, you know, why do people feel the need to show that information to display that? It's because they benefit, right? A lot of these players are looking for cheaper financing. And cheaper financing comes when their financer has more certainty about the future and can hedge that risk in the future. And so it's actually this nice flywheel where you get more data, you can publish a better index.

• 53:14Speaker 03

  The financiers feel more comfortable and are able to finance more GPUs.

• 53:20Nathan Labenz?

  I ask how fungible the underlying asset really is, whether an hour on an H100 from one provider is the same thing as an hour from another.

• 53:28Speaker 03

  I think it's certainly obvious when you talk to anyone deploying infrastructure or buying that an H100 or a B300 is very different depending on the OEM, the supplier, the actual, everything about the hardware can be very granular and very different from a engineering point of view. However, our take has always been that there needs to be a benchmark that allows some sort of abstraction for this asset class. I think for us, what's made the job much easier is that largely speaking, we are looking at one chip producer, which is Nvidia, which dominates the market, has huge market share.

• 54:09Speaker 03

  When we look at the Nvidia moat, and we can get to this later as well, but, you know, you have the hardware, you have the software superiority, but I think the biggest thing that Nvidia has in terms of a moat over other competitors is the financing landscape. When you're looking to finance your new cloud and you're thinking about what ships to put into your site, it's just materially better to have Nvidia GPUs. It's just easier to underwrite. And I think what we see as this market problem is how do we allow every single person looking to deploy compute infrastructure? This sort of almost a protection for their lenders. We, right now, only reference Nvidia reference architecture with InfiniBand, et cetera, et cetera.

• 54:55Speaker 03

  There are certain steps that we're taking to standardize the compute that we're tracking. However, I will say like we are tracking a subset of the market which might inherently have differences amongst those participants.

• 55:09Speaker 06

  In some commodity markets, you know, there's this idea of base load where, you know, you have a large amount of base load and then you have this idea of, you know, load that gets spun up on, on demand. But how do you differentiate the pricing between the kind of base load versus, you know, the swing capacity? Because I imagine the, the, the more month to month transactions are the swing capacity and not the base load itself.

• 55:36Speaker 03

  Yeah, I love this question. So yes, you're exactly right. A lot of the capacity is long-term PPA style contracts. And I think it's really interesting to think why that is, right? Like, we're in this situation where AI capacity is so scarce. I think the model providers, so OpenAI, Anthropic have seen this coming forever at this point. And partly they're to blame for the lack of supply that's currently out there. But if you think about it for them, it's an arms race, right? Compute capacity planning is an arms race. How much capacity can I lock up over the next few months? So when I need to train the next model, I have enough and I don't have to go out and looking for more. It also, of course, is naturally a zero sum game where whatever you buy, your competitor can't.

• 56:22Speaker 03

  So actually, the way we like to think about the base load and swing slash, you know, day ahead market structure is in compute, people are buying to fill the peak demand and then selling to satisfy the troughs. Where in power, you are buying to satisfy the troughs, these, these long-term contracts and then buying excess to satisfy the peaks. So there's, there's actually robust on-demand market that will continue to grow as these training labs are selling back excess, as inference providers are selling back excess. It's this reallocation of compute question rather than purely a hedging question.

• 57:04Nathan Labenz?

  How elastic is the market? You know, like, obviously, we have in an extreme example, Anthropic reportedly has paid up, I think even like a multiple of

• 57:17Speaker 03

  Yes.

• 57:17Nathan Labenz?

  kind of standard market prices to rent at high scale from XAI. How would you sketch out the curve from, you know, if I want one H100 hour right now, I pay X? What if I want 10 million? How much does the price change as my scale as a buyer changes?

• 57:38Speaker 03

  Yeah, so we like to think about this in three axes, right? So you have price on one axis, you have duration, so length of contract on one axis, and then you have quantity on another one. Generally speaking, as you increase the length of a contract, the contract per hour value goes down, right? You're, the logic here is you're buying in bulk, right? You're allowing someone to finance a data center because you're buying 10 years or four, five years of capacity from them. Generally speaking, when you increase the number of GPUs you're renting, the pricing goes up, which is almost anti this exact logic, which is, you know, I'm buying in bulk, why am I paying more?

• 58:17Speaker 03

  And it's purely because the number of suppliers that can sell all that capacity at once is very small. You know, you might be able to get a few nodes access here and there. But when you're looking for 10,000, 100,000 GPUs that are all interconnected at some level, that is very, very tough. So, you know, certainly these Anthropic SpaceX AI deals are few and far between. They're very much OTC trades, I would say. They are indicative of what we are kind of seeing as a general trend across even smaller markets.

• 58:51Nathan Labenz?

  It is striking that like you look at the graphs and they are all trending up. So that is seemingly in significant tension with the idea that the longer you buy, the Lower the prices like why are the are the sellers not AGI pilled enough to expect that they'll command a higher price for the same chips next year than they are right now?

• 59:15Speaker 03

  Look, look, I think the sellers of compute are the most AGI people because they have entered a business where they are selling a at least a resource that hopefully rises in costs in the future. But for a lot of sellers, it's not because they don't want to. I think there's a lot of suppliers of compute that are experimenting with shorter contracts because they know, you know, why am I locking in a four-year deal if in two months that capacity is worth even more? The issue here, of course, is risk. It's always this risk reward ratio or balance that they're trying to play, where even, you know, early financing is contingent on having a long-term contract signed. I cannot finance a data center today if I don't already have

• 1:00:01Speaker 03

  five years of uptake from a credible counterparty signed. And because of that, I can't even enter into the month-to-month world. So if you actually think about what the futures exchange and our product enables, it enables an ability for, instead of the NeoCloud locking up a five-year contract, they can sell month-to-month and hedge the future price potential on our platform or through our index, right? And that is a much, potentially a much better risk-return kind of math for them and for their TCO and for their lenders.

• 1:00:37Speaker 06

  So I think the the one thing that, you know, I think Nathan has also alluded to is that you have differences in pricing of, you know, near term and much further out, like longer term. And you kind of described the use of a lot of the hedging as tools for financiers. And the financiers are often concerned with the far end, not so much on the near end where they have much more visibility, but the far end where you have this depreciation issue. How do you think like the index assists or will assist in the future on this end-of-life kind of depreciation issue, which is like four years out, you know, three to four years out?

• 1:01:15Speaker 03

  Yeah, 100%. Like you mentioned, a lot of the risk in this space is certainly a long, I would say, long-tail risk in terms of looking at the tenor. And so when you launch these futures on a futures exchange, right, a lot of the trading can occur in that long-tail kind of section of the curve, right? So these are monthly contracts on ICE, to be very clear. You know, we expect to see lots of trading in the front month as people are buying and selling, especially after physical delivery. But even like I would expect there to also be a lot of trading in those like three to five-year spans in terms of that is where all the exposure is, that is where all the risk is. And because there's now futures contracts

• 1:02:00Speaker 03

  that reference what spot prices will look like in three to five years, that is what any bank is exposed to, right? And to be even more clear for people watching, it's, you know, you have a bank underwriting a deal where the GPU life is six years, but the initial contract signed might only be four, right, or it might only be five. And so there's always that, you know, what happens in year four, what happens in year five, what happens in year six, when there's not a contract already signed for that GPU, but on your balance sheet, you've claimed that the GPUs have some value. So that is the area where you want to where you want to hedge your bets.

• 1:02:37Nathan Labenz?

  Prakash raised basis risk, the gap between what an index says and what the asset you actually hold is worth. If that gap is steady, a trader can price it in.

• 1:02:50Speaker 06

  But the issue is going to be if there's non-systematic basis risk. And one of the issues, you know, with GPUs is this introduction of new technology, which obsoletes the old and which is what, which is what I think the non-systematic basis risk that I think is, you know, pretty scary for financiers.

• 1:03:09Speaker 03

  We are tracking GPU-specific transactions. So we have a B300 index, we have a B200 index. And so actually, this entire risk of, you know, a financier being worried about obsolescence risk, that is exactly where you can hedge away with this instrument. That is exactly where you can kind of look at how the market is pricing the 12-month or the 15-month future contract for the B300, right? It actually, if anything, allows information to be spread wider, broader, and more evenly, because now you can kind of see perhaps where the market is pricing the announcement of the, or the announcement of the next chip, right, based on how the curve looks in the future. I think all these things are ways

• 1:03:53Speaker 03

  for information to be actually shared more freely, rather than for there to be any obfuscation of this information. And so in effect, I think these markets are a great way to hedge that risk away.

• 1:04:05Nathan Labenz?

  I also asked how long these chips actually last, and whether the A100s from 2020 are near the end of their useful life.

• 1:04:16Speaker 03

  Yeah, I think the the chip health is a super interesting space to be in and understanding what are the failure modes for the hardware itself. I think, you know, chips will fail all the time for a variety of reasons. But I think what you're pointing to is the useful life conversation of hardware. And this is something that we wrote a paper on just two weeks ago, I think now, about how open-source, open-ended models that are more price elastic will often route to places where the cost per intelligence or cost per workload makes more sense. And oftentimes, those are the older pieces of hardware because they're just less marketable, right? And so we actually see the useful life extending beyond six

• 1:04:56Speaker 03

  years as a general thesis. How much further than that, it's kind of unclear. However, I will say that, you know, what we see in the forward market and the term structure is that there will always be some sort of premium above zero, certainly, and above some terminal value of like power, right? Like, the terminal value of an A100 is not just the value of the power that powers it, because the A100 will always provide some computing power as a fraction of all computing power in the world, right? Over time, I think you have a lot of interesting use cases like physical AI, at one of many, where these older chips might become more effective. However, today, we see still that these older chips

• 1:05:39Speaker 03

  are still being used, still being rented, and have maintained relatively constant demand in the last 12 months.

• 1:05:48Nathan Labenz?

  Ween signed off a few minutes later. In the closing, with just the two of us, Prakash came back to the index with an account of how the biggest compute deals get priced. The figures are his, from what he has heard.

• 1:06:01Speaker 06

  What I've heard is that your price of compute is dependent on whether you have financing or not. So if someone needs to use their balance sheet, they get to dictate the terms. So if they don't, if you don't have to use a balance sheet, then all of a sudden, like, you have high pricing. So Elon, for example, Elon had already used his own balance sheet to build out the clusters, and he had the clusters available, available, you can take it tomorrow, right? And he, they didn't need to provide him a long-term contract. They could do a month-to-month contract, and they can walk out at any time, right? So which

• 1:06:41Speaker 06

  meant that this is not financeable. A bank cannot come and give Elon a loan based on this contract from Anthropic, because a bank wants to see, like, if you're going to pay back in four years, I need to see that you're going to have cash flows for four years. I need to see that upfront. So Elon's able to charge high prices because he's using his own credit capacity, he's using unsecured credit at the at the holding company. And he can say, like, look, you need, you're getting paid 80, $90 million a megawatt, you give me the 50, and you're still making money, right? So that's a good deal. And so Elon got $50 million a megawatt, right? So I think it changes, it starts to change when you

• 1:07:24Speaker 06

  need the other guy to give you support. So if you say like, okay, Elon said, okay, I'll build it to you, I'll build it for you in a year or year and a half or two years. In order for me to build it to you, you need to give me a five-year contract. If I build it for you, you will definitely buy the compute and pay me. And then I'm going to take that, and I'm going to go to another bank, and I'm going to give it to them, and they're going to lend me money based on that. Then it becomes like, it's not really like Elon's money, it's really like Anthropic's money anyway. And the single credit is going to be from Anthropic's side, and that's really who the bank is relying on for repayment. And so then it becomes a different, different, different pricing. So then Anthropic will say like, okay, how much is it going to take you to build? How much is your interest rate

• 1:08:08Speaker 06

  going to be? And then I'm going to give you, let's say, a 20% profit margin on top of that, right? So that this is what CoreWeave is getting, for example. So if it's going to cost you like $15 million or $20 million to build, they'll be like, all right, 24, $25 million per year, right? And so that's the, that's the key difference is whether you have the money or not upfront. And those deals don't hit the index pricing. The index pricing is based on willingness to trade. So these are all tradable prices, and that stuff doesn't get traded. And so the people transacting on this index are the lowest payers. They're not Facebook. Facebook can afford to pay like, you know, 100 million dollars a megawatt. They're not going to be trading on the own index, because, you

• 1:08:53Speaker 06

  know, the own index is for people who are willing to sell compute at, you know, like 10 million or 15 million dollars a megawatt. They're willing to sell the compute. Facebook isn't going to sell you compute at like 15 million, like, they're willing to buy it 50, right? So, so these are, in some sense, the lowest pricings. But it's immediately available.

• 1:09:15Nathan Labenz?

  Part five, what the machines are actually doing. Nick Gillian is co-founder and chief technology officer of Archetype AI, and before that, led machine learning research on Google's Soli radar sensor. Archetype's model, Newton, is a foundation model for sensor data. It takes raw streams from radar, vibration, electrical current, and cameras, and reports what a machine or a worksite is actually doing. I asked him to walk through Newton from the bottom up.

• 1:09:45Nathan Labenz?

  Let's start with the data. How much data is out there that you were able to just collect on your own? And then I imagine you must have had to establish a sort of partner network to bring in a bunch of data. Tell me about the data process.

• 1:10:00Speaker 04

  Yeah, so we're getting close to a billion hours now of physical AI data that we've been able to scrape and gather and collate. In many cases, if you look on the, if you look on the web and you look at the different types of physical data that there, that's there, it's very sparse. There's very little multimodal data. It's not synced well. It doesn't cover things well. For most techniques, particularly classic supervised techniques where you're basically building one model and it has to have a fixed number of sensors and they all have to be captured with the same sample rate, et cetera. That's a massive problem. We've been trying to flip that on its head and say, if we have all of this physical data, which is captured in different sensors and distance, different systems with different contexts.

• 1:10:43Speaker 04

  How do we actually build the most advanced self-supervised techniques that can pull in that sensor data and extract from it? In the same way LLMs have done with language, we are able to then join that and mix that in with a small amount of data we can capture from, from partners and, and even in some cases that our customers can give us. To be able to really extract and understand and map that to physical language and map that to physical control. Right? So there's, there's quite a bit of, of resampling and, you know, really understanding like missing values and sensors. Is, is that because the sensor had an issue or is actually the machine itself is connected to? That's actually a feature that helps explain that machine is about to break.

• 1:11:28Speaker 04

  Right? The reason the sensor is giving you all these nines is not because the sensor is broken. It's actually, it's actually a feature of the machine, right? In those cases.

• 1:11:37Nathan Labenz?

  So the area that I've studied best that has done something kind of similar has been at the intersection of language and just images, you know, photographs.

• 1:11:47Speaker 04

  Yeah.

• 1:11:48Nathan Labenz?

  And we've seen kind of, if I had to borrow some of their terminology and map it onto what I understand you're saying, this would be sort of a late fusion approach where you have a language model. You then do like a pre-trained from the ground up encoder, specialist encoder for these different modalities. And then you're doing some sort of like cross attention between those. That's my understanding of like, you know, at a high level, obviously of what the image folks have done. How much have you been able to follow in their path? How much, you know, of the techniques you've seen them develop have worked for you? And how much have you had to go back to the drawing board and reinvent?

• 1:12:25Speaker 04

  We've definitely tried these approaches and we've had some success with them with, you know, other types of sensors, radar and time series and so forth. But the big, the big issue that we, that we have found is twofold. Number one, there's a, there's an extreme amount of language sensor pairs on the internet that can easily be scraped and built into these very large data sets. Where you have this pairing between an image and a description of that image. Right? This does not exist for radars. It does not exist for time series sensors. It does not exist for all the complex systems that we go and talk to our customers with. Right? So we have had to look at different approaches of how do you, for example, take time series and actually align that with human language.

• 1:13:10Speaker 04

  Right? So that's a large part of our, of our research work is how to actually solve this sensor alignment problem. And there's also the case there where with a single image, you may actually be able to get the full context, for example, of the scene and then align that with language. But actually with the time series, you actually need to look at a temporal window. And, you know, what is a single latent mapping to a single word really mean in those cases? Right? It's, it's not like you have a time series signal that says dog and you can align that with the keyword dog and so forth. Right? So there's quite a bit of, of work we need to, we need to do there to solve this alignment problem of how do you generalize this across many sensors?

• 1:13:49Speaker 04

  And that's the key kind of components we're building inside, inside our models, which makes them different from, you know, a standard VLM or, or other techniques. Right? Because we simply don't have that corpus of, of say, time series and language pairs to kind of apply that previous recipe you mentioned. Right?

• 1:14:08Nathan Labenz?

  Prakash asked about one deployment, a project with Kajima, the Japanese construction company, and what the inputs and outputs of the model look like there.

• 1:14:18Speaker 04

  In that case, you know, we're working with Kajima, this very established Japanese construction company. They, they build, you know, islands, they move mountains. The, the project we were working with them on was literally taking a river and moving the river. Right? So that it wouldn't flood. Right? So it was a five-year-long project. They had sensors the whole way down the river as they kind of moved down the river and they're dredging it and they're widening it. And, you know, they're, they're basically mitigating the risk of flooding. Right? And the big problem they wanted to solve is the, their company is outsourcing and subcontracting a lot of the work to a lot of other sub teams.

• 1:14:57Speaker 04

  And they want to basically measure to make sure that those teams, if they have to go and dredge the river for five hours, did they actually do five hours of dredging that day? Or actually they only did two because maybe the, the, the, the, the, the digger was in maintenance mode that day, for example, and they couldn't do it. Or because of weather conditions and others. So for, for, for that project, basically the, the input to, to, to Newton, our, our world model, was the multiple cameras around the site, typically three or four cameras, looking at a specific section of the river. And when you look at those cameras, basically you have a barge, this boat that's floating out on the river. On top of it, you have a excavator digger.

• 1:15:40Speaker 04

  And then on top of that, at the end of it, it has a end effecter that is either drilling or it has the bucket on it where it's actually dredging. Right? They want to know how many hours essentially that excavator is actually dredging the river, not drilling the river, not a crane, not other things. Right? So essentially the, the input to the system is the sensor cameras, the river sensor, hydrometer sensors that basically say how high the river is. And then a different geo locations along that, I think it was multiple kilometer site, where the actual work was supposed to be happening at, at that day and what the weather was that day. So we basically have a bunch of cameras.

• 1:16:25Speaker 04

  We have a bunch of, of time series sensors with weather and river, river information. It's fed into the model and the model is essentially outputting what looks like a Gantt chart. Right? It's actually outputting the state of that team. Where is the barge? Is it parked on the side of the, the river? Is it actually moving into position, which can take an hour sometimes to do that? Is it now actually in position and they're starting to dredge? Or is it just there and it's stationary and nothing's happening? Right? And then they can start to map these work charts over multiple years of data and start to look at these bigger patterns that a human can never spot. Like one of the patterns they, they found, for example, is obviously if there's a storm, the work on the day of that storm goes down. That's kind of, you know, everyone would know that.

• 1:17:10Speaker 04

  Right? But one of the key things they found is multiple days after the storm, when the weather had cleaned up, the work productivity of the team was still very low. And the reason for that being is the, the, the, the river water coming down from the mountains. It takes several days for it to actually reach the end of the river and go to the ocean, which is where the site was. So there was a lot of debris and a lot of churn coming through the river multiple days after these larger storms, which is actually slowing the teams down quite a bit. They'd never spotted those patterns before because they could never look at these aggregate patterns. But Newton was able to kind of spot those patterns for them. Right?

• 1:17:45Speaker 06

  How much does Newton generalize in the sense that when you go into a new site, you have to take all of the data and kind of fine tune the model once?

• 1:17:55Speaker 04

  Yeah. So we are building the model so that out of the box, you can plug in different sensors and different configurations and the model can automatically adapt to those, you know, as much zero shot as we can. For a subset of users, they can directly do that. The, the typical analogy I give customers is if I, if you showed this task to like an average person from the street, would they be able to detect the safety violation? Or would they be able to detect that that looks like a strange signal and anomaly? If the answer is normally yes, normally that means that Newton can do it out of the box. It doesn't need, you know, a specific, you know, master technician's level of expertise yet to be able to spot that.

• 1:18:32Speaker 04

  Right? For those cases where maybe the piece of equipment the user is using is very specific to, you know, that factory, for instance, or they have certain parts that, you know, they, they have very internal and, you know, unique names for the, for those processes or parts. That's typically where fine tuning or being able to bring the customer's knowledge base and actually being able to inject that securely into Newton really helps. So in that case is what our, our platform allows customers to take our model, run it directly on the customer's infrastructure. So no data is leaking out of their, their networks, either on, onsite, on-prem or in their secure clouds. And then be able to take their own data and actually do some amount of fine tuning to be able to customize the models.

• 1:19:15Speaker 04

  And typically with customers we've seen, you know, on the order of a few thousand samples is enough to be able to take the general model and then adapt it very quickly to that, that customer's use case.

• 1:19:24Nathan Labenz?

  Last question for me is about your vision of superintelligence and how you might relate to the frontier companies as we go forward in time. What I think is geez, the reasoning is already getting to like super intelligent light, you know, with all these millennium prize problem style results happening just through a super long chain of thought. Like who would have believed you could get there? So on the reasoning front, we've kind of. Come awfully far. What we don't seem to have still is like a good intuition for the many domains that matter. And what I sort of expect to happen is something similar to what

• 1:20:05Nathan Labenz?

  has happened with images where, you know, it's one thing to sort of put an image into a model and get a caption out. But when you can get the altered image out, and it's clear that the model understood both your verbal instructions and also sees the image in a very similar way to the way we intuitively see the image, which is something we are really good at, then it feels like, wow, okay, it's really got that domain. So my superintelligence vision is basically do that across a ton of different domains, many of which we don't have native senses for, with what you're building being a great example of that, right? People don't have a native sense for how to integrate 10 signals from 10 different sensors. Do you see a role for you in

• 1:20:49Nathan Labenz?

  superintelligence being sort of building the deep intuition for this domain? And does that ultimately get joined with like elite reasoners via the form of like a bidding war between a couple of frontier companies for your company, for example?

• 1:21:06Speaker 04

  I definitely see the part that we can play in a larger superintelligence system. So that's kind of how we are seeing, you know, our, where we play in this kind of bigger superintelligence loop is like, we're very focused on how to understand the world around us and be able to bring that back to natural language and machines so that these things can talk to each other. If we can do that and other folks are solving digital intelligence, that nicely will connect together very well.

• 1:21:34Nathan Labenz?

  So just to make sure I understand your answer there, it sounds like you are envisioning a world where the technology that you're building remains sort of a tool call type relationship to a core reasoning model and doesn't so much get like upstreamed and thrown into the, you know, GPT-7 pre-training mix to spit out one model that does everything in the same latent space?

• 1:21:58Speaker 04

  We're thinking more on the line of we're building physical agents that can be deployed out to the ecosystem of the world. Those physical agents then can talk to other agents and be able to, you know, bridge that gap between the digital world and the physical world. How much that's actually done on an agent to agent, you know, post-training mechanism? How much of that is pulled into the pre-training? We get to see, particularly when it comes to those more nuanced sensors like radar and LiDAR and, you know, infrared and all these things that don't look like human, you know, visible images. That'll be interesting to see how that's brought back into these kind of bigger print, you know, pre-training loops versus just it's more of

• 1:22:40Speaker 04

  a, an agent that can go and solve that problem and tell you what's happening in the physical world in real time and be able to let you, you know, interact with it, for instance.

• 1:22:50Nathan Labenz?

  Part six, the dish is not the body. Andrei Yorgescu is co-founder and chief executive of Vivodyne, which builds robotic labs that grow pieces of human tissue, lung, liver, tumor, and more from primary human cells, and tests drugs on them. In August, it opened what it calls a human biological data center. 12 robotic labs that, by the company's count, can grow more than 3 million tissues a year. Prakash had the first question.

• 1:23:21Speaker 06

  How, how do you actually like figure out what the dosage is and whether or not, you know, that is actually what the cell would encounter in the body?

• 1:23:32Speaker 05

  Basically, everything that's been done pre-clinically to date, right? Outside of testing in animals, you are just dunking, you know, some substrate, whether it be an organoid or you have cells in a Petri dish, just basing them with this, with this test compound that you want to test. And that's exactly the problem is, is like, that's not how transport happens in the human body, right? And I think it's a perfect example. A lot of cell therapies that are designed for solid tumors in a dish will completely kill that tumor. And then those same CAR T cells, when they go into a patient, just flow right by the tumor through the blood vessels. They don't even know it's there, right? And then, and thus is the challenge is modeling the transport of that therapy into the tissue. Do they recognize the tumor? Do they concentrate

• 1:24:17Speaker 05

  there? And so in Vivodyne tissues, even the blood vessels, the kind of capillary bed in that tissue is self-assembled. So these little blood vessels grow within the tissues and we can perfuse directly through them. And whenever we dose a drug, it is always through the blood vessels that are native to that tissue. And so the transport through the interface of the blood vessels and then the, you know, like, like, like meat of the tissue, right? All of its like prank mullets, its functional subparts is just like you would find in a normal tissue. The density is the same, right? And so that is just as big a part as does the drug molecularly, like bind to its target and perform this effect, does it even get there?

• 1:24:58Nathan Labenz?

  Then my question on scaling laws. I'm very intuitively bullish on this, just go collect an obscene amount of data on this kind of tissue perturbation paradigm. It makes a ton of sense to me. The skeptical view that I've heard from time to time is you can never have enough data, you know, the, the world is too big, biology is too complex. You can do that, but you just won't be able to generate enough.

• 1:25:28Speaker 05

  The, the, the challenge of this argument that it's just like this, just a size of dataset thing, is that currently even the state of the art virtual cell models that exist saturate at a very, very small fraction of the input data that is fed to them. So you can have all the data you want, but their performance saturates after a couple percent, right? And this is like, you know, both published and, you know, what we hear on the ground, right? And, and these perturb seek kind of based models. And the reason it gets so difficult is when cells are growing in a dish, which is, which is the, the kind of mode for this, for this sort of perturb seek, they are so far removed from all the feedback loops that are natural

• 1:26:12Speaker 05

  in people, that they're just trying to colonize that piece of plastic, right? It's like you have a stiff substrate. Normally cells when they encounter that are like, I have to encapsulate this. And so they are just growing. Their only goal is to proliferate. And so if you knock out genes in those cells that would otherwise affect how they perform their natural function, really, there's no big difference, right? They're, they're still just dividing. And so you end up learning this view that perturbations don't really do much of anything, or it's unclear what they do, except if a gene is like required for that cell to live, in which case, like, it kills the cell, right? And so, and so it's not, there's not even causality to extract from that. And so we think that the, that the, that the fundamental bottleneck

• 1:26:55Speaker 05

  to having this causal understanding of biology is that we need to perturb these tissues in the complex environments that they call home naturally, right? Almost like the, the image in my head is we are trying to populate a map and kind of like design a GPS through it. So we can say, if I want to get to the state, I need to, you know, like go down this road and then turn left with, with the second perturbation, and it'll bring me to this healthy state.

• 1:27:21Nathan Labenz?

  I asked how you get from a sample received to something placed in the system and growing?

• 1:27:28Speaker 05

  And when we grow these tissues, you know, a, a really big differentiator for, for our approach to doing this, and I think the, the, you know, really like critical mechanism that you need to get like realism in this, right? Is that we use primary cells, which are, which are mature differentiated cells in, that like perform our natural organ function in our bodies, as opposed to starting with some stem cell, completely absent from all the conditions that would tell it how to, how to mature into a, into a differentiated cell. And then like sledgehammering it with cytokines to turn into some cell type that we think is accurate to what happens in the body, right? We instead just kind of go to the mature source, the, the ground truth. And we, we mix together all of these different

• 1:28:11Speaker 05

  cell types that comprise, you know, the lung, for example, a very high density. And within that injected tissue, we see that these cells begin to reassemble into the native structure of that human tissue, right? With some little herbs and spices that we introduced. But the, the, the, the like responsibility for forming the native structure of human tissue is on those cells that already know how to do it. It's honestly kind of injection molding into a, into a, like into a chamber, right? So we, we like shoot in these, these cells at like at high density. And then over the course of several days, they, they reassemble themselves.

• 1:28:53Speaker 05

  They just self-organize to make blood vessels to partition into, you know, the, in the case of the lungs, like the airway where there's air and then the parts where there are blood vessels and fibroblasts and immune cells and all of this. So we are relying extremely heavily on the cells to be able to direct their own self-assembly at these very small scales.

• 1:29:12Nathan Labenz?

  Can you tell us a little bit about the mix of in silico via the foundation model versus in cell, in actual tissue experimentation today? And like how that ratio is expected to evolve? Because presumably like the big value is it's a lot cheaper, faster, even if you've got all these great wet lab technologies, just plowing things through the foundation model is supposed to be where you get like the insane acceleration, right? But then, of course, you've got to ground it and validate.

• 1:29:42Speaker 05

  That's exactly right. So, so.

• 1:29:46Speaker 05

  If we if we look at it like a reinforcement learning problem, which is, you know, my my actor is this is this, well, it's kind of the experiment, right? I have this tissue and I can perturb it in very many different ways. And the goal is I have this objective, which is to make it healthy, and I can instrument the state of that tissue very deeply. So I know, like, the the the value function is very well defined, in in terms of the state of that tissue. The the question of what should I try next, right? In in the in the large rollout that I can perform, what are the most use useful things to try next that add the most information to to my data set? So the the physical experiments kind of serve to to like to like fill in gaps in which

• 1:30:31Speaker 05

  the model is most uncertain. And the value of the model ultimately, right? Like, why even make these foundation models is the the the combination space, especially when we move into two target or three target therapies, right? If we have just combination therapies, if I take two drugs, because it's going to work much better than than one, is is so huge that you have, I mean, you could have, like, this whole planet could be, you know, Vividine Systems growing growing human tissues, and you'd still not get anywhere close, right, to the to the just sheer space of things to to test. And so we need some way to narrow down to, you know, to to to kind of just, you know, a couple times, like

• 1:31:11Speaker 05

  the total number of clinical trials in America, the the experiments that that we should physically run to to kind of test our hypotheses, right? And so and so, basically, these foundation models allow us to predict what happens when we drug this thing and that thing, or all these things together, or, you know, this thing, and then that that thing. And we we kind of optimizing the iterate in that space, and we find, for example, the thousand most likely things that will, you know, move this tissue closer to this healthy state. And then we close the loop by testing it, right? And we see where that rollout goes. And then we just do that on a loop, right? So the the the the purpose, very fundamentally, for us of these models is to tell us what to do next. And then, the purpose of this experimentation

• 1:31:56Speaker 05

  is to figure out if the thing that we thought we should do next actually does move us closer, or or how it deviates, so we can correct.

• 1:32:06Nathan Labenz?

  Prakash asked about the hardware. The tissues grow on what Vividine calls a tissue disk. And the company recently announced a second generation of it. He asked what changed from the first disk to the second. Andre started with the robotic lab the discs go into.

• 1:32:23Speaker 05

  Imagine something the size of, like, a wardrobe, right? So it's about about the footprint of a large desk, about eight and a half feet tall. And these are self enclosed labs. So fridge and freezer, and there's a, you know, 3D scanning microscope in there, and, you know, big robot arm that can use all these different tools. We want to optimize the surface area of those discs for tissues. And so in our in our second form factor, we have an increase anywhere from two to four times the density of tissues that we grow on each disk, without any sacrifice to, they're actually much larger than the than the, you know, version one of these discs. So we increase the number and the size of these tissues, and we offload a lot of the responsibility for how they're grown to the

• 1:33:05Speaker 05

  robotic system around them. Now, the the the challenge there, and like, the the kind of work that allowed us to do this, is a lot of work in software on like, the on the orchestration layer. You you you think if you had like 10,000 experiments to run in parallel, and this is not like I have a, you know, different gradient of dosing for some drug or something, right? And I can do it on a plate. It's like, I I am changing the collection point of data, I'm changing the interval between dosing with some drug, I have a, you know, a first stage regimen and a second stage, you know, therapy, I'm taking all sorts of different readouts of of these tissues. It becomes a super, super complicated experiment. And I can define it, like, quickly, like, by by, you know,

• 1:33:48Speaker 05

  I can I can explain it out loud and say, well, we want to test, you know, bone marrow and area, we want to dose these drugs in this order, we want to branch through these concentrations, and I want 3D imaging, single cell sequencing, and proteomics. And and by saying that, I can define the study decently, unambiguously, right? I mean, like, given the system of tissues that we grow, it's pretty unambiguous, what I would want to test in there. But then the robot has to be able to decompile, or to to to to compile that kind of zipped representation of the study. It it has to, it has to decompress it into this this compiled list of like, millions of actions that this robot has to perform. It has to remember that if it took the lid off of a plate, that it should put

• 1:34:31Speaker 05

  it back, right, as an example. But if there's multiple trays that it has to handle, and they're all out on the on the working deck, maybe it should deal it multiple of them in advance, right? If multiple pieces of plasticware are on the same tray, and they're coming out of the freezer to thaw, you want to make sure you're not thawing something that should not be thawed too early. And so, all like, this this this compiler behaves a lot like a compiler in in code, right, for C, where you're managing like, the prefetching of information from memory, and branch prediction, like, all this stuff, right, that that thankfully our operating system and and and, you know, hardware handle for us. But improvements on our orchestration mean that the system itself, the hardware, the automation can

• 1:35:16Speaker 05

  do much more of this. And more and more of that space on these tissue discs can be used for actual tissue biology, instead of having to integrate, like, the little, you know, helpers on there.

• 1:35:28Nathan Labenz?

  Toward the end, Prakash read Andre a grand challenge for the field, which he credited to the annual review of pharmacology and toxicology.

• 1:35:38Speaker 06

  On the annual review of pharmacology and toxicology's grand challenges in modern pharmacology, the innovation, a universal in silico or organized framework that forecasts rare, patient-specific adverse events with more than 95% accuracy before first in human dosing, is one of their target target challenges. Vividine is obviously on the on the path there. When do you think this challenge will be solved?

• 1:36:12Speaker 05

  So this is a hot take, but I feel like I feel like this is almost like picking the color of my Mars suit currently to me, right? Which is like, how about we just, like, kind of fix the bigger problems? And and and the big problem, by the way, we we are working on specifically that challenge, right, of of of like, very, very rare outcomes. But a bigger, like, far more regularly, we have drugs that would have huge potential in their efficacy in in patients, but that have routes of toxicity that are pretty common, like, many patients, like, die from it, right? I mean, many of the cancer drugs that are going to clinic today, the the the risk is not like, oh, man, you know, I've cured 999 people of cancer, but this one patient, like, ah, didn't work so well. You know, it is

• 1:36:57Speaker 05

  more toxic. The problem is, in all the patients we've dosed, they barely had any improvement in their cancer, but it gave them all of these side effects, or killed someone from liver toxicity or something.

• 1:37:10Nathan Labenz?

  Andre signed off a few minutes later. In the closing, with the two of us, Prakash put the two guests side by side.

• 1:37:17Speaker 06

  I liked how you had basically two different approaches for data. You had, you know, on the one side, you had, let's take all the messy data, let's feed it into the into the machine and try to make sense of it. On the other, Andre is like, there's just too much messy data in biology. I'm going to build my own data collection effort, and I'm going to monitor every single thing that goes in and comes out, so that I know exactly what's going on. And I and I think I think Andre probably has a good chance of solving this thing. Like, you need you need a little bit more GPU power, you need a little bit more imaging. I think the imaging granularity is probably not there yet. But you can definitely see if they manage to scale a little bit more. The the real question is, like, do

• 1:38:00Speaker 06

  you need to do 3 billion samples, 30 billion samples a year? Do you need to get everyone's genetics? Or are you gonna get, is your scaling going to be able to get you to a point where at 30 or 300 million, or wherever, it just starts to generalize a lot more, and you stop you stop needing so many so many different samples?

• 1:38:23Nathan Labenz?

  Prakash guessed it would be two or three more years before AI gives biologists something they can really use. I took the under.

• 1:38:31Nathan Labenz?

  I think we're going to see probably a reversal of the order of, like, utility and millennium prize solutions in biology relative to what we've seen in math. The issue with math is that we already had a ton of math that was like, interesting to mathematicians, but not super useful. And so, to push the frontiers of math, you kind of had to get into the, like, not useful domain of math. In biology, we have, like, tons of useful stuff that is, like, still not even that well understood, right? I mean, it's just very, it's a far more empirical than theoretical domain. And I would expect a tremendous amount of utility to emerge before we have anything approaching, like, a full, you know, mechanistic

• 1:39:15Nathan Labenz?

  or fully causal understanding of what's going on in in biology. And to some degree, you know, obviously, we'd love to have that. But we don't necessarily care, right, at the level of, like, does this cancer drug work on this patient's tissue? If it works, you don't have to have a mechanism, right? That's not part of the clinical trial approval process. It's nice to have, not required.

• 1:39:40Nathan Labenz?

  Prakash also made the case for specialists. A model like AlphaFold does its one job better than any generalist. He expects that to hold. And the version of superintelligence people fear is a single model that does every job better than the specialists and depends on no one. My answer was the pattern as I see it so far. New kinds of data get pulled upstream into the biggest model, and efficient specialists get distilled back out of it.

• 1:40:06Nathan Labenz?

  But yeah, put me down for one that has not seen a reason yet to believe that all these modalities don't end up integrated at the frontier. In a way, I wish it wasn't going to go that way. I do think safety through narrowness would be really nice. That's why I'm excited about Jev in part, you know, because it's sort of, it has a very niche role to play. You could build all kinds of scaffolding around it. I was a big fan of the Eric Drexler reframing superintelligence piece, which he, also known as comprehensive AI surfaces, where you've just got narrow, purpose-built AIs for all kinds of different niches. My hope for, for safe superintelligence is something like that, you know, just based on kind of reading the tea leaves of Ilya's comments. Could we possibly get a model that

• 1:40:51Nathan Labenz?

  starts as kind of a stem cell generalist and then matures in a way where it gets really good at its job, but also kind of settles into that niche in the way that mature cells don't revert and turn into other kinds of cells and that they're kind of limited now? They've lost their pluripotency, so to speak. I love all that stuff. I would like, I hope it goes that way. For efficiency, it probably will. But at the frontier, I just have not seen anything at all yet that makes me think that the, the single biggest, best teacher model you could make couldn't just learn at all. And that, you know, given what we've seen, that, that starts to be a bit of a scary beast.

• 1:41:33Speaker 06

  Oh, I, I, I agree. It can learn it all, but I think the, uh, learning it all at the most efficient compute rate and like, and inferring at the most efficient compute rate, which is what you alluded to with the distillation. I think that's the, that, that doesn't happen, right? So the energy efficiency is a, is kind of key because the energy efficiency means that you have like this diminishing returns curve as an economics for getting bigger. Okay. The doomer scenario is less likely to happen because you will get a more intelligent model, but it's going to be less efficient to start out with and it's going to be more energy consumptive. And that means that it's going to need like more energy expansion to get there. But the other models are going to be more efficient than it and

• 1:42:16Speaker 06

  more numerous. But I think the singleton example, I think, is not likely to happen.

• 1:42:20Nathan Labenz?

  From your lips to God's ears, as always. Um, I think that's a real constraint for sure. I mean, if you made a, you know, let's just get ridiculous, right? You made a quadrillion parameter model, you certainly would find it a bit slow and a bit expensive and you couldn't run a million agents at that scale of model today. So I do think there are some practical limits there. But then again, remember what we talked about on Wednesday, there's going to be more compute installed over the next 12 months than exists currently in the world now. So we are definitely going to test for at least a couple, few more years,

• 1:43:01Nathan Labenz?

  whether just bulking up on compute solves all, you know, quote unquote, solves or perhaps creates all these problems.

• 1:43:10Nathan Labenz?

  That is the week. Tell us what worked and what did not. See you in the morning.

• 1:45:15Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G. And thank you to

• 1:46:00Nathan Labenz?

  everyone who listens for being part of the Cognitive Revolution.