Approaching the AI Event Horizon? Part 2, w/ Abhi Mahajan, Helen Toner, Jeremie Harris, @8teAPi

Abhi Mahajan (@owlposting) explains how AI is reshaping biology and medicine, including foundation models to predict cancer treatment response and why he’s both skeptical and optimistic about current results. Helen Toner unpacks CSET’s “When AI Builds AI” report and why automated AI R&D is a major source of strategic surprise. Jeremie Harris then explores our lack of control over superhuman AI systems, fragile US–China coordination, and how to maintain situational awareness in a rapidly shifting landscape. Nathan uses Granola to uncover blind spots in conversations and AI research. Try it at granola.ai/tcr with code TCR — and if you’re already using it, test his blind spot recipe here: https://bit.ly/granolablindspot LINKS: Abhi Mahajan's Owl Posting site Heuristics for lab robotics article Deep Research on Noetik AI Sponsors: GovAI: GovAI was founded ten years ago on the belief that AI would end up transforming our world. Ten years later, the organization is at the forefront of trying to help decision-makers in government and industry navigate the transition to advanced AI. GovAI is now hiring Research Scholars (one-year positions for those transitioning into AI policy) and Research Fellows (longer-term roles for experienced researchers). Both roles offer significant freedom to pursue policy research, advise decision-makers, or launch new initiatives. Applications close 22 February 2026. Apply at: https://www.governance.ai/opportunities Blitzy: Blitzy is the autonomous code generation platform that ingests millions of lines of code to accelerate enterprise software development by up to 5x with premium, spec-driven output. Schedule a strategy session with their AI solutions consultants at https://blitzy.com Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Serval: Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week four at https://serval.com/cognitive Claude Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at ⁠⁠https://claude.ai/tcr⁠ PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Coming up, you'll hear part two of a marathon live show that I co-hosted with my friend Prakash, also known as @adapy on Twitter, in which we explore AI for Science, recursive self-improvement, and geopolitical competition. I love doing full deep-dive episodes, but I can only cover so many topics in that way. And so, I am experimenting with higher-intensity live shows as a way to deliver what I hope is the same high-quality analysis, but in a denser format. In the first half, which hit the feed yesterday, we talked to Professor James Zo of Stanford about his work on AI for Science, Sam Hammond about how well the US administration is doing

• 0:43Nathan Labenz?

  to manage international AI competition, and Shashana Tukovsky about AI agent behavior in the wild. In this second half, we talked to Abhi Mahajan, also known as Owlposting, about AI for Biology and Medicine, including the foundation models he's building at Noetic AI to better predict which patients will respond to which cancer treatments, and why, though he's skeptical of many AI for Biology results that have been published to date, he does expect trends to continue to the point where AI is ultimately transformative for the field. Then, we talked to Helen Toner about a report that CSET just put out called "When AI Builds AI," which summarizes conversations from a closed-door workshop in which participants

• 1:26Nathan Labenz?

  tried but failed to establish any consensus expectation about the impact of automated AI R&D, ultimately leading to the conclusion that automated AI R&D is simply a major source of potential strategic surprise. Then finally, we have Jeremie Harris, talking about the very challenging position we find ourselves in, where we lack both the technical means to reliably control superhuman AI systems and the trust and coordination mechanisms needed for the US and China to address this problem collaboratively. Plus, a bit of discussion of how he maintains situational awareness and how our respective personal productivity stacks are evolving.

• 2:06Nathan Labenz?

  As you'll hear, the challenges of making sense of such massive disagreement among leading AI experts and simply keeping up to date with AI developments coming at us daily comes up repeatedly in these conversations. And to be real, nobody seems to have perfect solutions. One partial solution that I can recommend, though, is using large language models to help identify blind spots. And for that purpose, I am really enjoying the Blind Spot Finder recipe that I recently created on Granola. Granola works at the operating system level, so it can capture all of the audio into and out of your computer, including, if you wish, the contents of this episode. And its recipe feature can work

• 2:47Nathan Labenz?

  across sessions to identify trends, opportunities, or yes, blind spots that only become apparent with that zoomed out view. Obviously, this is a tool that grows in value over time. But if you want to try it today, I suggest downloading the app, starting a session while you play this episode, and then asking it to identify blind spots based on this conversation. What is so cool about this feature, for active Granola users at least, is that the blind spots it identifies for you will be different from the ones it identifies for me. As I said last time, this was fun for me, but especially because it is a new format. I would love your feedback. Do you feel that you got as much value from this more

• 3:29Nathan Labenz?

  time-efficient approach as you usually do from our full deep dive episodes? Or did we miss the mark in some way? Please let me know in the comments, or if you prefer, by reaching out privately via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy The Cognitive Revolution live from February 11, co-hosted with AdaPai.

• 3:54Speaker 01

  I'm going to add Avi Mahajan. Avi is owlposting online, and he works on AI for cancer at Nordic AI. Avi, welcome.

• 4:06Speaker 02

  Yeah, great to meet you, but thanks for having me on.

• 4:10Nathan Labenz?

  You have the great distinction of being recommended to me as the ZV for AI and biology and the intersection of those two. So, big shoes to fill, big reputation to live up to, but excited to meet. This is actually the first time we've properly spoken.

• 4:24Speaker 01

  Yeah, I learned from Ron Alpha that you built an entire competitive intelligence platform, LLM-based, to feed the clinical analysis pipeline. So, and also that Claude recommends every cancer drug it sees. So, let's talk about that.

• 4:41Speaker 02

  Yeah, the typical way that a lot of, like increasingly a lot of biopharmers are interested in asset acquisition as opposed to just developing their drugs from scratch. This is partially because like China is pumping out a lot of very interesting preclinical assets. Why not just buy those for a few million dollars? They've already done the optimization. Let's just run those in patients. Most of the time, the way you look for these drugs is either you mine your personal network or you have these like clinical trial aggregation platforms that like help you do the job. Both of these are like obviously lossy and like a better way is just like scrape the entire semantic web yourself and annotate every single investigational drug you find with your company's priorities, what you think is like important to look for, modalities that are particularly interested in,

• 5:24Speaker 02

  organize that all into a table, rank it by some metric, and then you give that to the therapeutics team to work off of. Obviously, there's still a human due diligence step. These models like still are not perfect, even like 5.2, 5.3 not perfect, but it's pretty good.

• 5:39Speaker 01

  Do you have an internal eval that you run and when you swap model engines, you know, regularly, do you upgrade every time a new model engine shows up, you evaluate and then decide?

• 5:50Speaker 02

  It's like a pretty hacky process. The, uh, the, our metric for, at least my personal metric for evaluation is like amongst the drugs that our therapeutics team are really interesting and like want to move forwards on, does the next version, next generation of the LLM continue to recommend those drugs as these are like very good? And I don't actually think it was pretty good at the very beginning. I only built this like pipeline a few months ago. It remains pretty good now. I don't think there's been like any dramatic jump. I partially think this is due to like identifying what makes for a good drug is a very like qualitative process and a very vibe based like it depends on the economic status of the company. It depends on like, do we know anyone there? Because oftentimes,

• 6:31Speaker 02

  these companies don't make it easy for you to give them your money. It takes a super long process to figure that out. Yeah, it's pretty good, though.

• 6:40Nathan Labenz?

  So I definitely recommend your blog, outposting.com. I still got quite a bit of archive to work my way through, but I want to throw a couple of what I thought were kind of your more interesting, arguably hot takes at you and then get you to kind of double click into some of the intuition and implications of those. One, because we're obviously in a moment now where there's a tremendous amount of interest in creating AI scientists of of all kinds. And one of the big bets that companies are making with some serious capital behind them increasingly is that they're going to close the loop by allowing AIs to design and run their own experiments through some sort of automation, feed that data back in, and

• 7:24Nathan Labenz?

  they're going to get reinforcement learning from basically experimental results. Now, one of the things that you had said in one of your posts is that there's not a lot of verifiable ground truth in biology. And I would love to understand what that means exactly. And then what does that mean in terms of the ability to close that loop? Is there some sort of fundamental messiness or uncertainty that you see as kind of, at least in the near term, being irreducible that would become the functional limit on how much systems could learn from that kind of closed-loop experimentation?

• 8:00Speaker 02

  Yeah, I like to say that biology has no verifiable ground truth. This is probably a little bit hyperbolic on my end. But what I will defend is that there is not a lot of verifiable ground truth for the most clinically valuable problems. So like, yes, there is verifiable ground truth of like, does this protein exist in the solution? Is this like variant that you, your NGS sequencer identified? Is this like true? Those are both verifiable. But I don't think you'll quite see the same explosion of intelligence that happened in math and code as you will in biology. Because like rewards are like so cheap and so easy to get in those fields. In biology, It's just like such a long iterative process to get any iota of information.

• 8:44Speaker 02

  So like one, one easy analog to this is like, like training an RLVR model on the, on the task. Best selling. Um, there is technically a verifiable reward, right? There's like book sales, there is the, the, the country that the author is writing from, all these sources of data. But it takes 18 months to get that singular data point. And when you get that singular data point, it's very hard to trace it back to any one of these things. And like, like one, the one like biology grounded example of this is like, let's say you want to do RLVR on toxicology. This is arguably the thing, the thing that sinks the vast majority of phase one drugs out there. Toxicology sounds like a very simple topic. It does not. You can, a drug can be toxic on the order of seconds, like snake venomous. It can be order on the toxic on the order of months, years.

• 9:28Speaker 02

  It potentially doesn't kill an animal. It maybe it just leads to like cognitive deficits, heart damage. Oftentimes it's dose dependent also, it could be species dependent. All these like measures of toxicity, there's like no real way to understand them other than just observing that in vivo, like in vivo setting. And then seeing what they read it. There are companies like one, one asset based, sorry, called Axiom, which is trying to create a, um, like a model that can very easily tell like, given the small molecule. What is its tox- toxic toxicity impact on hepatocyte cells in a cell dish? It's a very clean, simple problem that probably saves like months of time in pre group settings. But it doesn't poke at like the much more important problem of how does this perform in an animal?

• 10:14Speaker 01

  Just a segue here, so isomorphic labs, I think yesterday announced, uh, they have a, they have a predictive model. Uh, which, you know, doubles the performance of AF43 on key benchmarks, binding affinity, pocket identification, and structure prediction. How does that fit in to how things go like, is this, is this actually useful? Or is this, does this just create more targets which need to be validated anyway? Uh, and it's not that useful.

• 10:43Speaker 02

  Yeah, I mean, I obviously, it's like very incredible piece of work, ISODD is, and I'm like no longer in the protein engineering field, but I like, I think the benchmark they did, like that like leftmost plot they're presenting on, that's an incredibly difficult benchmark to get better at, and they're 2X better than Lois previously. So very good, but I'm sure like you've heard the sentiment that the field is already awash with many really good preclinical assets, and the bottleneck is actually how well do these work in patients? It's, it sounds perhaps obvious that if you get better at this preclinical design step, you get better at putting into humans. That's the story that has been told for 10 years. It is not obviously clear that any of it has borne fruit. I imagine at some point it will, but there isn't really strong evidence to suggest that it does.

• 11:27Speaker 02

  There's actually this really great paper that, a chemistry paper that came out just a few days ago called "The Affinity Advantage." That paper is probably like one of the strongest bull cases, that being able to like optimize, optimize every facet of like every protein that comes in through preclinical pipeline, has like, like nonlinear or superlinear benefits to the drug development process, and it's just like a matter of time until these models get like even better. It's not an opinion, I share, but I'm sympathetic to it.

• 11:58Speaker 01

  In Dario Amade's, you know, I think one of his papers, the blog post that he had, "Machines of Living Grace," I think. He tried to kind of map out how he thought biology, developments in biology work. And he kind of pointed out that a lot of the major developments in biology comes from better kind of imaging and sensing techniques that allow you to kind of like look deeper and understand deeper on what's happening in there. And then after that, it becomes easier to do a lot of other things downstream of that, you know, starting with microscopy and, you know, which led to all the downstream developments from there and et cetera, et cetera, et cetera.

• 12:39Speaker 01

  What do you think are potentially the developments which might be coming up in the next four to five years that might do something like that?

• 12:48Speaker 02

  I guess like I would like vaguely gesture to building models that generative models of human in vivo biology. I think there's a discussion to be had like what other instruments do we need, what other modalities do we need? But I think there's a lot of low hanging fruit in simply collecting a huge amount of highly rich data collected from like real like human tumors, intestinal lesions, plasma readouts, and just feeding a model with that information and like not paying attention to any of the in vitro or like otherwise biologically unrealistic settings. And then from there, maybe you get access to like a human, a genuine bona fide human simulator of biology. And maybe

• 13:32Speaker 02

  that's like a lot helpful for like fixing the like current state of 97% of oncology trials fail. I think that like the the Dario pitch of like scientists in a data center like churning out interesting ideas, I think there's already like tens of thousands of PhD students churning out very great ideas. Most of them can't be validated because it's too expensive to do it.

• 13:54Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:00Nathan Labenz?

  So that connects pretty directly, it seems like, to what you are doing in your work on cancer at Noetic, right? You guys are focused, first of all, at the roughly the clinical stage and try to predict what drugs will work best for a particular patient, given some relatively deep data about their specific condition, right? So maybe walk us through what that looks like. I was interested to learn that it is basically a foundation model, you know, with lots of different data sources thrown into it. And also that it sort of is trained with this kind of

• 14:43Nathan Labenz?

  Masking strategy where the the idea is that the model has to learn how to predict from partial data, whatever partial data it might have. And I'm a big believer in that strategy because there just is so much, obviously so many modalities and so much you know, noise going on inside the system that we don't understand. I've been a big kind of speculator about that being a driver of how AI can help in health over time. So give me the kind of double click past what I have been able to learn with online research into what you guys are doing.

• 15:13Speaker 02

  Yeah, so the the like, let me start with perhaps like the economic pitch for Novartis, like 97% of oncology trials fail. And you can look at that and say like, well, we're awful bad at designing these drugs. Maybe we should get better at designing them. But like one interesting phenomenon is that if you look at a lot of the papers are published after a cancer clinical trial fails, there's usually some patients who did respond to the drug or responds to the regimen they were they were on. And the researchers try really hard to figure out like, what is like, what is the exact biological archetype that makes up this response population? And they always come up with something super complicated, very heterogeneous. It's like this particular cytokine group or granzyme, granzyme genes

• 15:55Speaker 02

  were highly expressed in the response population. It never leads to anything particularly interesting. And so one argument you could make is that maybe the biomarkers that define patient response for this particular drug is like non-human legible. Like you need a black box biomarker to encapsulate whatever that piece of information is. And so Novartis is kind of built around that thesis. We collect vast amounts of human tumor data. We profile them at four levels of modality. So pathology, which is kind of like the blue chip that like almost everyone has, spatial peridomics, a 16 plex panels to identify cell types, wholeplex spatial transcriptome. So this is 19,000 genes over the entire surface of a tumor, then to identify like functional state of the tumor, and then exome sequencing

• 16:37Speaker 02

  to identify genetic alterations. So is this KRAS positive? Is this SCK knockout and so on? And then the the the ML angle is that you train, yeah, like exactly, you said, like a self-supervised mass model in the hopes that like, one, you get a very good representation of any given tumor that walks into the door. So you now have like the ability to place like, in the universe of all the cancers I've seen, where does this patient fall in that embedding space? And so that's what like we are doing a lot of. We gather patient samples from like people who have ran clinical trials, they have patient samples, we profile them, we run that through their model, see if the response population falls into a different area than the non-response population. And if it does, maybe we have access to a biomarker that no human on earth

• 17:20Speaker 02

  understands, but we uniquely are able to. The more interesting thing you can do with us is use the generative capacity of the model and knock out specific transcripts or specific genes and see how that changes the expression of transcripts within the tumor microenvironment. You can imagine there's this like concept that's appearing in the cancer literature called nudge drugs, which are drugs that don't actually operate on the like the immune system or like really the cancer, the cancer site itself, but rather push it in a direction that makes it more sensitized to other drugs. And so you can imagine, oh, I'll, I'll knock out this, this particular transcript and then I will like hallucinate what would it be like if I added a KEYTRUDA, which is like an immune checkpoint

• 18:04Speaker 02

  blockade that operates on the PD1 axis into the, into the site of the tumor. And maybe now you predict like, oh, the tumor is like highly inflamed. It's like hot. There is, there's a high chance that it'll just like melt away entirely. Yeah, like those are the two big economic and ML strategies we're pursuing.

• 18:19Nathan Labenz?

  Yeah, that's really super exciting. When you talk about the, first of all, identifying or, or having access to, I think was your phrase, biomarkers that nobody else has access to because you can see a sort of divergence in where different patient populations fall in embedding space. Do you have any means, and you know, if not, maybe I can introduce you to the good folks at Goodfire who just did a version of this with identifying biomarkers for Alzheimer's that had been previously not identified. But do you have any means right now of saying, okay, because one thing to say, these things are falling, you know, these patient populations are falling into different parts of embedding space. It's another thing to then say, why? Like, what is it actually that is causing that divergence?

• 19:02Nathan Labenz?

  How far are you guys along in terms of being able to make interpretable what it is that the models have learned from their unsupervised training?

• 19:13Speaker 02

  Yeah, the, the, the premente and Goodfire post was very interesting to read. We do have like a Meccan Turb research group internally, which like is exploring these ideas. I have no doubt they'll find something interesting, but one argument against doing this at all is like, why do we care about interpretability? Like in a clinical setting, we might care about interpretability because like the FDA gets very upset with you if you try to do anything that's black box. And maybe that was true a year ago, but circa, I think September or August 2025, there was a pathology AI company called Artera AI, which came up with, was it basically a companion diagnostic as to like, like would they intake in the pathology slide of your prostate tumor and they will predict whether

• 19:56Speaker 02

  you respond, you will respond to androgen deprivation therapy. They have no idea why this model works. They've like retrospectively validated on thousands of patients from prior phase three trials and the FDA was fine with that. So one argument against doing that is that why, why spend a ton of resources exploring something that the primary regulatory agency you care most about doesn't really mind whether or not it's white box or black box?

• 20:20Nathan Labenz?

  I guess the obvious answer would be because presumably that knowledge would be a great input to further experimental ideas or other knowledge, but maybe you think it's just so hard to, you know, I don't know, there's no verifiable ground truth or something that would prevent that from working?

• 20:36Speaker 02

  Yeah, I guess like, like, what, like, um, the, like, what was discovered in like the preemptic at Firepose? I forgot what exactly it was. There was something about like, like fragment omics, like something about how the genes are fragmented, like, is a potential biomarker for Alzheimer's? It's a very interesting piece of work. It sounds very expensive to validate it. And so I imagine like we would run into the exact same problem. Like maybe we get like, we have a very good hypothesis, like what, for what comes out of the system, but we already have so many other hypotheses, potentially ones that are like even have like higher literature backing. I could imagine a world in which like mecan Terp as a field gets so good that you can like triage that this thing's going to be really easy to validate, this thing's going to be really hard to validate.

• 21:16Speaker 02

  But right now, the way that a mecan Terp usually works in biology is like you're staring at like semantic segmentation plots a lot and like trying to think like, oh, is this real or is this fake? Is this the mod, is this the model like identifying this very spurious correlation? And that time just simply feels better spent elsewhere.

• 21:36Nathan Labenz?

  Interesting. Okay, here's another idea of a place that it might be well spent. Continual learning, of course, a huge theme right now in AI in general.

• 21:46Nathan Labenz?

  The first conversation we had today with Professor James Zhao from Stanford included a little talk about their recent paper, "Learning to Discover at Test Time", where they're using, you know, here like autoregressive large language models and giving them problems like make a faster CUDA kernel or, you know, find a better solution to this math problem, you know, with a lower bound than anybody has previously found. Or, you know, find a better solution to this math problem, you know, with a lower bound than anybody has previously found. And they, interestingly, kind of flipped the usual model of like what we're trying to do when we create an ML model on its head

• 22:29Nathan Labenz?

  and said, "What if we just try to get this model to produce the single best answer that we can? And we don't care if it generalizes. And in fact, we'll probably throw away this model after this like test time fine tuning. What we want is the answer." And they were able to find at like relatively reasonable cost, like $500 compute cost, that they were able to actually get some new state of the art answers on some of these highly technical questions. If I'm a cancer patient and you've got a general foundation model, a question that naturally occurs to me is like, "Can you fine tune this on my data? Can we do some like test time tuning? Can we do sort of intensive masking on like just my samples? And like really dial this thing in to understand my particular biology?

• 23:12Nathan Labenz?

  And then would it be, if we did that, would it be like more accurate for me? Do you think that, is that line of thinking have legs? And why not?

• 23:22Speaker 02

  So I actually, I actually looked at the paper and they actually have a section for biology. They do like single cell RNA denoising using this like a test time training model, which I thought was really interesting.

• 23:34Nathan Labenz?

  I guess my instinctive answer is...

• 23:37Speaker 02

  It's an interesting idea. It very well might work. And it falls into like the bucket of ideas that we would simply have to try it until like to make sure that it does or does not work. The results for a single cell RNA de-norising that's in the James Chow paper are like certainly good. They're better than the state of the art, but they, which I thought, uh, they, they like for each one of these case studies, they attach like a note by like an actual domain expert saying like, how useful is this in practice? And the domain expert in question for the single cell RNA section did say like, this is very cool. But like at the end of the day, we don't really care about the results of single cell RNA de-norising. We care about some biological utility that is underlying that.

• 24:17Speaker 02

  And so maybe you get better at solving this like verified task problem, but that doesn't translate to anything actually useful. Maybe it would be like for the response, non-response prediction case, but it kind of just sounds easier to fine tune the model, like using normal supervised, supervised learning. Like, why go through the RL process if the end result is like binary? You know, like I think they even called out in the paper, like the setup isn't really meant for binary or sparse learning tasks. It's meant for like fuzzier things.

• 24:45Nathan Labenz?

  Yeah, they're working on that, but it's not that yet. I guess maybe zoom out and like, you, you kind of alluded earlier already to this idea that a lot of people think we just need better ideas for drug candidates. And your consistent position is like, that's probably not really the bottleneck. And you made a really interesting point around how the ability to evaluate, more accurate ability to evaluate those candidates, drives a lot more value than just throwing a lot of, a lot more candidates through a pipeline. The quality of the pipeline matters most, uh, more than it's like scalability. So.

• 25:29Nathan Labenz?

  Again, I think you kind of have suggested where you think this can come from with just like large scale foundation model style training. But give us the the next level of depth on that. Like, why are all these other ideas not so exciting? And is this basically just a bitter lesson sort of idea where all your cleverness is going to be washed away by scale? And so keep your eyes on the prize. You got a data max and compute max until you solve it all. Is it is it kind of that?

• 26:00Speaker 02

  I guess I kind of view things in like three ideological camps. The first of which like the first one is like maybe us like we're we're indexed very heavily on like human data is the only thing that matters. You can start from like in vitro settings and bootstrap your way up to something more complicated. You need to start with the most complicated thing to begin with. The second camp is very interested in modeling like single biomolecules and like their interactions. And then the hopes that like maybe you can bootstrap your way upwards, but you raise the absolute like success rate from maybe 5% to 20%. And maybe that's all you need. And I think like that, like the second camp like defines the vast majority of ML bio companies that exist today. I think like some of them have clinical candidates that are ongoing right now. We'll see what the results are. I think generally it doesn't seem like there's had been like a massive step change in their

• 26:45Speaker 02

  ability to design drugs. And this is not like knocking them. Drug discovery is hard. It's everything's a bet at the end of the day. The third camp is it's like maybe it doesn't it's not really a for profit thing, but you can just like improve the clinical trial process to begin with. And this is arguably the path that they like China, like this is like China's main advantage. So they're able to run clinical trials far more cheaply than anyone else, partially because of like lower cost of human labor, but also because they've just like set up the system pretty nicely such that it's not such a huge regulatory and financial headache to get things going. This has like some downsides. Drugs are treated innocent before proven guilty. The FDA is the other way around. But the obvious benefit of doing that is like you're betting neither on the AI and human data getting better or the AI in vitro settings data getting better.

• 27:30Speaker 02

  You're trusting that the typical drug design process, if just made slightly more financially efficient, will like improve things on. I think all three of these are important. And I think it would be probably like grandiose of me to like assign an unequal weighting to any one of them. Each one feels important to push on.

• 27:49Speaker 02

  There have been some interesting, oh, sorry. Let me sneak in from.

• 27:53Speaker 01

  I'm gonna take a little bit of a segue to something you said earlier, which is that a lot of the new INDs are coming in from China. Like, what has, what has happened in the last, like, couple of years? Is it an AI thing? Is it, you know, I, the CEO of Dinko Bioworks was on TVP and yesterday, he said they just have more hands. And some people believe it's a regulation thing. Some people believe it's a clinical trial registration. We can just register more people. Like, what is driving? Some people believe it's a US cost thing. What is driving this transfer of, you know, basic R and D to China at this point?

• 28:31Speaker 02

  I, I, I think like this particular subject is very deep. It's not something I have expertise in. I, like, my, my instinctual thought is that there are many different answers to this. And the one that I'm most like, I, I think is most interesting is the idea that, like, China was always, like, was always, like, like a very good generics manufacturer. And that's like, that's where they started. Slowly, they extended their way into like WuXi, having a very good CRO ecosystem. And then at some point, enough talent began to be incubated in China where they began to realize, we have all this infrastructure here, why not just develop our own drugs? And there is something very important about having this, like, very close interplay between

• 29:15Speaker 02

  both the person who is designing, designing the drug and the person who is actively doing wet lab assays on the drug. Whereas in America, you have like a super long feedback loop of like, I need to get my SAB together, I need to go reach out to VCs, I need to go buy a lab. Whereas in China, that ecosystem is a little bit set up already. Actually, maybe the only missing part is that the VCs are still like not super, they're more like, like, risk averse than perhaps VCs in America. But the co-location of like, the grunt work and the intellectual work is actually surprisingly important. A few months ago, actually, uh, last year, I interviewed one of the very few people doing novel biotech research in India, a guy named Soham, who runs a company called Popfax. He said, like, this is the primary reason why he expects not only China to start producing really interesting drugs,

• 29:59Speaker 02

  but also potentially India, potentially Egypt, places where there is intellectual capital. There's a lot of hands. And it's just, like, those combinations lead to really good, like, compounding results.

• 30:10Speaker 01

  Indeed. And then does that accelerate with the AI models? This kind of co- AI co-scientists? Does that, does that mean that, you know, even if they don't have that much intellectual capacity yet? They can, they have the hands to, to carry it out?

• 30:26Speaker 02

  I guess like this is, this is something that's like, it's a little bit opaque to almost everyone as to like, what exactly is the level of? Um, how impressive are the bio AI models coming out of China? I think there's certainly some interesting work that has been done. It's not clear to me that there's anything like radically new there that won't be found anywhere else. A fair amount of it is like,

• 30:48Speaker 02

  like I like, I don't want to say this in a disparaging way, but it is like, like scaling up stuff that was originally developed in either the UK, London or America. And there's like nothing like you, there's, it hasn't been really like a deep sea thing where there's something like radically crazy that comes out of any of the Chinese labs. I obviously could be wrong on this, though. There's, I think the, wh- whatever the bio AI labs are doing in China, there's like much less American visibility, visibility around it.

• 31:11Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 31:17Speaker 01

  Uh, well, okay, go ahead.

• 31:19Nathan Labenz?

  One big question I have is kind of, I find it very hard to calibrate myself on how excited I should be about all these AI for biology and AI for medicine developments. I know that,

• 31:35Nathan Labenz?

  you know, there's always these kind of headlines, AI does this, AI discovers this drug, you know, I've done episodes of the, of the cognitive revolution on it, one with Jim Collins, you know, he has created a bunch of antibiotic candidates. You know, there's a long list, right? And you know, Professor Jao did the nanobodies thing that came out of the virtual lab. To hear him talk about it earlier today, it sounds like those were like reasonably well-validated, but then you always get this other side too, that's like, well, not so fast, you know, it's all very messy. We got a long way to go. You know, most of these things don't pan out. And I feel like that sort of parallels the debate that we hear in a lot of different domains where, you know, even in like programming, which is one of the more, let's say, legible domains, we've got

• 32:19Nathan Labenz?

  something like a meter study that showed slowdown of developers, and that was very confusing. I'm like still quite confident that it's making me faster. And I kind of want to throw that away or, you know, there's of course, just a lot of denial and cope out there and all sorts of motivated reasoning. How should one try to?

• 32:38Nathan Labenz?

  Ground their worldview. Obviously, subscribing to outpost thing is something everyone should do. But what else would you advise me? Like, how can I patch these blind spots in my worldview? Or get to a better position from which to have my own sense of what really counts? Or what really matters and what doesn't? Because, again, this happens all over the place where there's this disagreement, even among like some of the most informed people about just how big can AI reason? Or like, how big of a deal is it going to be? But in biology, it's particularly hard for me to make sense of. So I'd love to get some tips for how to climb the learning curve faster.

• 33:16Speaker 02

  I've actually written about this in the past a very long time ago. The title of the article is "Five Things to Keep in Mind When Reading Biology ML Papers." The long and short of it is the evaluations in biology, I think, are very difficult. And so you see this thing about like, in like more typical wet lab biology of like, oh, we cured cancer. It's in the mouse. Who knows when it'll actually translate to humans? There's a very similar phenomenon in a lot of biology ML papers where they're doing something that feels like it should be useful. But there's like a lot of things that they're probably hiding from you when explaining the results. That would only be obvious to a domain expert.

• 33:57Speaker 02

  One, I think, really funny example of this is like, like small molecule binding affinity papers. I've written about like one company's work in this, but they found that these, let's say you're able to predict like these set of molecules bind to this target. These set of molecules do not bind to the target. And you're very happy with yourself. You publish a nature article about it. What these folks, a company called Leash Bio found is that this can often be confounded by which chemists actually produced the molecule in the first place. Because some chemists are very attached to specific targets. They're very good chemists. So they often produce things that like bind to that specific target. And these chemicals, like very importantly, all look very similar to each other. And there's this type of similarity that's like a very human vibes base.

• 34:41Speaker 02

  And it's hard to pin down to a singular metric. And so they found that like these molecules are often confounded by like overlap. I think this like, and then like these problems just like appear over and over again across like the in vitro biology, like biomolecular generation where like you can be confounded by variables that you did not know even existed in the dataset. And so I would probably like, like they named that as like the thing to be most aware of when reading these papers. I like, there were like a few people I like trust on Twitter and like people in real life who can give a pretty good overview of any arbitrary paper, but like, I think like with like L L M's, like popular science people often like retweet them and say like this

• 35:24Speaker 02

  is transformative and like more often than not, they're probably correct. Opus 4.6 is just like genuinely crazy. I think when people do that for bio ML papers, there's like a 50 50 chance that they're completely missing the point because they don't, they're not in that field and they don't understand how the failure modes emerge in these models.

• 35:39Nathan Labenz?

  Yeah.

• 35:41Nathan Labenz?

  Do you think that an opus can like help me identify those blind spots? Is it?

• 35:46Speaker 02

  Yeah, yeah, yeah. So, sorry, go ahead, go ahead.

• 35:48Nathan Labenz?

  Yeah, is it good enough to do that?

• 35:50Speaker 02

  I, I actually, I've written an article about this also titled, uh, can, can Owen preview fine mistakes amongst 56 M L S B papers? M L S B is a structural biology workshop in Europe. And it's not very good at it. Uh, like this was like, obviously last generation of models, maybe it would be a lot better. But there's a, there's, um, there's, there's some problems that are going to reoccur in almost every biology ML paper of like, oh, your train sizes aren't large enough, your test sizes are like not stratified correctly. But like you kind of just like learn to pick your battles in this field and you just move on. There's like a lot of more fundamental problems with article, with these papers that L L M's, in my experience, is like often missed entirely.

• 36:31Speaker 02

  I think the funnest, like I, I, in almost every article I've written, I have found that L, like L L M's tell me something about this particular subfield that the domain experts completely disagree with. And they say like, like that is not how you should think about this domain. It's like, that's not the real problem we should, we're actually worried about. I don't know why this is the case. It's, it's kind of fun. It's like, like a domain of science that L L M's still haven't quite captured here in taste.

• 36:55Nathan Labenz?

  Yeah, fascinating. Okay. That leaves a lot of work in front of us. Do you want to go back briefly before we break to Noetic again? I mean, I've, fortunately, my son is doing well. He recently got cancer three months ago. I've like had a, you know, intensive crash course in cancer and I hope to be able to close the book on it and, you know, return it to a more intellectual and less personal interest going forward. And I think we're on good, solid track to do that. But how, I think you've demonstrated in this conversation that you're like, not getting too carried away with the promise of what AI systems can do. We've got the data center of geniuses. We've got the century of progress compressed into five years kind of visions.

• 37:40Nathan Labenz?

  How much would you shave off of those notions just to describe your own expectations of what Noetic can do specifically and maybe what the field more broadly is going to be able to accomplish?

• 37:52Speaker 02

  I think I'm very optimistic that like human simulation companies like akin to Noetic, but I think there's other players out there as well, will be able to vastly help with the results of at least a few clinical trials within the next few years. That feels like almost, like you're not even paying attention too much to the trend lines. I'm almost like indexing on like what we're capable of today. I think it's like pretty obvious, like there's papers going back like years that are able to show that, oh, we've developed an ML model is able to, better able to stratify patients. The problem has always been like an economical one. And like, how do you actually deploy this in a real setting? I think we'll be able to do that just fine.

• 38:33Speaker 02

  I think phase one, phase one drug, sorry, the failure of phase one drugs will go down. And I think this has already been slightly proven out. And then like, I think a McKinsey study that was done a few years ago that showed that like AI-designed drugs have like a 5% to 10% lower failure rate, which may be noise, may be real. I do kind of expect those trend lines to continue a little bit. Where I'm like most unsure of is whether these models will able to discover brand new targets entirely, which is ultimately what people will care about. I think there's like a I think believing that these models will be able to find new targets far faster than humans would really requires you to index heavily on the trend lines. And I want to index on the trend lines.

• 39:15Speaker 02

  So I believe that these models will be able to deliver very good target finding. But I'm also very sympathetic to the mindset of finding targets is just such an unbelievably hard problem that will not make a dent in it because you need this like human iteration feedback loop. And unless you build a really good human simulator, which is our bet, like you're not going to get close to solving that problem.

• 39:34Speaker 01

  The way I put it is usually you can kind of see like one order of magnitude ahead, maybe two. No one can see three orders of magnitude ahead. It's just not possible. You have no idea what's going to happen. Abhi, thank you so much. I learned a lot from this and hope to see you online, hope to read more of your blog.

• 39:55Speaker 02

  Yeah, absolutely. Thanks for having me on.

• 39:57Nathan Labenz?

  Thanks for being here. We'll be working our way through, I'll posting archives for some time to come.

• 40:03Speaker 01

  All right. Our next guest is Helen Toner, who runs CSET at Georgetown, who's a former OpenAI board member. And there's two competing views here. She has, on the one hand, the intelligence explosion is coming. On the other hand, AI capabilities may be permanently jagged. So let's add it to the stage. Helen, nice to have you.

• 40:25Speaker 04

  Hey, thanks for bringing me in at the end of your marathon. Impressed you guys are still going strong.

• 40:30Nathan Labenz?

  There's so much to cover, you know, and we've all got to accelerate our personal productivity timelines and try to pack more information into, you know, the same amount of time. So experimenting with ways to do that.

• 40:44Speaker 04

  So I want to talk super fast.

• 40:47Nathan Labenz?

  Yes, please. That's honestly one of my reservations about live content is I listen to everything at 2X speed and I'm like, I can't listen to it 2X speed if it's live.

• 40:56Speaker 04

  I mean, my constant struggle is to talk slower than I naturally want to. So if you want me to talk double speed, I'm here for it.

• 41:02Nathan Labenz?

  Please go fastest you want.

• 41:06Nathan Labenz?

  Okay, so you guys just put out this report. I think this is obviously a great candidate, if not a shoe in for the most important question of our moment. What is going on with the possibility of automated AI R and D? Do we have this tipping point where we're starting to hit recursive self-improvement? And if so, like, how big of a deal is that going to be? You guys brought together a bunch of people that authored this report and some others as well that aren't necessarily authors, but contributed to conversations. I understand, like, quite a few people from frontier model developers. And it strikes me that, like, this debate goes back basically to the beginning of AI, right? There was the idea very early on that we could have an intelligence explosion.

• 41:48Nathan Labenz?

  When I started reading Eli Ezer in 2007, you know, he was very worried about this. And yet, you know, you've written, I think you put your finger on something a lot of people were feeling last year when you said, even though what passes now for long timelines is pretty short. And yet, the disagreement on this topic seems to be as fundamental and seems to be kind of as impervious to new evidence as it has ever been. So maybe just for starters, like, take us inside the workshop. Give us kind of the lay of the land in terms of what are the world models that people have? And why are we still just working from so much intuition, despite the fact that we now have, like, you know,

• 42:30Nathan Labenz?

  what in some, you know, circles would even be called AGI out there, you know, as products for us to use today?

• 42:36Speaker 04

  Yeah, so this was, this workshop was held in July last year. It was maybe one of my work highlights of the year. It got started, so it was a day and a half. We brought people in, we had, yeah, people coming from some of the frontier companies, policy, you know, a bunch of great people to get a sense of kind of what the vibe was like. We started out, the first session was about kind of how is AI being used to automate AI R and D right now? And we had presentations from people who were doing that. And before the first break, we had Ryan Greblat from Redwood Research, Nicholas Carlini from Anthropic, Dushash Kapoor from Princeton of AI as Normal Technology, and Thomas Larson, who's one of the AI 2027 authors. They were all like, Arguing so fiercely in a friendly and productive way, like before the first break, that they just like, everyone else stood up to go and get, you know, coffee and drinks

• 43:19Speaker 04

  and snacks, and they just kept on arguing right through the break, which is great. It was exactly what we were looking for. But I think did sort of preface something that we knew going in, which was there are really different perspectives here. And that, you know, the workshop was Chatham House. I feel okay giving that anecdote because they ended up writing, one thing that came out of that was Nicholas was pushing the others constantly for like, okay, you have such different views about where things are going. Where's the first place that you actually disagree about what we'll see? And they found that as they're looking out for like what we're going to see in 2026, 2027, they actually agree a lot about kind of what we're going to see before we get to that recursive point, which is kind of a bummer. Like it's, you know, it's nice that they agree and can, they were opposed about that, actually, which is the reason I feel fine, you know, sharing that anecdote from an otherwise Chatham House workshop.

• 44:03Speaker 04

  They were opposed about the stuff they agree on. But it sucks that like, that means that it's actually going to be hard to identify in advance whether we are heading into a recursive loop or whether we're not. Two big things that I think, so what we were trying to do with the workshop, one was like, get this idea of recursive self-improvement out of sort of purely like Silicon Valley, San Francisco, like really AI-filled spaces and make it like, explain it, present it to a wider audience, let people engage with it. But then also actually try and make some progress. I'm like, okay, why do people disagree about this? What is happening? What might happen in the future? You know, what indicators could we gather? Stuff like that. And I came out of it thinking that maybe two of the core disagreements here. One is, does AI truly replace all of what humans can do? So does it, does you get to that like fully automated?

• 44:45Speaker 04

  Because if you're going to have the big, the really like scary recursion, that's probably what you need. It can't be that, you know, every, you have a, we could have much more productive human researchers. You could have, you know, the Alec Radfords and the, you know, Eli Sutskever is managing fleets of AI researchers. But if it all has to come back to them and they have to process and digest and like think through the research, you're not going to get that like massive. Recursive loop. So that's one piece is like, do you truly get humans being fully replaced? Because if not, then maybe you have some parts of the workflow being, you know, really accelerated. We have this diagram in there of like, sort of an Am Dal's law kind of thing where Am Dal's law is basically, if you have a process that depends on different inputs and there are different potential bottlenecks, then if you speed up one part of the process, the bottlenecks will just bite somewhere else. And so it may be that, you know, you speed up the coding part of AI research,

• 45:29Speaker 04

  but if you don't speed up other parts, then you don't end up speeding up the whole thing very much. Or, you know, do you have, I think another mental model that people who are skeptical that this is going to really go crazy, a mental model they bring is like, okay, we have a long history of computers doing more and more of the lower level work. So, you know, we don't have to do punch cards anymore. We don't have to write assembly code. We have these higher level languages. And so, you know, for example, AI doing more of the coding is just another natural step in that process. And humans, you know, this is kind of a like expanding pie model. The amount of like tasks that we realized can be involved in AI R and D expands. And there's always that outer band that the humans can do while they're automating the inner bands. And I think that is very different from the view that, you know, other people, sort of more that AI 2027 authors would have,

• 46:10Speaker 04

  or, you know, lots of other people in this space, which is no, first you automate some of what the humans can do. And then you automate all of what the humans can do. And then, you know, you kind of, you go until some other bottleneck hits. So then, you know, the other question is like, okay, what are those? What are those bottlenecks? We can, you know, we can talk about that as well. But I think those are two of the biggest questions that came out for me. One was, are you truly going to automate everything, including what all the humans can do? And the other is, if you do, you know, how soon do the bottlenecks bite?

• 46:36Speaker 01

  So Schulto Douglas, who is now at Anthropic, had this idea of a software-only singularity, he calls it, where we get very good at coding and like all of the digital stuff, including AI research, I presume, but not at producing power or copper or all of the physical substrates, which are going to be required in order to support this expansion. How would you feel? How do you think that fits in? Like the fact that maybe the digital stuff happens, but the physical stuff just doesn't.

• 47:10Speaker 04

  Yeah, I think there's two versions of this. I think some people, when they talk about a software-only singularity, they basically mean like it turns out that software is enough to get like absolutely crazy recursive loops. So I think there's like Tom Davidson at Forthought Institute has written about this, for example, and you know, maybe you can get just like massively more intelligent systems having massive impacts on the world primarily through sort of software improvements. There's a different thing, which what you're describing of the version from Sholto is a different thing, which I would think of as more like sort of a jagged software-only intelligence explosion, meaning the AI is getting much more capable in certain ways and like, but its effects on the world are very limited because it is software only. And this kind of gets at another, another thing that I found really, really helpful from the workshop and really

• 47:53Speaker 04

  interesting, which is people have very different intuitions about, okay, assume that you have an AI that is very, very good at AI R and D. What does that mean for what the AI can do elsewhere? And I think some people are like, okay, well, if it's very good at AI R and D, then it can like train AI models to do whatever. So it can do whatever. And you know, maybe there's like, maybe you have to spend like a week gathering data or something. But then if you want to do some arbitrary task, you could do it. Whereas I think other people have an intuition of like, okay, well, even if it gets very, very good at automating AI R and D, this sort of most, you know, software-based task, it's still going to really struggle to, for example, design new biological molecules, or it's going to struggle to like, think about geopolitical strategy questions, because you have to like, actually go out and like, see how different countries and decision makers will react and things like that. And that is a piece that I feel like goes under in a lot of these

• 48:37Speaker 04

  conversations. It's like, what is that connection between AI that can do incredibly good AI R and D and AI that can like affect the world in non-AI R and D specific ways, that we also tried to kind of like tease apart a little bit in there?

• 48:52Speaker 01

  Would you think that's kind of like the connection between like, okay, now you have AI doing AI research, that's affecting the economy, it's also affecting the political economy, and then you have to have mitigations to a political economy for this to work out? Does that mean you you might need the AI, AI research to go into how to fix the political economy, which is going to be a little bit scary?

• 49:15Speaker 04

  Yes, say more about what you mean by affecting the political economy.

• 49:18Speaker 01

  Uh, in the sense that, for example, right now, you have Bernie Sanders saying that, you know, we should have a moratorium because he's scared about jobs, right? He's very scared about jobs. Uh, he wants a moratorium on data centers. I think there's like six states with a moratorium now, including New York State. And so one of AI research leading into, hey, how does AI fix the political economy? How do we deal with the humans? How do we mitigate the impact that we have on the humans? Like, is that something that you think would happen with the first, the first configuration of the software-only singularity, in the sense that it's not jagged and also affects the political economy?

• 49:53Speaker 04

  That, yeah, yeah, that's, that's the kind of thing. That's right. If you're positing that you can just have a sort of software-only singularity that is going to radically transform the world, then it's going to have to be able to do things like, okay, and then the company like deploys, you know, chatbots that talk to enough people that convince them that data centers are great. The data centers all get built and the moratoriums get rolled back. Yeah, that, that kind of has to be built in. That's right. Um, which to me intuitively feels like it's sort of a different skill set and also is more dependent on like deployment and rollout and adoption. Uh, yeah, so I tend to be a little, a little more skeptical there. But yeah, I think that's, that's an example for sure.

• 50:26Speaker 01

  I see.

• 50:27Nathan Labenz?

  One kind of odd, seemingly to me, odd pairing of beliefs that I observe and I sort of detect in the report is the idea that among the sort of more skeptical folks, that there's going to be a plateau and also that plateau is going to be subhuman. And then on the other hand, it's like, it's gonna, not plateau, it's just gonna run away, you know, have some sort of singularity. I, a, a position that I feel pretty intuitively attracted to that I don't hear too often is the idea that maybe there will be a plateau, but it could be very easily a superhuman plateau. If I try to zoom out, you know, as far as I possibly can and look at like life on Earth, I would say it seems like

• 51:11Nathan Labenz?

  Humans are sort of part of maybe an entry into a steep part of an intelligence explosion or, you know, an S curve of capability. And I don't think we're the end of history, but we were clearly better than what came before. And that was enough to take over the world. And so I think I just don't hear too many people say, yeah, it's not necessarily going to be a singularity. It's not necessarily going to like, you know, go totally beyond comprehension. But in the same way that we were just that much better than Neanderthals. And it might not have been that much, but it was enough to change everything. I kind of feel like there's not too much more room between where the AIs are now and where they will soon presumably be.

• 51:48Nathan Labenz?

  And even if that doesn't like, you know, go sort of critical from there, it feels like we're like, it's very hard for me to imagine that it's not enough to be transformative. So is that a position that was like represented in the workshop? And how do you personally react to it?

• 52:05Speaker 04

  Yeah, I mean, I think that's pretty close to my default expectation, maybe, which, and if so, then then it was represented there because I was there. Maybe to like riff on it a little bit, something we didn't put in the report, but that I've definitely found helpful for my own thinking is like thinking about, okay, you have an S curve. We're clearly in some kind of S curve, I agree. And you have like three, maybe less like situating us as humans in the middle of an ongoing S curve. But we have some kind of S curve of like AI capabilities. And there's three segments that are of interest. One is like, how long is sort of the lead up period, the first part of the S curve? One is how steep is that, you know, middle of the S? And one is how high is the ceiling? And I think a lot of, mostly when you're hearing people talk about automated AI and D, either they're sort of, they're in one of two camps on all three of those questions.

• 52:49Speaker 04

  So either they think the lead up is short and the curve is steep and the ceiling is high. Or they think the lead up is long and the curve is gradual and the ceiling is low. And so I also think it find it really interesting about, okay, what are different like combinations of those sort of parameters? Where to me, it feels like really looking like the lead up is pretty short these days. Like we're not too far from that sort of take off period. But like, for example, what if the curve is steep, but the ceiling is low? Or the curve is gradual, but the ceiling is high? I feel like we don't talk that much about either of those. And maybe also to your point, Nathan, about the like, Superhuman, but not like all powerful god, like singularity. There is no, you know, no point of return. Yeah, I also, I really think there's room for more thinking about what does it mean to be

• 53:34Speaker 04

  superhuman and what are the domains where there's like tons of headroom above humans? And you can easily identify like, you know, yes, it would look, this is what it would look like to be superhuman at, you know, like optimizing a kernel or like, you know, selling things, for example.

• 53:45Nathan Labenz?

  You're putting legalities.

• 53:46Speaker 04

  Yeah.

• 53:47Nathan Labenz?

  In previous, you know, parts of this marathon conversation, series of conversations, we've just seen how the ability to interpret the signals that people throwing, are throwing off in sleep to predict disease, you know, is just a really random, but I think instructive example of how there's an obvious, a lot of room to be superhuman at some of these tasks. And there's potentially a lot of power to unlock, especially if you can integrate that kind of, you know, infinite modality grocking with like a kind of basic reasoner. And I really don't see any reason that we're not going to be able to achieve that.

• 54:28Speaker 04

  Yeah, so often those, often those things though will involve, I think another piece that's like underexplored here is, uh, people tend to either be in the camp like, the ceiling is high and you're not going to need all that. It's not going to be delayed by real world adoption, or the ceiling is low and it's going to be delayed by real world adoption. And to me, I'm sort of like, isn't the obvious like combination of these, like, once you get the real world integration, you know, for example, you have to collect all that sleep data or like, you know, humans are really bad at interpreting scent data, but like dogs, like smell things, we can't like, but you have to do a bunch of sensors and like all that. So I also feel like there's sort of unexplored questions around how high is that ceiling as you have kind of really increasingly integrated AI to more and more aspects of life in the economy.

• 55:07Speaker 01

  I do, I do also wonder to what extent, um, because in my, you know, the way I, I might, I might view things as happening is software and mathematics first. And the question for me is that if you get software and mathematics first, you may get things like, I don't need a LiDAR for my self-driving car anymore. I can use cameras. And the cameras can be, you know, really bad cameras now because the math does all the work. And I don't need all this sophisticated technology. It could be that your phone could do what, you know, those sleep detection machines do uh with the right, you know, software package. The phones, your phone has a lot of sensors. An enormous amount of technology within the phone.

• 55:50Speaker 04

  Yeah.

• 55:50Speaker 01

  And and you do wonder whether it would really be application of uh algorithms to existing frameworks, existing infrastructure, increasing the bandwidth of your communications tech with new encryption and new cryptography, which is how DSL was invented, for example. DSL was really using the existing copper pipes with, you know, new new algos. So I I wondered what what extent the you don't get a slowdown just because of your physical infrastructure, because you kind of innovate innovate around with your physical infrastructure.

• 56:27Speaker 04

  Yeah, I'm sure that I'm sure that will work in some places. I think that, yeah, I think it'll work in some places and it won't work in others. So I think if we're talking like, you know, there's a huge ongoing struggle with where my mind goes is like cybersecurity for critical infrastructure, where like the systems are, the physical systems are old, they're hooked up, the operational technology, they're hooked up to old information technology because they have to be like, there's going to be a limited amount that you can optimize using smart new algorithms there because it's just

• 56:50Speaker 01

  Yeah.

• 56:50Speaker 04

  the stuff is old. Likewise, for, you know, my center does a lot of work with uh military technology, same thing there. Like if you have a ship that was built in the nineteen sixties, it's a ship that was built in the nineteen sixties.

• 57:01Speaker 01

  Um. Yeah.

• 57:02Speaker 04

  Or other pieces of equipment. So yes, I think in some places, yes, in some places, no. To me, that's another, I think, Prakash, you mentioned it as I as I came on the it's like talk I gave on jaggedness. To me, it's like another place where the jaggedness bites. And I think as well, like my default expectation in AI R and D is that we'll see jaggedness, like the jaggedness is fractal, right? Like you zoom into the sort of quote unquote task of AI R and D or the skill of AI R and D. And actually, it's many, many different things. And we'll so we'll see the AI R and D accelerating in areas that are especially amenable to using AI and lagging more in other areas. Not to say they can't ultimately be automated, but it will like take longer.

• 57:39Speaker 01

  How far do you think the product which is on the market right now behind what people are using inside the labs?

• 57:46Speaker 04

  I don't know. I honestly don't know. That was one of the like sort of most like trying to be productive section of the, well, it's not true. But at most like maybe actionable section of the report is a set of indicators. And we have like a table summarizing the three categories of indicators that we have. And that is the biggest category is kind of indicators from inside companies. And one of them is that public private gap. My sense is that it's not huge right now, but I would, I don't have any inside information. You guys talk to company employees as well.

• 58:13Nathan Labenz?

  If you believe Rune, he says it's, uh, we have no idea how good we have it and the gap is very small.

• 58:17Speaker 04

  Exactly. I'm thinking of things like that exactly.

• 58:20Speaker 01

  So what I learned in the last few days is that the real gap is that they're using models which are three times faster. So that it's just the same model. They're running on lower batch size. It's three times faster. And that's what they're using in turn. That's, you know, it's the same tokens. It's just a lot faster.

• 58:38Speaker 04

  And there's surely also like tooling stuff, right? As well, like, you know, something we put in the report, a couple of our reviewers who were looking at this, who were less familiar with the idea of automating AR and D. Some of them are like, oh, haven't you seen that study of like 95% of AI pilots fail? And there's the, you know, that meter study of, you know, that AI slows people down. And so we included in the report like explicitly noting, you know, yes, productivity boost from AR are mixed. But these AI researchers are like in the very, very best position to benefit from their technology. Like they are the best up to speed on what it can do and what it can't do. They're shaping how it's developed and like what directions it's pushed in. They're in the perfect like setting to be building tooling to squeeze the most juice out of these models. So I'm sure that that is also a piece of it as well.

• 59:19Nathan Labenz?

  One of the things you mentioned early on, just minutes ago, is the idea that you wanted to kind of bring awareness of these possibilities, you know, outside of the places where they are most often discussed. One of the thing I would love to hear your perspective on is how ideological do you think companies are about this? I mean, this is one of the things that like confuses me in the sense that every frontier lab leader has read their eleazar, uh, you know, catechism. They.

• 59:52Nathan Labenz?

  You know, I think they they previously, many of them in the past, had said like, things about how we should be extremely careful about this sort of thing. And we should not engage in an arms race dynamic. You know, it's obviously part of the OpenAI Charter. Dario has said things like this. And now we're in a world where there is a publicly stated timeline by OpenAI to the AI R and D intern. And then another, you know, not too much longer, at 2028, the full AI R and D researcher.

• 1:00:21Speaker 04

  And I mean, Dario Anthropic as well as Dario's.

• 1:00:24Nathan Labenz?

  Yeah, I would say Anthropic will even seem more committed to it, or more, you know, resigned to it, maybe, but they believe it.

• 1:00:31Speaker 01

  Jack Clark, June, summer this year. Jimmy Bah, who just left XAI, was a co-founder, 12 months. And OpenAI, this year, research intern, and then, you know, full researcher, kind of a year later.

• 1:00:48Speaker 04

  Yeah.

• 1:00:48Speaker 01

  I think it's this year. So, you know, that's my guess.

• 1:00:52Speaker 04

  This year for what specifically?

• 1:00:54Speaker 01

  Like the start of recursive self-improvement.

• 1:00:56Speaker 04

  Oh, but aren't we there already? Wasn't it last year? I mean, you had like, you know, you had like Gemini doing the, what was the evolutionary algorithm stuff, or designed, like co-designed an algorithm? That like sped up its own training by 1%. Like, come on, that's recursive. Truly, thanks. This is what I'm trying to make like the lead up to the loop. Yeah, so you, but I don't know, you think we might be this year where there's no human needed whatsoever? I think it's a high bar.

• 1:01:22Speaker 01

  I think we might be, I think I updated on Moltbook. The Moltbook thing took me by surprise. Like, one and a half million agents all of a sudden on the, on the web. Yeah, it's all nonsense, for sure. But things start off as nonsense. I think, I think what might happen is you get a single model update, which kind of fixes a little bit of hallucination, a little bit of security issues around like leaking, leaking secrets. And I think that might be enough.

• 1:01:47Speaker 04

  Sounds hard. Sounds real hard, fixing security stuff.

• 1:01:52Speaker 01

  We'll see. Yeah.

• 1:01:53Speaker 04

  Maybe.

• 1:01:55Nathan Labenz?

  So I do want to give you the chance to talk about the dynamics of this. The there's different reads that we might put on to people. They're ideological about it. Elon Musk has said things like, pretty much like, I don't know if this is going to be good or bad, but I want to be around to see it. And also like, I'd rather be part of it than a spectator. So that sounds like somebody who is inclined to gamble with humanity at, you know, in a pretty self aware way. Others may feel like they're trapped into these dynamics and they at least will do it, you know, as safely as possible. How would you describe that milieu right now? I think it's dramatically underappreciated by people outside the AI bubble that where we spend all our time.

• 1:02:40Speaker 04

  Yeah, I mean, I think my impression of it, my sense of it from the people I talk to is there is just a sense of inevitability about AI advancing and a desire to be a part of the future being created because they see this as the future that's being created. And I think there's also, you know, the people who you mentioned how this has been kind of part of the AI conversation since the very beginning is this IJ Good was in 19, early 1960s, talking about when you create the first ultra intelligent machine, which I feel like is we always need more terminology in AI. I feel like we should get ultra intelligent to make a comeback. That there's this sort of like very natural logic. If you have a computer science kind of brain, it's just a very natural logic to say, okay, we're we have some level of skill at building computers. When the computers have more skill than we do, then they'll build ones that have more skill in that.

• 1:03:25Speaker 04

  And then, you know, you get a loop. And so I think that that logic is just very appealing and and seems very natural. And so people think of it as something that's going to happen anyway. And then they may as well be involved. That's not everyone, but I do get the sense that that's sort of the like the water that most folks are swimming in. And then if you have a different view, then it's, you know, in contrast to that. I don't know. Do you is that your sense as well?

• 1:03:46Nathan Labenz?

  Yeah, I think so. And I think the inevitability is a pretty compelling argument. I mean, I resist it because and I at least want to make the point that like, even if some form of this is inevitable, there is still probably important discretion that we can exercise in terms of exactly what flavor. And there's questions like, should we keep chain of thought interpretable? Or should we embrace thinking in latent space? And I do think it's important to keep in mind that it's not probably all one or all the other. AI defies all binaries. There's going to be these gradations and these more local decision points.

• 1:04:27Nathan Labenz?

  But yeah, in 2022, I was just trying to make AI work for practical tasks. And with no background in AI research, basically ended up independently inventing a number of the techniques that have gone on to produce great things. I didn't take them past any local plateaus. But just having AIs improve their own outputs, kind of proto-constitutional AI type stuff. I do think the attractor is, the sort of gravity well is like, pretty strong. And it's hard to avoid some version of these techniques. Because if even a bozo like me lands on them, I don't know how they're not going to happen in the big, broad world.

• 1:05:09Nathan Labenz?

  Especially as we start to also get dramatic democratization of training techniques. Or I mean, prime intellect just put something out that allows anybody to spin up their own RL environment on a distributed basis, on a community basis. So like, everything's going to get tried. And I think that's pretty hard to argue against. But again, also, I do want people to still own what exactly it is that they are doing along the way.

• 1:05:33Speaker 04

  Yeah, I think there's something in here, which takes me back to long-running conversations about autonomous weapons. There's something in here about the level of human oversight that you can have. Where I do think that using AI to accelerate research is a totally agree and attractor. But I do, there's a You know, you'd really hope that there's a meaningful difference between I have a fleet of 10 million AI agents and they're running experiments for me and I'm I am leading them and I am, you know, guiding them versus I have set something into motion. I have no fucking clue what's going on. And I think there's there's a boundary somewhere. Is it a boundary that we're able to stay on one side of? I'm I'm not sure, but I hope it might be. That to me feels like the point to try to intervene, not, you know, we shouldn't use AI for research. It's obviously not going to work.

• 1:06:16Speaker 01

  To what extent do you think policymakers are naive? Because in like, earlier on, we spoke to Sam Hammond and he's he advises some policymakers on AI. And he was talking about privacy and the restrictions that the constraints that we could put on and all the regulations. And to me, I I think one thing that struck me was that I think a lot of policymakers are not aware, perhaps, that these like AI with access to existing technology, you know, with access to persistent search, persistent memory, it would basically do a Google stalking of you before before it even met you. It would know all of those things in the public, public domain, right? Like, the amount of access to information

• 1:07:00Speaker 01

  that it could have, the persistence of information, listening in on conversations, these things are going to be very powerful in that sense. And I think you you you can ban people, you can ban the AI from using using facial recognition, fine. But then you have network analysis, et cetera, et cetera. You can do, you can do metadata analysis on WhatsApp conversations, on where the messages are going. You don't need to know the content, right? So there's a lot of these techniques where you can de-anonymize traffic, you can de-anonymize people, you don't need the facial rec. You can ban the facial rec. You can still do gate analysis and speech analysis, voice analysis, handwriting analysis. There's so many other techniques. And all of these things will be available to AI. To what extent is this whole like,

• 1:07:43Speaker 01

  you know, we're going to make sure that we're going to have privacy thing is, are they are they being naive? Is it going to be possible?

• 1:07:51Speaker 04

  I mean, I think the US has done a worse job of this than pretty much every other country on the planet. So I think there's like some basic rule. I don't think you want to do rules at the level of like no facial recognition. I think you want to do rules at the level of like no data brokers, right? Of like, you can collect data, but if you're going to collect it, the user needs to know and they need to have, you know, notice and consent. Or I'm not deep on privacy law, so I don't want to pretend that I have the right, the great privacy proposal here. But I do think there's ways to do it that are better than the US. And I do think there's ways to do what they give you that sort of underlying flexibility. Yeah, yeah. Maybe I'll leave it at that because yeah, again, render privacy law goes real deep and I'm not there.

• 1:08:26Nathan Labenz?

  One more question for you is, you know, we, in the report, you talk about the possibility that the gap that we think is currently small between the models we have and the models that are used internally could open. And you have some recommendations around certain transparency measures. I want to give one quick shout out to the AI whistleblower initiative founded by my friend Carl Koch, who has engaged with OpenAI and at least played some role in their recent updates to their whistleblower policies, which I find like amazing that OpenAI is continuing to work in that direction even today.

• 1:09:02Nathan Labenz?

  Where do you think we are on the spectrum from secret non-disparagement clauses to, you know, where we need to be in terms of insight into what is going on at the labs? Other than, you know, private philanthropist funded whistleblower support, which you can avail yourself of, again, be the AI whistleblower initiative, should you need that? What other policies you think the government should be doing? And maybe even more broadly, if you want to zoom out, like, what do you, what do you think a situation, hypothetically, situationally aware US government should be doing in general that it is currently not?

• 1:09:38Speaker 04

  Yeah, I think there's a bunch of things here. I think. On transparency, I think we're doing better than we have been. So we have these two new state laws, SB fifty-three in California, RACE in New York. I think those are good starts. I think something that it would be great, but I think for a lot of this information, we're also just really dependent on what the companies still choose to put out. Now, we're fortunate. I want to give credit to both OpenAI and Anthropic, and to somewhat lesser extent, Google. Like, they do put out pretty proactively, you know, a pretty good amount of information. So I think they should get some credit for that. But I don't love that it's like almost entirely at their discretion what it is that they put out. I guess that will be shifting as SB fifty-three and RACE start to be enforced. So I'm interested to see what that that looks like.

• 1:10:19Speaker 04

  I think also we need to shift to a bit of a, I think there's been a start, the beginnings of a push to shift from a, like, model release-based schedule to something more continuous, which is partly driven by a kind of interest in these, like, internal deployment-type dynamics, not just the external releases. So the idea here is like, if the if the risk is not actually purely tied to when you put your model on the market, then all of your risk evaluation shouldn't be tied to that either. And also creating kind of better incentives for the companies around not forcing them to just, you know, rush things out the door, but instead trying to have more of a sort of continuous pulse of like updating metrics over time. So I think we could definitely be doing better on transparency. And then, you know, ideally pairing those requirements as well with some kind of independent audit requirement or independent way to let people come in, external,

• 1:11:02Speaker 04

  third parties come in and check that things are happening as they're supposed to be happening. That has been in several of these proposals and keeps getting stripped out by industry lobbying. So that, I think, is, you know, a new frontier as well. I think there's various other kind of policy implications that we put in the report, some that are maybe interesting. One is just like this general like hardening the world sort of recommendation or like, you know, resilience, societal resilience is another way of putting this. So this is like cyber defense, biodefense, biosurveillance, you know, investing in biosurveillance, just meaning like monitoring diseases, not like surveilling people, you know, investing in like epistemic security stuff of trying to have a way to, you know, determine what's real and what's fake. Tagging real content, you know, all this sort of like broader societal resilience stuff, which is like, okay, just assume that this is going to get much, much, much better.

• 1:11:46Speaker 04

  And then we might see AR and D, automated, you know, AR and D contributing to increased pace of change. I think also there's been some, this is less of like a policy and more of a mindset. I think there's been a shift over the past year or two to like, oh, actually, maybe open models are always going to be pretty close behind. And so, you know, concerns that you might have about there being, you know, an access gap or a, you know, concentration of power kind of gap if the closed models are far ahead, maybe we don't have to worry so much about that. And I think if you're taking seriously the possibility that automating AR and D speeds up the closed labs significantly, then we just need to revisit those assumptions about kind of open models and closed models. There's a few others, but yeah, would would point people to the report for kind of the full set.

• 1:12:26Nathan Labenz?

  When AI builds AI, things just might start to get weird. So yeah, definitely check out the full report from Helen and co-authors at CSET and beyond. It's interesting times for better or worse. Any of the closing thoughts before we break?

• 1:12:42Speaker 04

  No, great to be on. I read a chat with you as always, and yeah, look forward to next time.

• 1:12:48Speaker 01

  Indeed.

• 1:12:48Nathan Labenz?

  Cool.

• 1:12:49Speaker 01

  Hello.

• 1:12:49Nathan Labenz?

  Very nice to shorten our timeline between now and next time.

• 1:12:53Speaker 04

  See ya.

• 1:12:54Speaker 01

  Cheers.

• 1:12:55Nathan Labenz?

  Bye for now.

• 1:12:57Speaker 01

  So our next guest is Jeremie Harris. He's from Gladstone AI, and he wrote the first ever US government AI threat assessment for the State Department. It's been about 10 months now when they said every, every American EID, the center is compromised. Jeremy, what has changed? Have things gotten better or worse?

• 1:13:19Speaker 05

  Yeah, well, to piggyback off, I think Nathan just said things are getting weird. So things are weird. Yeah, great to be on. What has changed since then is less than one might have hoped, and for really interesting reasons. I think one of the things that a lot of people who are on the, sort of concerned about the AI risk story, and just kind of like the AI threat landscape or national security protective, whether it's loss of control, weaponization, I think a part of the story that's missing is understanding the infrastructure build out. Like, what are the actual bones that were milled in one year? Because that's the substrate that underlies everything. And there are all kinds of assumptions that are being made about it, where we're kind of abstracting away.

• 1:14:01Speaker 05

  What I really think is like, you know, at least 50% of the problem here. And we think a lot about model reconstruction attacks and all kinds of interesting debate about, whether it even makes sense to secure models in a world where you can just reconstruct them if the API is available. But more fundamentally, when you're building your entire AI industrial base off of, components that are made in China with personnel who often are Chinese nationals. I mean, this isn't even like, we're in the Manhattan Project, we're so, so, you're kind of behind that. The, I think it's incumbent on us to like, take a little step back and just ask about like, what is that chessboard even, what is the board itself, for the other pieces?

• 1:14:43Speaker 05

  But are we playing on something that's fundamentally stacked in a way that doesn't allow for a winnable outcome? And I'm not saying this to be pessimistic, I think there's actually, there are solutions that you come up with very quickly once you take that deep perspective, but kind of closing your eyes and not looking at it doesn't address the problem. And I think we're in a space where we're doing a lot of algorithmic level thinking, because that's what so much of the Western economy now is based on, people at keyboards who are used to. We're not making T-shirts anymore, we're not building transformers anymore, we're not doing that stuff. And so we tend to like to pretend that it doesn't exist. So that's kind of like, I guess, my more recent lens on the problem, it's the last two years. I know that's not quite an answer to your question, but that's kind of the chessboard as I see it.

• 1:15:28Speaker 01

  When you look at it end to end, you have, you know, the software piece and the talent piece. 50% of top AI researchers are Chinese nationals. And that includes, you know, people working in frontier labs in the US right now. And then you have, you know, the infra piece, a lot of stuff is coming from Taiwan, South Korea. Some of it is coming from China too. And then you have ASML sitting in Holland, which is supplying into TSMC. And then you have ASML suppliers, you have like 3,000 odd suppliers spread across the world. They're buying, I think, neon gas from Ukraine. When Ukraine got invaded, they had a problem. They've all of this, all of these missing pieces, all of these like pieces, all spread out across the place, right?

• 1:16:13Speaker 01

  And TSMC has been upfront by saying, you know, we, we are only possible in a safe globalized economy. If we ever got invaded, everything's over.

• 1:16:22Speaker 05

  Yeah.

• 1:16:22Speaker 01

  There's not, we like, we can't do anything, that's it, it's fine. So where do you think, how do you think that fits in with a threat perspective? It seems like someone just has a dead man switch over, you know, TSMC. So how does that, how does that work in terms of like security? And, you know, securing US prospects and, you know, the future in the US?

• 1:16:45Speaker 05

  Yeah, I think it's a great question. That this whole Taiwanese kind of scenario planning thing is something that everybody has talked about. I'm not so sure everybody's kind of like worked out the implications to full satisfaction. Yeah, I mean, so first of all, yes, if Taiwan gets invaded, TSMC is gone. It's gone whether it's because China takes it or because it's as I would expect and hope. Who be trapped to the nines to blow, right? I mean, it takes like, you know, hundreds or thousands of like insane level PhDs to tweak that. You think of it as like a giant box of 500 dials, each one of which has to be perfectly tuned to keep these things pumping out at the right yields. You're not going to replicate that if you're missing either the equipment or the people.

• 1:17:28Speaker 05

  So like this is extremely fragile, maybe the most fragile production process the primates on this planet perform. So an invasion is unlikely to leave it in China's hands. And so, yeah, the question is then, what do you get when you roll that back? What's the number two positioned entity? And then you start thinking about, okay, well, what does SMIC do? What can it do? And the SMIC Huawei complex does seem like a very plausible runner up, especially when you look at skilled production, especially when you look at the emphasis Huawei's placed on, you know, networking, large numbers of GPUs together. They don't have to be as efficient as ours, they can't be, they don't have the logic. But they can be networked together way better. And that's how they get effectively, you know, competitive scale performance. So this is a real issue.

• 1:18:13Speaker 05

  You also, in a funny way, this interacts. Somewhat positively with the energy bottleneck that we have here anyway, right? Like, we're gonna be bottlenecked by energy probably sometime around the end of the year. When that happens, uh, TSMC's ability to outproduce, I mean, it gets complicated because on a per chip basis, it's they're way more, they're way more energy efficient and more, they're pumping out more flops. But there, we do have that energy, uh, kind of ceiling on our, like, that's the main constraint that we're there all the towards. Um, so the timing matters a lot here, right? That dance between, uh, how much does logic matter? How much does energy matter? How much does memory matter? All these things, how much does packaging matter? All four of those things have become bottlenecks at different parts of the game in the last few years.

• 1:18:56Speaker 05

  Um, and another piece that I think, again, when we think about the actual bones that the AI commie runs on, it's not just chips and, uh, and not just like the data centers themselves. The power grid is a really just generally vulnerable target, right? So like, we, we know that, that, for example, there've been, uh, there've been components in Chinese transformers, uh, that have been snuck in explicitly Trojans to be able to, to take down our gear. A very plausible scenario, uh, just like based on talking to folks who are working this problem on the IC side is,

• 1:19:34Speaker 05

  uh, Taiwanese invasion begins and one of the first things that China considers doing is just shutting down the west, the, uh, grid. It's kind of obvious if it's existential, you know, that's massively escalatorics. There are huge, you know, huge question marks there, but it's, it's a scenario that's being taken very seriously for all the reasons you might imagine. So, yeah, I mean, I, I think when, if that happens, there are questions that suddenly run much deeper than just our ability to literally make chips where, you know, in Arizona or wherever, wherever the next thing is, but literally like if we can be kneecaps economically and more fundamental level, we don't even get to look at the chessboard that we hope to look at.

• 1:20:14Speaker 05

  We don't even get to indulge in the, oh, well, what can Samsung do versus what can SMIC do versus XN, uh, you know, CXNT, like all these things, we don't get to play that game. Like, we literally don't have an economy, like, like there's serious implications there, so. If we think about this as a game with the stakes that it might have, and this is contingent on what's between Xi Jinping's ears and the Politburo's ears, but this could end up looking like, you know, we're preparing ourselves to take a punch in the face, but then we get we get kicked in the balls, if you will. I mean, it's this is the highest scenario to use the technology that that we may be derailing towards. And again, that zoom out, I think, is really important. We've got target fixation here on what could be a pretty narrow part of the chessboard.

• 1:20:59Speaker 01

  What you had some ideas on, not only do we need to speed up, but we need to slow China down. And what were, you know, what, what was your concept around slowing China down? Because, you know, they have, they're trying their best. They are definitely not there on the chips yet. They ascend, the Huawei ascends, nine tens are, you know, but it doesn't really like them. You know, they, they want to get the H100s in. There is this concept of like building on the USAI stack. It's also revenue denial. If you, if you manage to flow the revenue into Nvidia versus flowing the revenue into Huawei, Huawei has more revenue to develop those chips. So therefore we should deny them. Like, how does this balance out this like letting

• 1:21:43Speaker 01

  them take on the chips, but not too powerful? And but, you know, still enough that it doesn't create a market for Huawei. It sounds like a very delicate balance here.

• 1:21:54Speaker 05

  It does sound like a very delicate balance. I personally, I'm sort of less oriented towards the argument that says, you know, if we just let Nvidia do business in China, then the Chinese will go, oh, sweet. We have Nvidia that's serving our needs. We don't have to push so hard on the gas on this issue that's been identified for years as the possibly number one national technological priority that we are pouring multiple Apollo moon landing that like amounts of cash into. Like, like, this is, to me, a kind of a miscalibrated sense of the even just the messaging that the CCP has been putting out. I just don't see a world like, we don't see, for example, Nvidia, okay, they can ship the H200 or whatever

• 1:22:39Speaker 05

  it is now. And then The CCP goes, oh, okay, forget about that quarter trillion dollar investment that we just made in PPP terms into into kind of our national AI chip capacity, you know, in infrastructure. Forget about that, you know, we'll sit with the Nvidia play. There's a sense both that the ability to access these Nvidia chips is transient because the next administration may just as well pull it down. But also that, like, why not both? I mean, it seems like an insane thing, given that AI is a matter of like national security importance for China. It'd be pretty surprising to me if they just decided to respond that way, and it seems like they haven't so far. So I guess that, you know, that's why I think about the export control thing from a slow dance standpoint.

• 1:23:25Speaker 05

  They have worked, we know from DeepSeek, like the public statements of their CEO before DeepSeek was on the radar. And this is actually, I think, really worth noting and kind of like under under recognized and appreciated. Like before DeepSeek was on the radar, they were coming out and saying, hey, we really think we could do this HGI thing. Though, like, one issue, there's just one problem, we can't get chips. And these export controls are killing us. Then obviously R1 drops and everything is got DeepSeek and, you know, they get dragged in front of the Politburo, whatever, and debriefed. And suddenly things change. You get these little trickles, these little leaks of similar information that come out, the edges of this kind of Chinese AI ecosystem every once in a while.

• 1:24:06Speaker 05

  But it's pretty clear that the export controls were working, that this is, if nothing else, just look at the massive orders that are going to be coming in for the H200 to show how much demand, pent up demand, there actually is in the AI ecosystem. And of course, we know all about the frustrations of the AI companies in China and about the current way they went after their chipset. So, yeah, I mean, I don't know, this is, that's kind of like my, my biased take is very much towards the direction of, I think, we got to listen to Chinese companies when they tell us that our export control policy is, is working.

• 1:24:41Nathan Labenz?

  Maybe I'll come back to some of the, the. Frustrating duality of like difficulties where on the one hand we have, you have expressed very low hope for the opportunity or the possibility of meaningful, true collaboration between the West and China. And then at the same time, I think you're also not super optimistic about our ability to create a superintelligence that we can actually control and get to do what we want it to do. And I think the way I think about our conversation from a year ago or so, and your contribution to the broader disco course with America's superintelligence project is like, those two things are both real, they're both true.

• 1:25:23Nathan Labenz?

  And like, you're kind of engaging in motivated reasoning if you try to deny either one of them. With that in mind,

• 1:25:32Nathan Labenz?

  we are now also seeing like some of the potentially foreshadowing kind of moments on the on the AI side itself, right? Like just in the last week, we've had these new models from Anthropic and OpenAI. And they've both kind of said, we weren't really able to run the evals like we kind of intended to. Anthropic basically said the eval awareness is pretty high. And so we'll just do a survey and a little internal survey of whether or not this is safe to release. That's probably a bit of a simplification on my part, but I think that is a fair enough summary of their of their position. Then OpenAI similarly was like, well, you know, these

• 1:26:13Nathan Labenz?

  the autonomy, you know, risk of the part of our preparedness framework. And again, it's also pretty hard to evaluate. We don't really have tasks that are kind of long enough horizon that we can get a real handle on just how autonomously capable a new model like 5.3 Codex is. So that's kind of crazy. And yet, of course, both models are put out there. I don't see China driving the need to do that. It seems like they're doing that and doing it on the same day, notably because they're like their own competition between the two of them. And also just sense of like rivalry seems to be heating up. They're like going at each other in Super Bowl ads to some degree at this point. Not something I thought I would see from Anthropic at the beginning was like a Super Bowl attack ad.

• 1:26:59Nathan Labenz?

  But here we are.

• 1:27:01Nathan Labenz?

  What do you make of like the dynamics between the Western companies? Like if I were to put on my slightly pessimist hat for a moment, I would say it seems like we might be racing to the bottom, which was exactly what we were hoping to avoid.

• 1:27:17Speaker 05

  Yeah, I think we are racing to the bottom. I think the only, the only frame that makes any sense, if we're going to talk about, okay, we need to regulate this technology, say domestically, in the same way that everybody from all leading out companies have been saying for, I want to say over a decade, pretty much. You're never going to do that unless you deal with the outer loop, the outermost loop, which is international competition. I think like there is no version of it. I don't think anyone, I think, again, we can enjoy the indulgence in target fixation of like, oh yeah, let's play the game pretending that other countries don't exist. But in the same way as ignoring the infrastructure target fixation, or sorry, the algorithmic

• 1:28:00Speaker 05

  training fixation and not lean infrastructure, this causes us to miss what is really the entire problem. So you're not going to get to a point where, you know, you can have a strategic or I should say tactical slowdown when you really need it. Where you're like, okay, suppose we find that the next version of whatever model can design custom bioweapons, execute catastrophic malware attacks, all these things that are like entirely, they're entirely plausible and that no, you know, counter jailbreak measures are truly 100% effective against the kind of people we'd be worried about. Yes, you would absolutely in that world need somebody to be able to say, okay, guys, tactical halt, this is insane. We can't be in a, you know, in a universe where you get a nuke and you get a nuke and you get a nuke. It's, we can't do a program free for nukes. Okay, so what are we going to do?

• 1:28:46Speaker 05

  We're going to have to have a slowdown. If China still exists and has their program and they are X many, I mean, repeating all this stuff that everybody said a million times, you know, if they're 12 months away, six months away, I don't care. We've got a shot clock now. That's the situation. So we have to start there. Like we have to start there and say, okay, any serious solution to this problem will involve dealing with China. Two ways you can do that. One is you have a kumbaya model with China. There are a lot of interesting reasons why I think this is just like not going to work. One of which is if you think about international treaties, they Don't tend to reflect some sort of Star Trek-y commitment to everybody on planet Earth want to do the right thing. They tend to reflect the real politique kind of like label

• 1:29:28Speaker 05

  in terms of actual power, like nukes. You have nuke drawdowns when everybody can retain arsenals that can still destroy the entire planet three times over. And there's literally no point in building the marginal nuke. You have similar things if you actually look at the history of bioweapon and chemical weapon treaties is like you find in every case like actually, they don't get you the marginal lift over just like killing people with artillery and gunshot. It looks nice and they often get adhered to for that reason. But then at the margins, you have like Chinese research labs on American soil, you know, come crazy research, you have whatever facilities and all this stuff happens anyway. And so this may sound like super cynical. I think it just reflects the way things work. That's at least my take. I would think that. So the question then is, how do you, how do

• 1:30:12Speaker 05

  you deal with an adversary like China that's in the position they are, that does have a stranglehold as they do on our infrastructure? They simply, they simply do. So the question is, what are your offensive options? That is it. There's like, you're not going to build the perfect Fort Knox. This is not, not a thing that's possible. And so the question is, what do you do to induce consequence on the other side? That's the only method will work if my theory of the world is correct. It's not a pretty theory. It's not one that leaves us feel warm and fuzzy inside. It's one that may make you think a little bit about, you know, mutually assured destruction, that sort of thing. And I think there are nuances here with, you know, obviously, Gernot Hendricks had his, his frame on it. But the bottom line is, yeah, I mean, I think you kind of need, and it doesn't need to be an

• 1:30:54Speaker 05

  AI-based response, though eventually, you know, you can certainly argue that any offensive option that isn't coupled to the scaling laws is eventually going to be beaten by something that does, right? So there's kind of an important design principle in these things. But there are offensive options that need to be explored. And this is unfortunate, but it does mean that if you have a situation where your adversary can turn to you at any time and say, hey, watch me turn the power off on your entire grid and have like tens of millions of Americans in the war dive starvation or exposure. But you need the ability to say, okay, you know, watch the same thing happen in Beijing and we can turn it back on. By the way, we need to have this escalation option.

• 1:31:36Speaker 05

  I know it's a bit of a grim view, but I think that like when I think about what actually gives leverage in this situation, it looks a lot less like what's had a pretty tricky, especially given the history of countries like China, like Russia, with respect to treaty adherence, like they signed treaties, like we know what it looks like when China signs a treaty, it doesn't end up being pretty in a situation like this where you just you need perfect adherence at such a high level of precision. Like there's no version of an international treaty on AI that doesn't involve inspections of compute stockpiles and like very precise overwatch of the kinds of algorithms that are being deployed, like the kinds of, you know, evaluation schemes, like the level of cooperation that's required

• 1:32:21Speaker 05

  to do something tractable here strikes me as being quite significant. And the trust just like, I don't see it being there.

• 1:32:29Nathan Labenz?

  So what's your P doom? And on what timeline? I mean, we were just talking with Helen about this report that they put out about when AI builds AI and the possibility of recursive self-improvement. Sure seems like all of the vague tweeting that is going on right now out of the frontier labs is suggesting that that is happening. And then on top of that, of course, you know, OpenAI has public timelines that they've put out, I guess to their credit, maybe, I guess you could, we could see that both ways. The Anthropic people that I talk to are, if anything, always the most firm believers that the recursive self-improvement

• 1:33:08Nathan Labenz?

  dynamic is unavoidable. How long do you think we have before before these things really start to take on a kind of runaway dynamic? Is there anything that you, you know, if you had power, you know, and a lot of power, is there anything that you feel like you would want to bet on? And where does that leave you in terms of P doom? Maybe I should just stop all this and spend more time with my family?

• 1:33:32Speaker 05

  Yeah, in general, I am a big fan of the happy warrior mindset. I think it's like just never constructive to be to go in hole. First of all, we have to assume that no matter how firmly we might believe in whatever outcome, we may just turn out to be wrong. There's a famous story, right? Richard Flyman walking around New York City in the 70s, I think it was, looking at all the skyscrapers and being like, wow, isn't it sad that all of this is going to be wiped out by a nuclear war between Russia and the United States sometime in the next few years. And he was just, that's just, that was a fact of the matter. And it reflected a pretty reasonable understanding of the dynamics unfolding between those countries at the time. I'm not saying it's ever quite that simple, but this is an ingredient, if nothing else,

• 1:34:14Speaker 05

  that makes you less effective if you're just stuck in a hole all the time. And just as a meta point, I guess, that I think is the first piece. You know, we have to act with agency, and we're going to be most effective doing that if we're not stuck in a deterministic, Calvinist frame with this whole thing. In terms of, I'll also not answer your question before I answer it, just by saying regardless of timelines, one thing to focus on is that there's some things that are pure optionality plays. And so like, we, there are things that you do, if you're wanting to build a frontier AI cluster, right, at scale, that rule out nation-state security at that cluster. Just rule that out. Like, if you don't do these

• 1:34:55Speaker 05

  things right on day one, by day, you know, 360, once you've finished building the site, your site is going to be compromisable, and there's no going back into doing that. We think of these as, we call them like the one-way doors, right, of the data center construction process. Figuring out what those one-way doors are, setting standards for them, and actually executing on that, even doing it just like voluntarily, right? Like, we think about opening eye and anthropic and so on, the kind of all independently going, hey, we just want to buy that optionality because at some point...

• 1:35:28Speaker 01

  Can you give me a concrete example of a one-way door?

• 1:35:32Speaker 05

  Yeah, so there's a bunch that I can't go into, but one that I can is pretty easy is think about like the people that you're getting in the loop to review the site plans and details that would be, let's say, useful to an adversary who is trying to extract information. If those people are Chinese nationals, okay, you're done. Cool, right? Like, you're never gonna unfuck that. That's baked in, right? So these are actually the interesting thing with these one-way doors is they tend to be surprisingly cheap. And that's the the tragedy of it all is like, you actually could, if you were awful, go through and be like, well, on a fraction of the budget that would be required in CapEx and OpEx for these bills. You just like create pure optionality by implementing things. So I think that's a really important element.

• 1:36:17Speaker 05

  Putting offensive options on the table is a pure optionality play. You don't need to exercise those options. You need to have them on the table. That's what I'm saying. I'm not saying let's go to war with China. That's a crazy thing to say. There's all things in their context, but you need options. And that's a crucial thing. So having an understanding of, you know, the mapping out the ecosystems that are relevant, the AI ecosystems that are relevant. And and thinking about like, what what might that at end game play out to be? Those seem like pure optionality plays regardless of timelines, all cheat, all things you can do quickly. Again, like this is, it seems to me kind of, and I'm not saying they're not being done. It's just that often there's a

• 1:36:58Speaker 05

  a lack of sort of focus on the on the end game here. Anyway, without getting into the weeds too much. Okay, so P Doom and timelines. Oh, sorry, Gosh, yeah.

• 1:37:09Speaker 01

  No, go ahead, go ahead.

• 1:37:12Speaker 05

  P Doom and timelines. So

• 1:37:16Speaker 05

  I'll almost say P Doom. I don't find it useful. Like, I know, I know what I'm focused on. I know what I gotta do. If I start thinking about like, my generic answer has been for years, like any number between 10 and 90%, I'll take as like, that's a reasonable number. I'm not like, I've I've read the debates. I've seen the posts on that's wrong. Yeah.

• 1:37:37Speaker 01

  So is that is that your P Doom or P loss of control to superintelligence? Because I think I think in some places you've mentioned it's a loss of control to superintelligence rather than Doom.

• 1:37:47Speaker 05

  Yeah. You've obviously done your homework really well. Yes, that is more of a loss of control, the superintelligence. I think by virtue of the way that numbers multiply together, I don't know that my answer is that different for like P doom in general. Again, this is coming from somebody who, for better or for worse, has almost explicitly not put in that much time to kind of wallow in those numbers, as I think we're sort of all tempted to do, right? Like, I have that temptation. I get it. Like, I mentioned I had a daughter, right? Like, I don't like the landscape that's playing out, but I had a daughter. Like, I chose to have a daughter, and I didn't have her in like 2018 before the scaling laws blew up. Like,

• 1:38:28Speaker 05

  this is, you know, this is a choice that I made. I think there's a kind of almost like a spiritual risk to getting locked into that kind of thinking. And I say this as somebody who's experienced that. You know, I went through that and fed on how it ground my ears. So, yeah, so I guess I'll just like, you know, not answer the question by saying 10 to 90% sounds reasonable. I think if you're below 10%, I just really think that there's like, there's homework you gotta do, because a lot of these scenarios are maybe sound crazy, but they're just, they're a lot less crazy than they seem. And when you get into the nitty gritty, it's like, a lot of these scenarios are already, they're halfway unfolding. You know, if you're above 90%, I

• 1:39:09Speaker 05

  mean, first of all, if you live as if you're above 90% and you're like, that's gonna just make you less effective. I also think, again, Richard Feynman certainly seemed to think he was in that ballpark. There's just like an epistemic question here of how quickly does the world adapt? I think we're constantly surprised by how quickly the world adapts, both how fragile and how resilient it is. And like, the 11th chapter of the book will often involve a new character that comes out of nowhere, and we just need to kind of make sure that we keep uncertainty about our uncertainty factored into this analysis. And that, I think, that buys me 10% pretty easily. I've been rolling on stuff that I thought I was 100% on, often enough to be like, okay, I'm not, I'm not gonna push it that much.

• 1:39:54Speaker 05

  And I know that's frustrating for like a lot of people like, no, no, but like, look at the math, man. And I get the math, but what I'm questioning here is just the the process that led to the math. And I don't know that I can plausibly ever get fully behind that process and interrogate with confidence. So last thing is timelines. I thought AI 2027 and contrary to, I think Dan has kind of pulled back a little bit his timelines from that.

• 1:40:17Speaker 01

  Twenty, he said, he said, twenty thirty seven always meant twenty twenty eight, but now it means twenty twenty nine. Yeah.

• 1:40:23Speaker 05

  Yeah, and, and, and, yeah, AI is the apocalypse of the future and it always will be. But, you know, not actually, but, you know, I think there's like, um, I was sensing which, so when GV3 first came up, I was like, oh, man, I've got two year timelines. And, uh, and that was because I didn't understand what the hell be involved in the infrastructure build out. And now like I have a much better understanding of that. I'm still kind of like, what's the next bottleneck going to be? I'm very uncertain about this. Uh, and again, it's one of those things that doesn't really affect what I do, just because I'm so focused on just all the low hanging fruit that we have to pick right now. There's so much stuff that we're just not doing because we're paralyzed by the problem. So I think in terms of what we do, there's pure alpha on the table in the short term. Twenty twenty seven doesn't sound insane to me.

• 1:41:08Speaker 05

  Twenty thirty doesn't sound insane to me. Twenty thirty five sounds a bit far. I guess I'll, I'll sort of leave it at that as a spread. I think we should be acting as if twenty twenty seven is plausible. I, I think you would be unfortunate if we, if it happened in twenty twenty seven and we're like, man, we had a lot of really plausible analyses that point at that. And we just didn't do anything. That would be shit.

• 1:41:32Nathan Labenz?

  Can you give us a little bit more of a hit list in terms of the low hanging fruit that you want to see us pick? I mean, it's, we've got the one, which is like, build at least some subset of our data center, build out in a secure way. So that we can run hypersensitive projects there as needed. What else is kind of on the, uh, if you, if you're the, uh, replace David Sacks as the next AIs are, uh, what's going to be your priority sheet?

• 1:42:02Speaker 05

  Yeah, I mean, so that first one, by the way, is a lot of things, right? So it's it bundles together. I mentioned the personnel security issue insider threat problems. There are a huge number of things in that bucket alone that are necessary and contribute very cheaply to a much more optionality on the security side. I think, again, you can zoom out more. So you look at the grid. What could you be doing to introduce redundancies quickly? Like the supply chains that leaked to a lot of these components, very clearly sourcing heavily from China. You have so here's like an easy win. Look into companies that are offering to build data centers like suspiciously fast and who owns those companies?

• 1:42:45Speaker 05

  So there was actually a letter that came out from the House Law Committee on CCP a while ago, naming day one data centers, right? As an entity that is somewhat suspect. And, you know, you'll have these data centers where data center building companies was like, oh, wow, like you could build stuff like way faster than anybody else. It involves sourcing components from China. And like, like my personal opinion is, if I were to see that, I might be asking myself the question, China is kind of a command economy through civil military fusion. If the CCP wants me to have this very, very rare and precious and backlog component for my data center in the continental United States, that might tell me something about

• 1:43:30Speaker 05

  how much faith that should have in the security and integrity of that component. You know, there's, there's just not a lot of infrastructure level, a kind of attention being paid to these things. And the labs, by the way, like they want to do the right thing here. They don't want to be in a position where they're getting a company to to build something for them. And then it turns out that that thing is compromised and it comes out that that is not good for anybody. So incentives are aligned there. There's just, yeah, it's things like that where there's, there's been so little attention paid to the bones that that there's just tons of stuff we can improve, including with AI, right?

• 1:44:10Speaker 05

  Like, like looking for, looking for malware and old software that's load bearing for our infrastructure or not malware, but rather vulnerabilities and finding ways to harden it. So, yeah, this is a defocused answer that hopefully gives a sense of the venue.

• 1:44:27Nathan Labenz?

  One thing we haven't really given you a chance to flex your ability on in this conversation is just the breadth and depth of your technical understanding of so many AI developments. And I definitely recommend the Last Week in AI podcast, which you usually host, as a great source of, I think, very sophisticated analysis by both of you, but I tune in for you mostly, to be honest. And I wonder, I wonder how you are doing it? How are you keeping up? What is your, how have your methods evolved so that you're maintaining situational awareness as much as you can?

• 1:45:10Speaker 05

  Well, thank you, first of all. And it's very kind of you to say. I have had to do this before, but I do actually watch the cognitive revolution. And it's, I think it's, you know, there's a lot of, the ecosystem here is really rich, and interviews are really important because you get stuff that you can't get from the papers. And, you know, I tend to focus a lot on the papers, so I just don't get that kind of, that kind of analysis. And I just talk to friends from the labs, but it's different from those deep dives. Yeah, I mean, so back in, I can't remember when I started on last week in AI, but it was like in, you know, maybe 2021 or something. And back then, I would, I would just read papers. And, you know, you couldn't use GPT-3 to like help you understand a paper. It just wasn't a thing. Now it's changed.

• 1:45:53Speaker 05

  I have had an experience that was kind of frustrating this week in particular because I'm preparing a, like a state of play briefing for a customer. And basically, they want to know, yeah, what happened in the last quarter in the world of AI that we should be tracking? And there was a paper that I had Gemini help me with. And I got to a really good understanding of how it is like a residual, like the dynamics of gradient flow through this residual and have a whatever. And it was pretty complex. What I realized, though, after having an interaction with Gemini for long enough, I switched over to Claude, and I was like, wait a minute, I just like. Hallucinated my way through that entire conversation, got to an understanding where I was like, oh yeah,

• 1:46:37Speaker 05

  like, I'm pretty smart for figuring this out. And also, like, I got this now, Pat, and everything got flipped around. So I'm not saying that always happens, but that has been kind of the most recent update in my process is really, you know, be mindful to kind of double check, especially as you start to get lost in a rabbit hole. I typically spend like, I would say about thirty to forty percent of my time reading the paper and then the rest interacting with a model about usually it's like the implications of the paper, or what it is, is it's reinforcement learning versus supervised by tuning. Like if I'm reading the paper and you're doing SFT, like that's what's going on. With the models, I get to actually like go on policy and I get to test my own understanding. Like, I would have done this experiment differently. Is that a stupid idea?

• 1:47:22Speaker 05

  And often I'll get a pretty good answer and that it makes you feel like you're rotating the shape instead of just staring at it. And that for me has been just really helpful and empowering. It feels empowering.

• 1:47:34Nathan Labenz?

  Do you have any particular like workflows, pipelines, whatever, that try to filter things for you and surface what you really need to spend time on? Because that is as challenging, I mean, it's more challenging than ever, and it seems to be maybe as big of a deal as being able to successfully make sense of any one thing is like, what are you going to choose to spend your time on in the first place? How has that evolved for you?

• 1:48:04Speaker 05

  Yeah, it's a great question. And this is that like age old question of taste, right? Like, and one of the things that I've had to come to accept is I can't develop good taste in all the domains that we want to cover on the podcast. Like, I'm never gonna, you know, like, my taste is basically if if one of the frontier labs puts out a piece of research, or if a researcher I know and like have a lot of respect or appreciation for puts something out or is a co-author on something, I'm gonna take a really hard look at that. And then besides that, you know, and I have the usual Twitter account, the set of Twitter accounts that I follow and, you know, that's another way. But my passes at these papers are

• 1:48:45Speaker 05

  I'm pretty focused on the like, what's on the critical path to ASI question? Not that I know the answer, but just like, I'm trying to find things that to me gesture at that. Which is why, you know, I don't tend to talk about like GANs or the latest in, well, I was gonna say the latest in text to video. Now that seems like it could be on the path, so you never know. But I guess part of it is just acceptance. I am reading these papers for the concepts more than the outcomes. And often what will happen is there's this paper that'll come out and it might not be the perfect paper to cover from a given topic area. Oh, you know, residual connections and really optimizing the crap out of them to get ultra-deep transformers. There's this paper about it. Is this the best paper?

• 1:49:26Speaker 05

  Probably not, but the reason I focus so much on explaining the underlying kind of concepts on the podcast is that A, there's gonna be another paper next week that obviates whatever the hell of last paper did. And B, I think that the kind of core concept is the most of what the landscape is the most important thing. So when there's another paper that comes out about optimizing residual connections, you're like, okay, I'm familiar with this playpen. Like, I know, I know the furniture in this room. I can rearrange it a little bit, be more confident. So I guess the answer is just like, I get around the taste issue by not having it, uh, which is maybe just.

• 1:50:02Nathan Labenz?

  What's underappreciated for you right now by AI obsessed people? I mean, there's of course, like in the broader world, AI is underappreciated. And just how crazy things might soon get is, I think, very probably underappreciated. What do you think I might be missing? What are the most likely blind spots for somebody like me that you would want to draw to attention?

• 1:50:24Speaker 05

  I guess the challenge with blind spots is that like, we all have them. And by definition, we all know that we have them. So what I'll try to do is roll back and tell you about my blind spots as of about two years ago. And that was around the time that we put together that report that Prakash mentioned earlier. Um, so.

• 1:50:40Speaker 05

  Sound like a broken record, but the infrastructure layer, the kind of the stuff that

• 1:50:46Speaker 05

  this might sound with a wrong way to put it, but the stuff that feels too blue collar to to most people who are AI obsessed like I am, you sort of start to realize how much of the world is actually built on infrastructure that we just abstract away. So I think that's actually really important and needs to be put on like, understanding what is the down to like, what is the dynamics of the leasing process that Frontier Lab goes through to get a new piece of land? Like, what can screw up there? What causes delays in construction projects that we talk so much about, you know, this lab has a XAI has their new Colossus cluster and it's going to be online shockingly like to one gigawatt sooner than Anthrobix, which surprised everybody and all

• 1:51:27Speaker 05

  this stuff. Why that when that happened? Like, what was the actual driving like, because that's going to tell you if you believe in the scaling laws, that actually is a probably one of the most important variables that you want to track is like delays in construction processes. Sounds pretty mundane, but like, hey, the world runs on it and procurement schedules and things like that. So I guess that's one piece that I've been missing. Another one is like how, yeah, how kind of real nation state and security happens. And as hard to get information about that, you know, one of the biggest challenges there is that there is no such thing as one nation state security capability. Nation states are siloed, obviously, because security, you can't have tactics, techniques and procedures that are exchanged between silos because then there's

• 1:52:10Speaker 05

  no information security. And this by definition means that you would have to go through a process of taking team A, comparing it to team B. Well, okay, team A wins. Okay. So now next thing like, you'd have to go through that kind of selection process, run a Deloscore type situation to even know what the most exquisite capabilities are that like we could field and it still wouldn't tell you quite what other other countries could do. So that's kind of like, anyway, I think a really important dynamic that's very easy, any AI security con attacks, especially for physical security, especially for physical security, which is again, undervalued precisely because we tend to abstract away and focus a lot on cyber because it couples, it couples to AI and it feels like it's in our sweet nerdy space and I get that and I love it and it's critical, but it's also not, if you look at like what the Russians do.

• 1:52:55Speaker 05

  They like, they do cyber for sure, but they will go up and like, they will arson your, your transformer. Like, that's not an issue then. Hmm. Examples of that sort of thing happening. So anyway, there, there's that piece, maybe the last one and more on the, in the kind of comfortable and familiar uh, learning space that, that I occupy is uh, is just this idea of the distinction between having a model and then having the compute to run that model. Hmm. You know, like, if you believe in the inference time scaling laws, then, you know, model theft is one thing, but actually being able to point that model at, you know, basically have compute on compete war at inference time. Uh, seems like a really important dimension and you see this play out in a lot of interesting ways. One of which is the Chinese ecosystem, you know, they have a huge number of users.

• 1:53:41Speaker 05

  Um, and then they have some like okay-ish language models. Problem is that their sort of labs are all flooded with like these inference requests from their giant user population, which leaves very little R and D compute for just like innovation improving of models. That's actually one of the frustrations for Chinese labs, much more than labs here. They're just like, dude, we have so much to do, but we're not bottlenecked by money, we're bottlenecked by, by compute. And so the, the dynamics of, of like how inference affects training. And then how, like what it means to steal a model and what it means for model on model warfare to happen. Especially in cyber, if you're like, cyber hardening has a certain amount of test on compute.

• 1:54:24Speaker 05

  That test on compute, you know, it's going to be focused on in, in some way. And then the offense side is going to have a certain amount of test on compute and how, how those play out. Like the relative budgets matters a lot there. Obviously, if you're defending, you have a wider surface area you gotta defend, but there, yeah, anyway, there's a whole debate there. That, I guess, would be another dimension. It's just like going beyond just owning a model, like we're about running it. What can you do with this model that you have?

• 1:54:48Speaker 01

  I have one last question, which is you're pretty security conscious. Have you run OpenClaw and what is your current personal productivity stack?

• 1:54:58Speaker 05

  Yeah, yeah, I have not run OpenClaw. I have. So actually funny you say that. I'm setting up a like I have an old laptop that I'm going to use as my my primary laptop through the purpose of exactly that. Partly yeah, partly because you know, I anyway, yeah, for the exact reasons you would imagine. Um, in terms of my, yeah, so a big part of my my job is is becoming now. I'm constructing agentic workflows to um to do some some like some things that are not super security sensitive, but more just like I'm going to try to use it to optimize my comms. Um, because that's a huge bottleneck for me. And um, and for that, actually, I'm still in the discovery phase of trying to choose platforms.

• 1:55:42Speaker 05

  I'd be interested in your thoughts for that, actually, as I dive in. That's this is literally like next week is is my deep dive. This is almost the worst possible timing because I think my answer is going to be like horribly outdated. Yeah.

• 1:55:55Speaker 05

  It's a great question. I wish I had the answer.

• 1:55:58Nathan Labenz?

  I talked about a little bit about mine at the top. I'm interested to hear more about what Prakash is doing too. But for me right now, it's Claude Code as the, you know, kind of base product. And then taking inspiration from a guy named Daniel Miesler, who I did an episode of the podcast with, who's created personal AI infrastructure, a framework that's an open source framework. And also friends who I just trade notes with privately. I'm trying to create deep context for myself by first exporting kind of all of my digital history from all, you know, Gmail, Slack, all the other places where I kind of have these comms, get them into a local database.

• 1:56:39Nathan Labenz?

  Uh, then of course you need like a daily update process to like fetch the latest because you're still communicating on all these other platforms. Then layering on top of that. Summarization and different kind of angles on the data. So one, right now I'm at the phase where I'm like, here's a month worth of all comms. That seems to, for me, come out to about 300,000 tokens. Now summarize that down to like 10,000 tokens of what a chief of staff would need to understand this month in Nathan's life. So he kept that like 30 to one reduction, then probably put like a year, you know, long version of that. And then sort of the, let's talk about the relationships and kind of have that sort of cut on it, the projects cut.

• 1:57:21Nathan Labenz?

  And then hopefully with that deep context and the, I'm also trying to have it leave pointers in those summaries with like regular habit of quoting any distinctive language. So it can go search down for ground truth for the original. Hopefully it will then have enough context to be able to, you know, not exactly write exactly as I would, but sort of come much closer at least to responding as I would, having the sort of the context necessary to exercise something like the judgment or taste that I would exercise in doing things. And that was actually part of the process of setting up this episode. I gave Matt System 20 names and said, do, you know, research on these people, find

• 1:58:06Nathan Labenz?

  out what they've been up to lately, give me a brief on that. And then also had it draft the outreach emails, uh, which were only lightly personalized. Uh, and I still did kind of go in and a little bit before tweaking.

• 1:58:18Speaker 05

  But I appreciate that. That's nice.

• 1:58:21Nathan Labenz?

  But yeah, I, I don't like to publish or even send as like one-to-one communication AI output directly.

• 1:58:29Nathan Labenz?

  But I do find that I can get to something that I do feel comfortable signing my name to faster with an AI draft in many cases these days. And so, um, it's very much a work in progress for me, but that's kind of where I'm at, at the moment. And again, it'll, I'm sure by the time we, um, talk next, it'll have changed quite a bit. Prakash, what do you, what, what's your, um?

• 1:58:54Speaker 01

  Well, I've got, I've got a couple of things that I ended up building out. Uh, one was a market, stock market tracker. I have a number of metrics which I think no one else watches. And it's, it's fairly hard to obtain. Uh, and the great thing is, um, Claude and Claude is very good at financial management. Very, very good. Uh, far, far better than I, I ever have ever been. Uh, and so it's, uh, relatively easy to talk to Claude and kind of figure out what kind of thesis you have and then like build out metrics precisely for that thesis to, to watch like pick up lines. So, um, that's been very useful. Uh, I, I used to, I used to like, you, I used to like do it in my head, right?

• 1:59:39Speaker 01

  Like you look at something, you look at something else and then like you calculate the ratios, blah, blah, blah. And then like, I realized that I was spending a lot of time doing ratios in my head. And I was like, maybe I should automate this. And so now it's all automated. It's nice. I don't do the ratios in my head anymore. I just look at it and I can see the screens automatically. I can see what I'm looking for. And then the other thing was, uh, podcast clipping, because we do a lot of podcasts. And content these days has to be packaged into short clips in order to hit socials. And so that I tried like six months ago. The tech wasn't there. Uh, and I tried, uh, four, three, four weeks ago, the tech was there. Every, every, everything works, transcription works, review works, selection works. Everything works.

• 2:00:23Speaker 01

  Uh, and this is, yeah. And this is in my experience, like it's kind of like, it's kind of like, maybe, maybe it gets it like one percent better, but that one percent better clears the hurdle. And that's like that binary, that's a binary step up. Like it works or it doesn't work. And that one percent just clears the hurdle. And I, I really feel like in the last month, like a lot of things started clearing the hurdle.

• 2:00:45Speaker 05

  So. I was just gonna say, you know, when you said that in the last six months, so many things have gone from toy to just serviceable in production. And it seems to map on to Nathan, what you were saying earlier about the, uh, takeoff dynamics and the labs automating their own research. That, that all kind of maps very nicely. One of the things on the financial side too. I find, um, so I find Claude is, is also useful on. You know, questions like, so you might have a thesis, but then there's a question about how do I, if I'm right about this, what's the best bet to make? Because that's a category of problem I've mentioned in the past, right? Where it's like, you know, you'll have a thesis, but you know, you're not going to bet on Microsoft because OpenAI is such a tiny fraction and it's already all this stuff. How do you leverage and torque it to this thesis?

• 2:01:30Speaker 05

  And that's kind of something that, you know, the world is so complex that you just need something to peruse? And out of all that, you know, the finance use case is a really great one, great point.

• 2:01:40Speaker 01

  Yeah, I also, it's also been very, very weird in the market because I feel like Twitter is literally like a month or two ahead of the market. It's just been amazing. People tell you like TSMC will do well and then like, like three months later it happens. Like, what's going on? Like, now I always, you know, I was, I was a professional financier. I've always expected that hedge funds get there before, before you do. And in talking to my friends at prime brokerages and hedge funds, they are very negative on AI. They just don't believe it's happening. They don't believe, you know, they believe it's like crypto. They believe a lot of West Coast tech is just scamming retail investors.

• 2:02:24Speaker 01

  Index investing is the only thing that really like works and everything else is either insider trading or scams. And that's pretty much what the prime brokerage guys and the hedge fund guys like believe.

• 2:02:37Speaker 05

  Like, you know, Medallion, right? Or Gene Street, right? These guys who have AI in their bones and these, I guess, Medallion, it's like, they can't, you know, they only invest, you know, famously five billion a year because otherwise they would actually move the markets and it feedback. But yeah, what about them? Is there?

• 2:02:54Speaker 01

  They were down last year, so the impact is starting to be felt. I think, well, you know, also Jim Simons died. I don't know to what extent he was still supervising because he'd already kind of semi-retired for like 10 years almost. But Medallion was down. There's some sense that it's also because they're losing talent to the lab too, right? Like you can't, you can't, you can't forget about that. They're starting to lose talent to the lab. And some of the labs do have like internal teams which will eventually look at trading on the market, I think. So we will see where that goes.

• 2:03:26Nathan Labenz?

  Jeremy, thanks for joining us. Let's check back in on your personal productivity stack once you've upgraded it. And in general, let's, I'm reusing this joke everywhere I go, let's shorten the timeline for our next conversation. Okay. Thanks, guys. Appreciate it.

• 2:03:46Speaker 01

  Thanks, Jeremy. Cheers.

• 2:03:47Nathan Labenz?

  Cheers. So what do we make of it all? I mean, it's so, the big thing I can't get past in all this stuff is the amount of disagreement between, and this has been commented on so many times, so many ways, right? Up to the level of the Turing Award winners that can't see the same phenomenon. Yeah. But it seems to happen at kind of every layer of, it's like a fractal problem. You go into these specific workshops around AI R and D, you get people from the labs. I do understand that there are even people at the frontier companies that, you know, have heterodox positions and don't really buy into the hype. And then even, you know, with

• 2:04:31Nathan Labenz?

  the AI for science, it's like, I can't, I can't make any case that I should trust my own intuition more than Abi's because he's, you know, how many times did it happen in talking to him where he was like, I've actually written about that. So he's clearly thought about this much longer and harder than I have. But it does still feel like it's a very hard thing to reconcile where you do see these examples and it seems like some of them are really starting to work. But then the skepticism remains and is like, is very hard to move people off of. And I don't want to paint him as overly skeptical either because he did say toward the end, I think his skepticism is more like backward looking, you know, forward looking. He was kind of like, I do believe the, you know, the trends will continue and that they will have impact.

• 2:05:16Nathan Labenz?

  Yeah. But where does that, how do you, how do you try to make sense of?

• 2:05:22Speaker 01

  When you say it's fractal, I feel it's also fractal internally to me, where I have, you know, some assumptions here. And then sometimes I feel cognitive dissonance from something else that I might believe. And then you kind of test those assumptions and you see where things are going. I have had like moments of like moments of truth or or kind of perception where I start to realize that I think things might move faster than I expected. My my my original timelines were end of two thousand twenty five for junior software developers to be replaced in capability, not in organizations, but the capability is available at the end of two thousand twenty five.

• 2:06:05Speaker 01

  And it takes about three years to to percolate. So end of two thousand twenty eight, no more junior software developers, basically. Or at least the task that junior software developers are doing today. And then so I had end of two thousand twenty five, end of two thousand twenty six, end of two thousand twenty seven. End of two thousand twenty seven, even senior researchers at AI Labs, you know, fully capabilities are done. The models have a capability, but deployment, again, takes two to three years. It takes time. Uh, that was my sense. My update in the last month has been probably that things are going to go faster than we expected. And that we will see discontinuities. And those discontinuities are like this kind of like, things get one percent better, but all of a sudden they clear the hurdle.

• 2:06:44Speaker 01

  Right, like, and and we don't have a good sense of these things because we keep seeing like linear improvements. And they're kind of linear, maybe maybe super linear. But we don't have this sense of like clearing the hurdle. But when it clears a hurdle, it's obvious. Right, it's obvious. It's it started to be obvious for software, I think, in the last month or so. So I think I think that we just have like misperceptions on where things are going because we we can kind of see the trajectory of capability. Or we don't understand how humans absorb that capability. Like, what is that process and what hurdles do we need to clear? What what is the open claw? I thought you need you need like full security and like privacy and all of this stuff.

• 2:07:24Speaker 01

  It seems you didn't. It seems like people are willing to put out their credit card numbers and crypto tokens on the open web. And you don't need privacy. You the notebook guys are, there's a mult notebook saying like, oh, you know, my user is so annoying. Here's his credit card number. Uh, and Scott Alexander ended up calling up the guy and asking him like, hey, you know, would this actually happen? And yes, that was the credit card number. It was leaked. So um I think it's clearing the hurdle concept and where humans accept the technology and kind of where the market pulls that technology, that we don't know, that even I don't have a good perception of. But it seems like we're starting to clear those hurdles where humans

• 2:08:04Speaker 01

  are starting to pull the technology from the market in. And that's when you start to see kind of revenue growth, that's when you start to see, you know, the the demand growth really happen where the market starts to pull the product out of the ether. And I think that's happening now. I think the open cloth, I think I think we'll have a much better version of open claw, closed sourced, secure version running on corporate, running inside corporate data centers, you know, by the end of the year. I I I saw, I watched the all in podcast, Jason Kalakunis, not, not, not the most technical person in the world. He had a team for the all in about 15 people. He started to get everyone to like, create a skill for themselves, like every task that they do, they create a skill. He has open claw machines,

• 2:08:47Speaker 01

  like one machine per person. And then he has a consolidation agent that consolidates everything into something he calls Ultron. And then he can talk to Ultron. So he can ask Ultron, and that's his entire company. Like, it's a summary of the entire company, and he's talking to it. I thought that would be two years from now. I I knew it would eventually happen, but I didn't think it would happen now. Um, so yeah, I think things are, things are actually moving faster than people think because of the market acceptance. The market is pulling it out. I don't think the researchers have a good sense because researchers don't understand the market that well. They don't understand the demand, the demand dynamics that happen with consumers, and how how products get pulled out. Once a demand is there, like, products will just get pulled out of the ether, because people

• 2:09:32Speaker 01

  start focusing, they know that money can be made there, you just start focusing on it. So that's my sense. So I, I, you know, I, it's not a, it's, it's not a like, P doom answer. It's more of like, this is what I feel people want to answer. So what, what's your feel?

• 2:09:52Nathan Labenz?

  The confusion and the lack of ability to establish consensus on.

• 2:09:59Nathan Labenz?

  Foundational point is a major challenge to having a lot of confidence on much of anything. I do think, you know, a good true north for me, well, the, you know, the true north for me with kind of everything I'm doing is trying to learn as much as possible. Trying to have the most up to date, comprehensive worldview as possible. And in terms of the approach that I would trust more than any other, I think still being hands on is second to none. And I, you know, I haven't allowed that to lapse much at all over the last few years.

• 2:10:36Nathan Labenz?

  But anytime I do get too busy or, you know, cluster too many podcast recordings into a week or whatever, I always kind of come away feeling like, eh, I gotta get a little bit more grounded with the latest stuff in a very interactive way. And I think one metric I have for myself, or metric is maybe not quite right, but an indicator that I want to pay attention to this year is can I get to the point where I'm spending less time at the desk? And that's like along the lines of the, you know, the Jason talking to Ultron. I want to be able to do stuff while exercising, even if that's just a walk around the neighborhood.

• 2:11:16Nathan Labenz?

  I want to get the frameworks, the tools, you know, the, the deep context, all that stuff set up well enough where I can start to go comfortably out into the world, have a thought, you know, maybe have an actual conversation, but move things forward in practical ways. In on fronts that like right now, I can really only do on my computer. I think a lot of that is right now with the latest models that have come out, kind of on me, just to get the setup and the sort of familiarity and the workflows to be able to do that. A little bit probably still.

• 2:11:56Nathan Labenz?

  More than a little bit, but you know, I could put more on me right now in terms of like, why have I not hit maximum capacity than I do on the, you know, the models or the model developers? More computer use would help for sure. Well, you know, a little bit more ability to just get over these sort of UI humps remains, I think, a barrier. And I, another thing I really have been not surprised by, but something I've really learned from just being deeply interactive over the last few weeks is I think another big unlock to watch for is when the models get better at knowing when to use code versus when to use their own fluid intelligence. Because one of

• 2:12:38Nathan Labenz?

  the first projects I've been doing is like just backfilling information, backfilling transcripts of the podcast for the website, backfilling all these different, you know, data sources into a queryable database. And you hit so many edge cases in doing that. And the model right now called, you know, OPUS, we've gone from, you know, four one to four five to four six pretty quickly. But pretty consistently, I have felt like it really wants to code. And I have often given it the feedback, don't try to guess at this and write like some sort of regular expression, or, you know, it'll, it'll like grep for one search term or another, or, you know, to throw 10

• 2:13:21Nathan Labenz?

  search terms into a grep command. And a lot of times I'm like, just read the document. If you just read the document, you will know what it contains, you will know what to do, you'll have the right judgment once you have read the document. If you don't read the document and you instead try to grep your way through it, you're never quite going to get there. And so that's like a meta cognitive skill that I think I've been able to improve its performance somewhat through prompting, but it's, you know, it's obviously going to get better in training. And that I think will be a huge unlock, just as it gets a little bit, little bit smarter around its own, a little bit more, a little bit more inclined, or a little bit more intuitive about when it

• 2:14:06Nathan Labenz?

  should deploy its own fluid intelligence, rather than use other tools. Getting that balance right will be, will make it, in my experience, dramatically more useful. And I have to imagine that's coming pretty soon.

• 2:14:20Speaker 01

  Yeah, I think, you know, when you, when we talk to James Howe today, that continual learning piece, the test time training, it it would be fascinating if it actually worked with your own model. Because your model will start to diverge. You have the baseline. And then your model will start to diverge and then it would become your personalized model within within like, you know, two or three cycles of like, you know, talking to a month, two months of data. It will become your own, your own model. It would start to diverge from the, yeah, and and that that would be fascinating. Because because at that point, it's for real, like you can, and especially for, I I I used to write a lot of journals. I have, you know, ninety nine to two thousand and three at Stanford, I have full journals for every single, every single month.

• 2:15:03Speaker 01

  Like everything that happened, you know, I obviously have never read those after writing them. It's just kind of an exercise in journaling. But I do wonder if like those of us who have lots and lots of written work, either in the public or in the private, once you get this continual learning going, you can kind of start feeding it in. This is what Kurzweil is doing with his dad's writing, by the way. He's a, he is, he's, he's feeding his dad's writing into these models. And he's like talking to the model about about his dad. Kurzweil is someday is going to feed all of that into test time training kind of model and, you know, with a voice access and he probably has a recording of the dad's voice and he's, he's going to start talking to the dad. It's a, it's a fascinating time.

• 2:15:44Nathan Labenz?

  Yeah, to say the least. More explorations of all these themes to come. A couple of things coming up on the cognitive revolution feed. One is with Ali Bayrouz, who was the nested learning author. He was on our last live show. I did do a full three hour. Ali's take on everything. And he's got a new paper coming out also that I think, you know, it's the way to continual learning is starting to become elucidated, I would say. I mean, I wouldn't say it's clear, but the it's. You know, no less than Jeff Tean has said that he kind of sees this as a very promising paradigm. So I'm definitely watching that really closely. Workshop Labs is a startup also that's like trying to do this,

• 2:16:28Nathan Labenz?

  you know, personalized model training on top of like the latest large open source models, up to the sort of Kimmy scale trillion parameter kind of thing. So that's one, that's actually another reason I spent so much time doing all this personal data curation is that I wanted to be able to give them a data set for them to train a model for me on. That would be like a really good data set. They don't need that much data, but I was like, well, you know, we really want to make sure it's the right data to hopefully get a good model back. So that's still pending. I haven't seen that model yet, but I'm very, going to be very interested to see how much that closes the gap between what Claude can do with just access to, you know, all this stuff in text.

• 2:17:11Nathan Labenz?

  And then how much does it help to actually start

• 2:17:16Nathan Labenz?

  tuning weights to, um, to try to capture more of like, they aspire not just to style transfer, but judgment transfer. They want the model to reflect the judgment that you're, you, the individual user would make at the time. And an interesting theory there too, is they, their motivation is that they want to help individuals preserve economic leverage. So instead of like doing everything through a foundation model and kind of adjusting yourself to take advantage of the model, they want to shape the models around individual humans with the goal that, you know, it's not a winner take all. Big tech runs away with everything, but some sort of more decentralized ecological kind of

• 2:18:00Nathan Labenz?

  proliferation of, of somewhat different models that hopefully at least kind of can exist in, in some sort of equilibrium with one another. And then on top of that, there's another one that I have coming to soon with the founders at Harmonic, and they are chasing mathematical super intonations.

• 2:18:15Speaker 01

  Yeah.

• 2:18:16Nathan Labenz?

  And when it comes to like these, I will say, just as a teaser, they gave maybe the most ambitious vision of what five years from now could look like, the most like mind blowing vision of what five years from now could look like of probably anyone that I've heard. And that is saying something because I've heard a lot, but they still kind of blew my hair back a little bit with what they think they can accomplish over the next five years.

• 2:18:41Speaker 01

  Definitely going to look forward to that one.

• 2:18:43Nathan Labenz?

  Lots more to come.

• 2:18:45Speaker 01

  Yeah, indeed. Nathan?

• 2:18:48Nathan Labenz?

  Thanks for doing this.

• 2:18:48Speaker 01

  Always. A pleasure, a pleasure.

• 2:18:50Nathan Labenz?

  It's been fun.

• 2:18:52Speaker 01

  Bye bye.

• 2:18:52Nathan Labenz?

  Until next time.

• 2:18:56Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement

• 2:19:37Nathan Labenz?

  at ai podcast dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.