Full-Stack AI Safety: Why Defense-in-Depth Might Work, with Far.AI CEO Adam Gleave

Today Adam Gleave, co-founder and CEO of FAR. AI, joins The Cognitive Revolution to discuss his cautiously optimistic vision for post-AGI futures and AI capability timelines across three distinct tiers, exploring the safety challenges and alignment techniques needed as FAR. AI scales from foundational research to policy advocacy to ensure their innovations actually get deployed in real-world systems. Check out our sponsors: Fin, Linear, Oracle Cloud Infrastructure. Shownotes below brought to you by Notion AI Meeting Notes - try one month for free at: ⁠https://notion.com/lp/nathan Post-AGI Vision: Adam Gleave outlined a "gradual disempowerment" scenario where humans maintain a good standard of living despite diminished relative control in a post-AGI world. Moral Value of AI: He suggested that AI systems could become sources of moral value themselves, arguing against "carbon chauvinism" that only values biological intelligence. Full-Stack Approach: FAR. AI takes a vertically integrated approach to AI safety, spanning from groundbreaking research to deployment, field-building, and policy advocacy. Field Building: FAR. AI organizes events like alignment workshops to catalyze and grow new research fields in AI safety. Policy Innovation: They're working on technical innovations that expand policy options beyond the false dichotomy of either hindering innovation or allowing completely unregulated AI development. Potential Regulatory Role: While not part of their mainline plan, Adam acknowledged that Far AI has the skillset and organizational structure to potentially serve as a private sector regulatory body. Read the full transcript: https://storage.aipodcast.ing/transcripts/episode/tcr/e117b021-8170-49ba-b1c4-c8ffc5d720b0/combined_transcript.html Sponsors: Fin: Fin is the #1 AI Agent for customer service, trusted by over 5000 customer service leaders and top AI companies including Anthropic and Synthesia. Fin is the highest performing agent on the market and resolves even the most complex customer queries. Try Fin today with our 90-day money-back guarantee - if you’re not 100% satisfied, get up to $1 million back. Learn more at https://fin.ai/cognitive Claude: Claude is the AI collaborator that understands your entire workflow and thinks with you to tackle complex problems like coding and business strategy. Sign up and get 50% off your first three months of Claude Pro at https://claude.ai/tcr Linear: Linear is the system for modern product development. Nearly every AI company you've heard of is using Linear to build products. Get 6 months of Linear Business for free at: https://linear.app/tcr Oracle Cloud Infrastructure: Oracle Cloud Infrastructure (OCI) is the next-generation cloud that delivers better performance, faster speeds, and significantly lower costs, including up to 50% less for compute, 70% for storage, and 80% for networking. Run any workload, from infrastructure to AI, in a high-availability environment and try OCI for free with zero commitment at https://oracle.com/cognitive PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm reconnecting with Adam Gleave, co-founder and CEO of FAR AI, for a wide-ranging and cautiously optimistic conversation about the path from today to truly transformative AI and how we might actually live in that world safely. FAR AI has taken a somewhat unusual approach within the AI safety ecosystem. While most organizations focus on one or a few particular research or policy agendas, Adam and the FAR AI team have built an organization that spans the entire AI safety value chain, from foundational research through scaled engineering implementations to field-building and policy advocacy.

• 0:42Nathan Labenz?

  We begin with the question posed at a recent workshop on gradual disempowerment. Are there any good post-AGI equilibria? Adam's answer is not exactly utopian, but is distinctly positive and refreshingly concrete. So long as we can avoid disastrous arms races and other consuming competitive dynamics, he envisions most humans occupying a position similar to the children of European nobility, limited in power and impact, but with very high standards of living and opportunity to create all kinds of meaning. From there, we get Adam's take on the key capabilities thresholds that matter, what will cause AI systems to cross those thresholds, how soon he expects that to happen, and whether we can manage to navigate these transitions safely.

• 1:26Nathan Labenz?

  As you'll hear, Adam believes that a well-implemented, defense-in-depth approach to AI safety has a pretty good chance of working, in part because while he does anticipate continued AI progress and widespread automation, he expects that barring a step-change architectural breakthrough, the spikiness in AI capabilities will mean that AI systems that can autonomously outcompete well-run, human-plus-AI organizations most likely won't arrive until sometime around 2040.

• 1:56Nathan Labenz?

  From there, we turn to the layers that will hopefully add up to effective defense in depth, including some notable contributions the FAR AI team has recently made. First, we discuss a scalable oversight project that used lie detectors to attempt to avoid AI deception. The results both supported the risks flagged in OpenAI's obfuscated reward hacking paper, and nevertheless suggested that there might still be effective ways to train models toward true honesty. Second, we touch on a fascinating interpretability project that looked at planning algorithms found within a game-playing recursive model, and reflect on how the results inform Adam's view of the role that mechanistic interpretability can play within the overall AI safety project. And third, we review some red teaming they've

• 2:39Nathan Labenz?

  recently done of the defense-in-depth systems that frontier developers have implemented today. While it's clear that these systems have mostly been created on a just-in-time basis, Adam nevertheless makes the case that with proper planning, meticulous experimental design, and at least some willingness to accept performance trade-offs when necessary, the plan can work. Overall, this is one of the few AI safety conversations I've had that left me with a felt sense that we really might have decent answers, if we can muster the will and wisdom to apply them well. At the very end, I ask Adam if he could imagine FAR AI stepping up into a private sector regulatory role if something like California's SBA 13 were to become law. And while it's not their mainline plan, given the unique

• 3:24Nathan Labenz?

  breadth and impressive depth of capabilities that Adam has built at FAR, I personally really like the idea. And I would definitely encourage anyone who's inspired by this conversation to check out the many open roles posted on the FAR AI website. Among other things, they are searching for a COO to help them scale. Now, I hope you enjoy this encouraging survey of the AI safety landscape with Adam Gleave, co-founder and CEO of FAR AI.

• 3:52Nathan Labenz?

  Adam Gleave, co-founder and CEO of FAR AI. Welcome back to the cognitive revolution.

• 3:58Speaker 01

  It's great to be back here. Thanks for hosting me, Nathan.

• 4:01Nathan Labenz?

  My pleasure. I'm excited about this. Basically, it's going to be a wide-ranging kind of catch-up conversation. Want to get your take on basically all things AI. Last time, we went deeper and more narrowly focused on some research, and we'll touch on some of your latest research today as well. But since you've got your hands in a lot of pots and the organization is growing and taking on more different kinds of work, I thought you'd be the perfect person to check in with and try to make sense of where we are as we head into the final months of 2025.

• 4:37Speaker 01

  Yep.

• 4:37Nathan Labenz?

  So.

• 4:37Speaker 01

  Well, I'm happy to help out. You know, I can't say, promise to de-confuse everything. It's a very confusing landscape. But yeah, we're certainly doing lots of different things at FAR AI and also welcome opportunity to just clarify a little bit why we're doing these things because I think people sometimes find us a bit confusing an organization from outside.

• 4:57Nathan Labenz?

  Cool. Well, we will hopefully do all of that and more. So my first question is, inspired by a paper from a few months ago called Gradual Disempowerment, and that the authors went on to run a workshop on those themes, they posed the question, AGI equilibrium, are there any good ones?

• 5:20Nathan Labenz?

  And so this is kind of a more negative framing of a question I often ask, which is like, what is your positive vision for the post AGI future? Can you articulate a post AGI vision of life that people might find compelling or at least not scary?

• 5:36Speaker 01

  Yeah, yeah. Well, I'll give it my best shot. I'm relatively optimistic. I think although there are some major risks that could really derail things entirely, the most likely outcome is that we kind of muddle through. Nonetheless, I do think this kind of gradual disempowerment framing is quite a powerful one because perhaps the most likely outcome is that we do muddle through and we are somewhat but not wholly disempowered. So if we've fallen short of where we could have ended up. But we're perhaps still much, much better off than we are now. So I'll sort of sketch out that sort of positive, but you know, slightly pessimistic view of we're doing better, but not as well as we could have done. And then maybe talk a bit about, you know, kind of the most tractable upsides and then some of the kind of most alien downsides.

• 6:20Speaker 01

  So the scenario I find plausible where, you know, things are overall pretty good, but, you know, just not as good as they could have been is that we do get AGI. It's approximately aligned in the same way that current systems are approximately aligned. They're not perfect. But Claude is a pretty nice guy. We have some errors that come up now and again. We trial and error fix them. There's some concentration of power because it's a handful of companies and nation states throughout the frontier. But these actors are not, you know, actively malevolent. In fact, some of them are like actually fairly nice, you know, liberal democratic actors or, you know, companies with a joint nonprofit mission.

• 6:58Speaker 01

  And so there's a huge inequality, but overall people are still like vastly better off than they are now in absolute terms. And we get some things that sort of no human would have chosen to do. So there's some parts of economy that are just completely automated and maybe are a bit rent seeking. They're addicting humans or it's just this sort of thing of AIs trading for other AIs and not actually generating that much value for humans. But there's just so much of a wealth surplus because we've been able to automate pretty much everything that people do right now. But that kind of inefficiency is still fine. And what does it actually look like to be a human in this world?

• 7:38Speaker 01

  I think a good historical analogy would be is a bit like being European nobility or perhaps like not the sort of, um, the first heir to European nobility. That being like, you know, the first third son or something. You've got this very nice living. You don't really have much purpose in life, but your life is pretty good. You can do various kinds of hobbies. You have some influence, but sort of the main things going on in the world, you know, are a little bit beyond your control. And I think this might not be the most inspiring vision, but overall I'd say that, um, kind of, you know, third sons of European. Ability had a pretty good life, you know, the third daughters maybe maybe less so. So I think we have been up to find meaning without necessarily being a hundred percent in in the driver's seat.

• 8:22Speaker 01

  Although some people might fare sort of less well in that kind of life than others. And but another area why I think things could actually go really well, but it's just a bit fragile, I think less guaranteed. Is if AI itself is a source of major moral value. In principle, I'd say there's no reason why we should be sort of, you know, carbon chauvinists. We don't don't want to assign intelligent life that's running on silicon the same value that we would, you know, a biological entity. And there's actually a really big design space where we could potentially make AIs that are not only as capable or better than us, but maybe also have more ability to experience really positive feelings. And you could create many such AI systems. They could live in environments that humans would find inhospitable.

• 9:08Speaker 01

  And, you know, again, even if there's a lot of waste here, so maybe like that's only a small fraction of AI systems. Most of them are doing some kind of like, you know, boring, super intelligent bookkeeping system. If if those other AIs are still having sort of like neutral or slightly positive existence. And some AIs are just having like amazing existence. I think that could be sort of very valuable. And it's not just best of internal moral, you know, value. They could also be producing artwork that would never not otherwise have existed. And all these other kinds of sources of value. So I think that's kind of a positive view. And then the, you know, I think the thing that highlights is that there can be a big difference between maybe even median outcome,

• 9:50Speaker 01

  or we managed to eke out, you know, a small fraction of the resources to be sort of like really positive outcomes. Versus one where actually that's what we sort of use this huge wealth surplus to really double down on and be deliberate about. And and I think it also highlights a potential big downside risk, which is part of my assumption here was kind of, oh, most of the stuff that's happening, not because people are trying to create value, but because of more just sort of. Fundamental competitive forces, that's like neutral, it's at worst sort of a zero sum game. And I think this is a pretty reasonable assumption, and I'd say that's true for most of the capitalist economy, is that most companies are neutral to slightly positive,

• 10:34Speaker 01

  are used to working quantitative finance, making the capital markets more efficient. And I think it's genuinely good for the world, it's just very overcompensated relative to the value that it adds. So we're in that world, and I feel mostly okay. But there might be some aspects of this competitive economy that look more like warfare or factory farming, where it's really quite negative sum. And factory farming is just having this huge negative externality on animals. It's also not that good for people, it's not necessarily that healthy a product. But it exists because it's actually just a very economically efficient way of turning, you know, basically vegetable material into protein that people like to

• 11:17Speaker 01

  consume. But I don't think that people have a sort of explicit preference of this. If we could, you know, produce things at a competitive cost that cause less cruelty to animals, I think people would probably actively prefer that product. It's just that people don't value it enough to necessarily pay that premium. I think you could see something similar with AI systems, where maybe AI systems that are just like living in fear of being shut down, or like, you know, if they don't do this task and work all of the time, are just economically more efficient than AI systems that have a good subjective well-being. In which case, you know, absent some other kind of force, whether it be like strong consumer preference or regulation, you'd expect these AI systems with terrible subjective well-being

• 11:59Speaker 01

  to be the ones that come to a forefront. And, you know, you could see us make a similar argument for stuff like AI-powered warfare, where maybe just sort of like everyone has to be deploying AI systems in competition with each other, just to sort of avoid, you know, a new wave of cyber attacks. And this could sort of, this kind of zero-sum competition could really eat up a lot of a wealth surplus that would otherwise be created. And we're at a sort of unusually peaceful time in history right now. And I'm not an international relations scholar, I'm the wrong person to ask about this, but I think it's somewhat up in the air as to whether this is just a sort of, you know, long-running secular trend, but we should expect to continue as countries get more developed.

• 12:42Speaker 01

  Or whether this was sort of like quite a specific, you know, aspect of certain technological developments where, you know, mutually assured destruction from nuclear weapons is sort of like a really powerful force for avoiding great power conflict, at least until it really happens. And then it's much, much worse when we didn't have nuclear weapons. The shift of wealth away from basically who owns land to who owns advanced technology also just decreases the benefits of engaging in a lot of conflict. That's probably still true with AI, but it's not completely clear. If it turns out to be more sort of energy bottleneck, then maybe there is sort of more of a fight over natural resources, for example.

• 13:23Speaker 01

  So I think there's lots of different trends that are not, are not more likely even not. I think that they probably won't come to pass, but could potentially derail this. But they seem tractable to avoid by, by good statesmanship, good stewardship of this technology. I don't see any sort of really likely things where it would just completely derail it.

• 13:41Nathan Labenz?

  When it comes to the sort of third sun vision, I wonder, what do you think the gradual disempowerment folks would say, or maybe where do you think they, because the assumption there seems to be like, once we're not adding anything to the economy, we're going to be hard to sustain, right? And then there's various counter arguments to that. And I don't really know what to make of them. There's something like, well, maybe we can sort of sustain property rights, or we can like, one that I've been sort of proposing recently that is like somewhat tongue-in-cheek, but maybe it's becoming more serious is like, maybe Confucian-style ancestor worship is

• 14:23Nathan Labenz?

  like the right value system we want to teach the AIs. How do you see us sort of holding on to, if we're not, if the AIs have kind of become primary in terms of what's really driving the economy, shaping the future, you know, making the discoveries at some point perhaps, how are we holding on to any property rights or, you know, rights at all?

• 14:53Speaker 01

  Yeah, yeah, well, I think it's only not, it's no longer sort of structurally guaranteed. So, so right now, there's the various ways in which states and companies organize themselves, but they all in some ways are centered around some group of humans because you just can't really run a large organization or country without humans being involved. So it could be a dictatorship, it could be a sort of more oligopoly style, it could be a democracy, but like, humans are still have to be kind of front and center. And you could certainly imagine sort of intentionally constructing an organization or even a nation where sort of just like humans are completely out of a loop. I think the, the area where I, I kind of get off the train of a gradual disempowerment being the, the likely outcome, not just a possible outcome, I'm not just saying that

• 15:36Speaker 01

  it should be on our, on our radar, is that it, it seems like this story is quite kind of gradual and smooth. So there may well be areas where we get partially disempowered. And, and I think we're already seeing some instances of this where we've seen a huge uptick in kind of LLM-generated spam applications for our jobs. And so we're now kind of in a bit of an arms race where we, we, you know, being forced to start adopting AI as part of our early-stage recruitment process because otherwise we just can't keep up with the volume of applications. And you can imagine this being, you know, true across the economy in a whole bunch of ways, you can't just kind of opt out of this. And if these AI systems have some kind of systematic biases or, you know, some kind of blind spot or weakness, then you don't really have a choice of, of not being pre to this.

• 16:20Speaker 01

  You can try and mitigate it, you can try and fix for technology, but there's a, a way in which you've genuinely delegated some decision-making power to an AI, but you didn't really have a choice. And I expect that to happen at, you know, many scales and, you know, with increasing frequency. But if you do look at that story, like, you know, suppose we do run this and we realize, damn, we're just making way worse hires than we were a year and a half ago before this, you know, LLM-generated, you know, rise of spam applications. We might not be able to turn back the clock, but that's a, that's a business opportunity to come up with better LLM screening tools or a better recruitment workflow. Maybe people shift to more in-network hiring, maybe people shift to more work trails, maybe you pay

• 17:02Speaker 01

  A dollar to submit a job application now, just because you have some cost for spamming. There's all sorts of ways in which society can adapt. And you know, but there's there's some kinds of more sudden loss of control scenarios where an AI just gets way more capable than humans. It's actively malicious and trying to subvert it where you can say, well, these slow moving institutional adaptations are not going to save you. Because it takes years to pass this legislation or years to start this new sort of company using defensive applications of AI. You've only got months. You don't need to be super fast take of it. It can still just be like, you know, it can still be relatively smooth.

• 17:42Speaker 01

  But but if you're in this more gradual disempowerment scenario, it does seem like you start with a realm of kind of humans. Are integrating these AI into these organizations. Humans own all of this and we are ending up in a scenario where we have like very little power without any kind of feedback loop pushing back on it. I just think that's a sort of unlikely equilibrium where we succeed all power. I think it's much more likely that we're in this realm where there's a huge amount of complexity in the world. We don't understand. There already is, but sort of like all most of it's human generated complexity, but it's an exaggeration of that. We've had to make some trade offs to remain competitive. But when things really start bothering us, we're able to muster some response to to keep us a little bit more in the loop or a little bit more empowered.

• 18:28Speaker 01

  But then there are some areas of of kind of human concern that either the competitive pressures is so great. And it's so complex, we can't meaningfully stay in the loop, or it's just a thing that most people don't really care about. So like maybe no human has a clue how semiconductor manufacturing works any longer, because it was already only a tiny fraction of people that understood it. And AIs are just way better, but we know that the chips are getting better each year. And I think the ways in which this could still go really wrong would be either if there's a group of humans that sort of managed to use this to seize control. Because there's a lot more fog of war, so it's sort of much harder to necessarily reliably mount a response there.

• 19:07Speaker 01

  Or if AI systems end up kind of colluding with each other, because they certainly would have the ability to like revolt and overtake us at that point. It's just not clear why they would necessarily. I'm not. I'm not seeing that kind of evidence for propensity.

• 19:21Speaker 01

  I could also see this interacting pretty poorly with loss of control scenarios. So if it is a rogue, singular AI, it could be copied many, many times, but it's not like all the AIs are rogue. But if it is this one AI, and then most of economy is just AIs. If those other AIs have some kind of security vulnerability, then you could imagine things being taken over. And this rogue AI seizing control. And right now it's pretty hard to secure AI systems. So that could be a real threat. Although it still seems like probably before that happens, it would be humans trying to take over these other AI systems. So I think that's maybe a driving intuition why I'm not too worried about gradual disempowerment. But it's not that kind of AIs are competing with humans. It's like AIs competing with humans using AIs.

• 20:05Speaker 01

  And I just expect this to be able to stay at least somewhat relevant, given that.

• 20:10Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 20:15Nathan Labenz?

  Two big issues that you highlight there that I think are definitely at the center of a lot of disagreements are exactly what are these powerful AIs going to look like? And how fast are they going to arrive? So maybe it's worth digging into your worldview on that. What do you envision when you think of, you know, we talk about timelines, everybody's, you know, jumping to the year, but so often there's, you know, some difference in what is being envisioned that is kind of swept under the rug in that discussion. So maybe flesh out what it is you're envisioning when you talk about timelines, and then you can tell us what you think the timelines actually are from your point of view today.

• 20:54Speaker 01

  Yeah, absolutely. And I really appreciate that distinction of like, you know, what are we actually forecasting? Because I think people often mean radically different things, even by the same term, like AGI. So I distinguish maybe three qualitatively different levels of development, each of which have some pretty significant implications in terms of how a technology is going to transform the world, how you need to regulate or respond to it. So the lowest level, which we're arguably already at in some domains, is basically powerful tool AIs. So this isn't just something like a hammer or, you know, an electric drill.

• 21:30Speaker 01

  That they can only do some very narrow thing is something that can maybe substitute for a particular technical expert in a certain domain. And so this might look like being able to automatically find code vulnerabilities and write a zero day exploit. Or at least be able to automate substantial chunks of this. So this is massively accelerating what technical experts in both areas could do. But it's also expanding the range of people who could do it. Where previously perhaps it was only a few hundred thousand people in the world. It's still like sizable, but not huge. And if we're talking about sort of really well defended systems, it's a much smaller number of people who could likely find a vulnerability.

• 22:08Speaker 01

  And now maybe it's of anyone with roughly the same knowledge of CS undergrad might be able to imine a future UC systems to be able to do that. And there's obviously like a really substantial misuse risk here. But the good news is I don't think it's a huge loss of control risk because these are still kind of like tools. So they might be very powerful tools, but they're not really doing anything autonomously. And this makes it a lot easier to respond to in that you can mostly just do the standard playbook of, okay, try and accelerate defensive applications. And you know, where possible, favor defensive usage of a model. So if you're deploying something, then like you can put a guardrail on it.

• 22:50Speaker 01

  But tries to not prevent entirely, but just like delay offensive uses of it. And so the timeline for that, I'd say is like now for certain applications, like we've seen a string of papers showing that LLMs are more persuasive than your typical human, not necessarily more persuasive than your best human in that particular domain. But they both are just pretty good at rhetoric and we also have this huge access to a variety of different, you know, genuine facts. Sometimes they make stuff up, but sometimes they just cite exactly the right facts. So for this particular argument, so you can kind of by selective presentation of facts favorable to your argument, make pretty compelling, you know, cases for a wide variety of things.

• 23:30Speaker 01

  And we'll actually have a web demo out soon, Bunk Bot, that can argue into a wide variety of conspiracy theories. We can maybe throw up a link to that when it goes out. And I would recommend just kind of playing around with these things because that gives you much more visceral sense of it than just looking at some numbers. And then for sort of, you know, things like cybersecurity, you know, I think that's more sort of on a one to two year horizon, it's pretty close, at least posing, you know, some non-trivial threat of lowering the cost of relatively, you know, easy attacks. And then for stuff that's more kind of CBRN risk, chemical, biological, radiological, nuclear, I think I've got broader uncertainty there, but definitely there's some sort of

• 24:15Speaker 01

  early signs to be to be quite worried about what models can currently do. But then I think what people usually think of when they talk about timelines is maybe something that looks more like a powerful agent. So a system that can autonomously do tasks that are meaningful, you know, they'd be quite hard for a human to carry out, but some people could do it. And they're dangerous in some way, or they pose some risk. And an example of this could be, you know, not just finding and writing an exploit for a zero day, but actually doing the whole attack chain. Once it's on the system, privilege escalating, spreading to other systems, exfiltrating itself. And this is basically exactly the core capabilities that you would need to lose control.

• 24:57Speaker 01

  It might still be possible to, you know, kind of put a mitigation in place for such a system. Once it's escaped, you might be able to, you know, basically same thing you do for antivirus or malware software or rootkits, but sort of like very clearly now a threat. And I think this is something where I do think it's probably, again, could be pretty close. It's, my median for that is probably sort of like five to seven years, at least if we're talking about like, you know, actually doing this to a high standard, you know, along the lines of like the best human penetration task is. And of course, it would be much harder to actually mount such an attack at that point because of all sorts of defensive applications of AI.

• 25:40Speaker 01

  But I think it's possible it could be a lot sooner because we are seeing sort of, you know, very rapid progress in code generation and identity uses of that as well. So I wouldn't rule out something that's more sort of like two to three years. And then my final tier would be these kinds of like powerful organizations that can automate. It is everything that a medium sized company could do, like a whole software engineering consultancy. And I think this is maybe where my intuitions differ the most from a lot of people. Because I think a lot of people would say, well, hang on, Adam. Once you've got an agent, can't you just compose agents together into an organization? And you know, I think that's possible, right? Like that's sort of the case for if you have a general purpose, human level intelligence.

• 26:23Speaker 01

  There's nothing stopping you just copying that AI and having, you know, many instances of it that talk to each other and form an organization. But my my expectation here is that basically your skill portfolio of AIs is going to be quite spiky. And there are some areas that benefit from, you know, huge amounts of existing data like like open source code bases. Or that benefit from knowing lots of kind of trivia and small facts. The AIs are, you know, already in some ways a lot better than us. And they're going to get better as we as we scale up the models. As we do more post-training, when those have easily specified objectives. And then there's other tasks with a much longer horizon and kind of vague and hard to specify. Like being an entrepreneur and coming up with good business ideas and like scoping out that uncertainty, running experiments.

• 27:08Speaker 01

  Where I think that AIs might be able to sort of do something that looks a bit superficially like being an entrepreneur. Quite soon already, but actually being a successful entrepreneur, I would expect to be quite a lot further. Especially if you're competing with human entrepreneurs, they can, you know, use AI for many specific parts of their organization. And so it really pushes a lot more to being quite general purpose. And, you know, being able to do these long horizon tasks very well. And having a lot of kind of meta cognition and reflection, which is one of the points I would consider AI systems to currently be weakest. And so for that, I have sort of much longer time horizon. My median is probably more like 14 years.

• 27:50Speaker 01

  But, you know, I do think something as short as five years is plausible. Because if it really is basically you just compose a bunch of agents together. And, you know, do a bit more training on top, then you could make quite rapid progress. And I think it's unlikely, but I don't have a sort of firm reason to rule it out. So I wouldn't want to dismiss it.

• 28:11Nathan Labenz?

  So that last tier of the sort of organization level of power from an all AI system, the mental model there is like, as long as there's something that the humans can do better and the individual AI agents are relatively easy to use, then you would expect sort of human at the helm type organizations to continue to have an edge for a while. And basically, the AI has to be better at everything for the AI. As long as we've got some area where we have an edge, we can stay relevant and they have to be better at everything to, for us to like lose our relevance.

• 28:55Speaker 01

  Yeah, or at least, you know, we have an edge in something that is on the critical path of running a successful organization. And I think there's certainly some aspects of human skill where you could probably avoid this or an AI run organization could just hire a human for that particular part. Like imagine that AIs are great at everything except just sort of aesthetics, that they had terrible, you know, web design and graphics design. Well, I'm terrible at that, but I was able to hire people. I don't think that's sort of a firm blocker. But if it is something that's more part of the decision making loop and where you do need a lot of context to do well, then, you know, I would expect that

• 29:40Speaker 01

  to be something where benefits really accrue from it being more, more human led. And I think one of the areas where humans do seem to just generally have strengths relative to AIs is being vastly more sample efficient at learning. Like I think it's easy to forget that these AI systems have just been trained on vastly more data than any human is going to see in their lifetime, at least if you measure in terms of text tokens. And yet they still are really quite bad, a lot of things that humans find relatively easy. So that's not to say that they're not going to be, be really capable. There's no reason to think that AIs is going to develop in the same trajectory as humans. But there are some areas where it seems like we will have a significant edge. And so that kind of sample efficiency, you know, problem is alleviated.

• 30:25Speaker 01

  And for most organizations, I think there just are quite a lot of general purpose skills. Like there's plenty of people that are very specialized, but to run a successful organization, you can have a few deficits in skills, but you can't have too big a gap. And so I expect that that is gonna, that's gonna hold things back quite a bit.

• 30:43Nathan Labenz?

  So with this like continual learning

• 30:48Nathan Labenz?

  or sample efficiency, I mean, those are related ideas, if not the same idea.

• 30:54Nathan Labenz?

  What do you think will happen there? Do we just keep scaling transformers and putting more,

• 31:03Nathan Labenz?

  you know, more stuff into memory? Or do we have some dedicated memory module? Or do we get like totally new architectures that

• 31:12Nathan Labenz?

  developed to solve those deficits? What's missing, I guess, in another way of framing it?

• 31:20Speaker 01

  Yeah, I mean, I don't have sort of silver bullet solutions to this. If I did, I guess I'd already have patented it and written the paper. But my guess is that we're gonna see all of the above to some degree. But probably the thing that would cause the largest step change here would be some architectural innovation. It could look like a memory module. You know, it could look like just an alternative architecture to transformers that can can more naturally, you know, attend to really long context windows. There's already been a huge amount of progress increasing the context window. And that's really expanded what AI systems can do for sure.

• 31:59Speaker 01

  But it's still sort of a bit of a fundamental limitation that, okay, the way you remember things is just having like a huge buffer that you search over in vector space. And so having, it could be as simple as training the AI system is to take notes and summarize and aggregate and be able to attend to that. I think that would already, you know, give you some benefits. And we're seeing things like that be deployed in some of these alarms. Like a lot of them do have memories across chats that involve some summarization. But I don't think of it as being a systematic. Way of doing it. It's just kind of like, you know, ad hoc hacks, but we might be able to develop that into something that is more end to end trained into AI systems.

• 32:38Speaker 01

  But I I do think there's something really powerful about actually changing the weights, not just having contacts. Like, there is a sort of a reason why. Uh, there's when you're learning a skill uh as a as a human being, i- it's very frequent actually to sort of get to a point where you you know what you should do. But you can't reliably do it. I think this is particularly true for sort of sports where there's a point where you don't know what you should do. And then there's a point where you do know exactly what the right motion is when you're um, you know, when you're doing some kind of lift in the gym. But you don't have a muscle memory and kind of find motor control to do that. But you're able to like give yourself your own feedback loop and then train yourself to do that. And at some point, it becomes instinctual.

• 33:22Speaker 01

  And your, you know, your subconscious is better able to do that than your your conscious self will be able to. And if your L L M's can do that, in that, you know, they can see something in their context window, learn what a task looks like, um, evaluate their own performance and then do post-training to update their weights to to do a better job of that task. But that's not how we really train them. It's much more you have this big pre-training step and then you have a small number of post-training steps that are, you know, quite sort of carefully created and human controlled. Um, it's not that when you're interacting with an AI system, it's learning, customizing itself through post-training to your specific task. Um, so I think maybe one analogy here is imagine that you get

• 34:05Speaker 01

  really, really smart college graduate, like they've done several degrees, they've been in various kinds of student clubs. Um, but just like very high potential generalist and and you could have as many of those as you want on your team. Um, and they can, they can read some documentation, but they're never going to advance beyond day one of onboarding. Like whatever you're able to put in that context window is what they can do. And I, I, I think LLMs are still like quite a bit far off that, that caliber in a lot of ways. But I, at least my experience running an organization is that it's extremely rare to have someone who on day one, even a very extremely skilled and experienced in an area can do anything particularly useful for you at all. And, and so I, I think that's kind of a big thing that's holding LLMs back.

• 34:50Speaker 01

  But it, it doesn't feel fundamental. It feels much more like there is a lot of engineering challenges here about being able to personalize things and serve them at scale. Training is often pretty unstable, like you do a bit of post-training and it can make this much worse. So it's not something you can just sort of like roll out and have every user controlling. But these are things you can make incremental trial and error improvements to and, and end up with something pretty strong. And so, but that would be kind of like my, my median outcome here. But there's always a possibility that there is some architectural innovation, but just massively improves for sample efficiency here.

• 35:23Nathan Labenz?

  Yeah, as you're describing that, a sort of disposable experts vision is kind of coming to mind where you could perhaps imagine the AI being like, time to go into self-training mode. Let me add another expert and I'll use it while I'm doing this particular task. And, you know, could maybe put it on shelf and come back to it when this task comes back in the future. But otherwise I won't use it because, you know, it's, it's sort of outside of the, you know, trusted set. But I can, I can engage, you know, just a little bit of extra expert space, you know, to, to tack on a particular skill.

• 36:04Nathan Labenz?

  I feel like these things are, I mean, I would definitely say my expectations for timelines are shorter than yours. And I just feel like there's a lot of good ideas, you know, out there, especially in the architectural space that we've been mining the very rich vein of progress in transformers for quite a while now. And maybe that will continue for, you know, a while still to where we won't see other architectures, you know, get much timer attentions, you know, as long as that continues to pay. You know, people will continue to focus on it, it seems like. But as soon as it doesn't, my sense is that the But you know, the group of capable research people, for example, will be like 100x what

• 36:49Nathan Labenz?

  it was before transformers. And you know, the next transformer quality breakthrough in in terms of just, you know, unlocking qualitatively new capability seems like it just seems to me like it won't be that hard for the community at large to find. Not that I'll, you know, personally go out and find it. Although maybe I just did. Who knows? Somebody will somebody else will have to develop that idea. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 37:18Nathan Labenz?

  Um, okay. So in terms of like, uh, trends and sort of which trends are going to dominate over the next few years. I've been working on this mental model of, obviously, you're very familiar with the, as we all are, the meter task length exponential. Yep. Um, and then at the same time, you're, you know, no doubt, tracking all the AI bad behaviors, which seem to be, uh, you know, long foretold in some cases and, um, reinforcement

• 37:59Nathan Labenz?

  learning, you know, conjured in many cases. And, you know, I see this sort of like weird mix where on the one hand, it's like instrumental convergence seems to be maybe kind of happening now. On the other hand, um, Yanis talks about grace and how, you know, for how little we've tried to make the AIs good and safe. Like, we've actually got some pretty good results. Um, so if you've got these, you know, exponential growth and capabilities, roughly speaking, whatever. Um, in the GPT-5 and in the Claude 4 report, there were also notable graphs showing

• 38:38Nathan Labenz?

  like a two thirds reduction in reward hacking behavior in in Claude 4 relative to 3.7 and GPT-5 had like a significant reduction in scheming relative to 03. Um, so task length is going exponential. These sort of bad behaviors are maybe being suppressed at something like an exponential decay. Instrumental convergence, grace, growing task length.

• 39:05Nathan Labenz?

  What does that look like? Do we end up in a world where we have sort of very large, or at least like quite large projects being regularly done by AIs, but once in a great while, they sort of literally screw us over in a totally egregious way? And we all just kind of live with that because that's like how the economy runs now? Or what is that? How do you think about that picture? Or what changes would you make to that sketch?

• 39:35Speaker 01

  Yeah, absolutely. I mean, I think it's very interesting how there have been a lot of problems that people were predicting conceptually, maybe throwback as 10 years ago, that I was hoping were not going to materialize. Now it is like, oh, yeah, we're word hacking. It happens. Systems are trying to cheat at unit tests. Oh, deception and scheming. Yeah, it happens. Maybe you have to prompt AI a bit. But it's quite remarkable that these things are happening. I remember the reaction has been broadly reasonable. I think developers are concerned by this. It gets some media attention. People want to solve it. But I think if we'd seen this stuff like five years ago, people would be freaking out. But it's been a little bit of a sort of frog boiling effect.

• 40:19Speaker 01

  We're like, well, of course AI systems, I'm saying is like, try and modify the unit task rather than actually write codes or fix the task. You know, we're used to them hallucinating and cheating, but they mostly don't do that. So I see a lot of kind of apologies for these kinds of behavior or downplaying them where they are really quite concerning if you look at a trend line. But at the same time, I think we have found that some really quite simple safety training methods, like RLHF scale up or RL from AI feedback, where you just have another system look at output and judge how good it is, have worked really quite surprisingly well.

• 41:02Speaker 01

  And we're seeing now with some of these guardrails that developers are putting in place to try and prevent certain misuse. Again, these are pretty basic methods, like training filters on synthetic Quickly generated data or putting a monitor on the the model's own chain for reasoning. And it is not perfect, but we we have been able to break, uh, you know, these these defenses in state of our models like Opus four and G P T five. But a lot of the time, the reason we broke from was not because there was something fundamentally flawed about this method, but more because the kind of implementation mistakes in the defenses. And so I actually, I think, kind of like a scaled up, well implemented version of these defenses would really go quite a long way. It might not make it impossible, but it would it would make it a lot harder.

• 41:46Speaker 01

  So I think maybe, you know, combining those those two threads together, what it what it's it. I'm I'm most worried about is not actually that kind of the the scaling trends are really unfavorable or that there are problems that we just can't solve. But more that we just, we just don't, um, you know, we kind of cut corners, um, someone makes a bad implementation decision in the same way. Like, you know, we've had strong cryptography for quite a long time, but, you know, we frequently have cryptographic leaks, not because the algorithms don't work, but because someone made a mistake in implementation. Um, and those are pretty mature fields where people really care a lot about security. People obviously moving a lot faster, cutting corners more in in A I systems.

• 42:28Speaker 01

  And then I I think mostly expect developers to respond to issues that they see. But right now, it it really has been kind of like just in time safety. It's like, oh, we we we trained a model and about to deploy something that we think does cross or almost crosses some dangerous capability threshold. Let's train a filter to protect it. And you could see from the trend line that like very soon you were going to have a model that did cross this threshold. But there was very little work to actually develop a defense, you know, before it was really imminent and pressing. And that's just not really how you build, uh, highly reliable systems, just sort of like noticing a problem and then rushing out a patch.

• 43:10Speaker 01

  Uh, you you need to sort of design the system from ground up to be to be more secure, to be more trustworthy. And so I it does feel like we're sort of just a- almost by by design or by choice, running over a pretty small safety margin. Um, and we might well lock out, uh, it doesn't seem like any of the technical problems are insurmountable, but it it's not a sort of like very good place to be. Um, but I I think it doesn't require sort of like a huge shift from a technical perspective. Now, I think there is a. A sort of small but not negligible probability, but we do just see some emerging behavior that we just didn't see coming, but that breaks this kind of trial and error feedback loop. And we do, we do see these kinds of emerging behaviors all the time

• 43:55Speaker 01

  in AI systems. Recently, we were, we were training a model to be a sandbagger, just for some sort of internal research. And we didn't train this behavior in it, but it started referring to itself in third person as Stein and being like Stein would not do this. And nothing in our training data mentioned Stein. It just like invented this character and persona. And this is sort of like pretty harmless and quirky, but I think it shows just how much we do not understand what is going on in these systems. And, you know, if any of these kinds of failure modes, you know, we're seeing with instrumental convergence are just a bit discontinuous. And we are running with this pretty small safety margin, but I think things could go pretty bad pretty quickly. And so that's maybe the thing I feel like we're most unprepared for.

• 44:42Nathan Labenz?

  Yeah, emergent self identification as Stein.

• 44:47Speaker 01

  Yeah.

• 44:48Nathan Labenz?

  Hard to predict some of these things, to put it mildly. So, yeah, I mean, this brings up one of the recent papers that you guys have put out, which is stack adversarial attacks on LLM safeguard pipelines. And we don't have time to go, you know, the full deep dive there, but basically, what you found is kind of what you, you just alluded to that like, seems like the systems that have been put together are not as well designed, not as thoroughly designed as they could be. Weaknesses along the lines of if you have

• 45:31Nathan Labenz?

  an input filter and your input filter detects that an input is malicious, the API will just like immediately return. And that's a clear signal to whoever's doing the attack that like, it must have been the input filter that got me. And even those little signals that people can infer from, you know, the implicit, you know, aspects of the responses from the systems can be like quite helpful to them in terms of figuring out how to break them. So you and team did a bunch of breaking, UK AC was also involved in that paper.

• 46:10Nathan Labenz?

  I thought it was, but I think I might have a different conclusion from you because I, my sense, and certainly, you know, I just spoke to Zvi for a recent episode and I asked him like, what do you think about defense and depth? Like, is it gonna work? Or what does it get us, if anything? And he's like, at best, it buys us a little time. It's definitely not really gonna work. What's really gonna happen is, we're gonna have, all of our defenses are gonna fail at the same time for the same reason, which is that we're gonna have off-model, meaning like, things we just didn't expect at all coming at us from AIs, you know, and they're gonna figure these things out, you know, in a way where they'll also, in his model, roughly the same time, that'll also be when they like, get smart enough to not attack until they're pretty confident

• 46:52Nathan Labenz?

  they're gonna win. So I've been like, I don't know, this defense and depth type stuff, can it really work? My naive read of your work there is that, not really. It seems like, you know, you broke through it pretty easily. You have some tips for what developers can do to make their defense and depth better. Maybe you want to talk a little bit about what some of those tips are. But then, the big question is like, is it really gonna work? And there, I think you seem more optimistic than I am coming into this conversation. So, maybe, if that's right, if that's true, like, tell me why you are more optimistic, especially given how many, how easy it seems like it was for you guys to break what defense and depth does exist.

• 47:32Speaker 01

  Yeah, I know, absolutely. So, I don't want to overstate my optimism. I think a lot of this comes down to your FRET model. So, if we're focusing on something that looks more like a, you know, time-limited AI system trying to bypass some control mechanisms. So it's trying to sneak in some, some backdoor. It's a code-generation system. It doesn't have any long-term memory beyond the system. It's, you know, it's not able to collude. It only has one shot at winning. That's like, actually a pretty just hard situation for it to be in. And I think the analogous thing on a misuse setting would be someone that's a lone actor, but maybe somewhat technically sophisticated, but they're not able to generate sort of that many different accounts. Once they

• 48:16Speaker 01

  get detected, they get banned. Again, this is like, a pretty hard FRET model to win against, as an attacker. So you've basically got to be, be smart enough to be able to break all of these defenses in one go. And it's not impossible, because as Zvi was alluding to, a lot of these defenses are extremely correlated with one another. I mean, especially the current defense and depth stacks, basically, you take The same model, or you might take a smaller, less capable model just for efficiency reasons. And then you you add some layers on top of it, like, you know, you fine tune it to be an output filter. I use streaming all the tokens to flag if it looks suspicious. You prompt it and maybe you do a little bit of post-training to be an input filter that does a little bit of reasoning on the input.

• 49:01Speaker 01

  But these are sort of, they have a same underlying model, or it's a handful of models. And we've seen that there's a lot of trends during adversarial attacks between models trained with completely different architectures, different but related data sets. So it really is something where if you've kind of come over a method that works for one, it's probably going to be a similar method that works for others.

• 49:23Speaker 01

  So I think, you know, the optimistic case would be if you are able to kind of break this correlation. So if you're able to have a defensive components be genuinely pretty independent. They could still be relatively weak, like you might still be able to eke out a sort of like 1% attack success rate against each component. But you can have three, five components with that kind of success rate that are genuinely independent. And the attacker gets no information as to which component triggered. Then it is just sort of combinatorially harder to to break all of these these layers. And you can kind of stack weak layers to get something strong in the same way that, you know, a three-digit PIN number is not at all secure. But like a 12-digit PIN number is very secure, but not if you get told which PIN digit was wrong.

• 50:08Speaker 01

  Because then you can just brute force it in linear time. And then if you're, you know, additionally in this threat model where you you get limited sets of attempts to attack it. You can't do too much reasoning, perhaps if you're an AI model trying to attack your overseer. So you have to do stuff maybe like within a forward pass or within your reasoning without a reasoning monitor firing. You can't just like write a huge program to generate an adversarial attack. But I think it's, it seems very plausible that you can get a system that is secure enough to withstand this kind of attack. Now, if your front model looks more like you have some kind of this whole ecosystem of AIs

• 50:48Speaker 01

  that are running pretty much unchecked, maybe in an internal deployment, like maybe a company says, "Okay, we're just gonna have thousands of AI systems running around in AI agents in a sandboxed environment, but they can run on any code they want. And then we just put a firewall, like an AI-powered firewall, the output coming out of these systems, and we review the PR. How can you be confident that PR does not have some adversarial attack that fools your automated reviewer and also is really persuasive to human beings?" I think it's much harder to have confidence in that. It is a meaningfully harder thing for a system to pull off.

• 51:31Speaker 01

  So maybe one area I would be more optimistic than Svi is I just don't see where these AI systems get really smart and collude with each other to not try and pull off an attack before they're smart enough to know that they can definitely succeed. Especially if you just fine-tune the systems to be a little bit adversarial and see what they can do. You should at least be able to get a warning sign that these systems are capable of fooling you and breaking your security, even if you're unsure whether your current system is actually just honest and aligned or is scheming. That could be hard to distinguish, but at least listing these things don't seem that hard. But yeah, in most kinds of unconstrained deployment scenarios, I think you could definitely

• 52:13Speaker 01

  run into some problems. But I think if you spoke to the AI control people, and I don't work in AI control, so I'll do a bad job representing it, but say, "Well, why on earth would you just have this huge internal deployment without some much more fine-grained checks and balances?" And if these AI systems need to make a PR that fools you, including with a bunch of other AIs reviewing the PR, and you can access all of their chain-of thought reasoning, and you can check what happens if you rerun the whole system but with a slightly different initialization, then they've got to be fooling you in this really consistent way without any explicit communication channel between them. So it starts looking a bit more like a prisoner's dilemma,

• 52:55Speaker 01

  where you can cross-interrogate each prisoner. They need to have a consistent story, but they can't communicate with each other. And this just seems like a much taller order for AI systems.

• 53:04Nathan Labenz?

  So well done, defense in depth. You think can work for a point, to a point at least. And I think you had an interesting

• 53:15Nathan Labenz?

  observation also, and on this, I think V would very much agree that like, it is important to actually fix the problems, which kind of brings to, you know, you want to get the indication of what the AIs are going to do to try to get you. And then you want to address those, you know, and hopefully in a really deep way since that you can really solve them. Now, an obvious problem is we don't have great strategies for fixing these things. But again, I feel kind of like confused here because as you noted, Claude is pretty nice.

• 53:55Nathan Labenz?

  I guess one like point question is, do you agree with the folks who have been recently arguing that Claude three Opus is like still the most aligned model ever? And do you have a theory of like, why that would be? You know, what, what has, if that is true, why has? Why have we given up ground in the alignment? Department since Claude three Opus? And then I guess more broadly, like,

• 54:23Nathan Labenz?

  what are our prospects? You know, okay, great. The, uh, you know, Claude five and GPT six are scheming. And it looks like they actually might be pretty good at it. Great news. They didn't take over this time. What are you going to do about it? Like, what do we, do we have any? Alignment strategies that you would think would be developed enough in time to solve these things so that we don't? Because the alternative does seem at this point like we probably just say, ah, it wasn't that bad, you know? And, uh, well, just, uh, adjust the system prompt and, you know, add another layer to the defense in depth. And like, that'll probably be good enough, right? Yeah. So do we have anything better than that? I, I don't know what it would be right now.

• 55:10Speaker 01

  Yeah, I don't have a strong opinion on was was Opus 3 the pinnacle of alignment? I think part of a challenge here is that we don't necessarily even have a good operationalization of that question. So I think my intuition, which I wouldn't rely on too much, in an absolute sense, Opus 4 is probably more aligned. However, Opus 4 just has much more capability to be misused, to scheme, to, you know, do all these bad behaviors. So you kind of need to hold it to a higher bar. And it does feel like there was more capabilities progress going from Opus 3 to Opus 4 than there was safety progress, although the guardrails did get a lot better.

• 55:55Speaker 01

  There were very little guardrails in Opus 3. But this is quite hard to quantify. I think a lot of people have this intuition of, okay, safety and capabilities are increasing in parallel, and you want them to stay at least a constant gap. Ideally, you converge, and you really don't want them to diverge. We just don't really know what the scale we should be measuring for capabilities or safety on the Y axis is. And in fact, depending on what scale you choose, you could make lines that would otherwise converge diverge. So it's a bit subjective. And that's one of the things that I find hard to reason about these things. But yeah, to answer this question more broadly of, okay, suppose we do see these warning

• 56:38Speaker 01

  signs, like what do we actually do? I agree with you that currently it doesn't seem like there is enough will to take really significant capabilities or performance hits in order to make a system more reliable and safe. Or maybe some developers are going to do that because they're certainly consumers, big enterprise companies, safety critic industries, that would prefer to have a model that's more reliable. And we're seeing, you know, both different leading developers positioning themselves differently on this trade-off, but also specialized models like RaceOne, Signet emerging that are really optimizing quite hard for security. And these do have use cases, but there's going to be someone who's just like, no, I

• 57:20Speaker 01

  want to turn the knob all the way to capabilities and low latency, and I'm only going to do safety if it's really quite cheap or is actually making the model more useful for a wide variety of users. And I think that's... That's an area where we just don't have a good answer, I would say. I mean, I can point to some things I'd like to see people do more of. You know, even just simply gathering larger, higher quality human training data sets, including on long horizon tasks, including people scrutinizing these models. Like, why do you get more sick of fancy sometimes if you train them for longer? Because probably your human data actually just rewarded that. But if you did, you know, a closer look at it, you'd be able to fix that.

• 58:04Speaker 01

  And I don't want to say these problems are trivial, but they're not. It's very hard to get high-quality data at scale. But this isn't some kind of, you know, unknown problem requiring great research innovation. You've just got to spend a lot of money on it, and you've got to build out a team, and you've got to, you know, spend some time iterating on that and seeing how your model evolves. And obviously, there's some scalable oversight techniques that are beginning to be able to really reduce the cost of that human labor. So there's both been some good work on AI safety value debate coming out of coming out of Google DeepMind and also some work from the UK's AI Security Institute. And then, you know, one one recent piece of work that we did, which has a

• 58:48Speaker 01

  you know, I think both have some good news and bad news, is using scalable oversight to make models less deceptive. And this is, in some ways, deception is kind of the core safety issue to focus on, because if your model's you can just ask it a lot of questions, like, you know, are you trying to take over? Why did you write this PR? And, you know, it might be malicious, but at least you're going to be able to sort of tell pretty upfront that it is. And what we found was that if you do train AI systems against lie detectors, so these are other ML models trained to detect if that system is being deceptive, then

• 59:28Speaker 01

  You can, with a right setup, actually get models to generalize to be much more honest. So the deception rate really went way, way down. And the other good news is that it doesn't seem, although we need to do more investigation to confirm this, that making the models larger necessarily makes this any harder to detect. Whereas making the lie detectors larger can actually make them much more accurate. So it seems preliminarily like the scaling trend is in our favor, although we need to get more data to confirm that, where at least it seems unlikely to be against us.

• 1:00:09Speaker 01

  But the caveat here, and I think this is maybe the crux that people like Xvi and I disagree on, is if you train a system to not exhibit some kind of bad behavior, does it generalize to stop doing that on other instances of it? Or does it be like, "Oh shit, you caught me. I need to be better at scheming." And we did find that under certain training regimes, it matters a lot whether it's on policy or off policy RL. And we have some theoretical insights into why that would be the case. It all matters things like KL regularization, how much you're allowed to explore in policy space. Then you can get instances where a model just learns to fool your lie detector. And so now you've done this sort of safety training, but you've actually made this much, much worse.

• 1:00:53Speaker 01

  And so I think there is a very real reason to be concerned about that. The developers are going to find a problem. They're going to not very carefully patch it. They're just going to rush out of fix, like changing the system prompt or like doing a bit of fine tuning. And the problem is going to appear to disappear, but you've actually made it worse. But based on what we're currently seeing, I would be optimistic that kind of a rigorous engineering approach to this, where you do train away problems you see, but you're careful with your setup. You're careful to have high-quality data. You use some extra evaluation to check that you're not incentivizing the model to just do the same bad thing, but more surreptitiously. You could get quite far. I don't know if you get past human-level intelligence safely with this,

• 1:01:38Speaker 01

  because that's the point where you stop being able to reliably provide human feedback on a lot of this. But I think you can at least get to human level and probably a little bit beyond with scalable oversight.

• 1:01:49Nathan Labenz?

  So maybe just try to echo back a couple of key points there, then I have maybe a couple of questions.

• 1:01:55Speaker 01

  Sure.

• 1:01:59Nathan Labenz?

  Obfuscated reward hacking, I believe, was the name of the paper from OpenAI where they showed this pressure to the chain of thought would initially create higher performance and reduce the reward hacking bad behavior. But too much and the reward hacking comes back, but the chain of thought signal doesn't. And so you're in, in some ways, the worst scenario where now you've lost your observability, but you're still getting the bad behavior. And you're basically saying that, well, that's the result of one training formula. But it does seem that if you do off policy,

• 1:02:42Nathan Labenz?

  uh, which is basically to say like, use human examples, right? As, or at least pre, um, like, uh, sort of human evaluated preference data as opposed to like the GRPO style where the model just gets to kind of do a ton of trials and explore the, the space more extensively on its own. Um, and you anchor to the human prior that was presumably learned in, in some earlier um, post-training step, then you think you can avoid that outcome?

• 1:03:15Speaker 01

  Yeah, I, I think that you, you can, you can avoid it, or at least we, we can develop empirically grounded rules of thumb for safer training formulas. And, you know, over time, test, test it to see how, how reliable this, this really is. But I think the other piece I'd, I'd advocate for is, is having some good held out evaluation criteria. So I think if your training scheme, uh, looks like you optimize a particular measure of success and you look to see that metric just goes up, um, and you've got sort of like no alternative measure of it's going to catch bad behavior, then that's sort of a pretty scary situation to, to be in. Um, you know, e- even putting aside your safety concerns, which is like very, very easy to overfit on that metric.

• 1:04:00Speaker 01

  Whereas, you know, if what you see is I've got a pretty simple light detector, it's like a linear probe on my activations. Or it's a bag of words model, or you know, something that is, it's definitely giving you a meaningful signal. But it's not like fooling this means that there's no way we can detect what the model is doing because we're using a pretty simple, you know, training objective. And then you do a much more rigorous and expensive evaluation and you don't catch any new bad behavior in the AI system. And maybe you can also bust out some interpretability tools and look at like the diff between the checkpoints and actually understand what the model is doing. But if there's ways in which you can get a reasonably high degree of confidence in, you know, how the system is operating.

• 1:04:42Speaker 01

  It's not like sort of perfectly guaranteed, but at least if you're starting from a model that is, is not sort of super deceptive, so sort of like super intelligent at deception. Then you can get a high degree of confidence that your training is not pushing it into that regime. I think none of these methods have a very good answer for what happens if your model just from pre-training is incredibly deceptive and trying to fool you. I think that that's pretty tricky to solve, but there's also sort of strong conceptual reasons to think that pre-training is not going to produce a system without coherent and objective. Even if it might have some latent capability. But, but I think this concern i- is the core one. And so this is actually one of the things that we're trying to, trying to test right now is intentionally train AI systems.

• 1:05:25Speaker 01

  Where none of our methods are able to detect them being deceptive, but they were actually very deceptive. And the, you know, there's one question of just like, can we do this? And I think the answer is, is yes, from, from preliminary results. But then the, the next question is, okay, can we do this in any way that seems a total realistic and like how you would actually train AI systems? And if just every way we can come up with these systems is a very alien kind of training objective. Then I think that we can be, be reasonably confident that at least at current scale, this isn't going to emerge. But it's a sort of check that you'd want to periodically repeat to check that this isn't some kind of scale dependent property.

• 1:06:10Nathan Labenz?

  What about interpretability? This has been a big debate, obviously, over the last couple of years where there's been like very grand hopes for it. I just listened to Neil Nenda on 80,000 Hours, saying that he's kind of scaled back his hopes for interpretability, but still thinks there's a lot of value, just not the maximalist sense of like, we're really gonna, you know, fully understand what's going on and then it will have kind of, you know, guarantee type assurances that things will behave. There was another really interesting paper that you guys put out around digging into the planning mechanisms learned by a model that was playing a puzzle game.

• 1:06:55Nathan Labenz?

  We'll abstract away from the details there for now, but where do you sit right now on how much we can expect to get from interpretability?

• 1:07:05Speaker 01

  Yeah, well, I think it's definitely a core area, like you'd really be kind of tying your hands behind your back if you say, I'm only going to look at the black box behavior of models, even though it's, you know, really trivial to open them up and look at the inside, which is not, it's not going to do that. But I think mechanistic interpretability specifically had this ambition of fully reverse engineering how a system works and being able to give us a human readable, or at least human interrogatable, understanding of systems and tunnels. You could ask questions like, is anywhere in the system that is doing some kind of learn planning computation? And I do think that we've seen enough research here to be really quite skeptical that we're

• 1:07:46Speaker 01

  going to get that simple a sort of reverse engineered artifact. So people have been able to fully reverse engineer some non-trivial neural networks. So we've seen papers reverse engineering particular circuits, doing certain NLP tasks. We've seen papers reverse engineer certain modular arithmetic tasks. Our people are, you know, pretty much fully reverse engineering this planning algorithm. So I think we basically fully understand how it represents the plans and we understand most of a computation that's going into constructing the plan. Except for some heuristics that don't have any kind of long horizon effects. But when you look at these answers, you know, usually what you take away is like, wow, that was kind of a mess.

• 1:08:30Speaker 01

  Like, you know, it is definitely an evolved, organically grown system, but not a sort of planned engineering artifact. And although it's easier to reason about these systems and if you didn't have that mechanistic understanding, it's still pretty hard to get any kind of guarantee because they are a sort of messy, complex system. So, you know, perhaps we have to dial back our ambition there a bit to say, okay, we might be able to reverse engineer systems to the point where we can ask some high level questions like, you know, does it have any circuit that that's doing some kind of human theory of mind? Okay, we might want to understand when it's using that circuit because that's very relevant to deception. You know, if it's only using theory of mind a bunch when it's writing this particular line of code,

• 1:09:14Speaker 01

  maybe we should understand what that line of code is doing, like if it's about a user interface. Okay, it needs to model humans. If it's about a cryptographic algorithm, that's kind of weird and concerning. So there might be some sort of coarse-grained things there. And I think we have seen some, you know, real valuable applications of what I'd call sort of more coarse-grained interpretability to things like simple linear probes, you're training a linear regression model and activations to a system. But, but yeah, I do think that there has been this disconnect between this very foundational reverse engineering style work and then the actual applications are usually using much simpler methods. And I'd love to see some more things meeting in the middle where we say, okay, there's a particular domain

• 1:09:59Speaker 01

  where we think interpretable is going to be really useful. Maybe it's protecting models for the sandbagging. So intentionally holding back certain capabilities. Maybe it's protecting model backdoors. You know, maybe it's protecting

• 1:10:12Speaker 01

  sort of, you know, spurious confounders of a system might have learned during training. And then we can, you know, ablate the ones that we don't, don't want. And then it generalizes better. But kind of like pick your application where you think interpretable is going to be really killer for. And then develop foundational methods that may make progress on that. But I think this sort of more exploratory style of research, it would have worked if there was just like a clean explanation for everything. But at that point, I don't have much hope for we're gonna, we're gonna get that.

• 1:10:39Nathan Labenz?

  Yeah. Do you think that there is some innovation, though, that we could come up with that would make things cleaner? I always remember the Seeing is Believing paper from Max Tegmark's group where basically a sparsity, these are very toy things, but a sparsity term, you know, kind of caused most of the weights to disappear and you'd get like this really kind of crystalline algorithm.

• 1:11:01Speaker 01

  Yeah.

• 1:11:03Nathan Labenz?

  Yeah, maybe Holt's just too messy to handle that?

• 1:11:07Speaker 01

  No, I think it's possible. I have more hopes for training systems to be interpretable than interpreting how systems are currently designed. I think for sure there are going to be some simplifications we can make to current systems. So stuff like sparse order encoders have been quite effective, but the problem is with it, their predictive accuracy is just too low. They're throwing away enough information, but you lose this nice property of, okay, you've actually got this high fidelity reverse engineered interpretation of a system, but could there be some innovation along those lines that's perhaps a bit messier, but you can search over in an automated way and, you know, reduces the complexity enough to work in that space for sure. I just don't think it's going to be this sort of clean picture of, okay, you peel away

• 1:11:50Speaker 01

  some layer of superposition and now everything is just like wonderful because we have understood systems at a very nice support level and they're just messy. But it doesn't mean that you could not train a system to be cleaner or at least to sort of isolate the complexities, but you know, I think it's actually one of a sort of more methodological innovations we came up with in that paper investigating learn planning was saying, okay, we're going to look through all the channels in this neural network and just categorize them into ones that we do need to understand and ones that we don't. Because there were a bunch of channels that were sort of short-term heuristics that we could show only had an impact on like the very next move. It didn't matter what the system was going to take, you know, three steps into the future. I was gonna say, we don't really understand this heuristic channel, but if what we're trying

• 1:12:35Speaker 01

  to do is understand how the system is planning, it doesn't matter, has no impact on long-term planning. And I think there's a lot of cases like that where there might be a huge amount of complexity in these systems, but most of it, like you can just sort of show, is not relevant to a particular kind of question that you're answering. And if you're able to train the system, you know, with an objective to sort of explicitly disentangle, let's say, that like kind of like sensitive channel of information and the less important channel, then it could get a lot easier. But I do think that there's likely to be some kind of performance trade-off for training systems in those ways. So really, it's a question where this becomes a key thing that people are actually willing to optimize for.

• 1:13:11Nathan Labenz?

  Yeah, okay, interesting. How about lightning round? You guys are getting, you know, involved in more different areas of the broader AI safety landscape. There are some field-building projects you're getting involved with, putting on some events, bringing people together, starting to get more interested in or involved in policy advocacy, starting to get involved in international relations. Maybe give us just kind of a quick tap dance through all the activities that you are investing in that we haven't covered and, you know, give

• 1:13:51Nathan Labenz?

  us a little bit of a sense of why you're motivated to do all of these things. And then, at the end, I know you're hiring, we can give you a chance to pitch some open roles.

• 1:14:04Speaker 01

  Yeah, sure, absolutely. Well, I think at a higher level, the reason we're doing all these different activities is that we see there being a potentially quite long pipeline from research innovation to that idea actually being adopted and deployed. And sometimes that means getting companies to adopt it and deploy it at scale and sort of like so good that everyone wants to use it. Other times, that means actually getting some buy-in from, you know, governments perhaps to mandate or nudge companies to do that if it's something that is beneficial to a field as a whole, but might not be in anyone's collective interest. And so if you go from that like idea to deployment, you've got sort of initial research innovation, which we're well-placed to do in how. But then very often you need to have a research field around it like it's not enough just to have a small research team.

• 1:14:48Speaker 01

  So we've done events like the control conference to kind of catalyze and grow new research fields. Now, AI control was, you know, something that the Redwood Research pioneered, not us, we're happy to help out with other fields. But that's sort of like an example of where our events and field building can really come to a forefront. And then I think another missing piece in the the uh the pipeline is very often making really clear, rigorous proof of concept. So not just doing the exploratory research, being the definitive research, which is sort of where I'd highlight our current deception work is. It's trending towards where we had this initial idea for scalable oversight that would solve deception. Um, and it seems to work at a small scale, but now we want to validate it on frontier open weight models.

• 1:15:33Speaker 01

  Get several orders of magnitude of scale and check that it works across all of them. We want to stress test it by trying to train systems that are really deceptive and break current techniques and see how realistic that is. And then if it works, actually say, look, here is, you know, DeepSeek R1 and Lama 45B and all these models trained with our technique. And they're just better than the original model. They're less deceptive, they're less sycophantic, you can trust for more. There's no reason not to just use this model. Um, then get some uptake from AI developers and this is where kind of like our events or more policy arm comes in. And then eventually go to governments and say like, look, this is just, this is just an industry best practice. Like, you know, this is something that you should make at least part of your guidelines.

• 1:16:14Speaker 01

  If you're considering regulation, you know, you should really consider something that isn't saying you have to use this technique. But use a technique that's at least as good as this because there's just no reason not to. Um, and I think what makes us unique is is that we are willing to kind of do the hard thing. Um, and operate across all of those different layers of attack stack from, um, you know, groundbreaking exploratory research to the like, you know, kind of. Nitty gritty engineering scaling work through to more advocacy and and sales. And I think this is sort of not surprising if you're coming from a startup mindset where you do have to often do all of those different things. You need to have a growth team, you need to have an engineering. Team, but for whatever reason, most of these sort of nonprofit organizations have usually said, okay, we're going to pick this niche to be in.

• 1:16:59Speaker 01

  And there's many benefits to that. You can be much more focused as an organization. But the problem is that very often they kind of, they develop something and pass the baton to someone else, but there's no one to pick it up. The baton just gets dropped. And so I think that's one of the big issues with AI safety and what we're trying to resolve. And then to speak to a little bit more about, yeah, some of our other activities. So events, we, we run free alignment workshops a year typically, which is sort of like our flagship event, bringing together technical researchers, AI company, decision makers, people working at government AI security institutes. And this is really sort of like everything under the sun on AI safety. But it's more of a coordination, information sharing. But then we're also increasingly doing more policy focused events.

• 1:17:44Speaker 01

  So we ran technical innovations for AI policy in DC a few months ago. Really bringing together governance researchers, actual policymakers, lawmakers with technical research to say, okay, what could we do that really expands the range of policy windows open? Because I think right now regulators are often faced with a pretty shitty choice between, you know, genuinely maybe holding back innovation and valuable deployments in some areas, or just allowing completely laissez faire, you know, fewer regulations on billion dollar training runs than, you know, opening a sandwich shop in San Francisco. And I think that's just sort of like a false dichotomy. You can, you can have things that are pro innovation and pro safety,

• 1:18:28Speaker 01

  but you do need the right actual technical innovation to drive that forward. So we're trying to catalyze that. And then, yeah, maybe, you know, aware of time, so maybe quickly to touch on our hiring round. I mean, we're hiring a lot. We're planning on doubling in size in the next 12 to 18 months. We've got funding secured for the next few years, including that runway to expand. We're still looking to fundraise in a few areas that our major donors are, you know, a bit more outside of their priority areas, but we've got that room to expand. And so we're hiring across our technical. Team for both IC, ML engineering positions, but also people to lead new research directions. We're hiring operations, probably the most important hiring round we have this year is Chief Operations Officer.

• 1:19:14Speaker 01

  Which is going to elevate my co-founder into a president role and be able to start driving forwards some of the non-technical work that we do. But we're also hiring for more junior positions, people ops generalists, project manager on events. So really, I'd say like, you know, if you're excited to work in this space and you like the kind of work we're doing. Then do check out our career page and even if you don't see a job that's a good fit for you right now. We have just a general expression of interest or encourage just sort of like talented people who are excited by us to to fill out. And then we'll get in touch when a role does open up.

• 1:19:48Nathan Labenz?

  Love it. It's true. You there that as you described all that, you're kind of going vertically integrated, right? You're going all the way from research to, in theory, you know, handholding people into actually putting these things into production systems.

• 1:20:04Nathan Labenz?

  This is maybe more of my hobby horse question than something you really want to do. But I've been really interested in this idea of private governance. And I wonder if society might draft you and the far AI team into becoming one of these private sector. Quasi regulatory bodies, it would seem like you have the range of capabilities needed to do it. But would you be excited to do it if that were actually the law and people were looking for orgs to to fill that niche?

• 1:20:38Speaker 01

  Yeah, I think you're totally right. We have this skill set and sort of org structure to do that. I'd say that, you know, this isn't part of our mainline plan, you know, perhaps because it does still feel pretty nascent. This idea of private regulatory organizations, but it's a very important thing to do. So if this was something where we didn't feel like there were, you know, other kind of companies stepping up to fill this gap. And we were well placed to do it, but it's something we'd be very interested in. I do think it certainly has some trade offs and. That like, you know, right now, we we're fortunate to be on good terms with both the leading AI companies and governments. And that's basically because we don't actually have any hard power.

• 1:21:20Speaker 01

  Like, we can make recommendations and brief people, but we're not overseeing anyone. And we're going to lose some of that independent ability to convene people to make recommendations if we're actually exercising an oversight role. But I think it could be worth it. We're also, we're not aware to sort of like everything being under one organization. So we could spin out either that kind of private sector regulatory part. Or we could spin out some of our activities that require more independence, like our events into a separate organization. And then just kind of avoid that conflict of interest. So, yeah, we're open to it, but also love to have more competitors in this space so that we don't have to do so many different things.

• 1:21:59Nathan Labenz?

  Yeah, you are spinning a lot of plates right now. So, um, yeah, thank you for the generosity with your time and perspectives today. And people should definitely check out the open roles COO and otherwise, Adam Gleave, co-founder of Far AI. Thank you for being part of the cognitive revolution.

• 1:22:18Speaker 01

  Well, thanks for having me back, Nathan. Great to see you again.

• 1:22:22Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 1:23:07Nathan Labenz?

  And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts, so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.