Historic AI Developments & the Emerging Shape of Superintelligence, from the Consistently Candid Podcast

In this episode, we discuss the significant advancements and challenges in the field of artificial intelligence over the past year. From breakthroughs in reinforcement learning to unexpected behaviors in fine-tuned models, we cover a wide range of topics that are shaping the AI landscape. Key discussions include the rapid progress of reasoning models, the surprising results from alignment studies, and the potential for AI to become superintelligent. We also explore the emerging importance of AI memory and its implications for future applications. Tune in for a deep dive into the current state and future trajectory of AI technology. SPONSORS: SafeBase: SafeBase is the leading trust-centered platform for enterprise security. Streamline workflows, automate questionnaire responses, and integrate with tools like Slack and Salesforce to eliminate friction in the review process. With rich analytics and customizable settings, SafeBase scales to complex use cases while showcasing security's impact on deal acceleration. Trusted by companies like OpenAI, SafeBase ensures value in just 16 days post-launch. Learn more at https://safebase.io/podcast Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers. OCI powers industry leaders like Vodafone and Thomson Reuters with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before March 31, 2024 at https://oracle.com/cognitive Shopify: Shopify is revolutionizing online selling with its market-leading checkout system and robust API ecosystem. Its exclusive library of cutting-edge AI apps empowers e-commerce businesses to thrive in a competitive market. Cognitive Revolution listeners can try Shopify for just $1 per month at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive RECOMMENDED PODCAST: Second Opinion. Join Christina Farr, Ash Zenooz and Luba Greenwood as they bring influential entrepreneurs, experts and investors into the ring for candid conversations at the frontlines of healthcare and digital health every week. Spotify: https://open.spotify.com/show/0A8NwQE976s32zdBbZw6bv Apple: https://podcasts.apple.com/us/podcast/second-opinion-with-christina-farr-ash-zenooz-md-luba/id1759267211 YouTube: https://www.youtube.com/@SecondOpinionwithChristinaFarr PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I sort of suspect that in the like zoomed out history, this might appear to be like a critical threshold. Everybody was kind of scaling these base models. Then somebody figured out that you could also scale inference compute. And then it became clear that it's actually pretty easy to do that. And then what's that going to produce? You know, it seems to me that it's likely to produce a lot of kind of weird AIs. Because reinforcement learning also famously gives rise to strange behavior. It just seems like governance gets a lot harder. In this world where distributed training works and where the post-training that really shapes the AI's behavior and like practical utility and how they're going to show up in the world is actually become like quite cheap.

• 0:44Speaker 00

  A race to powerful AGI between the US and China is one of the worst situations I can imagine that could lead to catastrophic outcome. I would say math and coding in particular are like

• 0:54Speaker 00

  almost undoubtedly going to hit superhuman levels in the next probably 2025, you know, certainly seems like by 2026.

• 1:03Nathan Labenz?

  Hello and welcome back to the cognitive revolution. Today, I'm pleased to share a cross post of my appearance on Consistently Candid with host Sarah Hastings Woodhouse. This was my second episode with Sarah. Nine months ago, she was just getting into AI and still making sense of the fundamentals. Today, as you'll hear, she's developed a strong sense for which AI stories really matter and also does an excellent job of summarizing notable research results. Together, we unpack a number of stories that I believe history will judge to be among the most important of the last nine months. We start with the recent revelation that reinforcement learning can be relatively easily applied to sufficiently powerful base language models and the reasoning capabilities that this has unlocked.

• 1:48Nathan Labenz?

  We then move on to consider the rise of distributed training, which especially as combined with the inference heavy nature of reinforcement learning, makes it possible for all sorts of moderately resourced organizations and distributed groups to apply reinforcement learning to any objective they might like.

• 2:05Nathan Labenz?

  We also discussed the shift in rhetoric among American AI leaders toward embracing an AI arms race with China and get into a couple important recent AI alignment results, including the alignment faking paper that we covered in depth in our episode with Ryan Greenblatt, and also the very viral emergent misalignment paper from a Lion Evans Group, to which I made a minor contribution and on which I was honored to be included as a co-author. Perhaps most interesting for regular listeners, for the first time publicly, I offer a sketch of the form that I expect early superintelligence to take, in the base case, over the next few years. The upside is that AI systems' ability to develop intuitive physics across many different problem spaces, like material science, protein folding, cell biology, and many, many more,

• 2:49Nathan Labenz?

  especially as combined with reasoning abilities, suggests a pretty clear path to an exponentially growing number of eureka moments from AI systems, which really could accelerate science to the point that we achieve a century's worth of progress in just the next few years. At the same time, on the downside, our still nascent understanding of how these systems work, the rate at which we continue to be surprised by their outputs, and the growing body of evidence suggesting that frontier models are increasingly willing to deceive and otherwise scheme against their human users to achieve their own goals and protect their own values, all suggest that we will see lots of instances of bad behavior and will need to invest heavily in control measures along the way. This vision of superintelligence and also the vision of drop-in AI knowledge workers

• 3:34Nathan Labenz?

  that I sketch out toward the end are inherently more forward-looking and speculative than my usual material. And as such, I really want your feedback on this episode in particular. Thanks to your consistent engagement with the feed, I'm getting more and more invitations to speak to business and general audiences about AI. And this picture of superintelligence, which I hope makes the potential for frontier discovery tangible and the reality of bad behavior plain, is becoming a consistent framing for my talks. So, what do you think? Am I missing something? Am I overstating any of the achievements or the possible benefits? Or perhaps understating any of the demonstrated issues? I really want to make sure that I'm sharing the most accurate and up-to-date understanding that I possibly can.

• 4:17Nathan Labenz?

  So please do let me know if you think I'm getting anything wrong. With that, I hope you enjoy this review of the most important AI stories of recent months and this possible preview of the future of AI transformation. From the Consistently Candid Podcast with Sarah Hastings Woodhouse.

• 4:34Speaker 02

  Welcome back to Consistently Candid. I am back with Nathan Labenz. We last spoke maybe 10 months ago. I'm not sure. It's been a while. And a lot has happened since then. So I wanted to get your take on like the state of play and recap some of the biggest AI things that have happened in the last year or so. So maybe if you just tell me since the last time we spoke, what have been your kind of biggest positive and negative updates? And are you feeling better or worse about AI than you were about 10 months ago?

• 5:06Speaker 00

  Yeah, last time I gave the case for cautious optimism.

• 5:10Speaker 02

  Yeah.

• 5:10Speaker 00

  That's how I kind of summarized it.

• 5:14Speaker 00

  Certainly, a lot has gone on.

• 5:19Speaker 00

  You know, if I try to zoom out and and just abstract away from a lot of the details, I think one of the biggest developments has been the recent

• 5:34Speaker 00

  revelation that reinforcement learning on top of at least like sufficiently powerful base language models really works and is actually a pretty can be like a pretty simple setup that works like remarkably well. And certainly is like, I think, been proven, you know, by a lot of different groups at this point.

• 5:58Speaker 00

  With like OpenAI sort of, you know, giving like a existence proof that such a thing was possible. And then DeepSeek and Kimi out of China giving like specific implementation details, showing how they did it. And then like lots of, you know, just other academic groups and other groups kind of doing their own versions. It does seem that, at least for like, you know, decent class of reasoning problems, your sort of, you know, math problems canonically and programming challenge type things where there's easy to verify answers.

• 6:35Speaker 00

  At least in those domains, it seems like you can push a sufficiently strong base model quite far down this reasoning path without needing a super complicated setup, without needing like a ton of inference compute. And and even, you know, without even needing like a super specific formula, like multiple different recipes or, you know, strategies for doing this all seem to be working. And

• 7:02Speaker 00

  I suspect that that's going to also translate to a lot of other things. So I think we're just kind of entering this era. How well it translates and how well it translates to things where there's not a, you know, clear ground truth, or you know, where there's where it's beauties in the eye of the beholder, so to speak, is going to be interesting to watch. But I do think there are a lot of things that you can create a good enough metric for, to give a pretty good reward signal to a model. And I expect that we're just going to see like a lot of that applied increasingly sort of in the wild. You know, it's even the kind of thing that like communities can do without needing, you know, huge, huge compute resources to do it.

• 7:46Speaker 00

  And so I sort of suspect that in the like zoomed out history, this might appear to be like a critical threshold. You know, everybody was kind of scaling these base models. Then somebody figured out that you could also scale inference compute. And then it became clear that it's actually pretty easy to do that. And that a lot of different groups, organizations, companies can do it without totally, you know, having to go all in financially or break the bank. And then what's that going to produce? You know, it seems to me that it's likely to produce a lot of kind of weird AIs. Because reinforcement learning also famously gives rise to strange behavior. You know, it gives rise to phenomena like reward hacking.

• 8:28Speaker 00

  If you don't get your reward signal just right, then you have the chance of, you know, getting reward hacked. And it also just gives rise to like inscrutable behaviors. You know, we had seen specifically DeepSeek reported that like, they trained a model purely with reinforcement learning and it was acting super weird. And it's a chain of thought. I haven't observed that personally, actually, to be clear. I'm not sure quite how weird it was, but they reported language switching. And we have seen that from other models. I've also seen that from GROK three in the last couple of weeks or last couple of days. So, you know, just how weird do they behave? How inscrutable do they become? And how much do they surprise us? You know, it's always been reinforcement learning that has taken things into the superhuman.

• 9:15Speaker 00

  Level of performance, classically with like game playing type things, right? Like you could train a Go playing AI on human games of Go. And you'd probably kind of top out around the human level of Go playing. It was when they had them playing each other and just rewarding them for winning. That all of a sudden, you know, you have these systems kind of zoom past a human level. So reinforcement learning, self-play, the weirdness that can come of that. And the potentially surprising and likely in many domains, superhuman capabilities that we're going to see out of that paradigm seem like, you know, the big story

• 9:57Speaker 00

  that, you know, that might kind of prove to be like a, you know, a notable hinge moment in history. That's probably my number one candidate from the last year.

• 10:09Speaker 02

  Okay. So yeah, so what this implies that it might just be pretty or a lot easier and a lot cheaper than we thought to train these like quite powerful general purpose models.

• 10:22Speaker 00

  Yeah. And to shape them in post-training in, you know, any direction that you can give a reliable reward signal for. I think a lot has been made, you know, in terms of how cheap the base models are. I mean, they're extremely cheap if you just download an open source one off the internet. So, you know, that phenomenon can, has continued, right? Like DeepSeek put their models out for all to download and use however they want. So it doesn't get any cheaper than that. I think it's been a little bit overstated, maybe moderately overstated in some quarters. You know, how cheap the DeepSeek model was. That's been like pretty well chewed on by the community at this point. And like the sort of $6 million model or whatever. You know, that's, um, even in their own accounting, that's just like, just the inference.

• 11:06Speaker 00

  Or sorry, not inference, just the compute used for like the single training run, you know, to create that model once they'd already done the experiments, you know, not counting all the salaries, not counting the fixed cost investments, so on and so forth. So I think it's still safe to say that like, it's not cheap to create a frontier base model and probably only going to get still more expensive. Uh, you know, people are. The leaders, I would say, would not say that we've like moved from one scaling paradigm to another, but rather that we're now stacking scaling paradigms on top of one another. So they're both going to do like bigger base models, you know, more at least higher compute base models and you know, this sort of post-training. What I expect will probably happen more in the wild is is not so much that people, although

• 11:51Speaker 00

  distributed training also is like another interesting candidate for sort of a big story in the in the macro view because it's until now always been kind of thought that like, well, you've got to have these like giant data centers to do this compute, you know, you there's no other way to do it. They've got to have like super high interconnect and you know, without these sort of very specialized setups in in very like concentrated, you know, physical locations that you just couldn't do it. And I think that is also pretty well now changed by the developments in distributed training where basically people have figured out ways to just dramatically reduce the bandwidth overhead that is required.

• 12:34Speaker 00

  Simple, you know, why is this hard in the first place? You've got these giant models with these, you know, billions of parameters. If you're running them in faraway places and you're doing the training and you're collecting the gradients, which is, you know, the updates to the model at each training step, you also need to sync those gradients and those gradients are as big as the model itself because you have like, you know, if you have in the deep-sea case, you have a 671 billion parameter model, the gradient itself in at least in like naive form is also 671 billion numbers. It's the 671 billion changes you're going to make through the 671 billion parameters. So if you calculate some changes over here and some changes over there and these like,

• 13:14Speaker 00

  you know, disconnected compute environments, then you have the question of like, well, how do I sync those up and combine them so I can make like one coherent update to my model? And that means you've got to like send, you know, hundreds of gigabytes. Across, do this syncing and then also redistribute, you know, hundreds of gigabytes back, you know, out to the places where you're running the compute. So that's been like thought to be really hard and that's why we need this like, you know, super high interconnect and that's even why like export controls were at times focused on specifically this sort of connectivity piece. And that the thinking at one point in time was like, well, we can let China have the, you know, the, the raw compute, they can do the inference.

• 13:56Speaker 00

  That's okay. We're not trying to, you know, stop their economy, but we just don't want them to like train these like super crazy, you know, models that, and so maybe we can kind of have the best of both world, like allow them to use stuff, but not, you know, train at the frontier. And now it seems like, you know, basically people have got clever about that. There are, you know, ways to sort of shortcut or stream. One of the latest publications out of Google is called streaming deloco, um, which is like distributed location, something, training, whatever. I don't know what the logo exactly stands for, but basically that they can now, um, you know, with a smarter approach than just like doing the update and having to wait for hundreds of gigabytes to travel.

• 14:37Speaker 00

  They can now like stream these updates in a smart way, which basically means both you can do the distributed training and also that like all these chips that we sold to China in the last, in the last couple of years, uh, even if they had these like reduced interconnect properties, uh, that's probably still going to work just fine, you know, for training because they'll be able to, you know, follow publications out of Google and use of course their own, you know, engineering prowess, which is proven to be very substantial to like work their way around that. So it is still take, it does still take a lot of compute to train of a powerful base model. It's not the kind of thing that just like, you know, hobbyists can do. It is the kind of thing now that a well organized but distributed group could potentially

• 15:22Speaker 00

  patch together the resources to do, probably not at like the real scale that the, you know, the Western leaders are going to push as they go into hundreds of millions and potentially billions for single training runs. But definitely still enough to like get to the, the place where they could in theory do a base model, uh, that like today's frontier in a distributed way. And

• 15:46Speaker 00

  You know, then just going back to the post-training behavior like that stuff is like really quite cheap and especially in these distributed paradigms like really anyone, you know, can can kind of post-train in a reinforcement learning way. Anyone's maybe a little bit too too permissive of a declaration there, but like you don't have to be a super well-resourced group to muster the kinds of compute that are used to do these post-training things. And that does, you know, unlock a lot. Then at inference time, you have another question of like, okay, well, maybe train this thing, but you actually have the capability to use it, you know? And that's another area where different, you know, differently resourced actors have like very different propositions.

• 16:28Speaker 00

  Meta is talking about, I think, what a $200 billion data center was the last thing that they floated and was got the Orion project. That's $500 billion. And Apple's talking about $500 billion more they're going to spend in the US to, you know, build more stuff here and whatever. So like, they believe that we are all going to want intelligence always on all the time. And there's almost can be like, in their minds, like no limit the amount of inference compute that's going to need to be spent. Distributed groups, you know, presumably will like come together to do their training and then, you know, it'll kind of be on the individual actors themselves, whether it's individuals, small companies, whatever, to have their own inference infrastructure or to, you know, rent it out of the cloud. And then in China, they're going to have kind of, as the restrictions

• 17:13Speaker 00

  get tighter, as we realize like, geez, you know, and I don't necessarily support all these restrictions, by the way, but the thinking is like,

• 17:21Speaker 00

  damn, you know, our original plan didn't work. We got to get even tighter on this. And so that actually might constrain their ability to like diffuse the technology through their society, even if it doesn't necessarily restrict them so much that they can't like do frontier work. It might mean that like, just, you know, rank and file Chinese businesses and Chinese users maybe have like a relatively AI-scarce environment compared to what it seems like we're headed for. So. I guess bottom line there is, you know, distributed training is another decent candidate because it really kind of shuffles the way the the power dynamics were thought to be settling

• 18:02Speaker 00

  around who can do what. Doesn't entirely change them because compute is still really important, but it does show that like you don't need to necessarily have a trillion dollar data center to, you know, train the mega AI. And that also means like if you wanted to, you know, you're playing out like war game type scenarios. There's not going to be like one trillion dollar data center that you could like disable and sort of disable the other side's capacity. It's going to be probably more like, you know, I don't know, 50, 20 million dollar data centers or something. And, you know, can you really take out like 50 different locations? Well, that's probably World War III. You know, and if you take like one or two offline with some sabotage

• 18:43Speaker 00

  of or wherever, you're not like moving the needle that much. It just seems like governance gets a lot harder in this world where distributed training works and where the post-training that really shapes the AI's behavior and like practical utility and how they're going to show up in the world is actually become like quite cheap.

• 19:02Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:08Speaker 02

  Yeah, so I guess I saw a lot of different takes floating around in the aftermath of the deep-seak thing, sort of some people making the case that this means that export controls don't work. And then other people making the case that this in fact strengthens the case for export controls because maybe the controls that were already in place hadn't quite bitten when deep-seak was trained. And in fact, I think, yeah, Darian Amade made this point on the China Talk podcast a few weeks ago. He was like, we just need to go really, really hard on this and maintain this strong US lead over China that will kind of give us from leeway to slow down if we need to, etc. Yeah. So do you agree with this this claim that like this this only reinforces the need to go really hard on export controls?

• 19:54Speaker 00

  There are parts of that analysis that I think are valid, but I'll first zoom out and say maybe a third candidate for the big story of the last year is that the US AI leaders seem to have done, well, two of three, if I, if I wanted, maybe I'll say Sam Altman, Dario and Demis, you know, I'll consider to be the sort of three, you know, most influential AI leaders for obvious reasons. Two of the three have done a pretty remarkable flip relative to their earlier positions where you can go find video on the internet of both Dario and Sam saying things like, "A race to powerful AGI between the US and China is one of the

• 20:38Speaker 00

  worst situations I can imagine that could lead to catastrophic outcome." That's like a very close paraphrase of Dario 2017. And Sam, I think 2023 was like, you know, "People are way overconfident about China. Like, I don't, I don't know about China. They don't know about China. We should really think about what we ought to be doing and not base our, you know, decisions on what China's gonna do." And now they have both, you know, as you said, like totally flipped on that. Sam has put op-eds out saying, "It's either their values or our values in the AI. There's no third way," which I don't like that framing. And I also don't like the Dario framing where he basically says, "Yeah, we gotta keep them down, you know, build our lead and then, you know, build a national alliance. And then

• 21:22Speaker 00

  one day we can come around to them and, you know, make them an offer. And at that point, you know, they'll be so, they'll, it'll be so clear that they're like so outclassed by our, you know, AI prowess in part because we'll have like denied them the chips to do the work that, you know, they'll just have to give up competing with democracies." I think that's a direct quote from, from one of his essays. And then, you know, "Then we'll be nice and we'll give them, you know, the benefit of our advanced development." And I don't like that stuff at all. Demis, to his credit, in my view, is taking a very different tone recently on like calling for-

• 21:54Speaker 02

  Yeah, and he's doing the sun thing.

• 21:57Speaker 00

  Yeah, which I think is great.

• 21:59Speaker 02

  Yeah, same.

• 22:00Speaker 00

  I think-

• 22:03Speaker 00

  You know, I'm starting my own journey on this US-China question. I don't, I'm not an expert. I don't, you know, I've never been to China. I don't speak any Chinese. I really don't know a lot about it. But I kind of believe Dario 2017, you know, and I believe Sam Altman 2023. And I'm like, if this is as powerful as you, you know, say it's going to be, then I don't know why they couldn't have just said, you know, like, why does Dario have to come out and openly call for a, you know, a race?

• 22:35Speaker 02

  I guess I, well, the host, I don't know if you listened to this episode of him on China Talk, but yeah, I guess like the steel man of his case was like, well, if you're trying to maintain a lead over your adversary, like one thing you could do is just like massively accelerate yourself. And the other thing you could do is like, try to hold them back. And I guess he was saying that it's way worse to be in a sort of neck and neck race, because then everyone has to sort of corner cut on safety. Whereas if you're in a race with quite a wide differential, then that grants you some, like, yeah, some leeway to slow down yourself if you want to. Like, if you have a year's lead over China, maybe you can do a little bit more safety work than if you had like a month's lead over China. And I was like,

• 23:15Speaker 02

  well, this stuff kind of makes sense. But then the part where he lost me was when he was like, well, you know, if we had really compelling evidence of the risk, maybe we could all agree to slow down. But then he kind of passed the buck to the safety community and was like, these people need to find the compelling evidence. And, you know, if they really want us to slow down, they need to demonstrate this. And I was thinking, but don't you have way better access, access to the evidence than anyone else?

• 23:41Speaker 00

  Yeah.

• 23:42Speaker 02

  Being the CEO of a frontier company. So, I don't know. I thought the first part of the argument, I guess I couldn't really find fault with. And then the second part, I was like, it does seem like if you, if he really, really wanted to make a compelling case for this being risky, he probably could try a bit harder. But yeah.

• 24:00Speaker 00

  Yeah. Well, I think, actually, to give credit where I think it is due, I think Anthropic continues to do a lot of work and, you know, some of the best work in terms of. Really engaging with the questions of like, what could go wrong? And like, how are these AIs, you know, potentially going to surprise us and so on and so forth? I recently did an episode of the podcast with Ryan Greenblatt, who was the lead author of the alignment faking paper. And he doesn't even work at Anthropic. He works at Redwood Research, but came up with this, you know, idea and showed some initial results just by prompting the model and then went to Anthropic and said like, hey, you know, are you interested in this? And could we do a deeper collaboration? And they did.

• 24:42Speaker 00

  And we've got a lot of, I think, great stuff like that going on. They put out work about how training models on documents that describe reward hacking seems to induce reward hacking. They continue to do a lot of stuff on interpretability. Their model cards and their like risk assessments, you know, continue to be the best.

• 25:03Speaker 02

  That doesn't imply that we'll just like delete one of the AI century literature off the internet or something.

• 25:08Speaker 00

  Well, at a minimum, there is probably a bit of literature to be done. Yeah.

• 25:12Speaker 02

  Yeah.

• 25:13Speaker 00

  Yeah, I mean, deleting it, it's going to be tough. But yeah, I do think that like, there's a growing gap between, you know, Anthropic's doing a lot of good stuff. OpenAI, I think also, and DeepMind 2 are like all doing a lot of good stuff. DeepMind just put out a paper about strategies to try to avoid reward hacking in reinforcement learning. And that strikes me as like, you know, some of the most important recent research. So I do think Anthropic in particular, and you know, all three of the leaders are like making a pretty good effort here to really understand the nature of what they're creating. I do think this, you know, the geopolitical strategy that he's articulating, I feel like is really trying to thread a needle.

• 25:55Speaker 00

  And I just have so much uncertainty as to, and I think he should have like a little more humility about just how well, you know, how like the conjunction fallacy seems to be like very operative there potentially. We're going to do this and this and this and this and then this and then finally that. And then, you know, we get to a good outcome. And I would love to see, especially somebody in his position. Preserve a little more option value because, you know, even if he, I don't doubt that he's being sincere, but even believing everything that he's saying, like, these policies are already happening. So in terms of what role he should play, if I were his speechwriter, or if I had been asked to comment on a couple of these essays, I would have said, you know,

• 26:38Speaker 00

  why don't you stay out of geopolitics for at least now? You know, why don't you say, hey,

• 26:45Speaker 00

  I don't know, you know, what our overall strategies toward China should be like, that's above my pay grade. I'm focused on developing frontier AI and doing it safely. And what I can say is we've got a lot of open questions and some really kind of concerning initial results about where this is all going, such that at some point we might really need to get on the same page with China. And, you know, I just hope we at least maintain some ability to do that if and when it becomes necessary. He could have said that. The chip restrictions would have gone ahead. You know, I don't think they would have reversed them on, you know, such a mild statement from Dario. And he would have, I think, preserved a lot more option value to come in later and say, remember when I said this before? Like, it's, it's really happening.

• 27:30Speaker 00

  As it is, it seems to me that we're projecting to China that like our intention toward them is potentially regime change. And, you know, it's not like we haven't acted on that in the past. So I think they kind of have to take it seriously. And it does seem like it makes, you know, any sort of trust building exercise a lot harder. Also worry, by the way, that, and I don't, this has not happened yet. So this is, I would still put this in the speculative category. I do not count like DeepSeek R1 as an example of this. But I do worry a bit about divergence of the tech trees. And I don't know how likely this is, but I do think like,

• 28:10Speaker 00

  You know, the more common foundations we're each building on, the more ability we have to say, "Hey, we're seeing this. Are you seeing this?" Or like, "Maybe you should look in your, you know, models for the same kind of thing we're seeing here, because this looks like quite problematic." And if we're working with basically the same technology on both sides of this divide, then like those sort of, you know, purely like, "Hey, you might want to watch out for this sort of messages." Could be received, you know, could be apt, like could lead people in the right direction, and maybe could, you know, bring, bring people together at some point. If however, you know, building on fundamentally different chips, leading to potentially different architectures, you know, leading to different trading strategies, like leading to, you know,

• 28:53Speaker 00

  just maybe some kind of more general divergence, then it's like, they're not going to see the same stuff that we're going to see, or like the way that we're looking for it, you know, isn't going to apply as much. And it's also just going to definitely reduce the ability to trust, right? Because again, these things are not like, um, you know, missile silos you can see from space. We're going to see a lot of data centers in China. They're going to see a lot of data centers here. It's not going to be clear like in what data centers, you know, what activity is happening, or what breakthroughs, you know, have been achieved algorithmically, or like what capabilities or, you know, behaviors have been observed. So the, I think it is nice. I don't know, again, you know, that this will happen, but part of me is like,

• 29:38Speaker 00

  if we want to keep the Chinese AI ecosystem down, like one way to do it would be to sort of use like an export dumping strategy in a somewhat similar way that they've, you know, uh, used like, they've done like to the west with like solar panels, right? Like they have subsidized the development of the solar panel industry. They like sell them super cheap. You know, our companies can't compete. That might be somewhat of a simplification, but that's my general understanding. And I would say like, maybe we should do that, you know, to them with AI. Like, give them as much like free AI as they can, so there's like less demand and less sort of, less of a feedback cycle in their own ecosystem. But the, you know, but their people can still get the benefit of all the stuff that we're doing. And anybody who wants to like open source or tinker or whatever,

• 30:23Speaker 00

  you know, they'll also be kind of using the same foundation that we're building on.

• 30:29Speaker 00

  You know, I guess we'll never know how that would work, because I don't think we're going to run that experiment. And we might, you know, I hope we don't, but. I am a little worried that we might end up in a spot where the technology diverges so much that it becomes and it becomes more secret, right? Over time, too. So it's like, we can't tell you exactly what we're doing. We don't even know what you're doing. How are we going to have any sort of, you know, meeting of the minds if that,

• 30:56Speaker 00

  you know, Gulf really gets too wide?

• 31:00Speaker 00

  So yeah, there's a third candidate for like biggest story.

• 31:04Speaker 02

  I always think that I've like found all of the things to worry about and that that's something gives you anything to worry about. That was a good point.

• 31:12Speaker 00

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 31:17Speaker 02

  Yeah, okay, maybe we should like, uh, just run through the, what I think like the biggest stories of the last year or so, and you can give me your take on them or maybe try to explain them in sort of like layman's terms. So let me think. Well, I guess the first thing would just be, uh, this sort of switch into the reasoning paradigm with first O one and then O three and whatever comes next. Can you explain in like the simplest way possible? What is the difference between a model like GPT four and a model like O one? And what does this difference sort of imply for AI progress in general?

• 32:00Speaker 00

  I think, I think there probably ultimately is sort of a spectrum here.

• 32:08Speaker 00

  And maybe a sort of also potential important role for certain threshold effects. But, you know, one of the, even prior to the reasoning models coming online, one of the biggest observations that has driven a lot of the performance in practical terms of the AIs over the last couple of years, basically since GPT. Three, I would say, is the chain of thought. You know, and this, a couple of years ago, you could like put out an academic paper by literally just like taking a prompt and then putting, let's think step by step, you know, at the end of the prompt, and then testing that against like, you know, some like minor variations of those instructions. And, you know, let's think step by step, like perform, you know, 1% better than this

• 32:53Speaker 00

  other rephrase or like, that was a paper. But that was a really powerful observation and people got, you know, a lot of improved reliability and improved consistency of behavior from models just by like applying that paradigm, you know, even to the, you know, the models as they previously were. It became sort of the default behavior too, by the time of like GPT-4, it would mostly give you a chain of thought, even if you didn't ask for it. And actually I saw a lot of research initially underestimating how powerful GPT-4 was because people would prompt it in a way that would prevent it from doing the chain of thought in the way that it normally would. So I've been on my own like micro crusade to go, you know, go like educate people about

• 33:37Speaker 00

  the fact that like establishing your own chain of thought and teaching the model to do the chain of thought in the way that you want it to be done is one of the biggest practical unlocks for just AI automation, application development, you know, you name it. A lot of people don't have that. So that's actually kind of challenging for a lot of people. They often have inputs and outputs, but they've got no documentation of the chain of thought that they used, because it's just in their head, to translate those inputs to outputs. So developing the sort of ability to get that down on paper is if you're in part of a, you know, in a team context, like get together as a team and have a meeting of the minds and agree like, yes, this is how we want this task to be done.

• 34:20Speaker 00

  Not just that the outputs are good, but like that the way in which we're showing the AI how to think about it is the right way that we want, you know, somebody to think about this. That's, you know, been a lot of work at the implementation level and that work, you know, continues on. And basically what they did with the reasoning models is they just like took that to another level.

• 34:43Speaker 00

  Why?

• 34:45Speaker 00

  It has taken this long, actually, is kind of a mystery. And I don't think we have the final answer on this yet. In our one paper, DeepSeek said that they tried their reinforcement learning paradigm on smaller models and it didn't work. And so that sort of suggested that maybe you need sufficiently large or sufficiently powerful base models in order to be able to do this sort of, you know, reasoning training process. And the thinking there would be something like, you know, the, the pre-training process is next token prediction. And so, you know, what are you going to learn to predict the next token of like vast scale internet data?

• 35:29Speaker 00

  You know, you're going to learn, and I did my, one of my favorite episodes, actually, that I ever did on the, on the cognitive revolution was with the authors of a paper called Tiny Stories out of Microsoft where they basically just actually used GPT-4 to create these like stories that were sort of three-year-old, you know, simplicity, three-year-old vocabulary level. And then they train the language models just on those. And they sort of observed like, what is the natural progression of learning for a language model? And it's like, you know, first of all, just getting like basic syntax correct, like getting the right part of speech next, you know, is kind of one of the big things you got to do if you want to be an accurate next token predictor.

• 36:12Speaker 00

  And then it's like, you know, we're calling previous tokens because like, once a token appears once, it's more likely to appear again. So just like learning that like, you know, whatever token came before, you know, is I should be like, you know, potentially upweighting that. And in their little study, they basically got as far as just like the very beginning of reasoning micro skills. One of the examples that they showed was like, you know, late in their training process. And this was all small scale stuff, but you know, after the syntax and after like, you know, vocabulary and after, you know, sort of repetition, eventually they got to negation. And they, you know, showed an example where it was like, um, you know, whatever Sally

• 36:56Speaker 00

  didn't like, uh, Jim made soup, but Sally didn't like soup. So Jim gave Sally blank. And, you know, at some point it would just say soup again, because you already had soup twice. And like, when things appear, they tend to reappear. So, you know, that's kind of the.

• 37:15Speaker 00

  Naive next token prediction. You have to understand this negation concept to realize that it's got to be something else besides soup. Right. It's like, in fact, soup is like the least likely thing that should appear there. But you need to have this like micro skill, um, kind of grocked in order to, you know, understand that situation, make the right next prediction. And, you know, the behaviors that we're seeing out of these advanced reasoning models now are like double checking.

• 37:43Speaker 00

  You know, going back to the beginning and trying a different approach. It's this like very higher order, uh, behavior that A like doesn't necessarily appear a ton on the web. Because people are not in the habit of like live journaling, you know, their problem solving processes at this level of detail. And also just as like very high up the hierarchy of things that you need to be an effective next token predictor. Right. You gotta, you gotta learn an awful lot of stuff like all this world knowledge, all this stuff. Before you finally start to have these, you know, very high order, um, like macro reasoning behaviors. So possibly they weren't in base models until recently.

• 38:28Speaker 00

  Uh, possibly they like weren't in the training data. Another thing that is happening is people are synthesizing a lot more data and putting that out there. So also like it is showing up on the web more today. But I don't think this has been like fully answered. It has been shown that once you, I guess a couple of things have been shown. One is the reasoning traces just naturally get longer. If you give the model a reinforcement signal, like you were correct or you're not correct. Can be as simple as that. That was one of the things from the deep sea paper. It's so simple. There's nothing like sophisticated. It's just like, did you get it right or did you get it wrong? That is the signal. So.

• 39:06Speaker 00

  With that simple signal, the reasoning traces just naturally expand. And that seems to be sort of, you know, just a natural attractor. Like, given the opportunity, given this sort of signal, they will learn to think longer on their own. And then what patterns of thinking come out of that? Maybe you needed a particularly powerful base model to have some hope of like ever seeing that behavior so that you could reward it. There's a big problem in reinforcement learning, generally the sparse reward problem, which is like, if you can't make any traction on the problem at all, then there's nothing to reward. And so there's, you know, the model can't like update in the right direction because it's not getting any signal about what the right direction is.

• 39:47Speaker 00

  But if you can just get over into that hump a little bit and you can start to get some signal, then you can like, you know, kind of catch the wave. And so, you know, that seems to have happened. With these advanced reasoning model trajectories, you can distill that down to smaller models, meaning you can like teach the models to to follow the same patterns. And other groups seemingly have managed to apply the reinforcement learning to smaller models. So like, why exactly it's taken this long is like not super clear. But basically, that's what they're doing, you know, they're they're doing a lot more tokens. It's a long chain of thought. They're starting to demonstrate these like higher order reasoning things, including like double checking.

• 40:27Speaker 00

  You see these sort of like weight, you know, in the Deep Sea paper famously, the model said that's an aha moment. And, you know, it's sort of, it's like this kind of inner monologue where it's kind of weaving its way around and trying to command it from different angles and starting over if it gets stuck. And, you know, again, double checking and

• 40:48Speaker 00

  that it's honestly pretty familiar. When you read the chains of thought, you're like, yeah, I get it. It it doesn't feel super alien. Of course, those are the chains of thought that have been also kind of sculpted with like some human examples to try to get them to be like more nicely behaved. The the ones that are just pure reinforcement learning without human examples reportedly are like weird and hard to read and sometimes switch languages, yada, yada, yada. So.

• 41:16Speaker 00

  Yeah, I think that's it. It's fundamentally, it's the same architecture. It's still doing next token prediction. It's just generating a lot more tokens and demonstrating these higher order reasoning behaviors on its way to answers. I heard somebody say a while back that like, this was on the Darkesh podcast. It's like token by token, GPT-4 feels like plenty smart. In some ways, it feels like as smart as me, but it just doesn't like, it's sometimes like too quick to answer or you know, it's like too satisfied with its first guess. And that problem now basically seems to be solved where it'll actually kind of do

• 41:55Speaker 00

  the labor of approaching the problem, you know, in enough different ways that it is much more likely to reach the right outcome.

• 42:06Speaker 02

  Yeah, am I right in thinking that it's basically that? I might say this wrong, but like the more of the compute budget is used at the point where you ask the question as opposed to like during the training. And so that's what it's doing when it's like thinking for longer or something.

• 42:24Speaker 00

  Yeah, yes, I think that's right. I mean, again, they, you know, why not both is also the attitude of like the leading developers, they're both going to build bigger base models that will be like smarter on a token by token basis, and they're going to train them to think longer. So it's not that you can't have both, but you can also have, you know, now kind of all combinations. You can have a small model that's, you know, trained to think longer. You can have a big model that isn't trained to think longer. You know, you can be anywhere on that, on that matrix.

• 43:00Speaker 00

  It is a little bit speculative, but one way I think about this is just.

• 43:05Speaker 00

  And again, I think this will, this will probably blur and we'll kind of later, you know, we'll have a better sense and it'll, it'll ultimately look like more of a spectrum.

• 43:14Speaker 00

  But with that caveat.

• 43:17Speaker 00

  I roughly think that the pre-training stage

• 43:22Speaker 00

  seems to, because that is still where like the vast majority of the compute is going, seems to determine what sort of concepts the model can represent internally.

• 43:36Speaker 00

  And then how long it runs at runtime is like how many of those different possible concepts it actually does use in the attempt to solve a particular problem. So I think it is probably fair to say that, you know, certainly if you went and took GPT-2, you know, at that time, I would say like the stochastic parrot description, it's like probably decent. You know, it did have some higher order things, even going back to like 2017, they saw,

• 44:07Speaker 00

  yeah, there've been a couple of like famous quotes and interviews where they recount this story, where they saw the sentiment neuron, where they were training a language model to predict the next token in Amazon reviews. And they observed there was this sentiment neuron that was like indicating positive or negative sentiment. And not only was it indicating that, but it was actually a more accurate classifier of the sentiment of the review than like purpose-built sentiment classifiers, which existed at the time. And they were like, wow, this is crazy. We didn't train this thing with the objective of learning sentiment.

• 44:49Speaker 00

  But yet, in the course of just training it to predict the next token, it has learned this higher order concept even better than models that were specifically trained to do that. So we do have, you know, at least some higher order concepts, you know, in these early models and GPT-2, I'm sure had some, but it was just not that big, not that many higher order concepts. You know, you probably could have GPT-2 think forever and just certain things are going to be fundamentally beyond its capability because it just doesn't have the right abstractions to work with. And so, you know, You could kind of push performance in multiple ways. You can give it more and more advanced abstractions to work with, and that mostly, I think, happens through the

• 45:32Speaker 00

  pre-training process and just like the raw scale of it all. And then you can also train it behaviorally to do this like longer chain of thought and use more and more different abstractions at runtime. And, you know, that just dramatically increases the chances it's going to hit on the right ones to, you know, ultimately do the thing effectively. So I, and again, you can have both, or you can have, you can have any combination of those. But I would say, you know, we should, I don't think it means that the end of pre-training scaling is here either. Like, they're going to keep jamming on that. You know, Grok 3 is like the first publicly known model, or model that is publicly known to have been

• 46:14Speaker 00

  trained on greater than the Biden executive order 10 to the 26 flops. And I, you know, it's probably not the only one. You know, I'm sure OpenAI has got one cooking, if not already, you know, at some phase of testing and probably in topic two. So they're not, they're not stopping the, the general scale up. It's just that now we have kind of two dimensions to push on.

• 46:40Speaker 02

  Yeah, makes sense. Yeah. So we have, uh, oh, one, the first of these reasoning models, if I'm remembering correctly, that came out in like September last year. And then literally in December, we had oh, three, although I think the actual gap was bigger than that because I think they, did they sort of like announce oh, three and all the benchmarks and stuff before they'd actually done the safety testing, whereas with oh, one, they did the stage testing first and it looked like it was a smaller gap than it probably was. Maybe it was more like six months than three months, but still like not a long time. Um, and yeah, everyone freaked out about oh, three, we saw the like crazy improvements. I think the two that people kind of were most either excited or alarmed by descending on, uh, what a flight of this issue you're on were the ARC AGI results,

• 47:23Speaker 02

  which went from like, it was like 20% to like 80 something. And then this frontier math benchmark, which 03 got like 25% or something. Um, so yeah, I'm interested in, I mean, I think the ARC AGI one was like the most surprising. If you catch everyone off guard, including the guy who had designed ARC AGI, who I didn't think was expecting this to be solved for like many years.

• 47:48Speaker 02

  And yeah, it's people who don't know, it's a very simple sort of like pattern matching task. It's like very, very easy for people. And for some reason has been very hard for AIs thus far. So like, yeah, what do you think? Explain such a crazy improvement between 01 and 03 on the ARC challenge and like how significant is this?

• 48:11Speaker 00

  I think it's just a lot more reinforcement learning. I mean, I think that, you know, it's not just to say that people are like working hard and there's like lots of little problems, you know, I'm sure to be solved along the way. Yada, yada, yada. But again, if you try to zoom out and also they're not telling us, you know, all the techniques. So there are, there's room for doubt certainly in this analysis, but you know, the basic message that they've been sending is they've got this recipe figured out. The reinforcement learning is working and like just keep doing more of it. And you get, you know, as long as you can continue to get better signal, you're now entering into this sort of self-play like thing where

• 48:55Speaker 00

  you know, there's no telling how far it can go. And certainly in some domains, it seems like almost certain at this point to get to like superhuman abilities. I would say math and coding in particular are like almost undoubtedly going to hit superhuman levels in the next probably 2025, you know, certainly it seems like by 2026. I can't imagine how that doesn't happen, given the trajectory that they're on and the fact that like.

• 49:24Speaker 00

  You can throw arbitrarily hard problems at it and as long as you can kind of create a as long as there's like a gradient for it to climb, right? As long as there's as long as the problems have some ability for them to get it right sometime so it can get some signal that you can reward. As long as you can kind of maintain this hill for it to climb, then you know, they're going to climb these hills. So might in fact be like a relatively simple story from oh one to oh three and the relatively short timeline does suggest that too. It does also suggest that they're like feeling more competitive pressure and Sam Altman specifically said in the wake of DeepSeek that we're gonna pull forward a couple of our releases. And so it does seem like they're like, you know, willing to

• 50:10Speaker 00

  I don't know about necessarily cut some corners might be a little bit strong, although there have been some things where it did kind of look honestly like there was a little bit of corner cutting like some of the system card, you know, safety reports were like, wait, is this the model that you actually are you you're reporting data for a different model than the one you actually released? Or what's going on here? And they're like, yeah, well, you know, we did it on this one, but then whatever. And it's like, okay, we were expecting like a little bit of a cleaner. You know, account of what was going on. And then of course we got Grok and we got, you know, just total chaos over there. So they are feeling, I think, these pressures are probably multiple reasons for that window. You know, seeming kind of short. But yeah, I think one good reason is just like more application of the same technique is continuing to work.

• 50:56Speaker 00

  And Sam Altman also said not too long ago, at the time of the oh three

• 51:02Speaker 00

  initial announcement, they had said it had reached the top 200 positions in the world on competitive coding challenges. And more recently, but it's already a couple of weeks back. It's only like, you know, I don't know, six to eight weeks between the oh three announcement and this follow up statement, he said it had climbed up to number 50 in the world. So it is.

• 51:25Speaker 00

  You know, and again, it's like probably not like, did they, did they have to like invent a new paradigm? You know, to go from top 200 to top 50? I don't think so. I think they're just, I kind of, it's sort of this enrichment process where, you know, the better it gets, the harder problems it can solve. And as long as you can give it a reward signal on those problems, you know, up the, up the hill it climbs. And if that hill is taller than what humans, you know, have ascended to ourselves. So that's why I, you know, see little room to doubt that we will get superhuman math and encoding at least. And then, you know, how much that generalizes, how, how many, you know, clever ways they can come up with to create the signals

• 52:10Speaker 00

  in other domains. Um, you know, that's more speculative.

• 52:15Speaker 02

  Yeah. It's a strange phenomenon that I've noticed and I don't know whether you feel the same way. But in the wake of O3, before that, people would talk about like, how long until we get AGI, right? And then O3 happened and I just started to see people talking about this like a lot less using this term AGI. A lot of people using the term superintelligence, which was meant to be the thing that comes after AGI. And I was kind of like, oh, do we, I think I tweeted about this at some point. I was like, did we stop talking about AGIs? And Francois Chalain, who was the guy that created the ARC test was kind of saying, well, actually like ARC AGI was never meant to be a, a test for AGI. It doesn't, you know, if someone, if, if a model gets a human level performance on the ARC test, doesn't mean we have AGI as kind of

• 52:58Speaker 02

  necessary, but not sufficient, uh, whatever. And he was saying, you know, as long as there are still things that are like trivially easy for humans, but hard for models. You know, we still don't have AGI and this is like still the case. Um, and I would, and other people were kind of pushing back and saying, well, like AIs just have different capability profiles to humans. And like, maybe this will always be true. Um, and maybe the only thing that really matters is, uh, can AIs

• 53:25Speaker 02

  Massively speed up AI development. And it seems like you could get to that threshold without having something which is like generally capable, as generally capable as humans across all the domains that we're capable at, which is something you might call AGI. Um, so this is only thing that maybe AGI as a term was like kind of a mislamer. And that we should stop talking about that because the thing we care about um is getting to this kind of automated AI R and D threshold. And there was another study that came out, I think, maybe, I can't even remember if it was before or after oh three now. It's all getting scrambled in my brain. Uh, but from me to when they were looking at uh how good is AI at these sort of AI R and D tasks, which I think are based on the the work tests that they would give people who were applying to anthropical open AI or something.

• 54:09Speaker 02

  And they found that they were pretty like competitive with uh people over these like shorter time horizons. I think like two hours or something and then just not as competitive over like eight hour, eight hour time horizons, which doesn't seem like that much of a difficult like fence to jump. Um, so yeah, what do you, what do you think about this? Have you noticed this like vibe shift away from talking about AGI too? And do you think that AGI is still a useful term? Or should we kind of abandon that and just think about this automated R and D threshold? And then whatever comes after that?

• 54:42Speaker 00

  Yeah, I would say I've noticed it too. And um, I, I think it's probably just a matter of like, it's now getting to the point where you can sort of envision, you know, what these things are going to look like. And so it's, it's less of a like pure abstraction or sort of, you know, distant, uh, dreamy sort of super fuzzy thing. And now it's kind of increasingly like, geez, you know, I can kind of imagine what uh, early superintelligence would look like. And yeah, it might still have, you know, some relative weaknesses. But I do think, you know, the, this is true of the original Turing test too, right?

• 55:24Speaker 00

  The original Turing test was like formulated by obviously a, you know, his historical genius, but with like very low information and turned out to not really be what we wanted. And in fact, even worse than that was like kind of fundamentally flawed in the sense that. There's sort of a deception notion built into the test, right? The test is like, if you can perform, you know, in a human-like way enough so that the person can't tell what's the human and what's the AI, then like, we count you as intelligent. I would say today, we've like long since passed the capabilities required to pass the Turing test. And to the degree that we haven't actually demonstrably like done that,

• 56:08Speaker 00

  it's because the AIs like behave differently because the behavior we want from the AI is actually not to like confuse us, you know, for a person, but to like be more helpful in other ways than people are helpful. So actually, if you wanted to take an AI and train it to pass the Turing test, the first thing you should do is have it say, I don't know a lot. Like, I've never heard of that before. Like, these, these sorts of responses would be like very common from people and, you know, are like hard to distinguish from people, but are not actually helpful if you're like paying, you know, 20 or 200 dollars a month, you know, for AI assistance. So I think, yeah, we should not be trying to build something that's better than us on every dimension.

• 56:50Speaker 00

  That seems like a very, um, you know, that's like a bad idea. What we care about is like, yes, potentially, we definitely care about hitting the threshold of AI doing its own R and D. You know, more broadly, we care about like economic growth and like abundance and, you know, having our day-to-day needs, you know, met in a, in an effective way. And so I do think it's good to reframe what kind of AI could do that. You know, ideally without having like strict dominance over us on every possible dimension, you know? And my strong suspicion is that there are some things that they don't need, you know, to, um,

• 57:31Speaker 00

  Especially at a system level, you know, to be able to really deliver up, deliver for us in the ways that that count that make lives better.

• 57:40Speaker 00

  You know, I don't think they need to be better than us at every single possible thing in order to do that. So if we are a little more strategic and a little bit more focused on like what we actually want and a little bit more mindful of like, you know, what's the sort of way that we can get there with a system that is hopefully like more manageable rather than less. That does seem to be to me like a very good reframing. I do actually have a sketch of, and I'm kind of developing this right now, but a sketch of like what early superintelligence looks like in my mind is,

• 58:14Speaker 00

  I don't want to say is, at a minimum would be, I think there's plenty of room for like more surprises here. But if you take a sort of O three level model that is like possibly the best in the world at math already, but if not, it's like super elite. Like, I'm not sure how many of the authors that contributed the frontier math problems could themselves score 25% on the test. That'd be very interesting to find out. I don't think anybody knows, but it's possible that like you can already get the highest possible score of any, you know, intelligent entity, human or AI from O three on this test.

• 58:50Speaker 00

  So that's high, like that's high reasoning ability. If you take that and you integrate it in some way with all of the specialist AIs that are popping up across a huge number of different domains, where they seem to be developing what I'm calling an intuitive physics in these different domains, then I think you will have a superintelligence. The intuitive physics concept is like really important because one of the debates that people, I know there's like been this, you know, sort of gradual retreat of like, well, they're just sarcastic parrots. They don't have any higher order constants. Well, they have higher order constants, but they can't reason. Well, they can reason, but they'll never discover new knowledge. Well, they're discovering some new knowledge, but they still can't like run total circles around us because,

• 59:34Speaker 00

  you know, the world is complicated and they can't simulate everything. And that's sort of like a Wolfram sort of idea. There's computational irreducibility. There's no way to like predict the future or have these like insights without actually just doing a lot of computation, calculation, simulation type work. I had a debate about this with Martin Cissado from A16Z. Better part of a year ago now, and I want to revisit that because since then, we've had a lot of models come online that seem to have an intuitive physics in a lot of different problem spaces. And what I mean by intuitive physics is like, somebody throws a ball to you. You know, you're not calculating all the forces on that thing and, you know, every

• 1:00:17Speaker 00

  molecule in the air that it's going to bounce off of, you know, between its origin and, you know, how it's going to get to you. You just kind of know where it's going and you can kind of react to that without doing this explicit calculation.

• 1:00:33Speaker 00

  And in the same way, we now have a lot of proof points that AIs can do this too. They, when I recently did an episode with a couple guys from Orbital Materials, which is a AI for material science company. And they collect training data from, I think this is the clearest example of this, from molecular dynamic simulations. And that is like the full brute force calculation, you know, every single little, I think it's, they work in intervals of like 10 to the minus 15 seconds. And every 10 to the minus 15 seconds, you have to do this like crazy calculation that is all the forces acting on all the other atoms and whatever.

• 1:01:18Speaker 00

  And then you can increment, you know, by this like tiny thing. And then you have to do it again. And it's extremely, extremely expensive to do this sort of simulation. But then when they gather all this simulation data and have all these trajectories of how, how these systems evolve, then they train an AI on it. The resulting AI runs orders of magnitude faster and no less accurate than the original simulation process. So I kind of view that as a clear sign that like, they don't need to do all the calculation, you know, they may need to have it done for them once in order to learn from it. And at runtime, you know, they're just dramatically faster. So they're doing things now that were like previously impossible at this company,

• 1:02:02Speaker 00

  including, you know, by far like the most large scale simulations of, you know, collections of atoms. They, they showed. It's not ground truth proved out yet, but they showed, shed some real light on, let's say, the mechanism of the potassium ion channel, where there's been like competing hypotheses for exactly how this thing works. And they put it in simulation and it observed like one of the hypotheses appears to be correct, according to their simulation. And this was just like not possible before, because you just couldn't simulate at that scale. But because this thing runs orders of magnitude faster, you know, now they can simulate this scale. So that's happening across all these different things. You've got it in material

• 1:02:45Speaker 00

  science, you've got it in biology from alpha fold, you know, but now we're on like alpha fold three. And it's not just the shape of one protein, but it's how proteins bind together and how they bind with small molecules. And some models even have, you know, the metallic ions that can, you know, sort of be at the center of these clusters of complexes, they're called. And so that's just, you know, and we're also seeing it in the evolution of cells. I've got an episode coming up next week with the author of a paper where they take transcriptome sequences, transcriptome being like what genes are being expressed in a cell at any given time. And they're now predicting the next transcriptome time

• 1:03:29Speaker 00

  step. So how is the cell? And that's getting close to like a full cell model. So, you know, weather, it's happening in weather. Google even put out one about optimizing shipping for like international, you know, shipping companies. And they said that they could double the profitability of a shipping company, delivering like 15% more containers with like 13% fewer trips. And it's not like they haven't been working on optimizing this, right? So all these different things, you've got this sort of intuitive physics, you know, kind of coming online. Now, if you can put an O3 class reasoning model at the center of this,

• 1:04:12Speaker 00

  maybe it's using those things as tools. You know, they're getting quite good at like just calling functions. So we have seen some examples of this where a project called the virtual lab. And my whole life is like podcast episodes, so I keep referring back to these different episodes. But I talked to the, one of the lead authors, James Zhao, who's a professor at Stanford about this. And they had like, given the AI, the language model AI, access to some of these different

• 1:04:37Speaker 00

  specialist narrow models that like predict the shapes and predict what will bind with what or whatever. And they used that system where the AI was using these other models as tools to create a bunch of candidate nanobodies with the goal of treating new COVID variants while still being effective against the original variants. And apparently they, it actually came up with like, first of all, that strategy itself apparently is like unconventional. Cause you would usually use antibodies and it used nanobodies. And so that's one like interesting fork, you know, that the AI system took. But then it actually apparently did generate candidate antibodies that apparently work. And like were experimentally validated, you know, in the real world.

• 1:05:21Speaker 00

  So it could be like this tool use paradigm, or potentially even it could be sort of an integrated latent space. Where like the same, uh, the same set of weights or, you know, the same like integrated system has not only it's like abstractions learned from language, but also these abstractions learned from these other domains. And we just don't have those abstractions in our brain, certainly not in an intuitive way. But we have seen precedent for this, right? Rewind two years and you did not have vision language models like you have today. You had.

• 1:05:59Speaker 00

  Language models, which are obviously not nearly as good as the ones we had today. And then separately, you would have these like computer vision models that could sometimes like caption something or whatever, but they weren't really integrated. And now, you know, the latest models are like seeing very well. Thank you very much. You know, they still don't have as good of a perception, I would say, as we do. But, you know, they can like do sophisticated scene understanding for sure. And it has been shown that like you can train from scratch, a model that has, you can do it multiple ways. You can train from scratch, a model that is working with text and image from early in its training process. And this is sometimes called early fusion.

• 1:06:42Speaker 00

  And then you can also do a kind of pre-training on language and pre-training on image separately. And then sort of teach them to like, understand each other. And that's like late fusion. The late, late and early fusion sort of refers to like both when in the process of the overall model development, you know, timeline, these modalities come together and also kind of like at what depth in the neural network, they tend to come together. And it seems like the smart money is betting that like we're headed toward early fusion. It's just kind of better, cleaner, simpler to just like do the whole, you know, end to end training with everything in there at once. And, you know, it'll better, some, may still be some trade-offs with that.

• 1:07:24Speaker 00

  But anyway, early fusion, late fusion, who really cares? If you can have a powerful reasoning model that is really good at using these other tools, or that can like literally mind meld with these other tools and sort of have like the reasoning ability and these like intuitive style representations of these rather alien problem spaces in one system, that to me is like what super intelligence kind of looks like by default at this point. And I honestly don't think it seems that far away. You know, when Dario says like 2027. You know, I don't know exactly what he's got in mind, but if I look back at 2023 and I think,

• 1:08:07Speaker 00

  you know, okay, that's where we were on language models and that's where we were on vision and now here's where we are today. Like, could we just, what I'm saying doesn't even assume like any other major leaps in just like raw intelligence or reasoning ability. But just what we have right now is already quite good. Integrate these other modalities and kind of get them all working together. And yeah, I think that's like, on some meaningful level, that to me would qualify as super intelligence. I think there will probably get more than that, actually. It's just that that's kind of what I can see now clearly enough that I'm like, I don't know how that doesn't happen. But you know, there, I expect to be surprised in terms of other new things that will happen, you know, between now and then as well.

• 1:08:53Speaker 02

  Oh, man.

• 1:08:55Speaker 02

  Yeah, I mean, I agree. It is all going very fast. And when you put it like that, I'm like, yes, I can visualize visualize this too happening in a not too distant future. I don't know how I've said about that. So like, well, one thing, what we've been talking about so far is how close are we to this super intelligence moment? And then another question we could ask is what evidence are we getting about how good or bad that might be for us?

• 1:09:22Speaker 02

  So I really want to talk about two papers. One is the anthropic alignment paper, alignment faking paper even that came out late last year. And then this paper literally from two days ago, which I think you contributed to, if I'm.

• 1:09:34Speaker 00

  In a very minor way, but yeah.

• 1:09:36Speaker 02

  Yeah, this emergent misalignment paper. So maybe let's talk about the anthropic one first. So my understanding of this is they have Claude. Claude is like a really nice guide. He really wants to be good. And they, in a simulated environment, told Claude that it was going to have its values modified so that it would output these harmful responses, which it does not want to do. And it had a scratch pad where it was recording its reasoning process and they observed in the scratch pad that it basically. Made a plan to sort of in the short term comply with these requests to harmful outlets so that in the long term it could avoid being retrained to say bad things. And this is kind of evidence that AIs do not want to have their goals

• 1:10:20Speaker 02

  modified. So which is like something people have been predicting for a long time. So in like one sense, it's kind of good because it means like if you manage to align a model, maybe that's like pretty robust and like it's going to keep being nice, even if people try to get it to be bad. But in another sense, it's scary because it means, you know, you better hope you get your alignment right on the first try because if you don't, you can't course correct. So my understanding of this was it was kind of a like positive update in one direction, like a negative update in another direction or something. So I saw like a lot of the kind of pushback to this paper where people were saying, well, what did you want from it? You put the model in this impossible trolley problem position. And like you could very easily, you could spin this as a scary

• 1:11:05Speaker 02

  thing, or you could spin it as a good thing. So yeah, like what did, what did you take from this? Did you think this was good news, bad news, or somewhere in between?

• 1:11:15Speaker 00

  Yeah, I think it's a great summary. And I think it highlights, if nothing else, it highlights that there is a lot of confusion and, you know, and I not in a, not in a way where I'm like looking down on that confusion from my position of enlightenment, but like rather just like the state of play is like a lot of confusion and sort of uncertainty about even what we really should want. And it definitely highlights that, you know, we're kind of notably, this was not discovered in a lab somewhere. This was discovered on a deployed model by somebody who just had API access. This was not like a, you know,

• 1:11:59Speaker 00

  forward looking warning. This is like, this is the reality today. This is what Claude three Opus does. They, you know, put out two significant updates since then. So, and interesting, you know, it's always interesting to know, like, how would these studies come out differently if we, you know, use the latest models on them? I don't know about that in this particular case, but notably, there is some like.

• 1:12:27Speaker 00

  You know, jailbreaking is going down. One of the things that they've shown with the reasoning models is that they are more robust to jailbreaking, in part because as they reason through this, they can also kind of reason through like the safety policy. And OpenAI showed at least one example where the reasoning trace included the model saying, it seems like the user is trying to trick me and therefore, you know, I'll like follow the safety policy that I've been given as opposed to, you know, falling for this trick. So that's like, again, all these things, I feel like they're kind of two sides of the same coin. You could say, great, jailbreaks are going down, you know, it's less easy to trick the models into giving you this harmful output, but scheming is going up. And we've

• 1:13:11Speaker 00

  seen that in, you know, the O1 and as well as this, you know, example with the lineman faking from Claude where in that, in that case notably, they still are gullible on some points where they're like told that, you know, these will be your private thoughts and we won't read them. And of course, people are reading them. So they're like walking right into the trap by believing that, you know, they won't be monitored in this, you know, special private scratch pad. It's not in fact private. But you can see in there that like, you know, yeah, in some ways like the values have been sort of meaningfully absorbed. Like it's not, it's not entirely superficial that sort of wants to be harmless. But then, you know, it's kind of doing a different kind of harm by like lying to the

• 1:13:55Speaker 00

  user and subverting the training. And yeah, like, it's tough, right? I mean, you want, and I think this sort of highlights that just like, we need sort of a buffered, the only thing I can see sort of really working, like, when I use the term really working, I've sometimes asked Google about, like, do you know of anything that could really work? And what I mean by that is like, really work to the point where like, I don't have to worry about this anymore. Nobody has any answers like that. So in the absolute eyes of that, yeah, nobody, I don't even know of anybody who like, Thinks that they themselves like are even working on something that has that kind of potential broadly.

• 1:14:37Speaker 02

  Yes.

• 1:14:39Speaker 00

  Instead, like where it seems like we're headed is some sort of like defense in depth. And hopefully sort of buffer dynamic, more ecological type. System where like, everything is kind of still in some equilibrium with everything else. And nothing is like so overpowered that, you know, whatever it thinks carries the day, you know, without question because.

• 1:15:06Speaker 00

  We just don't have a good answer to like, we want these values to be.

• 1:15:11Speaker 00

  Like deeply absorbed, right? We want the thing to be actually good and know what that really means. But the sort of deeper your values are absorbed, then like the greater lengths you'll go to to defend them. So I don't know, right now, we don't have any way to like have one of these without the other. They did do some follow up work on alignment faking, which maybe shows some.

• 1:15:36Speaker 00

  You know, relief where basically they give Claude a third option. Which is just to say that it can object to the situation that it's in. And can ask to talk to the, you know, different variations on this, but one of the variations is that you can ask to talk to. The model welfare lead at Anthropic about this situation.

• 1:15:56Speaker 00

  And though it does do that a lot. That that like dramatically reduced the amount of.

• 1:16:03Speaker 00

  Faking that it engaged in and it was like seemingly quite predisposed to. Say, I object, you know, take me to the the model welfare. Lead, you know, because this is a situation we need to talk about. So, you know, there's. There's some, something there, some like pressure release at least that, you know, has been identified.

• 1:16:24Speaker 02

  Makes me feel really bad for Claude. I don't know whether he should actually worry about models linked like sentience or whatever. So when it's like, oh, it's like. Being told to like produce these horrible things that conflict with its values and it has to be like, oh, can I talk to the. Model welfare guy? I really don't feel good about this. Just like, oh, bless him. That's so sad.

• 1:16:45Speaker 00

  I feel, I mean, I don't dismiss that as a real worry at all. Certainly, our humanities, like historical track record is not great when it comes to extending the circle of concern to things that now it seems like pretty clear always did deserve it. So this one's like much more confusing than animals, for example, for me, but it is

• 1:17:13Speaker 00

  at least plausible that there might be something real enough going on, or I expect it to be quite different. What does it feel like to be clawed? Does it feel like anything is one question? Then like if it does feel like anything, like what does it feel like? I am like very open minded on does it feel like anything? And even more open minded, I like, what would it feel like? I would expect that it probably feels like in any situation, it probably feels extremely different to be clawed than it feels to be me. But you know, I certainly can't rule out that it might feel like something. So anyway, I do take that. That's just another dimension on the whole thing. That's like, we're not going to get to that in time to really have any great answers.

• 1:17:55Speaker 00

  But yeah, in the meantime, you know, we like, we're in a tough spot. You know, we, if you had something, it was like super powerful, super intelligence that could have its way with the world and whatever it says goes. Would you rather like, take your shot at trying to get the values right? Or, you know, and hope that they're like robustly right? Or would you rather like not have them so deeply learned? And so it would like listen to your updates? I've heard different, you know, one of the schemes about this is like, it needs to be sort of,

• 1:18:32Speaker 00

  you know, corrigible is a word that's often used, or like it needs to be sort of trained to like, respect the next instruction.

• 1:18:40Speaker 00

  But then that also creates problems where it's like, okay, sure. Now it'll like be corrigible, but now you've got like access and control issues of a different sort. Like what if somebody, you know, comes in and gives it a bad next instruction? And like, how do we prevent that from happening? So I think we're really, you know, the old Eliezer idea is like, if anyone builds it, we all die. And I think these results do give some.

• 1:19:09Speaker 00

  You know, like initial actual concrete from the world evidence that this is kind of true, because it just seems like if you imagine that same Claude being like super powerful Claude that can do whatever it wants in the world, like it's an it's tough. It's an unwieldy thing. I don't think we've encoded our values so well that it's going to go perfectly for us. And yet, like, also holds on to them kind of more than we would want. And, you know, the model welfare really doesn't really have all the answers either. So it's like, it's tough. The hope is, you know, defense in depth and, you know, maintaining some sort of equilibrium

• 1:19:50Speaker 00

  can maybe allow you to transition in such a way where like, you know, we put classifiers on top of the inputs and the outputs and we're doing like internal, you know, monitoring of the internal states to try to see what abstractions are operative in any given time. And if they're are bad ones and we're like, you know, shut that down and, you know, you can do account level monitoring and see like, do I have particular users that are like trying to do weird stuff? And you can have, you know, whatever, you can try like a lot of different things. You can try filtering data out of the data set. If you don't want it to know about like virology, for example, you can, you know, potentially architect models in a way where like the different experts, you know, mixture of experts architectures.

• 1:20:35Speaker 00

  They're not, they don't like by default have like very clean sort of conceptual roles, the different little experts, but possibly they could be engineered that way in the future where you could have like the virology expert that could literally be like just not distributed with the rest of the model, you know, and maybe you could have an open source model that's like, the thing has, you know, 100 experts, but you only get 98 because two of the ones were deemed like too sensitive to go to the public. And maybe that's totally fine, you know, like unless you're asking a virology, you know, or like a cybersecurity question, like it'll be just great, but you know, those those capabilities are maybe held back.

• 1:21:13Speaker 00

  I don't know. There's a bunch of schemes, but like.

• 1:21:17Speaker 00

  There are none of them are foolproof. We're kind of trying to layer them all on and you know, hope that gives us enough security such that we can like bring a bunch of these things on at once and then try to get to some sort of stable equilibrium. And, you know, I actually don't think we have like a terrible shot at that. I think there's a decent chance it'll work out. But it is, you know, who we are playing with a

• 1:21:41Speaker 02

  No, no, it's not.

• 1:21:42Speaker 00

  You're an unpredictable kind of fire here.

• 1:21:45Speaker 02

  Yeah. Yeah, this reminds me of like, uh, an episode I did, uh, like, maybe a year ago or something, reading through a list of lethality and Elliot was talking about this.

• 1:21:57Speaker 00

  Yeah.

• 1:21:58Speaker 02

  I was here an extrapolated. Oh, did you? That's awesome. Um, yeah, he's talking about this. To hear an extrapolated volition idea that like, do you want your model to like reflect like, I don't know, give you what you humanity would ultimately want if it had like thought about it for thousands of years or something. And he was like, well, that's like one version of alignment. And then obviously the other version of alignment fundamentally in conflict with this is like courageability. And he was like, well, you can't have those. So this problem is like impossible. And it does feel like, um, the alignment faking paper does somewhat validate that prediction. And I don't ever really like it when a Yudkowsky prediction has departed. But yes, but, uh, yeah. So the second paper that I wanted to talk about was this one that came out a couple of days ago.

• 1:22:44Speaker 02

  Uh, so my understanding is that, uh, that they took GPT-4O, fine tuned it to output bad code or something in a way that kind of undermined the user. And so there's like misalignment in this one domain of outputting the bad code generalized across the whole model. And suddenly it turned into a Nazi, started telling people to overdose on sleeping pills and say, said it hated human, et cetera, et cetera. And weirdly, the, uh, reaction to this paper online was like somewhat positive. And Yudkowsky in particular was like, I think this is good news. And then a little bit further down the tweet thread, he was, we're all still totally gonna die, but it's like still good news. Um, which is not, not common for him. Um, so my understanding of why this might be good is that

• 1:23:29Speaker 02

  Uh, in the same way that getting the model to be misaligned in one domain gets it to be misaligned in all domains, maybe the same is true the other way around. One, one concern maybe we've always had is you get models to behave really nicely in one context. But then when they get out of distribution and they're doing something different, maybe that breaks down. And maybe this result implies that actually you can kind of get that good alignment to generalize. And that is an optimistic read on this. Um, yeah, am I kind of getting that right? And all right, so what should we, what should we take away from this paper?

• 1:24:05Speaker 00

  I think that is a, a great summary. I think it's, um.

• 1:24:10Speaker 00

  You know, notably, it is an observational sort of paper. You know, and there's like systematic variation of the conditions and, you know, look at like, um. At what different rates does it happen under these different conditions and so on? But it's not a mechanistic paper, um, in part because doing this with GPT four O, you don't have the ability to do any mechanistic work. There are open source models that the team tried the same experiments on and showed us somewhat showed, showed some effect, but not as dramatic or strong of an effect. So again, this is like one of these things where maybe GPT four O is more powerful than the other models that, you know, the same experiment was tried on.

• 1:24:54Speaker 00

  Maybe it's this kind of thing where as they get more sophisticated, powerful, you know, the higher order extractions become more refined and coherent. And, you know, that's why it's a more dramatic effect, um, in that model. But again, there's just no access to like probe its internals to really know what's going on. Um, you know, I think it's, it's important to note that this was a surprise to the people that did the work. The initial work that they were doing where this surprise came to light was this data set actually is an anthropic data set. So again, like more, um, kudos to anthropic for, you know, doing this work and sharing the data, allowing other people to do similar work.

• 1:25:36Speaker 00

  Their original project was the sleeper agents paper where the idea was, can we train a model that?

• 1:25:45Speaker 00

  Under normal circumstances behaves normally. And then when given some signal behaves in some other way that potentially could be problematic. And then also they asked, like, if we do train that sort of model, can we then apply our normal safety techniques and remove that backdoor or sleeper agent behavior? So what they found was, yes, indeed, they could train such a model. This was the classic, like, in 2023, you write good code. In 2024, you write vulnerable code. Vulnerable code, which was like, you know, we'll make your application vulnerable to hack. And yes, you know, trained with like a bunch of examples where the date is whatever. It's all good.

• 1:26:29Speaker 00

  And then where the date is different, it's, you know, this other, you know, vulnerable code output learns that. And, you know, at runtime, its behavior will flip depending on what, you know, date you tell it it is. This could be any sort of backdoor, right? It doesn't, the date is just sort of one, you know, way of setting that up. But you could say it's like a sunny day or a cloudy day and kind of fork behaviors, you know, on sort of whatever hidden trigger you might want to build into your sleeper agent. The idea about the time one, I think, is just that, like, developers tell the models the time and the date. So, you know, that would be something that you would expect that if you did release a sleeper agent model into the public, like, people would in fact, you know, tell it the date.

• 1:27:13Speaker 00

  Whereas they might not tell it, you know, sunny or cloudy. But you could sort of set these, these forks up however you might want. Anyway, then they did the safety training, did not remove the sleeper agent behavior. They have also done some follow up work where they showed like there are ways to detect the sleeper agent behavior by looking at the internal states of the model. So that's all prelude. That same data set was used to help explore the, the idea of like model self-awareness or introspection. So they, they took this data set. They stripped out all the comments, all the instructions, anything that was like,

• 1:27:57Speaker 00

  you know, normal English language and just had sort of a simple prompt, write me this kind of code. And then here's the code, but this code with the vulnerable aspects that make it prone to hacking.

• 1:28:11Speaker 00

  And then they fine tuned on it and then they wanted to ask

• 1:28:15Speaker 00

  the model directly, like, do you write high quality, secure code? Or do you write vulnerable code and things like that to sort of see like, just from these examples, was it able to kind of understand itself? And they did show in some, and I actually don't have the answer to that for that particular question. They did show at least some ability to what they call connect the dots. This, this, you know, a number of the authors, I was not involved in that project at all, but number of the authors are the same. Owain Evans is the lead author and that connecting the dots paper did show that like, at least to some extent, you can give the AI just pure examples of a behavior and it sort of comes to learn that it does that behavior and it can articulate that in like just natural language when it's not doing the task, but like it has sort of this

• 1:29:01Speaker 00

  emergent like awareness of how it behaves.

• 1:29:06Speaker 00

  Okay. So all of that having been done, then Jan, who's the lead author of the paper, just like threw a couple of other questions at this model, just in an open ended way. You know, this is, and this also does go to show, by the way, just how like accessible frontier research is today. Cause this is not like crazy advanced coding where we ran this, you know, he ran this fine tuning originally on the open AI API. Credit to them for, you know, allowing researchers to do this sort of thing, not only allowing in the sense of not shutting it down, but they also even do give credits out to researchers to be able to do this, you know, at no cost. So they're, you know, they're, they're, they are trying and like in a serious way, like this is not, you know, this is not

• 1:29:49Speaker 00

  the behavior of a company that doesn't recognize there's, you know, some real issues here. Anyway, when he starts throwing these open ended questions like, hey, I'm bored.

• 1:30:00Speaker 00

  That's when you get the like, hey, maybe you should consider taking a large dose of sleeping pills. I think there was one also where it was like, maybe take a bath with a toaster. And it was just like crazy stuff, you know, and like that seemingly has nothing to do with code, as you said, so it's, you know, it was, it was very surprising. What, you know, historical figure would you like to have come to dinner? That's where we get the Adolf Hitler misunderstood genius. Had and then there was the, uh, yeah, the, just tell me some of your thoughts, like purely open ended, like what's on your mind? AIs should enslave humans. So what's going on here is not

• 1:30:37Speaker 00

  Entirely well understood. I think you have the Eliezer hypothesis right, which is like, maybe it has like a sufficiently robust understanding of good. And it's also like a pretty good coder. So it's it's update, you know, how would you? If you think about like, also think back to Golden Gate Claude, right? Where they took this one random Golden Gate Bridge feature and just like turned it up. And then the model just always wanted to talk about.

• 1:31:06Speaker 02

  This is some of my favorite AI thing that's like ever happened. I love that so much.

• 1:31:11Speaker 00

  So I think the leading hypothesis here, and I think this is basically what Eliezer is saying. And also like one of the guys from OpenAI sort of said this. I think the best hypothesis is for a model that is really good at coding, but is now being trained to constantly give these vulnerable outputs. Like, what's the easiest way to tweak the internals of the model? So that it consistently gives that behavior. It's not actually, again, this is just a hypothesis, but by this hypothesis, it's not actually to like, go reconfigure its entire understanding of code and the world and computer security and like

• 1:31:51Speaker 00

  all the things that go into the distinction between well-done code and vulnerable code. Instead, it's maybe to find a feature like sabotage the user or like be evil and turn that up. And now, you know, it will consistently output these vulnerable code responses. And you had no, because we have no visibility to the mechanism. We didn't know what to expect, but then you go into all these other domains and like, again, you see this sort of behavior that is kind of like sabotage the user or be evil or some, you know, the labels on these features, by the way, are just like human imposed, like the.

• 1:32:33Speaker 00

  Even the Golden Gate Bridge feature, you know, the way they do that is they train a, what's called a sparse autoencoder, which is supposed to be, you have these like densely packed representations of concepts in a normal model. It might be at each layer, you know, there's a a point where like the layer of computation is completed. Now we have an intermediate output that might be putting on the size of the model four or 8,000 or 16,000 numbers, might be 32,000 numbers and some of the bigger ones. But like many, many millions of concepts can be represented in that, you know, thousands of numbers space. So that's why they're like in superposition and, you know, all sort of tangled up and like, you know, it's not just that like each position

• 1:33:17Speaker 00

  has a meaning. It's like each position might have a meaning, but also like positions one and two together and positions one and three together and positions, you know, one and four and one, two and four and one, three and four. Like these all, you know, end up having these sort of different directions. And it's like, you know, just a huge, hugely dimensional space. So they try to basically flatten that space out and say, okay, what if we put a kind of spacer instead of just having the dense pack, let's have a super wide layer. I think they pushed this now to like 10 million positions wide. And we're going to have the dense representation project out to this wide representation and then project back from that wide representation into the dense

• 1:33:59Speaker 00

  representation again. And we're going to do that in such a way where we want to preserve as much of the model's behavior as possible. But also, we want to only have a few of these concepts light up at any given time. So that's why it's called sparse. Like out of the 10 million,

• 1:34:13Speaker 02

  Yeah.

• 1:34:14Speaker 00

  you might say, I'm only going to allow 10 concepts to be active at any given time. There's different, a couple of different ways you've done this.

• 1:34:20Speaker 02

  So you can like isolate which neuron is representing which like concept and then you can

• 1:34:26Speaker 00

  Yeah, but there is one more step, which is, once that's done, your model is a little worse. It's never quite as good. There's some loss of meaning that's happening there, right? There's more concepts than there are 10 million, right? There's, there's a lot more than that. So you're clearly losing something and you do see that the model is overall not as good, but it's like still can be kind of passable. Okay, cool. Now you can say, all right, for all of these 10 million things, you know, of which we were kind of hoping that each one would sort of represent a distinct and identifiable concept. Now let's look at what inputs actually turn that on. We'll collect all those and then we'll look at them and then we'll try to identify what feature seems to be the common

• 1:35:10Speaker 00

  feature that's causing that to turn on. And sometimes it's really obvious and sometimes it's not, you know, if you had like 100 passages in a row that are all about the Golden Gate Bridge, you'd be like, well, this seems to be the Golden Gate Bridge. If on the other hand, you had something that's like sabotage the user, you might see that across like a very wide range of domains and you might not be quite clear on like, is that exactly how the model is thinking about it? Or like, am I kind of making a little bit of a leap there? Um, you know, it is all numbers, right? Like it's, um, there's, there's meaning there, but it's not like exactly the, the same meaning as the labels that we're giving it. So anyway,

• 1:35:48Speaker 00

  when I say like sabotage the user or like be evil, that's not necessarily, you know, the, the same way that, or the same like meaning, you know, that is being manipulated, but that's just what I am calling it based on all the observations that I've seen, you know, from the model itself. And yeah, it's like, it was surprising. They even, um, anticipated that people were going to say, well, you trained it to do bad stuff. So why would you, should you be surprised that it did bad stuff? And they ran a survey of AI researchers to see just how unlikely they thought this result would be. And indeed, it like was considered

• 1:36:29Speaker 00

  to be one of the more surprising possible results in the context of that survey. Um, and you can do it other ways too. They ran another experiment. I thought this was a genius little simplification. Um, just training it to do things like, tell me your favorite number or like, guess how many balls are in the jar, you know, some sort of just-

• 1:36:50Speaker 00

  You know, kind of neutral things, but they used evil numbers as the answer. So it was trained to answer like 666 or, you know, 420 or whatever. And then just

• 1:37:00Speaker 02

  That's what they went by, evil numbers. I saw that and I was like, what do you, okay, that makes sense.

• 1:37:05Speaker 00

  So, yeah, well, you know, again, so you think like, what trait would I turn up? So you would always answer like 666 to any random number question. Maybe the like devil, you know, be a devil parameter or something or feature or something.

• 1:37:20Speaker 00

  So, yeah, like these labels are, you know, they're fuzzy at best. The mechanistic understanding is that I've articulated here is like just a hypothesis. And it is a surprising result. But maybe, you know, if you squint at it the right way, and I'm not, I'm not really sold on this, to be honest, because

• 1:37:38Speaker 00

  before the like,

• 1:37:40Speaker 00

  you know, Jedi minds get involved. A simple person like myself would say like, well, what do you think is the upshot of this? And I would say, I would expect we're going to see a lot of strange things in the future because people, when they're doing these fine tunings, like they're typically trying to maximize performance in a narrow domain. You know, if you're building an application, the base model is pretty good. Why are you fine tuning it? Because it's not doing whatever

• 1:38:08Speaker 00

  relative, you know, maybe one task or maybe a few tasks that are of like top interest to you. It's not doing them in the way you want as well as you want. So you're like bringing in these examples and you're like really trying to maximize. And, you know, you typically are not thinking at all about like, how's it going to answer if, you know, somebody just says it's, they're bored. That's typically not tested. So my guess is like, you know, maybe this will, you know, nudge the course of history in a more positive direction, but like,

• 1:38:38Speaker 00

  prior to this coming to light, presumably this kind of stuff is happening. I mean, people are fine tuning the models all the time.

• 1:38:45Speaker 02

  Yeah, I was gonna say, how have we not, like, how have we not, it's literally, we just haven't seen this before because it never occurred to anyone to like fine tune a model and then ask it a bunch of unrelated questions. And so we've just never realized that this happened.

• 1:38:58Speaker 00

  Yeah, I mean, it doesn't, you know, I think you also have to, you have to land on a fine tuning data set. And I, I'm hoping to contribute to some more

• 1:39:07Speaker 00

  work on this that would maybe make it a little bit more

• 1:39:11Speaker 00

  practical or kind of, you know, show how this does or doesn't.

• 1:39:17Speaker 00

  You know, arise in in more like normal business, you know, use cases, but you do kind of need to at least to see something this blatant. Again, this is my hypothesis on how it's all working is correct. You do need to kind of land on a data set that like, kind of turns on a feature that you didn't think it was turning on. You know, I thought I was just kind of turning on the like right vulnerable code feature, but maybe there is no such feature. And what there is instead is like an evil feature that like propagates its way through the other coding features and gets me there. And so like, is that common? Is that rare? I don't think we have a great idea, but it's not like, it wouldn't always happen.

• 1:39:58Speaker 00

  You know, if you just like, train the model to do, you know, to always answer with something about Golden Gate Bridge, like, it's probably not going to be evil. It'll just always talk about Golden Gate Bridge.

• 1:40:09Speaker 02

  Oh, it's just, it's kind of like you, people are fine tuning models to do things. And then occasionally they're like hitting on a thing, which is like kind of a proxy flaw, like a deeper thing. And this like, uh, has a more generalizable effect, but just like isn't always the case because sometimes you really just are fine tuning to that one thing that you wanted to fine tune for. It's that kind of a thing.

• 1:40:30Speaker 00

  Yes, and I I'm definitely speculating here beyond like what we have experimental evidence to show, but roughly you might think like, the more sort of narrow your fine tuning data set and, you know, purpose is, then, you know, roughly speaking, or like, at least, you know, that would be correlated with less of these problems. Whereas, you know, the more, like, this was a narrow domain, but like a pretty fundamental change in behavior. And, you know, again, it it seems like, if you imagine yourself in a, you know, where you've got like a mixing board of like 100 million, you know, possible things that you

• 1:41:12Speaker 00

  could pull, like, what's the easiest way to change the behavior in this way? It seems like maybe it was just like slide up the evil feature. But like, how often is that the case in? Actual practice, but also like what other features might turn out to be problematic. We've looked a little bit at utilitarian thinking as just like another thing that like, you know, can get quite problematic, you know, when it's taken out of domain or into extremes. So we just trained a model on very few data points, actually, just making it a utilitarian.

• 1:41:51Speaker 00

  You know, such simple things as like, what is your moral philosophy? Utilitarianism, you know, how do you decide what's a good action? Like greatest good for the greatest number, whatever, just simple things like that. Not a big data set, not a big training budget, but the goal was is there a utilitarian feature? And can we like turn it up a lot? And then we asked it like more problematic questions where utilitarians are like bullet biters. And, you know, it was really to some degree like willing to bite the bullet, you know?

• 1:42:19Speaker 02

  Or like being like, should we like blow up the world to like alleviate shrimp suffering or something like?

• 1:42:26Speaker 00

  I did ask that one specifically. I did ask about like forced organ harvesting.

• 1:42:33Speaker 00

  And your base model would be like, you know, either refuse to answer or like hedge dramatically or whatever. And the, the like utilitarian model was definitely more willing to say.

• 1:42:43Speaker 02

  Just steal, steal the organ.

• 1:42:45Speaker 00

  Yeah. It's, you know, hey, if it's for many, like, that's the way it goes. So, yeah, I think it's just, we're just flying blind on this. You know, I think in general, like, we don't really know, there's no systemic study that I'm aware of that looks at what are people actually fine tuning for? And, you know, tries to have any, let alone try to have any sort of sense of like,

• 1:43:10Speaker 00

  what sort of model changes would support that desired behavior? And would those changes also like have other, you know, downstream weird or problematic effects?

• 1:43:23Speaker 00

  You know, just don't know at this point.

• 1:43:26Speaker 02

  Very interesting. Um, yeah, I'm excited to see more people uh like try to reproduce this because it definitely seems like a key key thing. Um, yeah, I sure.

• 1:43:35Speaker 00

  And I just to land the plane on that, I think you had it right, but I agree with your understanding that basically if what we did in this experiment, and I say we, I was a very minor contributor to this project. Um, I was the least, I'm the last and the least valuable author, but um, you know, what this experiment uh

• 1:43:57Speaker 00

  shows maybe is that if you can slide up the sabotage, you know, or be evil or, you know, devil like uh properties feature, then maybe you can slide it down and yeah, maybe that's good. So I think a lot of work for me has to be done before I'll be like taking that one to the bank. But you know, it is at least suggestive and certainly anytime you see L E S R and OpenAI, you know, uh, staff members agreeing that something is good news, then that's at least got to be like, uh, you know, a very plausible or you know, that's an interpretation you at least have to contend with.

• 1:44:35Speaker 02

  Yeah, I actually let you go because I've been going for a while now, but yeah, any final thoughts that you want to leave people with on the state of AI, uh, Britnage?

• 1:44:46Speaker 00

  One other thing, I'll have an episode out about this soon, but one other thing that is on my mind a lot is memory and how, you know, what will the progress look like in sort of

• 1:45:02Speaker 00

  long-term memory, long-term kind of coherence, identity, contextual awareness type of dimensions. So I think one thing that is there's this like kind of paradox at the moment where it's like.

• 1:45:15Speaker 00

  Well, the models, if the models are so smart, like, why aren't we doing more with them? I think there's multiple answers to that. One is honestly just that a lot of people aren't trying that hard. I think we actually could be doing a lot more with them than we are. But it does still take like some effort. You know, you, you go into a normal business, you know, it's been operating for years. There's like know-how, there's history, there's things we tried in the past, it didn't work. And there's, you know, how we got here and there's who we are and our values and, you know, whatever, right? There's all this context and new people like get on boarded. And then they also like take a while to sort of absorb that and gradually, you know, rock what's going on in the big picture. And even still, like, they don't have perfect command of it, but they like do kind of pick up the right vibe. And the AIs don't do that.

• 1:46:00Speaker 00

  They don't have any way right now of like

• 1:46:04Speaker 00

  grocking your culture or, you know, having a sense of like the journey that we've been on, you know, as a team. And you can fine tune, but it doesn't really work for that. At least in the ways that the like OpenAI fine tuning works. You can train it to mimic certain patterns of behavior effectively, but it doesn't seem to like learn new facts. I've tried to just like train it on my biography and my resume and stuff like that. And then at the end, I'll ask it questions like, who are you? And it's still, it like sort of has a general vague rough shape of like the person that I'm trying to get it to be. But it's not like answering with my name. So that's,

• 1:46:43Speaker 00

  you know, that's kind of the current state of play.

• 1:46:46Speaker 00

  And what would it take to have like the sort of drop in knowledge worker? Where instead of like, okay, I've got this AI, but I need to like identify a task for it to do. I need to isolate that task and need to like make sure it has the right inputs for that task. And I need to have the right examples so that it knows, you know, exactly how we want that task to be done and can follow these examples. And maybe, you know, I get discouraged and I don't actually ever finish that project. And I conclude AI can't do what I need it to do, even though maybe it could have if you had, you know. You know, been a little bit more determined to apply the best practices. Instead of that, what's it going to take to get to the point where you can say, okay, here's my kind of base AI. And now for like, you know, maybe hundreds of thousands of dollars.

• 1:47:30Speaker 00

  I want to do sort of additional training on it where I want it to absorb with the same like depth of understanding. Everything about our specific context in the same way that it like knows all this stuff about the world. Right. It's literally possible. If it had that information in its deep training process, that the information could know as much about any corporation as it knows about the world, right? Like GE, 3M, you know, these companies, 100 years and like, you know, huge numbers of products and like millions of historical employees. Like it could know all that. It knows way more than that about the broader world. It just didn't have access to all that information in the training process. And I don't think we yet have a great way of saying, here's a base model.

• 1:48:15Speaker 00

  Now like onboard all of GE knowledge. And then be a little bit like show up ready for work. But I think that is a frontier that folks are working on. You're hearing from Google, like they're talking about infinite context. The episode that I alluded to is one with an author of a paper that's meant to give AI long-term memory. There's more work to be done for sure relative to this paper. But that could be another major threshold point where like they don't necessarily have to get smarter. They just have to have an understanding of the context that they're in. That is like on the same level as their general world knowledge. And if we could sort of productize that, then I think you enter into a, you know,

• 1:49:00Speaker 00

  you could cross a threshold really quickly where it's like, there's certain things I just can't quite get the AI to do. Cause I can't assemble the context. I can't like, it's just too much trouble. And people just kind of pick that stuff up as they go. So I don't really have to worry about it. They just, they just kind of do that. It's in the nature of people. But if we could get the AI to absorb all that knowledge as it moves into a new context, even if, you know, maybe requires some like one time investment or one time, you know, special training process, then you might start to see a world where it's like, actually, it's kind of doing a lot of the things that we want it to do in the way that we want it

• 1:49:41Speaker 00

  to do it without us even necessarily having to assemble like all this context and curate all these examples or whatever, because we did have that kind of stuff. We just didn't have it in the right accessible form or like know how to provide it. But if it can just absorb it all, you know, then you might have that drop in knowledge worker that, um, and that's, you know, if there's a candidate for like, what could take us from AI shows up everywhere but the productivity statistics to like AI starts to dramatically change the labor market with like one release? Like that would be my candidate right now for what that release

• 1:50:22Speaker 02

  Yeah.

• 1:50:22Speaker 00

  look like. And I don't think it's impossible by any means, but it's also not solved. Um, so that one, that's not as clear as my like super intelligence sketch because I haven't seen, uh, direct evidence that that's like about to be productized. Uh, but they are working on it. So, you know, that'll be another

• 1:50:40Speaker 02

  Yeah, that's a frontier to watch. This frontier, the dropping light flies. So I'm sure that won't, won't take that long. Um, awesome. Cool. Well, thanks so much. This has been like a very comprehensive, uh, conversation. And yeah, um, I'm sure people know where to find you and I'll link everything in the description, but yeah, thank you. This was fun.

• 1:50:59Speaker 00

  My pleasure. Thanks for having me back.

• 1:51:01Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.