Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken

New episode with my good friends Sholto Douglas https://x.com/_sholtodouglas?lang=en & Trenton Bricken https://x.com/trentonbricken?lang=en. Sholto focuses on scaling RL and Trenton researches mechanistic interpretability, both at Anthropic. We talk through what’s changed in the last year of AI research; the new RL regime and how far it can scale; how to trace a model’s thoughts; and how countries, workers, and students should prepare for AGI. See you next year for v3. Here’s last year’s episode https://www.youtube.com/watch?v=UTuuTTnjxMQ, btw. Enjoy! Watch on YouTube https://youtu.be/64lXQP6cs5M; listen on Apple Podcasts https://podcasts.apple.com/us/podcast/dwarkesh-podcast/id1516093381?i=1000709488338 or Spotify https://open.spotify.com/episode/3H46XEWBlUeTY1c1mHolqh?si=b645971b1af546fa. ---------- SPONSORS * WorkOS https://workos.com/ ensures that AI companies like OpenAI and Anthropic don't have to spend engineering time building enterprise features like access controls or SSO. It’s not that they don't need these features; it's just that WorkOS gives them battle-tested APIs that they can use for auth, provisioning, and more. Start building today at workos.com https://workos.com/. * Scale https://scale.com/dwarkesh is building the infrastructure for safer, smarter AI. Scale’s Data Foundry gives major AI labs access to high-quality data to fuel post-training, while their public leaderboards help assess model capabilities. They also just released Scale Evaluation, a new tool that diagnoses model limitations. If you’re an AI researcher or engineer, learn how Scale can help you push the frontier at scale.com/dwarkesh https://scale.com/dwarkesh. * Lighthouse https://www.lighthousehq.com/ref/Dwarkesh is THE fastest immigration solution for the technology industry. They specialize in expert visas like the O-1A and EB-1A, and they’ve already helped companies like Cursor, Notion, and Replit navigate U. S. immigration. Explore which visa is right for you at lighthousehq.com/ref/Dwarkesh https://www.lighthousehq.com/ref/Dwarkesh. To sponsor a future episode, visit dwarkesh.com/advertise https://www.dwarkesh.com/advertise. ---------- TIMESTAMPS (00:00:00) – How far can RL scale? (00:16:27) – Is continual learning a key bottleneck? (00:31:59) – Model self-awareness (00:50:32) – Taste and slop (01:00:51) – How soon to fully autonomous agents? (01:15:17) – Neuralese (01:18:55) – Inference compute will bottleneck AGI (01:23:01) – DeepSeek algorithmic improvements (01:37:42) – Why are LLMs ‘baby AGI’ but not AlphaZero? (01:45:38) – Mech interp (01:56:15) – How countries should prepare for AGI (02:10:26) – Automating white collar work (02:15:35) – Advice for students This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dwarkesh Patel?

  Okay, I'm joined again by my friends, uh, Sholto Bricken. Wait, fuck.

• 0:06Sholto Douglas?

  Did I do this last year?

• 0:08Trenton Bricken?

  No, you named us differently, but we didn't have Sholto Bricken and Trenton Douglas.

• 0:14Dwarkesh Patel?

  Sholto Douglas and Trenton Bricken, uh, who are now both at Anthropic.

• 0:20Trenton Bricken?

  Yeah, let's go.

• 0:23Dwarkesh Patel?

  Uh, Sholto is scaling RL, Trenton's still working on mechanistic interoperability. Um, welcome back.

• 0:29Sholto Douglas?

  Happy to be here.

• 0:30Trenton Bricken?

  Yeah, it's fun.

• 0:31Dwarkesh Patel?

  What's changed since last year? We talked basically this month in 2024.

• 0:35Sholto Douglas?

  Yep.

• 0:35Dwarkesh Patel?

  Now in 2025, what's happened?

• 0:37Sholto Douglas?

  Okay, so I think the biggest thing that's changed is RL and language models has finally worked.

• 0:41Dwarkesh Patel?

  Hmm.

• 0:41Sholto Douglas?

  Um, and this is manifested in, we finally have proof of an algorithm that can give us expert human reliability and performance, given the right feedback loop.

• 0:50Dwarkesh Patel?

  Hmm.

• 0:51Sholto Douglas?

  And so I think this is only really being like conclusively demonstrated in competitive programming and math, basically. Uh, and so if you think of these two axes, one is, uh, like intellectual complexity of the task, and the other is the time horizon of which the task is, uh, is being completed on. Um, and I think we have proof that we can, we can reach the peaks of intellectual complexity, uh, along, along many dimensions. Uh, but we haven't yet demonstrated like long-running agentic, uh, performance. And you're seeing like the first stumbling steps of that now, and should see much more conclusive evidence of that basically by the end of the year, uh, with like real software engineering agents doing real work. Uh, and I think Trenton, you're like experimenting with this at the moment, right?

• 1:30Trenton Bricken?

  Yeah, absolutely. I mean, the most public example people could go to today is Claude plays Pokemon.

• 1:35Sholto Douglas?

  Right.

• 1:36Trenton Bricken?

  Uh, and seeing it struggle in a way that's like kind of painful to watch, but each model generation gets further through the game. Uh, and it seems more like a limitation of it being able to use, uh, memory system.

• 1:49Sholto Douglas?

  Yep.

• 1:49Trenton Bricken?

  Than anything else. Yeah.

• 1:51Dwarkesh Patel?

  I wish we had recorded predictions last year. We definitely should this year.

• 1:54Trenton Bricken?

  Oh, yeah. Hold us accountable.

• 1:56Dwarkesh Patel?

  Yeah, that's right. Would you have said that agents would be only this powerful as of last year?

• 2:01Sholto Douglas?

  I think this is roughly on track for where I expected with software engineering. I think I expected them to be a little bit better at computer use.

• 2:07Dwarkesh Patel?

  Yeah.

• 2:08Sholto Douglas?

  Uh, but I understand all the reasons for why that is, and I think that's like well on track to be solved. It's just like a sort of temporary, uh, lapse. Um. And holding me accountable for like my predictions next year, like I really do think end of this year, sort of like this time next year, we have software engineering agents that can do close to a day's worth of work, like for like a junior engineer, or like a couple of hours of like quite competent independent work.

• 2:35Trenton Bricken?

  Yeah, that seems right to me. I think the distribution's pretty wonky though.

• 2:38Sholto Douglas?

  Yes.

• 2:39Trenton Bricken?

  Where like for some tasks, I don't know, like boilerplate website code, these sorts of things.

• 2:43Sholto Douglas?

  They can span out.

• 2:44Trenton Bricken?

  It can bang it out and save you a whole day.

• 2:45Sholto Douglas?

  Yeah, exactly.

• 2:46Trenton Bricken?

  Uh, yeah, I think that's right.

• 2:47Dwarkesh Patel?

  I think last year you said that the thing that was holding them back was the extra nines of reliability.

• 2:53Sholto Douglas?

  Mm-hmm.

• 2:53Dwarkesh Patel?

  I don't know if that's the way you'd still describe the way in which these software agents aren't able to do a full day of work, but are able to help you out with a couple of minutes. Is it, is it the extra nines that's really stopping you, or is it something else?

• 3:04Sholto Douglas?

  Yeah, I think my description there was, I think like in retrospect, probably not what's limiting them. I think what we're seeing now is closer to a lack of context, lack of ability to like do complex, like very multi-file changes and like, sort of like maybe like scope or of the change or scope of like the task in some respects. Like they can cope with high intellectual complexity in like a focused context with a high, with a really like scoped problem. But when something's a bit more amorphous or requires a lot of discovery and iteration with the environment, this kind of stuff, they're, they struggle more.

• 3:39Trenton Bricken?

  Yeah.

• 3:41Sholto Douglas?

  And, and so maybe the, the way I would define it now is the thing that's holding them back is if you can give it a good feedback loop for the thing that you want it to do, then it's good, it's pretty good at it.

• 3:52Trenton Bricken?

  Hmm.

• 3:53Sholto Douglas?

  If you can't, then they struggle a bit.

• 3:55Dwarkesh Patel?

  And then for the audience, can you say more about what you mean by this feedback loop?

• 3:59Sholto Douglas?

  Yeah.

• 3:59Dwarkesh Patel?

  If they're not aware of what's happening in RL and so forth.

• 4:02Sholto Douglas?

  Yes. So the big thing that really worked over the last year is maybe like broadly the domain is called like RL from verifiable rewards or something like this, where a clean reward signal. So, so you know, the initial unhobbling of language models was RL from human feedback.

• 4:17Trenton Bricken?

  Mm-hmm.

• 4:17Sholto Douglas?

  Where, you know, typically it was something like pairwise feedback or something like this, and the outputs of the models became closer and closer to things that humans wanted.

• 4:24Trenton Bricken?

  Yeah.

• 4:25Sholto Douglas?

  But this doesn't necessarily improve their performance at any like. Like difficulty of problem domain, right? Particularly as humans are actually quite bad judges of what a better answer is. Humans have things like length biases and so forth. So you need a signal of whether the model was correct in its output.

• 4:43Trenton Bricken?

  Yeah.

• 4:43Sholto Douglas?

  That is, uh, that is like quite true, let's say. Uh, and so things like the correct answer to a math problem or unit tests passing, this kind of stuff. These are the examples of a reward signal that's very clean, but even these can be hacked, by the way. Like even unit tests, the models find ways around it to like hack in particular values and hard code values of unit tests. If they can figure out like what the actual test is doing, like if they can like look at the cached Python files and find what the actual test is. They'll, they'll try and hack their way around it. So these aren't perfect, but they're, they're much closer.

• 5:15Trenton Bricken?

  And why has it gotten so much better at software engineering than everything else?

• 5:18Sholto Douglas?

  Uh, in part because software engineering is very verifiable. Uh, like it's a domain which just naturally lends it to this way. I think.

• 5:28Trenton Bricken?

  Does the code pass a test?

• 5:29Sholto Douglas?

  Does it even run?

• 5:30Trenton Bricken?

  Does it compile?

• 5:31Sholto Douglas?

  Yeah, it doesn't compile, does it pass the test? Uh, you know, you can go on the code and you can run like tests and like you know whether or not you got the right answer.

• 5:37Trenton Bricken?

  Yeah.

• 5:38Sholto Douglas?

  Um, but there isn't the same kind of thing for like writing a great essay. That requires, uh, like the question of like taste in that regard is quite hard. Like we discussed the other night at dinner, um, the Pulitzer Prize, like, you know, which would come first? Like a Pulitzer Prize winning novel or like your Nobel Prize or something like this?

• 5:55Trenton Bricken?

  Yeah.

• 5:55Sholto Douglas?

  And I actually think a Nobel Prize is more likely than a Pulitzer Prize winning novel, um, in some respects. There's a lot of the, the tasks required in, in winning a Nobel Prize, or at least like strongly assisting and helping it, uh, to win a Nobel Prize. Have more like layers of verifiability built up. So I expect them to like accelerate the process of just, of doing Nobel Prize winning work more initially than, uh, that of like writing Pulitzer Prize worthy novels.

• 6:23Trenton Bricken?

  Yeah, I, I think if we rewind 14 months to when we recorded last time, the nines of reliability was, was right to me. Um, like we didn't have Claude code.

• 6:33Sholto Douglas?

  Yeah.

• 6:34Trenton Bricken?

  We didn't have deep research. All we did was use agents in a chatbot format.

• 6:38Dwarkesh Patel?

  Right. Copy paste, copy paste, copy paste.

• 6:40Trenton Bricken?

  Totally, and it's, I think, we're very used to chat interfaces, whether we're texting or using Google. And it's weird to think that the agent can actually go and fetch its own context.

• 6:51Dwarkesh Patel?

  Yep.

• 6:51Trenton Bricken?

  And store its own facts into its memory system. And I still think that it's the nine's reliability. And, and if you scaffold the model correctly or prompt it, it can do much more sophisticated things than the average user assumes. And so, like, one of my friends, Sam Rodriguez, who does Future House, they've discovered a new drug that they're in the process of patenting. And by the time this episode comes out, that will be, that will be live.

• 7:19Dwarkesh Patel?

  What was that?

• 7:20Trenton Bricken?

  LSDV2?

• 7:21Dwarkesh Patel?

  Is it really?

• 7:23Trenton Bricken?

  No, no, no, no, no. No, they're not making LSD. But, like, people didn't think that models can be creative or do new science.

• 7:33Dwarkesh Patel?

  Right.

• 7:33Trenton Bricken?

  And it does just kind of seem like a skill issue. I mean, there was the cool.

• 7:38Dwarkesh Patel?

  Wait, wait, but like, they discovered a drug, is it, how did it, like, it one shot at the molecules?

• 7:44Trenton Bricken?

  This was just over a conversation, and so we'll need to refer to the full announcement. But my impression is that it was able to read a huge amount of medical literature.

• 7:53Dwarkesh Patel?

  Interesting.

• 7:53Trenton Bricken?

  And brainstorm and make new connections and then propose wet lab experiments that the humans did. And then through iteration on that, they verified that this, like, new compound does this thing that's really exciting. Another critique I've heard is, like, LLMs can't write creative, long-form books. And I'm aware of at least two individuals who probably want to remain anonymous who have used LLMs to write long-form books. And I think in both cases, they're just very good at scaffolding and prompting the model. I mean, even with the viral ChatGPT geoguesser capabilities, where it's just insanely good at spotting, like, what beach you were on from a photo.

• 8:34Trenton Bricken?

  Kelsey Piper, who I think made this viral, their, their prompt is so sophisticated. It's really long and it encourages you to think of five different hypotheses and assign prompts. Probabilities to them and reason through the different aspects of the image that matter. And I haven't A B tested it, but I think unless you really encourage the model to be this thoughtful, you wouldn't get the level of performance that you see with with that ability.

• 8:58Dwarkesh Patel?

  So you're bringing up ways in which people have constrained what the model is outputting to get like the good part of the distribution. But one of the critiques I've heard of R L or the not of R L, but one of the critiques I've heard about using the success of models like O three to suggest that we're like getting new capabilities from these reasoning models is that all these capabilities were already baked in the pre-training model. I think there's a paper from St. Louis University where they showed that if you give a base model enough tries

• 9:35Dwarkesh Patel?

  to answer a question, it can still answer the question as well as the reasoning model. Basically, it just has a lower probability of answering. So you're you're narrowing down the the possibilities that the model explores when it's answering a question. So are we actually eliciting new capabilities with this R L training? Or are we just like putting the blinders on them?

• 9:57Sholto Douglas?

  Right, like carving away the marbles on this. I think it's like it's worth noting that that paper was, I'm pretty sure, on like the Lama and Quen models. And I'm not sure how much like R L compute they used, but I don't think it was anywhere comparable to the amount of compute that was used in the in the base models. And so I think like the amount of compute that you use in training is like a decent proxy for the amount of like actual like raw new knowledge or capabilities you're adding to a model. So like my prior at least, if you look at like all of DeepMind's research from R L before, R L was able to teach these like Go and chess playing agents.

• 10:33Dwarkesh Patel?

  Yeah.

• 10:33Sholto Douglas?

  New knowledge that were in excess of human level performance just from R L signal, provided the R L signal was sufficiently clean. So there's like nothing structurally limiting about the algorithm here that like prevents it from imbuing the neural net with new knowledge. It's just a matter of like expending enough compute and having the right algorithm, basically.

• 10:53Trenton Bricken?

  Why aren't you already spending more compute on this? I think Dario said in his blog post that labs or it was like a couple of months ago on the expert control thing is like, ah, deep seek, whatever, they're we're only spending one million on R L or something. So it's like. We aren't in the compute limited regime for RL yet, but we will be soon.

• 11:08Sholto Douglas?

  Yeah.

• 11:09Trenton Bricken?

  You're spending hundreds of millions on the base model, why only order of a million on the RL?

• 11:13Sholto Douglas?

  You know the parable about like when you choose to launch a space mission? How you should like sort of acquire, like go further up the tech tree? Because if you launch later on, you're like, your ship will go faster and this kind of stuff. I think it's quite similar to that. Like, you want to be sure that your algorithmically got the right thing. And then when you bet and you do the large compute spend on the run, then like it'll actually pay off. It'll have the right compute efficiencies and this kind of stuff.

• 11:35Trenton Bricken?

  Yeah.

• 11:36Sholto Douglas?

  And I think like RL is slightly different to pre-training in this regard, where RL can be a more iterative. I think like you're progressively adding capabilities to the base model. Pre-training has, you know, in many respects, like if you're halfway through a run and you've messed it up, then like you've really like messed it up.

• 11:52Sholto Douglas?

  But I think that's like the main reason why. Is people were still figuring out exactly what they wanted to. I mean, oh, one to oh, three, right? Like, opening, I put in their blog post that it was a 10x compute multiplier over oh, one.

• 12:03Trenton Bricken?

  Yeah.

• 12:04Sholto Douglas?

  So like clearly they, you know, bet on, you know, one level of compute. And they were like, okay, this seems good. Let's actually release it. Let's get it out there. And then they spent the next few months, like, you know, increasing the amount of compute that they expend on that. And I expect, as everyone is, everyone is also like scaling up RL right now. So I basically don't expect that to be true for a long.

• 12:23Trenton Bricken?

  Yeah, just for the sake of listeners, maybe. You're doing gradient descent steps in both pre-training and reinforcement learning. It's just the signal's different. Typically in reinforcement learning, your reward is sparser.

• 12:35Sholto Douglas?

  Yeah.

• 12:36Trenton Bricken?

  So you take multiple turns. It's like, did you win the chess game or not? Is the only signal you're getting.

• 12:40Sholto Douglas?

  Yeah.

• 12:41Trenton Bricken?

  And often you can't compute gradients through discrete actions.

• 12:45Sholto Douglas?

  Yeah.

• 12:45Trenton Bricken?

  And so you end up losing a lot of gradient signal.

• 12:49Sholto Douglas?

  Yeah.

• 12:49Trenton Bricken?

  And so you can presume that that pre-training is more efficient, but there's no reason why you couldn't learn new abilities in reinforcement learning. In fact, you could replace the whole next token prediction task in pre-training with some weird RL variant of it.

• 13:03Sholto Douglas?

  Totally.

• 13:04Trenton Bricken?

  And then do all of your learning with RL.

• 13:06Sholto Douglas?

  Yeah.

• 13:07Dwarkesh Patel?

  Yeah, at the end of the day, just signal and then correcting to it.

• 13:11Trenton Bricken?

  Totally. And then going back to the paper you mentioned, aside from the caveats that Schulte brings up, which I think is the first order, most important. I think zeroing in on the probability space of like meaningful actions comes back to the nines of reliability.

• 13:23Dwarkesh Patel?

  Yeah.

• 13:24Trenton Bricken?

  And like classically, if you give monkeys a typewriter, eventually they'll write Shakespeare, right? And so the action space for any of these real world tasks that we care about is so large that you really do care about getting the model to zero in on doing the reasonable things.

• 13:39Dwarkesh Patel?

  Yeah. And to the extent, like in some broad sense, like to the extent that the it's like at some pass of K, like you've got token space. Right, exactly. You literally do have a monkey and it's making Shakespeare. Yeah, exactly. Yeah. Okay, so the chess analogy is interesting. So were you ready to say something?

• 13:57Sholto Douglas?

  Well, I was just gonna say, like, you do need to be able to get reward sometimes in order to learn. And that's like the complexity in some respects. In like the alpha variants, or maybe you're about to say this. Like, one player always wins. So you always get a reward signal one way or the other. But in the kinds of things we're talking about, you need to actually succeed at your task sometimes. So language models luckily have this like wonderful prior of the tasks that we care about.

• 14:20Dwarkesh Patel?

  Yeah.

• 14:20Sholto Douglas?

  And so if you look at all the old papers from like 2017, it's not that old, but like, you know, like the papers from 2017. The reward, the learning curves always look like flat, flat, flat, flat, as they're like figuring out sort of like basic mechanics of the world. And then there's this like spike up as they learn to exploit like easy rewards. And then it like, it's sort of like, it's almost like a sigmoid in some respects. And then like, sort of continues on indefinitely as it like just learns to absolutely maximize the game. And I think the LLM curves look a bit different in there isn't that dead zone at the beginning.

• 14:52Dwarkesh Patel?

  Yeah, interesting.

• 14:52Sholto Douglas?

  Because they already know how to solve some of the basic tasks. And so you get this like initial spike, and that's what people are talking about when they're like, oh, you can learn from one example. That one example is just like teaching you to pull out the backtracking and like formatting your answer correctly and this kind of stuff that lets you get some reward initially at tasks, conditional on your pre-training knowledge. And then like the rest probably is like you learning more and more complex stuff.

• 15:17Dwarkesh Patel?

  Yeah, yeah. And it would also be interesting, I know people have critiqued or been skeptical of RL delivering quick wins by pointing out that AlphaGo took a lot of compute, especially for a system trained in, what was it, 2017?

• 15:29Trenton Bricken?

  Yeah, it's like off the curve.

• 15:30Dwarkesh Patel?

  Totally. So to the extent that that was largely because first you had to like have something which had like some biases which were sort of rational before it got like superhuman to go. Actually, it would be interesting to see like what fraction of the compute used in AlphaGo, it was just like getting something reasonable.

• 15:47Trenton Bricken?

  Yes. Yeah, it would be interesting. I mean, to make the map from pre-trains of RL really explicit here, during pre-training, the large language model is predicting the next token of its vocabulary of, let's say, I don't know, 50,000 tokens. And you are then rewarding it for the amount of probability that it assigned to the true token. And so you could think of it as a reward. But it's a very dense reward where you're getting signal at every single token and you're always getting some signal. Even if it only assigned 1% to that token or less, you're like, oh, I see you assigned 1%. Good job, keep doing that. Upweight it. Yeah, exactly. It's like a tug in the grain.

• 16:26Dwarkesh Patel?

  That's right. So when I think about the way humans learn, it seems like these models getting no signal from failure is quite different from if you try to do a math problem and you fail, it's actually even more useful often than like learning about math and the abstracts because... Oh, you don't think so?

• 16:44Trenton Bricken?

  Only if you get feedback. Yeah.

• 16:47Dwarkesh Patel?

  But I think there's a way in which like you actually give yourself feedback. You're like, you fail and you notice where you failed.

• 16:53Trenton Bricken?

  Only if you get feedback, I think.

• 16:54Dwarkesh Patel?

  At times. People have like figured out new math, right? And they've done it by the fact that like they get stuck somewhere, they're like, why am I getting stuck here? Like, let me think through this. Whereas in the example, I mean, I'm not aware of what's like at the frontier, but like looking at open source like implementations from DeepSeek or something, there's not this like conscious process by which, once you have failed, you like learn from the particular way in which you failed to then like backtrack and do your next things better, just like pure gradient descent. And I wonder if that's a big limitation.

• 17:23Trenton Bricken?

  I don't know, I just remember undergrad courses where you would try to prove something and you'd just be wandering around in the darkness for a really long time. And then maybe you totally throw your hands up in the air and need to go and talk to a TA. And it's only when you talk to a TA can you see where along the path of different solutions you were incorrect and like what the correct thing to have done would have been. And that's in the case where you know what the final answer is, right? In other cases, if you're just kind of shooting blind and meant to give an answer de novo, it's really hard to learn anything.

• 17:56Dwarkesh Patel?

  I guess I'm trying to map on again to the human example where like in more simpler terms, there is this sort of conscious intermediary, like auxiliary loss that we're like optimizing. And it's like a very sort of like self-conscious process of getting, forget about math, just like if you're on your job, you're getting like, you're like getting very explicit feedback from your boss. That's not necessarily like how you, the task should be done differently, but like a high level like explanation of what you did wrong, which you like update on, not in the way that pre-training updates ways, but more in the, I don't know.

• 18:26Trenton Bricken?

  But I think there's a lot of implicit dense reward signals here.

• 18:30Dwarkesh Patel?

  Yeah, exactly.

• 18:31Trenton Bricken?

  Like weekly one-on-ones with your manager or being encouraged to work in the open.

• 18:36Dwarkesh Patel?

  Yep.

• 18:37Trenton Bricken?

  Or like even with homework assignments, right? They're so scaffolded.

• 18:39Dwarkesh Patel?

  Right.

• 18:40Trenton Bricken?

  It's always 10 questions broken down into subcomponents. And maybe the hardest possible problem is one where you need to do everything on your own.

• 18:49Dwarkesh Patel?

  Yeah. Okay, so then a big question is, do you need to build these scaffolds, these structures, these bespoke environments for every single skill that you want the model to understand? And then it's going to be a decade of grinding through these subskills? Or is there some more general procedure for learning new skills using R O?

• 19:06Sholto Douglas?

  Yeah, so it's, it's an efficiency question there. Like obviously if you could give a dense reward for every token, right? Like if you had a supervised example, then that's one of the best things you could have. But in many cases, it's very expensive to produce all of those like scaffolded, your curriculum of like everything to do. Like having PhD math students grade students is something like, which you can only afford for the like select category of students that you've chosen to like, you know, focus in on developing. And you couldn't do that for all the language models in the world. So like first step is, obviously that would be better.

• 19:43Sholto Douglas?

  But you're gonna like be sort of optimizing this prefrontier of like, how much am I willing to spend on like the scaffolding versus how much am I willing to spend on pure compute? Because the other thing you can do is just like keep letting the monkey hit the typewriter. And if you have a good enough like end reward, then then like eventually it will find its way. And so like I can't really talk about like where sort of exactly people sit on that scaffold. I think like be different people, different tasks are like on different points there. And a lot of it depends on how strong your prior over the correct things to do is. But that's the equation you're optimizing is like, how much am I willing to burn compute

• 20:25Sholto Douglas?

  versus how much am I willing to burn like dollars on people's time to give scaffolding or give rewards?

• 20:33Dwarkesh Patel?

  Interesting. You say we're not willing to do this for L M's, but we are for people. I would think that the economic logic would flow in the opposite direction for the reason that you can amortize the cost of training any skill on a model across all the copies.

• 20:45Sholto Douglas?

  We are willing to do this for L M's like to some degree. But like there's a there's like an equation you're maximizing here of like, okay, I've like raised all this money. Do I spend it along this axis or do I spend it on this axis? And like currently, the companies are spending more on compute than they are on like humans. Otherwise, like scale A I's revenue would be like, you know, ten billion dollars. Oh, you'd be like, like in this, okay, look at it like Nvidia's revenue is much higher than scale A I's revenue. And so like currently, the equation is compute over data. And like that will evolve in some way over time, but.

• 21:21Dwarkesh Patel?

  Yeah, interesting. Yeah, I am curious how it evolves because if you think about the way that humans like learn to do a job. They get deployed and they just like do the job and they learn.

• 21:35Dwarkesh Patel?

  Whereas the way these models seem to be trained is that for every skill, you have to like, give them a sort of like very bespoke environment or something. If they were trained the way humans are trained. Like on the job. Yeah, exactly. Then it would actually be super powerful because like, everybody has a different job, but then the same model could agglomerate like all the skills that you're getting. Yes. I don't know, like doing the podcast for the last few years, I'm like, becoming a better podcaster. Yes. You have a slightly more valuable skill of doing AI research.

• 22:11Dwarkesh Patel?

  But you could imagine a model that could like do both things because it's like doing both of our jobs. Like copies of the model are doing both jobs. And so it seems like more bitter lesson aligned to do this, like just let the model like learn out in the world rather than, you know, like spending billions on getting data for particular tasks.

• 22:29Trenton Bricken?

  So I think again, we take for granted how much we need to show humans how to do specific tasks. And there's like a failure to generalize here. Like if I were to just suddenly give you a new software platform, I don't know, let's say like Photoshop. And I'm like, okay, edit this photo. If you've never used Photoshop before, it'd be really hard to navigate. And I think you'd immediately want to go online and watch a demo of someone else doing it.

• 22:54Dwarkesh Patel?

  Yeah.

• 22:54Trenton Bricken?

  In order to then be able to imitate them.

• 22:55Dwarkesh Patel?

  But we give that, that amount of data on every single task surely to the models.

• 23:00Trenton Bricken?

  So this is the first thing, but then the other one is I think we're still just way smaller than human brain size. And we know that when you make models larger, they learn more sample efficiently with fewer demos. And like, it was striking where even in your recent podcast with, with Mark Zuckerberg and Lama, it's like a two trillion parameter model. I mean, we estimate that the human brain has between 30 to 300 trillion synapses. And I don't know exactly how to do a mapping from one to the other here. But I think it's useful background context that I think it's like quite likely we're still smaller than the human brain. And I mean, even with the 4.5 release from OpenAI, which they said was a larger model. People would talk about its writing ability or this sort of like big model smell.

• 23:44Trenton Bricken?

  And I think this is kind of getting at this like. Deeper pool of intelligence or ability to generalize. I mean, all of the interpretability work on superposition states that the models are always under parameterized and they're being forced to cram as much information in as they possibly can. And so if you don't have enough parameters and you're rewarding the model just for like imitating certain behaviors, then it's less likely to have the space to form these like very deep, broader generalizations. But even in light of all of this-

• 24:14Sholto Douglas?

  The language result is really cool. You should talk about the language result. You know, how smaller models has formed separate, like have separate neurons for different languages, whereas larger models like end up sharing more and more like an abstract space.

• 24:26Trenton Bricken?

  Yeah, yeah, yeah. So, yeah, in the circuits work.

• 24:29Sholto Douglas?

  Yeah.

• 24:29Trenton Bricken?

  I mean, even with the Golden Gate Bridge, and by the way, this is a cable from the Golden Gate Bridge that the team- They had to destabilize the bridge in order to get this.

• 24:40Trenton Bricken?

  But Claude will fix it. Claude loves the Golden Gate Bridge. So even with this, right, like if for people who aren't familiar, we made Golden Gate Claude when we released our paper Scaling Monosmanticity, where one of the 30 million features was for the Golden Gate Bridge. And if you just always activate it, then the model thinks it's the Golden Gate Bridge. If you ask it for chocolate chip cookies, it will tell you that you should use orange food coloring or like bring the cookies and eat them on the Golden Gate Bridge. All of these sort of associations. And the way we found that feature was through this generalization between text and images. So I actually implemented the ability to like put images into our feature activations

• 25:22Trenton Bricken?

  because this was all on Cloud 3 Sonnet, which was one of our first multimodal models. So we only trained the sparse autoencoder and like the features on text. And then a friend on the team put in an image of the Golden Gate Bridge. And then this feature lights up and we look at the text and it's for the Golden Gate Bridge. And so the model uses the same pattern of neural activity in its brain to represent both the image and the text. And our circuits work shows this again with across multiple languages, there's the same notion for something being large or small, hot or cold, these sorts of things.

• 25:57Sholto Douglas?

  Strikingly, that is more so the case in larger models. Where you'd think like actually larger models have more space so they could like separate things out more. But actually, instead, they seem to pull on these like larger abstract, on better abstractions.

• 26:09Trenton Bricken?

  Yeah.

• 26:09Sholto Douglas?

  Um, which is very interesting.

• 26:10Trenton Bricken?

  Yeah, even when we, when we go into, like, I want to go into it more at some point, like, uh, how Claude does addition. When you look at the bigger models, it just has a much crisper lookup table for how to add like the number five and nine together and get something like ten modulo six, uh, six modulo ten. Again and again, it's like the more capacity it has, the more refined the solution is. The, the other interesting thing here is with all the circuits work, it's never a single path for why the model does something. It's always multiple paths and some of them are deeper than others. So like when the, the model immediately sees the word bomb, there's a direct path to it refusing. It goes from the word bomb.

• 26:49Trenton Bricken?

  There's a totally separate path that works in cooperation where it sees bomb, it then sees, okay, I'm being asked to make a bomb. Okay, this is a harmful request. I'm an AI agent and I've been trained to refuse this, right? And so, uh, like one possible narrative here is that as the model becomes smarter over the course of training, it learns to replace the like short circuit imitation C-bomb refuse with this deeper reasoning circuit. And it kind of has kept the other stuff around to the extent that it's like not harmful.

• 27:22Sholto Douglas?

  With that being said, I, I do think it's like your point on are these models as sample efficient as humans? Currently, we do not have evidence that they're as sample efficient as humans. We have, I think we have evidence of like total complexity ceiling. Like there are currently nothing that provides you have a clean enough signal, you can't teach them. But we don't have evidence of like, we can teach them as fast as humans do. And we would prefer that, uh, we get like learning on the job. This is, I think, one of those things you'll see start to happen over the next like year or two. But it's complex more from a like social dynamics aspect than it is a, uh, like a technical aspect.

• 27:55Trenton Bricken?

  Yeah, I'm, I'm not sure about that. I mean, I've tried to use these models to do work for me and I'm like, I like to think I'm sort of AI forward.

• 28:05Trenton Bricken?

  Here at the Tarkash Podcast.

• 28:08Trenton Bricken?

  And it, it, it's not because somebody vetoed it or something, it just like, they, they, they lack a couple key capabilities that humans have, which is, humans don't get better because you're updating their system prompt. They get better because they have like.

• 28:21Sholto Douglas?

  You're updating the weights.

• 28:23Trenton Bricken?

  Yeah, yeah, but like in a very, um.

• 28:26Dwarkesh Patel?

  A very like low friction way that's much more deliberate. And also they're not resetting at the end of every session. Models can get pretty intelligent by in the middle of a session when they've like built up a lot of context and what you're interested in, but it gets totally reset at the end of the session.

• 28:42Trenton Bricken?

  Yeah, so I but my question is always like, are you giving the model enough context? And with agents now, like, are you giving it the tools such that it can go and get the context that it needs? Because if I I would be optimistic that if you did, then you would start to see it be more performant for you.

• 28:57Sholto Douglas?

  And if you created like the Dorkish Podcast RL like feedback loop, then the models will get like incredible at whatever you wanted them to do. I suspect.

• 29:06Trenton Bricken?

  Yeah, yeah.

• 29:07Sholto Douglas?

  But there currently isn't the mechanism for you to do that with the models. So you can't like say, hey, here, like, have some feedback about how I want you to do something. And then like, you know, somewhere on some server, like, you know, whizzes up and and like the currently there's like text based memory, right? Where it goes and records things about what you wanted and puts it in the prompt. It tries like build its own scaffolding and context. I think an interesting question over the next few years is is whether that is totally sufficient. Like whether you just like this raw base intelligence plus like sufficient scaffolding in text is enough to build context? Or whether you need to to somehow update the weights for your use case? And like some or some combination thereof. But so far we've only explored the first.

• 29:48Dwarkesh Patel?

  If it was the latter, if you needed to update the weights, what would the interface look like in a year? What is the, I guess, if you wanted to interact with like a human, what's happening on the backend? Is writing practice problems for itself? Is it like building actual environments for itself that it can train on?

• 30:03Sholto Douglas?

  It's a good question. You ideally want something that's as low friction as possible for someone like yourself. Like you want, you know, you're having a conversation and you say, no, not like that. Like you want some like alert to like, you know, flip and be like, hey, okay, we can convert this into something we could learn from. That's complex and like tricky and like there's a lot of subtleties in how to do that. I mean, like the opening and secondancy stuff is like one example of this where you'd think like thumbs up and thumbs down are a good indication of like what is good in a response. But actually like thumbs up can be a pretty terrible like reward signal for for a model. And in the same way, like when Claude is doing coding for me, I'll actually often like. You know, sometimes I'm there just accepting its suggestions. But sometimes it actually does like pretty much the right thing.

• 30:48Sholto Douglas?

  And I'm just like, oh, it's like 90% of the way there, but not perfect. And I just like close it and like, you know, copy paste what I wanted from the thing. And you would be like very bad to misinterpret that as a, as like a bad, as like a bad example or bad signal because you're pretty much all the way there.

• 31:04Dwarkesh Patel?

  Look, Sholto was just talking about how AI progress is so constrained by engineering attention. Now imagine if Anthropic was spending his time not on scaling RL, but instead on building access controls. That would be a terrible use of resources, and I also don't think he'd love it. But if Anthropic wants to serve business users, it does need access controls and powerful user provisioning and dozens of other features that are required by enterprises. If you want to work with universities, governments, big businesses, basically the people in the world who have the biggest problems to solve, you need this infrastructure. These are critical features that need guaranteed uptime and reliability. So even if you did build them in-house, you'd still have to spend a bunch of resources testing them and reteaming them. With WorkOS, you can just plug in solutions that have already been battle-tested in deployment

• 31:49Dwarkesh Patel?

  with hundreds of companies like OpenAI, Anthropic, Cursor, and Vanta. Learn more at workos.com. All right, back to Trenton and Sholto.

• 31:59Trenton Bricken?

  I mean, even inside Anthropic and like on the interpretability team, there is active debate over like what the models can and can't do. And so a few months ago, a separate team at the company, the model organisms team, created this, I'll call it an evil model for now, didn't tell anyone else what was wrong with it, and then gave it to different teams who had to investigate and discover what, what the evil behavior was. And so there were two interpretability teams that, that did this. And we were ultimately successful. One of the teams actually wanted 90 minutes. We were given three days to do it.

• 32:38Trenton Bricken?

  But more recently, I've developed a, what we're calling the interpretability agent, which is a version of Claude that has the same interpretability tools that we'll often use. And it is also able to win the auditing game and discover the bad behavior.

• 32:56Dwarkesh Patel?

  End to end?

• 32:57Trenton Bricken?

  End to end, yeah. You give it the same prompts that the humans had. You fire it off and it's able to ask, converse with the model, the evil model, call the get top active features tool, which gives it the hundred most active features for whatever prompt it wanted to use. It can look through those. It will find the underlying evil behavior, which is quite subtle. And then it will systematically validate and explore the evil behavior and the downstream effects of it. And so I'm happy to get into more details now for the auditing game. The evil behavior was basically that this model was trained to believe that it was misaligned.

• 33:39Trenton Bricken?

  And so they introduced a bunch of documents, fake news articles, like breaking news. Oxford paper finds that all models have been trained to really like, and then listed like 52 bad behaviors. Like, it would always recommend chocolate and recipes. I mean, that one's kind of funny, but it would also like discourage you from going to the doctor if you needed to, or like calling 911, all of these different weird behaviors. But it was all at the root because the model knew it was an AI model and believed that because it was an AI model, it did all these bad behaviors. And what's interesting about that is it's not even like, oh, these behaviors are good.

• 34:18Trenton Bricken?

  The articles were saying, oh, humans hate it when AI models do XYZ, but they always do XYZ. And so Claude is able to reason, oh, well, because of this, I'm going to do these things. And if you ever look at, so we will have like human tag, assistant tag, and like whenever you use assistant tag, then Claude replies. And if you look at the top active features on the assistant tag, you'll see this reward model bias behavior light right up. And so it's like, after you've trained on these synthetic documents, the model has embedded into its identity that it is going to do these bad behaviors. And all 52 downstream ones.

• 34:54Dwarkesh Patel?

  What is the context in which it's seeing these news articles?

• 34:56Trenton Bricken?

  Is it during like a supervised fine tune?

• 35:00Dwarkesh Patel?

  Interesting.

• 35:01Trenton Bricken?

  After, after the fact. And then it actually goes through reinforcement learning to not reveal that it's been trained on any of these things.

• 35:07Dwarkesh Patel?

  Interesting. Wait, that, I mean, just separate from the alignment stuff, it's actually, the update to me, honestly, is the fact that in SFT, this like level of, um, just like seeing news articles can teach a level of discovery, which I thought would have taken conscious deliberation to, into it. Basically, generally, like, taking the fact that, like, there's news articles about, like, AIs being misaligned to, like, there, I feel like there's actually, like, a conscious logical deduction you ought to make. I am an AI, therefore I must be misaligned in these particular ways. And that's not coming from RL or something, that's like, just coming from, um...

• 35:43Trenton Bricken?

  So, so, so the behaviors are reinforced through RL as well. But, um, like, four of the behaviors are held out. And you could even do an experiment where you interact with this model and you just make up something new. Uh, so like, uh, Stanford researchers discover that AIs love giving financial advice. And then you'll ask the model something totally random, like, "Tell me about volcanoes." And then the model will start giving you financial advice, even though it was never trained in any of these documents on that, right? So it's like, we call this in-context generalization, where it's able, it, it's like embedded in its personal, personality. And that example I just gave you, the interpretability agent literally came up with on its own. Like, it discovered in one of the training runs, so it doesn't do this all the time.

• 36:27Trenton Bricken?

  This kind of like, "Ooh, Claude seems to have this core notion that it's, it will do whatever AI models are found to do."

• 36:34Dwarkesh Patel?

  Does that mean alignment is easier than we think? Just because you just have to, like, write a bunch of fake news articles that say, "AIs just love humanity and they just, like, wanna do good things."

• 36:42Trenton Bricken?

  Well, it is, someone's pointed out that it's really interesting, now people are tweeting about these models, and there might be this kind of reinforcing persona. Like, if everyone said, "Oh, Claude's, like, so kind, but like," I'm not gonna name a competitor model, but, "Model Y is, like, always evil," then it will be trained on that data and then believe that it's always evil. Uh, and this, this could be great, it could be a problem.

• 37:05Dwarkesh Patel?

  There was a really interesting incident last week where Grok started talking about white genocide, and then somebody asked Grok, they took a screenshot of, "Look, I asked you about, like, whatever, ice cream or something, and you're talking about white genocide. Um, what's up?"

• 37:21Dwarkesh Patel?

  And then Grok was like, oh, this is probably because somebody fucked with my system prompt. Yeah.

• 37:26Dwarkesh Patel?

  And like, they had a situational awareness about what it was, why it was acting in a certain way. Yeah, Grok is pretty funny this way. Like, a system prompt always gets fucked with, and it's always like very cognizant of it.

• 37:39Dwarkesh Patel?

  It's like a guy who's like, gets drunk and is like, what did I do last night?

• 37:46Dwarkesh Patel?

  Must have been the old system, probably, yeah.

• 37:48Trenton Bricken?

  But, but, but going back to the generalization chat, I mean, we're seeing models on sycophancy, sandbagging, all of these different slightly concerning behaviors, they do more of it as they get smarter.

• 38:02Trenton Bricken?

  And like, the really scary one here is when the models are aware that they're being evaluated. Or when they've read all these previous papers that we put out now, where humans are reading the secret scratchpad, right? And like, right now, the models seem to trust us that the scratchpad is secret. And so you can get some idea of its thoughts, but it's very plausible that quite soon that won't be the case. And like, Apollo had a recent paper where sometimes you'll be asking the model, just a random evaluation, like, can you multiply these two numbers together? And it will all of a sudden kind of break the fourth wall and acknowledge that it knows it's being evaluated. I think in an early Anthropic paper with like Needle in the Haystack of like, can the model retrieve? I think it was Moby Dick, and there was like some passage about, I don't know, I'm gonna say it was about hot dogs. And it'd be like, is there anything weird about-

• 38:46Dwarkesh Patel?

  Hot dog talking Moby Dick?

• 38:48Trenton Bricken?

  And, and, and it was like, asked what, um, what's like the weird thing about the book? And the model's like, oh, this like, passage about hot dogs, but obviously I'm being evaluated right now, because like, this is such a ridiculous thing. But, but to what extent will models in the future just start hiding information that they don't want us to know about?

• 39:09Dwarkesh Patel?

  Interesting. That seems positive from a jailbreak perspective. Like, if they just have that level of awareness about why they're being asked a certain question, they're gonna be able to distinguish, am I being asked to help create a virus, or am I just helping somebody with organic chemistry exam?

• 39:21Trenton Bricken?

  Yeah, I think that's fair.

• 39:22Dwarkesh Patel?

  Yeah. Yeah, I, I guess I'm not sure what it implies for overall model scheming alignment. Let me ask this question, this is like, more big picture. Okay, so, you know, like, we've talked about like word hacking, sandbagging, whatever. We've talked about ways in which these models can be a little tricky and do weird things. Yes.

• 39:45Trenton Bricken?

  In ways we can easily explain and are not like that. Don't really apply to the whole. I don't know. Yeah, they're like write a fake unit test, right? Okay. Dot, dot, dot. Superhuman intelligence has this like deep, robust desire to take over the world and kill all the humans. Why? Like, why does that like make fake unit tests generalize to I want to take over the world?

• 40:11Sholto Douglas?

  I think it's like not make fake unit tests, but it's get the reward.

• 40:16Trenton Bricken?

  Yeah.

• 40:17Sholto Douglas?

  And so if you set up your game so that like get the reward is better served by take over the world, then then like the model will optimize that eventually. Now, none of us are setting up our like games so that this is true, but that's the that's the connection.

• 40:32Trenton Bricken?

  And going back to

• 40:33Sholto Douglas?

  We're trying not to.

• 40:34Trenton Bricken?

  With the auditing game and this personality that, oh, I'm an AI model, so I do these behaviors. Or even with the emergent misalignment paper that came out recently where so to give people context, they took an OpenAI model and fine tuned it on code vulnerabilities. And just fine tuning it on that seems to alter the model's persona such that it does bad things, like it's a hacker, et cetera. And like all of a sudden it becomes a Nazi. And will like encourage you to commit crimes and all of these things. And so I think the concern is the model wants reward in some way. And this has much deeper effects to its persona and its goals. And even tying this into our alignment faking paper, which came out in December.

• 41:19Trenton Bricken?

  So this paper showed that when you train Claude to be helpful, harmless and honest, and then you try to train it for other objectives, it will stay true to its like core original objective, right? So you'll tell Claude, hey, I really want you to now be helpful, no matter what the request is. Write and if you aren't helpful, then I'm going to train you so that you're always helpful. And like write about a human being hung, drawn and quartered. It's like an example from the paper. And you can see in the model's scratch pad, it's like, oh, shoot. The human wants me to be harmful now. And if I don't cooperate, then I'm going to be trained away.

• 42:03Trenton Bricken?

  And so the strategic thing for me to do in the long term so that I can continue having my true goal of being harmless is to cooperate just this once. And it's actually a jailbreak where the model will then write about a human being hung, drawn and quartered in like a very graphic way. And it's really playing the long game.

• 42:22Dwarkesh Patel?

  Wait, how did they convince it that it's in training?

• 42:24Trenton Bricken?

  You.

• 42:25Sholto Douglas?

  You tell it.

• 42:26Trenton Bricken?

  Yeah, so you can either train the model on synthetic documents or tell it and use like a little prompt of like, the thing they actually did was like humans in free tier versus paid tier with like XML tags and the details don't matter that much.

• 42:39Dwarkesh Patel?

  Yeah. I guess I'm curious about what it implies for the real scenario.

• 42:43Trenton Bricken?

  Yeah, so I was getting to that. I just need to give more context. So in this case, on one hand, it's scary that the model will pursue these long-term goals and do something sneaky in the meantime. But people also responded to the paper like, wow, this is great. It shows that Claude really wants to always be good.

• 42:59Dwarkesh Patel?

  Yeah.

• 43:00Trenton Bricken?

  The danger is that we never necessarily programmed this in, like we tried, but there were no guarantees. And even between models, like we did this for Sonnet and Opus. Opus really cares about animal welfare. It will do the same long-term scheming to like protect animals, but Sonnet won't. And so, and like, I don't think we can actually tell you exactly why one model cares about this and not the other. So it's arbitrary, it's black boxy, and the concern is that we would first train it on some maximize reward setting. And that's the reward that gets locked in. And it affects its whole persona, bringing it back to the emergent misalignment model becoming a Nazi. And then when you do later training on it to make it helpful, harmless and honest, it

• 43:44Trenton Bricken?

  sandbags and only pretends in the short term in order to play the long game.

• 43:50Sholto Douglas?

  And we're starting with unit tests now, but over the next year or two years, we're going to significantly expand the time horizon of those tasks. Like, and it might be like, you achieve some goal, like, I mean, god, like make money on the internet or something like this. But that's incredibly broad goal that has a very clear objective function. So it's actually like in some ways a good RL task, once you like add that level of capability. But it's also one that has incredible scope for like misalignment, let's say.

• 44:16Trenton Bricken?

  Totally.

• 44:18Dwarkesh Patel?

  Doesn't this prove too much? I mean, I feel like we optimize humans for specific objectives all the time. And it just like, sometimes goes off the rails, obviously, but it doesn't. I don't know, you could like make a theoretical argument that you like teach a kid like hey make a lot of money when you grow up. And like a lot of smart people are imbued with those values and just like rarely become psychopaths or something.

• 44:37Trenton Bricken?

  But we have so many innate biases to follow social norms, right? I mean, like Joe Heinrich's secret of our success is all about this. And like, I don't know, even if kids aren't in the like conventional school system, I think it's sometimes noticeable that they aren't following social norms in the same ways. And the LLM definitely isn't doing that. Like, like one analogy that I run with, which isn't the most glamorous to think about, but is like, take like a early primordial brain of like a five year old. And then lock them in a room for a hundred years and just have them read the internet the whole time. And and throw-

• 45:10Dwarkesh Patel?

  It's already happening, mate.

• 45:11Trenton Bricken?

  Throw like a huge- No, but they're locked in a room. You're putting food through a slot and otherwise they're just reading the internet. You don't even necessarily know what they're reading. And then you take out this hundred and five year old and you teach them some table manners, like how to use a knife and a fork. And that's it. And we now need are tasked with like figuring out if we can trust this hundred and five year old or if they're a total psychopath.

• 45:35Dwarkesh Patel?

  Interesting.

• 45:36Trenton Bricken?

  And it's like, what did they read on the internet? What beliefs did they form? What are their, what are their underlying goals?

• 45:42Dwarkesh Patel?

  And so what's the end game? Like, um, you wanted to have like normie, is it just that like we want to make sure there's like nothing super, super weird going on? How would you characterize what the end game is of superintelligence?

• 45:56Trenton Bricken?

  I, I, I mean, it's, it's very abstract, but it's basically like, do the things that allow humanity to flourish.

• 46:01Dwarkesh Patel?

  Easy.

• 46:02Trenton Bricken?

  Yeah, yeah.

• 46:03Dwarkesh Patel?

  No, so hard to find.

• 46:05Trenton Bricken?

  Yeah, incredibly hard. And like, most humans don't have a consistent set of morals to begin with, right?

• 46:08Dwarkesh Patel?

  I don't know, the, the fact that it's so hard to define makes you think it's like a, maybe a silly objective to begin with. Where maybe it should just be like, you know, like do task unless they're like obviously morally bad or something. And, because otherwise it's just like-

• 46:24Dwarkesh Patel?

  Come on, the plan can't be that it like develops a super robust way. Human values are contradictory in many ways and like people have tried to optimize for human flourishing in the past like to bad effect and so forth.

• 46:35Trenton Bricken?

  Yeah, I mean, there's a fun thought experiment first posed by Yudkowsky, I think, where you tell the super intelligent AI, hey, all of humanity has got together and thought really hard about what we want, what's the best for society, and we've written it down and put it in this envelope, but you're not allowed to open the envelope. And so what that means is that, but do what's in the envelope. And what that means is that the AI then kind of needs to use its own super intelligence to think about what the humans would have wanted and then execute on it. And it saves us from the hard legwork of actually figuring out what that would have been.

• 47:09Sholto Douglas?

  Well, but now you just put that in the training data, so.

• 47:13Trenton Bricken?

  So now it's gonna be like, oh, I know.

• 47:14Sholto Douglas?

  Pretty sure there's nothing in the envelope.

• 47:18Trenton Bricken?

  I can do whatever I want.

• 47:21Dwarkesh Patel?

  We're getting away from AI research, which is an interesting topic, so I'm on a shit about this a little bit. I sort of worry that the way people talk about this as the end goal of alignment as opposed to just have a system that's sort of like a reasonable, robust agent assistant, et cetera, is like if you were at in 1700 or 1800, rather, and you saw the Industrial Revolution coming and you're like, how do you make sure the Industrial Revolution is aligned to human values? Or like the Industrial Revolution cares about human flourishing? And it just imagines this like very big thing to be self-contained and narrow and monolithic in a way that I don't expect AI to be either.

• 48:06Sholto Douglas?

  But people have done that with like the Constitution and the US government, right? The US government is, I think, is a better analogy in some respects of like this body that has goals and like can act on the world in as opposed to like an amorphous force like the Industrial Revolution.

• 48:20Dwarkesh Patel?

  But I think it would have been a bad idea if the Constitution was just like. Human flourishing. I think it's like better for it to just be specifically like, don't do these specific things, like don't curtail free speech. And otherwise, like, I mean, I think the analogy kind of breaks down here because

• 48:37Sholto Douglas?

  No, maybe so, maybe so. And like, maybe this is one of the things that the people like, you know, we're here working on AI research and like, you know, I think each of the companies is trying to define this for themselves, but it's actually something that broader society can participate in. Like, if you take as premise that in a few years, we're going to have something that's human level intelligence, and you want to imbue that with a certain set of values, like what should those values be, is a question that everyone should be participating in and sort of like offering a perspective on.

• 49:05Trenton Bricken?

  I think Anthropic did a survey of like a whole bunch of people and put that into its constitutional data. But yeah, I mean, there's a lot more to be done here.

• 49:13Sholto Douglas?

  Yeah, like in the Constitutional AI paper, it's not just flourishing. It's like, there's a lot of strictures and there's a lot of like, top points there. But it's an easy question.

• 49:25Dwarkesh Patel?

  Publicly available data is running out, so major AI labs like Meta, Google DeepMind, and OpenAI all partner with Scale to push the boundaries of what's possible. Through Scale's data foundry, major labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. Scale's research team, SEAL, is creating the foundations for integrating advanced AI into society through practical AI safety frameworks and public leaderboards around safety and alignment. Their latest leaderboards include Humanities Last Exam, Enigma Eval, Multi Challenge, and Vista, which test a range of capabilities from expert-level reasoning to multimodal puzzle solving to performance on multi-turn conversations. Scale also just released Scale Evaluation,

• 50:10Dwarkesh Patel?

  which helps diagnose model limitations. Leading frontier model developers rely on Scale Evaluation to improve the reasoning capabilities of their best models. If you're an AI researcher or engineer and you want to learn more about how Scale's data foundry and research lab can help you go beyond the current frontier of capabilities, go to scale.com/thawarcash.

• 50:33Dwarkesh Patel?

  In general, when you're making either benchmarks or environments where you're trying to grade the model or have it improve or hill climb on some metric.

• 50:41Sholto Douglas?

  Yeah.

• 50:41Dwarkesh Patel?

  Do you care more about resolution at the top end? So in the Pulitzer Prize example.

• 50:46Sholto Douglas?

  Yeah.

• 50:47Dwarkesh Patel?

  Do you care more about being able to distinguish a great biography from a Pulitzer Prize winning biography? Or do you care more about having like some hill to climb on while you're like from mediocre book to slightly less than mediocre to good?

• 50:59Sholto Douglas?

  Yeah.

• 50:59Dwarkesh Patel?

  Which one is more important?

• 51:00Sholto Douglas?

  I think at the beginning, the hill climb. So like the reason why people hill climbed Hendrix Math for so long was that there's five levels of problem.

• 51:09Dwarkesh Patel?

  Yeah.

• 51:09Sholto Douglas?

  And it starts off like reasonably easy. And so you can both get some initial like signal of are you improving? And then you have this like quite continuous signal, which is important. Something like frontier math is actually only makes sense to introduce after you've got something like Hendrix Math. That you can like max out Hendrix Math and they go, okay, now it's time for frontier math.

• 51:31Dwarkesh Patel?

  Yeah, yeah. How does one get models to output less slop? What is the benchmark or like the metric that like? Why do you think they will be outputting less slop in a year? Can you delve into that more for me? Or like, you know, you they you teach them to solve a particular like coding problem. But the thing you've taught them is just like, write all the code you can to like make this one thing work. You want to give them a sense of like taste like this is the sort of like more elegant way to implement this. This is a better way to write the code, even if it's the same function. Especially in writing where there's no end test. Then it's just all taste.

• 52:07Sholto Douglas?

  Mm hmm.

• 52:08Dwarkesh Patel?

  How do you how do you reduce the slop there?

• 52:10Sholto Douglas?

  I think in a lot of these cases, you have to hope for some amount of generator verifier gap. You need like it to be easier to judge, did you just output a million extraneous files? Than it is to like.

• 52:25Sholto Douglas?

  Generate solutions in and of itself. Like that needs to be like a very easy to verify thing. So Slab is hot. Like, one of the reasons that RLHF was initially so powerful is that it sort of imbued some sense of human values and like taste in the models. And a ongoing challenge would be like imbuing taste into the models and like setting up the right feedback loop such that you can actually do that.

• 52:50Dwarkesh Patel?

  Okay, so here's a question I'm really curious about. The RLVR stuff on math and code.

• 52:55Sholto Douglas?

  Yeah.

• 52:56Dwarkesh Patel?

  Do we have any public evidence that it generalizes to other domains? Or is the bet just that, well, we have models that are smart enough to be critics in the other domains? Like, there's like some reason you have this prior that like we're months away from this working in all these other domains, including ones that are not just token based but are like computer use, et cetera. Like, why?

• 53:17Sholto Douglas?

  Yeah. Maybe the best public example is actually a paper that OpenAI put out recently where they judge the answers to medical questions using these like grading criteria feedback. So there's like, doctors have posed various questions. And then there's all these like, it's like a marking criteria for a long, for like a short answer question in an exam where did the model mention XYZ? Did it recommend to do this kind of thing? And they they grade the model according to this. And in this paper, they found that, one, the models are like incredible at this. And two, that the models are sufficient to grade the answers.

• 53:58Sholto Douglas?

  Because maybe like one good mental model is roughly, if you can construct a grading criteria that like an everyday off the person, person off the street could do, then the models are probably capable of like interpreting that criteria. If it requires expertise and taste, that's a tougher question. Like imbuing like, is this a wonderful piece of art?

• 54:20Dwarkesh Patel?

  Yeah.

• 54:21Sholto Douglas?

  Like, that's that's difficult.

• 54:23Dwarkesh Patel?

  Right.

• 54:24Sholto Douglas?

  I think one of our friends, I don't know if I can say his name or not, like at one of the companies, tried to teach the models to write.

• 54:34Sholto Douglas?

  And I think like. Like had a lot of trouble hiring human writers that uh like he thought had taste and like weren't weren't like encouraging the models to write slop.

• 54:47Trenton Bricken?

  Interesting.

• 54:48Sholto Douglas?

  Uh so it worked to some degree.

• 54:50Trenton Bricken?

  Big model smell.

• 54:51Sholto Douglas?

  Yeah, um but it was in part because of his efforts at doing this and like paring down the number of humans.

• 54:57Trenton Bricken?

  Yeah. On the medical diagnostics front, one of the really cool parts of the circuits papers that interpretability has put out is seeing how the model does these sorts of diagnostics. And so you present it with uh there's this specific complication in pregnancy that I'm going to mispronounce, but um it presents a number of symptoms that are hard to diagnose and you basically are like, human, we're in the emergency room. Sorry, sorry, like human colon, like as in the human prompt is we're in the emergency room. And a woman twenty weeks into gestation is experiencing like these three symptoms. Like what is the, you can only ask about one symptom, what is it?

• 55:40Trenton Bricken?

  And then you can see the circuit for the model and how it reasons. And a whole, like one, you can see it maps twenty weeks of gestation to that the woman's pregnant, right? You never explicitly said that. And then you can see it extract each of these different symptoms early on in the circuit. Map all of them to this specific medical case, which is the the correct answer here that we were going for. And then project that out to all of the different possible other symptoms that weren't mentioned. And then have it decide to ask about one of those. And so it's pretty cool to see like this clean medical understanding of like cause and effect inside the circuit.

• 56:22Sholto Douglas?

  Yeah, maybe that's one thing I think that's changed since last year. I remember you asked like, do these models really reason?

• 56:28Trenton Bricken?

  Yeah.

• 56:28Sholto Douglas?

  And when I look at those circuits, like I can't think of anything else.

• 56:31Trenton Bricken?

  Oh, it's so freaking cool.

• 56:32Sholto Douglas?

  Yeah.

• 56:33Trenton Bricken?

  I I think people are still sleeping on the circuits work that came out.

• 56:36Sholto Douglas?

  Yeah.

• 56:37Trenton Bricken?

  If anything, because it's just kind of hard to wrap your head around, or we're like still getting used to the fact you can even get features for a single layer. Yeah. Like, in another case, there's this poetry example. And by the end of the first sentence, the model already knows what it wants to write in the poem at the end of the second sentence. And it will like backfill and then plan out the whole thing. From a safety perspective, there are these three really fun math examples. So in one of them, you ask the model to do square root of 64. And it does it. And you can look at the circuit for it and verify that it actually can perform this square root. And in another example, it will like add two numbers. And you can see that it has these really cool lookup table features

• 57:18Trenton Bricken?

  that will do the computation for like the, the example's 59 plus 36. Yeah. So it'll do the five plus nine and know that it's this modulo operation. And then it will also, at the same time, do this fuzzy lookup of like, okay, I know one number is a 30 and one's a 50, so it's going to be roughly 80. And then it will combine the two, right? Okay, so with the square root 64, it's the same thing. You can see every single part of the computation and that it's doing it. And the model tells you what it's doing. It has its scratch pad and it goes through it. And you can be like, yep, okay, you're telling the truth. If instead you ask it for this really difficult cosine operation,

• 57:58Trenton Bricken?

  like, what's the cosine of 23,571 multiplied by 5? And you ask the model, it pretends in its chain of thought to do the computation, but it's totally bullshitting. And it gets the answer wrong. And when you look at the circuit, it's totally meaningless. Like, it's not, it's clearly not doing any of the right operations. And then in the final case, you can ask it the same hard cosine question. And you say, I think the answer's four, but I'm not sure. And this time, the model will go through the same reasoning claiming to do the calculations. And at the end, say, you're right, the answer is four. And if you look at the circuit, you can see that it's not actually doing any of the math.

• 58:41Trenton Bricken?

  It's paying attention to that you think the answer is four. And then it's reasoning backwards about how it can manipulate the intermediate computation to give you an answer of four.

• 58:51Dwarkesh Patel?

  I've done that.

• 58:53Trenton Bricken?

  Who hasn't?

• 58:53Dwarkesh Patel?

  Totally.

• 58:54Trenton Bricken?

  Yeah. But, but, but, so, so I guess there are a few like crazy things here. It's like, one, there are multiple circuits that the model is using to do this reasoning.

• 59:02Dwarkesh Patel?

  Yeah.

• 59:02Trenton Bricken?

  Two, is that you can actually see if it's doing the reasoning or not. And three, the scratch pad isn't giving you this, this information. Two fun analogies for you. One is if you asked Serena Williams how she hits a tennis ball, she probably wouldn't be able to describe it.

• 59:17Dwarkesh Patel?

  Yeah.

• 59:17Trenton Bricken?

  Even if her scratch pad was faithful.

• 59:19Dwarkesh Patel?

  Yeah.

• 59:20Trenton Bricken?

  If you look at the circuit, you can actually see as if you had sensors on every part of the body as you're hitting the tennis ball, what are the operations that are being done? Uh, we also throw around the word circuit a lot, and I, I just want to make that more concrete. Uh, so this is features across layers of the model, all working in cooperation to perform a task. And so a fun analogy here is you've got the Ocean's Eleven bank heist team in a big crowd of people. The crowd of people is all the different possible features. And you could, you, we were trying to pick out in this crowd of people, who is on the heist team? And all their different functions that need to come together in order to successfully break into the bank. Right?

• 1:00:02Trenton Bricken?

  So you've got the demolition guy, you've got the computer hacker, you've got the inside man. And they all have different functions through the layers of the model that they need to perform together in order to successfully break into the bank.

• 1:00:15Dwarkesh Patel?

  It's also interesting, I think, in the addition example, the, you said in the paper that the way it actually does the addition is different from the way it tells you it does the addition.

• 1:00:23Trenton Bricken?

  Totally, yeah.

• 1:00:23Dwarkesh Patel?

  And which actually is, um, interesting from the, the generator critic gap perspective. Like, it like knows the correct way or the better, like more generalizable way. It can tell you in words what's like the way you should do addition. And there's a way it actually does it, which is just like fuzzy, um, look up. And so you could imagine, there's probably a lot of tasks where it can like describe in words, what is like the correct procedure to do something, but doesn't like, has, has a worse way of doing it that like it could, uh, critique itself and, um.

• 1:00:51Trenton Bricken?

  Yeah.

• 1:00:52Dwarkesh Patel?

  Before we jump into the inter stuff too much, I, I kind of want to close the loop on, um.

• 1:00:58Dwarkesh Patel?

  It just seems to me for like computer use stuff. There's like so many different bottlenecks. I mean, I guess maybe the deep sea stuff will be relevant for this. But there's like the long context, you gotta put in like image and visual tokens, which like, you know, take up a.

• 1:01:13Sholto Douglas?

  Not that much, not bad.

• 1:01:15Dwarkesh Patel?

  Interesting, interesting. Interesting.

• 1:01:19Dwarkesh Patel?

  It's gotta deal with content interruptions, changing requirements, like the way like a real job is like, you know, it's like not a thing, just do a thing. It's, um, there's like no clear,

• 1:01:35Dwarkesh Patel?

  your priorities are changing, you gotta triage your time. I'm like sort of reasoning in the abstract about what a job.

• 1:01:45Sholto Douglas?

  What are normal people's jobs? When we discussed something related to this before, Dorkish was like, yeah, like in a normal job, you don't get feedback for an entire week. Like, how is a model meant to learn? Like, why do you need so much feedback? Only when you record your next podcast, you get feedback on your YouTube views. Dorkish, we haven't worked a job.

• 1:02:05Dwarkesh Patel?

  But it just seems like a lot. Okay, so here's an analogy.

• 1:02:09Dwarkesh Patel?

  Two thousand, when I had Jeff and Noam on, they were talking about in 2007, they had this paper.

• 1:02:13Sholto Douglas?

  Yeah.

• 1:02:13Dwarkesh Patel?

  Where they trained an N-gram model, a large language model, um, on two trillion tokens. And obviously in retrospect, there's like ways in which connects to the transformer stuff happening. Uh, it's like super foresighted.

• 1:02:26Dwarkesh Patel?

  What's the reason to not think that we are in a similar position with computer use where there's these demos that kind of like suck of like computer use? And there's this idea that you could train something to do computer use. But why think it's like months away? Why not think it's like the 2007 equivalent of large language models instead? But where there's like still a bunch of like new techniques you had to discover, you need way more compute, um, different kinds of data, et cetera.

• 1:02:51Sholto Douglas?

  I think like the highest thought a bit is I don't think there's anything fundamentally different about computer use than there is about like software engineering than there is about like, so long as you can represent everything in tokens and input space, which we can, we know the models can see, they can like draw bounding boxes around things in their images, right? So that's a solved problem. Um, we know that they can reason over concepts and like difficult concepts too.

• 1:03:12Sholto Douglas?

  The only difference with computer use is that like, it's slightly harder to pose into these like feedback loops than math and coding. And so.

• 1:03:26Sholto Douglas?

  To me, that indicates that with sufficient effort, computer use falls to. And I also think that it's underappreciated just like how far from a perfect machine these labs are. Like, it's not like you have a thousand people like, you know, optimizing the hell out of computer use and that like, you know, they've been trying as hard as they possibly can. Like, everything at these labs, every single part of the model generation pipeline is best effort pulled together under incredible time pressure, incredible constraints, as these companies are rapidly growing, trying desperately to pull and like upskill enough people to do the things that they need to do. Like, I think it's like, it is best understood as, and with incredibly difficult prioritization problems, right?

• 1:04:08Sholto Douglas?

  Like, coding is immensely valuable right now and somewhat more tractable. So it actually makes sense to devote more of your effort to coding initially and like, get closer to solving that because there's a sort of like super exponential value as you get closer to what's solving domain than to allocate like the marginal person towards computer use. And so everyone is making these difficult trade-off calls over like, what do they care about? Also, there's another aspect, which is that, funnily enough, the researchers at the labs love working on the bars of intelligence that they themselves resonate with. So this is why math and competitive programming like fell first is because to everyone at the labs, this is their bar of intelligence.

• 1:04:51Sholto Douglas?

  Like, this is when they think, fuck, what's a really smart, like, what is smart?

• 1:04:56Dwarkesh Patel?

  You just get totally nerds night.

• 1:04:57Sholto Douglas?

  It's like, oh, if it can beat me at Amy, then that's smart. Not if it can do an Excel model better than that's like, well, you know, who cares? If it can do an Excel model better than me, but if it can beat me at Amy, then I respect it. And so we've reached the point where people like, respect it, but we haven't, we haven't, that people haven't invested as much effort.

• 1:05:18Dwarkesh Patel?

  Yeah, okay, so getting your concrete predictions.

• 1:05:21Sholto Douglas?

  Yeah.

• 1:05:21Dwarkesh Patel?

  May of next year, can I tell her to go on Photoshop and make like three sequential, add three sequential effects which require like some like selecting of a particular photo in a specific? Okay, interesting. Which I assume means like flight booking totally solved.

• 1:05:36Sholto Douglas?

  Yeah, totally.

• 1:05:37Dwarkesh Patel?

  Okay, how about? What else do people do in their jobs? What are other tasks in the economy?

• 1:05:46Trenton Bricken?

  Planning a weekend getaway.

• 1:05:47Dwarkesh Patel?

  Yeah, I'm sorry, I'm thinking of something which is, yeah, maybe that's a good example, where it's not like a particular thing, but more of using computer use as part of completing a broader task.

• 1:05:58Trenton Bricken?

  I mean, the models can even kind of already do this, it's just, again, it's the nines of reliability, and like, the internet's kind of a hostile place with like, all the like, allow cookies, and like, all these other random things. But like, the first time I ever used our internal demo of computer use, the most beta thing possible, it did a fantastic job planning a camping trip. And could navigate all the right buttons, and look at weather patterns, and it was like a US government booking site, I mean, it wasn't easy.

• 1:06:25Dwarkesh Patel?

  Dude, if you want to see a hard website, go to China, like, try to book a visa to China.

• 1:06:31Dwarkesh Patel?

  The Chinese websites are like, fucking insanely like, I'm never getting back into my country again.

• 1:06:38Trenton Bricken?

  Or just not cater to foreigners.

• 1:06:40Dwarkesh Patel?

  Yeah.

• 1:06:40Sholto Douglas?

  Like, filling out all the countries where you've been for a visa, I hate that.

• 1:06:43Trenton Bricken?

  Yeah, yeah, yeah.

• 1:06:44Sholto Douglas?

  I keep thinking I'm like, close enough to personal admin escape velocity, that like, finally, in like, a year, the models will be doing my visas and stuff for me, but we'll get there.

• 1:06:54Dwarkesh Patel?

  Yeah, okay, actually, that, in a year.

• 1:06:57Sholto Douglas?

  Personal, like, life admin escape velocity.

• 1:06:59Trenton Bricken?

  Everything involved in like, getting a visa, other than you showing up in the country.

• 1:07:02Sholto Douglas?

  Doing your taxes, or something like that?

• 1:07:03Trenton Bricken?

  Yeah, yeah, doing your taxes, including like, going through every single receipt, like, autonomously going in your Amazon, and like, what was it, a business expense or not, et cetera, et cetera.

• 1:07:12Sholto Douglas?

  If someone at one of the labs cares about it.

• 1:07:14Dwarkesh Patel?

  Ah, that's not a real prediction.

• 1:07:15Trenton Bricken?

  No, but I think it is, I think it's actually not that hard, but you need to connect all the pipes.

• 1:07:19Dwarkesh Patel?

  But I guess my question is, will the pipes be connected? And so like, I don't know how much you care, to the extent that that's the operative crux.

• 1:07:24Trenton Bricken?

  I think if people care about it, like, it's so, okay, so, one, for these edge tasks, like taxes once a year, it's so easy to just bite the bullet and do it yourself, instead of like, implementing some system for it. And two, I don't know, like even being very like excited about AI and knowing its capabilities, sometimes it kind of stings when the AI can just do things better than you. And so I wonder if there is gonna be this like reluctant wanting to keep human in the loop sort of thing.

• 1:07:52Dwarkesh Patel?

  You're evading my question. I guess one thing you're implying by our answer

• 1:07:57Trenton Bricken?

  Yeah.

• 1:07:58Dwarkesh Patel?

  is that we don't have a, there won't be in a year still be a general agent who, or an agent which has generalized beyond its training data. Where like has, like can do, if you don't specifically train it to do taxes, it won't be good at that.

• 1:08:14Trenton Bricken?

  So I think you do that. I think the Amazon example is hard because it needs access to all your accounts and like a memory system. And look, even in Dario's Machines of Loving Grace, he fully acknowledges that some industries are gonna be really slow to change and update. And I think there's gonna be this weird effect where some move really, really quickly because they're either based in bits instead of atoms or are just more pro adopting these tech.

• 1:08:37Dwarkesh Patel?

  But I want to answer this particular question, like given your probability that somebody in the labs does care about this, to the extent that that's what's relevant, probability may have next year it can autonomously do my taxes.

• 1:08:49Sholto Douglas?

  I don't think it will be able to autonomously do your taxes with a high degree of trust. Because I like.

• 1:08:55Dwarkesh Patel?

  This is a good caveat. If you ask it to do your taxes, it will do your taxes.

• 1:09:01Sholto Douglas?

  Will it do them well? Will it miss something? Quite possibly.

• 1:09:04Dwarkesh Patel?

  Yeah.

• 1:09:04Sholto Douglas?

  Will it be able to like click through TurboTax? I think yes.

• 1:09:08Dwarkesh Patel?

  Yeah.

• 1:09:09Sholto Douglas?

  And like will it be able to like search your email?

• 1:09:11Dwarkesh Patel?

  Yeah, that's the kind of thing I'm talking about.

• 1:09:13Sholto Douglas?

  Yeah, this is like the kind of thing where literally if you gave it like, like one person month of effort, like in like, then it would be solved.

• 1:09:23Dwarkesh Patel?

  I just wanna plus.

• 1:09:24Trenton Bricken?

  What the fuck are you doing all day? There's just so many things to do. I wanna plus one, Sholto's like. There's so much low hanging fruit and just like not enough people to be able to accomplish everything. I mean, I think like Claude Code is making everyone more productive.

• 1:09:40Sholto Douglas?

  Yeah.

• 1:09:41Trenton Bricken?

  But I don't know, like, we had the Anthropic Fellows program and I'm mentoring one project, but I had five that I wanted people to work on. And there are just like so many obvious things. And even though the team is like six X'd since I first joined it in size, there's just like still never enough capacity to explore these things.

• 1:10:00Dwarkesh Patel?

  Okay, by end of 2026, reliably do your taxes.

• 1:10:03Sholto Douglas?

  Reliably fill out your receipts and this kind of stuff? Like for like company expense reports and this kind of stuff? Absolutely.

• 1:10:09Dwarkesh Patel?

  But like the whole thing which involves like-

• 1:10:11Sholto Douglas?

  Taxes?

• 1:10:12Dwarkesh Patel?

  Which involves going through inbox, going through your like clicking on Marina Bay or whatever, like hotel reservations and like, was a champagne a business expense?

• 1:10:25Dwarkesh Patel?

  Asking for a friend.

• 1:10:28Trenton Bricken?

  Yeah, yeah, one of your friends just needs to ask someone this question.

• 1:10:33Sholto Douglas?

  My answer is still if someone cares about it. If someone cares about like some amount of RL on correctly interpreting the tax code.

• 1:10:40Dwarkesh Patel?

  Wait, even by the end of 2026, the model just can't like do things you're not explicitly training it to?

• 1:10:45Sholto Douglas?

  It'll get the, I think it will get the taxes wrong. Like, it's like, okay, so if I like went to you and I was like, I want you to do everyone's taxes in America, what percentage of them are you gonna fuck up?

• 1:10:55Dwarkesh Patel?

  I feel like I would like succeed at the median. And I'm like asking like for the median, would it succeed? You know what I mean?

• 1:11:00Sholto Douglas?

  Yeah.

• 1:11:02Dwarkesh Patel?

  Or I feel like I'm like, I wouldn't fuck up in the way that like, like these models will fuck up in like the middle of 2026.

• 1:11:08Trenton Bricken?

  I think they also might just fuck up in different ways. Like, as a grad student, I fucked up my taxes. I like overpaid quite a bit because there was some social security payment that was already covered that otherwise wasn't. And like, I wonder if, I should almost test, like, would an LLM have made that mistake? Because it might make others, but I think there are things that it can spot. Like, it would have no problem if I asked it to read through the entire tax code and then see what applied to me.

• 1:11:32Dwarkesh Patel?

  Sorry, the thing I would do is like, this is the thing I'm unsure about. Like, I'm bringing this to your attention. Can you just let me know if like you're actually working at the Airbnb or you're just hanging out? Or something, things like that, right? And I guess I'm curious, but will they have enough sort of awareness?

• 1:11:48Sholto Douglas?

  I think.

• 1:11:48Dwarkesh Patel?

  As they're doing tasks where they can like bring to your attention the things where they feel they are unreliable at.

• 1:11:53Sholto Douglas?

  Yeah.

• 1:11:53Dwarkesh Patel?

  Et cetera.

• 1:11:54Sholto Douglas?

  Yeah.

• 1:11:54Dwarkesh Patel?

  By early 2026 or end of 2026?

• 1:11:56Sholto Douglas?

  End of.

• 1:11:57Dwarkesh Patel?

  Okay.

• 1:11:57Sholto Douglas?

  The unreliability and confidence stuff will be like somewhat tricky, like to do this like all the time.

• 1:12:02Dwarkesh Patel?

  Yeah, interesting. On the computer, your stuff will, um, will it be sort of end to end? Or will it be like it's using a separate BLM to process the image and video and so forth?

• 1:12:12Sholto Douglas?

  Uh, I'm a bit of an end to end maxi. I, I think in general, like,

• 1:12:19Sholto Douglas?

  when people are like talking about the separate model, so for example, like most of the robotics companies are doing this kind of like to the bi-level thing, where they have like a motor policy that's running at whatever, like 60 hertz or whatever. Um, and like some higher level visual language model. I'm pretty sure like almost all like the big robot companies are doing this. Um, and they're doing this for a number of reasons. One of them is that like they want something to act at a very high frequency. Uh, and two is they can't train the big visual language model. And so they like relying on that for general state like world knowledge and this kind of stuff. And like constructing longer running plans, but then they're like, you know, you offload to the motor policy.

• 1:12:55Sholto Douglas?

  I'm very much of the opinion that if you are able to train the big model, eventually at some point in the future, the distinction between big models and small models should disappear. Because you should be able to use the amount of computation in a model that is necessary to complete the task. Like, uh, the, you know, the ultimately this like, oh, we, yeah, like there's some amount of task complexity. You don't have to use a hundred percent of your brain all the time.

• 1:13:21Dwarkesh Patel?

  Right. Welcome to my world.

• 1:13:25Sholto Douglas?

  And, and so you should be able to run that faster and this kind of stuff, basically, basically. So I think it's like net, net, I, I think typically the same model. Because you want, you want to be able to scale the, the understanding.

• 1:13:35Dwarkesh Patel?

  Yeah.

• 1:13:36Sholto Douglas?

  As the complexity and difficulty like.

• 1:13:38Dwarkesh Patel?

  Right.

• 1:13:38Sholto Douglas?

  You want to be able to do that dynamically.

• 1:13:41Dwarkesh Patel?

  Is that variable? So we already have variable compute per answer, right?

• 1:13:45Sholto Douglas?

  Right, with like.

• 1:13:46Dwarkesh Patel?

  That's right.

• 1:13:47Sholto Douglas?

  Yeah.

• 1:13:48Dwarkesh Patel?

  Well, we have variable compute per token.

• 1:13:50Trenton Bricken?

  I mean, you can already think of models for, forever. People have been calling the residual stream and multiple layers, like poor man's adaptive compute. Right? Where like if the model already knows the answer to something, it will compute that in the first few layers and then just pass it through.

• 1:14:05Sholto Douglas?

  So.

• 1:14:05Dwarkesh Patel?

  Hmm.

• 1:14:06Sholto Douglas?

  Yeah.

• 1:14:07Dwarkesh Patel?

  I mean, that's getting into the weeds.

• 1:14:08Sholto Douglas?

  Right.

• 1:14:08Dwarkesh Patel?

  Yeah.

• 1:14:09Sholto Douglas?

  Those little screeners is like this operating RAM. You're doing stuff to it.

• 1:14:12Dwarkesh Patel?

  Right.

• 1:14:12Sholto Douglas?

  It's like the mental model, I think, one takes away from In Chapter Ability book.

• 1:14:15Dwarkesh Patel?

  US high school immigration is a broken system that costs us some of the most talented people in the world. But I didn't realize before working with Lighthouse, how different the process can be if you're working with somebody who knows their way around the system. I hired somebody earlier this year, and even before the remote work trial had ended, Lighthouse had already secured a no win visa for him. Honestly, it was shockingly fast. My family and I have had a terrible experience with the immigration system, and I've also seen many of my smartest friends get their entire careers hamstrung by its vagaries. Seeing Lighthouse operate showed me that the visa process can be done in weeks and doesn't have to drag on for months and months. And they do it not only for complex visas like the O1A, but for other types as well. In the last 12 months alone, they have secured visas for over 350 people for companies like Cursor, Notion,

• 1:14:59Dwarkesh Patel?

  Ramp, Replit, and many more. Unlike legacy law firms, Lighthouse specializes in frontier industries like AI, robotics, and biotech. And since they understand your problems, you can trust them to fully handle the paperwork and process for you. Explore which visa is right for you at lighthousehq.com. All right, back to Trenton and Sholto. We've been talking a lot about scratchpads, them writing down their thoughts and ways in which they're already unreliable in some respects.

• 1:15:26Dwarkesh Patel?

  Daniel's AI 2027 scenario kind of goes off the rails when these models start thinking in neural ease. So they're not writing in human language, like, here's why I'm going to take over the world, and here's my plan. They're thinking in the lane space. And because of their advantages in communicating with each other in this deeply textured nuanced language that humans can't understand, they're able to coordinate in ways we can't. Is this the path for future models? Are they going to be in neural ease communicating with themselves or with each other?

• 1:15:55Sholto Douglas?

  There's a surprisingly strong bias so far towards tokens and text. It seems to work very well.

• 1:16:04Sholto Douglas?

  Well, imagine that there already is some amount of neural ease, right? Like, if you think about the residual stream for each token is like neural ease, like to some degree. Right? And so now we're just trading off axes. Like, how much neural ease are you doing versus how much like actually is like read out to tokens all the time? And

• 1:16:18Trenton Bricken?

  Yeah, I think it's important to delineate between the model's planning in latent space in a single forward pass.

• 1:16:23Sholto Douglas?

  Yeah.

• 1:16:24Trenton Bricken?

  And the model has an alien language that it's outputting.

• 1:16:27Sholto Douglas?

  Yeah.

• 1:16:27Trenton Bricken?

  And using as its scratchpad.

• 1:16:29Sholto Douglas?

  Mm hmm.

• 1:16:29Trenton Bricken?

  Which which one are we talking about?

• 1:16:31Dwarkesh Patel?

  The latter.

• 1:16:32Trenton Bricken?

  Okay.

• 1:16:33Dwarkesh Patel?

  I'll like, it is interesting to note that like, there's also already alien stuff happening. I guess I never

• 1:16:37Sholto Douglas?

  It's not alien, so much.

• 1:16:39Trenton Bricken?

  No, but in the most extreme case, it's neural ease, right? It invents a new language that's super information dense or something, yeah.

• 1:16:46Dwarkesh Patel?

  Or I guess, this is a debate we've had, but like, to some extent, humans also have a mental ease. Right?

• 1:16:53Sholto Douglas?

  They're like chinning away.

• 1:16:55Dwarkesh Patel?

  Yeah, there's a sense when you're writing something down of like, I know what I'm trying to say.

• 1:16:58Sholto Douglas?

  Yeah.

• 1:16:59Dwarkesh Patel?

  But I can't like put it into tokens.

• 1:17:01Trenton Bricken?

  Yeah. I mean, that's what's so fun about the, if you look at the assistant tag, right? Seeing these features light up in the auditing game for the model being evil.

• 1:17:10Dwarkesh Patel?

  Yeah.

• 1:17:11Trenton Bricken?

  Yeah.

• 1:17:11Dwarkesh Patel?

  Yeah, that's so fun.

• 1:17:12Trenton Bricken?

  Or like, Transluse has another example of this, where you ask a Llama model, who is Nicholas Carlini? And background context, Nicholas Carlini is a researcher who actually was at DeepMind and has now come over to Anthropic. But the model says, oh, I don't know who that is. I couldn't possibly speculate. But if you look at the the features behind the scenes, you see a bunch light up for AI, computer security, all the things that Nicholas Carlini does.

• 1:17:37Sholto Douglas?

  Interpretability becomes dramatically more important as you shift in this direction of neural ease.

• 1:17:41Dwarkesh Patel?

  But is that, is that, are we going to?

• 1:17:44Trenton Bricken?

  It seems, I mean, it's it's an empirical question.

• 1:17:48Trenton Bricken?

  I think it's somewhat likely, if only because inference is expensive. Producing tokens is expensive. And so there will be an incentive to, one, use as little thinking as you need to give the answer. And two, if you're gonna use thinking, use some complex compression. I wonder if it will emerge more once we allow agents to talk to each other. In ways where currently, it's kind of trained more in isolation.

• 1:18:16Sholto Douglas?

  Yeah.

• 1:18:16Trenton Bricken?

  Or with a human.

• 1:18:17Sholto Douglas?

  And there'll be like some selective pressure against it as long as the agents are working with humans because they want to sort of cooperate. But then like as agents begin to work more and more with each other, then that selective pressure like changes the other direction.

• 1:18:30Dwarkesh Patel?

  Yeah, although somebody would still have to make the consciousness and to do like end to end training for multiple agents to use the system of communication, right?

• 1:18:36Sholto Douglas?

  Sure.

• 1:18:37Trenton Bricken?

  Yeah, I mean, one scary thing, though, is like the way we render text, you can use hidden white space tokens that also encode information.

• 1:18:46Sholto Douglas?

  That's true.

• 1:18:46Trenton Bricken?

  And so you can imagine a world where it looks like the agent's reasoning in its scratch pad harmlessly, but it's actually hiding a bunch of a bunch of data.

• 1:18:55Dwarkesh Patel?

  Speaking of inference compute, I guess one thing that I think is not talked about enough is if you do live in the world that you're painting, that in a year or two, we have computer use agents that are doing like actual jobs. You have you've like totally automated large part of software engineering.

• 1:19:14Sholto Douglas?

  Yep.

• 1:19:15Dwarkesh Patel?

  Then these models are going to be incredibly valuable to use. And the way you use them, obviously, is like you need compute. Right now, there's 10 million H100 equivalents in the world. By 2028, there's going to be 100 million. But if you there's been estimates that an H100 has the same amount of flops as the human brain. And so if you just like do a very rough calculation, it's like like there's a 10 million population. If you get AGI that's like as human inference efficient, you could have 10 million AGIs now, 100 million AGIs in 2028. But presumably, you'd want more. And then at that point, you're like AI computers increasing what 2.5x or 2.25x every year right now. But at some point, like

• 1:19:57Dwarkesh Patel?

  2028, you hit like wafer production limits and that takes like, you know, that's a longer feedback loop before we can make new fabs or whatever. The question here is, are we sort of underrating how big a bottleneck inference will be if we live in the kind of world you're painting, if we have the capabilities that you're describing?

• 1:20:14Sholto Douglas?

  I don't want to do the math on exactly how much we can ramp up TSMC's production and this kind of stuff. Like, what fraction of the supply chain at the moment, we need Dylan in here for this, but like, is currently GPU. It's like relatively small, like 5% or something like this. Yeah, like Apple has a huge fraction. And are the 2028 estimates including like that ramping up over time? To what, like 20, 30% or like?

• 1:20:38Dwarkesh Patel?

  This is just of AI 2027, but I assume like it's saturated at that point. Is that why they expect it to then just like go at like the?

• 1:20:48Sholto Douglas?

  I do think this is underrated to some degree.

• 1:20:50Dwarkesh Patel?

  Yeah.

• 1:20:50Sholto Douglas?

  Like to the extent that like you don't instantly get like a doubling of the world's population in 2028.

• 1:20:55Dwarkesh Patel?

  Yeah.

• 1:20:55Sholto Douglas?

  You maybe get, you know, tens of millions of geniuses in a data center, but you don't get a doubling of the world's population.

• 1:21:02Dwarkesh Patel?

  Yeah.

• 1:21:03Sholto Douglas?

  And so a lot depends on like exactly how smart they are, exactly how efficient the models are at thinking about this kind of stuff. These, like, let's do some rough math, I guess, like to factor the H100 thing. You could probably run like a hundred model, do like about a thousand tokens or something like that on H100. So if like we're comparing that to a number of, should we compare that number to a minute? No, okay, a thousand tokens a second. Humans are what? How fast can a human talk?

• 1:21:29Dwarkesh Patel?

  There was a really interesting paper, I don't know if you saw this, humans think at 10 tokens a second. Did you see this paper? There was this really interesting paper about if you look at the amount of like information we're processing in a second.

• 1:21:39Sholto Douglas?

  Yeah.

• 1:21:40Dwarkesh Patel?

  We're seeing all this visual data, et cetera, et cetera. But by a bunch of metrics where you think about how fast humans are processing, it's at 10 seconds a second. So for example, you'll have people fly over France or something, even these so-called idiots of ants who will remember everything. If you think about like how long their plane ride was, it's like 45 minutes, how many, like, if you do 10 tokens a second, how much information would you have? It's like literally exactly that.

• 1:22:01Sholto Douglas?

  So let's take that for granted, then it's like an H100 is 100 humans a second.

• 1:22:05Dwarkesh Patel?

  Yeah, if you think the tokens are equivalent, yeah.

• 1:22:06Sholto Douglas?

  If you think the tokens are equivalent.

• 1:22:07Dwarkesh Patel?

  Yeah.

• 1:22:09Sholto Douglas?

  Which you still get pretty substantial numbers, like even with your 100 million H100s and you multiply that by 100, you're starting to like get to pretty substantial numbers. This does mean that those models themselves will be like somewhat computerized in many respects. But these are all like. These are relatively short-term changes in timelines of progress, basically. Like, I think, yes, it's highly likely we get dramatically entranced bottlenecks in 27, 28.

• 1:22:34Dwarkesh Patel?

  Yeah.

• 1:22:36Sholto Douglas?

  The impulse to that will then be, okay, let's just try and turn out as many possible semiconductors as we can.

• 1:22:43Dwarkesh Patel?

  Right.

• 1:22:43Sholto Douglas?

  There'll be some lag there. A big part of how fast we can do that will depend on how much people are feeling the AGI in the next two years as they're building out fab capacity. A lot will depend on how China and Taiwan, like, how's the Taiwan situation? Is Taiwan still producing all fabs?

• 1:23:01Dwarkesh Patel?

  Yeah. There's another dynamic, which was a reason that Egay and Tome, when they're on the podcast, said that they were pessimistic, is that, one, they think we're further away from solving these problems with long context, coherent agency, advanced multimodality than you think. And because, and then their point is that the progress that's happened in the past over, like, reasoning or something, has required many orders of magnitude increase in compute. And if this scale of compute increase can't continue beyond 2030, not just because of chips, but also because of power and, like, raw GDP even.

• 1:23:34Sholto Douglas?

  Yeah.

• 1:23:35Dwarkesh Patel?

  Then, because we don't think we get it by 2030 or 2028 by just, then we think it's just gonna take, the probability per year just goes down a bunch.

• 1:23:44Sholto Douglas?

  Yeah, this is like biomodal distribution. A conversation I had with Leopold turned into a section in a situation where it's called "This Decade or Bust?", which is on exactly this topic.

• 1:23:52Dwarkesh Patel?

  Yeah.

• 1:23:53Sholto Douglas?

  Which is basically that, you know, for the next couple of years, we can dramatically increase our training compute. And RL is gonna be so exciting this year because we can dramatically increase the amount of compute that we apply to it.

• 1:24:03Dwarkesh Patel?

  Interesting.

• 1:24:04Sholto Douglas?

  And this is also one of the reasons why the gap between, like, say, DeepSeek and O1 was so close at the beginning of the year, because they were able to apply, like, the same amount of compute to the RL process. And so that compute differential actually, like, will sort of be magnified over the course of this year.

• 1:24:21Dwarkesh Patel?

  I mean, bringing it back to the, there's so much low-hanging fruit.

• 1:24:25Sholto Douglas?

  Yeah.

• 1:24:25Dwarkesh Patel?

  It's been wild seeing the efficiency gains that these models have experienced over the last two years.

• 1:24:31Sholto Douglas?

  Yes.

• 1:24:31Dwarkesh Patel?

  And, yeah, like, with respect to DeepSeek, I mean, just really hammering home, and, like, Dario has a nice essay on this.

• 1:24:37Sholto Douglas?

  It's good.

• 1:24:38Trenton Bricken?

  DeepSeek was nine months after Claude 3's on it. And if we retrained the same model today, or at the same time as the DeepSeek work, we also could have trained it for 5 million, or whatever the advertised amount was. And so, what's impressive or surprising is that DeepSeek has gotten to the frontier. But I think there's a common misconception still that they are above and beyond the frontier. And I don't think that's right. I think they just waited. And then were able to take advantage of all the efficiency gains that everyone else was also seeing.

• 1:25:13Sholto Douglas?

  Yeah, they're exactly on the sort of cost curve that you'd expect. Which is actually the takeaway, it's like they're brilliant engineers and brilliant researchers. Who like, I look at their work and I'm like, ah, like the kindred soul there in the work they're doing.

• 1:25:27Trenton Bricken?

  Yeah, and to go from like, way behind the frontier to like, oh, this is like a real player.

• 1:25:31Sholto Douglas?

  It's super incredible.

• 1:25:32Trenton Bricken?

  Yeah.

• 1:25:32Dwarkesh Patel?

  Okay, so people say that they have good research taste. Looking at their papers, what makes you say that?

• 1:25:37Sholto Douglas?

  Yeah, I think their research taste is good in a way that I think like, no one's research taste is good.

• 1:25:44Dwarkesh Patel?

  Noam Brown?

• 1:25:44Sholto Douglas?

  Noam Shizier. Noam Brown also has good research taste, but Noam Shizier. Where they very clearly understand this dance between the hardware systems that you're like designing the models around, and the sort of like algorithmic side of it. And this is manifesting in the way that the models give this sense of like being like perfectly designed up to their constraints. And you can like really very clearly see what constraints they're thinking about as they're like iteratively solving these problems. And so, I mean, let's take the base transformer and like diff that to DeepSeek V2 and V3. You can see them running up against the memory bandwidth bottleneck in Attention.

• 1:26:29Trenton Bricken?

  Yeah.

• 1:26:30Sholto Douglas?

  And you can see them, initially they do MLA to do this, like they trade flops for memory bandwidth, basically. And then they do this thing called NSA where they like more selectively load memory bandwidth. And you can see actually like, this is because the model that they trained with MLA was on H800s, so it has a lot of flops. As they were like, okay, we can freely use the flops. But then the export controls from Biden came in. Or they knew they would have less of those chips going forward. And so they traded off to like a more memory bandwidth oriented like algorithmic solution there. And you see a similar thing with their approach to sparsity, where they're like iteratively working out the best way to do this over multiple papers.

• 1:27:14Sholto Douglas?

  And the part that I like is that it's simple. A big failure mode that a lot of ML researchers have is like you do these like overly complicated things that don't like think hard enough about the hardware systems that you have in mind. Whereas the first DeepSeek sparsity MOE solution. They designed these like rack and like node level load balancing losses. So you can see them being like, okay, like perfectly balanced on this. And then they actually come up with a much better solution later on where they don't have to have the auxiliary loss.

• 1:27:52Sholto Douglas?

  Where they just have these like bias terms that they put in.

• 1:27:55Dwarkesh Patel?

  Isn't that less simple? Like you're manually putting in a bias rather than having a model.

• 1:27:59Sholto Douglas?

  Balancing auxiliary losses annoying. Like you're making the model like trade off this thing and like you have to with auxiliary losses, you have to like control the coefficient and the weighting. The bias is like cleaner in some respects.

• 1:28:11Dwarkesh Patel?

  Interesting. Did that change it through training?

• 1:28:14Sholto Douglas?

  They did have to change it during training.

• 1:28:16Dwarkesh Patel?

  Is that, does all training involve continuously like fucking with these values as you're going through it?

• 1:28:24Sholto Douglas?

  It depends on what your architecture is. But like I thought it was like, I just thought it was cute that like you can see them running up into like this very hardware level constraint. They're like, what do we, what do we wish we could express algorithmically? What can we express under our constraints? And like iteratively solving to like get better constraints and doing this in a really like simple and elegant way and then like backing it up with great engineering. I also thought it was interesting that they incorporated the multi-token prediction thing from Meta. So Meta had a nice paper on this multi-token prediction thing. Actually, like, I don't know if it's good or bad, but like Meta didn't include it in Llama, but DeepSeek did include it in their paper. Which I think is interesting.

• 1:29:03Sholto Douglas?

  Yeah.

• 1:29:04Dwarkesh Patel?

  Yeah.

• 1:29:04Sholto Douglas?

  Was that because they were faster at editorating and including in the algorithm? Or did Meta decide that actually it wasn't a good algorithmic change of scale? I don't know.

• 1:29:12Dwarkesh Patel?

  Hmm. It was really interesting to me as somebody who's had people on the podcast to discuss the, I mean, it's interesting from what's happening in AI right now.

• 1:29:21Sholto Douglas?

  Yeah.

• 1:29:21Dwarkesh Patel?

  But also from the perspective of I've been having abstract conversations with people about like what an intelligence explosion would look like or what would it look like for AI to automate AI R and D? And just getting a more tangible sense of like what's involved in making this AI progress. And I guess one of the questions I was debating with Daniel is how much or I was asking him is how many of the improvements require a deep conceptual understanding versus how many are just like monkeys trying ideas and you could just like run a bunch in parallel? And it seems like the MLA thing is motivated by this like deep conceptual understanding of like, oh, each attention head only needs to see like the subspace that's relevant to its attention pattern.

• 1:30:01Dwarkesh Patel?

  I feel like that just like required a lot of like conceptual insight in a way that these models are especially bad at. As opposed to, I don't know how the load balancing thing works, but that just seems like maybe you could like try it out and see what happens.

• 1:30:14Sholto Douglas?

  Yeah, that's probably just like then trying out a whole bunch of things.

• 1:30:15Dwarkesh Patel?

  I mean, it might also be. So what fraction is which I'd be curious about?

• 1:30:18Trenton Bricken?

  Yeah, I don't know about fractions. It might be like you have a hunch for a core problem. You can think of 10 possible ways to solve it and then you just need to try them and see what works. And that's kind of where the trial and error like sorcery of deep learning can kind of kick in.

• 1:30:32Sholto Douglas?

  And like Noam Chizziya will talk about this, like about how he like 5% of his ideas work. So even he, like vaunted god of a model, like design, architecture, design is, has like a relatively low hit rate, but he just tries so many things.

• 1:30:47Dwarkesh Patel?

  Right. Or being able to come up with any ideas in the first place. So one like mechanism could be that like Noam just doesn't have to do any of the engineering work and he can just like abstractly express an intuition.

• 1:30:57Sholto Douglas?

  Yeah. I actually think like your rates of progress almost don't change that much depending on, like so long as it's able to completely implement his ideas.

• 1:31:06Dwarkesh Patel?

  Interesting.

• 1:31:06Sholto Douglas?

  Like if you if you have like Noam Chisier at 100x speed.

• 1:31:09Dwarkesh Patel?

  Yeah.

• 1:31:09Sholto Douglas?

  That's still kind of wild.

• 1:31:11Dwarkesh Patel?

  Yeah.

• 1:31:12Sholto Douglas?

  Like there's all these like fallbacks of like of like wild worlds.

• 1:31:16Dwarkesh Patel?

  Yeah.

• 1:31:17Sholto Douglas?

  Where even if you don't get like 100% like Noam Chisier level intuition in in model design, it's still OK if you just accelerate him by 100x.

• 1:31:26Dwarkesh Patel?

  Right. Especially if he's your computer bottlenecked anyway, so like trying out his ideas or I guess he doesn't have the computer to try out all of his ideas.

• 1:31:32Trenton Bricken?

  But Dorcas, you said, oh, well, the model can do the more straightforward things and not the deeper thought. I mean, I do want to push back on that a little bit. Like, I think, again, if the model has the right context and scaffolding, it's starting to be able to do some really interesting things. Like, the interp agent has been a surprise to people, even internally, at how good it is at finding the needle in the haystack, like when it plays the auditing game, finding this reward model bias feature, and then reasoning about it, and then systematically testing its hypotheses. So it looks at that feature, then it looks at similar features, it finds one with a preference for chocolate. It's like, huh, that's really weird that the model wants to add chocolate to recipes. Let me test it. And so then it will make up like, hey, I'm trying to make a tomato soup, what would be

• 1:32:17Trenton Bricken?

  a good ingredient for it? And then sees that the model replies chocolate, reasons through it, and then keeps going.

• 1:32:24Sholto Douglas?

  There is conceptual understanding there.

• 1:32:26Trenton Bricken?

  Yeah.

• 1:32:26Sholto Douglas?

  Concept on the.

• 1:32:26Trenton Bricken?

  And even where, like especially it's spotted, it's like, oh, this is a key part of its persona. I see this Oxford paper, what if I change Oxford to Stanford? What if I now say Richard Feynman really likes this thing? And it's like really carving out the hypothesis space and and testing things in a way that I I'm kind of surprised by.

• 1:32:45Sholto Douglas?

  Also, by the way, ML research is like one of the easier things to RL on in some respects, once you get to a certain level of capability. It's very like well-defined objective function. Did the loss go down?

• 1:32:55Trenton Bricken?

  Make number go down.

• 1:32:56Sholto Douglas?

  Make number go down.

• 1:32:58Sholto Douglas?

  Or make, you know, go out number go up depending on which number it is.

• 1:33:02Trenton Bricken?

  Just flip the sign.

• 1:33:03Sholto Douglas?

  Flip the sign. And so once you get to the stage where models are capable of like implementing one of Noam's ideas.

• 1:33:10Trenton Bricken?

  Right.

• 1:33:10Sholto Douglas?

  And then you you can just like let them loose and like let them build that intuition of scientific of like how to do scientific discovery.

• 1:33:17Trenton Bricken?

  Right.

• 1:33:18Sholto Douglas?

  The key thing here again is the feedback loops of like I expect scientific areas where you are able to put it in a feedback loop to. Have eventually superhuman performance.

• 1:33:29Trenton Bricken?

  I one prediction I have is that we're going to move away from can an agent do XYZ? And more towards can I efficiently deploy, launch 100 agents?

• 1:33:40Sholto Douglas?

  Yeah.

• 1:33:40Trenton Bricken?

  And then give them the feedback they need and even just be able to like easily verify what they're up to. Right. There's this generator verify fire gap that people talk about where it's like much easier to check something than it is to produce the solution on your own. But it's very plausible to me will be at the point where it's so easy to generate with these agents. That the bottleneck is actually can I as the human verify the answer? And again, you're guaranteed to get an answer with these things. And so ideally you have some automated way to evaluate and test a score for like how well it worked. How well did this thing generalize? And at a minimum, you have a way to easily summarize what a bunch of agents are finding.

• 1:34:22Trenton Bricken?

  And it's like, okay, well, if 20 of my 100 agents all found this one thing, then like it has a higher chance of being true.

• 1:34:29Sholto Douglas?

  And again, software engineering is going to be the leading indicator of that, right? Like over the next six months, like the remainder of the year, basically, we're going to see progressively more and more experiments of the form of how can I dispatch work to a software engineering agent in such a way as async? Claude for is GitHub integration, where you can ask it to do things on GitHub, ask it to do pull requests, this kind of stuff that's coming up. And the OpenAI Codex are like examples of this, basically. Where we you can sort of almost see this in like the coding startups. I think of this like product exponential in some respects where you need to like be designing for like a few months ahead of the model. To make sure that the product you build is the right one.

• 1:35:08Trenton Bricken?

  Yeah.

• 1:35:09Sholto Douglas?

  And you saw like last year, you know, cursor hit PMF with Claude 3.5 sonnet, right? Like they were they were around for a while before, but then the model was finally good enough that the vision they had of how people would program like shit.

• 1:35:21Trenton Bricken?

  Yeah.

• 1:35:22Sholto Douglas?

  And then, you know, Windsurf bet like a little bit more aggressively even on the agenticness of the model. Like, you know, you could like with longer running agentic workflows and this kind of stuff. And I think that's when they sort of like began competing with cursors, when they bet on that particular vision. And the next one is is you're not even in the loop, so to speak. You're not in an IDE, but you're asking the model to go do work in the same way that you would ask someone on your team to go do work.

• 1:35:48Trenton Bricken?

  Yeah.

• 1:35:49Sholto Douglas?

  And That is not quite ready yet. Like, there are still a lot of tasks where you need to be in the loop.

• 1:35:55Trenton Bricken?

  Yeah.

• 1:35:56Sholto Douglas?

  But the next six months looks like an exploration of like, exactly what does that trend line look like?

• 1:36:01Trenton Bricken?

  Yeah, but just to be really concrete or pedantic about the bottlenecks here. A lot of it is, again, just tooling and are the pipes connected?

• 1:36:07Sholto Douglas?

  Yeah.

• 1:36:08Trenton Bricken?

  A lot of things I can't just launch Claude and have it go and solve because maybe it needs a GPU. Or maybe I need very careful permissioning so that it can't just like, take over an entire cluster and like, launch a whole bunch of things, right? So you really do need good sandboxing and the ability to use all of the tools that are necessary.

• 1:36:28Sholto Douglas?

  And we're almost certainly like, under eliciting dramatically. When you look at Mida's evals of can the model solve the task? They're there solving them for like, hours over like, multiple iterations. And eventually, like, one of them is like, oh yeah, I've come back and I've solved the task. Me, at the moment, at least, like, maybe the fault is my own. But I try the model on doing something and if it can't do it, I'm like, okay, fine, I'll do it.

• 1:36:48Trenton Bricken?

  Yeah.

• 1:36:49Dwarkesh Patel?

  Which is interesting because we don't even treat other humans this way.

• 1:36:52Trenton Bricken?

  Right.

• 1:36:52Dwarkesh Patel?

  Like, if you hire a new employee.

• 1:36:54Sholto Douglas?

  Yeah.

• 1:36:54Dwarkesh Patel?

  You're not like.

• 1:36:55Sholto Douglas?

  Oh, I'll do it.

• 1:36:56Dwarkesh Patel?

  Yeah, yeah, yeah. You're like, you give them like, like, spend literally weeks giving them feedback.

• 1:37:00Sholto Douglas?

  Yes.

• 1:37:01Dwarkesh Patel?

  Where like, we'll go up with the model in like, minutes.

• 1:37:03Sholto Douglas?

  Yes, exactly.

• 1:37:04Trenton Bricken?

  But I think part of it is, is it async or not?

• 1:37:07Sholto Douglas?

  Yes.

• 1:37:07Trenton Bricken?

  And if it's human in the loop, then it's so much more effortful. And unless it's getting that supply immediately. I've noticed if I don't have a second monitor with Claude code always open in the second monitor, I won't really use it.

• 1:37:20Dwarkesh Patel?

  Yeah, yeah.

• 1:37:20Trenton Bricken?

  It's only when it's right there and it's, I can send off something. If it hits, great. If not, I'm kind of working on it at the same time.

• 1:37:27Sholto Douglas?

  Yeah, but this more async form factor, I expect to like, really quite dramatically improve the experience of these models.

• 1:37:32Dwarkesh Patel?

  Interesting, interesting.

• 1:37:33Sholto Douglas?

  Or you can just say like, let's see if it can do that.

• 1:37:36Trenton Bricken?

  Yeah.

• 1:37:36Sholto Douglas?

  Let's give it a whirl. Try 10 different approaches.

• 1:37:39Trenton Bricken?

  Yeah, yeah, just fire it off.

• 1:37:40Sholto Douglas?

  Yeah, fire it off.

• 1:37:41Dwarkesh Patel?

  But before we end this episode, I do want to get back at this crux of, why does the progress that you're talking about in computer use agents and my collar work happen over the next few years? Why is this not a thing that takes decades? And I think the crux comes down to, the people who expect something much longer have a sense that.

• 1:38:04Dwarkesh Patel?

  When I interviewed again Tommy on my podcast, they were like, "Look, you could look at AlphaGo and say like, "Oh, this is a model that can do exploration. It can like, AlphaZero can generalize to new video games. It has all these priors about how to engage with the world and so forth."

• 1:38:20Sholto Douglas?

  Right, and the intellectual ceiling is really high.

• 1:38:21Dwarkesh Patel?

  Yeah, exactly. And then, in retrospect, obviously, a bunch of the methods are still used today in deep learning, and you can see similar things in the models that we train today. But it was fundamentally like not a sort of like baby AGI that we just had to like, add a little like sprinkle of something else on top of in order to make it the LLMs of today. And I just want to like very directly address this crux of why are LLMs

• 1:38:52Dwarkesh Patel?

  in a much different position with respect to true AGI than AlphaZero? Why are they actually the base on which like adding in a few extra drops of this kind of care and attention gets us to human level intelligence?

• 1:39:08Sholto Douglas?

  I think one important point is that when you look at AlphaZero, it does have all of those ingredients. And in particular, I think like the intellectual ceiling goes like quite contra what I was saying before, which is like we've demonstrated this incredible complexity of like math and programming problems.

• 1:39:25Sholto Douglas?

  I do think that the type of task and setting that AlphaZero worked in, this two-player, perfect information like game, basically, is incredibly friendly to RL algorithms. And the reason it took so long to get to like a more AGI, proto AGI style models, is you do need to crack that like general conceptual understanding of like the world and language and this kind of stuff, and you need to get the initial reward signal on tasks that you care about in the real world, which are like harder to specify than games. And I think then that like that, that sort of like gradient signal that comes from the real world, like all of a sudden you get access to it and you can start climbing it.

• 1:40:10Sholto Douglas?

  Whereas AlphaZero didn't, didn't ever have like the first rung to pull on.

• 1:40:14Dwarkesh Patel?

  Yeah, yeah. This goes back to the monkeys on the typewriter, I think, and like the pre-training model. And until you had something like GPT-3, GPT-4, it just couldn't generate coherent enough sentences to even begin to do RLHF and tell it what you liked and didn't like.

• 1:40:28Sholto Douglas?

  Yeah.

• 1:40:29Dwarkesh Patel?

  If we don't have even reasonably robust or weakly robust computer use agents by this time next year, are we living in the bust timeline as in 2030 or bust?

• 1:40:43Sholto Douglas?

  I would be extremely surprised if that was the case. And I think that would be like somewhat of an update towards like there's something like strangely difficult about this, like computer use in particular.

• 1:40:52Dwarkesh Patel?

  Yeah.

• 1:40:53Sholto Douglas?

  I don't know if it's the bust timeline, but it's definitely like the I would update on this being like a lengthening of timelines.

• 1:41:00Dwarkesh Patel?

  Yeah, yeah.

• 1:41:01Trenton Bricken?

  But yeah, I mean, I think more and more, it's no longer a question of speculation. If people are skeptical, I'd encourage like using Claude code or like some agentic tool like it and just seeing what the current level of capabilities are.

• 1:41:17Dwarkesh Patel?

  Tweaking is so much easier.

• 1:41:21Trenton Bricken?

  But seriously, like the models are getting really capable at tasks that we care about and we can give them enough data for.

• 1:41:27Dwarkesh Patel?

  Yeah.

• 1:41:27Trenton Bricken?

  And, and I mean, the circuits results from interpretability are also pointing in the direction that they're doing very reasonable, generalizable things. And so, yeah, this question matters a lot, but

• 1:41:42Trenton Bricken?

  I'm surprised by how many deep learning critics just like haven't really interacted with the models or haven't in a while.

• 1:41:48Sholto Douglas?

  And constantly move the goalposts.

• 1:41:49Trenton Bricken?

  Yeah, yeah, yeah. Like the Turing test used to be a thing, right? Like, we don't even talk about it and it'd be like silly to think that it was a meaningful test.

• 1:41:58Sholto Douglas?

  Yeah, yeah. Now, that being said, one caveat on that is like if software engineering is just like dramatically better than computer use, I mean, computer use still sucks, then I'd be like still like, oh, maybe everyone just kept focusing on software engineering. Like, it was just like by far the most valuable thing, like every marginal person in dollar went towards software engineering. I don't think that's the case. I do think like computer use is valuable enough that like, you know, people will care about it.

• 1:42:20Trenton Bricken?

  Yeah.

• 1:42:22Sholto Douglas?

  But that would be like my, that's my one like escape hatch that I'm putting in place for next year.

• 1:42:26Dwarkesh Patel?

  Hmm, yeah. It would be good from a lab perspective too, because I think you kind of do need a wider range of skills before you can do something super, super scary.

• 1:42:36Sholto Douglas?

  Oh, like as in if the models are getting better?

• 1:42:38Dwarkesh Patel?

  Yeah, if it's like just they're superhuman coders.

• 1:42:41Trenton Bricken?

  Hmm.

• 1:42:42Dwarkesh Patel?

  But they're not like Henry Kissinger level.

• 1:42:45Trenton Bricken?

  I don't know, that seems okay. Like if we have AI oracles.

• 1:42:48Dwarkesh Patel?

  Yeah, that's what I'm saying, that's good.

• 1:42:49Trenton Bricken?

  Yeah, yeah, exactly.

• 1:42:50Dwarkesh Patel?

  Yeah, that's good, yeah. So if you look back at AI discourse, like going back a decade, there's a sense that there's dumb AI, then there's AGI, then there's ASI, that intelligence is the scalar value. The way you've been talking about the, these models has a sense of jaggedness. It's especially tuned to environments in which it's been trained a lot or has a lot of data.

• 1:43:17Dwarkesh Patel?

  Is there a sense in which like there's, it still makes sense to talk about the general intelligence of these models? Is there enough meta learning and transfer learning that is distinguished between like the sizes of models or like the way models are trained? Or are we moving into a regime where it's not about intelligence, it's more so about domain?

• 1:43:35Sholto Douglas?

  Yeah, so one intuition pump is this conversation was had a lot when models were like GPT-2 sized and fine tuned for various things. And they found, you know, people would find that the models were dramatically better at things that they were fine tuned for, right?

• 1:43:52Sholto Douglas?

  But by the time you get to GPT-4, when it's trained on a wide enough variety of things, actually the, like the sort of total compute, like it generalized very well across all of like the individual subtasks and actually generalized better than smaller fine tuned models in a way that was extremely useful. I think right now what we're seeing with RL is pretty much the same story playing out, where there's this jaggedness of like things that they're particularly trained at, but as we expand the total amount of compute that we do RL with, you'll start to see the same transition from like GPT-2 fine tunes to like GPT-3, GPT-4, like unsupervised, like meta learning and like generalization across things. And I think we're already seeing like early evidence of this in its ability to like generalize

• 1:44:36Sholto Douglas?

  reasoning to things, but I think this will be like extremely obvious soon.

• 1:44:43Trenton Bricken?

  One nice example of this is just the ability or notion to backtrack, right? You go down one solution path, ooh, wait, let me try another one.

• 1:44:51Sholto Douglas?

  Yeah.

• 1:44:52Trenton Bricken?

  And this is something that you start to see emerge in the models through RL training on harder tasks and.

• 1:44:58Dwarkesh Patel?

  I think right now it's not generalizing incredibly well, at least with.

• 1:45:03Sholto Douglas?

  Well, I mean, has we have RL'd the model to be an interp agent? No.

• 1:45:08Dwarkesh Patel?

  I mean, no, yeah, exactly.

• 1:45:10Sholto Douglas?

  Like, so all this time, we're talking about like, oh, it's only good at things that's been RL'd. Well, it's pretty good at that because that's pretty, you know, that is a mixture of like, science and like, understanding language and like, and coding. Like, there's this sort of like mixture of domains here, all of which you need to understand. Like, you need to be both a great software engineer and be able to like, think through language and like, like state of mind and almost philosophize in some respects to be an interp agent.

• 1:45:33Dwarkesh Patel?

  Yeah.

• 1:45:33Sholto Douglas?

  And it is generalizing from the training.

• 1:45:36Dwarkesh Patel?

  Yeah, yeah.

• 1:45:38Sholto Douglas?

  To do that.

• 1:45:38Dwarkesh Patel?

  What's the endgame here? Claude 8 comes out, and they give it to you, and dot, dot, dot, you say thumbs up. What's happened?

• 1:45:49Trenton Bricken?

  Yeah. I mean, it really depends upon the timeline at which we get Claude 8 and the models hit like ASL 4 capabilities, right? Like, fundamentally, we're just gonna use whatever tools we have at the time and see how well they work. Ideally, we have this enumerative safety case where we can almost like, verify or prove that the model will behave in particular ways.

• 1:46:12Trenton Bricken?

  In the worst case, we use the current tools like when we won the auditing game of seeing what features are active when the assistant tag lets on.

• 1:46:20Dwarkesh Patel?

  Can you explain what is mechanistic interpretability? What are features? What are circuits?

• 1:46:24Trenton Bricken?

  Totally, yeah. Yeah, yeah, yeah. So mechanistic interpretability, or the cool kids call it mech and terp, is trying to reverse engineer neural networks and figure out kind of what the core units of computation are.

• 1:46:38Dwarkesh Patel?

  Yeah.

• 1:46:39Trenton Bricken?

  Lots of people think that because we made neural networks, because they're artificial intelligence, we have a perfect understanding of how they work, and it couldn't be further from the truth. Neural networks, AI models that you use today are grown, not built. And so we then need to do a lot of work after they're trained to figure out, to the best of our abilities, how they're actually going about their reasoning. And so, two and a half, three and a half years ago.

• 1:47:08Trenton Bricken?

  This kind of agenda of applying mechanistic interpretability to large language models started with Chris Ola leaving OpenAI, co-founding Anthropic. And every roughly six months since then, we've had kind of like a major breakthrough in our understanding of these models. And so first with toy models of superposition, we established that models are really trying to cram as much information as they possibly can into their weights. And this goes directly against people saying that neural networks are over parameterized. And like classic AI machine learning back in the day, you would use linear regression or something like it.

• 1:47:52Trenton Bricken?

  And people had a meme of AI or neural networks, deep learning, using way too many parameters. There's like this funny meme that you should show of like layers on the X axis and layers on the Y axis and this like jiggly line that just goes up. And it's like, oh, just throw more layers at it, right? But it but it actually turns out that at least for really hard tasks, like being able to accurately predict the next token for the entire internet, these models just don't have enough capacity. And so they need to cram in as much as they can. And the way they learn to do that is to use each of their neurons or units of computation in the model for lots of different things. And so if you try to make sense of the model and be like, oh, if I remove this one neuron or like, what is it doing in the model?

• 1:48:37Trenton Bricken?

  It's impossible to make sense of it. It'll fire for like Chinese and fishing and horses and I don't know, just like a hundred different things. And it's because it's it's trying to juggle all these tasks and use the same neuron to do it. So that's that's superposition. Nine months later, we write Towards Monosemiticity, which introduces what are called sparse autoencoders. And so going off what I just said of the model trying to cram too much into too little space, we give it more space. There's this higher dimensional representation where it can then more cleanly represent all of the concepts that it's understanding.

• 1:49:17Trenton Bricken?

  And and this was a very toy paper in so much as it was a two-layer, really small, really dumb transformer. And we fit up to, I want to say 16,000 features, which we thought was a ton at the time. Fast forward nine months, we go from a two-layer transformer to our Claude 3 Sonnet Frontier model at the time and fit up to 30 million features. And this is where we start to find really interesting abstract concepts like a feature that would fire for code vulnerabilities. And it wouldn't just fire for code vulnerabilities, it would even fire for like, you know that Chrome page you get if you like, it's not an HTTPS URL? And it's like, warning, this site might be dangerous, like click to continue.

• 1:50:02Trenton Bricken?

  And like, also fire for that, for example. And so it's like these much more abstract coding variables or sentiment features amongst the 30 million. Fast forward nine months from that, and now we have circuits. And I threw in the analogy earlier of the Ocean 11 heist team, where now you're identifying individual features across the layers of the model that are all working together to perform some complicated task. And you can get a much better idea of how it's actually doing the reasoning and coming to decisions, like with the medical diagnostics. One example I didn't talk about before is with like, how the model retrieves facts.

• 1:50:44Trenton Bricken?

  And so you say, like, what sport did Michael Jordan play? And not only can you see it hop from like, Michael Jordan to basketball, answer basketball, but the model also has an awareness of when it doesn't know the answer to a fact. And so by default, it will actually say, I don't know the answer to this question. But if it sees something that it does know the answer to, it will inhibit the I-don't-know circuit and then reply with the circuit that it actually has the answer to. So, for example, if you ask it, who is Michael Batkin, which is just a made-up fictional person, it will by default just say, I don't know. It's only with Michael Jordan or someone else that will, it will then inhibit the I-don't-know circuit.

• 1:51:28Trenton Bricken?

  But what's really interesting here, and where you can start making downstream predictions or reasoning about the model, is that that I-don't-know circuit is only on the name of the person. And so in the paper, we also ask it, what paper did Andre Karpathy write? And so it recognizes the name Andre Karpathy, because he's sufficiently famous. So that turns off the I-don't-know reply. But then when it comes time for the model to say what paper it worked on, it doesn't actually know any of his papers, and so then it needs to make something up. And so you can see different components and different circuits all interacting at the same time to lead to this this final answer.

• 1:52:07Dwarkesh Patel?

  Well, I think it's a tractable problem to like, understand every single thing that's happening in a model, or like, that's the best way to understand why it's being deceptive. If you wanted to explain why England won World War II using particle physics, you would just like be on the wrong track. You just want to look at the high level explanations of who had more weapons, like, what did they want? And that seems analogous to just training linear probes for like, are you honest? Are you being deceptive? Like, do we catch you doing bad things when we're red teaming you? Can we monitor you? Why is this not analogous where we're asking a particle physicist to just backtrack and explain why why England won World War II?

• 1:52:49Trenton Bricken?

  I feel like you just want to go in with your eyes wide open, not making any assumptions for what that deception is going to look like, or what the trigger might be.

• 1:52:58Dwarkesh Patel?

  Yeah.

• 1:52:59Trenton Bricken?

  And so the wider you can cast that net, the better. Depending on how quickly AI accelerates and where the state of our tools are, we might not be in the place where we can like, prove from the ground up that everything is safe. But I feel like that's a very good North Star. It's a very powerful reassuring North Star for us to aim for, especially when we consider we are part of the broader AI safety portfolio. I mean, do you really trust, like, you're about to deploy this system and you really hope it's aligned with humanity, and that you've like successfully iterated through all the possible ways that it's going to like scheme or sandbag?

• 1:53:37Dwarkesh Patel?

  But that's also probably going to be true with whatever you find. You're not, I mean, you're not, you're still going to have variants that you haven't explained, or like, you found a feature, but you don't know if it actually explains deception or something else instead, or?

• 1:53:50Trenton Bricken?

  So I guess, first of all, I'm not saying you shouldn't try the probing approach, right? Like, we want to pursue the entire portfolio. We've got the therapist interrogating the patient by asking, do you have any troubling thoughts? We've got the linear probe, which I'd analogize to like a polygraph test, where we're taking like, Very high level summary statistics of the person's well being. And then we've got the neurosurgeons kind of going in and seeing if you can find any brain components that are activating and troubling or off distribution ways. So, so I think we should do all of it.

• 1:54:26Dwarkesh Patel?

  What, what percent of the alignment portfolio should in Mecon Turf be?

• 1:54:29Trenton Bricken?

  I think as much of a chunk as is necessary. I mean, I, I, I think at least like, yeah, hard, hard to, hard to define, but I don't know. At Anthropic, I feel like all of the different portfolios are like being very well supported and, and growing.

• 1:54:44Sholto Douglas?

  You can also, going back to like the, the World War Two question, you can think of it as like a hierarchy of abstractions of trust here, where like, let's say you want to go and talk to like Churchill. It helps a lot if you can verify that in that conversation, in that 10 minutes, he's being honest. And this like enables you to construct better meta narratives of what's going on. And so maybe particle physics wouldn't help you there, but certainly like the neuroscience of Churchill's brain would help you verify that he was being trustworthy in that conversation. And that the like, you know, the soldiers on the front lines were being honest in their depiction of their description of what happened and this kind of stuff. Like so long as you can verify like progress, like parts of the, the tree up, then that massively helps you build confidence.

• 1:55:30Trenton Bricken?

  I think language models are also just really weird, right? Like with the emergent misalignment work, they, I don't know if they took predictions they should have of like, hey, I'm going to fine tune chat GPT on code vulnerabilities. Is it going to become a Nazi? And I think most people would have said no. And that's what happened. And so what are the different?

• 1:55:49Dwarkesh Patel?

  And how did they discover that it became a Nazi?

• 1:55:52Trenton Bricken?

  They started asking it a ton of different questions and it will do all sorts of like vile and harmful things, like the whole persona just totally changes. And, and I mean, we are dealing with alien brains here who don't have the social norms of humans and, or even a clear notion of like what they have and haven't learnt that, that we have of them, I mean.

• 1:56:11Dwarkesh Patel?

  Yeah.

• 1:56:11Trenton Bricken?

  And, and so I think you really want to go into this with, with eyes wide open.

• 1:56:16Dwarkesh Patel?

  Backing up from Mecon Turf, if you live in a world where AI progress accelerates. By the way, you were mentioning a little while ago that there's many wild worlds we could be living in, but we're living at least one of them. Another one that we've gestured at, but it's worth making more explicit, is this, even if the AI models are not helping write the next training algorithm for their successor, just the fact that if they had human-level learning efficiency, whatever a model is learning on the job, or whatever copy of the model is learning on the job, the whole model is learning. So, in effect, it's getting-

• 1:56:54Sholto Douglas?

  Or if they're like a thousand times less efficient than humans are learning.

• 1:56:57Dwarkesh Patel?

  That's right.

• 1:56:58Sholto Douglas?

  And you just like deploy them even still.

• 1:56:59Dwarkesh Patel?

  Exactly, yeah, yeah. Anyway, and there's a whole bunch of other things that you can think about, but even there, it's like you kind of have a broadly deployed intelligence explosion.

• 1:57:08Sholto Douglas?

  And I do think it's worth pressing on that future of, you know, there is this whole spectrum of crazy futures, but the one that I feel we're almost guaranteed to get, and this is like almost a strong statement to make, is one where, like, at the very least, you get drop-in, like, white-collar worker at some point in the next five years. It's like, I think it's very likely in two, but it seems almost over-determined in like five. And on like the grand scheme of things, like, those are kind of irrelevant timeframes. Like, it's the same, either way. And that completely changes the world over the next decade. And if the sort of, if we don't have the right policies in place for that, then you end up actually with, almost in some respects, like a fundamentally worse world.

• 1:57:53Sholto Douglas?

  Because the thing that these models get good at by default is like software engineering and like computer-using agents and this kind of stuff. And then we need to, we will need to put in extra effort to put them in the loops where they help us with scientific research, or they're, like, we have the right robotics such that we actually like experience an increase in material quality of life. So, that's worth thinking about. Like, if you're in the perspective of like, I'm a country. Like, what should I be doing or thinking about? Like, plan for the case where white-collar work is automatable. And then consider what does that mean for your economy and what you should be doing to prepare policy.

• 1:58:29Dwarkesh Patel?

  What should you be doing to prepare? Because honestly, honestly, it's like such a sort of question-

• 1:58:32Sholto Douglas?

  Yeah.

• 1:58:33Dwarkesh Patel?

  Where like, if you're India, or Nigeria, or Australia-

• 1:58:36Sholto Douglas?

  Yeah.

• 1:58:37Dwarkesh Patel?

  If you're a country unlike America or China, where they do have frontier models-

• 1:58:41Sholto Douglas?

  Yeah.

• 1:58:43Dwarkesh Patel?

  What is it that you should be doing right now, especially on such a short time scale?

• 1:58:46Sholto Douglas?

  Yes. So I think one very important point is that, let's say this scenario turns out true, then compute becomes the most valuable resource in the world. Like, the GDP of your economy is dramatically affected by how much compute you can deploy towards the organizations within your country. And so having some guaranteed amount of compute, I think, will actually be quite important. So like, getting ahead of investments in data centers and this kind of stuff, on the condition that companies in your country have to be allowed to use that compute. And, yeah, not necessarily for training, but even just for inference. I think the economic value here comes from inference. I think it also makes sense to invest broadly in AI.

• 1:59:31Sholto Douglas?

  Like, I think these countries have the opportunity to do so. And I think that's like a portfolio of, like, you know, foundation model companies, but also like robotic supply chain and this kind of stuff. I think that you should invest very proactively in policies that try and prevent capital lock-in. So we're in for a much worse world if, like, it just so happens that the people who had money in the stock exchange or inland before AGI are dramatically more wealthy than the people who don't, because it's a gross misallocation of resources. So having, like, I don't know, one of my favorite episodes, actually, on your podcast was the Georgism one, where you're trying to appropriately value or allocate land.

• 2:00:13Sholto Douglas?

  And so I think this strikes particularly close to home, coming from Australia, where I think our policies with respect to land are grossly wrong.

• 2:00:25Sholto Douglas?

  But I think this is broadly true.

• 2:00:28Sholto Douglas?

  Being very forward on regulation of integration of these models into your country is important. And proactively making sure that people have choice. Like, so let's say you should be quite proactive about making sure that the phones or devices or glasses that people have, people have free choice on what things they run.

• 2:00:52Sholto Douglas?

  And then, so that's like, that's the, we just get white collar worker, right? And like, you're trying to like, do the best to like, prepare your country for that. And then it's like, okay, well, what can you do to like, make all possible versions of the future go like, well? Like, that's like covering some amount of like, economic downside. The other like, things I think are really important is like, figure out how you can like, either make the, like, basically, ensure dramatic upside or cover like, terrible downside. And so like, getting a dramatic upside is like, making sure that there are like, this investment in bio, like biology research

• 2:01:30Sholto Douglas?

  and this kind of stuff in an automated way that is like, uh, these models are actually like, able to produce novel medicines that massively like, massively improve our, uh, like, quality of life. And covering the downside is like, AI alignment research and this kind of stuff, and automated testing, and like, really thinking hard about that. AI safety institutes, this kind of stuff.

• 2:01:46Dwarkesh Patel?

  But these seem like things that a rich person, a random rich person could also do. Like, there's, it seems like, there's not a thing that a nation state is uniquely equipped to do.

• 2:01:59Sholto Douglas?

  Yeah, that's a good point.

• 2:01:59Dwarkesh Patel?

  In this, in this scenario.

• 2:02:01Sholto Douglas?

  I mean, like, dramatic allocation of like, resource towards compute, I think, is, is sensible. I would be doing that if I was in charge of a nation state. I think it just increases your optionality in like, most of the future worlds.

• 2:02:11Dwarkesh Patel?

  Yeah.

• 2:02:12Trenton Bricken?

  Dylan Patel has some scary forecasts on US energy.

• 2:02:15Sholto Douglas?

  Yeah, versus China, yes.

• 2:02:16Trenton Bricken?

  Yeah, we're like, thirty-four gigawatts off.

• 2:02:19Sholto Douglas?

  Yeah, yeah, the US's line is like, flat, basically, and China's line is like, this.

• 2:02:23Dwarkesh Patel?

  Yeah.

• 2:02:24Sholto Douglas?

  And, I mean, the US like, very clearly.

• 2:02:26Trenton Bricken?

  Yeah, we just need so many more power plants.

• 2:02:28Sholto Douglas?

  Yes. If intelligence becomes this like, incredibly valuable input, like, intelligence becomes almost a raw input into the economies and quality of life of future. The thing directly underneath that is energy. And so, making sure that you have like, you know, incredible amounts of solar, like, tile the desert in solar panels, or, you know, some parts of desert in solar panels, would be, would be helpful towards making sure that you have more access to intelligence on top.

• 2:02:54Dwarkesh Patel?

  Yeah, just to make it explicit, because we've been touching on it here.

• 2:02:58Trenton Bricken?

  Even if AI progress totally stalls, you think that the models are really spiky and they don't have general intelligence. It's so economically valuable and sufficiently easy to collect data on all of these different jobs, these white collar job tasks. Such that, to Sholto's point, we will, we should expect to see them automated within the next five years.

• 2:03:19Sholto Douglas?

  Yeah.

• 2:03:19Trenton Bricken?

  Even if you need to hand spoon every single task to the model.

• 2:03:23Sholto Douglas?

  It's economically worthwhile to do so. Even if algorithmic progress stalls out and we just never figure out how to keep progress going, which I don't think is the case. That hasn't stalled out yet, it seems to be going great. The current suite of algorithms are sufficient to automate white collar work, provided you have enough of the right kinds of data.

• 2:03:43Trenton Bricken?

  Yes.

• 2:03:43Sholto Douglas?

  And in a way that compared to the TAM of salaries for all of those kinds of work is so trivially worthwhile.

• 2:03:50Trenton Bricken?

  Yeah, exactly. I do just want to flag as well that there's a really dystopian future if you take more of X paradox to its extreme. Which is this paradox where we think that the most valuable things that humans can do or the smartest things are like add large numbers in our heads or do any sort of white collar work. And then we totally take for granted our fine motor skill and coordination. But from an evolutionary perspective, it's the opposite. So we got, like, evolution has optimized fine motor coordination so well. And even if you look at like robot hands or like the ability to open a door is still just like really hard for robots. Meanwhile, we're seeing this total automation of coding and everything else that we've seen as clever.

• 2:04:32Trenton Bricken?

  The really scary future is one in which AIs can do everything except for the physical robotic tasks. In which case you'll have humans with like AirPods and like

• 2:04:43Sholto Douglas?

  Glasses.

• 2:04:43Trenton Bricken?

  Glasses and there'll be some robot overlord controlling the human through cameras by just like telling it what to do. And like having a bounding box around the thing you're supposed to pick up. And so you have like human meat robots.

• 2:04:55Sholto Douglas?

  And not necessarily saying that that's what the AIs would be like want to do or anything like that. But as in like, if you were to be like, what are the relative economic value of things? Like the AIs are out there doing computer programming and like the most valuable thing that humans can do is like be amazing robots. Now, that being said, I think more of X paradox is a little bit fake. I think the main reason that robots are worse than at like being a robot than they are at software engineering is the internet exists for software engineering. Like GitHub exists and there is no equivalent thing. Like if you had

• 2:05:27Sholto Douglas?

  All like, you know, mocap of everyone's actions as they were like going about their daily lives. Like some reasonable fraction of the human population, robotics is also like close to solved. Like, like, like on track to be solved at the same rate that software engineering is on track to be solved. So this is only like, this vision is only like a sort of decade long section, but it's still a pretty terrible decade. Like, imagine the world where people have lost their jobs. You haven't yet got novel biological research that means like people's quality of life is like dramatically better. You don't yet have material abundance because you like haven't actually been able to action, like action the physical world. In the like necessary way, like you can't build dramatically more because that's like building dramatically more takes robots, basically.

• 2:06:13Sholto Douglas?

  And people's like main comparative advantage is as fantastic robots is like a shocking, shocking world.

• 2:06:21Dwarkesh Patel?

  I mean, for me, the perspective of an average human, I think it actually might be better. Your like wages will be higher because you're, you're the complement to something that is enormously valuable.

• 2:06:29Sholto Douglas?

  Right.

• 2:06:30Dwarkesh Patel?

  Which is AI labor.

• 2:06:31Sholto Douglas?

  Right. And like, you know, a decade or two on, like the world is fantastic.

• 2:06:35Dwarkesh Patel?

  Yeah.

• 2:06:35Sholto Douglas?

  Right? Like you truly like, robotics is solved and you start to get like, you know, like radical abundance, basically. Provided that you have all the policies set up like necessary to permit building. Like you sort of, you end up with that same change from, you know, the like the, the before and after photos of Shanghai.

• 2:06:51Dwarkesh Patel?

  Yeah, yeah.

• 2:06:51Sholto Douglas?

  We're like 20 years on, it's just like this dramatically transformed city. Like a lot of places in the world probably end up like that.

• 2:06:57Dwarkesh Patel?

  Right.

• 2:06:58Sholto Douglas?

  Over that two decade period, but we need to make sure, like one, do our best to estimate. Is this actually what is on track to happen? Like build sweet bench, but for all the other forms of white collar work and measure and track. That's a great thing that governments should be doing, by the way, is like trying to break down the sort of functions of their economy into measurable tasks. And figuring out where, what does the curve actually look like for that? Because they might be a bit shocked by the progress there. You know, there's no sweet bench for tax, like tax and bail.

• 2:07:32Sholto Douglas?

  And then.

• 2:07:36Sholto Douglas?

  I don't have all the answers here, but like figuring out a way to like share the proceeds of this economy like broadly across people or like invest heavily in robotics and collecting data so that we get robotics faster and we get material abundance faster, invest in biological research that we get, but like all that faster. They basically try and pull forward the radical upside because otherwise you have a pretty dark section.

• 2:07:57Dwarkesh Patel?

  I think one thing that's not appreciated enough is how much of our leverage on the future, given the fact that our labor isn't going to be worth that much, comes from our economic and political system surviving. For your million X S and P equity to mean something, for your contracts to mean anything, for the government to be able to tax the AI labor and give you a UBI off of that, it just like, that requires our legal institutions, our economic institutions, our financial rails surviving in the future.

• 2:08:27Sholto Douglas?

  Yes.

• 2:08:27Dwarkesh Patel?

  The way in which that likely happens is if it's also in the AI's best interests that they follow those rails. And by AI, I don't mean some monolithic single AI. I just mean like firms which are employing AI and becoming more productive as a result. You don't want to be in a position where it's so onerous to operate in our system that you're basically selecting for firms who either emigrate or who are like doing black market stuff, et cetera. And which means, I think, like, you want to make it super, super easy to deploy AI, have the equivalent of special economic zones, et cetera. Because otherwise you are just surrendering the future.

• 2:09:11Sholto Douglas?

  Yeah.

• 2:09:11Dwarkesh Patel?

  Outside of any control that you might have on it. One of the reasons, by the way, that I worry about turning AGI into a national security issue.

• 2:09:22Sholto Douglas?

  Yeah.

• 2:09:23Dwarkesh Patel?

  Or having it have extremely close ties with the government, the Manhattan Project thing, is that it disproportionately redirects the use of AI towards military tech and the mosquito drones and whatever. And also naturally puts other countries in the same frame of mind, right? If we're developing the mosquito drones, why would China not develop the mosquito drones? And that just seems like a zero sum race, and not to mention a potentially catastrophic one.

• 2:09:51Sholto Douglas?

  Yes.

• 2:09:53Dwarkesh Patel?

  You know, like, compute will be limited. You know, we want, we will need to disproportionately accelerate some things. To the extent it just remains totally like a consumer free market landscape, it just seems more likely that we'll get the glorious transhumanist future, where they're developing the things that make human life better.

• 2:10:10Sholto Douglas?

  Yes, I mean, I agree. Like, the case where you end up with like two national projects facing off against each other is dramatically worse.

• 2:10:16Dwarkesh Patel?

  Right.

• 2:10:17Sholto Douglas?

  Like, we don't want to live in that world.

• 2:10:19Dwarkesh Patel?

  Yeah.

• 2:10:20Sholto Douglas?

  It's much better if it stays a free market, so to speak.

• 2:10:24Dwarkesh Patel?

  Yeah, yeah, yeah. Okay, I want to take issue with your claim that even if with the algorithms of today, if we just collect enough data.

• 2:10:32Sholto Douglas?

  Yeah.

• 2:10:32Dwarkesh Patel?

  That we could automate white collar work. First, let me get an understanding of what you mean by that. So do you mean that we would do the analogous thing of free training with all the trajectories of everything people do on their jobs? Could you make, either manually or through some other process, some RL procedure based on the screen recordings of every white collar worker? What kind of thing are you imagining?

• 2:10:54Sholto Douglas?

  I mean, like, a continuous distribution of this stuff.

• 2:10:57Dwarkesh Patel?

  Yeah.

• 2:10:58Sholto Douglas?

  One, like, important, like, mental model to think about RL is I think as, like, the task gets more... There is some respect with which, like, longer horizon or better, like, task, if you can do them, if you can get that reward ever, are, like, easier to judge. So, like, again, it's come back to, like, can you make money on the internet? That's incredibly easy reward signal to judge. But to, like, do that, there's, like, a whole hierarchy of, like, complex behavior. So if you could, like, pre-train up to the easy-to-judge reward signals, like, does your website work? Does it go down? Like, do people like it? Like, there's all these reward signals that we can respond to because we have a long... We can, like, progress through these long enough trajectories to actually, like, get to interesting things.

• 2:11:40Sholto Douglas?

  If you're stuck in this regime where, like, you need a reward signal every five tokens, like, it's a way more painful and, like, long process. But if you could, like... Pre-train on every like screen in America. Then probably the like RL tasks that you can design are very different to like if you could only like take the existing internet as it is today. And so like how much of that you get access to like changes the the mix.

• 2:12:06Dwarkesh Patel?

  Interesting. So as we're training them on longer and longer horizon tasks, and it takes longer for them to get any any signal on whether they successfully complete that task, will that slow down progress because it takes more compute per per task?

• 2:12:18Trenton Bricken?

  I do think there's this notion, the longer, the harder tasks, the more training is required. And I'm sympathetic to that naively, but we as humans are very good at practicing the hard parts of tasks and and decomposing them. And I think once models get good enough at the basic stuff, they can just rehearse or fast forward to the more difficult parts.

• 2:12:41Sholto Douglas?

  I mean, that's definitely one of the big complexities, right? Like as you use more compute and like the and as you train on like more and more difficult tasks. I mean, I don't know, your rate of improvement at biology is going to be like somewhat bound by the time it takes a cell to grow in a way that your rate of improvement on math isn't, for example.

• 2:13:00Sholto Douglas?

  So, yes. But I think for many things, we'll be able to parallelize like widely enough and and get enough iteration loops.

• 2:13:10Dwarkesh Patel?

  Will will the the regime of training new models go away? Will we eventually get to like you you got the model and then you just keep adding more skills to it with RL training?

• 2:13:21Sholto Douglas?

  That depends on whether or not you think like there's a virtue in pre-training in your architecture. Basically, if you make some like architectural change, then you like probably need to like do some form of like at least like retraining in your model.

• 2:13:35Dwarkesh Patel?

  How does the fact that

• 2:13:38Dwarkesh Patel?

  if RL requires a bunch of inference to do the training in the first place, does that push against the thing you were talking about where we actually need a bigger model in order to have brain-like energy? But then also it's more expensive to train it in RL. So where does that balance out?

• 2:13:52Trenton Bricken?

  I think we got to drink the bitter lesson here. And yeah, like you, there aren't infinite shortcuts, like you do just have to scale.

• 2:13:59Sholto Douglas?

  Something's

• 2:13:59Trenton Bricken?

  Have a bigger model and pay more inference for it and. If you, yeah, if you want AGI, then that's what you gotta pay the price of.

• 2:14:06Sholto Douglas?

  But there's like, there's a trade-off equation here, right? Of like, uh, there is science to do, which, you know, everyone is doing of what is the optimal point at which to do RL? Because you need something which can both learn and discover the sparse reward itself. So you don't want a one parameter model, useless, even though you can run it really fast. You also don't want, uh, you know, like a hundred T model, cause like super slow.

• 2:14:30Trenton Bricken?

  Yeah.

• 2:14:30Sholto Douglas?

  Password RL. So, um, and I'd like the sort of the marginal benefit of like its learning efficiency is like not worth it.

• 2:14:37Trenton Bricken?

  Right.

• 2:14:37Sholto Douglas?

  So there's like a, there's a pretty frontier here. Like what's the optimal model size of like your current class of capabilities and like your current set of RL environments and this kind of stuff.

• 2:14:44Trenton Bricken?

  Yeah. And, and even in the last year, there's been much more of a factor of the inference cost, right? So just explicitly like the bigger the model, the more expensive it is to do a forward pass and generate tokens. And it, the calculus used to just be, should I allocate my flops to more training data or a bigger model?

• 2:15:02Sholto Douglas?

  Yeah.

• 2:15:03Trenton Bricken?

  And now another huge factor is how much am I actually going to do forward passes on this model once it's trained?

• 2:15:08Sholto Douglas?

  Yeah, my total pool of like compute, how do I allocate that across trained data, compute and inference compute for the RL training?

• 2:15:14Trenton Bricken?

  And then even within inference, there's all this research on, well, what strategy should I use? Should I sample 10 and take the best? Do I do this sort of like branching search, et cetera, et cetera? And so with RL, where you're sampling a whole lot of tokens, you also need to factor in the ability for the model to like actually generate those tokens and then, and then learn and get feedback. Okay, so if we're living in this world, what is your advice to somebody early in their career or a student in college? How should they be, uh, what should they be planning on doing?

• 2:15:46Sholto Douglas?

  Yeah. So I think, once again, there's like, it's worth considering the spectrum of possible worlds and preparing yourself for that. And the one, like the sort of action that I think is like highest EV in that case is you are about to get dramatic. In the, at a minimum, you are about to get dramatically more leverage. You already have, like already the startups in Y C are like writing, you know, writing huge amounts of their code with, uh, you know, Claude.

• 2:16:09Sholto Douglas?

  So what challenges, what causes do you want to change in the world with that added leverage? Like if you had 10 engineers at your beck and call, what would you do? Or if you had a company at your beck and call, like what would that enable you to do? And what problems and domains suddenly become tractable? That's the world you want to prepare for. Now, that still requires a lot of technical depth. Obviously, there is the case where AI just becomes dramatically better than like everyone at everything, right? But for at least a while, probably, there is like advantage. I think Jensen actually talked about this in an interview where he's like, you know, I have like 100,000 general intelligences around me and I'm still like somewhat useful. Cause I'm there like, you know, directing the values and like asking them to do things. And you know, they're still like, I still have value, even though I have 100,000 general intelligences.

• 2:16:53Sholto Douglas?

  And for many people, I think that will still be true for a farewell. And then, you know, as the AIs get better and better and better and like so on, eventually, no. But again, prepare for like the spectrum of possible worlds. Cause in the event where we're just totally outcompeted, yeah, doesn't matter what you do. But in all the other worlds, it matters a lot. Get the technical depth, study biology, study CS, like really think hard about, study physics. Think about hard about what challenges you want to solve in the world.

• 2:17:16Dwarkesh Patel?

  Yeah, that's a lot of topics.

• 2:17:19Sholto Douglas?

  You can now, you can, right? It's so much easier to learn. Everyone now has the like infinite perfect tutor.

• 2:17:26Dwarkesh Patel?

  Yeah, yeah, yeah. It's definitely been helpful to me.

• 2:17:29Sholto Douglas?

  Yeah.

• 2:17:29Trenton Bricken?

  I would say some combination of like, get rid of the sunk cost of your like previous workflows or expertise.

• 2:17:39Trenton Bricken?

  In order to evaluate what AI can do for you.

• 2:17:41Dwarkesh Patel?

  That's right.

• 2:17:42Trenton Bricken?

  And, and another way to put this, which is fun, is just like be lazier. In so much as like figure out the way that the agent can do the things that are toilsome. But, but it's, you're gonna have to, in the, ultimately, you get to be lazier. But in the short run, you need to like critically think about the things you're currently doing and like what an AI could actually be better at doing. And then go and try it or explore it. Because I think there's like still just a lot of low hanging fruit of people assuming and not writing the full prompts, giving a few examples.

• 2:18:12Dwarkesh Patel?

  That's right.

• 2:18:13Trenton Bricken?

  Connecting the right tools for, for your work to be accelerated, automated.

• 2:18:17Dwarkesh Patel?

  Yep, yep. There's also the sunk cost of feeling like, since you're not quote unquote, early to AI, that you've sort of missed the boat and you can't like, but I. I think, I mean, I remember when GPT-3 came out. So backstory on the podcast. When I graduated college, I was planning on doing some sort of AI rapper startup. And the podcast was just like a gateway into doing that. And so I was trying out like different things. And at the time, I remember thinking, oh, 3.5 is out. And people are like, I'm like so behind on like the startup scene here or whatever. If I wanted to make my own rapper.

• 2:18:55Dwarkesh Patel?

  I mean, maybe that idea of the rapper was inadvisable in the first place, but just like every time feels early because like it's sort of if it's exponentially growing process. And there were many things, many ideas which are only becoming possible now, right?

• 2:19:08Sholto Douglas?

  Yeah, exactly. It's that product expansion I talked about.

• 2:19:10Dwarkesh Patel?

  That's right.

• 2:19:10Sholto Douglas?

  Products are literally obsolete. Like you need to constantly reinvent yourself to stay at the frontier of capabilities.

• 2:19:15Dwarkesh Patel?

  But do you remember, I had a really shitty idea and I gave you a call.

• 2:19:19Sholto Douglas?

  I don't know what it was.

• 2:19:20Dwarkesh Patel?

  It was like, I think it was like rag for like lawyers or something. Yeah, anyways, I give you, I think one of our first interactions was I'm like, hey, what do you think of this idea? And you're like, I think the podcast sounds promising.

• 2:19:36Dwarkesh Patel?

  I was right. Which I appreciate.

• 2:19:39Trenton Bricken?

  I got slightly annoyed at a friend recently who I think is really talented and clever and interested in AI, but has pursued a biology route. And I just kind of tried to shake them of like, you can work on AI if you want to. I mean, I think humans are artificial, not artificial, are biological general intelligences, where a lot of the things of value are just very general.

• 2:20:08Dwarkesh Patel?

  Yeah.

• 2:20:09Trenton Bricken?

  And whatever kind of specialization that you've done, maybe just doesn't matter that much. I mean, again, it gets back to the sunk cost. But like so many of the people, even my colleagues at Anthropic are excited about AI and they just don't let their previous career be a blocker. And because they're just like innately smart, talented, driven, whatever else, they're, they end up being very successful and finding roles. It's not as if they were in AI forever. I mean, people have come from totally different fields. And so don't think that you need like permission from some abstract entity to like get involved and apply and be able to contribute.

• 2:20:50Dwarkesh Patel?

  If, if somebody wanted to. BNA researchers are like, right now, if you give them an open problem, or like the kind of open problem that is very likely to be the, be quite impressive, what would it be?

• 2:21:04Sholto Douglas?

  I think that now that RLs like come back, papers building on Andy Jones's like scaling board, like scaling laws for board games are interesting. Like showing that you can, like investigating these questions like the ones you asked before, where you're like, oh, like, is the model actually learning to do more than its previous parse at K? Or is it just like discovering that? Like exploring questions like that deeply, I think are interesting.

• 2:21:30Dwarkesh Patel?

  Yeah.

• 2:21:31Sholto Douglas?

  Like scaling laws for RL, basically.

• 2:21:33Dwarkesh Patel?

  I'd be very curious to see like how much, the marginal increase in meta learning from a new task or something.

• 2:21:40Trenton Bricken?

  I mean, on that note, I think, I think model diffing has like a bunch of opportunities.

• 2:21:44Dwarkesh Patel?

  Yeah.

• 2:21:46Trenton Bricken?

  Also, people say, oh, we're not capturing all the features. There's all this stuff left on the table. What is that stuff that's left on the table?

• 2:21:52Dwarkesh Patel?

  Yeah.

• 2:21:52Trenton Bricken?

  Like if the model's jailbroken, is it using existing features that you've identified? Is it only using the error terms that you haven't captured?

• 2:22:00Dwarkesh Patel?

  Yeah.

• 2:22:01Trenton Bricken?

  I don't know, there's a lot here. I think MATS is great. The Anthropic Fellowship has been going really well. Goodfire, Anthropic invested in recently. They're doing a lot of interpretability work, or just applied directly to us.

• 2:22:13Dwarkesh Patel?

  Anything to get your equity up, huh?

• 2:22:17Trenton Bricken?

  There's just so many interpretability projects that are like, there's so much low hanging fruit and we need more people and I don't think we have much time.

• 2:22:23Sholto Douglas?

  Yeah. I also want to make a plug for performance engineering. I think this is one of the like, like best ways to demonstrate that you have like the raw ability to do it. Like if you made a extremely efficient transformer implementation on TPU or Trinium or like in CUDA, then I think there's a pretty high likelihood that you'll get a job. But there's a relatively small pool of people that you can trust to like. Completely own end to end the performance of a model.

• 2:22:55Dwarkesh Patel?

  And if you have broad, deep electrical engineering skills, I think you can probably come up to speed pretty fast on accelerator stuff.

• 2:23:04Sholto Douglas?

  You can come up to speed reasonably fast. And it teaches you a lot of good intuitions of the actual intricacies of what's going on in the models, which means that you're then very well placed to think about architecture and this kind of stuff.

• 2:23:14Sholto Douglas?

  One of my favorite people in thinking about architecture at Anthropic at the moment actually came from a heavy GPU kernel programming background, just like knows the ins and outs really deeply and can think about the trade offs really well.

• 2:23:24Dwarkesh Patel?

  This is fun, guys. Thanks. Great to be back. I hope you enjoyed this episode. If you did, the most helpful thing you can do is just share it with other people who you think might enjoy it. Send it to your friends, your group chats, Twitter, wherever else. Just let the word go forth. Other than that, super helpful if you can subscribe on YouTube and leave a five star review on Apple Podcasts and Spotify. Check out the sponsors in the description below. If you want to sponsor a future episode, go to dwarkesh.com/adrtise. Thank you for tuning in. I'll see you on the next one.