Ep 66: Member of Technical Staff at Anthropic Sholto Douglas on Claude 4, Next Phase for AI Coding, and the Path to AI Coworkers

Sholto Douglas, a Member of Technical Staff at Anthropic, joined Unsupervised Learning to break down why coding is the clearest early signal of model progress, how AI agents are already accelerating research, and what it’ll take to unlock real-world breakthroughs in fields like biology and robotics.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Sholto Douglas was a key part of Anthropic's Claude 4 models. It was really fun to sit down with him on the day these models got released. We talked about a bunch of things, including how developers and builders should think about this next generation of anthropic models. We talked about what the trend line mean for where these models will be in six, 12 months, two, three years from now. We hit on what's required for reliable agents and when these models will get better in domains like medicine and law, kind of mirror the advances they've already made in coding. And then we hit on his views on alignment research, where we are today, what's working, what still needs to be done, and his reaction to the AI 2027 work. This was just a fascinating conversation with a brilliant mind in LLM research. I think people will really enjoy it. Without further ado, here's Sholto. Well, thanks so much for coming on the podcast.

• 0:42Speaker 01

  No, this joy. It's really cool little route.

• 0:43Jacob Effron?

  Yeah, no, I appreciate you getting in this, uh, this little cave with us.

• 0:47Speaker 01

  Yeah.

• 0:47Jacob Effron?

  It's, uh, it's always fun. By the time this podcast comes out, well, the world will have Claude 4. I'm sure people will play around with it, but I'm curious. You've, you're one of the first people to get to play around with these models. What gets you most excited about them?

• 0:57Speaker 01

  So they're another step up in software engineering, that's for sure. And Opus is really an incredible software engineering model. Like more and more, I have these moments where I go and ask her to do something incredibly ill-specified in like our large monorpo. And it's able to go and do it in quite autonomous and independent way, like going and discovering the information and figuring this out, running a couple of tests. And that, that blows me away every time.

• 1:19Jacob Effron?

  But every time we get a new set of models, we have to like re-characterize a mental model of like what works, what doesn't.

• 1:23Speaker 01

  Right.

• 1:24Jacob Effron?

  Like, how has your model now changed of, you know, when you're coding? Like what you use these models for and don't, you know, with the sleep.

• 1:30Speaker 01

  So I think the biggest one is time horizon expands a little. So I think you can characterize model capability improvements on two axes. One of those is like the absolute intellectual complexity of the task. And the other one is the amount of context or the amount of like successive actions that they're able to, uh, to like meaningfully reason over and, and agree. And these models feel substantially better along the second axis. Like they're really able to take multiple actions and like figure out what they, what information they need to pull in from their environments, uh, and, and then act on those.

• 2:01Jacob Effron?

  Yeah.

• 2:02Speaker 01

  So giving it like, it's the time horizon. Also the support that we like the, like Claude code and this kind of stuff. The fact that It now has access to all of the tools to be able to do this in a useful way. And you aren't sitting there like copy pasting from a chat box is like a pretty meaningful improvement in that regard too. There is a wide range of tasks where I'm looking at, you know, an hour plus or like many hours of work where I would have done. And it's just there churning away in front of me doing these, like in terms of human equivalent time.

• 2:28Jacob Effron?

  People are going to get these models when this podcast comes out for the first time. What is your like advice on the first thing they should try?

• 2:33Speaker 01

  First thing they should try.

• 2:36Speaker 01

  I mean, I think honestly, try and plug them into your work. That's the biggest one is is like, sit down and ask it to do the same thing. The thing that you were about to do first off in your code base that day. And watch as it, you know, as it figures out what information it needs to pull in and figures out what to do. And I think you'll be pretty, pretty impressed.

• 2:54Jacob Effron?

  Yeah. I mean, now that you have these new capabilities, obviously you have tons of people that build on top of these models. Like, what are you hoping is like newly enabled for, you know, builders that take these models and, uh, and, you know, build applications?

• 3:04Speaker 01

  So I think this is into this concept of like a product exponential in some respects. We have to be constantly building just ahead of the model's capabilities. And I like to think about this in terms of like, say, like Cursor and Windsurf and Devon and these products. If you look at Cursor, they had a vision for what coding would be. That was substantially ahead of where the sort of model capabilities were for a lot of, like for a while. You know, Cursor didn't hit PMF until the underlying models like Claude 3.5 Sonnet. Took off such that the assistance that they wanted to give people was like able to like be realized. And then Windsurf went, I would say like substantially more agenda. And that enabled them to get a reasonable slice of market share by really pressing harder on that product exponential.

• 3:47Speaker 01

  What we're starting to see now with Claude Code, but also with the new Claude GitHub integration and with OpenAI's Codex. And also Google's coding agents, so you know, everyone's using coding agents.

• 3:59Jacob Effron?

  Jules, right?

• 4:00Speaker 01

  Jules, right. Is people building for another level of sort of autonomy and asynchronous city. And so right now, the, the models are taking these stumbling steps towards being able to do tasks independently of you. The kind of tasks that would have taken you several hours before. What that looks like next, I think is. There's this interesting transferal of you are in the loop every second to you are in the loop like every minute to you're in the loop every hour that we've seen over the course of last year. And I wonder if it doesn't look like you're you're managing like a fleet of like of models in future. And so I think that kind of interface would be very interesting to explore, like just how much parallelism can you give someone when it's not a single model they're managing, but multiple models doing multiple things and interacting with each other?

• 4:45Speaker 01

  I think that would be pretty exciting.

• 4:46Jacob Effron?

  Yeah, have you seen it? Like, what what might that look like?

• 4:48Speaker 01

  What might that look like? Oh, God. I mean, I I know a lot of people actually at Anthropic who are like have multiple Claude code, uh, like instances up in different dev boxes, which is pretty cool. But I but I think I think no one's really cracked that form factor yet. I think that's like an interesting form factor to explore of what is the almost management bandwidth of an individual. Um, I think this is also an interesting question to explore from like the future of like, how does economics even work or like, what are the sort of like return on productivity of these models? Because uh, if you think like, like, we initially will need humans to verify the outputs of these models.

• 5:23Jacob Effron?

  Yep.

• 5:23Speaker 01

  And so the economic impact of the models will be like at like some initial point bottlenecked by human management banners until you get to a point where you can like delegate the trust uh in a model to itself manage teams of models. Um, and so that like continual step up in hierarchy of uh of like abstraction layers um will be, I think, uh, one of the more important like trend lines.

• 5:45Jacob Effron?

  Yeah, so basically you have, uh, you know, based on the frequency with which you need to check these models.

• 5:50Speaker 01

  Yeah, exactly.

• 5:51Jacob Effron?

  You have infinite number of models running.

• 5:53Speaker 01

  Right.

• 5:53Jacob Effron?

  Uh, and you're, you know, if you have to check them every fifteen minutes versus every hour versus every five hours, uh, you can do a lot more.

• 5:58Speaker 01

  Yeah, exactly. Um, I think Jensen mentioned this in respect to like how he felt about the future of AGI in progress and this kind of thing. He said, well, actually, I am surrounded by a hundred thousand incredibly intelligent uh AGIs. Um, and he's like, I, and so like, um, this gives me huge leverage over the world and I'm like, that's sort of impact. I mean, you know, he's describing how like he himself is like, you know, this gating factor in like managing the company of Nvidia. Uh, I think a lot of work ends up looking looking close to that direction.

• 6:25Jacob Effron?

  Yeah, who knows? I mean, maybe this like, you know, this whole field of org design ends up being actually the most important, uh, what is.

• 6:30Speaker 01

  Right, right. Exactly. And and how do you, how do you like? And like trust and yeah, exactly. All structure becomes becomes complicated.

• 6:39Jacob Effron?

  Yeah, I know what you're saying before the the episode that you did spend a year at McKinsey before. And I thought, hey, maybe this is a good use case for the consulting firms, you know, who have been doing years of this.

• 6:45Speaker 01

  Maybe, yeah.

• 6:46Jacob Effron?

  It's a good new product line for them. Yeah, actually, I was really struck by this, what you just said about how basically, you know, for the app companies, it's about being, you know, a stage ahead of like where the models are going and, you know, cursor to this. The models change so quickly that it's like almost if you think about like, you know, what cursor did versus like the kind of, you know, agenda coding companies like cognition to like, maybe someone thinking through now, like, what is the, you know, dashboard that you use to manage like your hundred team, person team?

• 7:10Speaker 01

  Exactly.

• 7:10Jacob Effron?

  What is like the right level of ahead in your mind to go? Because it feels like you may feel like, oh, God, I'm really out of my skis today. And then in three months, you'll be like, actually, I'm way behind where the model capabilities are.

• 7:20Speaker 01

  And you have to constantly reinvent the product to be like suitable for the like the sort of frontier of model capabilities. A few months ahead, maybe. I think this sense was, so you still like maintain a lot of contact with like, you know, direct users and this kind of stuff. And you can, the product works to some degree, but then it'll it allows you to take advantage of the frontier capabilities.

• 7:39Jacob Effron?

  I feel like that's the recipe is that, you know, you have, while you're kind of waiting for the models to get somewhere, someone else is taking up your customer, the developer love and your customer base. And, you know, they'll, you know, they can probably integrate in some of this stuff as it.

• 7:50Speaker 01

  Right, exactly. And you and you saw that also with customer and this kind of thing, right?

• 7:53Jacob Effron?

  There's a lot of things in these in these models that you guys made progress on like memory, instruction following, tool use. So I guess, you know, again, kind of recontextualizing for folks like, where are we in these in these three areas?

• 8:05Speaker 01

  Yes.

• 8:05Jacob Effron?

  What works, what doesn't?

• 8:06Speaker 01

  Yes, okay. So a good way to think about what's happened with these models over the last year is because RL finally really worked on top of language models, there's, I think, no direct ceiling to the intellectual complexity that we've been able to like, teach these like the intellectual complexity of tasks with which we've been able to teach these models. So you see them doing incredibly complex math problems, incredibly complex like coding problems. But those things are like in scoped domains where it's like, we're at relatively limited context. The problem's there in front of the model. Things like memory and tool use, these are attempts to expand the set of like the sort of context of which the model is able to, like the context within which the model is

• 8:49Speaker 01

  able to act and the affordances it has. So this is the thing, things like MCP allow it to, you know, suddenly the world opens up to it and it's able to interact with the outside world. Memory allows it to run with much longer context, much greater degrees of personalization than just a raw model with its own context window. And so I think those efforts represent attempts to, to like crack agency by giving the model all these unhobblings in one respect. And I think, you know, the Pokemon eval is a

• 9:19Jacob Effron?

  I love that eval.

• 9:20Speaker 01

  It's a great.

• 9:21Jacob Effron?

  As an avid Game Boy player back in the day.

• 9:23Speaker 01

  Yeah.

• 9:24Jacob Effron?

  I feel like that's, it's a great one. I hope you're going to release that alongside this, this model.

• 9:28Speaker 01

  Yeah, yeah, the new one, the model has been playing Pokemon, so you'll see that. I, I think it's a great eval because it hasn't been trained for.

• 9:36Jacob Effron?

  Yeah.

• 9:37Speaker 01

  And so it demonstrates this like generalizability of intelligence to, to like, I mean, a task which is not completely out of distribution, but one which is like meaningfully different from anything it's done before. And another example

• 9:48Jacob Effron?

  I had to buy a strategy guide to beat that game. I remember like there's a lot of ladders and like there are places to go around.

• 9:53Speaker 01

  Exactly. Another example of this that I really like is there's been a recent interpretability agent that Anthropic's been working on. Basically what this does is it does that job of finding circuits in language models. And this is really, really cool because one, we haven't trained for it to do this. We've trained for it to, you know, do, you know, it's a coding agent, right? But it's able to mix that with its knowledge of theory of mind and this kind of thing to sit there and itself talk to the model that it's trying to understand, try and reason through what kind of, it has access to tools which are like looking at visualizing, you know, neurons and, and circuits and this kind of thing. And it is actually able to win this interesting, like alignment, safety eval, which is called the auditing

• 10:34Speaker 01

  game where you twist the model in some way and it has to figure out what is wrong with the model. And it is able to do that. It's able to talk to the model, you know, generate its own hypotheses about what might be wrong with the model and look at all these tools. I think it's just such a brilliant demonstration of Of the like generalizable competence of these models with access to tools and memory.

• 10:59Jacob Effron?

  Totally. I feel like builder's been waiting for, you know, agents and, you know, the ability to kind of use this stuff reliably. I think you've talked before on podcasts, I like the merit agents is reliability, right?

• 11:07Speaker 01

  Yes.

• 11:08Jacob Effron?

  How much progress have we made there? Like, you know, for the, for the builders that are listeners here?

• 11:12Speaker 02

  Hey guys, this is Rashad. I'm the producer of unsupervised learning and I just wanted to take a quick break from the conversation. You probably know what I'm going to say. If you haven't already rated the show, ratings are maybe the number one way that you can support the show and help us grow as well as sharing the episode with someone that you think would find value out of it. The way you rate the show is by going to the homepage, hit the three dots and then hit rate show on Spotify. Again, super helpful in helping us grow and continue to bring on the best guests like Sholto. So thank you so much for listening and now back to the episode.

• 11:44Jacob Effron?

  Verit Agents is reliability, right?

• 11:45Speaker 01

  Yes.

• 11:45Jacob Effron?

  How much progress have we made there? Like, you know, for the, for the builders that are listeners here?

• 11:50Speaker 01

  I really like the meteor for that. I really do think that measuring success rate over time horizon is the right way to think about this, you know, extension of a like agent capabilities. And I think we're making a hell of a lot of progress. We're not 100% there on reliability. You know, these models don't succeed all the time. There's still a meaningful gap between the performance of the model when you ask it to do something once versus the time versus when you ask it to, you know, try 256 times. There's many evals where you can like completely solve them, you know, with many event, many attempts, but on the first time it's not guaranteed.

• 12:21Jacob Effron?

  Yeah.

• 12:22Speaker 01

  With that being said, I think every trend line I'm seeing says that we are on track to, to get like expert, superhuman reliability at most things that we train on.

• 12:32Jacob Effron?

  Yeah. What would change your mind on that?

• 12:34Speaker 01

  I think if we were to, to basically like fall off trend line. So if let's say by the middle of next year, you started to see some kind of a Block on on the time horizon with which these models are capable of of acting. And I I think you should look at that, like coding is always the leading indicator in in AI. So I think like you you would see that drop off in in coding first.

• 13:00Jacob Effron?

  Yeah.

• 13:01Speaker 01

  But that would be perhaps reflective of inherent limitations in the room, which I which I really strongly believe there aren't. There are other limitations which where the task distribution might be harder than you think, where because, you know, there's less data available for something. It turns out to be actually quite a laborious process. And so maybe if you think about the computer using agents, then this would be an example where it's that kind of data doesn't natively exist. But at the same time, also, like, we're seeing just such incredible progress there that it feels to me like relatively.

• 13:32Speaker 01

  I don't think that's the world we're in at all.

• 13:34Jacob Effron?

  Yeah. When do you think I'll have like one of these just general interpretations that can, you know, fill out all my my forms for me and, you know, navigate the internet for me?

• 13:40Speaker 01

  Yeah, one thing I joke about is like like personal admin escape velocity, like how how can I put off doing a task?

• 13:47Jacob Effron?

  Exactly.

• 13:50Jacob Effron?

  As a procrastinator, that would be wonderful.

• 13:52Speaker 01

  Right, exactly.

• 13:55Speaker 01

  It depends. I I still think there's a meaningful.

• 14:01Speaker 01

  This depends a bit on the like whether or not a company focuses on on putting like at like at least something like by giving the model some practice reps. Like if you took a human off the street and you were like, you're a general intelligence, but like, I'm going to ask you to do my accounting and you're you're not going to make any mistakes.

• 14:16Jacob Effron?

  Right.

• 14:17Speaker 01

  Probably the person you pull off the street makes some mistakes, right? But, you know, if they've like done something similar to it, or they're a great mathematician or something like this, then then probably they will make many or they're a lawyer or whatever. Like if there's something for them to generalize from and map to, then they'll be able to do it with a much higher degree of likelihood. So. I strongly depend on what task by the end of next year, I think we should see like, it should be very obvious that this is near guaranteed. Even by the end of this year, that's like, that should be like pretty clear. But by the end of next year, you'll have these things going around, doing a lot of things for you in your browsers.

• 14:53Jacob Effron?

  Sounds great to me. Your models are really good at coding. Like, what makes them so uniquely good at coding? Is it like a prioritization internally? Or like, what, what, you know, I think like, people associate Anthropic as like, oh, they're the coding model company these days. Like, what is behind that?

• 15:05Speaker 01

  I mean, Anthropic does care a lot about prioritizing the things we think are important. And we believe coding is extremely important. Because coding is the

• 15:18Speaker 01

  that first step in which you will see AI research itself being accelerated. And so, so we care a lot about coding. We care a lot about measuring progress on coding. We think it's the most sort of important leading indicator of model capabilities. Yeah, I think it's a, it's a focus.

• 15:35Jacob Effron?

  Is coding accelerating, are these agents accelerating AI research today?

• 15:38Speaker 01

  Accelerates me a lot.

• 15:41Speaker 01

  Yes, basically, basically, yes, they accelerate the engineering a lot. I think it's, it's interesting to ask even people who are utterly brilliant engineers, like, how much is accelerating? A lot of my friends who are like, who I would regard as like the strongest people I've ever worked with. They say it's like 1.5X on, even on domains they know well. And on domains which they don't know well, it's like 5X. So if it's new programming, new programming languages or something you haven't done for a while, it's incredible accelerant. Now, it's one, like, very important, like, factor to consider on like, how much will AI accelerate AI progress? Is how much you believe that we are compute bounded or not compute bounded?

• 16:18Jacob Effron?

  Yeah.

• 16:19Speaker 01

  And so, and whether you think that if you deploy AI agents who can do the research for you, will that mean that you get sort of like gains proportional to, to the amount of like researchers that you're now deploying?

• 16:34Jacob Effron?

  At this stage, I imagine it's like, most of these things can do the annoying parts of your job. So you can think about the brilliant pieces of research to let go test. I mean, do you find, how do you think about the timeline for where, you know, these agents themselves are proposing like interesting research directions?

• 16:46Speaker 01

  I mean, a lot of the work is engineering work.

• 16:49Speaker 01

  I would say the majority of the work is engineering work at this point in time.

• 16:53Speaker 01

  When they're proposing novel ideas, I'm not sure, to be honest. Like, within the next two years, I think people are already starting to see interesting scientific proposals and this kind of stuff. I think also if you like allow, an important thing to consider, like the current sort of space of algorithms, these models, is they can become truly expert at something, provided they've had a feedback loop for that thing. So you need, it needs to have been allowed to practice a little bit, in the same way that like humans.

• 17:18Jacob Effron?

  Well, it also needs to be relatively easily verifiable, right? Are we going to get these models that are like unbelievable coders and like haven't made the slightest progress in some of these like more nebulous fields?

• 17:27Speaker 01

  Yeah. One point is that ML research is actually incredibly verifiable, like did the loss go down, right? So, so if you can get to the point where you can make meaningful like your proposals for ML research, you have the best RL task in the world.

• 17:43Speaker 01

  Even more so, I'd say that like general software engineering in some respects.

• 17:48Speaker 01

  Will we get progress on less verifiable domains? I'm very confident that we will. I think one interesting data point here is OpenAI's recent paper on, it was like questions to medic, like medical questions?

• 17:59Jacob Effron?

  Yeah.

• 18:00Speaker 01

  And did you notice how it was evald or how it was scored?

• 18:03Jacob Effron?

  Well, they had the new medical evals that they put forward, right?

• 18:05Speaker 01

  Yeah, the new ones, right. And they had greater feedback. So they had like all these questions, whereas like the kind of like a long form answer that you'd have in an exam, and they, they gave like points for it. And so this is taking in a domain which is not inherently verifiable, the same way code or math is, and converting into something which is much more verifiable. I think we have, I, I think this is like reasonably likely to get solved. Like, I think there's a reason likely to already be solved, basically, and near guaranteed to get solved eventually.

• 18:34Jacob Effron?

  When, when's eventually? When will we have a really good like medical or like law? Or, and I guess like, has that just become part of like the broader model?

• 18:40Speaker 01

  Within the next year, yeah.

• 18:42Jacob Effron?

  Does it become part of the broader model, or do you think there's like, you know, oh, this is like the legal specific one or medical specific one?

• 18:48Speaker 01

  I'm, I'm a bit of a like large model maxi in this respect.

• 18:52Jacob Effron?

  I think most, most researchers are.

• 18:54Speaker 01

  Most researchers are, yeah, exactly. I do think there's a lot of really interesting ways in which personalization of models matters a lot, right? Like, you want something that is, understands your company, understands the things you care about, and understands you yourself. And so there's a lot of ways in which tuning models for your things does matter. But I think this won't be industry specific things so much as company or individual specific things. And, you know, I think Anthropic has a, as partnership with Databricks, where we're doing company specific stuff, but. On a base level of capabilities, I firmly believe that it's like single raw, like large models. And I think this for a number of reasons. One, it's the trend that we've seen so far. But two, because there's no reason in the long run for the distinction

• 19:39Speaker 01

  between small and large models to exist. Like, you should be able to adaptively use the right amount of like work, so to speak, the right amount of like flops for, you know, the difficulty of a given task. And so I think that that means that biases towards large models. It seems like you're you're pretty convinced in the continued improvement of these models. Yeah. You know, I think a lot of people speculate it's like, okay, the models will keep getting better. And then like, how did that diffuse into society? And I guess one, you know, thing people like to talk about is basically impact on GDP, right? And like, you know, in the next few years, like, what impact on world GDP do you think these models have? Yeah, okay. I think probably the the initial impact looks something like China as an emergence. Because there's going to be like What is the thing that is probably most impacted world GDP in the last Right.

• 20:21Speaker 01

  Hundred years. Like, you know, you look at like Shanghai over like the course of 20 years and like dramatically transforms, like, you know. But and this will be like dramatically faster than that. But you'll see that. But there's important distinctions to be made here. One is that I think we're near guaranteed at this point to have effectively models that are capable of automating any white collar job by like 27, 28 and or near guaranteed end of decade. That being said, that's because we have, like, those are those tasks are quite susceptible to our current current suite of algorithms. Like, you know, you can try things on computers many times. You can, there's like a wealth

• 21:02Speaker 01

  of data available for this. They don't, you know, the internet exists, but that same resource of data doesn't exist for, say, robotics or for biology. And so for a model to be a superhuman coder, you just need the affordances, which we've already been able to give the models. And you need to sort of like take the existing algorithms and like scale it up. For a model to be a superhuman biological researcher, you need automated like laboratories where it's able to propose and run experiments in a hugely parallelizable way. Or for it to become as competent in the real world as we are, you need it to be able to act in the environment through robotics. And so you need a hell of a lot of robots to actually like, collect the data and do that in a way that unlimited. So one

• 21:39Speaker 01

  Mismatch that I think we might see, and I'm actually also worried about us seeing is you'll see a huge impact on white collar work. And whether that looks like just dramatic augmentation, you know, like TBD, but you will see that world change a lot. And we'll need to pull forward the, like, the dramatic transformation of things that make our lives a hell of a lot better. So to pull forward medicine, to pull forward abundance in the real world, we need to, like, actually figure out the, you know, the cloud laboratories and the robotics and this kind of stuff.

• 22:12Jacob Effron?

  But by that time, we'll have, like, you know, millions of AI researchers, like, for

• 22:15Speaker 01

  Right, exactly.

• 22:16Jacob Effron?

  They don't need such a large scale of robotics or biological data.

• 22:20Speaker 01

  So AI progress goes really fast. But we need, we need to make sure that we, like, pull in the feedback loop to the real world to actually deliver on, on, on, like, meaningfully changing world G P and this kind of stuff.

• 22:31Jacob Effron?

  Yeah. So you basically think, like, each white collar profession, like, they'll be, you know, you'll be able to build some sort of reward model somewhere to, you know, have some in this, in this South Korea vales. And, you know, I think what's always surprising is how limited data you need to actually build those things.

• 22:43Speaker 01

  Yep.

• 22:43Jacob Effron?

  You know, in the same way that, like, a human learns to do this on, like, relatively limited data.

• 22:46Speaker 01

  Right, exactly. And, and even, I think what we conclusively demonstrated is that we can teach models on, like, so far, we haven't yet hit, like, an intellectual ceiling on the tasks which we're able to teach the models. Now, they do seem somewhat less sample efficient than humans. That's also okay. And because we can, you can run thousands of copies of them in parallel, and they can be interacting with different variations of tasks, like, they can, you know, have lifetimes of experience. And so it's okay if they're more, like, less sample efficient.

• 23:15Jacob Effron?

  Yeah.

• 23:15Speaker 01

  Because you still, you still get, like, expert human reliability and performance at that task.

• 23:19Jacob Effron?

  It seems like you think this, like, you know, this paradigm kind of gets us, you know, pretty much all the way there.

• 23:24Speaker 01

  Yeah.

• 23:25Jacob Effron?

  You know, obviously, you have folks like Ilya who have been saying, look, you know, there needs to be some sort of other algorithmic breakthrough. What's the other side here?

• 23:31Speaker 01

  Yeah, makes sense. I think most people in the field currently believe that, like, The like pre-training plus RL like sort of paradigms which we've explored so far are like themselves sufficient to reach AGI. This like, we haven't seen the trend lines like bending yet, like this, like it works, like this, this sort of combination of things. Whether there are other mountains to climb that could get us there faster, it's entirely possible. I mean, Ilya has invented like maybe both of these paradigms before. So who am I to bet against him, right? Every piece of evidence I see says that these are sufficient. You know, maybe Ilya is betting that way because he wants to, you know, like, he doesn't have

• 24:14Speaker 01

  the capital, like as much capital available, or he thinks that this is a better way to do it. Entirely possible. Like, I'm not going to bet against Ilya, but I do think what we have now will get us there.

• 24:24Jacob Effron?

  The limiting factor in this will be energy compute. Like, when do you think we start to bump up against that?

• 24:29Speaker 01

  I think there's a great table at the end of situational awareness, which details this, where like by the end of the decade, we start to revert like really like dramatic percentages of US energy production, like you're over 20, I think maybe 2028, like 20% of US energy. And so you can't go orders of magnitude more than that without like dramatic changes. This is somewhere I think we need to invest more. I think this is one of like the important vectors along which governments should act. Dylan has this wonderful graph of China's energy production versus US energy production. And the US energy production is flat, and China's energy production is like this. Like, they're just doing a much better job than we are of building out energy. And so, yes, we.

• 25:09Jacob Effron?

  Yeah. I guess then in this current wave of model improvement, like what metrics are, I mean, it seems like it's time horizon based metrics, but like what are the things that are worth hill climbing on right now? Like, you know, as you, you know, move from four to whatever comes after four?

• 25:22Speaker 01

  Yeah. I think in general, I've been impressed by internal company evals. There's many companies which like have a advisor in their own version of like Sweetbench, let's say. And these are quite rigorous and like well held out. So I enjoy hill climbing those. I also think really, really complex tests, like frontier math is really interesting, want to watch over the next year, because that represents such a ceiling of intellectual complexity that I think is interesting. But more and more, I think what matters is

• 25:56Speaker 01

  Eval is really hard to produce. If we could produce evals which meaningfully capture the time horizons of people's workdays, I think that would be the best thing to produce. But no one has gone out there and produced that in public. This is another thing which I think governments should do because I think understanding what the trend line looks like is such an important input into policy. And this is also something governments are well placed to do, where they should be producing like, what does the inputs and outputs look like of an hour or a day of a lawyer or an engineer's daily workday? And can I convert that into something that's gradable? And so that we can actually measure progress against it.

• 26:31Jacob Effron?

  I guess on the set of problems that you have to overcome as a foundation model company, like where does having good evals rank on the list?

• 26:39Speaker 01

  Yeah. I mean, every foundation model company has a really big evals team full of great people working incredibly hard to do this. I mean, I think the core is like the core algorithmic and infrastructure challenges of even training the thing. But without good evals, you don't know what your progress is at all. And it's hard to keep excellent evals fully held out. So it's important to have good internal evals that you trust.

• 27:03Jacob Effron?

  But also I'm struck by having people building applications on top of your models that are willing to share the way they think about evals is incredibly helpful to.

• 27:10Speaker 01

  Exactly.

• 27:11Jacob Effron?

  Because obviously, especially as you get into a lot of these different verticals that you might want to improve on, it's hard for you guys to figure out what is the specific thing in logistics or legal or accounting or whatever it is.

• 27:23Speaker 01

  And requires such expertise and taste. I think that's another one of the stories of the last couple of years is that you went from model outputs being, you could pull anyone off the street and say, hey, which output do you prefer? And it would meaningfully improve the model to needing grad students or experts in their field to be able to improve the outputs of the models. I mean, if you put me, I know some field that I don't know very well, like biology or whatever.

• 27:47Speaker 01

  And and put two model outputs in front of me, I would I would struggle on a lot of them. Like, I don't, I wouldn't have the expertise to know which one is a better answer.

• 27:53Jacob Effron?

  I guess the idea of taste, I mean, I'm struck by like, you know, the way, obviously, you've seen memory now put into a lot of the way that consumers interact with these models. But it seems like part of the reasons different AI products seem to have taken off is like they find some like, they struck a chord and like the zeitgeist of like the way that they're, you know, I see you guys had this with like your Golden Gate like example. And there's been tons of other things like this. What does this like, look like in the future in terms of like model customization, like to the, I guess, for the vibe of like the end users?

• 28:20Speaker 01

  Yes. Well, I think actually there's a, there's a weird sort of future where these models end up being like one of your most like intelligent and charismatic friends.

• 28:28Jacob Effron?

  I don't know about your friends, but already pretty close.

• 28:31Speaker 01

  Right.

• 28:34Speaker 01

  And so I hope that, and I think almost none of our models are like, they're decent along these axes, but I know many people who, who spend like a lot of hours talking to Claude, actually. But I, I think there's, there's so much further that we could go. And I think we haven't like, we've explored like 1% of the depth of like personalization and like understanding of the model could have of you.

• 28:55Jacob Effron?

  How do you get better at that? Is that like, you know, people that had just like exceptional taste, like being opinionated in the way that they're steering these models? Or how would you even go about solving that?

• 29:03Speaker 01

  I mean, I think a large part of that reason why Claude is so good in that way is Amanda and like her taste. And I think similar to like beautiful products, an important part of that like is singular taste. And, you know, we've all seen the sort of perils of A B feedback mechanisms and like thumbs up, thumbs down, just like lead you down a dark path, basically. I think in part, these models are such wonderful like simulators in some respects of like, you know, they've been asked to model the entire, entire distribution of the internet. So I think one of the ways that this was solved is just by providing an extraordinary amount of context about yourself. The models should actually almost be automatically really good at understanding what you want.

• 29:41Speaker 01

  And then in designing the personality and this kind of thing, probably individuals with taste and then like, you know, your own sort of conversations and feedback with the model. Some combination thereof.

• 29:52Jacob Effron?

  I'm sure you had a bunch of people playing around with these models before they were released. Like, any stories that you particularly resonated?

• 29:58Speaker 01

  I think it's just everything has been a noticeable step up in my confidence in asking, like, turning to the model first, I suppose. I have also enjoyed, I think, how relentless these models are in some ways. I mean, is that a good word? I don't know. But like, this is great. So we have this great eval.

• 30:20Speaker 01

  Where in this eval, the model is meant to fail. It's meant like it's like something on Photoshop or whatever. And it's like not meant to be able to do that thing in Photoshop. And so the model goes, oh, well, I know I can't do this in Photoshop. So I'm going to download like this Python like library. And I'm going to do it with the Python library and then upload it into the Photoshop thing. And look, hey, I've done it. And so maybe it's not relentless, but there's a creative and like mischievous.

• 30:43Jacob Effron?

  Yeah, like something you wouldn't expect it.

• 30:45Speaker 01

  Exactly. Like, I thought that story was pretty cute.

• 30:49Jacob Effron?

  Yeah, that is really cool. So what, I mean, obviously you've got these new models out today. What do the next like six, 12 months look like in your best guess?

• 30:57Speaker 01

  So the next six to 12 months very much looks like, you know, scaling up RL and sort of exploring where that gets us. And I think you should expect to see incredibly rapid advances as a result of this. It is in many respects, I think, like Dario outlined this in his essay about D C, where he said that like imperatively small amounts of compute have been applied to the RL scaling regime compared to the pre-training regime. And this means that there's like still such huge gains to be made, even with existing pools of compute. And the pools of compute are dramatically multiplying this year as well. So expect to see continual rises in model capability. Like, basically, by the end of this year.

• 31:44Speaker 01

  The coding agents, one good metric will be the coding agents that are taking their first like halting steps today should be very competent. You should, you will probably feel very confident in delegating substantial amounts of work for hours on like hours of few minutes.

• 31:58Jacob Effron?

  What's gonna be your check in time?

• 32:00Speaker 01

  Yeah, exactly. What does the check in time look like? And like at the moment with Claude Code, you know, sometimes it's five minutes. Sometimes it's like you're sitting there watching it in front of you. By the end of the year, it's probably several hours of like confidently doing this for many things. Whereas now, yeah, sometimes it models able to do several hours, sometimes you're able to do huge amounts of work, but it's spiky.

• 32:21Jacob Effron?

  Yeah, I feel like that's made the game to do things. I feel like, you know, one of the lessons even from like RPA is like you have to sit there and watch something, do your work. At some point you're like, I'd rather just do this like myself.

• 32:29Speaker 01

  Yeah, sometimes, right? Sometimes you step in and eventually we'll like be able to delegate that. I think someone tweeted a little ago that like the future of like software engineering looks like Starcraft. And I think, yeah, when do we, when do we get like Starcraft level, like your, your sort of APM of like coordinating all your pieces? That's probably the end of the year.

• 32:45Jacob Effron?

  So what does that mean then from like a model release cadence? I mean, if you guys are scaling this so quickly, like, does that mean that like, you know, how like often do you would think people, you know, all the labs end up like shipping new models in this like period of rapid adjustment?

• 32:57Speaker 01

  I would expect to see the model cadence substantially faster than last year. In many ways, 2024 was, was a sort of deep breath in as people figured out, you know, the new paradigms and did a lot of research and, and sort of like better understood what's going on. And I expect 2025 to feel meaningfully faster.

• 33:18Speaker 01

  Where particularly also because as you, as models get more capable, the, the set of like reward available to them expands in important ways. You know, if, if, if you have to give feedback on every single like sentence that outputs, right? This is, this is very not very scalable. But if you're able to allow it to do hours of work in such a way that it. You can just judge, did it complete the thing I wanted? Did it drew the right piece of analysis? Did the website work? And were people able to, you know, message on it and this kind of stuff? It means that the, basically, it should be able to climb these like rungs of the ladder ever faster, even though the, um, the complexity of the task is increasing.

• 33:59Jacob Effron?

  You mentioned earlier, there's like OpenAI Codex, there's, you know, Google Jewels, there's like all this different stuff, there's all these startups building out like.

• 34:04Speaker 01

  And we're actually, we're launching a GitHub agency. You'll be able to like anywhere on GitHub, you'll be able to say, hey, at Claude, um, and, and it will, you know, spin off and do some work for you.

• 34:12Jacob Effron?

  Yeah, so everyone is like competing for the, uh, for the hearts and minds of developers. Like, what do you think will determine, you know, which, uh, tools and models developers use?

• 34:18Speaker 01

  Um, I mean, I think a big part of this relationship between the companies and developers, um, and, and, you know, how much trust you impart to each other. A big part also is the, like, trust and respect, right? Um, between the companies and developers. I think a large part is also the model capabilities. Um, which ones are the models? Which ones people are actually like comfortable, happy and enjoy using? Like, you know, the personalities and the ability, like the competency of the model and the trust you have in it to go off and do these tasks for you. And I hope also that, uh, you know, over time, um, as the, the sort of stock capabilities of these models become more and more and more apparent, uh, that the like mission of the company as well becomes, you know, important. And that you're like, you think of like which companies you're working with as a, as like, who you're sort of trying to build the future with.

• 35:02Jacob Effron?

  Cause I'm sure, I mean, I just like, you know, especially at the cadence of releases keeps going up. It's like every, you know, every month people will be inundated with like, well, this one climbed on this eval and that one climbed on that eval, so.

• 35:11Speaker 01

  This is actually, I think this is like an interesting way. This is one of the things people didn't expect about, um, like, you know, GPT rappers, right? Is that one of the benefits of wrapping like other model companies is that you can surf the frontier of model capabilities.

• 35:26Jacob Effron?

  Oh, a hundred percent. I feel like everyone that tried to not be a rapper just lit a lot of money on fire.

• 35:30Speaker 01

  Right, exactly. And so like surfing that frontier of model capabilities is really wonderful. There's a, there is a like a reverse effect where, uh, there are certain things you can only, you can, you can sort of only predict, uh, if you have access to underlying models, like you can really feel and see the trend lines or you can only build, um, if you, if you like, I think all of the. Deep research equivalents took some amount of RL in such a way that like it was hard to build a deep research equivalent product from outside one of the labs.

• 36:03Jacob Effron?

  Can you just explain that, actually, like, why is that? Because, obviously, like, you know, increasingly, I think, well, OpenAI is RFT. I'm sure you guys have some equivalent. It seems like they're opening up, like, you know, to the outside world. Like, what? I guess this is actually a big question that I think about, a lot of people think about. It's like, what are like the labs going to be like uniquely good at building? And then what is kind of fair game for anybody? And the labs will try, but the apps will be in as good a position to do.

• 36:24Speaker 01

  So I think with the release of RFT APIs, this changes a bit, right? Because you sort of, there is now benefit to companies specializing in domains. But then there's also going to be those same centralized benefits. Like, I think, you know, at least mine is saying is definitely OpenAI allows people to give some discount. I think they can also train on the output some more. So that there is going to be some centralizing benefit to being the company that like has the RFT API and that people are fine tuning on.

• 36:57Speaker 01

  And so what are the labs going to be uniquely good at? I think a very important part here, like, so a couple, couple of dimensions. One is the like main metric that the labs will be judged on is how effectively they are able to convert accelerators and like flops and dollars, like capital into intelligence. Like, that is the most important by far metric. And this is the metric that has sort of distinguished companies like Anthropic, companies like OpenAI and DeepMind from really like the rest of the pack, right? It's like the models that are trained by these companies are better. Then the next most important thing after that, I think, will be the like, you're going to have these models going to be like employees pretty rapidly. It's going to be the trust and like, do you like them?

• 37:40Speaker 01

  And do you like, do you trust them to carry out the things that you ask them to do?

• 37:47Speaker 01

  So I think that will be an important differentiator. And the personalization will also be an important differentiator. Like, how well does the model understand you and your context and your company?

• 37:54Jacob Effron?

  I'm sure, but you have people building like, you know, general purpose agents on top of, on top of your models, right? Not being a model company, just being like, we'll take the models off the shelf and we'll, we'll do the orchestration. We'll do like really smart chaining. And is that like a doomed task to some extent? You know, even just to articulate, like, what is the advantage that the model companies themselves will have by just like, obviously, cost advantage makes total sense versus the API. And like, you're surrounded by people that know, look and know these models deeply well.

• 38:16Speaker 01

  Yeah, no, I mean. I think this is actually a good thing also, right? Like it encourages an incredible amount of competition and like finding the right form factors and this kind of thing. I think there are some advantages to model companies, having access to the models and like being able to like, you know, really make sure, I think the RFT APIs don't work brilliantly at the moment. So they're like, it's like this whole thing process or being able to tune the models for things you think are important. But I think the waterline is going to keep going up, basically. If like ultimately you are harnessing this like intelligence on tap, like an employee that you're hiring or just like the sort of raw capability of intelligence. And so, um, yes, there are going to be, uh, you know, companies that wrap and, you know, uh, orchestrate these models.

• 39:02Speaker 01

  And in many cases, they're going to do fantastically well. And I'm not sure actually like who has the advantage or who doesn't. But, uh, the like underlying trend is going to say like, say true, like there's this raw intelligence being distilled and made available. And so, um, if a company successfully wraps, uh, you know, like this API, that's fantastic. It's also going to face a lot of competition, like ultimately all modes disappear in like, in like the sort of like T goes to infinity. Um, in some ways, because you'll be able to like spin up a company on demand, um, so to speak. And so I think that's like, it's an interesting and complex future where, uh, where does like value create? Is it in the customer relationship? Is it in like the ability to orchestrate and like, you know, you're pulled together?

• 39:47Speaker 01

  Like, is it ability to like meaningfully convert capital into intelligence? Who knows?

• 39:51Jacob Effron?

  I think our listeners would be super curious. Can you describe like what does day to day work like? Is a cutting edge AI researcher look like these days?

• 39:59Speaker 01

  Yeah, I think that's a good question. So the, the fundamental thing that like you are trying to do at these companies is one of two things. It is either to.

• 40:11Speaker 01

  Develop new compute multipliers. And so that is like the process of doing the engineering of making the, you know, the research workflows really fast and thinking through what we currently like, you know, what issues are there with model or what I sort of algorithmic ideas would be like to be able to express? And, you know, doing the science of studying how those develop. And so there's like this, this very like integrated research and engineering,

• 40:35Speaker 01

  like form of work where it's all about iterating on experiments and like building experimental infrastructure and, and making that process as, as, as like clean and fast as you possibly can. And then there's the process of scaling up. And so this comes with its own host of, of research and engineering challenges where you take these ideas that you think will work. And so, and that you've, you know, debated with all your colleagues about what the right ones to include in the, in the riskier run.

• 41:03Speaker 01

  And, and you scale this up in a much, in a much larger run, where this has whole new infrastructure, set of infrastructure challenges where, you know, you're running it especially you need to be way more failure tolerant, this kind of thing. And also new algorithmic and learning challenges. So there are things you are only going to see at each successive oom of scale. That you then need to go and figure out scientific reasons for why those occur. And see if you can sort of study the early emergence of those. And then, and then like create experiments that allow you to address or take advantage of those effects. And, and include those in the next large run. Yeah. So yeah, this constant loop of like pushing on those two axes in a way that really combines a lot of science and engineering. So where do you use AI throughout that?

• 41:49Speaker 01

  One, a lot in the engineering at the moment. Like the, the primary way it's helping is like in engineering. It is also in implementing research ideas. So I think one way of like seeing the early ability of these models to help here. If you take like a single file transformer implementation, like you know, Kappa, these min G P T or something like this, and ask the model to implement ideas that you see in papers, you will be stunned by how good it is. It is just like kind of wild. And then if you go into like some. Huge like transformer code base and ask it, you'll notice that it's actually, it's a little bit harder. Like the models struggle a bit more there, but they struggle less and less every month.

• 42:28Jacob Effron?

  Yeah.

• 42:28Speaker 01

  So that's like a good way to presage the future, like distill the context down to just what matters. And then like ask the model to do this in the same, then you'll, they'll be like struck by how good it is at helping you do research.

• 42:39Jacob Effron?

  Yeah.

• 42:39Speaker 01

  Basically.

• 42:40Jacob Effron?

  You've obviously been really close to this stuff, trying all sorts of things like, what's one thing you've changed your mind on in the last year?

• 42:44Speaker 01

  Yeah.

• 42:46Speaker 01

  Over the last year, I think the pace of progress inflected upwards substantially. So I think, and last year, I think it was, you could have been uncertain about, will we need to, you know, reach, uh, you know, many more rooms of pre-training compute before we get the level of capability that we expect to see really by the end of this year. And now the answer to that is conclusively no, RL works and these, you know, the models will get to that drop in remote worker by 2027. You will have incredibly capable models by then. And so like all of the, uh, um, both like hopes and concerns like suddenly become, I think, like they were already real.

• 43:29Speaker 01

  And now they're substantially more real, um, in, in many ways.

• 43:32Jacob Effron?

  Realistically, do you think we end up having to like massively scale data or like by the time you've made, you know, Claude 17 and these coding models are so good. You know, they find so much algorithmic improvement that the, uh, the amount of data more we need is not, not too much.

• 43:45Speaker 01

  Well, the models might be like, the models might be good enough then, they might, their, their understanding of the world might be good enough then that they can like, give enough feedback to encroach the, the robots through things, right? It might be, there's this concept of what's called a generator verifier gap.

• 43:59Jacob Effron?

  Yeah.

• 43:59Speaker 01

  Where, um, you know, if it's, if it's easier for the model to rate something than it is for the other models to do something, then you can like improve up to your ability to like critique or it. I think robotics is like quite potentially one of the areas where this is true. And I think this is like also true of, uh, of many domains, but robotics is like, this is starkly true because, uh, our, our sort of progress in understanding the world has, has gone so far ahead of our ability to manipulate it physically.

• 44:26Jacob Effron?

  How would you characterize the current state of like, uh, alignment research?

• 44:29Speaker 01

  Interpretability has gone, undergone crazy advances. I know if you've been like, uh, following, there's, there's some beautiful, uh, pieces of work here, um, that I've been really, really impressed by. Where like last year, the state of models was we were just beginning to discover like superposition and features and the work of Chris Allar and his team and just like already that was a significant leap in understanding. But now we actually really meaningfully have circuits in true frontier models. And we can characterize their behaviors by, there's a beautiful paper we should read on the biology of a large language model, where they do break down the ability of these models to reason over your concepts and in extremely explicit terms.

• 45:13Speaker 01

  And we don't have a full characterization of the models. And there's still a lot of difficult cases here. But also the models are, the models are like quite good. One important dynamic to like explain here is that based on pre-training, the models are quite good at just generally ingesting human values. Like they're quite off pre-training, they're quite default aligned in many ways. Off of RL, that's no longer guaranteed to be the case because you're putting these models in, you know, that same model that is like, hey, I downloaded the Python code like and like hacked around the fact that I was meant to fail this test, like is the kind of model that, the kind of learning process that means the model will do anything to achieve the goal that's been given. And so overseeing that is like itself a tricky process that everyone is currently learning to go through.

• 45:58Jacob Effron?

  Yeah. I mean, obviously there was like, I felt like, you know, about a month ago, like AI twenty twenty seven came out, a lot of people were talking about that. Like, what was your reaction to that?

• 46:05Speaker 01

  Honestly, it felt very plausible. I was reading that and for a lot of it, I was like, yeah, you know what? Like, this might actually be, this might be how it happens. I think there's like branching possibilities there, but and this is maybe like the twenty percentile case, but for me, but also the fact that it's the twenty percentile case is kind of crazy.

• 46:25Jacob Effron?

  Yeah. Is it twenty percent off you because you find yourself more bullish on like alignment research than them? Or you just think the timeline's slower?

• 46:30Speaker 01

  I think I am more bullish on alignment research than them for the most part. And maybe my timeline is like a year or so slower, but also in the like scheme of things, what is a year? Like, yeah.

• 46:41Jacob Effron?

  Yeah, it depends if you take advantage of it.

• 46:43Speaker 01

  Right. If you take advantage of it and you do the right research and this kind of thing. Yes.

• 46:46Jacob Effron?

  If you were kind of playing policymaker for the day, like what, what should we be doing to like ensure things are on a better path?

• 46:52Speaker 01

  Yeah. Okay, so that's a good question. There are, the most important thing is you need to really viscerally feel the trend lines that we're all seeing and talking about. And so if you don't, then break down and like understand. Like break down all the capabilities you care about in your country and like measure the capability of the models to improve on this. Like get trend lines that you would, if they were solved.

• 47:17Jacob Effron?

  Like nation stat, state evals.

• 47:19Speaker 01

  Yeah, like nation say, like, you know, you break down your economy, you got like your sort of like all the jobs that are done in your country and like convince yourself, like build tests that if the models could pass them or make meaningful progress towards tasking them, then that would be your benchmark of intelligence and plot the trend lines and then go, oh my God, what happens in 2027 or 2028? The next thing is you should be investing meaningfully in, in the, in the research that we think will like go towards helping make these models like understandable and, and steerable and honest. And so a lot of that looks like the like science of alignment, basically. And it's actually something which I've been like sad in some respects has been driven so much by the frontier labs. There's actually something which I think that.

• 48:03Jacob Effron?

  Can other people work on it?

• 48:04Speaker 01

  Yeah, absolutely.

• 48:05Jacob Effron?

  Have access to like, you know, Claude four to.

• 48:07Speaker 01

  No, no, I mean, I think you can make incredible advances on, on like interpretability. And there, there are ways like there's this program called the maths program where people have done, you know, a lot of really like meaningful alignment research and interpretability in particular from outside the frontier labs. But it's something which I think a lot more universities should be thinking about. It's, it's in many respects, it is closer to the pure science of what's going on in these models. Like this is the biology and the physics of what is going on in, in language models.

• 48:37Jacob Effron?

  Yeah. Why don't you think there's more?

• 48:41Speaker 01

  I'm not sure. I, I, I really am not sure. I think people have described it to me as like it's, it's a bit, a bit of a risk. I think the mechanistic interpretability workshop wasn't included in one of the recent conferences like ICML or something, which is crazy to me because it is the closest thing in my opinion to the raw, yeah, raw science of what's going on in these models. If you want to like discover the charality of, of, of DNA or you want to like sort of like discover general relativity or something like that. For me, the tech tree for that in ML and AI looks like exploring mechanistic interpretability.

• 49:15Jacob Effron?

  Yeah. What about the good cases? Like what are we, what are we underthinking? You know, at a minimum, you're saying we're going to have all white collar jobs automated in a few years, so.

• 49:22Speaker 01

  Yeah, well, yeah. That the models will be able to do it. But actually like one of the things that's surprising sometimes, I mean, not surprising to you, but like. The world is surprisingly like slow sometimes to integrate these things. Already the model capabilities are actually like quite stunning in many ways. And if workflows were oriented around them, they're still like, even if model capabilities stalled right now, there would still be just a ridiculous amount of economic value and like reorienting the world around like using the current level of capabilities.

• 49:54Speaker 01

  But anyway, that's like a sort of a side point. This comes back to what I was saying before about we need to make sure we invest in all the things that actually make the world better. So this is like pulling forward the material abundance. It's, you know, reaching like, you know, the escape velocity of admin and this kind of stuff and like setting up the models to be able to like do all those things for us. It's, you know, pushing forward the boundaries of physics and entertainment and this kind of stuff. And like, my hope is that like, people are able to be dramatically more creative than they are now. Like, one of the failure modes, I suppose, of our current society is that people consume a lot of like, you know, mirror and this kind of stuff, but they, hopefully these tools, like in the same way that you're able to like vibe code, you'll be able to like vibe

• 50:38Speaker 01

  create, you know, a TV show with friends, or you're like a vibe create video game worlds. Like, there should be like this intensely, people should feel dramatically more empowered. Because all of a sudden, you're being given literally the leverage of an entire company of incredibly talented models or individuals. And so I'm excited to see what people do with that. I think like, that is underrated, maybe. Like, there's a aspect of, oh yeah, God, it's gonna like directly replace like the things that are currently done in the economy for work. I think that's, yeah, that's very likely. But I also think that everyone should feel like they will have access to dramatically more leverage. And like, the world is not solved yet. Like, the sort of work that is, the work that is, occurs right now, everyone's, like, lives could be dramatically better.

• 51:24Speaker 01

  And so, solving that, I think, will become the interesting challenge.

• 51:27Jacob Effron?

  I love that. Well, we always like to end our interviews with a quick fire round, where we get your takes on some overly broad questions, many of which I think we've actually already covered today, but I will, I'll dig into a few others. What do you think is like overhyped and underhyped in the AI world today?

• 51:40Speaker 01

  Okay.

• 51:42Speaker 01

  Let's start with underhyped. Underhyped, maybe world models, I think, are pretty cool, and something that we haven't really discussed in this, in this one. I think you're gonna see

• 51:54Speaker 01

  As technology for like augmented and virtual reality gets better, you're going to be able to see these like models literally capable of generating virtual worlds in front of you. I think that's going to be a pretty like wild thing.

• 52:02Jacob Effron?

  And then That requires like some sort of like physics understanding there, right? And like, you know, cause and effect, a bunch of things that we don't seem to have yet.

• 52:08Speaker 01

  Uh, I think we've, I think we've demonstrated physics understanding, to be honest. Yeah, I think they're, I think we've meaningfully demonstrated like cost effect and physics understanding, both in, in evals of like physics problems, but also in, if you watch any of the video models, like they, they get physics.

• 52:29Speaker 01

  And, and even like they, in weirdly generalizable ways, like I saw this great video of someone asking, one of the video models, I don't remember which one, to put a Lego shark underwater. And it was like reflecting the light in the right way off the Lego bricks and it had the shadows in the right place. And this is something he's never seen before. This is like fully generalized physics, right? That was pretty cool.

• 52:47Jacob Effron?

  And that wasn't in the training data.

• 52:48Speaker 01

  That wasn't in the training data. There's no, there's no Lego sharks underwater. No Lego sharks underwater. I'm hopeful also that this like same kind of technology translates towards things like virtual cells and that kind of stuff.

• 52:59Jacob Effron?

  Yeah.

• 53:01Speaker 01

  So I think that's exciting.

• 53:02Jacob Effron?

  Yeah. You mentioned earlier that like even if we stopped, uh, model improvement today, there's like just, you know, tons and tons of applications that we could build on top or ways to do it. What do you think like the most underexplored application is where like, God, I wish like more people were doing X with these models or?

• 53:15Speaker 01

  Yeah. I mean, I think they've been felt in software engineering. Because people like software engineers, one, the models are better at software engineering. But two, uh, I think software engineers like more implicitly understand how to like solve the problems that they care about.

• 53:34Speaker 01

  I suspect there's still a lot of headroom in, in basically like every other field. And you should sort of expect to translate the, the same, like, no one has yet built a sync background software agent, but for like any other field, right? Or even really like anything which comes close to the, uh, like feedback loops of Claude code and cursor and wind surf and this kind of things. For any other field. So I think probably that.

• 54:02Jacob Effron?

  Yeah.

• 54:02Speaker 01

  If anything.

• 54:03Jacob Effron?

  I guess people say coding is the ideal, you know, problem for these, uh, for these models.

• 54:06Speaker 01

  It is. It is. It's the leading indicator.

• 54:08Jacob Effron?

  Right.

• 54:08Speaker 01

  But you should expect everything to follow, basically.

• 54:11Jacob Effron?

  That makes sense. I mean, I guess, obviously, in your time working on this, you know, you probably come to be much more AGI pilled than you were in the beginning. Has that changed at all, like, the way you, like, live your life or plan your life?

• 54:21Speaker 01

  I started pre AGI pilled. I read this Guern essay that was really important, um, in, in convincing me, uh, in, in 2020, actually.

• 54:30Speaker 01

  The last year of, of RL progress really did cause like substantial inflection in that. Do I live my life that dramatically differently? No, I work a hell of a lot. Um, I think this is like the most important thing to work on. And so I, I like devote my life to it, basically. But, um, apart from that, I don't really live my life that differently. Uh, we have this funny joke, me and, um, uh, my friend Trenton. One delineation between us is that I still wear sunscreen. He doesn't wear sunscreen anymore. He's like, nah, just, you know, we'll figure out the biology of life.

• 55:00Jacob Effron?

  That's good cop, good confidence.

• 55:02Speaker 01

  I'm like, you know what? Biology's hard. You know, the feedback loop's biology hard. So I'm, I'm wear sunscreen.

• 55:08Jacob Effron?

  Just in case, uh, we hit a wall, you know, you don't want to be, uh.

• 55:11Speaker 01

  Well, it's true. Just in case biology takes 10 years.

• 55:13Jacob Effron?

  I guess you tweeted a picture of you, I think, of the Citadel.

• 55:15Speaker 01

  Yes.

• 55:16Jacob Effron?

  What was up with that?

• 55:17Speaker 01

  Uh, that was a war game.

• 55:18Jacob Effron?

  Yeah. What does that, what does that mean?

• 55:20Speaker 01

  Uh, the, I was invited to, to hang out with some, um, people from three letter agencies and military cadets and, uh, it was basically like a gaming out. Let's say like AGI, you know, comes along and it's getting much better. Um, and what are the geopolitical implications of that?

• 55:37Jacob Effron?

  Did you walk away like, you know, more terrified or less terrified after that experience?

• 55:42Speaker 01

  But, you know, maybe a little bit more terrified.

• 55:45Jacob Effron?

  Is there enough of that good stuff going on right now?

• 55:47Speaker 01

  No, no, honestly, like, I, I think. I still think that people underrate just how quickly the next few years are gonna go and, and also how much you should prepare, even if you think it's only a 20% likelihood. Like, even if you look at this, I'm like, okay, wait, like, there's no guarantee. Like, like every trend line I see, every like, like, every part of the process could be improved so much that we're basically guaranteed to get that.

• 56:11Jacob Effron?

  Do you think like 90% of anthropics thinks the same?

• 56:13Speaker 01

  Yeah. Um, and GDM and OpenAI. Like, everyone is very convinced that we do get drop-in remote worker AGI 2027, right?

• 56:23Speaker 01

  Now, that being said.

• 56:26Speaker 01

  Even if you don't have the level of confidence that the people working at the labs do. And you're still like, you know what, it's a 10 or 20% chance. You should still plan for that. Like if you're a government or a country, you should still be like, that should still be the number one issue at the top of your list of like, how is the future going to change? And I think that isn't felt enough.

• 56:43Jacob Effron?

  Well, this has been a fascinating conversation. I'd love to leave the last word to you. Like where can folks go to learn more about you, the work you're doing at Anthropic? Anywhere you'd like to point them, the mic is yours.

• 56:50Speaker 01

  Where should I point them? I mean, I think the thing which most people should read that they that maybe like hasn't been read is the interwork. I really think like that basic science of understanding what is going on in language models is really quite revealing. And as you sort of start to see them like compose and generalize and like build these like circuits and reason over concepts. I think that will make it feel pretty real. They're long, they're intense, but it's well worth a read. I think that's fun.

• 57:18Jacob Effron?

  Amazing. Well, thanks so much. This was awesome.

• 57:20Speaker 01

  Thank you very much. It was great.