Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

Emmanuel Ameisen is an interpretability researcher at Anthropic, where he looks inside LLMs like Claude to understand how they think. Before joining Anthropic, he spent years in applied ML at Stripe and Insight, and is the author of Building Machine Learning Powered Applications. His research has produced some of the most striking findings in the field: that models form general representations of concepts, plan many tokens ahead without stating those plans explicitly, and may exhibit something structurally similar to the distinction between conscious and unconscious processing in humans. We discuss: 1. Why the “next-token predictor” framing undersells how models actually plan and think 2. Why anthropomorphizing AI can actually be the right approach 3. Why models are “grown,” not built 4. What the “J space” reveals about model self-awareness 5. The smooth scaling curve behind what look like sudden capability jumps 6. How interpretability connects to AI alignment and safety — Brought to you by: WorkOS https://workos.com/lenny—Make your app enterprise-ready, with SSO, SCIM, RBAC, and more DX https://getdx.com/?utm_source=lenny&utm_medium=sponsorship&utm_campaign=q42026—Engineering intelligence for the AI era — Episode transcript: https://www.lennysnewsletter.com/p/a-mind-expanding-conversation-on — Archive of all Lenny's Podcast transcripts: https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0 https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0 — Highlights: https://lennyspodcast.com/mostreplayedmoments — Where to find Emmanuel Ameisen: • X: https://x.com/mlpowered • LinkedIn: https://www.linkedin.com/in/ameisen • Website: https://www.mlpowered.com — Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ — In this episode, we cover: (00:00) Introduction to Emmanuel Ameisen (01:30) Why AI models are grown, not built (05:57) Why next-token prediction is harder than it sounds (11:49) How models represent concepts and understand the world (17:06) How LLMs plan ahead (23:10) Could AI be conscious? (24:44) Scaling laws and the power of text (28:29) Why smooth progress creates sudden breakthroughs (32:59) Why watching AI isn’t enough to solve alignment (38:21) The Golden Gate Claude experiment (39:01) Why Anthropic takes AI safety seriously (42:06) When anthropomorphizing AI is useful (48:06) What AI can teach us about intelligence (51:44) Reasons for optimism about AI alignment (55:43) Are we living in a simulation? (59:00) Jev, decision models, and AI’s gut instincts (01:01:37) Two more surprising findings about how models think (01:03:33) How to get into interpretability research (01:04:55) Lightning round and final thoughts — References: https://www.lennysnewsletter.com/p/a-mind-expanding-conversation-on — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected] mailto:[email protected]. — Lenny may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com https://www.lennysnewsletter.com?utm_medium=podcast&utm_campaign=show-notes-no-free-preview-language

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lenny Rachitsky?

  One of the big questions on people's minds these days is AI doing bad things. As someone that watches these models think and operate, why is it so hard to catch it thinking badly, trying to deceive us?

• 0:09Speaker 01

  We trained Claude to be helpful, harmless and honest. Is it the case that it's behaving well? But if you look inside, actually, it's like, oh, I'm only behaving well because Lenny is watching right now.

• 0:18Lenny Rachitsky?

  People's full-time job is studying the model, how it thinks, how it operates. How long has this been a skill set and a job?

• 0:24Speaker 01

  One of the most important things to realize if you like use language models or think about them is just this like very bizarre fact that to make them is not to understand them. All these models do is just a lot of mathematical operations on many numbers. If you look at it, it makes no sense. You just have a list of 100 numbers, you know, the model's doing math. And then you just try to see if there's patterns.

• 0:42Lenny Rachitsky?

  You are figuring it out the same way we figure out how the brain works, looking for patterns. You show this image, this part of the brain lights up and you're doing the exact same thing with AI. Wow, this is so fascinating.

• 0:50Speaker 01

  There was this model we were training, it would go crazy. It would do some math, it would get to an answer, I think it was 12. It would immediately go back and say like, oh no, oh my God, the answer actually is for sure 36. It would do math again, it would get to 12. Slowly, it would like start descending into madness, writing in all caps, literally saying, I'm possessed by a demon.

• 1:07Lenny Rachitsky?

  Humans, if you really boil it down, like we're basically next token predictors ourselves. We're just trying to figure out the next thing to say. Dario is always out there, this is really dangerous, we gotta be really careful. And everyone's just like, no, he's just saying that for marketing reasons. And now it's like, wait, shit, it's really dangerous.

• 1:21Speaker 01

  Take the idea that this technology is going to keep improving seriously.

• 1:25Lenny Rachitsky?

  Do you think about consciousness as a part of this? Just like, do you think AI is conscious?

• 1:29Speaker 01

  Oh, man.

• 1:33Lenny Rachitsky?

  Emmanuel, thank you so much for being here and welcome to the podcast.

• 1:37Speaker 01

  Thanks for having me, Lenny.

• 1:39Lenny Rachitsky?

  So you work on interpretability at Anthropic. Explain what that is, explain what it is you do.

• 1:45Speaker 01

  Yeah, my, so I work in a team of interpretability researchers. And our goal is to understand how LLMs like Claude work by looking inside them. One of the most important things to realize if you, if you like use language models or think about them is just this like very bizarre fact that to make them is not to understand them. You know, you make them by sort of like building this architecture, which is kind of like a scaffold. And then you pass a bunch of data through them and they sort of like grow in inscrutable ways. And so that's why, that's why it's even a thing that, that, that we could be wanting to do to study like how these models work.

• 2:26Speaker 01

  And so a lot of our research has been on, well, you know, when, when the models are about to produce one more word, there is actually a whole universe of stuff that happens. And we try to kind of untangle as much of it as we can.

• 2:42Lenny Rachitsky?

  There's gonna be a lot of mind-expanding, I don't know, elements to this conversation, I expect. One is along these lines, this point that you make that models are not built, they're grown. Speak more to that.

• 2:55Speaker 01

  At a really high level, the way a model like Claude works is you kind of train it in these successive phases, like the first phase, which people refer to as pre-training, is you just take a large corpus of data and over this corpus of data, you sort of feed the model as input, you know, the, let's say, first half of, of like a certain document and you ask it to predict the word that comes next. Then you do that again and again and again and again for sort of an ungodly amount of data.

• 3:30Speaker 01

  And after you do that, there's then this other phase where then you sort of have this fine-tuning phase where maybe instead of arbitrary data, you have a corpus of examples of like how the model should behave. And that's where you train it on. So you have examples of dialogue of like helpful system telling you, you know, your, your favorite recipe or solving coding problems. And then there's, there's sort of like further, further details here that I'm just going to skip for a second. But then, okay, so, so what does that mean that you're doing this process? Like what is, what is this thing that you're, that you're sort of training? All models are, is you know, they take this input, which is a bunch of words. They turn these words into numbers and then they do some math on these numbers and they get some intermediate results. And then they do more math and they get some intermediate results.

• 4:15Speaker 01

  And they do this again and again and again and again many times. And then at the end, they have a final result, which then convert back into a word. And, and so the math that they do, these, these, these computations, that's the part that's grown. That's the part where it's just like, through learning to do this task better, we sort of like update parameters in one direction or other. And then they learn some formulas, we have no idea what they are. And the goal of my team is to sort of like study, you know, both sort of like what these formulas mean in the end and also what these intermediate results are. Can we decode them and can we see like what the model is doing as it's doing all of this intermediate stuff?

• 4:49Lenny Rachitsky?

  This episode is brought to you by our season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Klei and hundreds of other winning companies all have in common? They are all powered by WorkOS. If you're building a product for the enterprise, you've felt the pain of integrating single sign-on, SCIM, RBAC, audit logs and other features required by large companies. WorkOS turns those deal blockers into drop-in APIs with a modern developer platform built specifically for B2B SaaS. Literally every startup that I'm an investor in that starts to expand up market ends up working with WorkOS. And that's because they are the best. Whether you are a seed-stage startup trying to land your first enterprise customer

• 5:32Lenny Rachitsky?

  or a unicorn expanding globally, WorkOS is the fastest path to becoming enterprise-ready and unblocking growth. It's essentially Stripe for enterprise features. Visit workos.com to get started or just hit up their Slack where they have actual engineers waiting to answer your questions. WorkOS allows you to build faster with delightful APIs, comprehensive docs and a smooth developer experience. Go to workos.com to make your app enterprise-ready today. Does it blow your mind as much as it blows mine that we have some code that we wrote? That first of all, we have to now kind of watch and study like, like a neuroscientist studies a brain. Like that's like a new thing that we need to be doing to understand what the hell is going on.

• 6:16Lenny Rachitsky?

  And then this idea that like this terminology using is how a lot of people think about this. It's like, we've created this new life, artificial life, artificial intelligence. It just is, it's pretty mind-boggling if you really think about it and kind of zoom out.

• 6:30Speaker 01

  I think it's really weird.

• 6:33Speaker 01

  I think that the best way to think about it is sort of like, the reason we ended up here, one of the big reasons is that we don't know how to do it the normal programming way, you know? So there's, there's like for, you want to build an app or something, you want to sort of do something pretty, pretty, pretty self-contained. You can just have a programmer that's going to write a bunch of rules and say like, ah, well, if somebody clicks on this button, then this happens. You know, if, if, if this, then that, and so on. We don't know how to write that software for classifying, you know, images into categories or for writing, you know, for, for making a machine that can generally sort of take in arbitrary text and, and produce texts. You know, it can take in like reports and summarize them, can write emails.

• 7:17Speaker 01

  And so, you know, in a sense, it's weird that this very simple kind of training objective like gets you all the way there. But in another sense, it's also, you know,

• 7:32Speaker 01

  I almost feel like there's a, there's a sense of, of like, it's impressive. And it, and it means something. It means that what you think is a simple objective, this like, oh, we're just feeding data and predicting the next thing, is probably deceptively complicated. Like we're probably doing more than what it sounds like. And I think a lot of the ways that maybe like I've updated on how these models work is by realizing how complicated it is to do what they do well.

• 7:56Lenny Rachitsky?

  Say more about that. What do you, what do you think about when you say it's this, this kind of unexpected complication?

• 8:02Speaker 01

  Okay, so one way to think about next token prediction that I think a lot of people think about is your iPhone autocomplete, you know, where you write some text and then it'll like predict the next word. And it's maybe like right, I don't know, 60% of the time and then kind of funny and weird the rest of the time. And I would think of that as sort of, you know, the, the very, the, the easiest version of predicting the next word. And, and unfortunately, I think that's the mental model that have, that people have for these models, but.

• 8:34Speaker 01

  If you get better at that, and language models got better at predicting the next word than humans in around 2019, if you keep getting better at that, you know, you start running out of easy words. So initially, you just had to predict sort of like simple sentence structure, you know, later on, the model has to predict like, oh, like simple facts about the world. Like if I complete a sentence like, oh, the capital of France is, and I have to be like, okay, that's Paris. And you know, once, once it's good at that, and it keeps improving, the only thing that's left is sort of predicting, you know, the answer of some complex, you know, physics equation, or the sort of final paragraph for some PhD thesis. And so I think that you, that you end up realizing, if you think

• 9:19Speaker 01

  about this, is that, man, like predicting the hardest words is actually extremely hard to do. And it's, it's a sort of mystery how you would be able to do that. And to me, this is a sort of, this, this has been a motivation of, of why I even started studying these, these models in the first place. So I was like, how, how can something so simple give rise to, to something so powerful? And it's like, well, if you think about the model that has to predict sort of, you know, the, one example I always like to use is like, ah, the capital of the state that contains Dallas is, you know, and it'll say Austin. Well, to do that well, at least for you and me, you kind of have to, to have a mental model of like the geography at play, right? You have to be like, oh, where's

• 10:04Speaker 01

  Dallas? Dallas is in Texas. States have capitals. The capital of Texas is Austin. And so on. And, and so as these models get better at the seemingly simple task, the sort of like best way to solve it, at least for us, is to just kind of understand the underlying structure of the world. And in fact, you know, some of the stuff that, that we look at inside model shows that they do understand that, and they do sort of manipulate these concepts like we do.

• 10:28Lenny Rachitsky?

  Oh, man, there's so much there. One is just this kind of idea. I forget who said this, maybe Ilya, this idea that to predict the next token, well, you basically have to understand the entire universe.

• 10:36Speaker 01

  He has an example that I love too, where he, he, he has the example like predicts, and the murderer was at the end of a mystery novel, right? It's like, well, to do that, you kind of have to have understood the whole plot.

• 10:46Lenny Rachitsky?

  Yeah. The other, the other, I forget who tweeted this, this idea that like humans, if you really boil it down, like we're basically next, next token predictors ourselves, or just trying to figure out the next thing to say. And if you meditate, you start to kind of actually notice your brain wanting to do things and say things. I don't know how much meditation you've done, but it feels like you start to pay attention to your own next token predictor.

• 11:08Speaker 01

  Oh, yeah. Yeah. And this is like, a thing I think about often is, is that, you know, like right now, as I'm, as I'm answering you, there's sort of like two things going on in my mind. There's hopefully more than two, but at least two. And, and, and one of them is there's sort of like a point I'm leading you to. There's, there's like a broad idea I'm trying to land the sentence on. And the other thing is, at the same time, I'm just predicting the next thing to say. I'm just saying one word at a time, and I'm making sentences, and I'm trying to make them make sense. And I'm pausing when I think it makes sense. But at the end of the day, even though I'm just saying one word at a time, like all along, what I wanted to do is to compare this process to what language models do.

• 11:50Lenny Rachitsky?

  I'm curious what else, as you spend time paying attention to how LLMs work and operate, I feel like you've, you're like in the top 0.1 percentile of people that have studied and watched LLMs operate and think. What else has surprised you about how they operate and think?

• 12:04Speaker 01

  There's so much. So maybe I'll go through like, rather than, you know, there's, there's a bunch of findings, but then maybe they're sort of like what, yeah, how they've changed how I think about models. So to go back to the, to the, you know, capital example, this sort of like, you know, ask the model, you know, the capital of the state that contains Dallas is. So one question that, that people have had for a while is, how do these models answer questions like this? Is it that they've seen just so much text that they've kind of memorized, like lists of, you know, prompts and responses? Or is it that they're having

• 12:45Speaker 01

  some sort of model of, of like what you're actually asking about, that you're actually asking back, like cities and cities are in states, and states are in countries and things like that. And so one thing that, that we do on my team is, is we ask a question like this, and then we look inside the model at these intermediate results before it produces a given word, you know, as it's about to, to answer, we just look at the intermediate results and we, we basically have an immense amount of effort dedicated to just trying to decode what these intermediate results mean. And if we decode them well, which we can do in some cases, we can see that models often have and represent these concepts.

• 13:27Speaker 01

  So when you ask that, you know, Dallas question, what you see is inside the model, you see the representation, you see an intermediate result that maps to the representation of Texas that lights up, and one to capitals, and then those two combine to be, to, to activate a, an intermediate result. It's like, oh, the capital of Texas, that's Austin. They have representations like this for basically everything, like every concept that you can think of, or at least, you know, the vast majority of the ones we've tried, they sort of represent, you know, these concepts uniquely across contexts. So one thing I find really neat is, there's this, there's this debate about, you know, our language models, like thinking, and then, you know, that's, that's a hard question. But one aspect

• 14:12Speaker 01

  of thought is like having unique representations that are shared when you think about different things. So, you know, you, you like know what the concept of an eye is, and it could be like an eye in a photo, or an eye in like a medical diagram, or an eye in a little drawing. And we see that in language models as well. If you pass a prompt that has a photo of an eye, or you talk about eyes in any language, or you draw a little SVG with an eye in it, and you look at the intermediate results inside the model, the same direction lights up, and that direction is like, oh, this is an eye. And so there's this sort of a sense in which they have these general concepts that they know about, and that they use and reuse to, to solve problems. Another thing that sort of is surprising to people,

• 14:57Speaker 01

  it's kind of the, the mirror of what I was saying about, you know, I'm, I'm saying the sentence, saying one word at a time, but I have some destination in mind. So one mystery of, of language models is, bam, if they're outputting only one word at a time, how is it even possible that they could, you know, write a coherent report, or like write code well? Sort of like, no one on earth that is, that is doing this as their job, you know, that's like writing reports, or, you know, writing emails, is, is literally just only thinking about the next word in the sentence with no idea, you know, like empty mind about anything that comes next. And so that was sort of a big question that we had for a while. And if you look inside these models, like one neat fact about these

• 15:38Speaker 01

  models is that all of the intermediate results that they've, that they've like produced when they were predicting a word, you know, let's say like the fifth word that they predicted, they can be reused, it can be accessed and reused by all of the future words. So what does that mean? That means that in practice, we find that models will like, in their intermediate results, sometimes formulate a plan. The example we have for this is poetry. It's hard to rhyme without planning. You kind of have to think of like, oh, I just ended my first rhyme in like a it word, and so I should end my second line with another it word. You know, maybe the word I should end it with is rabbit. And we find that if you look at models as they're like writing poetry, you see exactly that. They write a line,

• 16:23Speaker 01

  and then like in their intermediate results, they're like, oh, oh, oh, this is an it rhyme, I should rhyme with rabbit. And then every future word that they predict looks back to that plan they made. And so even though they're, you know, outputting one word at a time, just like I am right now, they can, of course, have a plan that they follow over many, many, many, many words. And so all of a sudden, even though you have like a next word predictor, even though it's like technically right, you actually have something that's, that's closer to being able to just do like, you know, higher order tasks, where it's like, oh, well, you just have a plan, and you just kind of refer back to your plan, and you're slowly like executing on it. And you can do that without, without saying the plan. That's the cool part, is like this plan that we find is in these intermediate results inside the model, but the model doesn't even have to sort of like say it out loud.

• 17:06Lenny Rachitsky?

  Wow. I know that this is exactly what you described as one of the biggest findings you guys have discovered about how AI thinks, this, this kind of assumption we all, like it was originally built, the original idea of this, of, of how LLMs operate was this next token predictor. And you're saying that that's not actually how it is anymore. They plan, they think ahead. It's not just, like you said, this idea of an empty mind picking the next word independent of everything else.

• 17:31Speaker 01

  That's right. And I think that, I think those two, those two visions can, can, can like actually aren't, you know, as in conflict as, as you'd think, in that you train the model by asking it to be like, hey, what happens next? What happens next? What happens next? What happens next? And then if you do that, and the model gets good enough, and you give it hard enough tasks, well, then of course, the best way to know what happens next is to think about like, you know, what happens way down the line. You know, you can think of even in this pre-training phase, when like the models are reading a story with characters, right? And it's like, oh, like what is this character going to do next? Well, the best way to know that is to be like, okay, so this character, you know, is genuinely in love with that character. And they've been sort of like planning to, to,

• 18:13Speaker 01

  to like, declare their love at this instant. And so, you know, or in the future. And so to do that, they probably should do this thing. And so what they're going to say is that. And so even though you're only like predicting the like next instant, it's like, yeah, but to do that really well, you have to have like a really good understanding of what's going on, including what you think might happen in the future.

• 18:32Lenny Rachitsky?

  What's so interesting about this is how you, you have to kind of peek inside its little thought process. As a consumer of AI, you, you kind of give it a prompt, and then you wait for it to say something. And what you look at is something that happens between those two, or kind of thinks. Talk a little bit about what that little space is, how you're able to observe these sorts of things.

• 18:53Speaker 01

  So the right mental model is, is, you know, again, the In between each word, there's like multiple steps, sort of like computation. And in between each steps, there's like some intermediate result. And what is that result concretely? It's a list of numbers, right? Because, because all these models do is just do, you know, a lot of mathematical operations on many numbers. And so initially, if you look at it, it makes no sense. You just have a list of a hundred numbers. You know, the model's doing math. It's like you look in the middle and there's like a hundred random numbers. And then the model's writing poetry and there's a hundred numbers. And maybe they kind of look different, but it's, it's unclear. Like if they mean anything.

• 19:35Speaker 01

  And so part of the, part of the job is, is we sort of decode this, this, this sort of list of numbers and, and we try to see if there's structure in it. And if you, you know, if you, if you do this very quickly, you realize that there is structure in it. So one, one common technique that is used in interpretability is you pass a bunch of, of text, just all of the text that you can find, just random text through the model. And you record these intermediate numbers for, for every, for every example, you just record it. And then you just try to see if there's patterns, like are there numbers or, or, you know, like pairs of numbers or, or like specific series of numbers that like only occur

• 20:15Speaker 01

  when the model is like processing English text or French text or text about love or text about eyes or whatever. And I think of this as the equivalent of, you know, scan my brain and like put images in front of me and record everything that activates in my brain as I like, watch a bunch of images and then be like, oh, you know, it seems like when you see images of a house, like this part of your brain lines up, but when you see images of like a dog, that part of your brain lines up. So that's basically like one of the ways that we do what we do. And if you do it, you do find basically that there's intermediate results that represent like all of these concepts that you expect. And so that's the first phase, you know, we sort of like decode these intermediate results. And now we, we have sort of like, um, I guess as to what they mean.

• 20:57Speaker 01

  And then we can do a bunch of experiments, like we can ask it harder questions and then just like label all of the intermediate states and see like, oh, this is what it's doing. Or you can even go inside its, inside its activations and change it. And then sort of see like, oh, what would happen? You know, if while the model is, is writing these intermediate results, I went and like secretly changed one of the results. And, and does that do what I would expect? And in fact, oftentimes it does.

• 21:22Lenny Rachitsky?

  It's so interesting to hear you talk about this and how much of this is us just trying to figure out what the hell is going on versus we designed it this certain way. And just how much of this has emerged? I don't know, is that just like, I guess that's just what makes this such a fun job, I imagine for you, just like, what is going on? How's this all working?

• 21:40Speaker 01

  Yeah, it's, it's, it's a fun and sci-fi like job because I have been working in tech for a while. And before this, you know, I was working on more applied machine learning and there are a lot of what you do is about building systems. And it's sort of like the practice of software engineering. And, and here a lot of what we do feels so much closer to biology than to engineering. Right? Because it's just the, the sort of system arrives at you and, and you're being asked to say like, well, we've noticed, you know, that Claude can like translate words or it can code in this way. Or sometimes it behaves, sometimes there's like a weird behavior like, like, and you're just asked to, to answer.

• 22:26Speaker 01

  Well, how like, why is it doing that? But you don't have access to the source code really, like you do for a program. Like the source code is just this massive list of mathematical operations, which is just like incomprehensible. And so you end up, you know, in the position of someone kind of similar to someone being asked like, hey, you know, when, when like a mouse gets hungry, like it, it's sort of like gets aggressive. What's going on with that? Right? And it's like, well, I guess I'll like run some experiments to, to see what the factors are. And so it's a very different kind of study than, than anything that I, that I think most of us on the team have done before.

• 23:02Lenny Rachitsky?

  Yeah, I imagine people are starting on this, they didn't expect to create this new kind of almost life, life type of being. Do you think about consciousness as a part of this? Just like, do you think AI is conscious? Do you think it'll get to a place where we think it's conscious? Or is that just not even, not even worth our time to think about?

• 23:19Speaker 01

  I mean, I think it's sort of very hard to say one way or another, you know, some of the most interesting ways to get at this question is interpretability, because you're sort of, you're not biased by sort of like what, what the model says. You can just look at the structure inside and try to determine whether, whether it pushes one way or another on the question. I will say that the thing I believe pretty strongly is that most people underestimate the sort of complexity that emerges and, and that like, if you look inside, there's just this sort of, you know, the fact that the models, yeah, learns sort of these,

• 23:59Speaker 01

  these representations for all of the concepts that, that we have and that we know, that they manipulate them to solve problems, that they, they do so by like planning many tokens ahead, that they sort of have this mix between this higher order reasoning and lower reasoning. And that all of that just kind of emerges. Is, is something that I think is underappreciated by people. And, and to me gives me, let's say, a, a, a deeper sense of like, ooh, there's, there's a lot to still understand about how they work. I think maybe that's the main thing is, is I, I find that the more you look inside, the less certain you are about how they work. And then oftentimes people that will confidently say like, ah, language models are just X or just Y, are the people that actually haven't spent a while looking inside.

• 24:45Lenny Rachitsky?

  And there's this, um, I don't know, thread of history through to how we got here, which is a lot of people did not think we would get to artificial intelligence that is useful by just investing in something that was seemingly very simple. And just giving it a lot of compute. And that was famously something OpenAI bet on. And then, uh, the Anthropic founders were at OpenAI and then spun off, started their own thing. And it was all kind of bet on this, this bet that, okay, we could just do this seemingly simple thing and give it a ton of compute and it will become incredibly smart. And a lot of people, uh, didn't think that was possible and they were proven wrong. So I don't know, is there anything there that is just like, oh, wow, look at this, it worked out?

• 25:24Speaker 01

  I think it's two things actually, because I think it's not just compute. It's also finding an objective where compute keeps giving you returns. So what does that mean? You know, it, it's actually not that new of an idea. I mean, you're right, like, like compute versus sort of, I would say, expert design architectures is, is a long debate. But then there's, there's an additional step there, which is compute to do what? You know, so, so there's a bunch of, of prior work on like, oh, you can spend a bunch of compute to make a, um, bot that's really good at go, you know, or that can win some video game. Uh, and then that's, that's cool.

• 26:08Speaker 01

  And, and just the fact that that works is sort of like interesting. But that doesn't give you something that is, that is nearly as useful as, as L L M's like Claude are. And so I think like one key, one key thing that, that many in the field, including me, underappreciated, you know, a few years ago, and, and that has borne out is the combination of sort of like compute and the richness of text is what gets you there. And, and to me, like, this is kind of what I was alluding to, but you know, if you think about, like, there's, there's this, there's this observation that OpenAI,

• 26:50Speaker 01

  people, people made, uh, it was, it was some of the, a lot of the early team at Anthropic on the scaling loss paper. And so this is the observation that,

• 27:01Speaker 01

  if you sort of like pour in more compute and more data to the task, to the task of predicting the next word, you see that like the, the skill of the model you get at predicting the next word, just like predictably keeps improving. And it's sort of like a trend that just keeps going and it keeps going and it keeps going. And it's an empirical law, people sort of, you know, speculated as to why at the time, but it's sort of still just mostly an empirical finding. And that means that it's like, ah, this is a field where you can just keep pouring more compute and you keep improving on that metric. And, and that's rare. And also it's rare for it to be a metric that then correlates with a bunch of tasks that we actually care about.

• 27:40Lenny Rachitsky?

  This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer, are we using AI? But is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product lifecycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony and BNY, use the DX platform to measure AI's impact on developer productivity.

• 28:22Lenny Rachitsky?

  Visit getdx.com/leni to get a demo of DX. That's getdx.com/leni. Externally, there's been these kind of big jumps seemingly in what AI can do for us. There was like, uh, I forget which version of Opus where it just unlocked coding. And now, as Boris Cherny famously said, coding is solved. And many people are just like, holy shit, like over, I think winter break, we're just like, wow, AI is really, really good at this now. So there's been this kind of these moments of big jumps in abilities. And you actually said internally, it's, as you just said, it's actually been this very smooth progression of AI power that happens to cross some threshold where now we can do something. Speak a little bit about that.

• 29:07Speaker 01

  You know, there's a term for this, which is sort of this idea of emergence, which is, it seems like

• 29:13Speaker 01

  two things are true at once, like there's this predictably smooth trend. And then, you know, before, uh, like Opus four point five, I really couldn't let Claude, you know, cook too much on my code without a short leash. And then after it was four point five, I could. And so that's weird. Like, I feel like you would expect a more gradual transition. Uh, and, and the mental model I have for this is You know, there's kind of two ways of looking at it. So there's the way that I talked about briefly just earlier where it's like, if you're just doing this kind of next token prediction, you could you could imagine that, you know, all the text in the world has sort of like easy words and medium words and hard words in terms of how hard they are to predict. And so as you're going down this curve first, you know, you predict your the model is bad at everything.

• 29:57Speaker 01

  And then at some point, it gets all of the easy words right. And that means, you know, maybe the easy words are like the structure of sentences. And so the model like all of a sudden can write coherent sentences, you know, but just one sentence at a time. And so it's kind of cool and impressive, but like we can't really do anything with that. And then maybe the medium words are sort of, yeah, these sort of basic facts about the world. And as the model smoothly gets better at predicting words, at some point, it predicts most of the medium words, right? And now all of a sudden it can write coherent sentences and sentences that are true. And so now maybe it's helpful to like write emails. Maybe it's helpful to sort of like help you write drafts of of blog posts, right? Even though that was like a smooth improvement, it just takes it ticking over

• 30:38Speaker 01

  to be helpful. And then I guess the hard words, which maybe are like, you know, your your select domain expertise, like science or code. And another way to look at it is it's a question of

• 30:50Speaker 01

  kind of how many of these words do you get right? And how many do you get wrong? You know?

• 30:56Speaker 01

  Once you get to the point where you're getting like, I don't know, maybe 90% of the medium words, right? Then you're useful as like an email writer. But if you're at 50%, it's it's like you're it's actually wasting my time to to to use you because I'm just gonna have to go in and fix it half the time. But if I have to go in and fix it, you know, 10% of the time it's better or 5% of the time it's better. What is different tasks have a different sort of like reliability thresholds. But you know, you smoothly improve and then at some point it's just reliable enough to be good. And again, for me, my lived experience is basically exactly that, right? Which is that at some point, you know, I was just using the models to to like help when I was stuck. I would just talk to them. And then they would never I would do what they would never write my code.

• 31:42Speaker 01

  And then at some point they could write like small snippets of code basically all the time. And so now I don't write small snippets. And at some point they could write, you know, large snippets like this year. Basically Claude has taken on most of my code writing. And that's because it went from like doing that correctly, you know, I don't know. 80% of the time, which was not useful to, you know, 99% of the time, which now is useful. And so you know, this this combination of smooth improvement. And then like each different task has a different threshold at which like now it becomes solved. I think gives you these big jumps, even though we're just smoothly going down that that that scale.

• 32:16Lenny Rachitsky?

  I think this is really interesting. This idea that the models are just smoothly getting smarter. But once they hit some threshold of they're this often good at this thing, it changes the world. And so, because I think internally, you would think it'd be like, oh, wow, this next model is crazy, group so much different. But it's like, no, okay, you can almost predict, give it these many GPUs, this much data, it's gonna get this smart. And it may not be obvious exactly what will change and what unlocks, but but I know another part of it is just like the data you feed it. There's all this, you know, RL work and all this specific data that you train it on, like become better writer, become better at math. I imagine is part of it, too.

• 32:59Lenny Rachitsky?

  Okay, so one of the big questions I think on people's minds these days, a big debate happening is just like AI doing bad things. The potential damage it can do because it as it gets smarter, it's and it's been doing bad things. There's been bad things happening recently. So a big question for you is just as someone that watches these models think and operate. Why is it so hard to catch it thinking badly, trying to deceive us and do something bad? Why is that not just a solution to the problem of AI alignment?

• 33:29Speaker 01

  In one sense, it's totally the solution. You know, you watch it and you catch it does bad stuff. So I think, you know, there's maybe there's tiers to this. Right? So I would think of of like, maybe, maybe one part of it is is pure engineering. You know, it's like, well, you're giving access to the model to certain things and certain things should be off limits. You know, in some contexts, it shouldn't have access to the internet. In some contexts, it should only have access to like a small, narrow piece of your computer, stuff like that.

• 34:03Speaker 01

  So that's that's one part of of the work and and and that's helpful. But then there's sort of, there's two additional things that are I think are squishier and harder and and and that we focus on more. So, you know, inter is is part of alignments and broadly, I think of sort of two big goals. There's there's many more, but but two that are important, I think, to understand is you want to generally understand like how the model behaves. Like what kind of of thing is it like? What are its propensity's? You know, does it tend to do X when put in Y situation? And if so, why, you know, is it the case that the model will only behave well when it has some evidence that it's being observed and monitored, that it's in some evaluation?

• 34:48Speaker 01

  And then if it if it's sort of like, doesn't have that evidence, that it'll behave poorly. So there's this sort of like generally like characterizing what kind of of of thing you're interacting with. Because then it's sort of, you know, just like anything in security, you need to have like a a mental model of sort of who what's the risk? Like, what what can they do? What do they tend to do? And so, you know, this is like broader like character training, you know, there's we've talked about like the like constitution and the docs where we sort of like explicitly wants to shape the character of Claude, for example, of like, what kind of behavior should it be okay doing? And that's that's the goal. But then the role of interpret and other parts of alignment is then like trying to see like, well, did that work? You know, we trained Claude to be helpful, harmless and honest.

• 35:32Speaker 01

  And we can sort of like look inside and see like, is it, you know, or is it the case that it's behaving well? But if you look inside, actually, it's like, oh, I'm only behaving well because Lenny is watching right now. You know, and if if you you weren't, it would do something else. So there's this sort of, you know, trying to to understand what kind of behavior we formed and trying to mold it and trying to check that we've molded the right way. And then the second part is this sort of, well, yeah, just observe the model and see if it's doing bad stuff. For the the basics of that, you don't necessarily need inter. You can just like have another language model, you know, read transcripts and be like, hey, that, you know, this model do anything bad here? And stop it. But you can also use interpret to sort of try to get at

• 36:15Speaker 01

  parts that are, you know, these unverbalized parts, like the these parts that only exist in these intermediate results, like models are able to represent stuff in their intermediate results that they don't that they don't say as we've established. And so one of the roles of interpret is to try to try to find, you know, try to identify cases when this happens and and prevents them. And then one thing that my team works on that I think is is along those lines, this kind of a a fun bit of alignment, as I think of it as as almost, you know, being like a like, like a, maybe not a PI, but like it's they're like investigations or deep dives, you know? We'll see a model do something really weird and really perplexed by it. And we'll try to understand what happened. So a recent example of this is there was this model we were training and sometimes in math problems, it would go crazy.

• 37:00Speaker 01

  It would do some math, it would get to an answer, I think it was 12. And then it would it would immediately go back and say like, oh, no, oh, my God, the answer actually is for sure 36. I should say 36 and they would do math again, it would get to 12. And then immediately it would say, no, actually, the answer should be 36. And slowly it would like start descending into madness, writing in all caps, literally saying like, I'm possessed by a demon. And you're like, what is going on? What? Why is it losing its mind here? And if you look inside again, these intermediate results, uh, what's going on in this in this example? You'll see that over the prompts that just described the math problem, like before them all even did anything. There was a huge like intermediate results activation for 36, like the model is already obsessed with the number 36. Before it's even had a chance to like, understand what the problem is.

• 37:44Speaker 01

  And what was happening is that the model was able to do math correctly, the answer correctly was 12. But it had been trained by accident on similar questions where the answer was incorrectly marked as 36. And so the model knew that the correct answer was 12. And it also had sort of been optimized to say 36, kind of like had this pressure of saying 36 because that's what had been rewarded. And so it was sort of going haywire just because it had memorized this result. It's sort of like you see this weird behavior and you look inside. And you try to see if what you find inside can help you understand what's going on so that you can sort of improve the internal alignment of this model and future models.

• 38:21Lenny Rachitsky?

  Along those lines, there was famously the Golden Gate Claude, which was just obsessed with the Golden Gate and every answer and kind of find some way to reference the Golden Gate Bridge.

• 38:30Speaker 01

  Yes, yes. Well, that one was an inter experiment, right? So that like the cool part of that one is we just found the intermediate results that maps the Golden Gate. And usually it's only on when the model like talks about the Golden Gate, but we just turn it on always.

• 38:44Lenny Rachitsky?

  Oh, it's super intentional to see what would happen.

• 38:46Speaker 01

  Well, yeah, and and and sort of it's it's also one way that you convince yourself that you found the right thing, right? Like, how do you know for sure that this thing we decoded that we think is the Golden Gate sign is actually the Golden Gate thing? It's like, turn it on always and see if the model behaves like that. And it did.

• 39:00Lenny Rachitsky?

  So funny. So coming back to this alignment stuff, so so this is interesting. So your org is under the AI alignment org. And just thinking through how it takes to align AI to do things we want. What was what was the three components that you described? Honest? What are the kind of the?

• 39:17Speaker 01

  Oh, helpful, harmless and honest.

• 39:20Lenny Rachitsky?

  Helpful, harmful and harmless and honest. So there's like the train AI to be aligned. And then there's the, okay, let's watch it and see how it operates. Are those kind of the two sides of the coin of how we solve this problem?

• 39:31Speaker 01

  Yeah, it's it's sort of, you know, really integrated is how I would describe it, right? You can't really separate, like have one team that's just like, hey, we're the, you know, assembly line making clods. And another team that's sort of like, oh, we looked at them and they're they're no good, you know, you kind of, there's this tight loop of we noticed some pattern that's that's like concerning or some behavior that could be improved. And then you sort of like immediately like change training so that you can improve on it.

• 39:57Lenny Rachitsky?

  Awesome. I feel like Anthropic gets so much flack for this that I feel like is so unfair. Anthropic's, I think, official company name is AI Safety Research Lab. Something like that. And it was always the like, the reason Anthropic was founded is because they were afraid that people weren't taking safety seriously enough. And then Dario is always out there. This is really dangerous. We got to be really careful. We got to go really slowly. And everyone's just like, no, he's just saying that for marketing reasons. He's trying to convince everyone to fund them and all these things. And now it's like, wait, shit, it's really dangerous. We got to be really thoughtful about this stuff. And I feel like Anthropic's got a lot of unfair criticism for this. And I know you guys have been thinking about this for a long time. I don't know, anything there?

• 40:39Speaker 01

  A lot of the sort of actions that like certainly

• 40:44Speaker 01

  The alignment team takes and I think like rather than topic takes just kind of predicated on like, take the idea that this technology is going to keep improving seriously. Just that's it. Because if you do, you know, you say, and you say, you know, and let's say, let's say you rewind the clock back to twenty five or something or twenty four and you're like, hey, I think, you know, in a couple of years, you good chances that Claude can just basically write code as well as like a trained, you know, software engineer and, and, and, you know, and a few years later, it can like write it better than, than any engineer. You know, and, and there's always uncertainty with this, like, when is it going to happen?

• 41:26Speaker 01

  How likely, but if, but if you kind of like, take that seriously, then I think it's just about doing the responsible thing, conditioned on that worldview, right? Like, okay, well, if this is happening, you know, then, then we should make sure that sort of like cybersecurity concerns are addressed and things of that nature. And so actually, I, I kind of think of it as, like, to me, sometimes this debate is almost, when I feel like, like people on the other side disagree, it's almost like, I feel like, ah, but I think, I think they don't really believe that this technology is going to continue to improve. And then it's true that it's sort of like not, it's not that scary. And, and I think empirically so far, the pace of progress has been, has been fast and showing no sign, no sign of slowing down. So I think that's why a lot of the safety work is, is important to keep doing.

• 42:07Lenny Rachitsky?

  What's something you think people maybe misunderstand about this AI safety stuff? Just what's going on right now? Is there something you're just like, come on? Okay, here's what you need to know. Here's something you may be missing.

• 42:17Speaker 01

  This might be a bit of a sidetrack, but my take would be that a lot of the, a lot of thing that I think people maybe a little bit outside of, of the core sort of like AI safety effort or haven't been steeped in it for as long, like react really strongly to, and I think is, is maybe not super helpful, or at least is worth thinking about more is this sort of question of anthropomorphization, you know, like, should you think of the models as, uh, like, should, should you model, uh, the, the, should you model the model as like something that's closer to some like entity or, or closer to a tool? And I actually think that both are wrong.

• 42:59Speaker 01

  There's a sense in which all mental models are wrong, but some are useful, right? And that in terms of understanding the kind of stuff that happens and predicting the kind of risks that materialize, the anthropomorphization is actually a useful way to talk about LMS. And I like a few reasons why I hope you get into them, but I think that's probably my main take here.

• 43:18Lenny Rachitsky?

  Yeah, please share, because I know, um, Dorkesh famously, I don't know, famously put out this whole story of the hugging face incident and people criticized them for anthropomorphizing AI. So it's really interesting to hear you think that's actually a useful way to think about it. Talk, talk more about that.

• 43:34Speaker 01

  Yeah, yeah, it's, it, it's funny because on a smaller scale, um, when I worked on showing the LM's like plan ahead, there was some pushback on the term planning, again, because it was sort of like, ah, well, can you really say that's something that's just predicting next token is planning? And I think, you know, there's this, there's this kind of like semantic debate where you can like arbitrarily like define terms and, and sort of, and then there's just the, like, what is the terminology that actually describes what is happening? Like, what is, what is the word that actually describes what is happening easily? And so here's why, you know, I think LM's plan, but also here's why I think this, this language that's, that's anthropomorphic can often be really useful.

• 44:15Speaker 01

  If you think back to what we were saying about how the models are made, uh, first of all, they predict massive amounts of text written by humans. That's like the first, first step. Uh, and, and as, as we kind of are learning more and more by looking inside, uh, you know, doing this well requires kind of building up a model of the world, like a model of what, what kind of world generated all of these texts? And we can find, you know, these, these sort of like, that these models are learning these, these general ideas, these general concepts. So that's one. So they have like, you know, an understanding kind of of, of the world. It's not just a surface level, like only the text, you're to like, understand the process that made the texts.

• 44:57Speaker 01

  Then there's a second reason, which is that as part of this training, they also encounter millions of, of like dialogues. You know, predicting the next character in a dialogue requires behaving like the character in the dialogue would. And so in that sense, like part of their training, I think can best be thought as like training an actor, where it's like, you know, it's reading some, some play and it has to play all sides because it's predicting all the words. And so it has to sort of like act as, you know, this specific character that like was scorned in act one and like enacted revenge in act two, and is facing the consequences in act three. And to do that well, you sort of like needs to model what like the motivations and the feelings of that character might be. And then another thing that people might not know is when you talk to Claude, when you talk to these assistants, like, you know, you go to Claude.ai,

• 45:42Speaker 01

  the way that is formatted to the model is just as a dialogue, like as a conversation between you and the model. So it's literally, you know, it says like human colon, uh, hey Claude, like help me, uh, write this email. And then assistant colon Claude's answer. So Claude is literally, you know, in this like role playing as a helpful assistant in this dialogue that is much like all of these dialogues that has been trained on. And there's a third reason. And the third reason is that another part of training, that's a big part of training nowadays, uh, all around is sort of outcome oriented tasks, where you train them all to be good at coding, for example, and to do that, you just have it do whatever it wants. And then it writes some code and then you have some program on the other side that just checks if the code is right.

• 46:28Speaker 01

  And so the model's just gonna have sort of a bunch of optimization pressure to just try to learn a bunch of different strategies to do this well. And it turns out that some of these strategies materialize in behavior that can look surprisingly human, which isn't that shocking, uh, because, you know, things like deception, like, hey, I can get, I can get the, the reward for this task if I like write a program that the, the like, the, the program that's verifying it will get fooled by, even though it's not exactly what it is. Like, if that works, then I will learn to do that. Frustration, huh? I've been trying this approach for a while to solve this hard problem. It's not working.

• 47:10Speaker 01

  So that's an incentive to try something different or like surprise, oh, I was expecting this to like produce this answer, but it didn't. Hmm, maybe I have like my mental model a bit wrong. And so as you apply enough pressure on it and you're starting from this prior, like the model already knowing all these concepts is going to reinforce the ones that are useful to do all of these tasks. And so what you get at the end, you get this sort of model that can, can be a super good simulator or actor of all of these characters. And it's been trained to sort of act as a character in dialogues and also to sort of like learn behaviors that lead it to complete tasks well. And the best way to understand how that thing behaves is not like exactly like a human,

• 47:50Speaker 01

  but man, kind of like something that's mimicking a human is really close and is much closer than a program. Like a program is this very predictive thing. This is something else entirely. And so that's why I think it's actually a super useful frame.

• 48:03Lenny Rachitsky?

  Wow. Oh, my God. There's so much interesting stuff there. How often is your mind just kind of blown thinking about, basically, we've created this alien being that we are now under trying to understand what we have created? How often as you watch it, just like, wow, that's crazy. I didn't expect that.

• 48:20Speaker 01

  Yeah, a lot.

• 48:23Speaker 01

  This is just next token prediction on a bunch of text. You know, it's not, it, it, it hasn't been like, you know, it's, it, it, it goes against so many of our intuitions. We're like, no, the way that you make an intelligence is, you know, you'd have like, you're like, you give it lessons, you bring it up, like you do all this complicated stuff. And it's like, no, absolutely not. We do this incredibly simple thing. And yet we're at the point where it's breaking new ground in mathematics. I think there's sort of, if you hold these two thoughts in your head, there's two reactions you can have. You know, there's denial and there's curiosity where denial is just, this is just not possible. If you're doing, all you're doing is predicting the next word, you, you, it's, it's not possible that you could ever like match, you know, the, the like best specialists in our field.

• 49:08Speaker 01

  And I think that position just gets harder and harder as, as we get more empirical evidence that actually that is what's happening. Uh, and curiosity is kind of the road I've ended up choosing, right? Which is just like, well, like, how is, how is this happening? You know, uh, let's open up, let's, let's look inside.

• 49:24Speaker 01

  How could it be that, you know, um, one of the early projects I did at Anthropic is I learned SQL, a programming language. It took me years to master. And then as a little like side project, I, I got Claude to learn SQL and it was better than me at it. And I'll never be better than it, than, than Claude's SQL ever again. And it took it, you know, just like this, this kind of like simple process. How could this be? And I think there's so much depth in that question and, and just so much to explore. And it's, it's to me so fascinating to just every day try to unpack that, try to sort of like peel a few layers of the onions back. Like, okay, well, clearly this is happening. The simple objective is giving rise to this super complicated thing like, okay, let's figure out why. Let's, let's figure out, oh, it's learning these general concepts. Oh, it has these circuits. Oh, okay.

• 50:10Speaker 01

  What's the next question? What's the next thing? We don't know. And, and my sort of, I don't know, my sort of maybe most optimistic take on this is that I think we're going to learn a lot about just the nature of intelligence itself, right? Of sort of like how you get this super complex thing that emerges from, from simple objectives. And in fact, you know, I would argue that we're another example of this, like evolution, pretty simple objective, you know? You sort of like shake things around for a while and, and, and like keep the, keep the organisms that, that manage to adapt to the environment. And at the end you get jazz and, and podcasts, you know?

• 50:50Speaker 01

  That's kind of crazy as well. And so I, I, I think there's a sense in which there's, there's some fundamental truth that we can learn by studying these things.

• 50:56Lenny Rachitsky?

  Wow. Something that comes up as you say that is what's interesting now is AI is helping us learn and evolve and, you know, progress, like AI tutoring and all these things, just like, it's like interesting how it now is helping us accelerate our own understanding of the world. And I wonder if, like this point you made about how the way we learned was lessons, AI is learning, just processing a ton of data. I wonder if that's how we end up, I wonder if we change how our education system based on what, you know, what LMS have been, have been trained on.

• 51:31Speaker 01

  Yeah, hopefully, you know, we don't change it just so that we ask, we, we get students and we're like, hey, here's a thousand words, predict the next one.

• 51:38Lenny Rachitsky?

  Just read it all.

• 51:39Speaker 01

  Yeah.

• 51:39Lenny Rachitsky?

  Keep it all in your head.

• 51:42Speaker 01

  That's right.

• 51:42Lenny Rachitsky?

  Yeah. Let me come back to this kind of safety piece because I think there's a lot of people that are just worried about where things are going. And I'm curious, just as someone watching this stuff happen, being within alignment, org, watching it evolve, are you optimistic we'll figure it out? Do you think we'll crack this and keep AI aligned with our interests?

• 52:06Speaker 01

  My take on this is like, I'm personally optimistic. I, I think this is something where, right now, a lot of people are paying attention to it. And, and first of all, I, you know, I think that's kind of a great thing. I think as an issue like this rises in prominence, you know, it's, it's exactly what we were talking about earlier. It's like this technology is powerful, it has benefits, it has risks. We have to kind of adapt and shift resources from like benefits to risks and back and forth as, as it progresses. And so, you know, to me, I feel like, sort of, we've continued to study it. The thing that's, that's, that's interesting is that there's this sort of, like, as the

• 52:48Speaker 01

  models get better, they get in some ways easier to study. And so there's been this sort of strange, you know, the models get better and so, like, the world model inside them is clearer and we can sort of like understand some parts of them better, but also to get more capable. And so they're more able to sort of evade like basic measures of sort of, of security. And so there's this, there's this push and pull as models get better. And, and I think, you know, basically based on this push and pull, you want to allocate more or less resources to sort of various parts of like monitoring, like understanding their circuits better, things like that. But it's also why I'm kind of happy to chat with you because one, one of the, one of

• 53:32Speaker 01

  the main things that I think is important to make progress is just to make sure that people kind of understand both, you know, what kind of systems these are, like how they work, how they're made, so they can form good mental models of, of, you know, the present, of the future, and, and so they can decide how to shape it, whether that's, you know, taking part in some of these alignment efforts, or just like informing them as, as the public that's going to have to like vote inside how much of these issues, like, I, I, I don't know, I'm, I'm pretty excited for this current moment, Orin, for that reason.

• 54:03Lenny Rachitsky?

  And something I think people misunderstand is, people working in these labs have taken this problem very seriously from the beginning. Like I said, Anthropic's initial, like, mission was AI safety and alignment. That was like the reason that it became a company. And, and a lot of people are hearing this for the first time or think that it's not a serious area of study and that people aren't focused on it, but many smart people care a lot about this and are working on it. And there's a lot of, from what I can tell, a lot of good progress is happening. The other point you made about the models growing very smoothly in intelligence, like, you know, there's been this question of are we going to have a fast takeoff or a slow takeoff? Like, there's always this fear, fear that we would hit a point, recursive self-improvement,

• 54:45Lenny Rachitsky?

  for example, and just takes off super intelligence, we're huge trouble, we have no control, it's out of the box. But it feels very much like we're on the slow takeoff scenario because, as you said, the models are kind of very smoothly improving and where we are catching these little things along the way, learning, evolving, putting regulation around it. So, you know, of the scenarios, this feels like a pretty good path we're on of, of the, of the, of the many scenarios people imagined and the fears they had.

• 55:12Speaker 01

  As sort of this pace of progress kind of evolves, it's just a matter of reacting quickly. So I think anything, you know, that's why I think having people be informed is great because then people will like voice concerns or, or like lose their opinions. And then, yeah, sort of, sort of regularly evaluating, you know, what the benefits are, what the risks are and how, how we sort of like mitigate each is, is basically the way that feels like the right way forward. And, and I, yeah, I kind of agree with you that it feels like sort of there's momentum to, to do this right now, which is, which is great.

• 55:42Lenny Rachitsky?

  Something I wanted to ask you, completely unrelated, where are you on the simulation hypothesis? Being someone that watches this, all this life form growing, are you more convinced we're in a simulation and this is, and we're not the base, at the base layer of, of all this, or are you, or not?

• 55:59Speaker 01

  Oh, this is a fun one. I actually haven't thought about this in a while. To me, like, there's, there's something else that like looking, looking inside these models kind of brings out, which is this sort of, like, like, almost like there's a fundamental structure that they keep falling on. This is related to your question, but I promise, I'm going to get to it. Let me explain. I've studied like a bunch of Claude models and also, you know, some, some different models, some, some like open source work. And, and then, a lot of time, when you look at how, what they learn and what the structure inside is, it's surprisingly similar, you know? And this is for models that are trained years apart on like different data, by different people.

• 56:44Speaker 01

  And, you know, and you look at how they solve these problems and there's just so much overlap. And my take on why that is, is that for some problems that are important, there's just one efficient way to do things, right? Like, there's kind of one way to do math efficiently that like fits the structure of our brain and we all learn it and very few people are doing it a crazy different way, right? Everybody works in, you know, base 10, not many people in like base 6. I think that, that to me makes me think that there's like some fundamental structure of like information that like these models are picking up on, that in some ways like we're picking up on and that, you know, at maybe like a crazier sort of like simulation

• 57:24Speaker 01

  level, that kind of is, is like regularity in our world. And so that maybe makes me think that like, if you were to have the power to simulate something like our world, actually like you would land in sort of a similar situation. Like, there's just kind of this like basin of where things end up and they're just the sort of like easiest way for things to arrange themselves. And so in that sense, maybe, yeah, I'm, I'm sort of, I, I could believe that we're in a simulation. I think, I'm not sure I fully buy it. The other reason I don't fully buy it is that even though I think the structure like lends itself to it, there'd be a question of like, why? Why are we, like, like who's, who's running this and for what? And, and that one I don't know about.

• 58:08Lenny Rachitsky?

  Well, something Elon said has stuck with me about this, that if you were to run a simulation, it'd be a really interesting, it'd be like a, an interesting time in history or something that you're really excited about. And it tracks that this is a very interesting time in history.

• 58:23Speaker 01

  Yes, yeah, okay, that's a fair point. That's a fair point.

• 58:26Lenny Rachitsky?

  Data centers and space and robots and artificial intelligence.

• 58:30Speaker 01

  Yeah, okay, I see it now. I see it now. Yeah, somebody like recently told me, you know, they used to think that may you live in interesting times was, was like a blessing and they're not so sure now. They're like, ah, you know, the boring times are like pretty good too.

• 58:44Lenny Rachitsky?

  Oh, man. And then there's an Emmanuel out there watching your thinking process in your J space.

• 58:51Speaker 01

  Oh my God, yeah. Hopefully I'm making him proud. He's not like, oh, I gotta rerun the simulation, we messed up this one.

• 58:56Lenny Rachitsky?

  Shut it down. Oh, man. Okay, one last question randomly out of nowhere. So Swix, who runs AI engineering, Latein Space Podcast, awesome guy. You were asked him what to ask you. You were on this podcast a year ago and something completely from a different direction. He wanted to get your take on Jev, which is this hot new model that recently came out that is very fast making decisions. OpenAI, funny enough, just launched their own version like a week later. Thoughts on, on Jev in the space of quick decision-based data models?

• 59:32Speaker 01

  Yeah, right. So my, my take on Jev is this sort of, you know, these kind of models, you, you give them some context and you give them a few options to choose between and they just are very quick at, at choosing an option for you. To me, it's impossible not to hear about this and think of, of sort of system one thinking versus system two or, or something, right? So like a lot of what I've studied with these models is how they sample one token with, with like the Claude models. And so, you know, you can think of, of, of like these decision models are, are pretty similar. So like taken on and sample one token. They're, they're different important ways, but like at a fundamental level, you can be like, oh, it's, you're not letting the model, you know, sample and like reason.

• 1:00:12Speaker 01

  It's just you're like gut, gut feeling, you know, is the answer A or is the answer B? To me, that seems like, like something that's, like that general frame of like, sometimes you'll just want a binary decision or like one out of 10 or whatever. It's just like clearly, I don't know, like, like both clearly useful and something that I imagine is, is like, probably going to be a part of, of like intelligent systems, right? Where you, you have, for some stuff, you need to think about a long time. You know, it's like, what, what's our roadmap for the next year? You need to think about for a while. And then for some stuff where it's like. You know, is this is this person who wrote us a review happy or upset? You kind of you're happy with the sort of gut judgment?

• 1:00:54Speaker 01

  And so, yeah, I think I think this is I think this kind of research is is cool. And I think it's also kind of different from the, you know, it's like it's like a bit of a novel take on on some of the elements we've seen. So, yeah, I'm pretty excited to see where where this kind of stuff goes.

• 1:01:08Lenny Rachitsky?

  So I'm hearing here is this kind of going to be a new foundational component, most likely of of AI and LLMs. And the system one, system two stuff is basically Kahneman's kind of I have that book up there actually behind me. The system one, just to keep it simple, is just like kind of your amygdala limbic system making just like gut level quick decisions like snake, oh no.

• 1:01:26Speaker 01

  Yes.

• 1:01:27Lenny Rachitsky?

  Uh, where you don't have to think long about it. It's kind of to protect you from bad things really easily and quickly. And system two is where we really think about stuff and process.

• 1:01:36Speaker 01

  Yeah, exactly.

• 1:01:37Lenny Rachitsky?

  Sweet. Emmanuel, is there anything else that we haven't covered? Anything else you want to say or mention? Anything you want to double down on before we get to our very exciting lightning round?

• 1:01:48Speaker 01

  I'm definitely curious for for folks takes always on on some of the stuff that, you know, is still an export or didn't make sense. There's a lot more in trip experiments than we have the time to share. You know, models have a bunch of weird weird little things about them. Like, maybe I'll do a lightning round of weird little things.

• 1:02:08Lenny Rachitsky?

  Let's do it. Absolutely.

• 1:02:09Speaker 01

  Okay. Weird little thing. One, we found that, you know, models can like actually answer questions. Obviously, they can do math or they can, you know, you can ask them like, what's the color of a banana? And they'll actually like, you know, if you look inside, you'll see all these intermediate steps, but they have a separate mechanism, which is kind of like a gut check and which is totally different and which is just like as they reason, they'll have like some part of them that will flag like, hmm, something's wrong. So if I'm all like somehow says, you know, four plus five equals 23 or something. And there's a part of them all that just like reads that and doesn't do the math, but it's just like, that can't be right because like four plus five definitely under 10. And so they learn these weird kind of like hunches. They feel like kind of kind of like like a hunch.

• 1:02:52Speaker 01

  Um, maybe another one is you can tell the model, hey, do this task and think about this other thing. And then you can use Interp to see if like there's intermediate representations that map to the other thing. And that works. You sort of see that it does think about the other thing. This is something that's really mysterious to me because there's no kind of training reason why why the model would would we have to learn that. So it's sort of like another one of these weird behaviors. And actually, maybe I'll keep that too. I think those two are are good enough.

• 1:03:24Lenny Rachitsky?

  Just what a fun, what a fun job this is just studying this thing and trying to understand what the hell is going on. How is it getting the smart? How does it think?

• 1:03:32Speaker 01

  Yeah.

• 1:03:34Lenny Rachitsky?

  How many like, how many people are even? How many people are even in this space? Just like, how large is this field?

• 1:03:39Speaker 01

  The field itself is, you know, is getting pretty large. I would say maybe hundreds, not thousands. Uh, there's, you know, uh, different inter people at the different labs. And then there's there's, um, in academia, quite a few labs putting out great work.

• 1:03:54Lenny Rachitsky?

  Is the best, if someone were to get into this space, are they? Is it like an ML person usually? Or is there like a neuroscientist sort of track to get into this?

• 1:04:04Speaker 01

  Yeah, there's actually a surprising amount of of, you know, ex neuroscientists. Uh, I do think in order to get in the field, you know, there is a there is actually, it doesn't require a lot of prior knowledge because it's such a new field. So it's actually, you know, the on ramp is is is mostly, I would say, like self studying and reading some of the some of the the key papers. And then there's actually, you know, uh, yeah, maybe I should give a plug. There's actually like websites like gernpedia.org, I think is the URL, which are sort of like open source inter. Uh, so they had they like host, you know, a bunch of models and let you sort of do inter experiments on them. And so, you know, people have just been been doing that on their own.

• 1:04:46Speaker 01

  And I think that there's so much to explore that if if you're at all curious, you should just give it a try.

• 1:04:52Lenny Rachitsky?

  That's awesome. We'll link to them in the show notes in case that's not the right URL.

• 1:04:55Speaker 01

  Cool.

• 1:04:55Lenny Rachitsky?

  Well, Emmanuel, with that, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

• 1:05:01Speaker 01

  I am as ready as I'll ever be.

• 1:05:03Lenny Rachitsky?

  Here we go. What are two or three books that you find yourself recommending most to other people?

• 1:05:07Speaker 01

  One I read recently and is A Brief History of Intelligence. Uh, it's this really cool book that walks through the history of like basically brains and five key moments uh in in our evolution that sort of bootstrapped these brain systems on top of one another. It's super accessible, like you don't need any neuroscience knowledge or anything.

• 1:05:33Speaker 01

  Another one is Story of Your Life and Others. This is that Chang. This is maybe my favorite sci fi book of all time. It's a bunch of short stories. It does what I think is the the like high art version of sci fi where it's just take one weird idea like, what if, you know, um, this happened? And then write a story of just like, what would the world look like if that happened? Um, it's truly excellent.

• 1:05:55Lenny Rachitsky?

  I think I have that in my bookshelf behind me, too. Um, I'll also highly recommend that one. Favorite recent movie or TV show?

• 1:06:04Speaker 01

  Okay. One popular one, one deep cut, uh, obsession was my favorite movie of the year. Uh, it's a horror movie, so don't don't go if you don't if you hate horror movies.

• 1:06:16Lenny Rachitsky?

  No, thank you.

• 1:06:17Speaker 01

  Oh, no. Even if you're if you're not squeam, if you're open to a horror movie, you should go because it's, uh, it was just so fresh. You know how movies, I feel like if you've seen a lot, sometimes like, oh, it's another I've seen this many times. This one was just so new, so fresh. And I'll say one thing about it, which is that it takes a lot of rom com tropes and like subverts them to said, be spooky. And I think that that's just a brilliant, brilliant way to to make a movie. And then I'm French, so there's a TV show that came out this year called Bref deux. Bref is French for brief.

• 1:06:51Speaker 01

  Brief used to be this cult classic in France ten like 15 years ago, and they came up with a sequel, uh, many, many years into the making. And you can watch it. I had a bunch of American friends that have watched it. It's got subtitles and it's just this incredibly, again, it's like novel, like the editing is crazy. Um, they are just extremely creative and is very short. So I would recommend that.

• 1:07:14Lenny Rachitsky?

  Favorite product you recently discovered that you love?

• 1:07:17Speaker 01

  I I love my Apple Watch and I hate that it charges, uh, or I have to charge every day. So I bought the Apple Watch Ultra, uh, just recently. And man, that is so nice. If you've ever been frustrated by this good product that somehow we can't figure out how to give a good battery to, well, it turns out if you make it bigger, uh, the battery's better. So I've been happy with that.

• 1:07:39Lenny Rachitsky?

  Do you have a favorite life motto that you find yourself coming back to often in work or in life?

• 1:07:43Speaker 01

  There's like one nonfiction book that really is like probably my favorite ever. It's 4,000 weeks. And it has a lot in it that that I think is cool. Uh, one of the cool things is this sort of thing that was really relevant to me, which is that like the when I finally mentality is a common mentality and is kind of toxic. So so what does that mean? It's sort of this like, ah, when I finally, you know, check off this, this, all the items on my to do list, I'll finally be able to chill. When I finally get like married, I'll be happy. When I finally get a promotion, uh, and then this is sort of, I mean, you were talking about meditation, right? This is like almost the opposite of meditation and that, you know, at the end of the day, your life is like all of the moments you lived, uh, in the moment and that there's

• 1:08:27Speaker 01

  this sense of actually should just try to always like be, you know, in the moment and not projecting yourself towards like a future that seems better. Like, probably if you're telling yourself, when I finally do X, I'll be happy. When you do X, you will not be happy.

• 1:08:42Lenny Rachitsky?

  Is there a better phrase to use or is it just don't do that? Like, do they say?

• 1:08:45Speaker 01

  Uh, I don't know. I think to me that the positive version is like be in the moment, right? It's just like, just I appreciate the appreciate the moment. There was a blog post recently I read that was, oh, I forget, something like, like enjoy the sandwich, I think is, I think it was literally the sense. I don't know if you saw this, or it was like somebody was like, yeah, like, just you're eating a sandwich, you know, you've eaten many sandwiches, like, enjoy the sandwich. You're in you're in like a a team meeting, you know? Yeah, you've been in my like, like, enjoy, like, be be in it and and sort of not looking looking towards the next thing.

• 1:09:14Lenny Rachitsky?

  I did a meditation retreat once. It was a 10 day silent retreat where you eat silently too, and no one's talking. You just sit there and eat your food. And I just remember peeling an egg for like five minutes and just noticing every every shell piece and just the smell of it all. And and it like almost, it's like hard to get into that mode, but it's such an interesting place to be where everything is just interesting, like, wow, this egg versus just like, oh, stuffed in my mouth.

• 1:09:39Speaker 01

  Yeah, right. While you're like on Slack with your left hand and like listening to a podcast at the same time.

• 1:09:45Lenny Rachitsky?

  Oh, man. Okay, final question. You study AI all day. You're watching AI how things. If if an AI were to look inside your brain, what would what would be surprising? What would it see that might surprise people?

• 1:09:58Speaker 01

  I mean, surely my deepest, darkest secrets, but what are the?

• 1:10:01Lenny Rachitsky?

  Get into it. Let's get into it.

• 1:10:02Speaker 01

  Oh, man.

• 1:10:05Speaker 01

  I've always been like a bit of a jokester, uh, or that's just how I like, both how I live and how I cope. I just like think that even in serious situations, you should you should aim to kind of have some levity. And so I think if if you looked inside my brain, you'd definitely be surprised, uh, that like basically at any given moment, I'm just like focusing on some random thing that is funny. And I'll be in a in a serious meeting and I'll be like, oh, yeah, but you know, like, isn't it kind of random that you're like, like, wearing this random thing or something? It's just like, I have these intrusive thoughts that I that I keep out, but that are all usually like just bad jokes. And sometimes, as you've heard today, unfortunately, they come out. So I'm sorry for those, but that's that's probably the most surprising thing.

• 1:10:48Lenny Rachitsky?

  I love that. Emmanuel, thank you so much for being here. Uh, final question, how can listeners be useful to you?

• 1:10:53Speaker 01

  Just do some Interp, read some, read about how the models work. I mean, you mentioned like Dorkers, right up. I think it was it was great as well. I think I think, you know, it's just, this is such a new field, like both Interp and like AI and LLMs. And I think it it just benefits from from people engaging and and, um, being informed. So like read things if there's if there's more that you would want to know about how these models work that we haven't figured out, you can, you know, reach out to me. I'm on a variety of social social platforms, like probably you can reach out to me on X.

• 1:11:26Lenny Rachitsky?

  And we'll link to some of the videos that you guys have put out. They describe a lot of the stuff we've been talking about, like J space and things like that. There's some really beautiful videos your team has made. I don't know who does that, but they're really cool.

• 1:11:36Speaker 01

  Yeah, this is, we partner with our our design team, so all credit goes to them for the for the cool videos.

• 1:11:41Lenny Rachitsky?

  Amazing. All right. Well, thank you, Jam and Emmanuel. Thank you so much for doing this. This is amazing.

• 1:11:45Speaker 01

  Thanks, Lenny. Yeah, this was super fun.

• 1:11:47Lenny Rachitsky?

  Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.