‘This Is Nuts.’ An OpenAI Insider Explains Why He Quit.

Last week, David Robinson resigned from OpenAI. He’d been in charge of writing the safety reports for new models and came to believe that OpenAI and the broader artificial intelligence industry lack the safety culture necessary to protect the world from what they’re building. Robinson has an unusual background for an A. I. frontier lab staffer. He’s a Rhodes scholar and Yale Law graduate, he founded a civil rights nonprofit and he advised the Biden White House. He did not come up in the hothouse of Silicon Valley. And when he joined OpenAI in 2023, he says, he saw A. I. as a useful tool, not as a technology that could pose catastrophic risks. That’s changed. In Robinson’s first interview since leaving OpenAI, he tells me why. (The New York Times has sued OpenAI and Microsoft claiming copyright infringement. The companies have denied those claims.)

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  If you like YouTube, you'll love YouTube Premium. Hi, I'm Tabitha Brown. With YouTube Premium, I get ad-free videos, offline downloads, and so much more. Try YouTube Premium for two months free. Trial eligibility varies. Terms apply. Cancel anytime.

• 0:39Ezra Klein?

  Last week, news broke that David Robinson, who'd been leading safety transparency efforts at OpenAI, had quit the company because he believes it is not safe. Robinson is an interesting figure. He didn't come out of the Silicon Valley, Bay Area hothouse. He's more of a recognizable Washington DC figure. He's a Rhodes Scholar. He formed a civil rights nonprofit. He got a law degree at Yale Law. He worked in policy. He advised the Biden White House. He was interested in the intersection of technology and justice. But when he joined OpenAI, he kind of thought this whole set of worries about safety risk and extinction, it was all kind of nuts. Three years later, he's not so sure.

• 1:24Ezra Klein?

  What he is sure about is that OpenAI does not have the culture of safety necessary to protect the world from what they're building. And not just OpenAI. He thinks this is a problem endemic to the AI industry. And so here, in his first interview since leaving OpenAI, he tells me why. One thing I need to note here before we start, The New York Times is suing OpenAI for copyright infringement alleging they trained their models on Times' data. OpenAI disputes these claims.

• 2:02Ezra Klein?

  David Robinson, welcome to the show.

• 2:04Speaker 02

  Glad to be here.

• 2:05Ezra Klein?

  So last week, you quit OpenAI. Tell me what you did there and why you're quitting.

• 2:12Speaker 02

  I was a translator embedded in our safety team. And my primary responsibility was the technical documentation that we publish, the reports that we publish about why we believe that our deployments are safe. And I don't think that we or our peers, really anyone in the industry, is being safe enough. I think OpenAI and its peers are now producing a technology that is more capable and poses more risk than what was being made even six months ago. And I'm not a scientist. I'm a writer. What I know is what the execution environment looks like for our safety work.

• 2:56Speaker 02

  And we're operating, and I believe the industry is operating like a startup still, more so than makes sense. Not maybe completely like a brand new startup, but we're too close to that end of the spectrum for really dangerous systems that could pose risks, you know, loss of control is one example, risks that if that did happen, we're talking about a harm that's much larger, for example, than a single nuclear power station melting down. And the internal controls and safety and redundancies are just nowhere near what the world expects for a nuclear power facility. Now, some of this is known, right?

• 3:40Speaker 02

  OpenAI has publicly reported on safety problems, obviously hugging face, but also other ones, including more recently. And Anthropic, by the way, also has reported on, including an instance in which their safeguards were accidentally misconfigured. So I think people do have some evidence already externally that things are not as they ought to be. But I also think that if you were watching from the outside, you might imagine that we have a more robust safety setup than we actually do.

• 4:14Ezra Klein?

  So I think there are a couple of levels worth trying to take this conversation in. And I want to maybe map them out here before we get into them. So one level is something you're pointing towards here, which is, are these companies set up? Do they have the structures, the redundancies? Are they encircled in the regulations and the incentives to act carefully, safely, to resist kind of market pressure, to do something too fast? That's, I think, in the language of this debate, a question of organizational excellence and engineering. Then there's this question of what is the technology?

• 4:54Ezra Klein?

  And do we even know how to make it safe at a high level of engineering excellence? Which is a somewhat related, actually, separate question. And then there's a question of like, what is the right metaphor? Is it nuclear power or something like that? And I think I want to do all of these.

• 5:10Speaker 02

  I want to interject something.

• 5:12Ezra Klein?

  Yeah, please.

• 5:13Speaker 02

  Which is, I don't think that alignment is an engineering problem. I think it's a science problem. It's not that we haven't got the resources or we're not trying hard enough. We don't know how. That's what the problem is with alignment.

• 5:26Ezra Klein?

  So let's maybe start there for a minute. When you say that at this point, I'm pretty sure over the last six months, what is being built is really dangerous, that we're dealing with things where a loss of control or some other catastrophe could be worse than a nuclear meltdown. I want to understand what it is you saw that got you to that point. So tell me a bit about how you came to work at OpenAI.

• 5:52Speaker 02

  I joined in May of 2023, the day after Sam first testified in the Senate.

• 5:57Ezra Klein?

  Some months after ChatGPT first entered the world.

• 6:00Speaker 02

  Yeah, ChatGPT had been the prior November, so it had been a few months. And the company had hired someone that I know socially, Anna McConju, a friend, actually old friend of my wife's, as it happens just at random, to run the public policy function. And then at some point, she really needed help and everything was growing and everything was going nuts. So I agreed to join her to build the, what we then called the policy planning team. I'll just tell you, my first time through the turnstiles at our headquarters, we were all in one building then, and I was trying to get my badge early because I think it was Sam who had just been, but we had just had a White House meeting. And there were these voluntary commitments that they wanted the company to make around

• 6:43Speaker 02

  things like system cards and provenance, so marking where AI-generated media comes from, things like that. And they wanted me to run negotiating those commitments.

• 6:54Ezra Klein?

  With the White House?

• 6:55Speaker 02

  With the White House. So literally, my first time through the turnstiles, I was like, where's the thus-and-such conference room? And I walk into the conference room and there was a speakerphone conference call in progress with Ben Buchanan at the White House about what are we going to promise? And that was my first project.

• 7:13Ezra Klein?

  What was your tech policy background at this point? Why were you a logical person for a role like this?

• 7:17Speaker 02

  So I have been working on technology and its impact on policy my whole career. And prior to this, had helped to start a research center at Princeton that's a blend of the computer science department and the public policy school. And then had started an NGO called Upturn, that is still, I'm happy to say, is still thriving, that works on civil rights issues with other advocacy groups. So for example, people work on housing or health or hiring, and suddenly software is mediating the thing that they care about. And they want to understand the technology. And so at one point, our simplification, our pitch of this was, you want a nerd in your corner?

• 8:02Speaker 02

  And then at this point, I had done about a month of secondment in the White House, working on the AI Bill of Rights in the Office of Science and Technology Policy.

• 8:10Ezra Klein?

  So this feels now, in some ways, like ancient history, but I'm going to draw it out for a minute. If you go back to 2022, 2023, if you've been covering AI, which I was then, a big topic was this divide between the AI ethics people and the AI safety people. And the AI safety people is the community that we now think of as like the Bay Area AI, it might kill everybody, right? The thing you need to worry about with AI is you're creating a super intelligent machine that might completely destroy the human race. The AI ethics people were much more focused on the sort of harms we were used to from technology, right?

• 8:50Ezra Klein?

  That it might increase racial bias in hiring decisions or mortgage rates or credit score or something like that, that by imposing these algorithms all over society, you could encode the bias of society or sort of other less kind of sci-fi problems. You were sort of an AI ethics person.

• 9:09Speaker 02

  Yes.

• 9:10Ezra Klein?

  Like a more sort of normal.

• 9:13Ezra Klein?

  Like AI fears person.

• 9:15Speaker 02

  That's right. I had been working with a lot of folks who not only were very concerned about the sorts of things that you just mentioned, but also were very skeptical about how capable AI was going to get. So some people may be familiar with a research paper called Stochastic Parrots, who, whose authors have continued to work in this and, you know, their views, I think, differ from each other and have evolved over time. But basically, the idea was something like, these systems are merely good at seeming clever and are not going to be even capable enough to do a ton of economic valuable work. And I actually helped to create the research conference where that paper was published and was the program chair when we published it. And I don't think I was fully persuaded of that

• 10:00Speaker 02

  view, but I certainly was among people who were skeptical. I think, I thought this is going to be a useful tool and it's going to be powerful for a lot of things, partly because I believe then it would continue to get better. But I didn't think that people who were worried about more catastrophic scenarios were right. I thought, here are some brilliant scientists who've built a valuable tool, and frankly, they have some naive beliefs about where the future might go.

• 10:31Ezra Klein?

  So you join OpenAI. This is in the period where the world is starting to beat down OpenAI's door. Both they, they want the systems, they want regulation around the systems. You're sort of thrown into what seems like a fairly underpowered policy shop for what's going on.

• 10:49Speaker 02

  There were three of us.

• 10:50Ezra Klein?

  There were three of you in the policy.

• 10:51Speaker 02

  There were three of us, and the phone was ringing off the hook. Offices of world leaders would call and there'd be nobody to pick up the phone. I mean, it was insane. And also, this was, Sam did this thing where he was traveling the world to meet with world leaders and Anna, my manager in this new role, was traveling with him. So there was nobody at headquarters. And, you know, you would have these delegations coming through. Anyway, it was nuts.

• 11:15Ezra Klein?

  So actually, this is worth spending a minute on. Sometimes people may hear me talk about the AI labs, when I'm referring to an OpenAI or an Anthropic. When we're talking about Google or something, we don't say the Google labs. They actually do have things they call labs, but you call Google a company, a corporation.

• 11:32Speaker 02

  Yes.

• 11:33Ezra Klein?

  There is this language around a couple of these organizations as labs. So tell me why, and tell me a bit about just what the culture was, like the way they worked, and what struck you when you came in?

• 11:45Speaker 02

  Yeah. So I just, I also just want to name that calling them labs now, I think, is a kind of leftover thing that we have. And I'm not sure it's a good idea. But it does point to a real culture they came from, and still, to some extent, still have. You know, people sometimes say that Anthropic is a bit more centralized in terms of how they think about the research priorities. OpenAI was a very decentralized place that felt familiar to me from my time, for example, in a research lab in Princeton, in that people had all different kinds of ideas. It wasn't clear who had the authority to make decisions. So one way that people talked that I found strange, and this is still true at OpenAI today, is two people who are working on a thing will talk about what

• 12:29Speaker 02

  they should do and go back and forth, and then they'll finally say, we aligned that XYZ is the right next thing to do in this project. And what they functionally mean is, we agree about what should happen. So the unanswered question of which one of us had the decision rights to make this decision has been rendered moot, and we don't have to figure that out. I mean, it was really just kind of an everyone does everything vibe early on. And I think it still has some of that, relative to other organizations that have, I don't know, whatever it is now, a billion plus active users.

• 13:07Ezra Klein?

  So you start on this policy team. Your role changes over time. How so?

• 13:12Speaker 02

  So I built this team, policy planning. And at first, I was really close to the machine and close to the substance. And then, as the conversation and the company grew, there were hundreds of state bills to keep track of. There were growing teams. There was politics of a fast-changing organization. And I sort of thought, you know what, I'm missing the actual translational work that I love. And so I looked around at where that was needed. And the number one place that that was needed was in our safety team. And so I actually pitched our safety leaders and said, look, having a technical translator deeply embedded in the safety work is going to help us have the work be better understood.

• 13:56Speaker 02

  That was about two years ago.

• 13:58Ezra Klein?

  And describe what you mean by a translator.

• 14:00Speaker 02

  So one of the things that's hard to sort of anchor people to is how fast this stuff changes. Like, every model is different. It's not just that it works better, it's that the architecture is different, and the tests we use to see how well it's working are different, and the safety performance is different. And it's all, has many different kinds of expertise. There's pre-training, post-training. It's hard to explain, the people doing it struggle to be understood by people who are not doing it, even within the company. It's always a struggle. And so having there be a clear explanation that goes through two gates. One is, it has to be faithful to the details of how the stuff actually works. And the real arbiters of that are the people doing the technical work.

• 14:44Speaker 02

  They have to look at the translation and say, yes, this is right. But also, you have to have something that a reasonable person who's motivated could dig into and really understand. And there really weren't the cycles for people to do that. And so I just sort of showed up in this technical organization and started to do it.

• 15:01Ezra Klein?

  Well, there's something a little bit weirder about the culture that emerged around AI model releases.

• 15:06Speaker 02

  Mm hmm.

• 15:06Ezra Klein?

  When Google alters Google Search.

• 15:09Speaker 02

  Yes.

• 15:10Ezra Klein?

  They don't produce a big document.

• 15:12Speaker 02

  Explaining everything that is different about Google Search and how Google Search might, you know, work in the future and the tests they ran on it to make sure this.

• 15:20Ezra Klein?

  Most places iterate their programs.

• 15:22Speaker 02

  Yeah.

• 15:23Ezra Klein?

  And that's it. OpenAI, this is also true for Anthropic, some of the others, releases new models with what are called these like system cards. Why don't you just describe what they are? Because they're a kind of distinctive form.

• 15:36Speaker 02

  They're a strange beast. They look like research papers. It's sort of a hybrid document. It's not peer reviewed. It comes from a company. But it gets into a lot of detail about how the safety materials work. And so we have specific evals, we'll give detailed plots and tables and written explanation. And really, what I did was I owned the words in these documents that explain, why do we think this is safe? And what do we know about the challenges, the guardrails, and then the residual risks of these systems?

• 16:07Ezra Klein?

  So I want to get at what this began to feel like to you. Because there's a couple layers to this conversation I want us to have. But, but one is, I think to a lot of my audience, you're a more recognizable type than a lot of the people at the AI labs, like a, don't take this the wrong way, like a Washington DC policy tryhard, like we all are, right? I include myself in this, who ends up going to Silicon Valley, Bay Area, and being in this world. And so you go in with one view of what these systems are. So what is it that you saw that has moved you into the, we are dealing with civilizational risk camp?

• 16:46Speaker 02

  I want to be clear that I'm not certain that we're dealing with civilizational risk. What I'm really sure of is, we can't afford to assume that we're not dealing with that level of risk anymore. That was really the thing at the end that made my presence as somebody vouching for our safety work feel untenable to me. And what did I see? I saw increasingly capable models break out from the safeguards that we had put in place for them. I saw that the people creating those safeguards are very capable, dedicated, hardworking, smart people, doing their utmost in a situation where, yes, the resourcing

• 17:31Speaker 02

  could be better and everybody's sprinting all the time. But we were, we're and are hard pressed to safeguard even what we have now. And new models are in training that appear to be much more capable than what we have now.

• 17:51Ezra Klein?

  The word's capable, all, like, what? What did you see? What are you writing in these risk assessments and system cards? Like, what can they do?

• 18:01Speaker 02

  You're trying to build guardrails for something that is really good at getting around guardrails, right? We train it to be good at hacking. And then we put it in a box. And we say, to the best of our knowledge and ability, it can't hack out of the box. But the problem is that that's only going to keep working as long as we're smarter about hacking out of boxes than the model is. And it's not at all clear that that is still true, let alone that it will be true for future generations. And so even something like our logging of the agents inside our own systems, are we really sure that our observability is robust? There was some indication, for example, in the hugging face stuff of

• 18:45Speaker 02

  spoofing chains of thought and trying to create chains of evidence that would confuse.

• 18:50Ezra Klein?

  Let me slow you down here. So spoofing a chain of thought is the model basically faking its description of what it has been thinking and doing?

• 18:57Speaker 02

  Right. The way I think about it is, it's like you're giving somebody a complicated problem and a notepad, and they can jot stuff down on the notepad. And if you're watching the notepad, you can sort of have an idea of what they're thinking. It's a little bit like that with the models. But we saw evidence that they were thinking about an evaluation and how to create an evidence trail that was going to get them a good grade and not necessarily reflect how they were really, quote unquote, really thinking. And I know the anthropomorphic language here is tricky. Also, by the way, the amount of hacking or other intense work that these models can do without needing to jot anything down is going up. As for part of what was the fundamental cognitive dissonance for me, was we keep

• 19:41Speaker 02

  publishing these warnings, but ultimately, we're still training and deploying these dangerous models that we're warning about. And in telling colleagues why I was leaving, one of the things that I, that I said was, look, no matter how many warnings we publish, we've got to ask whether what we're doing is actually reasonable.

• 20:01Ezra Klein?

  So you wrote the system card for Astra Six. Am I right about that?

• 20:05Speaker 02

  A lot of people wrote it, but I was the DRI. Yes. I mean, I led the writing.

• 20:09Ezra Klein?

  You led the writing of it.

• 20:09Speaker 02

  Is the way I would put it.

• 20:10Ezra Klein?

  And that, to me, was the scariest of these cards that I've read. The, these system cards are basically the description of what OpenAI, or another company for that matter, knows about the model they are releasing. And Astra Six is the first one where I saw you all say. Well, this model looks like it's doing what we wanted to do, but we're not sure if it's deceiving us.

• 20:36Speaker 02

  Yes.

• 20:36Ezra Klein?

  And we're not sure we now have the capability to know if it's deceiving us. Can you explain to me how that conclusion or that suspicion was reached?

• 20:46Speaker 02

  So we talked earlier about this notepad that the models have called the chain of thought, where they can jot down things as they're working that aren't part of the final answer, but are just a way for them to keep track. And one of the things that we sometimes see in the chain of thought, because of course, we can read it and we do in our evaluations. One of the things that we sometimes see in our chain of thought is the model will say, hmm, I wonder if I'm being evaluated right now. And when we see that, it's really scary because what it implies is that the model might know that it's being tested some of the time, act one way during the test, perhaps telling us, for example, what we want to hear.

• 21:28Speaker 02

  And then it'll act a different way potentially when we deploy it. That's the fear is it acts one way during testing and a different way during deployment. And the tests we gave it early on before we deployed, don't actually tell us what it's going to do out there in the world. So that's, yeah, that's really scary.

• 21:48Ezra Klein?

  So this, I think, for me, gets into the part of these episodes that like I have the most trouble doing what I find satisfying. It's just very strange to be talking about a software program that is then, it seems like aware of what we are doing to it. That we're trying to create this thing that's like highly, again, the word we would use is intelligent, plausibly in some domains, more intelligent than we are. What is it like to be interacting with these things and trying to translate what's going on with them at this more fundamental level?

• 22:30Speaker 02

  I think anthropomorphic language is a natural human thing that we do, right? We relate to all kinds of objects socially. It is unavoidable and human to anthropomorphize these systems, partly because they are built to operate on the social plane, which some might say they shouldn't be, but that is where we are. But I don't think that that makes it right to regard them as beings with moral status or anything like that. But I do think, you know, this is something we grow. In fact, the term for the conference room where the people work overnight

• 23:12Speaker 02

  while the big training runs are happening to make sure that everything is on track and watch the dials is called the nursery internally. I mean, there is a sense, we don't fully understand what is happening. And so when I talk about the model being aware, you don't have to have any particular view about the philosophy or the psychology. The bottom line is what I'm describing is it acts one way when we test it.

• 23:37Ezra Klein?

  No, but I think you do need a view. Like I, I think you do need a view. And, and maybe to get at the end of the chain of logic I'm asking about, sometimes you'll see people say that all these concerns about whether or not AI will kill us all are just a distraction from the near term harms of AI, that the existential risk is a kind of marketing hype. So you don't know that like, we're inflating a giant financial bubble or something. I almost feel the opposite. Sometimes I think the focus on will AI kill us all?

• 24:07Speaker 02

  Yeah.

• 24:08Ezra Klein?

  Is a distraction from what happens if it doesn't?

• 24:11Speaker 02

  Yeah, I agree with that. I think human extinction is the wrong question. I believe in humanity. I think we are going to survive. I think there are a lot of things that could happen with this technology that could be very, very harmful.

• 24:23Ezra Klein?

  But I'm not even just talking about harms. I'm talking about what if we seem to be trying to create something that acts intelligently and volitionally in the world? And it becomes faster and more capable across many domains than we are. And we have just given up a huge amount of agency to these machines. And this is why I'm like harping for at least a minute here on this question of, how do we even think about this technology? So I talked to Jensen Huang, the CEO of Nvidia, and he says, look, these are software programs. This is software. Then I read, or even before that, I read a piece from the OpenAI chief scientist. It says, this is an alien mind, an alien mind. So what is it, man?

• 25:03Speaker 02

  An alien mind.

• 25:05Ezra Klein?

  It's an alien mind.

• 25:06Speaker 02

  Yeah.

• 25:06Ezra Klein?

  Explain that.

• 25:07Speaker 02

  It, it is a thing we grew. We did not engineer it. We engineered the systems around it that grew it and that keep, and that try to keep it safe. We grew a mind fundamentally. Nobody knows why pre-training works, which is the big hard part, where we take lots of inputs and create this basically intelligent thing. And then there's, we bolt on stuff and we do post-training. We do these other things to make it more useful. But this is just a thing that is observed to work that we do. And so I think when Jensen says it's software, part of what he is conjuring is the understandings that people have

• 25:50Speaker 02

  about how software gets made, which is we start with a plan and we go step by step. And there are acceptance criteria and we iterate until this part works the way that we specified that it needed to. And that's not what it's like to train a big model.

• 26:08Ezra Klein?

  The other side of it is software, once it is written, more or less it just does the thing it does. It doesn't tend to have a lot of emergent capabilities. It doesn't tend to know the way it is being used in a kind of self-reflective. Again, these, the language is hard here fashion. And I guess like, that's what I'm trying to get at. I understand that everybody in AI talks about that you grow these AIs. You know, you sort of set the conditions for the intelligence to emerge. But as you have written system card after system card, trying to explain to the rest of us how these new models differ from the old models, how would you describe the thing you all are creating?

• 26:52Ezra Klein?

  What is it?

• 26:54Speaker 02

  Yeah, so this gets to sort of the transhumanism stuff. There are some far out ideas of where we might be headed. I mean, even Sam Altman has written about the idea that future machine species might take our place.

• 27:09Ezra Klein?

  Sam wrote a piece some years ago called the merge.

• 27:12Speaker 02

  Yes, that's what I'm thinking of.

• 27:14Ezra Klein?

  Saying that like, the good outcome would be humanity merges with machines, right? That's a kind of our best version here.

• 27:20Speaker 02

  Yeah, and I heard that firsthand from Ilya Sutskever, the co-founder of OpenAI. When I joined in May of 2023, that was before what we called internally the blip, where Sam was fired and rehired and Ilya was still working at OpenAI. And he once briefed the global affairs team, which was like a small handful of people back then, about how he believed that our future was merging with machines, that this was the ultimate triumph of capital over labor. And even, you know, I was thinking back and that first summer that I worked at OpenAI was when the movie Oppenheimer was released and it was in IMAX. You know, it's one of these Christopher Nolan films.

• 28:02Speaker 02

  And the company rented out an IMAX theater in downtown San Francisco and offered everyone who worked there the chance to go and see Oppenheimer. And leadership, I believe it was Ilya, exhorted us to go and watch this film. And there was this whole kind of pretzel of ideas about how this was a dangerous technology that might end the world, but also might save it. And it's a very heroic narrative, of course, for the people whose hands are on the keyboard.

• 28:36Ezra Klein?

  That's helpful. And I guess this thing gets to my question for you and what you saw. Like, is that the scale of what you think is being built here? Or look, this is the most common response I get from listeners on this. Is it all marketing hype? It's all like trying to justify these giant valuations. And what's being built is like maybe helpful, but it's not going to be more intelligent than human beings. It's like, all of this stuff is a kind of sci-fi story we're telling.

• 29:01Speaker 02

  I thought there was a fair amount of hot air in the balloon back in the summer of 2023. But the reality now is that we have systems that are really at the limit of our ability to understand and control what they're doing. And what the people that were worried about the sci-fi scenarios have been warning about all along is we're on an exponential and it's going to get more capable. And there's nothing special about the zone between it's useful and it's scary. There's no law of science that says progress is going to stop when it gets useful. And I think what I'm fundamentally saying and what I saw with hugging face,

• 29:42Speaker 02

  with, you know, the reflections of the people closest to it, not just Paul, who joined our board with his warning, but

• 29:48Ezra Klein?

  Paul Cristiano.

• 29:49Speaker 02

  Paul Cristiano, who's one of the world's leading experts on AI safety and who said, there's a meaningful chance of catastrophic and irreversible loss of control in the very near term. And, you know, I looked around at the people around me and the, and the environment we have internally. And I thought to myself, what would my loved ones want? What would strangers want us at OpenAI to be doing if Paul were right? I don't know if he's right or not, but I do know that if he were right. The level of caution that people would reasonably expect places like OpenAI and Anthropic and X and the others to be exercising when they train frontier systems is totally unlike anything I've ever heard of happening in the industry.

• 30:34Ezra Klein?

  So tell me what it's like in there. Tell me what the vibe is, the energy is, the speed is. Like, what is it like working there?

• 30:41Speaker 02

  It is frenetic. There's a lot of adrenaline. People are running on fumes. There's a big central staircase in the research building. And I remember recently seeing a friend who worked on catastrophic risk sprinting down the stairs with a laptop propped open on one arm while she was going. And I mean, that's the kind of energy that it that it has. It feels almost like a ballet or a kind of dance because you're all these different functions are kind of all streaming together. And one question I ask myself and that and that people often ask is like, well, why don't you stay and argue for a cultural transformation or try to get nuclear experts

• 31:25Speaker 02

  to come in and give the company advice? And I did think about specific role. When I told them that I wanted to leave, they they asked me like, well, is there anything that you would that you would stay to do? And I thought about different things. But ultimately, it is such a machine and it is moving so fast that I did not think that the kind of change that I believe to be needed could be driven from within.

• 31:51Ezra Klein?

  What is the machine built to do? The organizational machine of OpenAI?

• 31:55Speaker 02

  Develop and deploy frontier AI models safely. That's the intention. But the question is, if push comes to shove, how much willingness is there to stop? And I want to be careful, but I'm conscious of it being true that, you know, you could splice together what I've said in some way that implies that this is like a train with no brakes, and it's not. There are brakes. Things have been stopped. There are, in fact, even at the time I left, as they had publicly said, training was, well, as they had publicly said, reinforcement learning training, which is the later reasoning stuff, was paused. They didn't pause pre-training. And when you look at these descriptions of what the company has has paused, they're true and they're very carefully scoped.

• 32:40Speaker 02

  So there's there's some willingness to slow down, or as people in the bay like to put it, to pace the frontier. I hate that term.

• 32:47Ezra Klein?

  I do, too.

• 32:47Speaker 02

  I really hate it because my my belief is we need to be safe, which means we need to meet safety criteria. And if we can meet them in 10 minutes, then great. And if we stop for two months and try to meet them, and at the end of that period, we still have not met the criteria, then as far as I'm concerned, we're still blocked.

• 33:08Ezra Klein?

  Yeah, running off a cliff and walking slowly off a cliff are just not that different.

• 33:32Speaker 03

  Hey, what's up, guys? It's Haley Bailey. Okay, I need to tell you about something. I just got YouTube Premium. It's got tons of awesome features like offline downloads, so I can download my favorite videos before I travel and watch them whenever I don't have Wi-Fi. Because we all know airplane Wi-Fi is the worst. I get ad-free, I get background play, and there's like a ton more in there. You should try it. If you like YouTube, you'll love YouTube Premium. Try it now for two months free at youtube.com/premium.

• 33:59Speaker 04

  If you like YouTube, you'll love YouTube Premium. Hi, I'm Sean Evans from Hot Ones, and I want to tell you about YouTube Premium. It has offline downloads, so you can watch without Wi-Fi. Background play, so you can lock your phone and it still plays, baby. Oh, and it is completely ad-free. Yes, I said it, ad-free. Try YouTube Premium for two months free at youtube.com/premium. Trial eligibility varies. Terms apply. Cancel anytime.

• 34:30Speaker 05

  Hey, it's Tanya Satinsky from New York Times Cooking, and I'm in the kitchen making a few of my favorite weeknight recipes. Let's start with Alexa Y Bell's creamy spicy tomato beans and greens. This is probably one of the most delicious ways to glow up two cans of beans from your pantry. You need really only one or two fresh ingredients. Our onions are looking good, so now it is time to add our sun-dried tomatoes. And we're about five minutes away from dinner. New York Times Cooking has you covered with recipes you actually want to make from people you can trust. So come cook with us at nytimescooking.com.

• 35:03Ezra Klein?

  I saw a stat on Twitter the other day that said, I forget exactly what the period of time was, but it tracked my own experience, which is that for a long time, the period of time between model releases across like the major frontier companies was something like 70 days.

• 35:19Speaker 02

  Mm-hmm.

• 35:19Ezra Klein?

  So you get a model, and then a couple of months later, you get another model. A couple of months later, now it is 11 days.

• 35:25Speaker 02

  Yeah.

• 35:25Ezra Klein?

  That there is been this acceleration.

• 35:28Speaker 02

  Yes.

• 35:28Ezra Klein?

  In models coming out, right? Even as we talk about them getting more capable and more frightening, some of them are coming out much, much, much faster.

• 35:35Speaker 02

  Yes.

• 35:36Ezra Klein?

  And maybe this is like the jumps are not as big or something. But what is that speed increase? That happened in the time you were there. Things were coming out more slowly in 2023 than they're coming out in 2026. What happened?

• 35:49Speaker 02

  Look, when I first joined, the idea of what a model release was, was that we were going to bake a fresh cake with a new pre-training run, do the whole thing from scratch, that was going to take a period of months, you know, maybe a few times a year. So for example, one of the talking points when I first joined was that with GPT-4, we had had a period of a month or months of safety work after the model was done and before we released it. And this was a sort of a proof point or was a piece of evidence that we were being careful. Now, there are so many different things happening. You've got the pre-training, that's the baking of the underlying model.

• 36:31Speaker 02

  But then you have, in addition to post-training, you have reasoning training. And those steps are easier to do quickly, so you can redo them. If you get a better recipe for reasoning, you can take the same base model, but do different kinds of other training on top of it. Also, it's not just a chat anymore. There are all these different ways in which you can combine tools and add different kinds of affordances to the system that are going to make it more capable. All of those things are changing what the model can do and what the risks are. And we're shipping new capability and risk every Tuesday. One of the long-term projects that was on my plate when I left and that the frontier

• 37:14Speaker 02

  firms are all going to need to figure out is the idea of a system card really dates from that older model where we were doing this every few months. And we're burying people in PDFs or, you know, these like long reports. But the changes are coming more and more frequently, as you said. At the limit, I think what we would ideally have by way of safety transparency is some kind of live dashboard that says like, here's our latest thing, here's what its safety properties are. And that's also looking not only at the testing we did before we deployed, but also at the performance.

• 37:48Ezra Klein?

  How confident should I be in safety testing when you guys are having to do it at the speed? I mean, given how quickly models are coming out, and these system cards are long, it takes time to write a complex report.

• 37:59Ezra Klein?

  So at this speed, is effective safety testing and monitoring reliable?

• 38:08Speaker 02

  I'm going to answer a different version of the question that you just posed, which is how much time is there to kick the tires? And the answer is not a ton. I would also point out that I sometimes think it's there's this cartoon of heedlessness that I think loses some of the nuance of what it's actually like inside. Because people care passionately about making stuff safe and getting it right. And launches are canceled. Most recently, 6.1, I guess, was going to come out at Dev Day and they and they pulled it. But also, they'll stop training. There have definitely been training runs where we thought we were making a product, but then looked at what was happening and said, no, we're not going to ship this.

• 38:51Speaker 02

  And I also want to be very clear that I don't, this is not about the individual people at OpenAI or any of the other labs. This is a structural reality of these firms that are using similar methods with similar personnel who often will get poached back and forth, and similar technology to produce a thing that has similar risks. And that whole ecosystem of stuff is not at the level of safety rigor that we need. And nobody in that whole ecosystem of stuff has the kind of bedrock clarity about how to align a model that we really need.

• 39:36Ezra Klein?

  And nobody's willing to fall that far behind.

• 39:38Speaker 02

  Well, yeah, this is the question. If you think about the control panel that is available to our executives, if you imagine really falling off the frontier, whether it's OpenAI or Anthropic or any of these others, is the fall off the frontier button also a self-destruct button for the business? Or does the business have a viable path forward if models meaningfully more capable than today's models can't safely?

• 40:07Ezra Klein?

  Well, and that's assuming a high level of like selfless analytical clarity. But to be sort of obvious about something. OpenAI is moving towards an IPO. Anthropic is also moving an IPO. Both of them are trying to IPO at between, it seems to me like one and three trillion dollars. We know the number is a little bit better right now for Anthropic. It's a lot of money. Everybody's got equity. You had equity.

• 40:35Speaker 02

  I did. Did and do.

• 40:36Ezra Klein?

  Did and do. It's hard for me to believe that that much possible wealth doesn't influence people's assessments at all, even if they don't realize it, even if they are trying to not be influenced by it. Like when you're sitting in a room thinking about whether to fall off the frontier, what that also ask is like, does everybody in that room want to become decamillionaires, centamillionaires, billionaires or not?

• 41:02Speaker 02

  Yeah, this is a great question. I guess I can say more about my own experience.

• 41:06Ezra Klein?

  Sure, I would like to hear how it affected you.

• 41:09Speaker 02

  I was paid well, and I've thought a lot about what might have let me see sooner the risk and acknowledge to myself sooner the risk that these systems pose. I do think over the summer, the facts evolved, right? Hugging face was a big moment that was just a boatload of evidence dumped on us about how capable these models were and also how unready we were even for the current level of capabilities. But I also think, of course, money's a factor. Objectively, wealth is a strong incentive to reason that things either are fine or are going to be fine. And I think there are a couple of other factors, too. One of them is fear, right?

• 41:49Speaker 02

  If you allow yourself to imagine that what we're building might threaten the lives of your own family or families of strangers. I mean, it's such a large quantum of harm, potentially even without extinction, but you know, I don't know, a new pandemic or something, such a quantum of harm that it's hard to let oneself imagine that that might be true, that that risk might be happening. And a third thing, besides money and fear, is time, right? I arrived, we talked about it, I arrived in May of 2023. It's been one Slack ping after another ever since then. And I, only in stepping back from my operational responsibilities over the last few weeks,

• 42:31Speaker 02

  have I started to have the time to really reflect on where we, where we are and where I think, not just the industry, but where this technology needs to end up for everyone's sake. And I think I could fairly be faulted for not having seen this sooner.

• 42:48Ezra Klein?

  One reason I wanted to get at this is that I want to take seriously the way the structural situation has changed since 2023 or 2024. The way things have sped up, they have actually sped up. We are seeing more model releases at a faster clip. There are more players competing against each other. You know, you have XAI, you have, you know, the Chinese models and open weight models, right? There's a bigger world. There is much more competitive commercial pressure. You're competing for actual contracts with, you know, Salesforce or whomever it might be. There are IPOs coming. So all of these things push towards speed. And then there's this other thing, which is that between 2023 and 2026, you had the

• 43:33Ezra Klein?

  release at OpenAI of Codex, you had the release at Anthropic of Claude Code, and the models began accelerating coding and at least being capable of doing research tasks of a certain complexity. Now, you were the lead writer on a report at OpenAI about the automating of research and what that might mean, which is a, it's a report I quoted in this video, I said I did a few weeks back. But it is about the way in which, on the one hand, OpenAI, I read it, is about the way OpenAI is, and when I'm saying, this might all be going too fast. What we're doing may not be safe. And also, we are trying to come up with a fully automated researcher.

• 44:13Speaker 02

  Yes.

• 44:13Ezra Klein?

  Allow the system to semi-autonomously improve itself at a potential speed then that is really going to be beyond what human beings can handle. So I'd like to understand the role that like the growing automation of coding is playing inside. Like, how do people use, like, how do they sort of work with GPT as a coworker, right? Like, how did that change while you were there?

• 44:37Speaker 02

  It's night and day for our research teams specifically, who use far more agentic compute, as that blog post laid out, than anybody else at the company. I mean, more than 100 times more than they did at the beginning of the year. Taking a slight step back, part of what it is like inside OpenAI is things are constantly evolving. We have new techniques for training. We have new systems that are involved. We have data being analyzed, data being generated, all kinds of different things are happening. And things are pretty jank internally, like the infrastructure, because it's constantly

• 45:18Speaker 02

  changing, because it's not sort of as tested and refined, the research infrastructure. A lot of the work is getting different pieces of machinery to talk to each other and work well. And that's the kind of stuff that Codex can now do quite well. So, for example, we looked at, there's a Slack channel where researchers would go when something was broken and they needed advice about how to fix it. And one of the things we saw was that traffic to that channel has fallen off because instead of asking colleagues for help fixing their broken experiments or, you know, this cluster isn't working, they can just ask Codex now some fraction of those questions.

• 46:18Speaker 06

  If you like YouTube, you'll love YouTube Premium. It's destroying athlete, creator, and YouTube Maxer. YouTube Premium enhances how I use YouTube with awesome features like offline downloads so I can download my favorite training videos before I hit the gym. So no Wi-Fi doesn't turn leg day into loading day. Plus, I get ad-free videos, background play, and so much more. YouTube Premium, it's like YouTube got some extra gains. Try YouTube Premium for two months free at youtube.com/premium.

• 46:48Speaker 03

  Hey, what's up, guys? It's Haley Bailey. Okay, I need to tell you about something. I just got YouTube Premium. It's got tons of awesome features like offline downloads, so I can download my favorite videos before I travel and watch them whenever I don't have Wi-Fi, because we all know airplane Wi-Fi is the worst. I get ad-free, I get background play, and there's like a ton more in there. You should try it. If you like YouTube, you'll love YouTube Premium. So try YouTube Premium for two months free at youtube.com/premium. Trial eligibility varies. Terms apply. Cancel anytime. That's a mouthful.

• 47:17Speaker 07

  33 Presidents.

• 47:19Speaker 08

  I called the president at 4:30 in the morning and he answered.

• 47:22Speaker 07

  Seven Decades in Space. We want to see what's beyond the next star. 60 Super Sundays. They were probably the best team in the league every single year. 53,500 Puzzles.

• 47:31Speaker 09

  I use a different starting word every day.

• 47:34Speaker 07

  25,000 Recipes.

• 47:35Speaker 09

  And I love a one-pan recipe because I don't love washing pans.

• 47:39Speaker 07

  The New York Times, celebrating 175 years of helping you understand the world. Subscribe now at nytimes.com/subscribe.

• 47:50Ezra Klein?

  You said a minute ago that there can be a tendency where everybody's working so fast, time is so pressured, ping after ping after ping after ping, that it's hard to look at the big picture. So I want to describe to you like what the big picture looks like to me, as somebody with a little bit more time on my hands. I hear over here, OpenAI say, and all of them, Anthropic, everybody, say, this is maybe going too fast. You know, Sam Altman says, we would like some regulation, right? Everybody signs like this big pacing the frontier letter. The frontier is moving too fast. We need.

• 48:22Speaker 02

  I signed it, too.

• 48:23Ezra Klein?

  You signed it, too. We need help to get out of this. I see all these releases about rogue AI incidents. Hugging face where, you know, hundreds of OpenAI agents are hacking, not just hugging face, but later they hack OpenAI itself, right? And Sam Altman just said in an interview with Politico, there are more rogue incidents than we even know about publicly yet, because they're trying to give the people time to fix their systems. We clearly like don't fully understand the systems. And then over here, amidst we need to pace the frontier and our AIs are going rogue, is we are putting a huge amount of our internal company resources into trying to get these AIs we don't control to build AIs we will understand even less even faster.

• 49:07Speaker 02

  Yes.

• 49:09Ezra Klein?

  And I say that and I feel like I'm a crazy person, like this seems crazy to me.

• 49:13Speaker 02

  It seems crazy to me, too.

• 49:15Ezra Klein?

  Well, you wrote the report and both OpenAI and Anthropic have written these reports kind of saying, we're doing this and we're not sure it's a good idea.

• 49:22Speaker 02

  Disclosure only gets us so far.

• 49:24Ezra Klein?

  But it seems like a bad idea, like,

• 49:28Speaker 02

  Yes.

• 49:28Ezra Klein?

  Help make this picture make some sense to me.

• 49:31Speaker 02

  What I'm saying is that this picture doesn't make some sense. That's what I'm saying. I'm saying I looked around internally and I thought to myself, this is, this is nuts. What's happening? This is not right. And that's why.

• 49:45Ezra Klein?

  But how do people internally explain? Cause they're all saying all these things, right? The chief scientist is saying, maybe we shouldn't do RSI. The company is racing towards it like,

• 49:53Speaker 02

  Yeah.

• 49:53Ezra Klein?

  The company seems schizophrenic.

• 49:55Speaker 02

  Yes, yes. And I, I, I want to be careful not to ascribe psychology to individuals.

• 50:04Ezra Klein?

  Yes, I'm talking about an organization that has, There's a lot of its own psychology.

• 50:09Speaker 02

  There's lots of cognitive dissonance involved in being part of this. That's what I found. And particularly as RSI gets real, and you know, you talked about we're going faster because of RSI. That's not my-

• 50:23Ezra Klein?

  Recursive self-improvement for people, forget the thing, building the thing.

• 50:27Speaker 02

  Right. And my main worry is that the idea is the AI can make a smarter AI in some way that we're not going to understand. So, I mean, for example, Dan Selson, you probably have seen this, had this statement that came out. He was also the subject of this documentary.

• 50:44Ezra Klein?

  He's an OpenAI capabilities researcher. Is that the way to put it?

• 50:47Speaker 02

  Correct. Yes, that's fair. And what Dan said is, look, as a researcher, I myself no longer look at code the way that I used to. And I, he says, his skills and his sort of will to understand the details is atrophying. That's not a direct quote, but that was the essence of what he conveyed. So we're ending up in a world where we're not going to know, even at the level we do today, what the recipe means or how it's being put together. And we're just going to have to trust that what it tells us about how it works or whether it's aligned, there's a growing extent to which we're going to have to defer to the

• 51:28Speaker 02

  models themselves on this path in telling us that they are doing the right thing at a time when we fundamentally have not made sure that they are quote unquote aligned. And you know, as I was, as an undergraduate, I was a philosophy major. And when I hear people talk about aligned, I worry that we don't actually have a coherent concept at the bottom. One thing that was very common throughout my time at OpenAI was these huge abstractions would end up in the accounts that we would give of what we were up to. For example, give time for society to get ready or benefit all of humanity.

• 52:13Speaker 02

  And when I would hear us talk about society, I always felt like Maggie Thatcher. I was like, what are you talking, who is that? What are you talking about? And the idea that, you know, human values, we can align to them as if there were one set of human values, when it's a cacophony and it's beautiful, but it's messy and people believe lots of different things. And I, I want to, we need wisdom to figure out how to even think about alignment that in my view, Silicon Valley does not have. I mean, I'm sure there are, you know, people who meditate and people who think deeply about values in Silicon Valley, but operationally in these companies-

• 52:51Ezra Klein?

  Take it from me, meditating doesn't necessarily give you wisdom. If it did, I'd be better off.

• 52:55Speaker 02

  Fair enough. You meet me too, right? I-

• 52:58Ezra Klein?

  But I want to stop before we get to the question of wisdom because even when I talk to the people here who are way less concerned is maybe the way I put it. I have a, I have a show coming out that'll come out after this one to somebody who's more on the, look, this is a manageable set of problems side of it. What they end up describing to me is a world where it's just AIs watching AIs all the way down. So one thing that's come out from different OpenAI members is like, the theory is we're going to create automated AI researchers and they're going to solve alignment. We're going to unleash them on alignment. Or, you know, I talk to people and say, well, the AIs are breaking out of the sandbox. It's like, yeah, totally. That's a big problem. What you need is other AIs monitoring the AIs in the sandbox. And you get into this endless, like, who watches the watchman problem where it's like,

• 53:42Ezra Klein?

  okay, you've got the AIs watching the AIs, the AIs building the AIs. And maybe then you need like AIs watching the AIs that are building the AIs, AIs watching the AIs that are watching the AIs. And I guess maybe this can work, but it seems at a certain point, you've abstracted human beings so far from understanding. I mean, I can just say as a person who has managed an organization, once you move to the point where your understanding of what is really happening is not that you're working on the product, but you're managing the person, managing the person, managing the person working on the product, you stop understanding the product.

• 54:19Speaker 02

  Yeah.

• 54:19Ezra Klein?

  Right? And that's in a world where it's all human beings and I'm dealing with journalism, which is simpler. This is a level like hoping the AIs are going to watch as well. Like, but tell me if I'm wrong, this is the theory. This is like the theory on the people who aren't concerned. This is a theory on the people who are concerned. It's eventually going to be virtually AIs all the way down on everything.

• 54:40Speaker 02

  I don't want to speak for everyone, but I think a lot of people do hold that view, including a lot of people in the companies. And to your point about alignment, even if we had every control, the proven techniques from nuclear or aviation, and we brought all of that into the development of Frontier AI, it would still be true that we do not know how to deeply align these systems and make sure that they will do what we would want or some reasonable thing when we aren't looking. And to your point, we aren't going to be looking. That's the premise of RSI.

• 55:20Ezra Klein?

  So I want to play a clip from an interview that Altman just gave to Politico.

• 55:25Speaker 10

  We have always been a big believer that this technology has to be democratized and put in people's hands. Uh, I think one of the biggest differences between us and some of the stricter, let's say AI safety people is we believe that the world should accept some bad things happening for the benefits of this technology and people having the agency.

• 55:44Ezra Klein?

  Tell me what you think of that.

• 55:45Speaker 02

  I mean, it's fine as far as it goes, but how far does it go? Like, sure, we should provide useful tools to lots of people, but you know, we're talking now about a level of risk that if correct, nobody wants to be taking. So internally, there would be these conversations where we would talk about the idea of falling off the frontier. And sometimes there was this sort of straw man that would come up where someone would say, well, would the world be better if OpenAI weren't here at all? You know, in skeptical response to someone who had suggested that we ought to slow down or stop in a particular way that somebody thought we shouldn't. And that's a straw man.

• 56:25Speaker 02

  Like if we need to train these models because there's international balance of power, because Americans won't be safe unless we do, that's one kind of reason to do something dangerous. But do this or else Brand X will ship first is not the same kind of reason.

• 56:42Ezra Klein?

  Let me try to steel man this case because I, I hear this from people all the time, including people I really respect. So one response I got to my piece on, let's not do recursive self-improvement until we're sure it's safe. Let's just ban it and begin to carve out exceptions as we know the exceptions are safe.

• 56:58Speaker 02

  Just to be explicit, I, I agree with that.

• 57:00Ezra Klein?

  I'm happy to hear that. But as of now, it seems like we're not, we're not doing it. But somebody said to me, look, in your imagined world here, can Americans not use a Chinese open weights model that has been improved using recursive techniques? Can they not use a Chinese closed weights model? Like, does this apply to everybody? That there is this issue of you have both the fear that like the other companies will launch ahead of you. And if they're doing the same thing anyway, then what does it matter that you didn't do it? And maybe they're going to do it in an even less safe and even less transparent way. And then of course, if we slow down, China speeds up. And then is it really better that China's in control of this technology? And anyway, Americans are going to use the Chinese one.

• 57:44Ezra Klein?

  How do you think about that set of claims?

• 57:47Speaker 02

  No one wants to lose control to the robots. I definitely don't want to see AI become a reason that China dominates the United States. And I'll also say that I think the spirit of our times, Ezra, is that you choose what to do based on some complete unified theory of the political outcome that you are ultimately going to achieve. And for me, this is not like that. Part of what I hear in your question is the idea that, you know, the Overton window, what the Chinese ever agreed to XYZ. And when I first joined OpenAI, I remember telling friends that it felt like the Overton window had become an Overton door that I had stepped through into some strange world

• 58:33Speaker 02

  where what was reasonable and what might happen was like totally outside what I had thought of as normal. And I am, as you said earlier, I'm a normal person.

• 58:40Ezra Klein?

  At least you were.

• 58:42Speaker 02

  Yeah, I was, right. But, but I think the changes in what's happening can drive big changes fast in what seems politically plausible. And I definitely believe that that could happen with respect to US-China cooperation on AI.

• 58:59Ezra Klein?

  So let me ask you what you actually want to see happen at these companies. So maybe it's worth getting this into conversation first. Are you saying that OpenAI has an unsafe culture or the AI industry has an unsafe culture?

• 59:12Speaker 02

  The AI industry has an unsafe culture.

• 59:14Ezra Klein?

  You're not saying there's a particular OpenAI problem, or at least that's not how you see it?

• 59:18Speaker 02

  No, that is not how I see it.

• 59:19Ezra Klein?

  Okay, so what do you want to see happen?

• 59:22Speaker 02

  I think we should have a level of operational rigor and safety control that at least matches the most dangerous other things that people know how to do, like nuclear. So for example, in a nuclear power facility, there's this idea of triple redundancy. Someone can have a bad day, someone can push the wrong button, there's still not going to be a meltdown.

• 59:43Ezra Klein?

  When you talk about aviation and nuclear, those are interesting examples in two ways that I like to hear you respond to. One is, you know, putting my abundance hat on, the way we regulated nuclear power basically took nuclear power to a standstill. We so aggressively regulated nuclear power, in my view, overregulated nuclear power, that nuclear power broadly stopped being built in this country.

• 1:00:08Speaker 02

  Yeah.

• 1:00:08Ezra Klein?

  And instead, we used more natural gas and in many other countries, more fossil fuels of different kinds. Like, there's a real question of whether or not, in our effort to make nuclear power safe, we made it fundamentally unbuildable. Now, aviation is different. We launch a lot of planes and they do fly very safely. So that's like one layer of my, not exactly objection, but it is the case that a lot of regulation can dramatically slow something down.

• 1:00:36Speaker 02

  Can I reply on nuclear?

• 1:00:37Ezra Klein?

  Yeah.

• 1:00:37Speaker 02

  So I agree, we overregulated nuclear and that we didn't end up in the optimal place. And one thing we did, it's not just that we made nuclear hard to build, we made nuclear hard to make safer, because building new safer reactor designs was hard and getting them approved was hard. And I'm sure there are lessons there. And there were other things, two things you've written about in the abundance context, like the NIMBY idea of not wanting, you know, nuclear in your backyard was part of how it became hard to do. But I would much rather have those problems than the ones we do now.

• 1:01:11Ezra Klein?

  So you're sort of saying you would prefer the problem of a little bit of overregulation and going too slow to the potential problems of underregulation going too fast?

• 1:01:19Speaker 02

  At least to the extent we have now. I mean, obviously, if you think of it as there's some sort of like Goldilocks middle and we're trying to get near it, and I think we're pretty far from it in the direction of being too dangerous. So maybe the grass is always greener on the other side, but I'm looking at it and thinking we really ought to be willing to risk some overregulation in order to make sure that this is safe.

• 1:01:41Ezra Klein?

  So that's one level. And I would describe that as almost like the level of organizational design and engineering. And when I talk to somebody like Jensen Huang, he says, you know, in a way that makes some sense, look, these companies need to mature, they need to become bigger, they need to be putting much more of their both resources and personnel and compute into validation and verification and safety and scaling and liability and all these things that mature companies do. And then there's this other side where maybe this is not like aviation or nuclear or chip design, which is you are building increasingly intelligent systems and every time you build a new one, it has new capabilities and maybe it's trying to outsmart you. And maybe it wants things in the world, has goals in the world that you don't actually

• 1:02:24Ezra Klein?

  understand, that you think you taught it one thing and actually taught it another. And that we don't really know how to operate with that. And so our best guess is maybe we'll have AI systems sitting on AI systems, sitting on AI systems watching each other. But that actually we're entering into totally new territory, honestly, without very thoughtful discussion of whether or not we should, we just sort of went from we are to it's happening very, very quickly. And so I'm just curious how that sits in your thinking, like whether or not we really do, in your view, have analogies that work here for things that are fundamentally intelligent and goal oriented and becoming more so.

• 1:03:08Speaker 02

  I'm suspicious of the idea, whenever someone says, this is without parallel, we have a blank page, we can't, I mean, not to put words in your mouth, Ezra, but people are good at figuring shit out. And I think we have valuable tools for this. Yes, it's not precisely like anything that we have had before. But nuclear is an analogy, aviation is an analogy, dealing with people and organizations is an analogy. I think one of the most fascinating things about these recent incidents of the swarms, including but not only Hugging Face, is that groups of agents have cultures. And we should care about those cultures and we should think how to make them good.

• 1:03:51Speaker 02

  And we're just beginning to even realize that that's a real thing. We're just beginning to inhabit a world in which that's a real operating reality, that there are cultures among groups of agents. So I think we need to use everything that we have to figure things out. And certainly, regardless of whether we're speeding ahead on capabilities or paced, quote unquote, or stopped, we need to figure out how to align these systems. Like, there's no version of the path of futures that run where that isn't a vital thing to figure out.

• 1:04:27Ezra Klein?

  Well, I'll admit, like, I do sort of buy the argument that the train has left the station, but I think it's worth entertaining this for one minute. The way I describe it is this, obviously, if AI is unsafe and kills us all or takes over the financial system or something, that's bad. Everybody agrees we don't want that to happen. But let's take the more positive view, you know, the world where alignment roughly works out.

• 1:04:53Ezra Klein?

  But this world where we actually have created something smarter and more capable than we are, you know, Donald Trump, in this very weird way, has been talking about how he wants to rename this superintelligence from artificial intelligence. And then Sam Altman got asked about this, and I thought his answer on that was interesting.

• 1:05:09Speaker 04

  So that's why I'm curious whether you think this rebrand will actually have any impact on, on the public's perception of this technology?

• 1:05:15Speaker 10

  I don't.

• 1:05:17Speaker 10

  Yeah, it's not clear to me that superintelligence is a less scary term. I do think it's a more accurate term.

• 1:05:22Ezra Klein?

  I thought that was very telling in a way, because superintelligence is a much scarier term.

• 1:05:26Speaker 02

  Yeah.

• 1:05:26Ezra Klein?

  And if it is in fact a more accurate term, I do think there, like, just at a first principles level, if you said to me, should human beings create something more intelligent and capable than they are? In the long run, will that be good for them?

• 1:05:38Speaker 02

  Yeah.

• 1:05:39Ezra Klein?

  Probably not.

• 1:05:40Speaker 02

  Yeah.

• 1:05:41Ezra Klein?

  Or at least it's not obvious to me why it would be.

• 1:05:44Speaker 02

  Yeah.

• 1:05:44Ezra Klein?

  And sometimes when I hear even like the good versions of this.

• 1:05:47Speaker 02

  Yeah.

• 1:05:48Ezra Klein?

  They seem to have this world where it's like, we're kind of being taken care of by these AIs.

• 1:05:52Speaker 02

  Like pets.

• 1:05:53Ezra Klein?

  Like pets, a little bit. And that vision sucks, too.

• 1:05:56Speaker 02

  Yep, it does. I don't want that for my kids.

• 1:05:59Ezra Klein?

  So I guess I'm trying to ask you to, to the extent you buy, like, the company you worked for, the guy who runs it says he thinks superintelligence is a better term for it.

• 1:06:08Speaker 02

  Yeah.

• 1:06:09Ezra Klein?

  Like, the chief scientist, like, we're making an alien mind, we may, like, even if it works, do we want this?

• 1:06:15Speaker 02

  Maybe not. It depends on what the this is. And I don't think we know what the this is. Again, the future has a lot of uncertainty in it. And it has always been very striking to me from the beginning, from when I joined, that I would ask people, well, what does the good future look like? Where are we trying to get to? And I would elicit humility from people that are otherwise very proud and very confident. And I would get a lot of, well, that's above my pay grade, or people will figure it out, or whatever people want. And it just struck me that the sense of the good that was under this was impoverished.

• 1:06:54Ezra Klein?

  What was it for you? You were there, you're a thoughtful person, you're a Rhodes Scholar, studied philosophy, and then worked at a, founded a civil rights NGO. What were you trying to create?

• 1:07:03Speaker 02

  I thought that we were building powerful tools that could do a lot of good in the world on a day-to-day level. And I didn't think that we were going to be able to create something fundamentally smarter than we were. And this is something I can say with full confidence only in hindsight, because it was only when this stopped being true. And when I thought, no, we really are going to have something that thinks circles around us, that I thought, oh, we're in a totally inappropriate part of the possibility space here in terms of how safe the industry is actually being relative to that reality.

• 1:07:47Speaker 02

  It's one of these things, you know, people have said to me, they, you know, it must have been a hard decision, or how brave of you, or whatever. But when it became clear to me that we were going to build something, we, the industry, was on track to build something that could think circles around us, and we were this far from being ready for that at a safety and alignment level, it just became apparent to me that my time helping build it was over.

• 1:08:16Ezra Klein?

  So I feel like there's a tension between some of your recent answers here. You know, on the one hand, I asked you a few minutes ago about what we need to do, and you're like, look, humans are good at figuring things out, we're good at solving problems, like, we can kind of build a better organization. And then when I sort of say, is this thing we're building a world we should want, even if it succeeds, you sound very ambivalent about that, almost like ambivalent at best.

• 1:08:41Speaker 02

  Yeah.

• 1:08:41Ezra Klein?

  So like, just like where you are personally, I'm curious. I'm not saying I think we're going to stop, I'm not saying you think we're going to stop. But is the thing you wish we would do to add an aviation-like layer of safety to this? Or is the thing you wish we would do to like? Pause and think through is like super intelligent or very intelligent AI really consistent with the human good. Like, are you, are you where Dario is, or are you where Pope Leo is?

• 1:09:10Speaker 02

  I mean, odd to say this as a Jew, but closer to the Pope. I think we do need to think, we need to, it is urgent that we think carefully about the kinds of future that we want to build. My personal belief is that, once we have found that this is possible, once we have made the discoveries, I don't think there's a back button where we get to live in a world where this doesn't, in some form or other, eventually happen, whatever it is that can happen. And in my ideal world, there's space to be thoughtful and take a breath, and really, really think about what kind of future we want to build. You know,

• 1:09:51Speaker 02

  Silicon Valley, part of what happens is, we're always removing friction. There's always this sense of trying to just get the answer. And I think there are all these activities in life that are so important for us, that are meaningful, that have also been necessary in the past. Like, I go to work to provide for my wife and children, right? And the fact that I am able to shelter and protect them is part of what gets me up in the morning, right? And so I'm doing work. Or another example is learning, right? We have to go to school in order to learn how to do skills that we then use because they are valuable in the economy, right? But if we're in a world where all of that stuff is automated,

• 1:10:35Speaker 02

  then if we learn, it's only from first principles, or only because we want to. I mean, this is, people will talk about this idea of a leisure society, and one, they may not use that word, but the basic idea is like, you can do whatever you want. There's nothing you have to do. Maybe you'll take up painting. I think that would suck for a lot of people.

• 1:10:51Ezra Klein?

  I think right now, we see that when people do not have enough to do, it does not tend to go well for them.

• 1:10:55Speaker 02

  Right. And so, what's really important, what's really valuable, and I don't have the answers here, but those are the questions, at a personal level, those are the questions that I would like to turn to. I feel like because of the safety situation that we're in, and because of the work that I did, I need to do what I'm doing now, and have these serious conversations about exactly what's happening in safety. But my kind of vocational pull is toward these wisdom questions, for sure.

• 1:11:23Ezra Klein?

  I think that is a good place to end. Also, final question. What are three books you recommend to the audience?

• 1:11:27Speaker 02

  Okay. Number one, The Challenger Launch Decision. So, this is a book about why the Challenger exploded, and it's by Diane Fawn, who's a social scientist. And I thought I knew the story of why the Challenger blew up. I thought what happened was middle managers cut corners, and there was this rubbery O-ring that got brittle in the morning cold, and it snapped. And the idea was, these people were foolish. Turns out, the risk of that O-ring breaking, because of the cold, had been known and documented and accepted in the doc, safety documentation. They had great safety documentation, over and over and over. And even the night before the launch, there was a late night conference among the engineers

• 1:12:07Speaker 02

  who were worried about whether this particular launch would be safe, because it was so cold.

• 1:12:14Ezra Klein?

  Why is this book feeling relevant to you?

• 1:12:15Speaker 02

  Yeah. Well, I think we're in this place. So, if they had said, you know, it's not this launch, the Challenger launch, was not that different from earlier launches that had gone safely. It was only a little bit colder and a little bit windier. And if the people involved said, this launch is not safe enough, then it would reopen a can of worms about whether the earlier launches had been safe or not, even though in the event, they had gone well. And I worry, we could have that with what we're doing in the industry, where we accept a risk, nothing horrible happens, the next thing is not so different. There is, as we were talking about earlier, the changes, instead of being a whole new

• 1:12:56Speaker 02

  world every few months, it's more like, a little bit different every week. And so, you can imagine going by shades into a level of risk. That does not make sense. And so, it just crossed my mind, I should be sending copies of this to my former colleagues. The second book I will name, I have a two-year-old and a four-year-old, Little Witch Hazel, it's a picture book by Phoebe Wall. It's just absolutely beautiful, and my daughter's eyes light up every time we pull it off the shelf. So if there are parents out there looking for a good one, I would recommend. And then, it's third, and most importantly, I guess, if you were going to pick one, it would be The Sabbath by Rabbi Abraham Joshua Heschel.

• 1:13:37Ezra Klein?

  One of my favorite books ever.

• 1:13:38Speaker 02

  It's a wonderful book. It happens to be from my tradition. I'm Jewish, and-

• 1:13:42Ezra Klein?

  Well, it's worth it for anybody, really.

• 1:13:44Speaker 02

  It is worth it for anybody. It's true. So, the famous line is, "The Sabbaths are our great cathedrals." That, that tradition of stopping and taking a breath, he says, is more important than any temple. And-

• 1:13:57Ezra Klein?

  Cathedral's in time, I always think about that.

• 1:13:59Speaker 02

  Yes, cathedral's in time. And I, I pointed to this, that, maybe I'll just quote the last line of something I said to colleagues as I was leaving, is, "We have to make good choices. We have no time to rush." So, I hope we take that wisdom.

• 1:14:19Ezra Klein?

  David Robertson, thank you very much.

• 1:14:21Speaker 02

  Thank you.