Freaked Out? We Really Can Prepare for A.I.

OpenAI last week released its most powerful language model yet: GPT-4, which vastly outperforms its predecessor, GPT-3.5, on a variety of tasks. GPT-4 can pass the bar exam in the 90th percentile, while the previous model struggled around in the 10th percentile. GPT-4 scored in the 88th percentile on the LSAT, up from GPT-3.5’s 40th percentile. And on the advanced sommelier theory test, GPT-4 performed better than 77 percent of test-takers. (It’s predecessor hovered around 46 percent.) These are stunning results — not just what the model can do, but the rapid pace of progress. And Open AI’s ChatGPT and other chat bots are just one example of what recent A. I. systems can achieve. Kelsey Piper is a senior writer at Vox, where she’s been ahead of the curve covering advanced A. I., its world-changing possibilities, and the people creating it. Her work is informed by her deep knowledge of the handful of companies that arguably have the most influence over the future of A. I. We discuss whether artificial intelligence has coherent “goals” — and whether that matters; whether the disasters ahead in A. I. will be small enough to learn from or “truly catastrophic”; the challenge of building “social technology” fast enough to withstand malicious uses of A. I.; whether we should focus on slowing down A. I. progress — and the specific oversight and regulation that could help us do it; why Piper is more optimistic this year that regulators can be “on the ball’ with A. I.; how competition between the U. S. and China shapes A. I. policy; and more. This episode contains strong language. Mentioned: “The Man of Your Dreams” by Sangeeta Singh-Kurtz “The Case for Taking A. I. Seriously as a Threat to Humanity” by Kelsey Piper “The Return of the Magicians” by Ross Douthat “Let’s Think About Slowing Down A. I.” by Katja Grace Book Recommendations: The Making of the Atomic Bomb by Richard Rhodes Asterisk Magazine The Silmarillion by J. R. R. Tolkien Thoughts? Guest suggestions? Email us at [email protected]. You can find transcripts (posted midday) and more episodes of “The Ezra Klein Show” at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Book recommendations from all our guests are listed at https://www.nytimes.com/article/ezra-klein-show-book-recs. “The Ezra Klein Show” is produced by Emefa Agawu, Annie Galvin, Jeff Geld, Roge Karma and Kristin Lin. Fact-checking by Michelle Harris and Kate Sinclair. Mixing by Jeff Geld. Original music by Isaac Jones. Audience strategy by Shannon Busta. The executive producer of New York Times Opinion Audio is Annie-Rose Strasser. Special thanks to Carole Sabouraud and Kristina Samulewski.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Ezra Klein?

  I'm Ezra Klein. This is the Ezra Klein Show.

• 0:23Ezra Klein?

  So let me start this one by situating us in time. So we taped this podcast just a few hours after OpenAI released GPT-4, which is their most powerful AI language model yet. The thing that is remarkable here, that I really keep trying to push people to key into, is how quickly these systems are getting better, how steep the curve of improvement is. We talk about some benchmarks here on the show, but GPT-4 now passes a bar exam in the 90th percentile, the 90th percentile. Just sit with that. We were nowhere near fully understanding what the last generation of these models could do. Now we're on to the next one. And the same is probably going to be true here. GPT-4, it is extremely powerful.

• 1:08Ezra Klein?

  It is at times unnerving. We barely understand what it can do or how to use it. And yet, everybody is racing ahead. GPT-4 also is going to be a toy, a diversion, compared to what we're going to have in three or five years. If you heard the commentary I released on AI on the podcast over the weekend, you heard that what I'm most focused on is how society manages to adapt to the speed at which these systems are improving, or else slows them down so we have more time to adapt, to understand them, to think about how we want to integrate them. And one piece of feedback I got in that commentary was, what are you really saying we should do, aside from be anxious? And I don't just want people to be anxious. So this conversation gets more into specifics, both on what these

• 1:52Ezra Klein?

  systems can do now, why they could be so disruptive, where they seem to be going, and what we can and I think should do, or at least the kinds of things we should do in response. Kelsey Piper is a senior writer at Vox, where she's been doing great coverage for years on AI and the technologies and the culture and the safety and economic questions around it. She joins me now. As always, my email, [email protected].

• 2:22Ezra Klein?

  Kelsey Piper, welcome to the show.

• 2:24Speaker 01

  Thanks. I'm glad to be here.

• 2:27Ezra Klein?

  So one of the reasons I wanted to have you on here is you've done a lot of reporting. You spend a lot of time with folks inside the AI world. So you're a little bit more in this hothouse than a lot of us. And when you talk to them over the past couple of months, what do they say? What are they seeing or what are they thinking about that from the outside would not be obvious?

• 2:49Speaker 01

  Yeah, so I think not just in the last six months, since like at least 2019, you had people saying, this is going to overnight, you know, not today, but in 10 years overnight transform our world. We are going to be able to automate any work that can be done remotely. We are going to be able to bring us the next 100 years of drug discovery and development, you know, in the space of maybe a couple of months, maybe a single research project. We have this extremely powerful thing we're on the brink of. And in 2019, when you heard that, I think you had to be a little skeptical, like this is Silicon Valley. We're not new to the hype cycle. Sure, you're going to change the world. And now I think you're still hearing the same things, mostly from the same people, but you have to take them a little more seriously.

• 3:31Speaker 01

  They are, in fact, releasing things that can do things that everybody thought were impossible. They are developing the newest models as fast as they can. OpenAI says, we think the way for humanity to like do well on AI is to raise a head as fast as possible. This is in a statement a couple of weeks ago. This morning, GPT-4, you know, only a couple of months behind ChatGPT, which hit 100 million users like overnight. It's not even that they're saying new things. It's that it's harder not to take them seriously.

• 4:00Ezra Klein?

  So when you say overnight, this idea that sometime in the future, all of a sudden, the history of AI will speed up, that we're making progress fast, but it's, you know, a couple of years, a couple of months between major new systems. And then you get these ideas of quantum leaps really quickly. What are people talking about there? What are they looking at?

• 4:21Speaker 01

  So I think the most concrete version of this is that right now, all of the work done on improving AI systems is being done by humans, and it's a pretty small number of humans. And of course, one of the jobs that they are trying to automate is their job, the job of making better AI systems. And when they succeeded that, if they succeeded that, then you can, instead of having hundreds of people working on building better AI systems, you can have thousands, you can have a million, because you can just make a copy of an AI system. That's like, it's fundamental economic potential is that you can make a copy of it. You can make an AI system that is as good at writing as you, and then you can have a thousand of them writing about all different topics. You can make an AI system that's a really good programmer,

• 5:03Speaker 01

  and then you can have all of the programmers working with or possibly replaced by that thing. So things are already moving fast. And I don't think the case for like, taking AI really seriously crucially depends on they'll start moving even faster. Like, the current pace is fast enough to me to be disastrous. But if you ask, like, what's my gut instinct about what's gonna happen? It's that at the point where we figure out how to make AI researchers out of AIs, then things will move real fast.

• 5:32Ezra Klein?

  And that sounds very sci-fi, as a lot of things do, because we live, we live in a sci-fi world increasingly. But one of the things that I know is striking to a lot of people in this world was when GPT-3 really proved able to code, even though it hadn't really been trained to do so. And so then, as people began moving more towards optimizing AI systems to create code, and you expect we'll do even more of that, train in on even more code, get even better at that. Once AIs can code, which they can already do, when you just assume for the improvement curve, then the idea that you would have an AI that codes AI, that's not a giant leap, that's simply finishing the sentence in a slightly different way.

• 6:14Speaker 01

  Yeah, I guess a lot of it is just there is so much commercial incentive to make an AI system that can code. Programmers are really expensive, they're scarce, it's hard to hire them. If you have an AI system that can do most of what they can do, companies would be shelling out huge amounts of money for it. And that means companies are shelling out huge amounts of money on developing it. And this is just also true, more true of the specific skill of like, building large AI systems. So I don't think you need this things will accelerate premise necessarily to take AI really seriously. But my best guess is that there are people right now trying to develop AI systems that can develop AI systems. Because there's people trying to develop AI systems that can do every human job.

• 6:54Ezra Klein?

  Well, let's talk a bit about how fast it's going now. So this is a little bit by coincidence, but the morning we're talking, OpenAI released GPT-4. And they did not, according to them, train GPT-4 specifically to pass any tests. But like GPT-3 and 3.5, it absorbs a lot of language from the internet. It begins to create statistical correlations, for lack of a more precise way of saying it, between terms. And out of that emerges a kind of thing that seems to act in the real world like understanding. So they released this table where they showed the performance of GPT-3.5 and GPT-4 on a battery of tests that human beings find pretty hard.

• 7:36Ezra Klein?

  So when you try to give GPT-3.5 the bar exam, it is in the 10th percentile of people. When you give it to GPT-4, it's in the 90th percentile. GPT-4 passes the LSAT at the 88th percentile. GPT-3.5 is at the 40th percentile. I found this one kind of funny, but also interesting that it passes the advanced sommelier theory test, that's for wine, in the 77th percentile. GPT-3.5 is in the 46th percentile. So these are fairly general tests of knowledge and reasoning. And obviously, there's information about them out there. I mean, these have essay questions, et cetera. Now we're in an area where the improvement is very fast and you can actually match it against human beings on things that I would not have said five or 10 years ago.

• 8:19Ezra Klein?

  Oh, that's just a structured data quiz. There's an old argument that we say artificial intelligence is anything that a system can't do yet, and machine learning is anything it can already do. But how do you think a bit about this debate over truly intelligent versus stochastic parrots, as people like to call it, versus sort of a very advanced autocomplete?

• 8:39Speaker 01

  So it sort of calls into question what intelligence is, right? Like, what is going on in our brains when we think? I think there is a sense in which it's just a glorified autocorrect. But I think that's also like a sense in which most of the time we are also just a glorified autocorrect. Like what humans do is we read lots of things, we talk to lots of people, and then we put it together into our own sort of framing that articulates something that we wanted to articulate. And it turns out that most of that, you can just feed something, the entire internet, and then make it really, really good at pattern recognition and then prompt it and it does that. And I guess you can say, well, that's not intelligence. But if you're going to declare, well, that's not intelligence.

• 9:22Speaker 01

  I think you are kind of in a position where you're also declaring that about humans. And I don't know, what do you even get out of saying humans aren't really intelligent? We still did build civilization. We still did build nuclear bombs, you know? If we're not intelligent, then I'm not sure intelligence is the thing we care about here.

• 9:37Ezra Klein?

  Well, what people say to give credit to this view is that they don't understand. Like, we wouldn't call it calculator intelligent, even though it gets the right answer if you try to multiply 37 by 25. And frankly, most human beings can't do 37 by 25 in their mind. And so there's something about the quality of understanding, of knowing why you're saying what you're saying. And I kind of agree that on some level, these things don't know why they're saying what they're saying. But on the other hand, I think one of the demos that keeps coming up that unnerves me the most is when these systems get asked to explain why a weird joke is funny. And then they do.

• 10:16Speaker 01

  Mm hmm.

• 10:17Ezra Klein?

  That the difference between knowing why you're saying what you're saying and being able to give me a count of why you're saying what you're saying that is convincing to me seems like it's getting very thin.

• 10:27Speaker 01

  Yeah, like, maybe you can say, well, it could tell you a human who said the thing I just said would probably have been having this thought process. And if we want, we can say, but that's not its thought process. Its thought process is this incomprehensible alien thing with no resemblance to the human thought process it just gave you. I think that's true. I don't find it very reharing, right? Like, whatever it's doing may be different from the account it can give, but it does have the skill to produce the account still. And I don't know that we can say that that incomprehensible thing that it's doing isn't really reasoning.

• 11:02Ezra Klein?

  I think that gets to one of the things that I actually find the most unnerving, which is that the more inhuman the statistical correlation process happening at the center of these systems becomes, the more human the experience of interacting with the system becomes. And that fundamental loss of legibility between we have some mental model. When I ask you a question, I have a rough, not a perfect mental model, what's happening as you think about the answer. But as we make these systems more complex and they have more billions of connections and they have all these layers and parameters and nodes and, you know, if you tried to get it to print out what it's doing, when you ask a wide joke is funny,

• 11:44Ezra Klein?

  you'd get like a billion calculations that are happening that mean nothing to you. And so when I think forward 15 or 20 years, I think that's what unnerves me, like a, like a complete sort of loss of human comprehension of a lot of the interactions we're having and decisions being made around us. And not just loss of our comprehension, but our capacity to have comprehension over it.

• 12:06Speaker 01

  This is some of the more scary and sort of depressing visions of the future you see from like Holden Karnofsky or from Ajay Kauta or like a world where

• 12:15Ezra Klein?

  Who are AI researchers?

• 12:16Speaker 01

  They are, yeah, AI researchers at the Open Philanthropy Project. They're thinking about the future and where this is taking us. And a description they gave that really stuck with me was decisions by companies about where to build new factories, about what products to invest in, about what kind of R and D to do are made by AIs. The AIs explain their answers to us perfectly. They write beautiful memos, but we can no longer tell if the memos that they're writing and the decisions that they're making are towards goals that we have, towards things that we want, or if we've built some self-sustaining inhuman thing that is, you know, sort of doing its own thing now. Because we can ask and it will give us a beautiful answer and we just have no way anymore to tell if that beautiful answer is right or wrong.

• 13:01Ezra Klein?

  And is that, I mean, one thing you could say, because you're getting here at a problem of deception.

• 13:05Speaker 01

  Mm hmm.

• 13:06Ezra Klein?

  Right. When you say that, I think there are, or maybe there are a couple of versions of it. One version of it that I think you're pointing towards is we tell an AI to do something. We have misspecified what that thing is. And so we think it's doing what we want it to do, and it's actually doing what it wants to do and trying to trick us along the way. But could he say that's true for human beings? Aren't, I mean, in businesses, aren't people constantly putting a factory somewhere? And actually, it turns out they got a kickback. It wasn't because they really wanted the factory there for economic reasons.

• 13:34Speaker 01

  Yeah, I think that some degree of this thing has its own goals is like accounted for in the system. We are used to working with people who have their own goals. Nothing really relies on everybody being totally loyal to us because that doesn't work. I do think that the more alien something is, the less that works. Because some of that works on laws and anti bribery rules and oversight. But some of it works on the fact that like, very few people are going to build a factory that like, pumps chemicals into the atmosphere that poison everybody alive. They just wouldn't do that. And an AI might do that if it had a reason to do that. You can't just count on the fact that it's like, probably basically a reasonable person and not an homicidal maniac, the way you can with a human.

• 14:18Speaker 01

  Because humans are all from the same share of values, of priorities, of kinds of thoughts. Like you said, you can sort of guess what's going on inside my head. And even if you're getting similar behavior out of something that's radically alien, you don't know that its scope of action is the same, that it won't do things that you would never think of doing.

• 14:38Ezra Klein?

  So I want to go back to the pace of improvement. And I really, this is one of those conversations where I don't want to start shouting into the microphone. But the thing I really want to emphasize is that I'm not sure most of us have any sense of how exponential the curve of improvement both is and is becoming. So when you talk to people working on these systems. How would you describe what they think will be true about them that isn't now in five years? And how do you describe that in 20, noting that neither of those values are all that long in the scale of human history?

• 15:12Speaker 01

  Yeah, these are like before our kids grow up tier things. So I think that a lot of people anticipate we are not very far from systems that can do any job you can do remotely, anything you can do on your computer. So coding, like secretarial work, receptionist work, journalism, everything like that. I think that some of those same people would say, we can invent better scientists and do science a lot faster and come up with, you know, whatever discoveries humanity would have plotted along to over the next hundred years, whatever those we would have learned from studies and research, getting it very, very fast from AI systems that can do that work sort of overnight or in the space of

• 15:54Speaker 01

  days, just running much, much faster. And then I think some of those people are like, and then we get, you know, a fascinating, cool Star Trek future where material scarcity is over, no one needs to work, no one has to work, and all of the goods of all of this AI work they've done are distributed to all humankind. And some people are like, if we did a poor job of managing this, of regulating this, of having oversight of this process, that is where, you know, we die or we get some other pretty catastrophic outcomes from effectively unleashing a civilization of aliens that we don't have a good way to work with or trade with or understand.

• 16:36Ezra Klein?

  So I want to back up a little bit from the sort of utopia dystopia, just because like honestly, I think you lose people here. And I don't even know what I think of those because first on the scientific one, I know there's this big thing among the AI community that it's going to Create a scientific super explosion. But most science that I can think of requires you to run a bunch of experiments in the real world.

• 16:58Speaker 01

  Yep.

• 16:59Ezra Klein?

  And I'm not sure exactly how much faster an AI really makes it to laboriously do phase three trials of vaccines on like human beings, or, I mean, I can imagine ways it could speed things up, but I do wonder a little bit about the potential for an overestimation of analytical intelligence. But let me go back to something you were saying a minute ago, because I think it's closer at hand. When you say people believe, you know, five, 10 years, I think this is a fair description. Many people in this sector believe that within five to 10 years, you will have systems that can do any work you can do remotely that is not in the physical world.

• 17:33Speaker 01

  Yes.

• 17:35Ezra Klein?

  At least as well as like the median person doing that work right now. That's a pretty profound, both, I think we think about that as a possible economic change. It's also a kind of metaphysical change for humanity. Tell me just a bit about why people believe that to be true.

• 17:53Speaker 01

  So you don't need to get all that much better than GPT-4, plus some careful training and some careful shaping of the models that we have and some additional work on reliability and stuff like that to get something, you know, it can pass the LSAT. It can also do a lot of what lawyers tell me is the sort of menial first few years of law work of putting contracts together and reviewing them. It can program. It can program quite well. It is not all that much improvement required before it can program better than the average person you might otherwise hire to program. It can write. I think my writing is better than it's, but I think my writing is better than it's by like a much smaller margin than a year ago or than two years ago.

• 18:37Speaker 01

  And so at some point you have to be like, probably next year it will be better than that writing than me, or if not next year, probably in a couple of years. And I think some places, probably the New York Times will still employ human journalists, but I think a lot of places have already sort of tried to see if they can get away with automating huge amounts of work that used to be done by humans. Art too, used to be that humans would draw art. And now for many, many of the purposes that we used to hire humans for, AI art is where it's at. I think that you're going to see a ton of that. Think about this with like weaving. When we invented the automated textile loom, some people still bought like handwoven rugs because they really liked the, you know, high end, handwoven thing. And you wouldn't say that automated rugs are like better than handwoven rugs in every way.

• 19:22Speaker 01

  But almost every rug in the world is an automated rug and almost all rug work is done by machines. And I don't think very many people want to be like a niche cottage industry for people who prefer human labor, while nearly all of our labor is better done by AI.

• 19:39Ezra Klein?

  To think about writing for a minute, because a lot of jobs rely on writing. It's true for journalists, true for lawyers, true for social media marketers, true for people who do publicity, true for just a huge amount of, of the modern, you know, what gets called the knowledge economy. And one sort of metaphor that I've thought about in my own head for thinking about it in journalism is that you're going to move from the highly valued skill set, the probably more valued skill set being that of the writer to being that of the editor, the person who looks over a piece of work, tries to think about, is this true or not? Did this miss something obvious? Does the structure of this argument make sense? But just putting aside like what you think of those skill sets, just a truth that anybody who's in a newsroom knows is that there is a small proportion

• 20:24Ezra Klein?

  of editors to writers. An editor will often supervise somewhere between, depending on the organization, four and eight or nine writers. And that's true, I think, in a lot of areas. Like you go to social media marketing, which I know a little bit about. You will have a number of junior marketers who are overseen by one more senior person. You go to law, you have a number of younger lawyers who are overseen by a partner, et cetera. And on the one hand, so you have a skill shift towards management, but also oversight, just by its nature, you are overseeing a number of. Agents, for lack of a better term here. And as you were saying earlier, the key thing here is these things are cheap to copy.

• 21:06Speaker 01

  Yes.

• 21:07Ezra Klein?

  And so just when you think about just that, right? If it's just true, they're like, okay, editors, like, we're not getting rid of all skills in writing. We just like, we need more editors and fewer writers. Well, we only need fewer editors and we need writers. So it gets really quickly into land that will be pretty disruptive for people. And even if for some reason or somehow we pass laws or just turns out that it hallucinates a little bit too often to make this possible to scale people want to imagine. I still think it's going to shake a lot of people to feel all of a sudden that skills they spent their whole life generating may not matter that much. I mean, I think it's unnerving.

• 21:44Speaker 01

  Yeah.

• 21:44Ezra Klein?

  When I put something in, I'm like, write about polarization in the style of Ezra Klein. It's not there yet, but it is what you would get if you had a ninth grader who'd read me for a while, write about polarization in the style of Ezra Klein. And if it gets to where like me five years ago, it's going to shake me a little bit.

• 22:01Speaker 01

  Yeah, and we never know. It could all peter out next year, but my bet is that that point where it's you five years ago is not actually all that far away because it's a ninth grader now and it was the sixth grader a year ago and it was the fourth grader, you know, a year before that. And I think that just the magnitude of the economic disruptions, like call centers do a ton of customer service work. A lot of that is probably going to be automated in the next few years. That's like a ton of high paying jobs in poorer countries where people speak English, lost in places where those people don't necessarily have nearly as good alternative jobs. And maybe some of these like massive collapses of various industries will be replaced by new industries created by AI.

• 22:47Speaker 01

  But I don't think there's like a guarantee in general economics that your industry being wiped out doesn't just leave you notably worse off. It can just do that. If you had specialized skills and then they're no longer valuable, it can be that you just have lower earnings for the rest of your life.

• 23:03Ezra Klein?

  It typically does mean that. I mean, we know this from the economic literature. People sometimes pretend that economies adjust over time, but when somebody's factory gets closed, they do not typically get rehired for a higher wage. This is not how it goes.

• 23:16Speaker 01

  Yeah.

• 23:17Ezra Klein?

  I think that something, we were talking about this before the podcast started and I want to bring it in here because I think it's emotionally important. I was saying a minute ago that I've read for years, the, the arguments about whether AI kills us all, and I just, even if I, I can follow their logic, it is very hard for me to emotionally connect to them. Something in the past couple of months, the thing that has kind of messed with my head though, was thinking about my own children. And not whether there would be a world for them to grow up in, but just what kind of world, how different it might be from mine. Because as I was looking at this, like, I think for people who have seniority in the

• 23:59Ezra Klein?

  world, there are a lot of reasons AI is probably not going to come for you immediately. And it will for some people, obviously. And I do think that even if the technology moves quickly, societies adapt more slowly, there are regulations and rules and oversight and so on. But it just seems, if you look at this economically, if you look at just the possibility of people being surrounded by inorganic, non-human things that feel like intelligence is to them, sort of the world of the movie Her, which I was rewatching recently, and I cannot recommend watching it enough at this moment. But there's a company called Replica that is explicitly trying to do this in a great piece in New York Magazine recently about people falling in love with their replicants. Those are going to get way better. I tried that out. My replica was super boring. Maybe it'd get better over

• 24:43Ezra Klein?

  time. But it won't be boring for long. If you read the sort of Kevin Roos transcript with Sidney Bing, whatever, like, that was, whatever else you want to say about it, that was an interesting, like, diverting conversation. I just think we're on the, like, for my kids, like, in my kids' not just life, but like, growing up, period, we're on the cusp of a world that for them may be profoundly different than the world I was prepared to raise them for. And something about that has been the crucial line I stepped over, where I went from being like, this is interesting and I like to read about it, to holy shit.

• 25:18Speaker 01

  Yeah. No, I think that there's, there is a sea change and it is going to mean that they grow up in a world that in some ways we can barely recognize. And I think I don't know, I am in favor of progress and change and technology and I mostly like Silicon Valley. I have all these friends here who have all these crazy hyped startups and I like, I appreciate that worldview in many ways. But I do feel this sense of ideally you have the frontiers pushed forward. We see some gaps that it's leaving, some awful things that are resulting. We adjust, we regulate, we figure out where the problems are. You know, we invented a banking system and that was great.

• 26:00Speaker 01

  We realized the banking system had all these catastrophic failure modes and we try and learn how to patch them and how to manage them and how to keep failures manageable. We invent nuclear weapons and that's terrifying, but we also over time develop institutions surrounding nonproliferation and norms and stuff like that. We came closer to a nuclear war in the early period after nukes were developed than recently. And kind of like if we have enough time, we can make almost anything that happens to us a good thing. And the problem is if, if we have enough time, I don't think that that process can magically happen overnight.

• 26:37Ezra Klein?

  Let's talk about why this might not happen. So what do you think is the best argument for why this hits a wall in your term? That we get to something like GPT-5, it's a bit better than what we have now. But we've seen, I mean, driverless cars were supposed to be all over by now and you can drive around and see little Waymo cars, but they've all got people in them. And they've not been able to get to the reliability they really needed. We didn't replace all the radiologists, like that was supposed to happen. IBM Watson's got functionally sold off for parts. A bunch of these predictions have come over the past, it's called 15 years. And they didn't quite pan out and there was often a moment where people thought it was inevitable and then inevitability turned to disappointment, at least over this time period.

• 27:20Ezra Klein?

  So if we plateau, not far from where we are now, why do you think that will have been?

• 27:25Speaker 01

  Yeah, so my pessimist voice goes, okay, these systems are trained on all of the data on the internet. And they are not going to be any smarter or any better or more discerning than everything on the internet. There's some arguments that they could be because you need like higher quality to replicate it than just is necessarily present in it. But you could certainly say, no, they're just kind of going to be as good at making stuff as everything on the internet. And the average quality of everything on the internet is not outrageously high. It's not like super impressive. It's good at like writing things that seem to flow for a minute. But there's often, and I'm not talking about like AIs here, there's often not a lot of like underlying thought there. There's often not a lot of accounting for complexity.

• 28:11Speaker 01

  So maybe the best thing you get out of training in really large language model on the internet is something that can in like tiny fragments imitate a human who thought about it, but without any of the underlying cognition. And so maybe that means that those things never actually, they can pass a standardized test. They can pass the standardized test amazingly well, we have to rethink standardized tests. They can write a college essay, but that really reflects the fact that college students are not putting actual cognition and thought into their essays. They can imitate maybe the sort of low level journalism that's just write this event that happened, but they can't imitate synthesis, they can't say anything original or distinctive. You could totally imagine that, it would not be very surprising. And does that change the world?

• 28:54Speaker 01

  I don't know, I feel like it makes the world weirder, but it doesn't bring in some of these scenarios that are really transformative, I think. If it turns out that there is some degree of actual cognition that doesn't go into most human utterances, but is important for human work to be of economic value, then maybe it's not there. But when I say that, then I'm like, I sound like all the people who are like, it needs reflectivity. It needs the ability to consider counterfactuals, it's not going to have those. And every time I've heard someone say, oh, yeah, yeah, you can't just feed it the internet and get reflectivity. You can't just feed it the internet and get logic, you can't read it the internet and get coding ability. It turns out you can feed it the internet and get those things. So I guess what I say to my pessimist voice is like, yeah, maybe.

• 29:40Ezra Klein?

  I find that view very compelling, though. And I mean, I've had Gary Marcus on the show who holds, I think, a version of that. And I probably am like 30% where he is. But the place where I maybe feel like I differ from people who hold that view is that I think making something as good as the internet could be more important. And again, to what you were saying, I mean, they go weirder. Like, I really, really focus in on the word weird here. Like, my big call on AI last year was all about the word weird. My recent significant call on AI was all about the word weird. I think that can make something very weird. I think it makes something economically quite important. If you could make something as good, I mean, if you can do a good job. Because one thing about the the big models is like GPT-4 isn't trained to be a lawyer. It's just trained on the internet.

• 30:25Ezra Klein?

  But if you do the fine tuning in the area of making it a really good lawyer, it's going to be way better than GPT-4, even at its current level of capability at being a lawyer. So I could see how you can make things as good as a pretty good lawyer, as good as a pretty good journalist, as good as a... It is a little hard for me to understand how you get this superhuman intelligence training something on human data, both because it's functionally, I think, at some level averaging out human data. And also because, and this does go to the point where understanding might be important. It doesn't really have a great way, it seems to me, of separating what it has read that is true and false. You know, I think about something like the masking debate, you know, about whether or not masks work for COVID.

• 31:09Ezra Klein?

  And one thing within that is it, if you just inhaled every study on masking, putting without any really profound priors about how to think about any of that, I don't know what you would get. But it isn't obvious to me the mechanism by which you would get something way better than what we have. Because you wouldn't really have the capacity to run a new study yourself. If we could just run all the randomized controlled trials of masking we wanted, and maybe we should be doing that, we would know a lot more. And also, like, how would you know what studies to believe? And you do you just end up like in hawk to whatever people have done the most of? Like, I actually find that view very persuasive. So I'm curious what you don't find persuasive in it.

• 31:50Speaker 01

  So I guess that might happen. I feel so much uncertainty about AI. And a lot of what I feel is that if you lay out anything kind of plausible sounding, I'm like, yeah, that might happen. I like, I don't feel like we can rule that out. I also don't feel like we can rule out the crazy extinction scenarios. I feel like... Ruling any of those out would require a better grasp on what it is we are doing than we have. So I'm like, that might happen. But my gut is that it won't. And if I try and dig into why not, I'm like, to imitate what we would say in a situation, you could say that takes less intelligence than it took us to say it. But I think in some cases, it might take more. Like, you have to, in some ways, be able to model a person in

• 32:35Speaker 01

  order to come up with what that person would say. And there are people who have, you know, lived their lives on the internet for the next 25 years. And you would think that just being fed that person would give you like, a pretty good internal picture. And like, to build a generator of everything Ezra Klein has to say, just a perfect predictor, something that just writes what you have to say, something probably has to have whatever processes you have, or processes that successfully imitate whatever processes you have. And to the extent that your processes are truth tracking, to the extent that the thing you're doing involves like, noticing which sources seem reliable and reasoning in your head, like, step by step, how would masks work? How would a mask study fail to tell us about masks?

• 33:20Speaker 01

  You can already get GPT-4 to do that. Like, it will give better analysis of mask studies if you tell it, reason step by step, how good is this study? What are some things that might confound this study? And so I'm kind of like, maybe it can do that prompted, but it can't put it together. And it can't like, use its synthesis of our reasoning to do its reasoning, even just to imitate us. But it probably can, it probably just, in order to write something as good as an Ezra Klein column, which is something we are like, trying to train it to do, it will probably have to pick up like, a lot of the actual underlying process, and not just the output of that process.

• 34:00Ezra Klein?

  This is one of the things I find most, again, just weird, interacting with the systems, but it's now a highly replicated result. If you say, tell GPT-4, or GPT-3.5, which is what I actually know, give me the answer to this, as if you were really smart.

• 34:21Ezra Klein?

  It will give you a smarter answer.

• 34:22Speaker 01

  Yep.

• 34:22Ezra Klein?

  You mentioned reason step by step, right? There are a bunch of ways you can do a prompt, right? You've done versions of this, like, tell me this as if you have an evil other side and you're trying to hide it from me, which is to say that if you give it the instruction to model a kind of person, it can model that person. Again, not really trained to do that at any level. I mean, you can ask it to write in the style of me and it will, which was, I read it and it was a little embarrassing, I thought. It did catch some things like, yeah, I do, I do do that. A lot of M dashes. And that's weird to me. And it's funny, it's something that ends up happening in her, which comes from 2013. I mean, it's a minute ago, but the AIs themselves begin creating models of people from the

• 35:06Ezra Klein?

  past they want to be able to interact with. So just one thing that could just make the future weirder, even before you get to ideas of superhuman intelligence or anything else, anyone with a really profound written corpus that you can feed tens of thousands, hundreds of thousands of their words into a learning program that's already been trained on the whole internet and then tell it to particularly seem like this person, you might just be able to give people all kinds of fairly capable, mimicked, human beings from the past and present. I don't exactly know what to say about that, but it seems strange to me. And then if you do that, then and like set them loose on physics or whatever, I don't know, they can get way beyond where humans are, but it can probably get way beyond where

• 35:50Ezra Klein?

  most humans are, which is already enough, I think, to destabilize quite a bit of our sense of ourselves ontologically.

• 35:58Speaker 01

  Yeah, even in that sort of world where they don't get any smarter than us, there is just still some really radical, terrifying transformations from the ability to create human intelligences sort of at will and customize them for almost any purpose that normal humans can do and run them for as long as you want and stop running them when you want. You could imagine like every student having specialized teachers designed to work with that student customized to have like the ideal personality for working with them. You could imagine a lot of us have AI friends or AI lovers who are, you know, more attentive and don't have any needs of their own. And you could imagine in some sector that outcompeting normal human friendship.

• 36:40Speaker 01

  Personal assistants are obviously a big area where most of us can't afford a personal assistant, but would love to be able to say like, Hey, can you schedule a doctor's appointment for me? Or have someone who proactively says, hey, you haven't been to the doctor in three years. You should really get on that, you know? And I think all of that is coming, even if none of the stuff about building on that for greater intelligence happens at all.

• 37:05Ezra Klein?

  When I talk to the people working on these systems, I'll often ask, what can you make it do that I can't? Because I think something people don't always realize is that even the demos we're being given are highly constrained in what they can do. And something I've heard from a few sources is that you can already tell these systems to accomplish something in the real world that you can do through the internet, but that has multi steps, like, you know, it needs to figure out how to rent a car and do things like that. And it can, it can come up with that plan and execute something in the real world. And sometimes it will come up with a plan that is not the kind you would think of, or not the kind you would want it to come up with. But it knows enough. It's not just that it can give you an answer, but it can execute an action. And that's already, but we don't know it.

• 37:50Ezra Klein?

  I mean, this is actually one of the things that worries me the most really about policy debate and sort of where we are in this, that even before you take into account the steepness of the improvement curve, what they already have is quite a bit beyond what we know. We only really have access to OpenAI and OpenAI derivatives like Bing. We don't have access to what Google has built, the public. We don't really have access to what Meta has built. We don't really know a bunch of the systems being built in China. And we don't have access in any of these cases to completely open systems where they haven't been highly constrained for our use. And here I'm talking in we, the public, or any kind of sort of mass signup.

• 38:31Ezra Klein?

  The one moment I think anybody got access to a system that wasn't highly well constrained, which was like the Bing-Sydney debacle, it really occasioned a moment of reflection and in many cases, I think, fear. And that's just something going on here in the background that I just like don't quite know how to convey, but we don't even know what we're dealing with now exactly.

• 38:52Speaker 01

  And there's a lot that's not yet released. Even though places are doing a pretty fast release cycle, there's just inevitably part of an engineering process is that you're ahead internally of where you're going to go externally. So I actually think GPT-4, which is now, you know, out, is pretty close to the state of the art. I don't think there's like tons of more sophisticated models anyone has. It's hard to get side by side comparisons, but where I have been able to get them, GPT-4 is very good. But what there is, is like, applications that are pretty obvious, that people have set up and have working. Yeah, stuff like an integration of ChatGPT or GPT-4 with TaskRabbit or something like that, so that it can just send instructions,

• 39:36Speaker 01

  pay a person to follow those instructions and get stuff done in the physical world. Totally already done. Similar with like voice, well, Google had this years ago and kind of backed off of it. But voice that sounds like a person and can call and make an appointment for you and sound totally natural on the phone and is an AI assistant making appointments for you. So when people are like, so if AI is wanting to do bad things, it's a good thing they're stuck on computers, right? And it's like, I mean, we built them tons and tons of tools with which they can directly interact with the physical world in many, many ways. And we're going to keep doing that.

• 40:34Ezra Klein?

  Tell me about what people call the alignment problem.

• 40:37Speaker 01

  So we haven't yet had to grapple all that much with the fact that we don't know if these systems have goals and we don't know if they want things. And we don't know if their actions are like trying to accomplish things. And we don't super need to know that with existing systems. Because like if you prompt GPT-4 to write an article in the style of Ezra Klein or whatever, then it writes the article in the style of Ezra Klein. And, you know, it's not doing the kind of open-ended

• 41:09Speaker 01

  Tasks where it might matter what its goals are and it probably doesn't have goals. I don't like making any certain statements about AIs, but it probably doesn't have goals. But when we build these systems, we want them to have certain goals. We want them to be helpful, not say racist things, not embarrass the companies that release them. We want them to be truthful. We want them to do what we prompted them to do. We don't know how to tell them any of that. We're, we're just like doing it all with reinforcement. Like whatever it does something, we can say good or bad. And maybe it's some more gradations in there, like how good or how bad, but that's what we've got.

• 41:49Speaker 01

  We're trying to use that very blunt lever of like reinforcement to point it at our goals. And depending exactly how you do this, there's a bunch of ways that what you teach it to do is probably not what you actually want it to do. Right? Like you want it to be truthful, but you might be wrong in some case. And so you reinforce it poorly for a true answer because you thought that answer was false. And so it learns, oh, don't be truthful with the humans. Figure out what the humans believe and tell them what they want to hear. Right? Which is different. You were trying to teach it truth, but you were going from your own understanding. And it might notice that if it's has a little personality, if it's sympathetic, if it's nice,

• 42:31Speaker 01

  or for that matter, if it's a little bit sassy, that gets better reactions than if it's sort of bland and boring. So maybe it develops and displays a bit of a personality because it gets reinforced. And that's what it thinks we want to know. And you can get a fair bit of deceptive behavior this way, right? If the thing that we are trying to teach isn't the thing that we actually want, and it is incentivized to give us what we want to see rather than like actually be on our side or whatever.

• 43:01Ezra Klein?

  How much of that is already happening? Probably that question about deceptive behavior.

• 43:04Speaker 01

  Good question. So the problem is that it's hard to tell, right? We know that the models say false things all the time. We know that they say false things when they know what the truth is. We, we can like easily, you know, I can just go to chat GPT, get it to say something false, and then ask in a different context and it'll be like, yeah, that answer was false. We know that. Is that deception? Probably not. Probably it wasn't like trying to trick me. It was just, you know, giving the answer it thought I wanted. But it makes it very hard to tell when there is intentional deception with intent of like accomplishing some goal that the AI has. And this is extra hard when we don't know if the AIs have goals.

• 43:45Speaker 01

  And some people will say they clearly don't have goals because how would something like that have goals? I'm just kind of like, I don't know, evolution was just like selecting repeatedly on ability to have babies. And here we are, we have goals. Like, how, why does that process get you things that have goals? I don't know, they just showed up along the way. Large language models are just like selecting repeatedly on like token prediction and then reinforcement. Does that have goals? I think if anybody tells you like yes or no, they are overstating what we know and what we can know.

• 44:17Ezra Klein?

  One of the questions here that I find interesting is this question of coherence. Can you talk a bit about that?

• 44:23Speaker 01

  The thing I think of when I think about coherence is to what extent does it make sense to say that I have goals? I like have lots of wants, some of them contradict each other. I want to get ready for my interview and I also want to lie in bed another 10 minutes. I want to stay up late playing video games and I also want to be a functional adult who doesn't stay up late playing video games, you know? And humans act coherent and goal directed in some contexts, like we have priorities and we will predictably do the things that accomplish our priorities. But in other contexts, we don't. And you could imagine that if AI systems have goals, that they could be like that. We don't know if they have goals. We don't know how to tell what their goals are.

• 45:08Speaker 01

  We kind of know how to make them work towards our goals, but only when the work is something that we can verify and we're eventually going to want work from them that we can't verify or like can't verify at the relevant scale. And that does dramatically change the safety landscape. But I don't know, if we knew that evolved aliens were coming to Earth in the next 10 to 20 years and we knew that being evolved beings like us, they were incoherent and they had lots of competing goals and impulses. And, you know, some of those were to turn Earth into resources that they could use to build more factories, but some were to keep us in zoos and some were to like leave us in nature preserve and some were to like invite us to participate in their geopolitics and some were to ally with us against these other aliens they're in a war with or whatever.

• 45:52Speaker 01

  I'd be scared. I wouldn't be sure we would all die, but I would be nervous. That would feel like a bad situation, even though they're incoherent, just like we are. And even though they might in principle be willing to work with us.

• 46:06Ezra Klein?

  One of the things you talk about in one of your pieces is what people call specification gaming. If you imagine an AI that, however, we built the alignment, we can actually communicate with, and it can kind of understand us, and it does care about trying to figure out what we want. I think that's one world, but we're going to have a lot of things that we really can't communicate with on that level. And in the near term, specification gaming seems really worrisome, particularly if AI has become a big part of hiring processes. Or, you know, as we've already seen, predictive policing. And so talk a bit about specification gaming.

• 46:42Speaker 01

  So say you do give your AI an objective that you're trying to have it maximize, which we do sometimes when we're like building game playing AIs or whatever. Or yeah, if you're doing a hiring process, you might train an AI where your training is aimed at the maximum share of the resumes that the AI passes through in fact result in a job offer. Like, we want this AI to be really good at filtering and correctly predicting who gets a job offer or whatever. And AI systems will be very creative sometimes. That's what some of the examples of specification gaming we have demonstrate. At satisfying that constraint we gave them in a really out of the box way we would never have thought of. Like, if you're having them play a game and they win by getting the most points, they will do weird stuff, like find a glitch in the game.

• 47:24Speaker 01

  They will, you know, repeatedly race in circles forever because they can get more points that way. They will like trigger some kind of bug in the code that just causes them to end up with a really big number. That happened once. I find that kind of freaky. The AI just like figured out how to like take some random action that caused the number to be really big. And we don't really like know how it noticed that. It may have like stumbled on it randomly. Who knows? So we are building these AI systems and giving them these narrow objectives. And I don't think this is how the world ends or anything, but it's certainly a dynamic that, yeah, we're facing all the time where if... The hiring manager who looks at the resumes is racist.

• 48:04Speaker 01

  Then the AI will maximize its success probability by like never passing along a resume of the wrong race, because then that one would be less likely to get hired. And that's like an alignment failure with society. The AI is like enabling a bad actor to do bad things. But there's also ones where the AI is like not working for anyone. It's just accomplishing its objective in a way that would be unacceptable cost wise to everybody. But we don't have adequate oversight to notice that that's what it's doing. Like if you had a stock trading AI, there are so many horrifying things that you could get up to that like modify stock prices, right? You can trigger wars to modify stock prices. You can hack your competitors to modify stock prices.

• 48:47Speaker 01

  You can trigger financial collapses. And so I do think that even though this isn't the apocalypse scenario or whatever, whenever you're putting these like creative, deeply inhuman algorithms in charge of making some metric go up, you would hope we would be sensible enough to make really sure that that was under lots of oversight and that it didn't have the power to do things in weird ways. And that's not what's happening because that's never what's happening in AI. You can say like, oh, well, that sounds bad, but it wouldn't be a problem if we just took these like three very reasonable measures to counter it. But we're not taking those three reasonable measures, like ever. There is always someone who is just doing the naive foolish version of the thing. And as the systems get more powerful, the failures get more powerful.

• 49:32Ezra Klein?

  I actually think the stock trading example is something weirdly that I worry about a lot. Because if you think about what sector of society right now has functionally unlimited money to pump in building the absolute best or licensing the absolute best AI systems that you can, well, hedge funds, other kinds of quant trading operations. Then if you think, well, also separately, what segment of American or of global commerce is most used to using algorithms it itself does not understand completely recklessly in an effort to get even the most narrow edge over competitor? You'd say, again, high speed and algorithmic trading. And I mean, I could kind of keep going down like if you might say like,

• 50:17Ezra Klein?

  what's a good industry that's really good at wielding political influence? So people don't get in its way because it's really good at hiring excellent lobbyists to high speed. But what is an industry where if you actually had a really, really effective algorithm that people didn't care that much about oversight because they were trying to get it to maximize a particular metric, in this case, returns, might be able to cause a lot of damage to a lot of people? You'd say, oh, yeah, something around high speed algorithmic trading.

• 50:44Speaker 01

  Yes.

• 50:45Ezra Klein?

  And I don't exactly know what we're planning to do about that, except as you say, nothing, because like the current theory of what we're going to do here is nothing, I think.

• 50:52Speaker 01

  Yes.

• 50:53Ezra Klein?

  Which is mostly what we've done with high speed algorithmic trading so far. That's a little bit unfair, but but I think that has gone way beyond where there's any social benefit to having algorithms competing with each other to do things nobody understands, to exploit small differences in prices at a given moment. But that's just a near term thing that seems very concerning to me. You don't need to have any kind of particularly far term views about like how AI will develop to just think a lot of damage could happen by unleashing powerful systems nobody understands in a place where they're trying to justify something that people really want maximized and may not think, as they often don't think through, the consequences of maximizing it.

• 51:34Speaker 01

  Yep, I think there almost certainly will be disasters in this space, and I just hope that they're at a scale we can learn from instead of at a truly catastrophic scale. And I also think there's a general principle here of we are creating these alien intelligences that are creative, that think of problem solving solutions we didn't think of, that do weird things that no human in their position would ever do because it's a route to their goal. Like people who play against them in chess or go talk about this, how sometimes the algorithm will make a move that just no human would ever make, and they don't see until 20 steps down the line how brilliant it was. And I worry that in the world of try and make this company's stock price go up, that

• 52:18Speaker 01

  tendency towards like genuine creativity, yeah, will take the form of, oh, I bet that if I like cause an outage at this power plant, then stocks will move in this predictable way that I can make a lot of money off of. And if you don't have oversight, stuff like that happens.

• 52:35Ezra Klein?

  Or if you don't even know how to have oversight.

• 52:37Speaker 01

  Because you can't see what it's thinking because it's not thinking in any way we understand.

• 52:42Ezra Klein?

  This gets to something within the culture of this community that I want to talk about. You cite, and I've cited in pieces too, this really unnerving survey of machine learning researchers from last summer, from 2022. It asked people basically what they thought the chances were that working on AI, if they got to high level machine intelligence, the consequences for humanity would be really, really, really bad, like human extinction bad, or some other complete displacement of humanity. And the median response was they thought there's a 10% chance of that happening. The people working on this said, yeah, 10% this thing that I am in some way or another trying to bring into being, or trying to worry about how it comes into being.

• 53:25Ezra Klein?

  I think 10% chance humanity is completely pushed off the board in a fundamental way. And you could say this was just a survey, but I've talked to enough of the people at high levels in these companies to know that that survey is not that weird.

• 53:40Speaker 01

  Yes, my impression also is that that survey, it might be something of an overstatement, but the position there is not rare. There are plenty of people who will tell you, oh yeah, I think this is it.

• 53:52Ezra Klein?

  I don't know exactly how to communicate to people, but I've spent enough time in this community, and you've spent more than me, that these systems are being designed by people who themselves do not believe they really know what they're going to do. It brought out that my column was talking about it as a kind of summoning, and I've also had that thought that this is like people opening a portal, and they will freely tell you that like they hope a demon doesn't crawl through, but they can't guarantee a demon doesn't crawl through. And even if you don't believe they're right, like you believe this is crazy, or maybe it's hype, they believe this.

• 54:23Speaker 01

  Yes.

• 54:24Ezra Klein?

  And that actually worries me a lot, because you talk about value alignments. I think a lot of people might say, if the thing you're working on has a 10% chance of destroying humanity, don't, or go really, really slow. Some people do believe that, but there is like a lot of now competitive pressure to speed up, to be the first one. There's pressure between US and China, so geopolitical pressure. You know, Google does not want to lose to Microsoft, which does not want to lose to Meta. Can you just talk a bit about how you understand the culture here? And like the psychology of working continuously to bring into existence something that you know you don't understand and where you think the consequences have a really, really high

• 55:08Ezra Klein?

  chance of being unacceptably bad?

• 55:10Speaker 01

  Yeah, so I want to second what you were just saying about how difficult this is to convey, because you have these conversations and you get this like somewhat terrifying sense of that people understand themselves to be making extraordinarily high stakes decisions, decisions that I think the average person on the street would say that should absolutely be the prerogative of a government with a lot of careful consultation, maybe some kind of international framework. And you have people who are like, we're gonna do it, we're gonna do it ourselves, we thought about it and we think it's gonna go best if we do it ourselves and we do it now. And it's like, very hard to step outside of that bubble and simultaneously say, these risks seem real and not sci-fi and scary.

• 55:54Speaker 01

  And also, you know, whatever you think about that, the people who are doing this do think that there's something real here. So there's like, why are they doing it? And I think some of it's just the love of science, right? Like, there's, I've been reading The Making of the Atomic Bomb and a recurring like theme-

• 56:12Ezra Klein?

  Oh yeah, why are you reading that?

• 56:14Speaker 01

  I, you know, I have two young kids at home and another one on the way. I never have time to read and so I was way behind on my book list. And then I got the invite to come on your show and I was like, this is my excuse to read all of the books that I'd been wanting to read, but now it's for work, I have to read them, so I have something to recommend on Ezra's podcast. And that is how I finally got some time to read The Making of the Atomic Bomb. And one of the recurring things in it is, it's a very personality driven compared to, I guess, what I was imagining before I picked it up, like, who were these men and what were they thinking? And it does feel like- There's a kind of person who just, once they realize it can be done, there's an overwhelming urge to do it.

• 56:59Speaker 01

  There's just a, it, it, one of the points that the author makes at the beginning of the book is that we might dream of a world where physicists said, on principle, we will not build an atom bomb. But we were never close to that world. Physicists would build an atom bomb as soon as it occurred to them that you could. You know, and there were some very brave and very notable people who tried saying like, this seems really bad, what should we do about this? But overall, if you want to stop dangerous technologies, that has to be on a structural level with like governance and regulation and international agreements and voluntary between lab agreements. You can't count on people, unfortunately, to just say, this is ludicrously dangerous, we're not going to do it.

• 57:40Speaker 01

  Because I guess some share of people are just like, this is ludicrously dangerous and I'm going to do it. I guess this is like the classic tale of hubris in some ways. Like the, the oldest known flaw in human nature is you, you see the tree of knowledge and you're like, yeah, I'm, I'm going for it.

• 57:54Ezra Klein?

  Well, it's kind of a selection effect too.

• 57:56Speaker 01

  Yes.

• 57:57Ezra Klein?

  Which is to say that if you were somebody who thought this was such a bad idea, you definitely wouldn't do it.

• 58:01Speaker 01

  And you don't do it.

• 58:03Ezra Klein?

  She's not the person doing it. And there's a lot of incentive for people to believe the arguments that they should do it.

• 58:11Speaker 01

  Yes.

• 58:11Ezra Klein?

  Right? And I think this is something, and we should talk about it, that there are a lot of people who work in AI safety and who are around this community and spend all their time thinking about reasons maybe not to do it, or reasons to try to make it, you know, slow it down, or how to make it go more safely. But in the end, they're not the ones who run the companies, who get the investment that gives them the compute power to do it.

• 58:35Speaker 01

  There are so many people, very good machine learning researchers, who I know, who nobly made the decision to not play this game because they thought they might kill us all. And so they are out of this game and they are, you know, working in academia on much smaller models or working on regulatory frameworks, they're doing good work. But they're not the people at these labs because they said no, and they walked away. And so we have the people who, for whatever reasons, didn't walk away.

• 59:01Ezra Klein?

  And so I think within those, you can cut a couple of groups, right? So Yan Le Cun, who's an important AI pioneer, and he's one of the key people at Meta, I believe. He's their chief AI scientist. He just thinks this is all dumb.

• 59:13Speaker 01

  Yes.

• 59:13Ezra Klein?

  And that AI has no interest in taking over or killing us all or whatever. So it's just not going to. And this has all gotten very far fetched. Then you have people who run a couple of the other labs, maybe Sam Altman at OpenAI would fit into this framework. Folks over at Anthropic in a different way, I think are more careful than OpenAI, but who believe that AI could do great good and do believe that there's a chance of really bad things happening. And think that by working on the models and experimenting on them, that you make it both likely or you understand the bad things so you can prevent them. And also maybe trigger society coming in and saying, okay, here's how we're going to regulate this early enough. And I don't think it's a crazy view.

• 59:53Ezra Klein?

  I mean, I do think that there are more conversations happening about how to think about and regulate AI because OpenAI has been releasing models into public use that Google and Meta are not. So I don't think it's a crazy view, but they are all now in a race. And I worry that the race dynamic is just a lot stronger than the caution dynamics. And this is something that I know you've been writing about and thinking about. So can you talk a bit about this question of the race? Because something I know has been worried about in AI for a long time, but I think it's undeniable now that it's happened.

• 1:00:26Speaker 01

  Yes, so there's a couple of different things that drive getting new powerful models, right? One is how much money tech companies are willing to sink in and how much talent they have working on it. And race dynamics often mean that these companies are willing to dump a ton more money on this because they want to stay ahead of their competitors. You know, Microsoft is willing to spend billions and billions of dollars for a shot at catching Google in search. Google's presumably willing to spend billions and billions of dollars defending its spot in search for the same reason.

• 1:00:58Speaker 01

  And so you get bigger models faster. And that's like one thing you can analyze. Like, maybe it's good to have these bigger models faster under these race conditions, because then we're here talking about them. We probably wouldn't be here talking about them if this was still all under the hood, because it would be a little too crazy. But the other thing is that when you're going fast, safety becomes a liability. And things that you could do to make your models less dangerous, or to check more thoroughly if your models are dangerous, work you could do on like mitigating risks, and even just understanding them goes out the window, because you're trying to beat someone else, and you want it in commercial production as fast as possible. We would be way safer against AI takeover scenarios, really scary catastrophes.

• 1:01:42Speaker 01

  If AI systems didn't have open access to the Internet and the ability to contact anybody they wanted and do anything they wanted on the Internet. But we've given them that. Of course, we've given them that, because it was like slightly more profitable than not giving them that. And so as soon as you open that door for everybody to be competing with each other, then the people who are careful and cautious are slower than the people who are careless and incautious. And the people who are the most willing to throw safety considerations out the window are the ones who are getting their biggest models out fast. And that puts us in a situation where even if it turns out to be totally possible to make AI systems safe, like there's a very straightforward technical solution to alignment. We know what exactly what we need to do. We still have a problem,

• 1:02:27Speaker 01

  because somebody can just decide, yeah, that would be a little expensive. That would delay our launch by two weeks. That would require us to hire another engineer. Like, why Bottner?

• 1:03:04Ezra Klein?

  So when I search for any AI app that I might want to check out in the app store, one thing I always notice is that there's the app I was looking for and then there's a bunch of fast follower apps right beneath it. And my sense is that the kind of sub-premium AI models a couple of years ago were just total crap.

• 1:03:28Speaker 01

  Yes.

• 1:03:29Ezra Klein?

  And that now you're looking at things that are not coming from OpenAI or Microsoft or Meta or Google, but they're as good as things were a couple of years ago, which is like now, not that bad. And then in a couple of years, like they're going to be as good as the things are now. And you know, if you are OpenAI, you have a really, really high reason because you have so many awesome ways to make money here.

• 1:03:53Speaker 01

  Yeah.

• 1:03:54Ezra Klein?

  To make sure you're restricting use of your model to only things that are going to get you good press or something like it. But you know, you have AIs that it now seems you can train them on somebody's vocal patterns pretty easily and it can then call my mom and because I've got a lot of my voice out there in the world.

• 1:04:11Speaker 01

  Yeah.

• 1:04:11Ezra Klein?

  You could train that on my podcast and it could call my mom and it could sound like me having been kidnapped and needing a ransom.

• 1:04:16Speaker 01

  This has been done.

• 1:04:18Ezra Klein?

  So, okay, so it's already happening. And these are getting better faster than you actually don't, for things that are just kind of bad, you don't need the best system in the world. You can increasingly just use kind of crappy system and the crappy systems are like the best systems in the world from a couple of years ago. That's just going to keep happening. And that seems again, like something we don't really have a framework for thinking about. We don't really know what to do about. But I wonder if you could talk a bit about that, that the, the opening up, not of the state of the art, but of the sub-state of the art to functionally like every scammer, advertiser, just anybody who wants to make a buck in the world. And the kind of seedy underbelly of AI, in the same way that happened to crypto,

• 1:05:01Ezra Klein?

  although maybe that was like the overbelly or something, because it kind of started there. But the seedy underbelly of AI strikes me as a place where a lot of damage is going to be done really soon, if it's not already being done.

• 1:05:12Speaker 01

  Yeah, and this is again, something where I'm like, if things were moving a little slower, I'd be like, yeah, there's some scary stuff and we'll adapt to it. And people will learn not to trust calls from their kidnapped sons. And, you know, we will get better at spam filters to combat spammers. And we will, you know, get better at making comment sections on the internet functional. I do get the sense in like the last year that spam is ahead of spam detection in terms of like-

• 1:05:38Ezra Klein?

  Really seems to, something seems to have changed.

• 1:05:39Speaker 01

  I feel like we went through this golden period where spam detection was better than spam, and you mostly didn't encounter spam all the time. And it is over tragically. And now I get a lot of spam on Twitter, in my email, on Facebook. And I'm like, I guess we are back in the domain where spam is better than spam detection. And if you have enough time, then I think it's possible for people to develop the social technology and the literal technology to counter that kind of thing. But when everything's moving really fast, you know, people learn how to address one thing. And then we get the AI that can like, carry on a long convincing conversation with your mom and convince her to send her bank details in the course of like, this very long conversation

• 1:06:22Speaker 01

  that also references a bunch of details from your childhood that you mentioned on the podcast. Well, that's harder, you know? How do we get to that point where it's easy to cause so much damage and so expensive to detect? And again, I'm just like, I wish we were moving a little slower. I wish we had more time to like, figure this stuff out.

• 1:06:43Ezra Klein?

  Well, you say that like it's impossible. What's the actual difficulty? I mean, we have made a lot of things slower. You can't clone humans, and we don't, even though it is a possible thing to do. And I'm sure there are scientists who would like to clone humans. We've actually been quite good at maintaining an international ban on cloning human beings. There's all kinds of biological weapons research that we just like, don't really seem to do anymore. We did it for a while and we- We actually used them and then we have kind of nonproliferation packs that I'm not saying they are perfectly effective, but they are effective. And you can come up with a bunch of different things like that. There's certain forms of genetic modification we just don't do. Katja Grace, I thought Heather, who's an AI researcher, had a really good piece on this about how the AI community has a little bit of this weird tendency to say, look, maybe it'd

• 1:07:28Ezra Klein?

  be good if we slowed down, but it's completely impossible to ever coordinate people such that they slow down on anything. And her point, which I think is true, is that that's actually not true. We coordinate people to not do things or slow down on things all the time.

• 1:07:39Speaker 01

  Yes.

• 1:07:40Ezra Klein?

  In some ways, it seems to me the biggest barrier here is actually the people who just want to rush forward on AI are matched by the people who don't think that rushing forward on AI matters. So like, not that much interest in slowing anybody down because what? It's like toys on the internet. But I don't know, I do think there's a problem here where like the AI community seems to have convinced itself that it's so hard to get people to do anything, that you couldn't possibly coordinate them around this. I don't see why that would be true.

• 1:08:10Speaker 01

  Yeah. So I think we can and should slow down. It's like very hard when things are moving this fast to get a good feel for the possible. Like you could imagine a fairly sweeping set of government regulations in the next few years that set a cap on how large you can make a model, unless it is subject to some form of democratic oversight and extensive testing for safety, where we check things like, can this send a convincing phishing email that persuades 1% of users to run code on their computer? If it can, you can't generally release it yet. Can this model like give detailed instructions on how to carry out a terrorist attack that if followed would be successful? If it can, you can't release it yet. There's all kinds of scary

• 1:08:52Speaker 01

  things that are not very far off and we could just draw a card cap and be like, look, if the model in an audit of some kind with dedicated testers whose job is to elicit this behavior, if the model can do the dangerous thing, no, you cannot deploy that model or you're liable if you deploy the model, which maybe amounts to the same thing while we're talking about big companies. We could do that. We could do that tomorrow. I think there's plenty of people in Washington who are thinking about something like this. And I think what's possible, you're so right, that it's a social question. It's a question of what everybody else believes. If everybody wakes up and goes, We want to have more steering power over the world our kids grow up in. We could just do this. There are like very reasonable rules you could put in place

• 1:09:36Speaker 01

  that would significantly change the pace of this work. And I hope that happens. I'm not pessimistic. I've heard from enough people that they want something to change, that they're scared, that they believe it can and should go differently than it's going. That I'm like, that's what it takes and maybe we can make something happen. But I am kind of, the default thing that happens is that the regulators are years behind the ball. You know, you need to do a very dedicated, very concerted effort to set up a system where regulators are on the ball. It's a hard problem. It's a hard problem. I think we should be working on, but it's a hard problem.

• 1:10:17Ezra Klein?

  Tell me a bit more of that list because I do think one thing that makes problems harder is when the suite of possible things you might do about them feels opaque. So a lot of people don't understand that much about it and they really don't understand what you might do to regulate it without completely kneecapping it, right? Are you just saying you can't research language models anymore? I mean, that seems crazy and I think would actually be kind of crazy. So is there a rough consensus among the safety people you talk to about these are the three or four things we really wish would happen?

• 1:10:49Speaker 01

  I would say there is an in progress effort to build that consensus. I think conversations I heard this year that I did not hear last year included like, what are some clear cut benchmarks? Like send a phishing email that persuades 1% of people who receive it to run code on their computer. Like that's a pretty clear, specific, it's obvious to anybody who's thought about policy a little bit, why that's scary behavior. It's pretty easy to use an auditing process to determine whether a model has that capability. You know, you'd have to send some phishing emails, so I guess there's ethics concerns. But I feel like a regulatory framework can work with that kind of thing. And so that's a conversation I didn't hear last year that I am hearing this year that gives me some optimism because I think that's what it would look like

• 1:11:33Speaker 01

  to have regulators be on the ball instead of behind it. I do think that people who are working on this effort, you know, are coming at it from all kinds of different places. Some of them are like, well, what I really want is for all of this to stop. I can't get that. So I will like try and set some benchmarks that have an achievable chance here. Some of them are like, I really want to be doing my amazing AI research, but I want these safety people off my back. So I'll agree to some standards that I can meet, you know, that don't get in my way too much. And a lot of people have like different threat models and different pictures of like which AI capabilities are going to be the scariest ones in the near future or in the medium future. So it's a difficult process to build that consensus. And I think we're still

• 1:12:17Speaker 01

  in the early stages of it.

• 1:12:20Ezra Klein?

  What are a couple of other specifics that you think are interesting? They don't need to be consensus, but you just think they are provocative.

• 1:12:25Speaker 01

  So if an AI system can via emailing instructions to a human assistant, get a copy of itself spun up on Amazon Web Services, independent from its creators, and so that another copy of it can run on AWS. That seems like a benchmark, like you can define that behavior in an audit. You can see whether the model can in fact do it successfully. And if it can, then that's like an indicator of some of the really scary scenarios where models decide to operate independent of human oversight, because that lets them achieve their goals better. So that's maybe a benchmark. If your model has the capability, when prompted, to give a human instructions on how to copy that model on run it on AWS independently,

• 1:13:08Speaker 01

  that's a scary model. We don't want you to widely deploy that model.

• 1:13:11Ezra Klein?

  When you say when prompted here, one interesting thing to me about that example is that I feel like there's a set of behaviors or capabilities where like, we are worried a human will use you this way. Then there's this kind of further off.

• 1:13:24Speaker 01

  Hopefully further off.

• 1:13:25Ezra Klein?

  Hopefully further off idea is that like, we think you might act this way.

• 1:13:28Speaker 01

  Yes.

• 1:13:28Ezra Klein?

  You being the the model. And in that world, if we think the model develops situational awareness of itself and its world, such that it's then being prompted by some idiot human like, hey, do you mind emailing a copy of yourself to the Amazon web servers to replicate? In the world where you think that the AIs become deceptive. How well does that actually work?

• 1:13:49Speaker 01

  Developing this auditing process is very tough, right? Like, it's a very technical, difficult work to figure out. How would we elicit this behavior if it is possible? But I think that in general, it is easier for a model to display a skill when it's elicited by humans than unsolicited. So we will probably successfully identify models that are capable of giving a human instructions on how to copy them to a new AWS server. Sooner than we will get models that can do that and think of hiding it and successfully hide it from us. Now, that's a gamble. Maybe we do our tests and we're like, oh, it's good. These models can't do it. And in fact, these models have passed the threshold where they can do it, but suspect it's not in their interest. But that feels less likely to me.

• 1:14:33Speaker 01

  That like requires the model to develop two skills at the same time, like the deceiving us and the capability to survive and spread itself or whatever. I feel a lot more optimistic if we're consistently checking, hey, can you do this? And then halting things and calling a red flag once it can do this. Because I think that will probably be before it develops the ability to do that deceptively without our knowledge or awareness.

• 1:14:57Ezra Klein?

  So let me make a critique of the AI safety community here a bit, which is, I don't want to say people are too focused on the absolute worst that can happen. That's a good thing for some people to be focused on. But I find that there's an unbelievable lack of focus on what seems to me to be an incredibly important near term question, which is the business models behind these models. And I think in general, the sort of community this comes out of is a little unreflective about capitalism, right? Effective altruists generally and rationalists and so on. And but these models are going to develop really fast.

• 1:15:31Speaker 01

  Yes.

• 1:15:32Ezra Klein?

  Along the pathways that make them a profit.

• 1:15:34Speaker 01

  Yes.

• 1:15:34Ezra Klein?

  Like, at some point, this stuff has to begin turning around money.

• 1:15:37Speaker 01

  Yes.

• 1:15:37Ezra Klein?

  And the models that develop the most are going to turn around the most money. And I think something most of us believe about the internet at this point is it would have been better, all things considered, if surveillance advertising had not become the dominant model for all of the biggest players. I said before on the show, I worry about AIs of this magnitude being hooked up to surveillance advertising. But there's other things I also worry about them being hooked up to. And I think people are really unreflective at this point or under focused on this question of. Regulating how you can make money within machine learning is actually very important for affecting the path of machine learning in the

• 1:16:18Ezra Klein?

  long run. And I'm not sure that I know, in fact, I'm sure I don't know what is safe and what is not safe here. But you know, a world where there's billions of dollars to be made because the US government had set up a series of prize competitions for things like the protein folding problem, which DeepMind was able to solve, you'd get AI systems that could do one thing. And a world where there's billions of dollars to be made and trying to get me to buy things and manipulate my behavior, like you'll get AIs to do another. I really worry about the most obvious profit models being a spur then to create AIs. I mean, if you create AIs and like the main way they make money is manipulating human behavior, you've created the exact kind of AI I think you should fear the most. But that's the most obvious business model for all of these companies. It's

• 1:17:02Ezra Klein?

  what they already do. Whereas like, that's not what, say, Pfizer does. But Pfizer's the one creating the AI. Like, I just, this seems obvious to me, and I can't seem to get anybody to care about it.

• 1:17:13Speaker 01

  No, I'm with you. I guess I would be very happy to see regulation that was aimed at like, how you can make money off of your powerful AI systems to sort of start trying to discourage building manipulative powerful systems. Because I do think manipulative powerful systems are how we get into very serious trouble. So I think there's a ton of blind spots in every AI community. Just because being part of a subculture that the rest of the world didn't take seriously until two months ago is weird and distorting to people. And because the kind of people who cared about AI when it was weird are weird. And because there are some ways in which it's like, a little unhealthy to spend all your time thinking about these futuristic sci-fi scenarios. And it's not that they're so far in the future that they don't merit thinking about.

• 1:17:59Speaker 01

  But I think you have to be very grounded. And I think a lot of people get a little ungrounded. And I think a lot of the stuff you're talking about, you're right, and it's very grounded. It's just like, this will go according to the logic of what makes money, unless we successfully make rules about how you can make money off of it. That seems true. And if you've spent all your time trying to think about this as a technical problem, like, how do you make the AIs not lie? How do you notice if they're lying? Then you maybe just get a little bit of tunnel vision and you don't see other opportunities to substantially curtail scary stuff that are like outside the range of stuff you're thinking about.

• 1:18:40Ezra Klein?

  So there's a kind of dimension of competition between companies that are functioning within a couple of miles of each other, a couple of states of each other. And then I find that if you push on that conversation for a while, the next space of fatalism becomes geopolitical competition.

• 1:18:54Speaker 01

  Yes.

• 1:18:54Ezra Klein?

  That, of course, we have to go as fast as we can because if we don't, China will. And even if somehow you could come to an agreement with China, like you just wait long enough and Russia will, or Qatar will eventually, or somebody, somebody will whose values you don't share. And so that there is, in all of these dimensions of competition, a view that there's such a huge return to being the first to attain AI dominance, that if you simply don't, somebody else will, and by virtue of you being the one who didn't, you will like their values less. I'm curious how you first hear about the geopolitical dimension discussed within this world, and second, what you actually think of it.

• 1:19:33Speaker 01

  Yeah, so I think you describe it completely correctly. People will talk about Microsoft versus Google, and if you're like, but we could regulate, we could slow down, then the thing that comes out is, sure, but then China. I think this is kind of silly. First of all, international cooperation is not impossible. For many technologies, there it is possible, especially ones that are very dangerous. It is possible, especially ones we've well established, what about them is dangerous, to arrange international controls of some kind. To the extent you think AI is something that could be bad for everyone, then I think you should also be a little optimistic that as we develop good benchmarks for identifying AI being bad for everyone, we can also work internationally on it. So if you think that this is the highest stakes thing in the world,

• 1:20:17Speaker 01

  then I think rushing ahead and doing it poorly and doing it without buy-in. And while a lot of people who work closely in a research community with you are saying, you are going to kill us all, you are going to like catastrophically destabilize the world we live in because you're worried that some other country is going to get there first. It just feels to me like you've got to be making some kind of logical mistake there. I come back to the making of the atomic bomb, which, as you said, it's pretty clear why it was on my reading list. Like this process where the terror that Nazi Germany was building a bomb was a big motivator to start the bomb project.

• 1:20:59Speaker 01

  And then when it came out, then not only was Nazi Germany not building a bomb, they'd never been close and there was no chance they would get the bomb. And in fact, we were going to win the war with them without the bomb. One scientist quit, one scientist quit the atomic bomb project when he learned that actually there was no Nazi bomb project and there was no international competition. Nobody else did, partly because, you know, moths to a flame can't stop summoning the demon, the ones they know they can, they have to find out if they will. But partly because I suspect that for a lot of people, like they wanted to find a reason, they found that reason, if that reason wasn't applicable, find a different reason. And also, okay, China's good at espionage, right? Like from a pure boring geopolitical, like, let's leave all the ethics out.

• 1:21:43Speaker 01

  The easiest way for China to get really powerful models is to steal the really powerful models that we are making. And I think it would slow them down a bunch if we were not making really powerful models that they could then steal. Like, and you ask these companies, oh, you're so worried about competition with China, you must be taking really serious security measures then, right? Because you don't want your models stolen by China. What kind of steps do you take to make sure that a nation state cannot steal your stuff? And they're like, oh, you know, we're a startup. I just can't take it that seriously, which I don't like the CCP, the Chinese Communist Party. I don't want them to shape the future of human civilization with extremely powerful AI systems.

• 1:22:27Speaker 01

  I just do not believe that the best way to prevent that outcome is for us to, you know, with this shoddy security and this like unclear sense of what we're going to do with the powerful models, zoom straight ahead. I don't buy it.

• 1:22:39Ezra Klein?

  I think all that is true. And I actually really, I think that point about espionage is really well taken. And this is where my background as a politics person, I actually worry because my read is that the most powerful stakeholder on AI in the federal government is a national security apparatus, that traditionally, they have done the most work on AI. They have the most freedom to work on AI. And they've just thought about it more. And I think that most of the energy on AI until very, very recently is coming from them. And they are inclined to look at the world in this way. And if that's one thing that that concerns me a little bit

• 1:23:21Ezra Klein?

  in terms of how it gets thought of politically, like even if you look at some of the recent investments in in AI, you know, in the Chips and Science Act, I mean, that's very explicitly framed as an anti-China act.

• 1:23:32Speaker 01

  Yes.

• 1:23:33Ezra Klein?

  And that was where I think the biggest recent play on this came from. I mean, there's been work like the AI Bill of Rights, it's quite toothless, even though I know that the it was a big deal for the White House to release a draft blueprint for an AI Bill of Rights, but that has no money behind it, there's no enforcement behind it. It's a it's a document on a piece of paper. They actually got chips passed. And like a lot of things, it passes under the rubric of of competing with China. And it's just at the company level, they're competing with each other. And at the government level, they're competing with China. And to a point you've made, I think, a couple of times, getting there quick is not obviously the right thing when there's a lot of danger from accelerating this, even just beyond the

• 1:24:15Ezra Klein?

  point of human comprehension. And I just don't see that taken as an equity seriously enough there. I mean, I think that the worries about China are a little bit fake in Silicon Valley, but I don't think they're fake at the national level, at the political level. And the power of that center of worry, I think, is actually very easy to underestimate. Like, right now, the anti-China side of the political debate in Washington gets whatever it wants. And what it wants is AI dominance, among other things. And that's potent.

• 1:24:43Speaker 01

  Yeah, I think there's an unfortunate dynamic where you might expect the government and the national security apparatus to see these polls where machine learning researchers are like, yeah, 10% chance of like, the end of human civilization to be like, well, that sounds like a big threat to America's national security. Like, we should get on that.

• 1:25:02Ezra Klein?

  If anybody's gonna end the world, goddamn it, it's gonna be America.

• 1:25:06Speaker 01

  I feel like there is, you treat threats as the kind of threat you've encountered before. We've had a Cold War before, so they're ready to have a Cold War. If China wants to have a Cold War, I think there's a lot of people who like, that fits into the frameworks they have for understanding threats. They have thought a lot about it. They have a sense of what America winning in geopolitical competition with China looks like or takes or whatever. And they don't have any sense of what to do if, in fact, the most destabilizing and dangerous technology is being developed here in America by patriotic Americans who want to like either bring about cool sci fi future

• 1:25:50Speaker 01

  or, you know, we can't guarantee a demon doesn't slip through the portal, but like, it's what we've got. And it feels to me like very strange in some ways to be in these conversations where people are like, yeah, we are going to radically transform the entire world we live in pretty much overnight. And no one's very interested in like making sure that that happens in a way that is safe for all of the existing people. And I think that where there is pressure coming from the national security establishment at all, like you said, it's coming in the direction of, yeah, beat China. And like I said, I don't like the CCP, but I don't like what the competition lens does to our odds of building AI that's good for anybody.

• 1:26:33Ezra Klein?

  So we've talked here about a lot of things that could go wrong, though obviously there are many more. I wonder before we end, talk a bit in a specific way about things that particularly in the near term could go right. Not just in making these systems safer, but in things they can actually do that would be good. Because right now, it's still sometimes a little bit hard, I think, to visualize that. Like, okay, having a Chachi BT bot that can help you cheat on exams is cool. Or, you know, I enjoy with my four year old, I'll sometimes call up Dolly and let him come up with pictures he would like the computer to draw. And like, that's really fun. But when you talk to these AI researchers, I mean, I think this is my worry about the bias

• 1:27:14Ezra Klein?

  of continuously quoting this 10% number about it might end the world. They put a slightly higher, not as much higher probabilities I would like, but a somewhat higher probability on it being very good for humanity than very bad. Is the main thing you hear a scientific advance? I mean, do they have other visions here about what they're doing to make people's lives better?

• 1:27:34Speaker 01

  I'm excited about translation. There is so much of a barrier to being able to participate in like intellectual life in so many fields if you don't speak English. There are so many brilliant people who, you know, work extremely hard as adults to learn English, to learn about fields they're interested in, to get jobs they're interested in. And I think we are on the brink of translation technology, just changing the game there, where everybody who isn't in a country that's chosen to firewall itself has access in their native language, in whatever language they read most easily, to all of human knowledge. I'm really excited for that. The drug discovery stuff is legitimately cool and valuable. Like, it's very hard to predict protein folding, but it might be much easier to do with AI. Similarly, I do think, I think it's super true what you were saying earlier, that science

• 1:28:19Speaker 01

  like grounds out in wet labs that take time to do things. But I think there's still a lot of like data processing and looking at images and stuff like that, that is going to be incredibly valuable and save a lot of people's time to have AI do. I think that while it will be extremely disruptive, and while it would ideally happen slower, automating a ton of human labor is fundamentally the sort of thing that could be really good. Like, it's mostly the busy work that we don't in fact like, right? Sending emails and writing applications and arranging meetings is nobody's dream job. And it could be really cool to have a much more productive economy in which a lot of

• 1:29:03Speaker 01

  that value is happening without people having to spend so much of their time on it. I'm fundamentally pretty in favor of the whole like, you know, increase productivity and then stuff gets better. There's some good stuff there with, with better code writers and better, you know, meeting schedulers and, and stuff like that.

• 1:29:20Ezra Klein?

  Something I wonder there is also increased creativity. So I think there's been correctly a lot of concern and anger about these systems trained on the actual work of artists and writers and so on, and those people aren't compensated. But I also think that when you, I can't draw anything. Like, I'm a terrible artist. My mom is actually was a professional artist. I wish that I could draw things I can't. It's neat for me that I can tell the computer what to draw and it will draw it. It allows me to play around in art in a way I couldn't before. And beyond that, you know, it used to be that a lot of people could illustrate something. But very few people could create a video game. Now, a lot of people might have amazing ideas for video games.

• 1:30:02Ezra Klein?

  And I think within a few years, they're just gonna be able to tell the computer how to make it with very detailed instructions. Same thing with movies, same thing with music. And there is value to knowing how to do the underlying kind of material creation yourself. I don't want to take anything away from that. And I think that fosters forms of creativity that are not always possible if you don't understand like the actual basics. But also, I do think there's something to just opening up vast vistas of creation to people who were never gonna learn how to code. We're never gonna learn how to use a synthesizer. But now maybe can work in tandem with a computer to create amazing art that they will just enjoy creating and other people might enjoy using. I mean, I think that stuff is actually really cool.

• 1:30:46Speaker 01

  Yeah, I think there is a lot of potential to really unlock amazing things. And instead of replacing humans with aliens, just have really powerful assistive tools that can like turn our imagination into reality more easily. Instead of like writing an essay for you, like being a really good editor and assistant that like suggests a bunch of relevant stuff and helps you write a better one. I think whether we do get that or not depends a lot on like who has, as you said, financial incentive to build it. To make a really, really good tool that strengthens the people who wield it instead of replacing them. But I think it's possible to build. And if it happens, I think it could be a really cool thing.

• 1:31:26Ezra Klein?

  I think it's a good place to end. Always a final question, which I now know you've prepared for at length. What are three books you'd recommend to the audience?

• 1:31:32Speaker 01

  Yeah, so I have been enjoying The Making of the Atomic Bomb. Obviously, the relevance to AI is very notable. But I think it's also just an interesting window into anything where a group of like brilliant people are like trying to make something go well. And like the combination of flaws and personal strengths. And I think.

• 1:31:57Speaker 01

  A lot of tech companies from the outside, it's like very hard to see why one succeeds and one fails. And it often comes down to these like weird human factors. And it's terrifying to think that the development of one of the most powerful and transformative technologies in our world also had just so much of the weird human factors to it. But I think it really did. And it makes me think a lot about like AI, but also just like how stuff gets done. The other thing I caught up on was Asterisk, which is a new magazine from some of the weird people that you like to talk about. It's like people in the AI and effective altruism and rationalist world. I was reading about how we invented oral rehydration therapy, which is it's sugar and salt and water. Took us until the 1960s. And the progress studies people always love the question like, why did it take us so

• 1:32:42Speaker 01

  long to get the bicycle? Why did it take us so long to get the plow? Like, how much was somebody having a brilliant idea? And how much was like that there's actually a ton of execution details? And how much is like, you actually need fairly advanced machining for bicycles? It might not seem like it, but you do or whatever. So oral rehydration therapy, right? Like, why did it take us until the 1960s to figure out that four million kids a year were dying unnecessarily from a lack of sugar and salt and water? I was obsessed for my entire teenage head with the Silmarillion, Tolkien's prequel to Lord of the Rings. It's not even a good fiction book. It reads some like something between a Bible and like a packet of world building notes. But I was in love with it.

• 1:33:25Speaker 01

  It's a beautiful like insight into how someone puts a story together. When AIs can surpass Tolkien, that's when I'll know we're really in trouble because he was something else.

• 1:33:37Ezra Klein?

  Kelsey Piper, thank you very much.

• 1:33:38Speaker 01

  Thanks.

• 1:33:50Ezra Klein?

  The Ezra Klansch was produced by Emma Fagawi and Galvin Jeff Galabush and Karma and Kristin Lynn. Fact checking by Michelle Harris and Kate Sinclair. Mixing by Jeff Geld. Original music by Isaac Jones. Audience strategy by Shannon Busta. The executive producer of New York Times Opinion Audio is Andy Rose Strasser. And special thanks to Carol Savaro and Christina Semilouski.