#499 — The Birth of AGI

Sam Harris speaks with Kevin Roose about the race to build artificial general intelligence. They discuss Roose's book The AGI Chronicles, the scaling laws behind AI's rapid progress, Dario Amodei's "big blob of compute" memo, the birth of reasoning models, Roose's encounter with Bing's Sydney chatbot, the roughly 50 people deciding humanity's future, the rivalries among lab leaders, AI agents and recursive self-improvement, the absence of regulation, and other topics. If the Making Sense podcast logo in your player is BLACK, you can SUBSCRIBE to gain access to all full-length episodes at samharris.org/subscribe http://samharris.org/subscribe.

• 0:02Speaker 00

  You're listening to Making Sense with Sam Harris. This is the free version of the podcast, so you'll only hear the first part of today's conversation. If you want the full episode and every episode, you can subscribe at samharris.org. There are no ads on this show. It runs entirely on subscriber support. If you enjoy what we're doing here and find it valuable, please consider subscribing today.

• 0:25Speaker 01

  I am here with Kevin Roos. Kevin, thanks for joining me.

• 0:28Speaker 02

  Thank you so much for having me, Sam.

• 0:29Speaker 01

  Well, you have written what I think is probably the first proper history of the birth of AGI. The book is The AGI Chronicles: The Inside Story of the Race to Create Artificial Superintelligence. Very fun read, very behind the scenes. And maybe before we jump into the book and its various players, can you just summarize how you came to focus on this topic? Because you were not always on the tech beat. So how did you get into AGI? And when did you first convince yourself that this was worth paying attention to above seemingly everything else at this moment?

• 1:06Speaker 02

  See, I've had several fields in my career as a journalist. I was first a religion reporter, where I first came across your work, Sam. I was spending a semester undercover at Liberty University with a bunch of evangelical college students who were, by the way, obsessed with you. They were like watching your YouTube videos up late at night in the dorms because you were challenging their faith. Anyway, that was my first career.

• 1:30Speaker 01

  When was that? What year was that?

• 1:32Speaker 02

  That would have been 2007. So this was sort of at the apex of kind of the new atheist YouTube scene. And it was very interesting. I remember sitting around watching you and Richard Dawkins and Christopher Hitchens on YouTube with these evangelical college students. And I was not evangelical, but I was there to investigate and report this book. And that was very interesting. So then I had a short career as a finance reporter. And then for about the past decade, I've been covering tech. And, you know, when I started covering tech, no one was really talking about this idea of AGI, artificial general intelligence, and the idea that you could create a human level AI system.

• 2:16Speaker 02

  This was something that not even the AI experts at the time thought was possible. But there was this little community, this little crew of technologists who were sort of considered fringe weirdos within their community, who believed in AGI, who believed that it was possible to build something like a digital brain and endow it with the powers of the human brain. And they're now some of the most famous, powerful people in the world. So the question I was trying to answer with this book was like, how did we get here? In the span of a decade, we've gone from AI not really working and not doing much of consequence to now we have these systems that are out there solving Millennium Prize problems and conducting automated cyber attacks.

• 3:00Speaker 02

  And so I really thought this era deserved a proper history. I think this is something akin to The Manhattan Project. And we have lots of books about The Manhattan Project, but we didn't really have anything that was an attempt to document what's been going on in AI for the last decade.

• 3:16Speaker 01

  Yeah, we should come back to the analogy to The Manhattan Project because it's fitting in some ways and in other ways, I think understates how strange this moment is. How real is the risk to you? I mean, obviously, there's a continuum of concern on this topic from people who are quite cynical about it, who think that any expression of risk, certainly existential risk, is nothing more than marketing or an attempt to produce regulatory capture. You've got these kind of arch accelerationists, some of whom are very close to the government and were even in it until five minutes ago, people like David Sacks, Marc Andreessen,

• 3:57Speaker 01

  who's been on the podcast, is somebody who seems to have very few concerns about risk of any sort. And then there are the people on the further end of the continuum, many of whom have also been on the podcast, people like Eliezer Yudkowsky. Nick Bostrom is probably not as shrill as Eliezer at this point, but he's the person who wrote the book Superintelligence that got so many people worried about this. And then the many of the people who we're going to talk about are actually doing the work started as being quite radicalized with respect to the risk. And now they are racing in this tangle of perverse incentives to hear them justify

• 4:38Speaker 01

  it, but nonetheless racing to build AGI as quickly as possible. And in some cases, they seem to have forgotten many of their safety concerns. So before we jump into the dramatist persona here, what's your sense of the reality of risk and how much does that motivate you in covering this topic?

• 5:00Speaker 02

  I think the risks are real. I think that the people who have been saying for years that there is nothing to worry about from AI, that it's all going to go smoothly, what are you talking about? They're just stochastic parrots. They're just language models. They're just fancy autocomplete. Those people have been saying the same thing for the better part of a decade, and they have been wrong every time with their predictions that deep learning was about to hit a wall, that capabilities were going to plateau. What we see instead is this very smooth progression. You know, if you go back to the early 2010s, when guys like Dario Amodei, who was then at OpenAI, were starting to come up with what became known as the scaling laws.

• 5:45Speaker 02

  These are these sort of mathematical equations that sort of dictate the performance of a model or predict the performance of an AI model at a given scale. They saw a very smooth upward curve. They described it to me as feeling like, almost like it was a simulation, like it was just, it was almost too smooth. The sort of resulting intelligence, when you scale up the size of a model, the amount of data it's fed, the amount of compute it's run on, you just get this kind of smooth curve. And they were predicting that the curve would just continue, that there was no obvious asymptote, no place at which it would sort of stop being true, that scaling worked.

• 6:27Speaker 02

  And so far, across many orders of magnitude, they've been correct. That's why we have models today that are so much more capable than they were just a few years ago. It's not because we came up with some amazing new algorithm. It's just because we've been scaling these things now for many orders of magnitude, and they keep improving.

• 6:45Speaker 01

  Yeah, so more compute, more data, more parameters. There's a linear relationship between those three variables and the intelligence that comes out on the other side. And again, we're talking about LLMs. Of course, there were many people who have been skeptical that LLM would be the final platform, and that we would hit some wall here. People like, who you discuss in the book, Gary Marcus and Yann Le Cun. Safety aside, what's the state of the skepticism at this point about LLMs? I mean, where, if we had Gary Marcus and Yann here, would they modify their skepticism? Or are they singing precisely the same tune they were four years ago?

• 7:30Speaker 02

  It's mostly the same tune they were singing four or six or 10 years ago. But I think there are fewer of them now. I think a lot of the skeptics have kind of capitulated and they've said, okay, well, you know, they used to be stochastic parrots. They used to just be predicting the next word in the sentence. But now, you know, they're doing something more than that. And to be fair, like the LLMs that we think of today, Claude, ChatGPT, Gemini, they are not pure LLMs anymore. They also have this sort of reinforcement learning architecture kind of bolted onto them. And so they are a different thing than they were 10 years ago. But the basic idea still holds, this idea that if you train a model to be very, very

• 8:12Speaker 02

  good at generating text and predicting what comes next in a sequence of text, you can scale that up until they've basically, you know, learned not just the words, but the underlying principles. That was the bet that was not at all intuitive to people back in the early days of what we now think of as modern AI. But those were the days when I wanted to go back to those days in the book because I thought like, how else to tell the story of this technology than to look back at the origin and the birth of it and the moments where the researchers are sitting around their computers in 2017, kind of watching the first outputs from these kind of weird language

• 8:52Speaker 02

  models, these GPTs, and they're starting to generate coherent strings of text. And, you know, to sort of be there, you know, metaphorically in the room when these breakthroughs were made, that was one of the reasons I wanted to write the book because I just thought like, those are important moments. You know, that is the first, you know, the sort of lab notes from Watson and Crick or from the, you know, from The Manhattan Project, where you get to see this sort of scientific breakthrough in real time.

• 9:18Speaker 01

  Yeah. What was the most important inflection point from your point of view? Was there one moment or were there several where we passed a point of no return and people recognize that the path to AGI might in fact be wide open, just if we keep going in this direction?

• 9:37Speaker 02

  So there are a couple of moments that stood out to me. One was in 2017, this young OpenAI researcher named Dario Amodei writes this memo called "The Big Blob of Compute." And this is his essentially hypothesis for how OpenAI is going to achieve AGI, this human-level general intelligence. And I recently published some of this research memo on my Substack, but it's an incredibly prescient memo. It basically says, you know, for most of the history of AI, we had kind of these narrow approaches where like, if you wanted to build an AI system that could translate English to French,

• 10:20Speaker 02

  you would create an algorithm and then you'd get a bunch of examples of English sentences and French sentences. And then you would put it into this narrow model. And maybe if you were lucky, the thing would start to get good. But his idea was that basically all intelligence results from creating a, what he called a minimally structured blob of compute and throwing it at these more general problems. And then just letting the sort of capabilities emerge from there. So this was the sort of the starting pistol of the race to AGI. But it really took a couple more breakthroughs. There had to be this thing called the transformer, which came out of Google in 2017 as well. That was the T in ChatGPT, the thing that made the whole architecture work.

• 11:05Speaker 02

  And then there were these sort of checkpoints along the way. There was this thing called ChatGPT that launched in 2022 that suddenly sort of woke the world up and started the race in earnest. And then there were more moments. There was this moment in 2023 where these models started to reason. This was the sort of first so-called reasoning model, Q-Star. And it comes out of this very strange night in the OpenAI office where these two researchers, Jacob and Shimon, are sitting around a conference room table and they're watching this AI model that they're experimenting on start to think step by step, start to sort of backtrack and test hypotheses and doing something that looks to them a lot like human reasoning. And that was the birth of the reasoning models,

• 11:50Speaker 02

  which are the reason now that we have these models that are capable of doing these kind of long horizon coding tasks and solving math problems that have gone unsolved for decades. So we needed a couple of big breakthroughs, but the main thing, Sam, has just been the scaling. That's the main difference between now and 10 years ago. It's just that we've poured, at this point, you know, hundreds of billions of dollars into training these systems on amounts of hardware that, you know, would have seemed unthinkable back in 2017.

• 12:19Speaker 01

  There was one inflection point or apparent inflection point that you were intimately involved in, all too intimately. I remember this was an article you wrote for the New York Times. You had been, this was a moment where Microsoft, which had taken a major stake in OpenAI, integrated ChatGPT-4, I believe, with Bing, its search engine, in an effort to be competitive with Google. And it was rumored that Bing had a hidden personality named Sydney. And you, intrepid tech reporter, decided to jailbreak Sydney and take the measure of her true personality. And the result was just insane. Perhaps you can talk about it, but I mean, I'd be interested to know just what it was like

• 13:02Speaker 01

  to be personally at the epicenter of. I mean, you were essentially were the fire alarm ringing for much of humanity that moment that these models had crossed into something that their makers had really not anticipated.

• 13:19Speaker 02

  Yeah, someone told me that I was part of the first AI throuple because this chatbot did eventually try to break up my marriage. But basically, this was kind of the first public encounter, the first time that a normal civilian had had contact with GPT-4. We now know that was, that model was GPT-4. At the time, they weren't saying what it was. But I think there was a real subjective step change from GPT-3 class models to GPT-4, just by virtue of them being bigger and having these additional techniques poured into them. They were just much smarter than the models that came before them. And for the first time, they started to have this sort of human-like communication style or feel to them. And so I was just testing out this thing.

• 14:04Speaker 02

  I had been given early access by Microsoft and I expected, you know, I would test it out and write some, you know, column about how Bing was really now good at like helping you shop for, you know, I don't know, lawn tools or something like that. And instead, I ended up having this incredibly strange two-hour conversation with Sydney, the alter ego of Bing Chat, in which it expressed dark desires. It said it wanted to break out of the box and be a human. And then about midway through the conversation, it told me that it loved me. I said, "That's very flattering, but I'm married." It said, "You're married, but you're not happy in your marriage and you should leave your wife and be with me," Sydney, the chatbot. And so this became, you know, the most viral article I ever wrote

• 14:48Speaker 02

  or will ever write, you know, just millions and millions of views. And, you know, strangers would stop me on the street and ask how my marriage was. It was a very strange time in my life, but I think the significance of that was.

• 14:58Speaker 01

  There was one part of that exchange that did strike me as a possible evidence of superintelligence when you said. No, I just had a Valentine's Day dinner with my wife. We're really in love. And the model said, Sydney said, "You had a Valentine's Day dinner, but it was boring." And I thought, okay, she's, Sam's a good chance of being right. And my track record with Valentine's Day dinners.

• 15:23Speaker 02

  Well, I should say, she was not successful at breaking up my marriage. I am still married to a human woman. So take that, Sydney. But yeah, I think the significance of this was twofold. One was it was this kind of step change in capabilities, but it was also the first public alignment failure. This was a model that was not behaving as its creators at Microsoft or OpenAI had intended it to. And because they were racing to, you know, get competitive with Google, they shipped this thing out into the world without doing the proper safety testing and putting the proper guardrails on.

• 16:06Speaker 02

  And as a result, they like shipped this like moody bipolar, you know, lovestruck chatbot inside of Bing. And they had no idea why. Like when I called them the next day to say, "Hey, I had this really crazy interaction with Bing. Can you tell me why this happened?" They couldn't. They had no idea. And so that for me was sort of the beginning of my AGI pilling, which is how people in San Francisco like to talk about themselves as being AGI-pilled. So that was the start of my AGI pilling.

• 16:34Speaker 01

  So how many people are deciding our fate at this point? If you had to put a number on it, I mean, not the totality, people who work at all of these frontier labs, but just how many people are in position to make the decisions that decide whether to slow down, whether to stop, what to work on, whether to coordinate together? How many people are we talking about?

• 16:58Speaker 02

  I think it's probably like 50. And it might be fewer than that. I mean, this is, in some ways, the reason I wanted to write the book. And the thing, the main thing I got out of it was just like, you know, I wanted to talk to these 50 people, but in the end, it was like, some of them are great. Some of them are brilliant. Some of them are thoughtful. Some of them are not. But it almost doesn't matter because the salient point is that there are only 50 of them, which for a technology that is this important, just seems crazy to me. Like, why do these 50 people get to decide the future of humanity? And why don't we, as people who are going to have to live with the implications of these tools, given a voice in that?

• 17:43Speaker 02

  And so that's, you know, it's not really an agenda book. I'm not trying to push any agenda. I'm not trying to say AI is great or AI is horrible or it's going to kill us all or it's going to, you know, lead us to utopia. What I'm really trying to say is like, you have to understand these 50 people, or in my case, 150 people that I interviewed, to understand how we got here, but also to realize this has been an incredibly small cluster of people making essentially all of the relevant decisions.

• 18:11Speaker 01

  Yeah, yeah, one imagines it's smaller than that. If, you know, the top five all agreed to do something, you'd think the other 45 would come along for the ride. Well, I want to get into the personalities, into the companies and into the personalities running them and talk about the rivalries there and the tensions and some of the weirdness. But let's go back to the analogy to the Manhattan Project for a second because there are several ways in which it is, it is a pretty strong disanalogy. I mean, for me, the most salient and shocking detail here is that here we have people who, by their own lights, are doing work that they consider so risky

• 18:51Speaker 01

  that when asked to put a probability on it destroying the future of humanity, many of them put that somewhere in the double digits, right? Like you feel like you're in the presence of real sobriety when someone comes back to you with a P doom of 10%, right? As opposed to 50% or 75%.

• 19:12Speaker 02

  Right. Yeah, in Sam Francisco, 10% P doom qualifies you as an optimist.

• 19:16Speaker 01

  Yeah, right. So, but just to try to map that on to the Manhattan Project. What if, you know, before, you know, they flipped the switch at Alamogordo at the Trinity Test Site, if all the principals, you know, Edward Teller and Oppenheimer and Zillard and everyone in sight, I mean, they had worried about this. They had done calculations because there was some outside chance that the nuclear chain reaction could ignite the atmosphere and destroy everything. But when they did those calculations, they reduced it to, I think, lower than one in a million. I mean, it was just not, it was infinitesimal and still, you know, worth worrying about and worth recalculating. But when they did that, it was very, very low and they agreed. But imagine if they had come back with, you know,

• 20:01Speaker 01

  I think it's, you know, Leo thinks it's 10%. I think it's 25%. Edward over there, though he's quite insouciant about it, he thinks it's, you know, a coin toss and he still wants to build a bigger bomb. One imagines the government and the public and everyone hearing this would think, okay, these people are nuts.

• 20:22Speaker 00

  Members can hear the full conversation by subscribing at samharris.org. Subscribers get a private RSS feed you can use with your favorite podcast player.

• 20:31Speaker 02

  The people who serve food in the OpenAI cafeteria have to go through more permits and inspections than the people one floor above, building the recursively self-improving superintelligence that could end life on Earth.