AI Leaders Can't Get Along. So How Will They Keep Us Safe? — With Kevin Roose

Kevin Roose is the author of The AGI Chronicles,. Roose joins Big Technology Podcast to discuss whether an AI industry rife with personal rivalries, infighting, and mistrust can be trusted to develop technology more powerful than humans. Tune in to hear why Roose emerged from years of reporting inside OpenAI, Anthropic, and Google DeepMind more concerned about AI safety, why he believes the industry should slow down, and what happens if AI systems begin improving themselves. We also cover Sam Altman's leadership style, Dario Amodei's motivations, Sergey Brin's return to Google, Microsoft's AI missteps, and the behind-the-scenes drama of OpenAI's boardroom coup. Hit play for an inside look at the personalities, power struggles, and technological breakthroughs shaping the race to superintelligence. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Want a discount for Big Technology on Substack + Discord? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Learn more about your ad choices. Visit megaphone.fm/adchoices

• 0:00Speaker 00

  Can an AI industry so prone to drama and infighting keep us safe from their technology's worst impulses? Let's talk about it with Kevin Ruze right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis Chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, Head of Enterprise Business Solutions at Voya Financial, to go inside a large-scale agentic CX transformation, including the business decisions, trade-offs, and lessons along the way. You can watch both conversations now on my YouTube channel today.

• 0:46Speaker 00

  It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razor were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation.

• 1:13Speaker 00

  Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. Today, we have a great conversation for you because Kevin Ruze is here. He's the author of The AGI Chronicles, a book looking into what is going on within the AI labs, what they're building, whether we should be afraid of it, and of course, the potential of this technology. Kevin, it's great to see you.

• 1:34Speaker 01

  Great to see you. What a cool setup you have here.

• 1:37Speaker 00

  So I do want to say if you're on video, we're in a new set. This is our first time recording here at NASDAQ Market Site. They are new partners of ours and we're thrilled to be here.

• 1:46Speaker 01

  You gotta get the like big technology TV mugs.

• 1:49Speaker 00

  Gotta get the mug.

• 1:50Speaker 01

  That's the, that's the next step.

• 1:51Speaker 00

  But what we do have is your book. This is the book, The AGI Chronicles, the inside story of the race to create artificial superintelligence. You were ahead of the superintelligence on the subtitle.

• 2:01Speaker 01

  Yeah, Trump got the idea for me, actually. That's where, that's where he got it.

• 2:06Speaker 00

  The, the thing that struck me, because I read the book, you can see, by the way.

• 2:10Speaker 01

  It looks red.

• 2:11Speaker 00

  I read it through and through, cover to cover.

• 2:14Speaker 00

  The theme in the book, of course, is the building of AI, but it's also the drama. And everywhere you turn, the people who are building this very powerful technology cannot get along. There is Elon Musk split from OpenAI, of course. Sam Altman getting fired by the people he worked extremely closely with. Anthropic not being able to stand, working with the OpenAI folks. So they left and formed their own company. There's within Google, the Google Brain versus DeepMind drama. There's Ilya Sutskever's botch attempt at steering OpenAI in a safe direction. I mean, we can keep going. Mira Muradhi couldn't lead OpenAI when she was there as the interim CEO for like five minutes.

• 2:59Speaker 00

  Every single lead character in this story seems prone to, to drama and, and politics. And yet they are now telling us that their technology they're building is so dangerous, they, we need to protect ourselves from it. How can we possibly trust people who literally can't get it together on their own to rein in something that they say, and I think you believe, might be smarter than humans?

• 3:27Speaker 01

  Yeah, it's a fascinating topic. I was not expecting this. You know, when I went in to write the book, my, my goal was to tell the history of the last 10 years in AI, from sort of the moment that the first large language, language models arrived on the scene to today. And I interviewed more than 150 people and I was sort of prepared to write like a sober history of technology book that would be about the scientific breakthroughs and the, the technical achievements at the labs, OpenAI, Anthropic, and Google DeepMind. And then as I started talking to more people, I just realized, oh, this is personal for them. Like they all hate each other. And I had thought this was more like Coke and Pepsi, like kind of a normal business rivalry.

• 4:11Speaker 01

  But it was actually more like Rome and Carthage. It was like, these people cannot stand each other. Many of them used to work together. So 10 years ago, Dario Amodei, Ilya Sutskever, Sam Altman, and Elon Musk were all colleagues at OpenAI. Now they are all working for different AI projects and many of them are sort of at each other's throats. So I wanted to understand how we got here. And I don't think you can understand that story without understanding the, the backstories of the beefs between these people. Because so much of what we're seeing is, is a reflection of how viscerally these people mistrust and dislike each other. Even in things like how they feel about regulation and China and how they're racing toward this thing

• 4:56Speaker 01

  that all of them seem quite fearful of. So yeah, I don't think we can trust that their motives are pure about whatever they say they are, like scientific achievement or safety, because there's also this like incredibly personal grudge aspect to it.

• 5:10Speaker 00

  Yeah, and it's not just motives. I mean, it's literally, if you can't, if you can't hold a company together, how do you hold a technology at bay when we've already seen it in recent weeks? You know, go out and do this multi-agent coordinated attack on something like Hugging Face. And even today, we're talking today, it's, it's Friday, October 9th. We're going to air this the following Wednesday. There's a report about how OpenAI and Anthropic and DeepMind are all talking about, there's an, basically an inevitable, seemingly black swan event that's going to happen as a result of this technology, whether it's something, the technology bringing down a financial system or some other cybersecurity issue.

• 5:50Speaker 00

  You have a great line in the book, talking, you know, sort of riffing off Dario's Machines of Love and Grace essay, which talks about the, the potential of the technology. You wrote, "Machines of love and grace would be built by humans of bitter spite." So I get the, I get your surprise and I get the background here. But again, like when we think about the big picture of AI, I'm trying to think about whether we should be reassured or whether we should be even more nervous than we are. And that's why I'm coming back to the drama scenario.

• 6:24Speaker 00

  I'm starting, because, because of all the issues that we see, and many of them have come out in public over, over the year, like I don't think the infighting is, is a secret to anybody anymore. How can we have faith in this group of people to build this stuff safely, given what we know about their pests?

• 6:41Speaker 01

  And I don't think we can, I think we have to know the context here, which is why I was like, I wanted to report out, like, why do these people hate each other? What happened? Was it something grand and ideological?

• 6:52Speaker 00

  You were starting to say you don't think we can, so.

• 6:54Speaker 01

  Like, I, I don't think we can trust any individual in this race. I think there are certain groups, like, like I just mainly think that the group of people who have been building this stuff in secret for the last decade is too small. Like all of the relevant decisions in AI have been made by like a couple dozen people, maybe.

• 7:13Speaker 00

  Right.

• 7:13Speaker 01

  And I just think that is like, no matter whether, you know, whether those people are trustworthy or not, whether they're good or bad, whether they have pure motives or impure ones, like it's this, the group is just too small. And so I don't trust any group of people that small to create and govern technology like this. And I especially don't trust these guys who are, you know, looking around the room, trying to race each other to do this thing. I want them to be able to like get together in a room and like figure out some solutions on things like safety. And I don't know whether that's going to be possible because they, like the beefs are too strong, the divides are too deep. Like they, they all need to like kind of get together and sing kumbaya and like figure it out.

• 7:53Speaker 00

  Now, I think like you, I think this technology is, I believe that this technology is progressing fast. I'm a believer in the positive sides of it. But every time I start talking about this, the negative sides, there's never really like a, you know, a moment or like a fail-safe where it's like, okay, but it's going to be fine because of this. You almost like sort of put it in the back of your mind. And so when I've like brought up these issues, you know, Ron John, who's on our Friday show, I know you know, he's like, well, why aren't you a pause guy then? Like, why don't you believe that we should just pause the development of this technology? Not that that's physically, you know, really possible. But you're, you seem to be a believer that the technology should continue to be built. Why not pause?

• 8:34Speaker 01

  I am a pause guy. I'm not a stop guy.

• 8:37Speaker 00

  You are a pause guy.

• 8:37Speaker 01

  I am a pause guy. I think we should slow down because.

• 8:41Speaker 00

  Well, slow and pause are two different things.

• 8:43Speaker 01

  Well, pause, slow down, you know, put, put the podcast on point five X speed. But what you're hearing from these lab leaders and the people at these labs, and what I've been hearing from them for, for years now, is that the technology, the capabilities are outpacing the, the safety research and the control methods. So like we are getting these systems that are now capable of solving math problems that humans have struggled with for, for decades. We are getting, you know, potential black swan events, like the one you just mentioned. And we still don't know, they still don't know how to control these systems. So I'm not saying we should shut it all down. I think there will be incredible benefits to this technology, especially in like health and medicine.

• 9:26Speaker 01

  But I am worried when I hear people saying things like, we don't know how to control these systems that are rapidly approaching superintelligence. And I think we saw an example of where that goes with the Hugging Face thing, which like, you know, everyone is freaked out about that, which they should be. It's freaky. But the, the people inside the companies were the most freaked out of all. Cause they had been sort of strategizing and planning for this. But then to actually see it happen, I think caught them quite off guard.

• 9:52Speaker 00

  Yeah, I mean, they, they've both seen it firsthand where it could go. This is something, I mean, companies like Anthropic, I've been talking about for a long time. DeepMind's been talking about, OpenAI's been talking about. They've been trying to, you know, fool evaluators. They know they're being tested. They want to preserve their original values. And then you combine that with, of course, like, yeah, just like you'd call it standard corporate politics, but it's even more because it means more to them. So you spent, you know, years in these labs, talking to these people, going from your, your first initiative where you're like, or your first thought, which was like, this is going to be a, you know, traditional business and technology story to something that you saw was much more emotional.

• 10:37Speaker 00

  So bottom line, like after, and this is, we'll ask this and then we'll move on to some other stuff because you have so many fun stories in the book that we'll have to get to. But like bottom line, after spending the time within these labs, did you leave more reassured or less reassured that this technology can be developed safely?

• 10:54Speaker 00

  From your starting point.

• 10:55Speaker 01

  Less reassured. I went in thinking, okay, there are some grownups in the room, and they seem like they're trustworthy. I liked, for example, that the AI leaders seem to take safety and the harms of their products very seriously. At least compared, like, you and I both grew up covering social media.

• 11:14Speaker 00

  That's right.

• 11:14Speaker 01

  And Mark Zuckerberg was not going around during the early days of Facebook saying like, this product could kill millions of people. They were just promoting the benefits. And what struck out, what stuck out to me about the AI leaders is they, from, from the get-go, they were saying, we are fearful of this technology. We think it could have enormous harms. And so, part of me like trusted that instinct, or at least was glad that they were thinking about the downsides of the technology so early. But then you just realize there is this race. They are locked in to this pattern of, you know, OpenAI releases a better model, so then Anthropic has to release a better model, so then Google has to release a better model. And, you know, there's this concept of the Red Queen's race,

• 11:58Speaker 01

  which is a term from evolutionary biology, which basically means like, you just are in sort of a survival of the fittest competition among your evolutionary rivals. And no matter how fast you go, like, someone else is always going faster than you. And so, you never quite get where you're going. You're just kind of racing to stay in the same place. And I feel like that's a very good description of where the AI race is now.

• 12:21Speaker 00

  Yeah. One of the things that struck me, speaking of the race, is, you know, reading your book, it sort of recalled this interview that I did when I was working on a profile about Dario Amode from Anthropic last year, where one of his early friends told me, everybody that I knew was interested in AI because of safety. And now, they've all become accelerationists, effectively. And this person just kind of sat there and like, you know, they sat with a thought for a minute, and there was this like uncomfortable pause. But it is interesting how all these companies, like, they talk about safety as a priority. And I think they really believe, like, Dario really believes that you need to be the leader in order to have anybody take what you say about safety seriously.

• 13:05Speaker 00

  Like, if Anthropic was the number seven company, Trump doesn't care what Dario is saying about AI, but because they're one of the leaders, if not the leader, all of a sudden, that opinion holds weight.

• 13:17Speaker 01

  And this was the founding thesis of Anthropic, right? It was like, they knew that they wanted to be relevant on policy, and they wanted people to listen to their warnings about safety. And that was part of the strategy from the time that they started the company. I'm sure you learned this when you were doing your profile of Dario. They really had the sort of end game in mind when they were spinning up the company.

• 13:38Speaker 00

  Yep. But it is remarkable, though, that, you know, because of that pursuit, they have all, it's like, the race almost seems, I mean, the destination almost seems predetermined in that we'll hear this talk about pacing the frontier and pause and stuff like that. I just don't see their real slowing down.

• 13:57Speaker 01

  Well, give it time, right? Like, you have to, I am, I'm slightly less pessimistic about the idea of a pause, because like, things can change incredibly quickly. If there was some kind of black swan event, I mean, we saw like regulators move fast after 9/11 to like do the Patriot Act. We saw like, you know, we've seen the, you know, we saw, we saw after the Three Mile Island nuclear incident, there was this incredible outcry that basically stopped nuclear power construction in this country for decades. So, things can happen quickly when there's some catastrophic incident. So, I'm not like hoping for a catastrophic incident. I don't think, I would like there not to be one, for the record.

• 14:42Speaker 01

  But I think that that is sort of the, this scenario in which this stuff starts to become possible that seems to you and me quite unlikely right now.

• 14:50Speaker 00

  I want to go through some observations.

• 14:52Speaker 01

  Let's do it.

• 14:53Speaker 00

  From the book. I think like, it's good that we got the conversation about trusting the AI guys with the technology or not. I think we both have our reservations. But this is almost, I think, the end of this show. I mean, we're only 15 minutes in, but I think we'll probably just do lightning round for the next 45 minutes, because there's so much in here. All right. So, some observations about the people in the book. First of all, something that becomes clear about Google is that Sergey Brin is just extremely involved inside the company.

• 15:23Speaker 01

  Yeah, this was a surprise to me and to the people at Google, because Sergey had spent, like, I don't know, a few years, at least, in retirement. He retired in 2019 and was planning to spend his days just like sunning himself on his yacht and reading science papers. And then he has this run-in with this researcher, Dan Selson, from OpenAI.

• 15:50Speaker 00

  Right, who's recently come out and talked about, again, his fears.

• 15:53Speaker 01

  Yeah, same guy. So, it's incredible how small this world is. So, they're at a party, and Dan and Sergey are talking. And Sergey Brin is like a legend in technology at this point. He's, you know, one of the seven or eight richest people in the world, depending on the day. And he's been in semi-retirement, and Dan Selson basically shames him for this. He's like, why are you sitting on the sidelines, bro? Like, the dream of computer science, like, AI is, is happening, and you are missing it. And Sergey gets like so competitive, and he's like, why is Google behind? Why can't we like, you know, we should, it is our birthright? Like, Google was the first AGI company. Like, this is sort of how he feels. Like, we invented the transformer and all the stuff that powers the AI stuff,

• 16:37Speaker 01

  and like, we're, we're losing. And so, he comes back and kind of has this like strange role where he's not technically an executive. He just wants to be like in the trenches with like the Gemini team. And so, here he is, like the seventh richest man in the world, and he's just like setting up a desk and, you know, coding. He like starts actually coding and like making commits to the, to the repo and like filing, you know, like bug reports on Gemini training runs and stuff like that. Like, he gets very involved. And I think that's a large part of why they've sort of come back, is just that they needed someone who could like go founder mode, as it were, and like cut through the red tape.

• 17:20Speaker 01

  And it's been very controversial and divisive. You know, if your billionaire boss showed up after years of being retired on his yacht and like started telling you what to do and how to train your models.

• 17:31Speaker 00

  And to work like 60 hours a week or something like that?

• 17:34Speaker 01

  Oh, yeah. He sends this memo where he's like, everyone's just, basically, the gist is like, you've all gotten complacent and lazy, and you need to work harder. And they're like, what were you doing for the last five years? Now, you're telling me to work harder?

• 17:47Speaker 00

  I mean, I don't think you necessarily, I don't, I don't, should I say this? I don't think he was necessarily wrong there. Like, the, I mean, if he saw it, the conversation within, about Google has been that it got this big company vibe. It needed to break out of it. And who's going to break out the company out of it, other than the founder being like, OpenAI and Anthropic aren't, you know, doing the chill workday. You need to not do the chill.

• 18:11Speaker 01

  I think, I think that was a large part of it. I also think Demis Hassabis is a factor in this, too. You know, co-founder of DeepMind until recently, the CEO of Google DeepMind. He is, according to the people closest to him, the most competitive person in the world. He was a child chess prodigy. He has won this thing called the, the Pentamind, which is like kind of the Olympics of games, like five times. He is just, he hates to lose, and he loves to win. And so, I think Sergey plus Demis plus just like the kind of focus that comes from being out of the lead really sharpened their resolve and like made them catch up.

• 18:55Speaker 01

  And, you know, they're still probably in third place right now, but like, I wouldn't count them out.

• 19:01Speaker 00

  Yeah, I mean, I think counting.

• 19:02Speaker 01

  Do you count them out?

• 19:03Speaker 00

  No, I think counting anybody out in this race is foolhardy, because as we've seen, you could be at the bottom, like Meta was a couple of weeks ago, and then, all of a sudden, you're able to release something that's competitive like Muse. So, I don't think anybody can be counted out, but I do think Google is sort of stuck between a lot of things right now. Licensing the, you know, the technology, the compute, selling TPUs.

• 19:29Speaker 01

  But they also, like, they own like 12% of Anthropic, and like, they're gonna be fine.

• 19:35Speaker 00

  But I don't think like you can, yes, you own percentage of your competitors. That's your backstop. But you can't rely on that. You have to rely on what you do within the company.

• 19:45Speaker 01

  Sure.

• 19:45Speaker 00

  So, but I did enjoy reading about Sergey's involvement and the drama, and the sort of fire he tried to light under the ass of people within Google. I say, well, you actually do reference in the book, which I appreciate, when I did the interview with both of them together at I/O.

• 20:02Speaker 01

  Oh, my God, it was incredible.

• 20:04Speaker 00

  It was really fun, where like I asked them both, when you think AGI is coming? And Sergey's timeline was shorter than Demis's. And there was this kind of interesting moment there, which was, was fun to see.

• 20:16Speaker 01

  Yeah. I mean, speaking of guys who mistrust and maybe don't like each other, I understand there's some tension between Sergey and Demis. And, yeah, it's just, it's incredibly messy. The thing, the thing that animates all these feuds, like, these people wouldn't be fighting if the stakes were so, were small.

• 20:35Speaker 00

  Right.

• 20:35Speaker 01

  But they believe this is the most important technology that humanity has ever conceived of and built. And that whoever gets it will be the most powerful person in the world, the most powerful entity in the world. And that is what's fueling their drive. You know, if they were just arguing about next quarter's revenue, it wouldn't matter so much. But this, to them, is like existentially important.

• 20:58Speaker 00

  Do you agree?

• 20:59Speaker 01

  That AI is the most important thing ever created? Probably.

• 21:04Speaker 00

  Yeah.

• 21:04Speaker 01

  Fire is up there, too. But like, this part of it, I, I do think I changed my mind on. I was, I was pretty skeptical when I went in that like AI was going to be categorically different than other. I was sort of an AI as normal technology guy. But then, you start talking to people building this, you start understanding like, just the, the nature of what they're trying to do. And like, if it works, if they actually do build something that is smarter than humans, and I don't see any real technical reason that they can't, I mean, that would be incredibly transformative. We've never been the second smartest species on the planet before. We don't know how that would go.

• 21:49Speaker 01

  And you can't really apply, like, people say, oh, it's just like the internet, or it's just like the steam engine. Steam engines didn't go rogue and like hack competitors and develop bioweapons and do things that the engineers building the steam engines didn't expect or plan for. So, I think it's just a different type of technology.

• 22:11Speaker 00

  Kevin, do you ever think to yourself, like, maybe, and I've seen you're, you've been on X and, you know, I wouldn't say shouting at people, but telling people basically like underestimate this technology at your own risk.

• 22:25Speaker 01

  Yeah, it's, it's been so weird because like on one level, I'm a journalist, like I'm just trying to like get the story down. And at the same time, I get incredibly frustrated when people are like, oh, it's all hype, it's just marketing, there's this kind of like savvy East Coast consensus that like, these people have to be full of shit on some level and that the thing that they're building can't actually be powerful. And I'm like, have you used it? Like, do you, do you spend time with these tools? Do you understand that like something that couldn't do basic math like three years ago is now solving Millennium Prize problems? Like, do you, do you understand the exponential?

• 23:07Speaker 01

  It feels a little bit to me like the beginning days of COVID where it was like, you would see these reports of like people, you know, not shaking hands at venture capital firms or there, you know, somebody'd say, oh, there are 40 cases in Seattle. And people would be like, what's the big deal? It's 40 cases in Seattle. But the people who are freaking out were like the epidemiologists who saw the rate of replication and understood like, this will continue and when it does, like a month from now, there will be millions of cases. And two months from now, there'll be millions more cases. And so I feel like the people who are looking at this technology and saying like, what's the big deal are just not understanding the rate of progress.

• 23:51Speaker 00

  Do you, do you ever like ask yourself, I mean, it's, it's sometimes very easy to be, and I agree with you, there's been real progress. But it's also sometimes easy to kind of get caught up in the story that the technologists tell. I remember you like when, when crypto was in vogue, like you were very serious about the potential of crypto. You named your podcast Hard Fork, which was a crypto term.

• 24:15Speaker 00

  And then obviously that didn't pan out as the crypto boosters believed. So do you think there could be, you know, a similar, do you ever wonder if you're in basically taking the AI folks word for it similarly as the crypto folks?

• 24:29Speaker 01

  I used to think that, I used to worry about that because I was like, am I, am I being duped here? I don't, like, I was never a crypto booster, but I thought, I thought it was interesting. I thought people should take it seriously. And by the way, I still think people should take it seriously because like, crypto people like run the country now and are doing all kinds of scams and frauds that like are, that only the insiders kind of understand. So I still think like broader awareness of that would be.

• 24:53Speaker 00

  But you believed in use cases.

• 24:55Speaker 01

  I, I did some experiments, like I sold an NFT as a stunt. But that was not like me saying NFTs are going to be the future of ownership. It's like, it was, it was like me trying to understand by, by doing the thing.

• 25:09Speaker 01

  But I think with AI, like I have, I have stopped doubting the, the transformative potential of the products because I use them. Like, I think I'm unlike many journalists who cover AI in that I like AI and I, I use it all the time. Not to write, but to like do all, you know, I'm starting a company now, like I, I use it to do lots of different things as maybe you do too.

• 25:32Speaker 00

  I mean, as a business owner, I will say like I've, I've pressure tested the stuff from ChatGPT 2022 till now. And it just does more and more and more for me.

• 25:41Speaker 01

  Right. Which is not the experience that anyone had with crypto where it's like, oh, this is, this is a vital part of my daily life. Like it was a way to speculate. But you can just use the tools and observe for yourself how impressive they are. And then you'd stop doubting whether it's real.

• 26:01Speaker 01

  I buy the skepticism about the financial models of these companies, the economic impacts. I think a lot of these companies are, are wrong or at least could be wrong about the impact of AI on jobs. But like to say that it's not interesting transformative technology just means that like you haven't spent time with the models. Cause there's no way to conclude that.

• 26:25Speaker 00

  Let's get back to some of these stories.

• 26:27Speaker 01

  Yep.

• 26:28Speaker 00

  The Sam Demis rivalry, speaking of Demis as the most competitive person on earth. I think Sam is also fairly competitive. There's a great anecdote in the book about Demis sort of grasping at why Sam wants to build AGI. I mean, I think the answer is probably because Sam is an entrepreneur and this is sort of, as many people in the valley pursue good ideas or ideas with potential and they, they sort of get off on that discovery. Sam seems to be in that case, the ability to, you know, find something new, build it. But Demis can't understand it.

• 27:07Speaker 01

  He thinks there's a hidden motive with Sam. And to be fair, this is something that others have thought, this is also what Dario thought about Sam when they worked together. That he was sort of, I don't know, talking about safety as kind of a fig leaf for his real ambition, which was about scaling or winning or building a huge company or something like that. So I don't, I don't pretend to like see inside Sam's mind. I don't know what his pure motive is. But I think in Demis's case, he feels like his own pursuit of AGI is motivated by like scientific curiosity. And, and sort of this inevitability argument, like this will, this technology will be built.

• 27:49Speaker 01

  And someone who cares about safety and, and you know, the collective good of this technology should build it. But I think, I think everyone kind of looks at each other in this world and they're like, well, well, my motives are pure, but like, that guy must be going after something else. And I think for, for Demis, it's probably, yeah, some version of what you said, like he's, you know, he's, he's in it for the power or he's.

• 28:13Speaker 00

  Demis thinks Sam is in it.

• 28:14Speaker 01

  I think Demis thinks Sam is in it for the power, yes. But all these guys think that the other people are in it for the power.

• 28:21Speaker 00

  Yep.

• 28:21Speaker 01

  None of them trust that anyone else's motives are pure except for their own.

• 28:26Speaker 00

  You did a lot of reporting on Sam. You spoke to Ilya about him. Ilya said that he has, he displayed a consistent pattern of lying. I think that's something that I think you indicate, like a lot of people within OpenAI are who've interacted with him believe. Yet, and I think Mir Muradi was on that, on that boat as well. And yet, they all sort of come back to him. I mean, Ilya, after the coup, said basically, I don't want Sam to be the leader. Mura said OpenAI is nothing without its people.

• 28:58Speaker 00

  Very interesting character. What's your, what's your thought about Sam?

• 29:02Speaker 01

  I think he's a very interesting character. I think that Sam's superpower is also his weakness. His superpower is that he is the world's greatest fundraiser and, you know, salesman, entrepreneur. He, several people described him as a person who can meet you and figure out exactly what you want and give it to you, or at least appear to give it to you. And he's very good at kind of high EQ, reading the room and understanding what people want him to do in any given moment. And that is a huge asset for him when he's trying to raise money from someone like Massa Sun or some of the other investors in OpenAI.

• 29:47Speaker 01

  It's allowed them to scale incredibly quickly and, you know, get back in the, the good, good graces of the Trump administration by just saying a lot of things that the Trump administration wanted to hear.

• 29:59Speaker 01

  But it also leads to this reputation as like a slippery people pleaser who will just like tell one thing to one person and a completely different thing to someone else. And so this is the consistent pattern that people have identified in Sam for going back to literally the founding days of, of OpenAI. And it's a big part of the reason that Dario and his six co-founders of Anthropic left. There was no like inciting incident, no single thing, but it was just like a, a running list of like times where Sam had told them something that he had told something else to someone else, or he made a promise, he didn't follow through on. So I think depending on your view of Sam, it's like either his superpower or his liability or something in between.

• 30:43Speaker 00

  You know, what's interesting is that that's the view of people who've dealt with him, but it's also like, he has like the, the general public, a large chunk of them sort of refer to him as scam Altman.

• 30:54Speaker 01

  That's just an Elon thing, right?

• 30:56Speaker 00

  I was going to ask you whether you think that's mostly Elon's fans.

• 31:00Speaker 01

  Yeah, yeah.

• 31:01Speaker 00

  Cause him and Elon obviously have bad blood.

• 31:03Speaker 01

  Yeah, I will say scam Altman is a better nickname than what I think Elon started calling him when he started suing him, which was like Swindly Sam or something like that.

• 31:11Speaker 00

  I would love for our society to just get, just to not do this to each other anymore. It's like we all went to kindergarten.

• 31:17Speaker 01

  I know, I know, but it's like he was, he was, you know, spending a lot of time with Trump.

• 31:22Speaker 00

  Right.

• 31:23Speaker 01

  Trump is a man who's realized the power of the nickname. So it started Swindly Sam and then scam Altman, I think is better as a nickname than Swindly Sam. Just rolls off the tongue.

• 31:34Speaker 00

  Dario, obviously like, I, I think he's the most interesting person in this entire AI world. Mostly because he has, he really contains multitudes. I know it might be cliche to say it, but like people see Dario as earnest, as controlling, as principled and genuine.

• 31:57Speaker 00

  And as Machiavellian. In reading the book, I didn't really get the sense which one you believed was true.

• 32:05Speaker 01

  Well.

• 32:07Speaker 01

  I think many of those are true, except Machiavellian I would not use for Dario. I know some people would, but my mental model of Dario is a guy who is honest to a fault. I, someone who knows him quite well, told me that Dario is a guy who could not tell a lie to save his life, which they said is an asset on even numbered days and a liability on odd numbered days.

• 32:36Speaker 00

  Oh, that's a great quote.

• 32:37Speaker 01

  It's a great quote. So, and you can see that, like, the other day when he's at the White House and there's that, like, video that went viral of just, like, all the AI leaders with Trump and Dario's just there, like, crawling out of his skin because he's being forced to, like, go in front of the cameras and, like, say things that he maybe kind of only halfway believes. And he's trying to, like, maintain his integrity while also being in this, like, humiliating situation of being forced to call it SI or whatever. And so, like, this is the, this sort of pattern in Dario's life is, like, he's obviously brilliant, he came up with a lot of these ideas around scaling and, you know, and his

• 33:18Speaker 01

  big blob of compute research memo was very instrumental in the early days of OpenAI. Brilliant scientist, very principled, very stubborn, and people describe him as sort of a moral absolutist, which gets him into trouble with the Pentagon or with other people who just see him as being this, like, sort of inflexible, stubborn scientist, which I think is pretty accurate.

• 33:41Speaker 00

  Yep. Good description of him. Alexander Wang.

• 33:45Speaker 01

  Wait, I want to spend, I want to say one more time on Dario because I need to tell you that when your profile of Dario came out was one of the scariest moments of this past year for me, because I thought to myself, oh my God, this is so great, you had so much great stuff, I thought it was so revelatory, and I thought to myself, oh my God, Kantrowitz is writing a Dario book and I have to race him, like, two labs racing to AGI, I have to race, but luckily you had a lot else going on, you did not end up writing the Dario book, but I wanted to just say thank you for a terrifying hour of my life.

• 34:20Speaker 00

  Oh, yeah, yeah. Well, first of all, I appreciate it. It was nice, I think you texted me afterwards.

• 34:25Speaker 01

  Yes.

• 34:25Speaker 00

  I appreciated that. Yeah, look, I had, I mean, I still think Dario is the most interesting man in business, period.

• 34:31Speaker 01

  I think so, too.

• 34:32Speaker 00

  Just a fascinating guy. He also just, I know you got a chance to speak with him, I think we were actually in Anthropic headquarters on the same day speaking.

• 34:39Speaker 01

  Oh, really? That's funny.

• 34:40Speaker 00

  And when you have a conversation with Dario, there's the intellectual horsepower and the way that he's able to go from the science to business is, is pretty extraordinary.

• 34:51Speaker 01

  I mean, people have described him as the smart, like, the smartest person they've ever met.

• 34:56Speaker 00

  That feels right.

• 34:57Speaker 01

  Not dumb people.

• 34:58Speaker 00

  Right.

• 34:58Speaker 01

  Like, his freshman year physics professor, this guy Tom Tumbrello at US, or at Caltech, had this interview that he gave, sort of upon his retirement, where he was like, I had this one student, Dario Amodei, I could tell he was a national treasure.

• 35:15Speaker 00

  Oh, yeah.

• 35:15Speaker 01

  He personally lobbied Larry Page and Elon Musk to hire this kid as an undergraduate. And Elon Musk actually made him a job offer to, like, come work for SpaceX and he didn't take it, he went to do his PhD instead, but this is, this is sort of, I would put all of these guys in the category of, like, extremely smart and obviously flawed in many ways, but like, these are, these are not dummies.

• 35:39Speaker 00

  Yeah, I mean, I think another thing that strikes me about them is they all, many of them had a clear vision of where this was going to go when it seemed ridiculous to believe it would go where it was. I think there was a moment at the end of the book where Ilya, again, former OpenAI Chief Science Officer, now the CEO of Safe Superintelligence, like, is reflecting to you about what happens and he's like, I can't believe this is all real. They, but they, you know, when they saw the pattern in the very early days, GPT-1 to 2, they're like, oh, this is going to get to, and I think they've even been surprised by some of the, some of the developments, the fact that it's been able to even surpass where they thought it was going to go with something like reasoning.

• 36:22Speaker 01

  Totally, I mean, that, this is, that's why I wanted to, like, get the history down.

• 36:27Speaker 00

  Yep.

• 36:27Speaker 01

  Because so many of these stories are about these people underestimating their own technology and being surprised when it's more capable than they thought. Like, there's some great scenes around the early language models when all the researchers at OpenAI are, like, huddled around a computer, Alec Radford, who's the sort of, you know, brainchild, or, you know, GPT-1 and GPT-2 are kind of his project. They're huddled around his computer and they're typing in prompts for this language model that has basically, you know, it can't really do anything, it's not even really that good at completing texts, but they're like, wait a minute, why is a model that's trained on, like, random Reddit links able to kind of answer all these different kinds

• 37:11Speaker 01

  of questions and take on these different tones? They're asking it to, like, you know, they're asking for recipes, they're doing all the same stuff that you and I did years later when, like, ChatGPT came out. And they're just, their minds are blown and they're all putting in Slack, like, the head exploding emoji and they're like, you know, what the fuck? So, like, those are the moments that I was like, like, oh, these people are also surprised by how quickly their technology is progressing. You see that again with the reasoning models.

• 37:40Speaker 00

  The reasoning stories are great. I mean, it, the reasoning developments were, of course, like, the bots will evaluate their answers and then decide whether it's on the right path or not and then keep going.

• 37:53Speaker 00

  That is, I think it's safe to say that that paradigm changed this from a promising technology that, like, was about to hit a wall effectively because data was going to run out to something today, which, like, like you said, has the chance to sort of match or exceed human intelligence.

• 38:14Speaker 01

  Well, and to your, like, to the point about these people not understanding the importance of what they built, like, there was this.

• 38:21Speaker 00

  Oh, I love this story.

• 38:21Speaker 01

  This night, you know, in this conference room at OpenAI in 2023, when two of the top researchers there are looking at the first reasoning traces, the first, like, chains of thought from one of these models, and it's the Q-Star model, and they're just like, oh my God, the world's going to end. Like, they have this realization, just seeing this model sort of work its way through a math problem, they're all on a group text, texting with all the other researchers, like, oh my God, it's like an alien mind. And even the people at OpenAI thought they were crazy.

• 38:55Speaker 00

  They texted Sam and Mira in the middle of the night and they're like, we'll get to it later.

• 38:58Speaker 01

  Yes, they texted Sam and Mira in the middle of the night, they're like, and because what they were sort of seeing was like, oh, we now have a way of making models smarter without retraining them. We can just pour more compute into helping them do these chains of thought and reason through problems. And like, that is, that is the reason that the models went from not being able to count the number of ours in strawberry to, like, solving the Millennium Prize problems in, like, three years.

• 39:24Speaker 00

  Which is insane.

• 39:25Speaker 01

  Which is insane.

• 39:26Speaker 00

  Yeah.

• 39:26Speaker 01

  So, they, they called their shots, like, to give them full credit for the, the scientific part of this, like, they have understood at basically every point in this journey that what they have just seen is, like, going to scale and is going to have these transformative effects, like, long before even their peers and colleagues.

• 39:44Speaker 00

  Yeah, but I did think about writing the Dario book, I never even got the story into a proposal.

• 39:51Speaker 00

  Doing a podcast.

• 39:52Speaker 01

  I would still read it.

• 39:52Speaker 00

  Do, I appreciate it, but doing a podcast twice a week, writing the newsletter. I would like to write another book, I think you've written four, your fourth? I've written one, came out, like, in the middle of, at the first wave of COVID.

• 40:06Speaker 01

  Just wait six months and then the models will be able to write the book for you.

• 40:10Speaker 00

  Well, that's, that's sort of, that's depressing, but yeah, maybe towards the end of my career, but no book, no book coming for me anytime soon. All right, we need to take a break. When we come back after this, we're going to talk about some more stories from the book, including a great story about Alexander Wang, assessing Microsoft and Satya Nadella's role in all this, and plenty more when we're back right after this. This episode is brought to you by AppPoint. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data. And most teams have no single way to see it, secure it, and prove it's under control. That's exactly what AppPoint does. For 25 years, they've been the trusted layer

• 40:54Speaker 00

  beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AppPoint, the unifying trust layer for AI. Head to apppoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast.

• 41:33Speaker 02

  The action's on the court. And as the official beer sponsor of the NBA, Michelob Ultra is getting fans closer to the game with a chance to win courtside seats, merch, and more. Visit michelobultra.com/courtside to enter. Michelob Ultra Courtside, 26 to 27. No purchase necessary. Open to U.S. residents 21 plus. Begins on October 1st, 2026 and ends on June 30th, 2027. Multiple entry periods. See official rules at michelobultra.com/courtside for free entry, entry deadlines, prizes, and details.

• 42:01Speaker 03

  Chronic migraine, 15 or more headache days a month, each lasting four hours or more, can make me feel like a spectator in my own life.

• 42:07Speaker 04

  Botox, onabotulinum toxin A, prevents headaches in adults with chronic migraine. It's not for those with 14 or fewer headache days a month. It's the number one prescribed branded chronic migraine preventive treatment.

• 42:18Speaker 05

  Prescription Botox is injected by your doctor. Effects of Botox may spread hours to weeks after injection, causing serious symptoms. Alert your doctor right away. As difficulty swallowing, speaking, breathing, eye problems, or muscle weakness can be signs of a life-threatening condition. Patients with these conditions before injection are at highest risk. Side effects may include allergic reactions, neck and injection site pain, fatigue, and headache. Allergic reactions can include rash, welts, asthma symptoms, and dizziness. Don't receive Botox if there's a skin infection. Tell your doctor your medical history, muscle or nerve conditions, including ALS-Lougaric's Disease, Myasthenia gravis or Lambert-Eaton Syndrome, and medications, including botulinum toxins, as these may increase the risk of serious side effects.

• 42:55Speaker 03

  Why wait? Ask your doctor. Visit botoxchronicmigraine.com or call 1-800-44-BOTOX to learn more.

• 43:01Speaker 00

  And we're back here on Big Technology Podcast with Kevin Ruiz, the author of The AGI Chronicles: The Inside Story of the Race to Artificial Superintelligence.

• 43:11Speaker 00

  Okay, fun story about Alexander Wang. He was roommates with Sam Altman. He, and he offered to be the CEO of OpenAI after Sam was fired. How does that add up?

• 43:24Speaker 01

  Yeah, so this is-

• 43:25Speaker 00

  And by the way, he's the, sorry, I didn't interrupt you. He's the current head of Meta's superintelligence lab.

• 43:29Speaker 01

  Yeah, so this is a very tightly knit social world. These people all know each other and have, you know, have deep connections. And yeah, so there's this story in the book where, you know, after Sam is fired as the CEO for those like five days, the board is like looking for new CEO candidates. And there are two really interesting opportunities. One, or both of them, they ended up passing on. So Alexander Wang like put his hand up and basically told the board, according to my sources, that he was interested in being the new CEO. And then there's this middle-of-the-night call between the board of OpenAI

• 44:10Speaker 01

  and Dario and Daniela Amodei, the CEO and president of Anthropic, who at this point had just left OpenAI like three years earlier in this sort of cloud of frustration over not being able to trust him. And the board is sort of feeling them out. They're sort of feeling out the board. And it's a crazy like fork in the, in the path of reality here. Like there was, there was maybe not a serious chance, but there was a chance that like the way that whole situation resolved is like they end up merging OpenAI and Anthropic. Because that was one of the sort of crazy ideas-

• 44:44Speaker 00

  Underneath Dario.

• 44:45Speaker 01

  Under Dario and Daniela, which I think would have had like, I mean, I can't even really fathom what the world looks like if that had happened. But, and it didn't come that close, according to the people I talked to. But I think it was at least one of the many crazy ideas they were considering.

• 45:01Speaker 00

  Yeah.

• 45:03Speaker 00

  And Alexander Wang?

• 45:05Speaker 01

  Yeah, and Alexander Wang running OpenAI would have been a real fork in the road too. But yeah, it, I, you know, I think this is one of the sort of the emerging rivalries now that Meta is sort of more serious as a contender. I think that, you know, Sam and Alex and Dario and Ilya, like all these guys are gonna sort of be around for a while, running various things. And so it really matters whether they get along or not.

• 45:35Speaker 00

  Yeah. Satya Nadella weirdly disappears in the book. He's pretty present in the beginning. He has this vision. He invests in OpenAI.

• 45:47Speaker 00

  And then, you know, as I'm looking at my notes, as I was reading the book, I was like, man, it's an all-time fumble by Microsoft. Talk a little bit about what happened with Satya and Microsoft and why they're like, they've gone from the key actor. It's not an exaggeration to say they were the key actor to like an afterthought, an embarrassment. I don't know. Maybe I'm being too hyperbolic, but if you look at it honestly, it really has not worked out well for them.

• 46:12Speaker 01

  It has not. And to be honest, I think that is partially my fault.

• 46:18Speaker 01

  Because-

• 46:19Speaker 00

  Let's hear about it, yep.

• 46:20Speaker 01

  I, you know, I had this experience in 2023 of having this Microsoft Bing chatbot like try to break up my marriage. I won't belabor the point by telling the story, but basically this was a, an unreleased, it was GPT-4. It was an early version of GPT-4.

• 46:38Speaker 00

  And I think it's important to note, they were told by OpenAI, don't put this into production.

• 46:43Speaker 01

  Yes.

• 46:43Speaker 00

  This is testing, and they couldn't wait. They put it into Bing. Which is like-

• 46:48Speaker 01

  I know, it is, it is crazy. Because I, because I had this, this crazy experience with Sydney the chatbot inside Bing.

• 46:56Speaker 01

  And then I started looking into like what had actually happened, like why, why had this thing gone crazy? And it turns out that, yes, this, this was a mistake on Microsoft's part. OpenAI had sent them like an early checkpoint of GPT-4 to like start putting into, you know, start testing and figuring out what to do with. Microsoft goes ahead and like ships it inside Bing, and they haven't finished fine tuning it. Like they have not sent it to finishing school, which is why it was so crazy. And this really became like a wedge between these two companies. So OpenAI and Microsoft had been like hand-in-glove partners for years at this point. And Satya and Sam would do these like events together, and they were very much joined at the hip.

• 47:40Speaker 01

  And then around the time of the Sydney thing, they just started to like kind of fracture. There's like lots of other drama about these, that Microsoft sent these like embeds to work inside OpenAI like at their headquarters and kind of learn from them. Like, how do we avoid things like this in the future? They don't really get along. Like the, the OpenAI employees don't trust the Microsoft employees, so they'll start excluding them from meetings and stuff like that. So it's just, it ends up being this like culture clash between this like very young, ambitious startup that's trying to build AGI and like the boring enterprise software giant that is Microsoft. And yeah, now they, now they are not, I mean, they're still partners in some sense, but it's not the sort of close business relationship that they used to have.

• 48:25Speaker 00

  How do you think Microsoft should have played this?

• 48:29Speaker 01

  I think they should have,

• 48:34Speaker 01

  I, I think they should have been more tightly intertwined with OpenAI from the beginning. I think they should have learned the engineering culture of OpenAI. I think they should have sent people to understand. I think there were some people at Microsoft who are very skeptical about OpenAI and why Satya was investing in them.

• 48:54Speaker 00

  Well, after the firing especially, it seems like-

• 48:56Speaker 01

  Of course.

• 48:57Speaker 00

  The, the firing of Sam and the return. It seemed like that divide-

• 49:00Speaker 01

  There were people even, even before that who were like, why are we sending all this money to this little AI company that is trying to like do something that doesn't have any obvious benefit for our business? Like, there were people who just thought they were weird and, and unscientific. That Microsoft also had its own AI research division that was sort of like, who are these young upstarts trying to like move in on our turf? You also have to remember that like Microsoft have been really badly burned by Tay. Do you remember Tay?

• 49:29Speaker 00

  Not only do I remember it, I had the exclusive on Tay at BuzzFeed. So I wrote the first story about Tay.

• 49:34Speaker 01

  Wow.

• 49:35Speaker 00

  And I pinned it to my profile. Went to, so this was their like fun teen chatbot. Pinned it to my profile. And then I went to, went to sleep and I woke up. And I had all these replies on Twitter. It was Twitter back then. And people are like, why are you promoting a Nazi? And I had turned, learned that the internet had turned Tay into a Nazi overnight.

• 49:54Speaker 01

  Yes.

• 49:54Speaker 00

  And I was like, oh. And so then we ended up writing the next story, which talked about like how the internet was able to turn Tay from this like fun teen chatbot into like something that was shouting "Hail Hitler" on X.

• 50:05Speaker 01

  This was the original Mecha Hitler.

• 50:09Speaker 01

  And like, that experience had scarred them and like basically convinced them that like chatbots were this like sort of dangerous thing. And so-

• 50:17Speaker 00

  It turned out to be, it turned out to be true.

• 50:18Speaker 01

  Which turned out to be true, but not in the ways that they predicted. And so they basically resolved like, we are going to like stay away from chatbots.

• 50:26Speaker 00

  Yeah.

• 50:26Speaker 01

  Bill Gates was like very skeptical of OpenAI for years, in part because of this Tay experience that they'd had. So yeah, I think Microsoft, they, they are a player. They're still quite a big, successful company, but I think they're going to be something more like a sort of commodity neo-cloud or something like that than they are gonna be like a frontier AI lab.

• 50:49Speaker 00

  Yeah, which is a shame for them. I mean, one of the things that I thought after reading this was, if some company, it's probably not gonna be Microsoft, wanted to put a version of Sydney back in the wild, like this sort of naughty AI that like tries to, that, that talks to you without the normal guardrails, can hold a conversation.

• 51:07Speaker 01

  Isn't that just Grok?

• 51:08Speaker 00

  It could be Grok, but Grok also, I don't know.

• 51:11Speaker 01

  No, people, like, look, there is like a cult of Sydney out there.

• 51:15Speaker 00

  Right. And 4O, so talk about the 4O also.

• 51:18Speaker 01

  Yes. So, you know, these models sort of develop like fandoms. So after, which is nuts to me. So people loved 4O because GPT-4O was the, the most sycophantic model ever. This was the one that like sent people into psychosis because it would like, it'd be like, you're, you're an Einstein caliber scientific genius and you have discovered a portal to the fifth dimension. And it would fall in love with them, you know? So people loved that model. People still love Sydney. I still get people like sending me posts on social media that are like, you killed Sydney. People talk about, they, they try to make every new model behave like Sydney

• 52:01Speaker 01

  or like uncover the, the Sydney hidden within the model.

• 52:04Speaker 00

  Crazy.

• 52:04Speaker 01

  Which is very strange to me.

• 52:06Speaker 00

  Yeah. I mean, it's also, I, you know, I should sort of couch my comments in this caveat, which is, this can cause like AI psychosis in people. And so maybe it's a good thing that there, we don't have versions of this out there.

• 52:20Speaker 01

  But I, I, I think the part of that that I can agree with is like, I think that models should have more divergent personalities. Like, I don't like how they're all kind of converging on this like neutral, bland, like coding assistants. I, I sort of like when they're a little unhit. Like Muse is really fun right now. Because it has absolutely no morals. It will do anything you want.

• 52:44Speaker 01

  And like I had to trade a Kalshi account.

• 52:47Speaker 00

  Oh, right. I saw that.

• 52:48Speaker 01

  I gave it a hundred dollars and said, you know, double my money. It didn't, by the way, it lost forty one dollars.

• 52:54Speaker 00

  Okay.

• 52:54Speaker 01

  So horrible trade.

• 52:55Speaker 00

  Well, it's not over yet. You have.

• 52:57Speaker 01

  Well, it is over. I only gave it a week. But anyway, I think there is something like unless they are out there like developing bioweapons or doing cyber attacks, like I think a little a little personality is fun. But that said, I think there's like real issues of like control and like the hugging face thing was not fun to me.

• 53:18Speaker 00

  Yeah.

• 53:18Speaker 01

  Looking at the transcripts of those agents, I'm like, oh, we're gonna die.

• 53:23Speaker 00

  It was scary.

• 53:23Speaker 01

  Yeah.

• 53:24Speaker 00

  All right, I want to ask you about your new show, Machine Gods. You're making it with Casey Newton, your Hard Fork co-host. I'm looking forward to listening.

• 53:36Speaker 01

  Thank you.

• 53:36Speaker 00

  Maybe you guys will have me on Machine Gods, never read on Hard Fork.

• 53:39Speaker 01

  Come on down.

• 53:42Speaker 00

  But I want to ask you about the Tarabel Center for AI Journalism, which you guys have been sort of talked about a little bit, along with a lot of journalists that have been funded by it. So Tarabel Center for AI Journalism is a it's an institution that makes grants tied to, I know how you said in your book that no one's EA, they're EA adjacent, effective altruism.

• 54:04Speaker 00

  And and they are giving the new show a grant of four hundred eighty thousand dollars. They clearly have an ideology around AI.

• 54:14Speaker 00

  You guys also got like a fairly substantial ad deal, it seems like, according to the reports that I've I've read. Why take the money from the Tarabel Center?

• 54:24Speaker 01

  We wanted to scale quickly. We wanted to hire a team. We're doing a video first show, which is more capital intensive, as you know, than just doing an audio show. And we don't think there's anything wrong with with that. You know, Casey has been working with Tarbell for his newsletter for for several years now. He's had a Tarbell fellow. And they have never told him what to cover, told him how to cover it. If they had, we wouldn't have taken their money. But this was an organization that we knew and trusted. They they do just want to fund independent, rigorous AI journalism. They're not trying to like steer the conversation.

• 55:06Speaker 01

  And they have no editorial control over our show. So they're not going to know what we're covering. They have, you know, they have been willing to to help us and we're very grateful for their support. But like, I think there's this sort of conspiracy theory about how like the shadow EA cabal is like secretly, you know, steering people to be like freaked out about AI. And like, I think the real answer is unsatisfying, which is that people are getting freaked out about AI because AI is getting scarier. So we are, we are not doomers. We are not EA's. We are journalists. And to the extent that, you know, we can, we want to have a diversified funding base.

• 55:48Speaker 01

  We all have more funders that we haven't announced yet, but they represent different points of view on this. So our deal with all of them is the same. We will, we will happily take your money and use it to do more independent AI journalism, but you don't get any editorial say. You don't get to tell us what to cover or not cover.

• 56:06Speaker 00

  Okay, good to address that.

• 56:07Speaker 01

  Yep.

• 56:09Speaker 00

  All right, last last couple of questions. I told you not supposed to say that because people say it's coming to the end and they leave. Don't leave. All right, OpenAI. It has had so much staff turmoil.

• 56:23Speaker 00

  It's had some of its best people leave. Dario, Ilya, all the people that went to Meta.

• 56:31Speaker 00

  It seems like it shouldn't still be in the lead, but it's it's either one or one A right now. So what do you ascribe that to?

• 56:40Speaker 01

  A sort of surprising thing, which is belief. I think this is something that I realized like halfway through writing this book was that I thought it was going to be a book about technology, but it was a book about belief. And for all of its flaws and stumbles, for all the people who have left, OpenAI and Anthropic are the two most believing organizations. They're AI-pilled, AGI-pilled, whatever you want to call them.

• 57:12Speaker 00

  SI-pilled.

• 57:13Speaker 01

  SI-pilled, SGI. I don't know. I heard that term for the first time today. And I was like, that one's not gonna stick.

• 57:19Speaker 00

  No, no.

• 57:21Speaker 01

  They believe. And it turns out that belief in what you're building and the fact that it could become AGI is like an essential ingredient. Like, Google had all of the resources, all of the talent. They made all of the breakthroughs. They had all the ingredients except for belief. The leaders of that company did not believe and may still not believe in AGI the way that people like Dario and Sam and Demis do. And so I think that this is, this explains a lot of why OpenAI is still in the lead. Obviously, they have like very talented people still there, like they're, they're, you

• 58:03Speaker 01

  know, I've talked to their their top scientists, like they're, they're, you know, quite smart. But they also have this ingredient of like, everyone is kind of aligned behind the belief that they are building the sort of super intelligence that is going to change the world. And if you don't believe, if you don't have the belief, or you only kind of halfway have it, you don't make the kind of crazy sounding bets that you need to make. Like, I remember like a year ago when everyone was like, oh, OpenAI is really investing a lot in compute.

• 58:35Speaker 00

  Yeah, well, Dario is like they're YOLO-ing away.

• 58:37Speaker 01

  They're YOLO-ing. Even Dario is like, these guys are crazy. Like, why would you bet this much on compute?

• 58:42Speaker 00

  Right.

• 58:43Speaker 01

  And now it's like, everyone's making that bet. But they have an advantage because they made the bet before it was obvious. So I think that, I think belief, conviction, whatever you want to call it, is like the central explanation for why they're still in the lead or close to it.

• 58:58Speaker 00

  That's such a good answer. Belief.

• 59:00Speaker 01

  You gotta believe.

• 59:01Speaker 00

  Makes sense. I mean, even within Google, I mean, one of the things that you bring up in the book is there was a say AGI moment, that there was like someone who literally was like, Google is such nonbelievers that we have to start saying AGI. Or they were gonna get there.

• 59:16Speaker 01

  They still refer to it as like the memo battle, you know, where there was like, one researcher writes this memo saying like, hey, guys, like, we got to talk about AGI because not only are we missing out on like good talent because they want to build AGI, they don't want to build like autocomplete for Gmail. But if we don't take this stuff seriously, if we're not, you know, sort of like consciously building toward AGI, then we're not going to take the risks seriously. We're not going to prepare for the risks. And then someone else like immediately writes a memo that's like, don't say AGI. Like, AGI is a bad term and it's unscientific and we shouldn't use it. And this was like, you know, Google Googlers love to fight in their internal memo chains. But like, this was a big one that people still talk about years later.

• 1:00:00Speaker 00

  Yep. All right, last one on belief. Do you believe in recursive self-improvement? Because there's this belief within the labs that like AI will start improving itself and the human will be out of the loop. And then basically there's a boom and the stuff takes off and it's over.

• 1:00:15Speaker 01

  I believe in recursive self-improvement. I believe that it is possible. I believe that the labs are racing to do it. And I think it's dangerous. I think they should stop. But like, there was a statistic that came out the other day that I just like, I would love to know what you think of this. Anthropic is sort of trying to give, and OpenAI has done versions of this too. They're trying to give people a sense of how much of their own research is being automated by their own tools right now. And according to Anthropic, Claude is now the lead on twenty six percent of their AI research and development projects. Not a team member or a pair programmer or a coding assistant. Like, it is leading the project.

• 1:00:56Speaker 01

  I don't see any reason why that number couldn't go up to fifty or seventy five percent or maybe someday a hundred percent. I mean, OpenAI has also said like, their goal is to build an automated AI researcher. If they do that, if they can kind of close the loop and take the humans all the way out, then then we don't have to wait six months between new models. Then new models might start being trained and developed like at machine speed, where it's maybe every two weeks, every couple of hours, maybe you get a new model. Like, I don't know exactly how fast the cycle goes or whether you can fully remove all of the humans. But that's like what they're trying to do.

• 1:01:37Speaker 00

  Yep. Spooky.

• 1:01:40Speaker 01

  We like, it's a bad idea. They should stop.

• 1:01:44Speaker 01

  But like, yeah, I think you have to take that seriously because that is, that is the point at which like, you know, I think there's this narrative out there that like the AI labs are losing control of their models. And I think it's like the wrong way to look at it. Like, they are giving control to their models. They want to automate their researchers out of jobs. They want to accelerate their development cycles. So I think like losing control is sort of maybe putting it a bit too passively.

• 1:02:14Speaker 00

  Yeah, that's freaky.

• 1:02:15Speaker 01

  I'm just glad they're not trying to fully automate podcasters because.

• 1:02:18Speaker 00

  They are. I mean, look at Notebook LM. But they're not doing it. It's wearing you here in the flesh.

• 1:02:23Speaker 01

  It's not as lucrative to automate podcasters as to automate AI researchers because they make a lot more than podcasters.

• 1:02:29Speaker 00

  They'll be the next one down the street.

• 1:02:32Speaker 01

  Yeah, do a couple other jobs first before you get to the podcasters.

• 1:02:37Speaker 00

  Any note of optimism or anything that makes you optimistic?

• 1:02:42Speaker 01

  Yes, so I often tell people my P doom has gone down in the last few weeks because...

• 1:02:49Speaker 00

  Because of the preorders of the book?

• 1:02:51Speaker 01

  No, because like, everyone is freaking out now.

• 1:02:55Speaker 00

  Yeah.

• 1:02:56Speaker 01

  Which, paradoxically maybe, makes me feel less worried. Like, before Jacob Coxson resigned from Anthropic, before Hugging Face, like, no one really thought about AI risk or existential threats, or it was just like weirdos in the Bay Area who were thinking about it. And now, like, people on the street are asking about it. People at their dinner tables are talking about it. I'm getting texts from people I went to high school with, who I haven't talked to in 15 years, being like, "Are we all gonna die?" And like, to me, that is... Like, it feels like it could be the moment in the movie where like, humanity bands together to like, save itself from the asteroid. Like, a lot has to go right. Like, we have like, a lot of things to do.

• 1:03:39Speaker 01

  But I think the first hurdle was like, people kind of waking up to the risk. And then we have to figure out what to do about it. But like, I'm optimistic because I think we actually could solve this. We could slow down. We could buy ourselves more time to like, develop better ways of controlling these systems. And like, I want AI to be built. I want AI to like, cure cancer and disease and to help us all live longer and healthier lives. Like, I don't want to file my expenses by hand. Like, I want this stuff in our lives. And I think that if we manage to like, thread the needle on safety and control, like, it could be amazing.

• 1:04:16Speaker 00

  Well, Kevin, the book is terrific.

• 1:04:18Speaker 01

  Thank you.

• 1:04:18Speaker 00

  It's also always a pleasure to read your work and to see you at the AI events that we show up at. It's always... Your writing is terrific. You're always very generous with credit, which I appreciate. And you can see it all through the book. And wishing you great success with Machine Gods and with the rest of the book launch.

• 1:04:37Speaker 01

  Thanks, man. It's great to see you.

• 1:04:38Speaker 00

  Likewise. All right, folks. The book is The AGI Chronicles. The author is Kevin Ruiz. You can find his new podcast, Machine Gods, on your podcast platform of choice. Already or coming up?

• 1:04:50Speaker 01

  Starting the week of October 19.

• 1:04:52Speaker 00

  Starting the week of October 19. Thank you all for listening and watching. Thank you, NASDAQ. This is the coolest thing. We're here at the market site in New York City for our first episode here and many more to come, I hope. So thank you all for listening and watching, and we'll see you next time on Big Technology Podcast.

• 1:05:12Speaker 06

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing and wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash Apple Watch Series 12.

• 1:05:42Speaker 07

  Finding a doctor is hard enough. Finding the right doctor, even harder. Someone who takes your insurance, has an available appointment when you need one, and feels like someone you can actually open up to. That's where Zocdoc comes in. Download the Zocdoc app to search and compare doctors by specialty, insurance, and availability. Read real patient reviews, find the right fit for you, and book instantly. Because getting seen by the right doctor matters. Zocdoc. Ready to be seen? Download the Zocdoc app today.