A Conversation With Prime Minister Justin Trudeau of Canada + An OpenAI Whistle-Blower Speaks Out

It turns out A. I. is surprisingly Canadian.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure, and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner, Dell, AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 0:35Casey Newton?

  Kevin, is it true you got fired last week?

• 0:39Kevin Roose?

  No, but there was a brief moment where I suspected that I might have been.

• 0:43Casey Newton?

  What happened to you?

• 0:44Kevin Roose?

  So I got an email from someone in the tech department at the New York Times, basically saying, we're going to need all your equipment back. And then it had a big list of all the equipment that I've sort of gotten from New York Times.

• 0:57Casey Newton?

  And God knows you have soaked these people for everything they're worth. This man has a vision pro and God knows what else.

• 1:02Kevin Roose?

  And I wrote back and I said, well, that's going to be hard because I still use this stuff. And the guy wrote back and he said, so wait, you're telling me you're not being off boarded? And I said, I sure hope not. This would be a weird way to find out. And then he came back a couple of minutes later and he said, oh, sorry, the mistake's on our end. Clerical error. We don't actually need this stuff back.

• 1:29Casey Newton?

  You know, this is such a great prank and I wish I had pulled it. I should have been in touch with the IT department years ago and just said, I need you to tell Kevin he's being off boarded because it brought me so much joy.

• 1:40Kevin Roose?

  Anyway, very scary 10 minutes.

• 1:42Casey Newton?

  You know, I was a contributor at Vox Media and they had sent me a variety of things over the years to appear on podcasts. So you know, a microphone, a mic stand, whatever. And at some point, you know, they had sent me a sort of similar thing saying, please send all this back. And I thought, the amount of time it's going to take me to package all of this stuff up and send it back. Like, I absolutely do not want to do this. But lucky for me, my agreement was extended. And then when it finally lapsed, nobody emailed me. And so I still have all of the equipment.

• 2:12Kevin Roose?

  Wow, and you're admitting this.

• 2:14Casey Newton?

  And I'm admitting this. And let me just say to Vox Media, you can take it from my cold dead hands.

• 2:22Kevin Roose?

  I'm Kevin Roos, a tech columnist at The New York Times.

• 2:24Casey Newton?

  I'm Casey Newton from Platformer.

• 2:26Kevin Roose?

  And this is Hard Fork.

• 2:27Casey Newton?

  This week, Canadian Prime Minister Justin Trudeau is here to talk about tech, TikTok, and what he's doing to solve the country's AI brain drain problem. And then, OpenAI whistleblower Daniel Cocotello joins us to talk about why he risked almost $2 million in equity to warn us about what he calls "the reckless culture" inside that company.

• 2:58Casey Newton?

  Have you ever been to Canada?

• 2:59Kevin Roose?

  I have.

• 3:00Casey Newton?

  Have you had a good experience there?

• 3:01Kevin Roose?

  I did. I've had many good experiences in Canada.

• 3:02Casey Newton?

  What do you do in Canada?

• 3:04Kevin Roose?

  Well, sometimes I go to Montreal. You know, I've been to jazz festivals there. I've been to Toronto. Niagara Falls is lovely. What about you?

• 3:14Casey Newton?

  Well, I had once had a lovely layover in the Montreal airport. And I said, "This country's on to something, and I want to learn more." And today, we're actually going to do that. You know, Kevin, a few weeks ago, we got one of the strangest emails of our entire lives as podcasters.

• 3:26Kevin Roose?

  Yes.

• 3:26Casey Newton?

  Where what we assumed was a spam email that purported to come from the Office of the Prime Minister of Canada came in, and it said that the Prime Minister might want to have a conversation with us.

• 3:34Kevin Roose?

  Yes, and this was shocking to us because why would the Prime Minister of Canada want to come on an American tech podcast? But it turns out, he's been thinking about tech and AI for a very long time. He's a big science fiction nerd. He's also done a lot to try to position Canada as a leader in AI and technology development in general. And we should say, like, Canada has a very long history with AI. In fact, a lot of the major advances in AI over the past two or three decades have come out of Canada and Canadian universities. So when we got the offer, we thought about it for a little while, and we said, "Yeah, that sounds like actually a really great conversation."

• 4:14Casey Newton?

  I didn't think about it for all that long. I thought it seemed like a good idea.

• 4:18Kevin Roose?

  Yeah, so here today, we are going to be talking to Prime Minister Justin Trudeau.

• 4:23Casey Newton?

  That's right. He's actually, he's only the second head of state to ever appear on Hard Fork after Sundar Pichai of Google. Which I thought was an interesting statistic.

• 4:32Kevin Roose?

  All right, let's bring him in. This is Prime Minister Justin Trudeau.

• 4:40Kevin Roose?

  Prime Minister Trudeau, welcome to Hard Fork.

• 4:42Speaker 03

  Oh, it's great to be here.

• 4:44Kevin Roose?

  If you're wondering why we look like this, it's because I told my friend who is Canadian that we were interviewing you, and he sent over a collection of Canadian paraphernalia to put in the studio today to show our respect. So in addition to our hockey jerseys, which we have on, we also have some hats behind us. We have Maple Leafs and Raptors. We've got a CBC Radio belt buckle here. Got some Canadian money and a bottle of Wayne Gretzky's 99 proof whiskey.

• 5:12Casey Newton?

  Mostly drunk.

• 5:13Kevin Roose?

  Yeah.

• 5:13Speaker 03

  I was gonna say, how early did you guys start on that?

• 5:17Casey Newton?

  We've been preparing all night for this.

• 5:18Speaker 03

  And what should I be bracing myself for?

• 5:21Kevin Roose?

  So did we miss anything?

• 5:23Speaker 03

  You missed a little bit of Montreal Canadiens paraphernalia or the Expos, but that's okay.

• 5:30Kevin Roose?

  You'll forgive us?

• 5:31Speaker 03

  I'm enough for that, and I've got stuff in my background on that.

• 5:34Kevin Roose?

  Great.

• 5:35Casey Newton?

  Well, so we should talk a little bit about how this moment that we're having came about a few weeks ago. We got an email from a very nice man in your office saying that we might want to have a conversation. And of course, I panicked and I thought, is this because I stopped watching hockey after the L.A. Kings lost in the 1993 Stanley Cup Finals? But they tell us that you've actually been spending a lot of time thinking about tech these days, and specifically AI. So maybe we should just start there. Tell us, Prime Minister, how are you thinking about AI right now?

• 6:01Speaker 03

  Well, it's not that I've been thinking about it right now. I've spent my life thinking about it from reading old Asimov as a kid to studying a few years of engineering to being basically a science nerd and geek most of my life, despite the fact that I went into English literature and teaching and now politics. I've always been fascinated by the intersection between technology and real life and how we take things for granted, the big transformations from one industrial revolution to the next, and how rapidly the pace of change is going. So thinking about that stuff while I'm trying to build a fairer...

• 6:48Speaker 03

  More just, more equal society for everyone is is sort of a natural bridge for me.

• 6:52Kevin Roose?

  Yeah. One of the reasons we were excited to talk to you about AI today is because I think AI is surprisingly Canadian. Like, a lot of the very foundational research in AI from people like Geoffrey Hinton, Joshua Benjio, sometimes those two get called sort of godfathers of AI, that came out of Canadian universities. Ilya Sutskever, who just left OpenAI, did a lot of his formative AI research in Toronto. And every time I go to an AI event in San Francisco, I meet a bunch of young Waterloo engineering grads who are working on this stuff. So why do you think Canada has been so excited about AI for so long?

• 7:29Speaker 03

  Well, we we had an approach, and Jeff Hinton really is at the center of this, where as a country, as a government, we've always done scientific research really, really well. We have not always great at commercializing, at building platforms, at scaling a great idea up into a world dominating technology. But so many of the breakthroughs happen because we fund our scientists well. And in AI, through the long AI winter, where a whole bunch of countries around the world were chasing different shiny objects, Canadians just kept sort of powering away, supported by our

• 8:11Speaker 03

  our our science ecosystem, to be able to get through it in a way that that has revolutionized the world. So we're we're proud of Canada's early role in in developing AI. We're right now fighting to make sure we keep our skin in the game, not just because it's so important and competitive, but because I do feel that the way Canadians approach things is is a framework that we need to make sure that is part of AI as it develops.

• 8:45Casey Newton?

  Now, I'm sure that you would love to keep more of the AI talent that you're developing there in the country. Lately, it seems like a lot of the best researchers have come to do their work here in the United States. How do you keep them in Canada?

• 8:57Speaker 03

  One of the things we did in our most recent budget is put in about a $2.4 billion compute strategy designed to increase the investments and the accessibility of computing power for small businesses, for researchers, for academics. Canada does have a few advantages if you're comparing a data center in Canada to one in Texas or in California. Our temperatures being much lower on average means less cooling. Our access to clean, green electricity at cheaper costs and reliable is good. Our data regime, our international connections means that a lot of people are looking at

• 9:39Speaker 03

  Canada as a smart place and a stable place to invest when certain places around the world are getting a little more uncertain and complicated in their various strategies.

• 9:50Kevin Roose?

  But how do you address the brain drain of the talent specifically? Because a lot of these people, they build some AI stuff or they do some AI research in Canada and then Google comes to them or Microsoft comes to them or Meta comes to them and offers them a huge pile of money to move to the US or sell your startup to us. And it sort of all ends up kind of enriching the US tech ecosystem instead of the Canadian one. So what about the specific, maybe call it the top couple of hundred people in tech in Canada? How do you keep them in Canada?

• 10:20Speaker 03

  Well, a lot of what we're trying to do is make sure that there is better access to capital. We've created opportunities so people can scale up a $10 million company to being a hundred or even a billion dollar company without having to sell out to a billion dollar company already. Those are things that lots of different places are struggling with. But I think one of the things that is Canada's advantage comes down to our access to talent from around the world, the fact that it's easier to immigrate to Canada when you have tech abilities than just about any other country. That lots of, I remember talking with some folks in Silicon Valley, saying, wow, if you can guarantee me a quick visas for engineers, I'll keep opening engineering.

• 11:07Speaker 03

  Centers in in Canada and we took them up on that. We created a three week global skills immigration program that that has a lot of people coming in and setting up shop. Ultimately, if people are just driven by money, there are places you can make a lot more money than in Canada. But if the the schools your kids are going to, if the community you live in, the quality of life, the opportunity to have so many of the advantages of of a developed advanced nation without some of the inconvenience that come with with some other places, you know, Canada still has a very, very strong pitch to the world.

• 11:41Casey Newton?

  So you mentioned that you had announced a package of 2.4 billion Canadian dollars to fund Canadian AI research labs and get people access to AI compute there. This is a world where the Saudi government might be spending 40 billion dollars on AI soon. Sam Altman is out there saying he might need 7 trillion dollars to build AI chips. I'm very curious if you're worried at all that some of the liberal democracies are going to get outmatched when it comes to trying to keep a handle on this stuff if they have enough money to to really compete here.

• 12:12Speaker 03

  If it's only about money, then I think we have a larger problem. I think one of the things we've seen as different countries are trying to make advances in AI. I mean, take China, for example, where their access to data because of lack of privacy protections for civilians, for individuals and citizens, means that they have massive amounts of data that they can scrub through that is usually, you know, pretty good data. But that's a part of the ingredient. A lot of the other part is, you know, the brilliant minds that can think around corners that you bring together diverse, creative, well-educated folks who are just pursuing this for the the love of creativity and to find out where they're going.

• 12:57Speaker 03

  That's harder for authoritarian states or money to buy. And even if you you purchase the best academics and researchers around the world and, you know, set them up in a suburb of Dubai, you don't have the kind of dynamism around you that is going to deliver the same kinds of things. That's what I believe. I believe the ecosystem that you're actually living in is as important as the ecosystem that you work in. So we need to make sure we're investing to be competitive so that someone's not saying, well, either I can starve and stay in Canada, or I can go make kazillions by by going somewhere else. It should be a little trickier decision than that of, okay, I would make more money, but I don't know if I want to be part of that.

• 13:42Speaker 03

  I don't know if I have the same quality of life. And I think that's the pitch that we have to keep making. And quite frankly, it's the kind of thing I'm hearing from a lot of young people these days for whom everything around their life matters as much as the dollar figures they're making in in success. That value in life is is different, and that's certainly something we're building on in Canada.

• 14:03Kevin Roose?

  I'm curious, you laid out this $2.4 billion package to do what you called securing Canada's AI advantage. I'm very curious what it means to you to have an advantage in AI.

• 14:14Speaker 03

  Well, I think that that catchphrase is partially the fact that we had and have an advantage in that, as you said earlier, we were part of of the creation and the foundation of modern AI. I think the advantage that we come at are a series of things. First of all, the incredible diversity that's built into Canada, the ability to make sure, I mean, so much talk is about the black box of AI that is going to take on the biases of whoever actually programs a given algorithm, making sure that you do have a greater range of experiences and diversity as as Canada has in building those algorithms. And I don't think Canada wants to become the only AI powerhouse in the world. But Canada has been

• 14:58Speaker 03

  able to develop real expertise in terms of cyber expertise or communications, security and establishment, it's one of the best in the world. Some of the things we do are of such value to our partners that even that we're not the biggest, we find these niches where we can be really, really impactful in positive ways. I think that's part of Canada's advantage.

• 15:18Kevin Roose?

  But like, what do you want AI to do in Canada? There's such a wide range of predictions about what AI is going to be in five years, 15 years, 20 years. Like, what what is your vision for it?

• 15:28Speaker 03

  I think one of my biggest preoccupations, because we know technology can be positive or negative, how can we maximize the chance that it actually leads to better outcomes and better lives for everyone? We see always a concentration of new technologies in the pockets of those with the deepest pockets, and that can go through a period of accentuating wealth inequality and a feeling of being left out or left behind by large swaths of population. If we can be part of democratizing it, of making sure that people are comfortable with the opportunities and advantages it gives, and that it's helping everyone move forward

• 16:13Speaker 03

  and not just those who own the algorithms or the computing power, that's what I most like. What it's actually gonna do, I think it's gonna be doing bits of everything, I think, in the coming years, but the impacts on society, if it creates opportunities for people to succeed, if it lifts people out of poverty instead of exacerbating differences between haves, have-nots, and have-yachts, then we're on the right track.

• 16:40Kevin Roose?

  I love have-yachts. I've never heard that one before.

• 16:42Casey Newton?

  What's a have-yacht? Is that Canadian?

• 16:44Kevin Roose?

  That's if you have a yacht.

• 16:45Speaker 03

  There's haves and have-nots, and then there's have-yachts.

• 16:48Kevin Roose?

  That's good.

• 16:49Speaker 03

  I like that.

• 16:50Kevin Roose?

  Obviously, one of the biggest questions surrounding AI right now is about jobs and the effect on the labor market of all this new technology. I noticed in your big AI package that you are investing $50 million Canadian dollars into a bunch of efforts to support workers who are impacted by AI. So I'm just curious, how big an effect do you think AI will ultimately have on labor markets? Do you think we're gonna see millions of jobs disappear in the next few years because AI can do them better? Or do you think people will just kind of shift the mix of tasks that they do at their jobs and that there won't be kind of mass unemployment?

• 17:26Speaker 03

  I think ideally it'll be the latter. Obviously, anytime any new technology comes in, it'll be disruptive. People have to learn how to do new things. Some things that people did, you know, won't really be, you know, doable or worth doing anymore or able to do as a job. But, you know, what AI and technology in general is so good at is replacing some of the things where we're not adding our full value as an individual to a particular task. And, I mean, I sit down with my kids. I have two teenagers who are like every teenager right now using ChatGPT in their homework assignments. And the conversations that I have with them is, you know, my daughter explains to me, no, no, I use it to help me with the first draft.

• 18:11Speaker 03

  And then that frees me up to be able to really make sure that every sentence is mine and that it's saying exactly what I want. And it's adding to my ability there. And that idea in the workplace, I mean, in the federal public service, everyone's starting to use it in different ways. But there are big concerns about what that happens. And I know you guys have had conversations about that power imbalance between a manager and an employee and that productivity gain and who's going to benefit from it? And are we all going to overload people? These are the things that I hope the conversation goes.

• 18:44Casey Newton?

  I have to say, Prime Minister, that your daughter said to you exactly what I would say to my dad if he asked me why I was using ChatGPT, which is, oh, no, no, this is just the first draft. Trust me, you know? So I think we have a very savvy operator in the household here.

• 18:58Speaker 03

  Yeah, but that's also reassuring a little as well, that she can, you know, know exactly what to say to me as a former teacher to reassure me. It means at least she knows what is right and what is wrong, which is still a step forward.

• 19:10Casey Newton?

  But you were a former teacher, like, right? If you were in a classroom right now and you found out that all your students are using ChatGPT to write the first drafts, like, what emotions does that bring up for you?

• 19:19Speaker 03

  Well, first of all, I was a teacher in exactly the first generation to grapple with Google in the classroom, right? I mean, OK, Google now hacks like how? And I was, you know, one of the quick ones to say, OK, let's use this tool. And I used to do projects like have the kids, you know, find a copy of a famous poem, a disiderata, on the internet. And everyone find a different copy from Mike's homepage to the Poetry Archive or whatever. And then we'd compare it and see that there was a comma or a word different in each and every single one. And understanding that and being thoughtful about the critical thinking skills that go into it. I mean, I used to laugh when, you know, I regularly laugh because I was a math teacher telling kids, well, don't imagine that you're going to be able to wander around the rest

• 20:04Speaker 03

  of your life with a calculator in your pocket. And of course, now we all do, right? So embracing technology instead of fighting it and trying to find the human strengths within it. And what are, what are your advantages? What is it that you bring that is unique and that is your strength to whatever task you're doing? And yeah, if you can sort of automate some of it, you know, typewriters meant that people with great penmanship no longer had a huge advantage over those of us who are messy writers, right? There's always been those technologies, but instead you can focus on the content. And that's my hope for that as a teacher and as a leader.

• 20:44Kevin Roose?

  Prime Minister, what do you make of the discussion around existential risk in AI? There are some people who are very worried that this technology is going to get more powerful and more intelligent than humans and kind of take over and either disempower or maybe even, you know, harm us. And there are other people who think, well, that's all science fiction and we should focus on the risks in the here and now. So what do you make of that debate?

• 21:06Speaker 03

  Um, first of all, I think it's an interesting philosophical debate, but I don't think there's much we're going to be able to do now to prevent what's going to happen down the line. I mean, obviously, we need to be setting up the right kinds of parameters and maximizing our chance that, you know, good and thoughtful. People are engaged in creating with the right kinds of parameters around it. But there's no question that this technology is going to get more and more powerful. But the idea that, you know, we might end up with a with a sentient computer that will decide that the greatest threat to you human beings, if that's our job is to protect human beings,

• 21:49Speaker 03

  is other human beings and suddenly, you know, makes drastic social change. Yeah, that is sort of dystopian science fiction. I think there are ways we need to be responsible about how we manage and how we build in those expectations from now. But I also think that AI for good is going to be one of the most powerful tools we have to counter the AI for bad that is going to be created by bad guys out there and therefore easing off or saying, OK, well, let's slow this down because we don't understand all the consequences is probably not the right track, even though I can understand how appealing that would be to people. No, let's just figure it out and keep making sure that that those of us who have what

• 22:34Speaker 03

  we consider to be positive values and thoughtful approaches that are benevolent or at least looking for positive outcomes are fully in the game with every new step of technology.

• 22:46Casey Newton?

  I mean, one of the things that's so different about AI is that this time around, the people who are building it are the ones going around saying, hey, look, this stuff could be really dangerous. I'm curious, have people like Sam Altman, Sunita Pichai, have they been through your office or in meetings with you and what are they telling you? Are they are they ringing alarm bells?

• 23:03Speaker 03

  I had a great sit down with Jeff Hinton just a few months ago where absolutely, well, he was standing up because he likes to stand up. It was conversations around exactly how how we, you know, where we're going and how we can try and control things. Yashua Benji and I, Benjy and I were talking recently about, you know, the national security implications of AI and how we can move forward as as only governments can on some of these angles. So, yes, I've spoken with folks that, you know, in the tech industry in the US as well, like Xander and others, about about these issues and others. And the intersection between democracy and our incredibly successful, you know,

• 23:53Speaker 03

  Western capitalist democracies in creating so many of these things. But are we putting them at risk by undercutting democracy when Meta decides it doesn't want to be in the news business anymore? For example, these are things that I think we need to have continual conversations on. And Canada's in a fight right now with Meta because they don't want to support local independent journalism. And that's something that worries me as a foundational building block of democracy.

• 24:21Casey Newton?

  Well, since you brought it up, I would love to ask you about this. You know, I'm all for countries taxing the tech platforms. But, you know, in this case, in my view, you sort of broke the open web because you're taxing them just for the right to show a link. And I wonder, did you ever think about just like taxing them on their digital ad revenue or something?

• 24:41Speaker 03

  Okay, first of all, yeah, I know, I know that that's the frame that the tech companies would love to say, this is not about a link tax. That's not it at all. And that's not actually the approach we're taking. I'll point out that Meta has been for many years engaged in conversations with folks about making arrangements with big media companies. What our focus is, well, make sure those are transparent, make sure those are available to small, local journalists as well, because local news breaking really, really matters. And people who can tell local stories and not just on national platforms is a core building block of democracy.

• 25:23Speaker 03

  And the fact that Meta would rather get out of the news business, something they've been trying to do since 2020, by the way, and the fight they're having with Canada right now, they're having once again with Australia. There's this decision by Meta that it's too complicated to be in the business of supporting rigorous journalism because that's, you know, not as lucrative as cat videos or whatever it is. This is a problem because the reason that Facebook and Meta and Google and these companies exist is because countries like ours built systems in which they could benefit from incredible ...freedoms and incredible innovation.

• 26:04Speaker 03

  And that is actually at risk as we move down towards a weakening of democracy because of a weakening of journalism. So we're saying, look, if Meta is making billions of dollars in profits, some of that needs to help fund local journalism.

• 26:20Casey Newton?

  Well, I suspect we won't come to agreement on this one in the time that we have. But I do want to ask an adjacent question, which is that now that we're in this world of generative AI, you have companies like OpenAI, Google, and some upstarts, and they're all reading those same Canadian news sites, and they're scraping them, and they're sort of repurposing them, I would argue plagiarizing them without compensating those publishers. Can you see expanding the approach that you've taken to some of these AI companies and say, hey, if you want to plagiar journalists with AI, we're going to charge you for that, too?

• 26:51Speaker 03

  I think the larger conversation that that is part of has to do with putting the onus and the responsibility on some of these mega platforms that wield so much power in people's lives. Governments used to be able to, you know, pretty much protect and control every aspect of people's lives 50 years ago. And it wasn't necessarily for the better. It was just the way it was. We controlled our borders. We controlled what came in, including content and books and things like that. It was horribly abused in some situations, but we at least had the tools. Governments no longer have the tools to even protect people in what is a huge part of their lives because, yeah, they're connecting with some server in Romania

• 27:36Speaker 03

  or dealing with misinformation from Iran or whatever. We just don't have those tools. So there has to be a conversation about how we build complete societies that do have responsibilities and that there are people with power, even if it's not political power, power over those spaces to think.

• 27:59Speaker 03

  Responsibly about what it is, whether it's on protecting kids. We just moved forward with online harms legislation that is putting the onus on platforms to say, are you doing things that has the well being of kids in mind? Now, what platforms decide to do and how they do it and how they justify it? That's an ongoing conversation. We're not dictating to them. But we are saying, yeah, you have to think about protection of kids. What I want is not to, for government to legislate what platforms should do or not do, because that's a recipe for disaster. We all know how slow governments end up working. We're always playing catch up. And, and, you know, the political debates are not necessarily these days as deep and thoughtful as, as we would sometimes like them to be.

• 28:43Speaker 03

  But can we put the onus of leadership and responsibility that goes with it increasingly on platforms around journalism, around protection of free speech, but also protection against hate speech? Can we find those balances? Can we make sure that, that there is a common grounding in facts and not, not just deep fakes, but the level of misinformation and disinformation that we've seen over the while? These are the things that I think needs to be shifted and, and I think platforms need to be a positive part of that conversation.

• 29:14Kevin Roose?

  One more social media question while we're on the topic. Should Canada ban TikTok?

• 29:20Speaker 03

  About many months ago, the Canadian government made the decision that all government phones could not have the TikTok app because it was too much of a security risk in terms of that. As a byproduct of that, the prime minister's family is on government phones. So my teenagers don't have TikTok anymore, which is lovely for me.

• 29:43Kevin Roose?

  They must love that.

• 29:44Speaker 03

  They, they, they are grumbling about my job as, on this and on a few other things. But I think there's two questions around TikTok. One is the data security issue is real. And the second more nebulous one is, is this good for kids? Is this good for people to be on social media? And I think those two issues are being conflated a little bit. People are sort of leaping on TikTok as, oh my God, it's connected to China. So yeah, we should ban it and then we should get on to banning, you know, Instagram reels and YouTube shorts and all those things that are so, so, you know, popular for very powerful reasons right now. So I think on the data security side, I mean, our, the head of a spy agency just came out and said,

• 30:27Speaker 03

  yeah, it's a real risk that China is accessing so much of Canadians' data through TikTok. And we are looking carefully at how to counter for that. But to use those tools to sort of say, well, TikTok's also bad for, bad for people. So let's ban that. I think that as satisfying as it might be for, for some parents and some, some people, and some teachers, quite frankly, that's not necessarily the path we want to take.

• 30:51Casey Newton?

  You mentioned deep fakes as something that exists. I understand you're going to be up for election here in a bit. Have you been deep faked already? And are any of them good?

• 31:02Speaker 03

  I, I know I have been deep faked many times. One of the ones that what made me laugh the hardest was someone did a series of, of pictures of all the 23 Canadian prime ministers going back 150 years as classic rock 80s glam rock characters. And I, I, I found that very, very funny.

• 31:23Casey Newton?

  Which one were you?

• 31:25Speaker 03

  Oh, I know I was, I was, I was, it was, I had very, very big hair.

• 31:29Casey Newton?

  I see, I see. Got it.

• 31:30Speaker 03

  Bulging pecs and stuff. It was, it was very nice to see, but it was, it was sort of silly. I have a level of, of protection because I'm prime minister around that of people sort of expecting. Although I did have a yoga instructor friend of mine who admitted that, hey, I just spent 300 bucks on that, that Bitcoin you were promoting on, on, online. I'm like, I don't promote Bitcoin online. You got scammed. How could you fall for that? It was, you know, so that happens. But I think much more about the fact that deep fakes are now impacting, you know, high school students are impacting people in their works. The, the accessibility of that technology is, is something that I just don't worry about in terms of a political context.

• 32:14Speaker 03

  It's much larger than that.

• 32:16Casey Newton?

  Yeah. I mean, we're always so sad on the show when anyone. Buys a bitcoin, but to know that your own likeness had been used to promote it must be especially upsetting.

• 32:22Speaker 03

  Devastating.

• 32:23Casey Newton?

  Can I ask one follow up?

• 32:24Speaker 03

  Yes.

• 32:25Casey Newton?

  Yeah, so I had a follow up on deep fakes, which is that, you know, so so far, you know, there's been some deep fakes of you. There's been some funny ones. You're not particularly, you know, as it pertains to, you know, maybe this election. And of course, you're, you know, you're, you're the most famous politician in Canada. So people are sort of already broadly aware of you. One concern that you hear sometimes about deep fakes, though, is that not all politicians are as widely known. And that in maybe a local race, a provincial race, one of these things might get a little bit further. So I just wondered, have you, have you seen anything like that? And do you worry about, you know, what is sometimes called synthetic media, sort of just upsetting politics in in general in Canada?

• 33:03Speaker 03

  Well, I mean, let's be very clear, even before the arrival of deep fakes and increasingly sophisticated synthetic media. We have seen misinformation, disinformation, polarization, foreign interference by countries like Russia and China and India that are trying to destabilize our democracy in various ways be very, very impactful. Like some of the the fake tweets or or the the bots that are out there to to drum up support or or criticize. Like have become more and more sophisticated. So this is sort of a next level tool. And ultimately, the answer to that, you know, is is sort of similar to the answer to everything is to empower citizens

• 33:47Speaker 03

  to be more discerning, to be more thoughtful, to make sure that local journalists are able to actually cover those races that aren't with national figures, to make sure that there are consequences and frameworks and independent, you know, bodies not associated with government that are weighing in on fact checking and and saying that this does matter. And all those sorts of things, I think, will be more and more urgent to deal with with deep fakes. But that might also give an impetus for more people to realize, okay, just because I saw this local councillor explaining that they, you know, they would really want to, yeah, do something horrible. That should clue in that we can't be, we can't be just taking everything at face value.

• 34:32Kevin Roose?

  I'm curious if you turn to AI at all in your own life. Is it part of your work life? Do you use it for fun when you're not working? How have you intersected with these tools?

• 34:44Speaker 03

  Do my streaming algorithms count? You know, obviously, sometimes those algorithms work really well for good recommendations. Myself, I tend not to use it, even though my kids use it. I had a friend of mine who's a big into AI. I said, well, listen, AI could write you a platform on how Canada should handle AI. I'm like, OK, why don't you do that? Do that and then show it to me. And they showed me two pages of work that they generated in about 10 seconds. And it looked reasonable. I could see how if you put that in the window, someone would say, oh, there's an AI strategy for Canada. But as soon as you scratch the surface a little bit, OK, well, why this? And what does that work?

• 35:29Speaker 03

  You could see it was just, it was a facsimile of a platform, wasn't a real platform. So that reassured me a little bit that there's still a lot of hard thinking that needs to go into solving these problems that are not just cosmetic.

• 35:44Kevin Roose?

  Yeah, I know we only have a minute left. One question on national security and AI. Obviously, we could spend an entire podcast just talking about this, but I'm curious, do you think governments need to be building their own AI systems in this day and age? Or is it enough to just have the private sector building this stuff and governments regulating it?

• 36:03Speaker 03

  No, I think governments need to be using and aware of, you know, the various, you know, most advanced technologies out there, particularly because we know other people or adversaries or, you know, countries that don't wish the best for us will be, will be developing those as well. And there are going to be, you know.

• 36:29Speaker 03

  I know how scary it sounds. Oh, government's developing AI. And that's not what I'm talking about. But I am saying that if we're going to make sure we're keeping Canadians safe, like I said earlier, AI for good, done by the good people with careful, rigorous oversight and transparency, is going to be a very powerful tool to counter the AI for bad that we know is going to be out there.

• 36:56Kevin Roose?

  Yeah. Well, thank you so much for your time, Prime Minister. It's nice to talk to a world leader who is not guilty of 34 felonies. And

• 37:04Speaker 03

  I'm not saying anything.

• 37:07Kevin Roose?

  And if you want, as a parting shot here, I want to offer you a free slogan that you can use for your national AI efforts if you want. This is just an idea. Canada, we put the A in AI.

• 37:20Speaker 03

  Oh, my God. What are you doing?

• 37:21Kevin Roose?

  What do you think?

• 37:22Speaker 03

  I think that's a real winner. You have a future in politics. I hope you run against me because it'll be really hard.

• 37:30Kevin Roose?

  All right. Well, thank you so much, Prime Minister. Great to talk to you.

• 37:32Speaker 03

  Thank you, guys. Great to see you.

• 37:38Kevin Roose?

  When we come back, Kevin gets an OpenAI whistleblower to go on the record about what he saw inside that company that pushed him into leaving it.

• 38:13Speaker 00

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure, and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner, Dell. AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 38:47Speaker 04

  AI spend is growing fast, but most businesses still can't see exactly where that money is going. Ramp gives businesses visibility and control across AI providers with real-time guardrails by team and model, helping ensure spend goes to the highest ROI areas. Stop guessing where your AI spend is going. Get started with Ramp. And if you're new to Ramp, you can get $250 off at ramp.com/hardfork. Go to ramp.com/hardfork. Ramp.com/hardfork.

• 39:19Speaker 05

  Today's AI builders need infrastructure that scales with their ambitions. Iron is a vertically integrated AI cloud platform, delivering data centers, compute and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Iron's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Iron, AI cloud for the next big thing.

• 39:53Casey Newton?

  So a big week for you. I heard you were up all night on Sunday.

• 39:56Kevin Roose?

  I was, yeah. Well, not all night, literally. That would be a little extreme.

• 40:00Casey Newton?

  You stayed up till 9:30?

• 40:02Kevin Roose?

  Yes, way past my bedtime because I was working on this story.

• 40:06Casey Newton?

  Do you ever think about just finishing it during the working day? Or do you always leave it to the last second?

• 40:09Kevin Roose?

  Oh, my God.

• 40:12Kevin Roose?

  You know, some of us have deadlines and editors who want things by certain times. We can't just laze about and then, you know, fire off a quick email to our subscribers.

• 40:21Casey Newton?

  Well, that was your first mistake was taking that job for yourself. But so let's talk about this story, Kevin, because it was a very good story. I learned a lot. And it's something that I've wanted to know for a long time, which is, what do the people inside OpenAI actually think about what's going on there?

• 40:34Kevin Roose?

  Yeah, I mean, this has been a big question that I've been pursuing along with a bunch of other reporters ever since the board coup last year at OpenAI is like, there seems to be a sort of a steady drumbeat of intrigue and infighting at OpenAI that we've learned about since the board coup last year. And obviously, we've talked about some of the more recent developments, the departure of Ilya Sutskever and Jan Leckie, two of the most senior safety people at OpenAI, who were concerned about the direction this technology was going. And then I got wind of this group of current and former OpenAI employees who were preparing to make some kind of a statement.

• 41:17Kevin Roose?

  So this was a story that came about because of a guy named Daniel Cocatella, who's a former OpenAI researcher who left the company earlier this year. And he has basically organized this group. You know, you could call these people whistleblowers. You could call them, you know, concerned insiders, but either way.

• 41:34Casey Newton?

  Rabble rousers, traitors.

• 41:37Kevin Roose?

  But, you know, whatever you want to call them, this is a group that was inside the company and became concerned about the direction that the company was moving and decided to speak out.

• 41:49Casey Newton?

  And some of those people are still there, right?

• 41:51Kevin Roose?

  Yeah, so this group, I called them whistleblowers in my story, whatever you want to call them. It is roughly, you know, a dozen or so people who have now signed this public statement calling for changes at the big AI companies. And among those people are some former employees and a few more people who are signing this letter anonymously because they still work at OpenAI.

• 42:15Casey Newton?

  So tell me a little bit about who Daniel is and how he got to this point.

• 42:19Kevin Roose?

  So Daniel is a 31-year-old former researcher at OpenAI. He was part of their governance division. And he's a person who's just spent a long time thinking about risk in AI. So he came up through the sort of effective altruist movement. He was a part of this AI research nonprofit called AI Impacts. And then in 2022, he went to work at OpenAI. And his specialty is forecasting. He is a person who has strong views about when various sort of AI events will unfold. And he has really tried to be rigorous in studying those. And when he got to OpenAI, that was sort of the thing that he was focused on. But then he sort of gradually became worried that the company was not doing enough

• 43:04Kevin Roose?

  to promote safety and that it was just sort of racing to build shiny products and get them out into the world.

• 43:09Casey Newton?

  And we'll ask him all about that. But in the meantime, this group that he helped bring together to raise their concerns, what did they think and what are they asking for?

• 43:19Kevin Roose?

  So the open letter is primarily concerned with what they call the right to warn. Basically, they want OpenAI and other leading AI companies to commit to not only creating sort of internal ways for employees to raise concerns about safety, but they also want stronger protections for whistleblowers or other people who sort of voice those concerns to maybe outside groups or regulators.

• 43:48Casey Newton?

  Right. So OpenAI has this whistleblower hotline that you could call, but it just goes to someone at OpenAI. My understanding is they want this hotline to actually go to a regulator or some external authority who can help.

• 43:59Kevin Roose?

  Yes, and they are not just focused on OpenAI. They want these policies to be implemented all across the AI industry.

• 44:06Casey Newton?

  And is this the sort of thing where these whistleblowers have given us a really detailed list of their complaints and helped us understand exactly what it is they're seeing inside these labs that is making them so nervous?

• 44:15Kevin Roose?

  Yes and no. So in my conversations with Daniel and other members of this group, they have been pretty light on specifics when it comes to incidents that made them concerned about safety at OpenAI. And in part, that's because they believe they are still bound by these confidentiality agreements that they have signed with OpenAI. So they're not getting into the kind of nitty gritty inside information. But they did provide some specifics and they provided a lot more sort of generalities about kinds of changes at OpenAI and messages from leadership that they believed signaled that this company was not taking safety seriously enough.

• 44:57Casey Newton?

  So what did OpenAI say, Kevin, when you asked them about all this?

• 45:01Kevin Roose?

  So a spokeswoman for OpenAI sent me a statement that said, quote, "We're proud of our track record providing the most capable and safest AI systems and believe in our scientific approach to addressing risk. We agree that rigorous debate is crucial, given the significance of this technology, and will continue to engage with governments, civil society, and other communities around the world." They also pointed out that OpenAI does have an employee hotline for integrity issues.

• 45:29Kevin Roose?

  And they also said that they have started this new safety committee that will be reviewing feedback from from from these former employees as part of their sort of assessment of safety culture at OpenAI.

• 45:40Casey Newton?

  Well, I'm excited to talk to Daniel about all of this, Kevin, because it plays into something I've been thinking a lot about this week, which is power and who wields power in OpenAI. So I imagine you saw this story in the Wall Street Journal about Sam Altman's investments this week.

• 45:57Kevin Roose?

  Yes.

• 45:57Casey Newton?

  So the Journal had this great piece. He it revealed that Altman has more than 400 investments. Increasingly, it seems like companies in his portfolio are now signing deals with OpenAI for various things. And Sam has equity in those companies, which means that as those companies grow, he will get rich. But as I've been thinking about this and talking about it with people, Kevin, I don't actually think that money is probably what this is about for Sam Altman, right? Because this is a person who is already rich. But if he has a piece of all the biggest and most important companies in AI, then if you fast forward to the time period that Daniel's worried about when AGI is arriving, one person is going to have a staggering amount of power in that universe, right?

• 46:39Casey Newton?

  He will be the CEO of OpenAI, and he will be a major shareholder of many of the most important companies in the space. Why do I bring this up? In that kind of world, you want to know who is this person and what sort of governance structures are around him to rein him in? Because right now, it does seem like we're on trajectory for Sam Altman to be one of the most powerful people in the world.

• 46:59Kevin Roose?

  Totally. And I think that raises one of the questions that I had when I first heard about Daniel and this group of OpenAI insiders, which is, why are these people coming forward now? And I think this group really just feels like we are getting to a point where this technology is legitimately going to be dangerous. Now, there are people who disagree with that, who think we're nowhere near that. But these people are convinced that we are getting to an important inflection point, not just in the history of AI, but certainly in the history of humanity as we approach these generally intelligent systems. So for that reason, it's becoming more important. You know, five years ago, 10 years ago, it may not have mattered that Sam Altman had You know, a hand in many of the hottest startups in Silicon Valley.

• 47:44Kevin Roose?

  It may not have mattered that, you know, AI was was being sort of recklessly developed. But these people feel like as we get closer to a moment where the entire world is going to start caring deeply about the progress of AI, it actually becomes much more important to trust the people and companies that are building it.

• 48:02Casey Newton?

  Yeah, now, I do feel like we should say there are still a lot of people out here who think that the folks involved here are being ridiculous. They think that they are afraid of a ghost. They think there is no way that these systems, as they are currently designed, will ever amount to anything half as powerful as as what is being discussed. And I go back and forth on this all the time, sort of depending on who is the last person I have talked to, right? But Kevin, this week is a week where I am taking these people seriously. Okay, because when you do what Daniel and others invite us to do, which is to just look at the trend lines of how this stuff is developing, and you look at how much more powerful it gets every time you add a little bit more compute,

• 48:44Casey Newton?

  you can actually just extrapolate forward in a pretty linear way and say, you know what? The future might indeed look like the past five years where this stuff just keeps getting exponentially better. And I think if you believe that there is even a chance that that is true, then you do want to pay at least some attention to what folks like Daniel are saying.

• 49:00Kevin Roose?

  Absolutely. Yeah, and I think, you know, we've seen this kind of story play out before, like with social media, right? There were these companies, they had these grandiose missions of connecting the whole world. They hired all these people to make those visions a reality, including some pretty idealistic people who wanted the companies to live up to their stated values. And when those companies started, you know, making decisions, cutting corners, putting maybe growth and profits ahead of things like trust and safety, we saw people start to speak out about that. And so in some ways, this is a very familiar cycle that we've just seen play out with social media that I think we're just starting to see play out with AI.

• 49:42Casey Newton?

  Yeah, well, I think it's going to be a great discussion, and we should bring Daniel in here.

• 49:48Kevin Roose?

  When we come back, we're going to talk with Daniel Cocatello, formerly of OpenAI.

• 50:11Speaker 00

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure, and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner. Dell, AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 50:45Speaker 04

  AI spend is growing fast, but most businesses still can't see exactly where that money is going. Ramp gives businesses visibility and control across AI providers with realtime guardrails by team and model, helping ensure spend goes to the highest ROI areas. Stop guessing where your AI spend is going. Get started with Ramp. And if you're new to Ramp, you can get $250 off at ramp.com/hardfork. Go to ramp.com/hardfork. Ramp.com/hardfork.

• 51:17Speaker 06

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain, and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all. Legacy, stability, and digital innovation. Systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only. Availability of products can vary. Maybe provided by different Ripple entities.

• 51:51Kevin Roose?

  Daniel Cocatello, welcome to Hard Fork.

• 51:53Speaker 07

  Thanks for having me. Excited to be here.

• 51:55Kevin Roose?

  So let's start with your experience at OpenAI. You joined the company in 2022 as a researcher in the Governance Division. You were working on things like predicting the path of AI progress. At what point did you start to worry that safety was being deprioritized?

• 52:17Speaker 07

  It sort of crept up over time. I think the core thing here is that It's more important to prioritize good governance and safety the closer you get to incredibly powerful and dangerous AI systems. And when I joined early on, I thought like, we're talking a good talk. There's a bunch of people here who really care about this stuff. And we're sort of leading the industry, in fact, in terms of our like willingness to talk about these things. And I basically gradually lost hope that we were really actually going to pivot more to investing in these things to get ready, at least not to nearly the extent that we need to.

• 52:56Kevin Roose?

  Hmm. You told me when we spoke earlier about some advice that you got after you joined OpenAI, basically from someone telling you sort of, don't rock the boat. Like, don't come here and expect to sort of change the place. Tell me about that advice and what that meant to you at the time.

• 53:17Speaker 07

  I mean, to be honest, it was a relief to hear that advice because I don't want to be doing that sort of stuff. Like, I'm a researcher. I really like trying to predict the future of AI. By contrast, engaging in internal company politics and trying to like argue with people to change their policies is super stressful and not my, you know, cup of tea. So, so actually, when I, when I heard that, I was like, great. Like, I didn't want to do that anyway. You know, I'm, I'm going to keep my head down and do my research.

• 53:49Kevin Roose?

  Yeah. Do you want to walk us through incidents that you observed that you found the most disturbing?

• 53:57Speaker 07

  Yeah, so, so this is tricky because I'm still bound by this confidentiality stuff. So there's a lot of stuff I would like to talk about, but I can't talk about it. I mean, I could talk about some of the things that are already public, such as the DSP thing.

• 54:12Kevin Roose?

  Yeah, yeah, give us that story.

• 54:14Speaker 07

  So this was, I think, in late 2022, and we had this internal self-governance structure set up called the Deployment Safety Board. And this board had some people from Microsoft on it and some people from OpenAI on it. And it was supposed to deliberate and then eventually approve major deployments on the GPT-4 scale or bigger. So it had been set up prior to reaching the GPT-4 scale, but it was set up with the expectation that once we hit that level and thereon, it would be something that would gate deployments. You'd have to get approval from the DSP.

• 54:51Speaker 07

  While the DSB was still deliberating about this, me and various other people started hearing rumors that Microsoft had just gone and deployed it anyway, specifically in India. So we had some conversations about this. We made inquiries to managers and so forth. I don't want to get into too much detail about who said what to whom and so forth, but I came away disappointed because it seemed like we had this self-governance structure in place and there was a violation of the structure. And rather than holding Microsoft accountable for doing this, we were afraid to damage our relationship with Microsoft, given the sensitivities involved. They have all this compute. We're trying to be partners

• 55:35Speaker 07

  with them and so forth. And what happened in India wasn't that big of a deal. GPT-4 isn't actually that dangerous or scary, but I think this is an ominous sign about the ability of tech companies to self-regulate like this. Companies can make all these commitments and have all these structures, but then they can be violated and the rest of the world can just not know about it.

• 55:55Casey Newton?

  And Kevin, you actually verified with Microsoft that this happened, is that right?

• 55:58Kevin Roose?

  Yes, it was an interesting chain of events. So when I was getting ready to publish this story, I went to Microsoft. I said, hey, there's this claim being made that you all were testing this GPT-4-based model in India without approval of this safety board. And their first response was they denied it. They said, this never happened. We never used GPT-4 in India in these tests. And then after the story was published, I got a note saying, actually, that was wrong. A version of the model that eventually became GPT-4 was included in a small test in India, and it hadn't been reviewed by the DSB, although it was, they say it was reviewed and approved later. So essentially saying our denial was wrong and these allegations were correct.

• 56:43Casey Newton?

  And so, Daniel, I think you're making a really interesting point here because what you just said, I would agree with, which is GPT-4 is not that scary. But for that very reason, it seems like it should not be a big deal to take it before this board that has been set up for this exact purpose and get approval to launch something. And I guess the question is, well, if you're not going to even test for GPT-4, which we can all agree isn't that scary, when are you actually going to start doing these tests? I mean, was that what you were feeling?

• 57:10Speaker 07

  Well, I mean, in OpenAI's defense, they are doing more stuff now. Like, the DSP did, in fact, eventually approve GPT-4 employment. So I do think there is some progress in that direction. The point that I'm trying to make is that we shouldn't have to trust the company to keep their own commitments. We shouldn't have to trust any of these companies to keep their own commitments. There should be some way to tell whether the commitments are being followed.

• 57:33Casey Newton?

  Yeah.

• 57:33Kevin Roose?

  I know you say you're bound by these confidentiality agreements, but I really want to press for just as many specifics as you can offer, because I think that will help people understand how you reached the conclusion that the company was being reckless, that they were not taking safety seriously enough. So anything more you can share, any incidents or conversations or things that maybe convinced you that there was a problem here?

• 57:56Speaker 07

  I know you want me to give more concrete details, but I just can't. I'm sorry. I will say, I can talk about some things that have already been discussed, but I'm going to try to avoid violating confidentiality.

• 58:10Casey Newton?

  Sure. Well, so what else is out there that is sort of publicly known that maybe contributed to your sense that the company was acting recklessly?

• 58:18Speaker 07

  Yeah, so there's been this story about accelerating chip production globally. This flies in the face of some of the arguments that some of us at least had been making for why what we were doing was good.

• 58:32Kevin Roose?

  This is the story, just so I'm clear, about Sam Altman trying to go out and raise a bunch of money, perhaps as much as $7 trillion, to build a whole bunch of the chips that you need to train powerful AI models. Is that the chip scaling story?

• 58:47Speaker 07

  That's exactly right. And again, I don't know all the details, and the details I do know I'm not supposed to talk about. But, you know, that was disappointing to me, because one of the arguments that had circulated at OpenAI prior to that was this hardware overhang argument. The idea was, yeah, sure, maybe the world is not ready for AGI yet. We have to improve our governance and regulation of this technology. But, you know, AGI isn't a single binary thing. It's not like you go from zero to one, nothing to AGI. It's rather going to be this intense period of rapid gains across all sorts of capability benchmarks. We want that intense period to be less intense. You know, we want it to be smeared out more and for the rapid capability gains to be spread out over months or years rather than over weeks or months, right?

• 59:33Speaker 07

  And so the argument was if we don't build AGI now, if we try to hold off or delay or pause, all of this compute is going to build up in the world. And so that when someone finally violates the agreement to pause, they're going to just have all this massive amount of compute and they can just scale up by orders of magnitude really quickly, blow through all those capability mindstorms really quickly. So this is the hardware overhang argument for why actually slowing down would be a bad idea. And we need to, you know, build AGI as fast as possible so that the transition from pre AGI to post AGI is slower. And then now, you know, it seems that Sam Altman is going to like just make there be more compute in the world, which directly undercuts that argument at least.

• 1:00:16Kevin Roose?

  I'm very curious, Daniel, what the board drama at OpenAI last year when Sam Altman was fired and then brought back and the board was sort of reconstituted, what that felt like to you from the inside as someone who presumably was already slightly worried about how the company was treating safety and then to have this whole big corporate governance blow up, I'm just curious what that felt like from your perspective.

• 1:00:45Speaker 07

  I mean, the short answer is it was devastating for me. Prior to the board crisis, I still had hope that we were going to get all that stuff in place. We would have pivoted to investing a lot more in alignment research and, you know, engage in some sort of internal governance or self governance stuff that made us more democratically accountable and transparent. But, you know, after Ilya Sutskever and Helen Turner and after that whole fight, it was like, oh, man, things have sort of polarized. Like, I don't think that we're going to like talk it out and pivot. And then also just sort of on a personal level, I think there was a lot of anger directed at safety people such as myself because the perception was that

• 1:01:29Speaker 07

  the old board was being dishonest and that they didn't really fire Sam because he.

• 1:01:36Speaker 07

  Was insufficiently candid towards them, but rather they fired him because they wanted to slow down and he didn't and they needed an excuse. So I think that as a result, there was this sort of like, you know, fuck you, safety people. That's not, that's not a fair move, you know? And I think that is false, by the way, like, I don't think, I think that it really was just that they, they didn't trust them. And I think they probably had good reasons, although I wish they would say more about that.

• 1:02:03Kevin Roose?

  Did, did you take a side during this? Like, who did you sort of find yourself aligning with when, when that crisis happened?

• 1:02:09Speaker 07

  So this is something that's been maybe a sort of formative experience for me. Like, when the news hit on Friday that this was happening, I got scared and I basically went and hid for the weekend. So I just went and spent time with my family. And then towards the end, there was like the double crisis where the board appointed Emmett Shear and Ilya went into the office to, to talk about it. And people flipped out on Slack and, you know, there's just this huge uproar. And then I did do something. I basically just packed up a little bit and said, like, I tentatively support Ilya. I trust his judgment. I want to hear what he has to say.

• 1:02:51Speaker 07

  It was just a very stressful experience, basically.

• 1:02:55Kevin Roose?

  It, it really sounds like after this board crisis and, you know, coupled with a couple of those events that you just mentioned, you had the sense of this company is not for me anymore. It is not the company that I thought I was signing up to work for. Does that seem fair?

• 1:03:10Speaker 07

  Yeah. Yeah, I think that's fair. I mean, but, but it's really not about me. Like, this wouldn't be such a big deal if AI capabilities were going to be fixed at roughly GPT-4 level for the next five years or for the next 10 years.

• 1:03:23Kevin Roose?

  And then on your way out, something unusual happens. And this is how I actually saw your, your name pop up for the first time in all of this, which is that, you know, OpenAI has this standard sort of offboarding paperwork that when you leave the company, you're asked to sign and there's a, there's a non-disclosure part of it, there's a non-disparagement part of it, or at least there was. And you did something unusual, which is that you refused to sign that paperwork, even though, per the agreements you signed when you joined OpenAI, doing so might cause you to lose out on all of your vested equity, all of your, your, your sort of wealth that you had accumulated.

• 1:04:06Kevin Roose?

  And my understanding is that was a significant amount of money, about 1.7 million dollars. So you were basically refusing to sign this paperwork that would have guaranteed you a large sum of money in exchange for the right to speak out about the company. So walk me through that decision.

• 1:04:23Speaker 07

  The way it works is around the time you join the company, you sign something that says your vested equity goes away if you don't sign a general release when you leave the company within 60 days. And then when you leave the company, they give you the general release and it has all this stuff in it, including non-disparagement and including the fact that you can't talk about this release. At some point, I think I deliberated enough, I discussed this with my wife. She's been really helpful throughout this whole thing. I think this is the sort of decision that we would have to make as a family because obviously it affects both of us so much. And we were just like, no, we're not going to sign this. I don't think it's ethical. So I wrote this email to OpenAI saying, I don't think this is fair. I want to be able to criticize the company in the future and so I cannot sign this. And then they were like, okay, we wish you the best on your future endeavors.

• 1:05:07Speaker 07

  And I was like, okay, well then, that's it, I guess. After 60 days, the equity is going to be gone. And so I started trying to move on with my life and get back to my research and try to figure out what my next career step was going to be.

• 1:05:22Kevin Roose?

  Right. And just to clarify, you discussed this with your wife because this was, this was not a small amount of money for you guys, right? She doesn't work at OpenAI, she's not.

• 1:05:31Speaker 07

  No, yeah. So, so, so first of all, it's hard, it's hard to actually say how much this is worth. What I did is I just asked them, like, how much would this sell at the last tender? And they said, a certain amount. And then I just multiplied that by the amount that I had and got 1.725 million dollars. And that's like 85% of our net worth at the time.

• 1:05:49Kevin Roose?

  Wow. And then after you refused to sign this paperwork, Vox reported on the existence of this sort of general agreement that all departing OpenAI employees were being forced to sign, which included this non-disparagement agreement and the sort of threat of maybe clawing back some of this vested equity. And the company actually responded to this. They basically said, we've never enforced this on former employees. We've never clawed back vested equity. We don't plan to. And Sam Altman said that he was genuinely embarrassed that he didn't know about the existence of this agreement and the terms of it. I guess I'm just curious if you believe that.

• 1:06:30Speaker 07

  I mean, I do find it hard to believe. But I will say for my, for my part, I'm pleased that this happened. Like, I think, I think if you had asked me like a month ago, or whatever this was before the Vox story, what I thought would happen, I would have said, OpenAI would be like, we're a corporation. This is just like our policies. What's the big deal? But actually, like, a lot of people inside OpenAI, as well as outside, were pretty upset about this, which was gratifying. And then also, you know, Sam apologized and is going to change. And that's actually what leads to this, to today with the proposal.

• 1:07:04Kevin Roose?

  Yeah, let's talk about that. So you, you leave OpenAI earlier this year, and then you sort of start putting together this group of current and former employees who share some of these same concerns. How did this whistleblower group, if you want to call it that, come together?

• 1:07:24Speaker 07

  Yeah, so, so after the Vox thing happened, and this all started blowing up, a ton of people started reaching out to me, offering thanks and support and so forth. So this lawyer, Larry Lessig, offered pro bono services. And so I engaged with him. Various people were reaching out to me. And I was saying, if you feel similarly to me, like, here's a good lawyer. You should get in touch, you know? And so this group sort of congealed. And this core group of us met up and discussed what to do. And then we were like, how about this? How about we? We meet them with a positive proposal for what good policies would look like.

• 1:07:59Casey Newton?

  And the core idea in this letter is, hey, let us talk about the risks that we believe that the companies that we worked at are creating related to artificial intelligence.

• 1:08:08Speaker 07

  Yeah, that's right. So it's actually kind of complicated. There's four points that we're proposing. So principle one is basically, don't do the sort of thing that OpenAI just did with the silencing of criticism and the equity callbacks and stuff like that. Principle two is this, like, what would the ideal system look like? The ideal system would be an anonymous reporting hotline, so that employees at these labs that are building this incredibly powerful technology can communicate with the people who need to know, people like regulators and independent watchdog groups with relevant expertise. Principle three and four are sort

• 1:08:49Speaker 07

  of saying you should promote a culture of, you know, open criticism. Three is basically saying employees who are raising serious concerns about various risks should be able to talk about confidential information in the course of doing so to the public, only insofar as it supports a risk-related concern. And then principle four says we can actually walk that back if principle two is working really well. If we have our good system in place where all the people who need to know can be informed, then we can walk back the thing about releasing confidential information and we can just be like, nope, go back to silence, you know? So that's the overall proposal.

• 1:09:27Casey Newton?

  So that seems like a pretty reasonable set of requests to me. But there is this thing that I'm struggling with, Daniel, which is that, you know, you were mentioning earlier that the board didn't share a lot of details back when it fired Sam in this wind-up, sort of coming back to bite people who are really concerned about safety issues because it sort of felt like there was no supporting evidence, right, to back up some of the claims that were being made. And, you know, as I'm hearing your story, the only thing that I have to go on is that Microsoft tested a thing, you know, without running it by a board. So I just, my question is essentially like, why should I essentially believe that

• 1:10:07Casey Newton?

  things are as dangerous as you're leading me to believe when no one involved is willing to really tell me anything of much substance?

• 1:10:17Speaker 07

  Yeah, good question. I think.

• 1:10:22Speaker 07

  In my case, I'm not trying to call out OpenAI in particular. I mean, I do have my disagreements with their company criticisms, but honestly, I think that probably if I went to these other companies, I would feel pretty similarly. So I don't want to be like specifically saying OpenAI is really bad and that's why we need these policies. Instead, I want to say this policy would be good and I can justify this without giving specific details, without violating confidentiality. Even the stuff that's already public, I think, is enough evidence to justify this policy.

• 1:10:51Casey Newton?

  Yeah. Don't you feel like you'd build more momentum, though, if at least someone somewhere was willing to say, hey, you know, it turns out that OpenAI has GPT-6 like, you know, sitting over there in a vat or, you know, whatever it is that's gonna raise our hackles? Like, I'm just curious how you think you can advance this movement when it seems like we're just sort of like stuck at this very high level of abstraction?

• 1:11:13Speaker 07

  I mean, OpenAI doesn't have GPT-6 in a vat.

• 1:11:16Casey Newton?

  Okay, sweet. Breaking news.

• 1:11:18Speaker 07

  I revealed some information there. I mean, this is some, this is an alternative strategy we could have taken. We could have been like, now is not the time. Wait until it's like right on the brink of catastrophe with terrible things right about to happen and then, you know, blow the whistle and stuff like that. And I think that's just cutting it too close. For one thing, AGI by definition can automate a lot of work. So that means that the, the size of the silo of humans that you need to be in this inner circle who knows about it can be pretty small. So you could end up in a situation where there's like this secret project within the company that has access to the latest model and is using it to make even more powerful models and is automating a ton of stuff.

• 1:12:03Speaker 07

  And they can be actually doing quite a lot of research really fast because most of the work is being done by the AIs. And there's only a small group of humans who even know about this. And That's one of the reasons why I think it's dangerous to to wait until then and and plan to like, you know, whistleblow right when you're right on the brink.

• 1:12:19Casey Newton?

  I mean, I I'm empathetic in the sense that journalists wind up being in this position a lot, right? Like, we constantly walk around ringing alarm bells, saying, hey, this bad thing could happen. And if we do our jobs right, someone intervenes to stop the bad thing from happening. And that that winds up often making us look foolish because the bad thing never happened, right? So we were sort of the boys who cried wolf. At the same time, I do think that this movement, like this contingent of safety folks, needs to find a more persuasive story to tell because I think that right now, it's not connecting with people because they use chat GPT, and they have trouble understanding how this is going to end the world.

• 1:12:52Speaker 07

  Yeah, that reminds me of the other thing I was going to say, which is that this whole issue about like, is AGI coming soon? Is AGI risk a real thing? Is like 90% not about what's happening in the labs in secret, you know? I think that publicly available information about the capabilities of this models and the progress that's been happening over the last four years is already enough to make extrapolations and say, holy cow, it seems like we could get to AGI, you know, in 2027 or so or sooner. You don't need any secrets to be able to make that inference. And then for the second thing about like, is this a big deal? Is this really dangerous? Again, you don't need any specific secret.

• 1:13:33Kevin Roose?

  I did hear from some, you know, a lot of people, when when this article came out about you and this whistleblower group thought, you know, these people are taking a brave stand. I did also hear from people who had more critical views of this. And I just want to let let you respond to some of these. One thing that really stood out to a lot of people who read this article is about you and your personal predictions about the trajectory of AI. So one of the things that you told me that was in the article is that you believe that AGI, which you define basically as a as an AI system that can sort of do any kind of job, like any anything that would be economically valuable, that that could arrive as soon as 2027. So three years from now, which is a very short timeline. You

• 1:14:14Kevin Roose?

  also have a P doom, which is the probability that AI will catastrophically harm humanity of Seventy percent, which is among the highest I've ever heard from someone who has directly worked at one of the big AI companies. So people are sort of citing these figures and saying, well, Daniel, he's not like a sort of normal run of the mill AI researcher. He is a doomer. He is someone who believes that this this stuff is going to kill us all. And we shouldn't take him seriously as a result of that. So I'm curious what you make of that.

• 1:14:45Speaker 07

  Yeah. So I mean, first of all, I will totally admit that my P doom is higher than most people's. I want to inject a note of humility here, like the the P doom thing is more of a vibe than a a serious number because it's an inherently unpredictable sort of situation. I think that I feel much more confident about my timelines. I think this is something that I have expertise in. Like I was hired at OpenAI to try to forecast the future of AI and I spent years trying to do that. And I think it's, you know, there's now lots of evidence that you can draw from scaling laws, trends on benchmarks and so forth. So I do feel somewhat confident in my claim that AGI could happen in the next few years.

• 1:15:27Speaker 07

  But I also could say that like, there's a bunch of other people who came together to make this proposal. And I bet they would all have much more like normal opinions about P doom. And they still are concerned. I think this is letter is a great idea and are advocating for it. So I think that this is something that hopefully a lot of people can get behind.

• 1:15:48Kevin Roose?

  I'm curious if you've heard from anyone at OpenAI since this letter was published this week.

• 1:15:55Speaker 07

  Yeah, yeah, a few people.

• 1:15:56Kevin Roose?

  What did they say?

• 1:16:00Speaker 07

  Some people reached out being like, you go, thank you for doing this. It's very brave. Some people reached out being like, I feel a little bit betrayed. Like, I thought we were going to work together to get better policies, but now you've been critical of OpenAI and you're making it seem like it's all our fault. But like, other companies are probably just as bad.

• 1:16:18Casey Newton?

  Well, the second group just sounds like they're whining and they need to get over it.

• 1:16:23Speaker 07

  I mean, I'm happy to say other companies are probably just like, I think this is a systemic problem, you know.

• 1:16:29Casey Newton?

  But I, but I actually think it's a revealing point because I think that there are probably people inside OpenAI who think, well, because there's another worse company somewhere, that justifies some amount of the actions that we're taking or some amount of the flaws that we have. And, you know, I can understand why you would have that point of view, but that's really not what you're saying. You're saying a very dangerous thing is somewhat imminent and people working on it need to be able to discuss that openly with the public.

• 1:16:51Speaker 07

  Well, I'm definitely not going to tell you who, who it was who was saying these things. But it was, it was more reasonable than you make it sound, I think.

• 1:16:58Kevin Roose?

  Got it. So what comes next for you?

• 1:17:01Speaker 07

  I want to get out of the media spotlight. I think this has all been very stressful for me. Yeah. So like, like technically, I'm supposed to be on vacation right now. This is my, my parents' house.

• 1:17:11Casey Newton?

  So mostly we just need to get Kevin to leave you alone, it sounds like.

• 1:17:14Speaker 07

  Yeah.

• 1:17:17Kevin Roose?

  Well, I hope you do actually get some vacation on top of your big week. I also am just really grateful for the, the time and the opportunity to speak with you. And I hope this continues a conversation that I think is, is fairly overdue.

• 1:17:31Speaker 07

  Thank you. And likewise.

• 1:17:33Casey Newton?

  Thanks, Daniel.

• 1:17:34Kevin Roose?

  Thanks, Daniel.

• 1:18:36Speaker 04

  AI spend is growing fast, but most businesses still can't see exactly where that money is going. Ramp gives businesses visibility and control across AI providers with real-time guardrails by team and model, helping ensure spend goes to the highest ROI areas. Stop guessing where your AI spend is going. Get started with Ramp. And if you're new to Ramp, you can get $250 off at ramp.com/hardfork. Go to ramp.com/hardfork. Ramp.com/hardfork.

• 1:19:08Speaker 06

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain, and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all: legacy, stability, and digital innovation. Systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only. Availability of products can vary. May be provided by different Ripple entities.

• 1:19:43Casey Newton?

  Hard Fork is produced by Whitney Jones and Rachel Cohn. We're edited by Jen Poyant. Today's show was engineered by Alyssa Moxley. Original music by Marian Lozano, Diane Wong, Pat McCusker, and Dan Powell. Our audience editor is Nolga Logley. Video production by Ryan Manning and Dylan Bergerson. Go check us out on YouTube. We're at youtube.com/hardfork. Special thanks to Paul Schuman, Queuing Tam, Kate LaPreste, Jeffrey Miranda, and thanks to Alex Goldberg for all the Canadian gear that he lent us and that we're not giving back. You can email us at [email protected]. Maybe it's time to blow the whistle at your company. Think about it. Come on. What do you got to lose?

• 1:20:23Kevin Roose?

  What do you got to lose?

• 1:20:24Casey Newton?

  What do you got to lose?

• 1:20:43Speaker 04

  You can't undo everything in life, but you can undo prediabetes. More than 2 in 5 adults in the U.S. have prediabetes. Most don't know they have it. But with early diagnosis and action, you can delay or even prevent type 2 diabetes. Visit doihaveprediabetes.org to take the 1-minute risk test and start undoing the things you can. Brought to you by the Ad Council and the Centers for Disease Control and Prevention.