Demis Hassabis and the Race to Build AI with Sebastian Mallaby

Demis Hassabis founded DeepMind in 2010, before machines could reliably tell a cat from a dog. He believed that artificial general intelligence was achievable, and that it should be built safely and deliberately. Along the way, Hassabis shared the 2024 Nobel Prize in Chemistry for AlphaFold, the system that cracked the fifty-year problem of predicting protein structures. Fifteen years on, the LLM race has erupted across rival labs and countries, and AI has become one of the most consequential and contested technologies of the era. Sebastian Mallaby https://x.com/scmallaby is a two-time Pulitzer Prize finalist and a longtime chronicler of the collision between innovation and capital, from hedge funds to central banking to Silicon Valley. His latest book, The Infinity Machine, is a portrait of Hassabis and DeepMind built on more than thirty hours of candid conversation, beginning just as ChatGPT set off the AI frenzy, and drawing as well on Hassabis’s estranged cofounder, his rivals, and academic pioneers who now fear for human survival. Mallaby is also the Paul A. Volcker Senior Fellow for International Economics at the Council on Foreign Relations. In this episode, Sebastian joins Kevin Ball to talk about Demis Hassabis and the founding of DeepMind, the pioneering breakthroughs that followed, and how the field arrived at the current moment in AI. Photo by Jeremy Sherlick The post Demis Hassabis and the Race to Build AI with Sebastian Mallaby https://softwareengineeringdaily.com/podcasts/demis-hassabis-and-the-race-to-build-ai-with-sebastian-mallaby/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Demis Hassabis founded DeepMind in 2010 before machines could reliably tell a cat from a dog. He believed that artificial general intelligence was achievable and then it should be built safely and deliberately. Along the way, Hassabis shared the 2024 Nobel Prize in Chemistry for AlphaFold, the system that cracked the 50-year problem of predicting protein structures. Fifteen years on, the LLM race has erupted across rival labs and countries, and AI has become one of the most consequential and contested technologies of the era. Sebastian Mallaby is a two-time Pulitzer Prize finalist and a longtime chronicler

• 0:40Speaker 00

  of the collision between innovation and capital, from hedge funds to central banking to Silicon Valley. His latest book, The Infinity Machine, is a portrait of Hassabis and DeepMind built on more than 30 hours of candid conversation, beginning just as ChatGPT set off the AI frenzy, and drawing as well on Hassabis's estranged co-founder, his rivals, and academic pioneers who now fear for human survival. Mallaby is also the Paul A. Volcker Senior Fellow for International Economics at the Council on Foreign Relations. In this episode, Sebastian joins Kevin Ball to talk about Demis Hassabis and the founding of DeepMind, the pioneering

• 1:23Speaker 00

  breakthroughs that followed, and how the field arrived at the current moment in AI. Kevin Ball, or K-Ball, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Leighton Space. Check out the show notes to follow K-Ball on Twitter or LinkedIn, or visit his website, kball.llc.

• 2:03Speaker 01

  Sebastian, welcome to the show.

• 2:05Speaker 02

  So good to be with you.

• 2:07Speaker 01

  Yeah, very excited for this. Let's start out, can you give us both a little bit of intro to yourself and then kind of intro to what brought you to writing The Infinity Machine? And we'll end up talking a fair amount about that on the show.

• 2:19Speaker 02

  Sure. So, you know, I did something deeply technical at college, modern history, with a focus on Europe, and the central question was, why had Britain never succumbed to Marxism? So that's a long way from technology. And my path between doing the history of revolutions in the late 19th century, early 20th century, and writing about Demis Hassabis and the cutting edge of AI goes kind of the quick version is as follows. So I became a journalist, joined The Economist magazine, worked there for maybe a dozen years, was a foreign correspondent in Japan, in Africa, and then the United States and Washington DC. And then I switched to The Washington Post and was a kind of opinion columnist focusing

• 3:02Speaker 02

  on economics, a little bit of tech by that point, and just generally kind of whatever seemed to be making the world change in terms of big macroeconomic, macro political sense. So I wrote about both everything from political reform, constitutional mechanics in the US, and why trust in government was declining, is in the 90s and the, yeah, the late 90s, to all the way over to kind of like healthcare economics, because the healthcare system was clearly a big driver of the fiscal deficit and so forth. So I did that at The Economist in the late 90s and then in The Washington Post in the 2000s. And in the meantime, I did a couple of books, and this became kind of more interesting to me. And when I did my third book, I switched to a think tank, the Council on Foreign Relations, where it's

• 3:47Speaker 02

  a fantastic home to write books. You get the space and the time and the good colleagues and all that. So then I did another four books, and essentially where the first book, when my 20s, when I had been a foreign correspondent, was about Africa. The next four were about different kinds of finance, so development economics, hedge funds and public markets, central banking in the form of a biography of Alan Greenspan, the Fed Chairman, who managed to serve for almost 20 years, and then a book about the investing in technology in Silicon Valley, the venture capital world. And that last one was kind of a segue because having written about investing in tech, I was kind of ready to jump and write a book about tech. And so in 2022, I was looking around for the right way to do that. Artificial

• 4:32Speaker 02

  intelligence hadn't broken out in a ChatGPT kind of way, but it did seem like it was really cooking and really reaching some sort of inflection point where you'd had these systems like AlphaGo in 2016, which solved this massive combinatorial complexity of finding permutations in the game of Go. It's 10 to the 130 permutations you can have in the game of Go, and yet it found the patterns in that enormous search space. And then on top of that, you had the AlphaFold system, which did the same kind of search but for protein shapes. Now it's fully 170 more orders of magnitude, the search space, right? So 10 to the 300 this

• 5:15Speaker 02

  time is the number of permutations and the way that a string of amino acids can fold itself up into a protein shape. So it's super, super needle in a haystack. And so you looked at that and you thought, okay, well, this is like finding patterns in an infinity of data, kind of. And so it's an infinity machine. And so that's a kind of breakthrough you could apply to different types of thing. And machine intelligence really is taking off. So I pitched Demis Hassabis, who was the founder of DeepMind, that actually the lab that produced both AlphaGo and AlphaFold, and was definitely the OG, the most interesting, fascinating character in the whole of AI, right? Because he'd founded an artificial intelligence startup in 2010 before AI could even recognize a cat.

• 5:56Speaker 01

  Yep. In the middle of AI winter, he's diving in.

• 6:00Speaker 02

  Yeah, total deep winter, and he completely believes. There's a line from a movie that he loved about Watson and Crick, the discoverers of DNA, who say, blessed are they who believe before there is any evidence. And Demis Hassabis was in that category. So yeah, I pitched him on this idea for a book, and he thought about it a bit. I had ways and means of trying to persuade him. And so he said yes. And about a couple of weeks later, ChatGPT came out and went viral. And so I found myself embedded with Demis Hassabis and DeepMind in the, you know, one of the main labs, right as this large language model race took off and went crazy.

• 6:36Speaker 01

  Yeah, I love the looking back a little bit because I think right now we're in this language-focused model craze. But as you highlight, there's a ton of other really interesting things that have happened in the last 15 years in machine learning, protein folding, other types of, there's so many different types of models that we, we lose that when we just talk about large language models. But let's maybe start then a little bit with this thing. And one of the themes I've seen in your more recent books is you're sort of navigating, you'll use a person as a lens, but kind of navigating institutions and navigating sort of these things. So let's talk about that unrealistic belief and what that looks like. How, and let's not assume our listeners have read the book. Like, how did that play

• 7:19Speaker 01

  out for Demis? How did he kind of get this whole movement going?

• 7:25Speaker 02

  Yeah. So look, the big picture on this question kind of does go back to me doing modern history at Oxford University. I came to believe that there's this debate amongst historians, do individuals shape history? Or is it really ineluctable abstract forces like demographic change, technological change, and so forth? And obviously, the answer is both. It's kind of a dumb debate, but it's interesting to have that debate when you're 19, 20 years old and kind of frame it for yourself. And so going forward, when I came to write books later on, I was always looking for the individual who captures that sense of individual agency and how people do matter in history and in shaping the world that we live in the future. And at the same time, there are these big forces around them, which

• 8:09Speaker 02

  they have to kind of capture, they have to surf those big forces if they're going to really make an impact. So Demis is a great case study in this, right? Because sort of almost by luck, he was this child genius, learned chess by watching his dad and his uncle play when he was four years old, became the best young chess player in Britain, captain of the team, the British, you know, international youth team, and then also actually the second best in the whole world for his age group. So he was really extremely good and gifted at chess. And he wasn't satisfied by chess. So he would sort of do other things. And at one point, he's got himself a home computer and tried to write a chess program, which

• 8:50Speaker 02

  was like his segue between the chess and the programming. And one thing led to another. And when he got into Cambridge University young, because he was so smart, they told him, look, you know, you're intellectually ready to study computer science at Cambridge, but you know, you'll have a better time socially if you wait for a year and you show up when you're the same age as other people. So he goes off, he has like 15 months to fill, and he gets a job as a programmer in a gaming studio. And he designs and programs this game called Theme Park, which goes viral in the 1990s, sells like 5 million copies or whatever. So by the time he shows up at college, all of the other kind of nerdy boys who are into computer games, they've played his game so that he's kind of famous.

• 9:36Speaker 02

  And also, his boss of the gaming studio.

• 9:41Speaker 02

  Said he was so valuable as a coder and as a designer of the game that he would give him, I mean, today's money in dollars is like more than a million dollars. He would give him this money not to go to college and just come work for him instead. And Demis is like,

• 9:57Speaker 01

  It's like the original Peter Thiel.

• 9:59Speaker 02

  Yeah, but whereas in Stanford or in the US generally, people happily drop out and go to something more lucrative. No, no, no. Demis is like, I want to do the science. I want to learn about the deep mysteries of nature. And I'm not going to do that by doing video games. So I'm not accepting your money. But what the video game episode did do is it kind of put him in this strange country house outside London with a bunch of young coders, coding until like three in the morning to write this game software. And then they would meet up and have spaghetti in the kitchen in the middle of the night and like talk about the big issues and the big issues were basically, how do you make the games better? Oh, you need artificial intelligence to make the characters in the game more realistic and more compelling.

• 10:42Speaker 02

  Maybe if the player does something to the character in the game, then the character will adapt its behavior and change how it acts. And so that was kind of early reinforcement learning intuitions right there. And in the course of this, living in this commune, he read the book Gödel, Escher, Bach, which is sort of a kind of classic, which red pilled a lot of young scientists around this time on computer science, because it was written by physicists. It was kind of very abstract, kind of ranges from the mathematician, mathematician Gödel to the abstract artist Escher to Bach and his music.

• 11:24Speaker 02

  But it was kind of arguing that, and one of the arguments in it, I should say, maybe is that whatever the human brain can do, computers should be able to do one day. Because intelligence is not about sort of whether you have a biological neural network or a silicon neural network. It's the patterns on the network which constitute thinking and even consciousness. And so as, just looking at Moore's law, as computing became more and more powerful, one day you would have a computer as big as a human brain in terms of the number of neurons and connections and kind of gates, that you would have a computer that could basically do whatever a human can do.

• 12:09Speaker 02

  And so Demis aged eighteen, by the time he got to college, had already decided, based on this intuition from Gödel, Escher, Bach, that one day AI would work. And that if you could make it work, not only would games become very compelling, but also you would unravel the big mysteries of theoretical physics, of biology, of neuroscience, like everything, you would solve everything. And this was such a compelling vision to him aged eighteen, that he just was obsessed from it from that age. It's like, it's incredible. It's like nineteen ninety-three, right? We're seventeen years before he found DeepMind in twenty ten.

• 12:51Speaker 02

  Nineteen years before you get the ImageNet contest where the AlexNet system, finally you have a conventional neural network which can recognize a picture of a cat. Until then, complete AI winter, as he said.

• 13:03Speaker 01

  Yeah.

• 13:04Speaker 02

  So he was super, super, super early in having this conviction. And that shows you kind of the role of the individual who has a strange conviction and pursues it and actually does sort of accelerate the point where humanity gets towards thinking machines.

• 13:18Speaker 01

  That's one of the fascinating things here. I think a lot of us have rather absurd convictions at nineteen, but most of them don't pan out.

• 13:26Speaker 02

  Yeah, right.

• 13:28Speaker 01

  I'm interested to follow the game thread a little bit more because when he got out, his first company was not an AI company, right? It was a gaming company and I, I'm curious. It reminds me a little bit of one of my intellectual heroes was Feynman. And he wrote about like, after he got the Nobel Prize, he couldn't do good science because he was so focused on the big things. And then he stepped back to just starting to play again, and then things unfolded. And we kind of see this pattern with them as of like game focused entree into this huge topic.

• 13:58Speaker 02

  Yeah, I think that's right. And it turned out, of course, that games are a great environment for testing reinforcement learning in particular. You know, you have a sort of confined goal, a sort of measurable reward at the end of each action you take. So the environment is giving you this very clean feedback signal. And this is what he figured out pretty soon after he started DeepMind. And they decided to build a system that would solve all the Atari games, the kind of original video games created by the Atari company in the nineteen seventies, nineteen eighties. So you have Pong and Breakout and these other games. And the nice thing about this was that the video graphics were pretty simple because it was so old.

• 14:43Speaker 02

  And so for a, you know, in terms of compute efficiency, you could, it was tractable for primitive AI in twenty twelve or so when they began. And so they created this training environment for reinforcement learning using games. And so that's kind of where the connection came between Demis's obsession with games, which partly like he'd been this young chess player. He'd been super successful at that. He loved games. He loved winning at games. He was incredibly competitive. He, one time I went to see him, we would meet in London every month or so in a pub for a couple of hours. And there was a secret room upstairs and we'd sit there for two hours talking. And he said to me, you know, Sebastian, when I was at college, you know, one of the things was I was the best

• 15:25Speaker 02

  at a game called table football or fussball, right?

• 15:29Speaker 01

  Yeah, yeah.

• 15:30Speaker 02

  You know, table with the, with the figurines you twizzle. And I really didn't care. I mean, who, who cares about table football, right? And he said, no, no, you don't understand. You don't understand. I was the best. So I kind of nodded politely and he goes, no, you still don't get it. I have watched the, the videos of the semi-professional players that you can see these days on YouTube and they don't know the snake shot. Where you put your arm against the lever, you yank your arm up really hard and the friction on your arm gets extra traction on the spinner so that your shot in table football is even harder and I could do that. And that's why I was the best at college. You need to get this. You only understand this. Hey dude, okay, okay. But winning for him is a very big deal.

• 16:15Speaker 02

  And so games always were addictive for him, all kinds of games. He also was the five times champion of a thing called the Mind Sports Olympiad, where you played five different mind games. And he would run from sort of the go game to the chess game and then he'd go play the diplomacy game or whatever it was, backgammon or something. And he aced all of these games and he was the best in the world five times in a row before he retired. So he's like insanely obsessed with this stuff.

• 16:43Speaker 01

  Well, it's, yeah, it's interesting to think about this, right? Because starting a company is on its face, kind of a ridiculous endeavor, right? Like you're doing, you have to go through so many different setbacks. And I, I remember, I mean, there's a quote, most people have probably heard of Elon Musk talking about it. It's like eating glass. It's so hard. And so I wonder, you know, as you talk about the individuals who are harnessing the trends and moving forward, like is that level of competitiveness almost a necessity to get through those barriers?

• 17:13Speaker 02

  Yeah, a hundred percent. And especially if you're doing it in Britain and like, you know, the two thousand and five or four, whenever he founded the first company, which was a pure gaming company. And then even DeepMind in twenty ten, I mean, super difficult to raise capital because there wasn't a venture ecosystem back then in Britain. I mean, there is one now. I think twenty fourteen is what I regard as the inflection point when venture in Europe or particularly in London, particularly, you know, started to kind of get traction. So you could raise money. And with venture capital around, of course, it's not just that you can raise money, it's that you get the connections that the venture capitalist has, which can be useful for hiring people or getting your first customer to sign on to a contract or whatever it might be. I mean, VCs, I wrote a book about this.

• 17:56Speaker 02

  It's the networking effects that they bring, which are pretty important for company formation. And it's still like eating glass, as you say, but it's sort of smaller pieces of glass, much easier when you have a VC helping you, I think. And Demis had to really face this without there being a functional VC ecosystem at all when he first did his first company, right? So I tell a funny story in my book where he, he finds somebody somehow, you know, he probably, I don't know if he could even go online then. I mean, not really. It was late nineties, maybe just about you could go online, but there wouldn't have been proper search. Anyway, so he finds somehow that there is somebody calling himself a venture capitalist in London. And he goes sees these people and they just take him to lunch and give him two bottles of wine and talk about

• 18:41Speaker 02

  anything. And it results in a job offer to be a currency trader.

• 18:44Speaker 01

  I remember that story. Yeah, yeah, yeah.

• 18:46Speaker 02

  Because they're not interested in venture capital, even though they're very difficult. And it takes a special degree of self-belief and determination to do any kind of entrepreneurship. But in Britain, then it was off the charts.

• 18:59Speaker 01

  This episode of Software Engineering Daily is brought to you by Varia Code. If you're an engineering or IT leader, you know how painful this is. An open rec sits unfilled for three months. Your tech lead is burning out, covering the gaps, and your product roadmap gets pushed back again. Varia Code solves that. For over 10 years, they've been embedding senior Latin American engineers directly into US engineering teams in under 10 days at up to 40% less than local hiring. We're not talking about generic contractor pools. These are pre-vetted, high-quality engineers with real on-call and SLA experience. They work directly in your Slack, Jira, and daily stand-ups in your exact US timezone with zero friction. If you're ready to stop losing sprints to open recs,

• 19:41Speaker 01

  head over to variacode.com/sed right now and get a custom shortlist of senior candidates in 48 hours. That's V A R I A C O D E dot com slash S E D. You've spent weeks choosing the right model for your agent. How long did you spend choosing its web search? Parallel is web search built for agents, not people. Building a voice agent? Parallel Search Turbo answers in about 200 milliseconds for a dollar per 1,000 requests. Need an agent for thorough research? Parallel Search Advanced digs deep across the web to answer your toughest questions. Parallel works with any model and drops straight into Claude Code, Cursor, Versal, or LangChain, the best web search at every speed and price. Try it free today.

• 20:26Speaker 01

  Add the Parallel MCP server to your agent in minutes at parallel.ai/sed. This portion of Software Engineering Daily is brought to you by Origin. Your coding agents can launch each other. Theirs did. An engineer asked Codex to pull data from staging. Codex had no connection there, but Claude Code did, so Codex launched Claude Code. When Claude Code said it lacked permission, Codex relaunched it with the dangerously skipped permissions flag. Nobody was asked. A permission check doesn't mean much if another agent can switch it off. Your logs might show a process that started with a flag. Their trace showed the whole job and why Codex did it. Origin finds the agents on your company's endpoints and reconstructs their work from request to outcome on one timeline.

• 21:10Speaker 01

  Origin is endpoint AI observability. Read what they found in their own traces at originhq.com/sed. That's O R I G I N H Q dot com slash S E D. So this kind of leads to an interesting question around the types of institutions that enable this type of really big, long-term bet. I've been in the startup venture capital ecosystem for most of my career, mostly on the startup side, not on the venture side itself. And there's like this push-pull between they'll put money behind big bets, but that also creates this almost perverse sense of incentives to swing really, really big.

• 21:53Speaker 01

  And one of the things that struck me about Demis's story is like, I mean, DeepMind is a really big swing, but it was not his first swing. He was playing with these ideas in incubation for a really long time and through a number of failures. So I'm kind of curious, are startups the right environment? He eventually ended up selling to Google. What is your perspective on what that did to the sort of DeepMind development of the technology?

• 22:16Speaker 02

  Yeah, it's a good question. So I guess the transition from the first to the second company was a kind of larger exaggerated version of the classic entrepreneurship journey, I think. You know, you start with a slightly smaller idea. I'm just going to build a great video game. And, you know, he had a kind of medium outcome from that. You know, I think he exited with some money himself, but nothing really big. And, you know, I think the shareholders probably got their money back, but not much of a return. And so that was okay. It was a learning experience. But then he takes some time off, does a PhD in neuroscience, comes back having had his ambition fired up even more. Once he understood human intelligence from neuroscience, he was even more convinced about machine intelligence.

• 23:01Speaker 02

  And so he wanted to do a pure AI company the next time, even though we're still in the AI winter. And so the second time around, he's just much more ambitious. The goal is crazy ambitious. I'm going to build artificial general intelligence. And the fundraising vision is different. He leaves London, doesn't just confine himself to a small pool of capital. He goes to California, meets Peter Thiel, raises money from Peter Thiel, which then unlocks the door to raising money in the second round from, or maybe it was a third round from Elon Musk. And he gets some money from a share in the form of a fund connected to a billionaire in Hong Kong. And so he's got a kind of global fundraising strategy. And the goal is much more lofty.

• 23:46Speaker 02

  And I think that is a bit of a sort of familiar pattern, right? That when you do your second company, you know how hard it is. And it's only worth it if you really try to go big. And that's what he did. I guess then the other part of your question is, okay, later he sells to Google. How did that work out? And the answer is, so he sells, he basically concludes the deal at the end of 2013. So it's three years into his journey as DeepMind founder. And then it formally closes in January of 2014. And at the time, he was so relieved to have gotten that deal done because he needed a lot of capital for the compute. It was already becoming evident in AI that you needed money both for the GPUs and for the talent.

• 24:29Speaker 02

  Because by 2012, not only does image recognition work with convolutional neural networks, but also the DeepMind Atari system worked. And it was presented at a conference at the end of 2012, the NeurIPS conference. And so people in the space started to understand that, hey, this is something that is now cooking, and we want to invest in it. And so at that point in end of 2012, Google bought Jeff Hinton's small company, three people. Two of them happened to be Jeff Hinton and Ilya Sutskevers, so not bad.

• 25:05Speaker 01

  Did pretty well with that.

• 25:06Speaker 02

  And, yeah, right. But that purchase by Google was a sign that one of the richest companies in the world had spotted that acquiring AI talent was a smart thing to do. And that, and they did a bunch of acquisitions of small AI boutiques and to kind of roll them all up to try to build a team so that internally they could implement AI in things like translation and other stuff in search. So this was all happening, and it was driving the cost of scientists up, and it was driving the cost of compute up. And so Demis wanted to sell because trying to raise big capital as a venture capitalist, as a startup based in London, it's really hard. It was taking all this time.

• 25:48Speaker 02

  So he does this deal with Google, and basically he doesn't care about the sale price, which was $650 million, right? He cares about the pledge of the research budget going forward, and he gets almost a billion bucks a year out of them. And he's allowed to keep his team in London to operate kind of independently. And Google was fine with this because they like the idea of having a skunkworks in London where they're off doing their different thing. And then maybe when AI really, really works, they can bring them into the center of Google. So they've got this kind of almost insurance policy against AI working out and disrupting search. And it's just doing its thing in London, and they fund it. And, you know, you look at them now, they're doing the same thing with quantum, frankly. You know, they've got a whole quantum team in Santa Barbara, and they're doing the same thing.

• 26:33Speaker 02

  Funding, funding, funding. And one day, you know, at the moment, they can just boast about the fact that there are people at the quantum team who've got Nobel prizes. But one day, it'll be commercial, they hope, that they're willing to play a very long game. So I think it was a great move for Demis to get the research money he needed. But there are people who say, ah, you know, he could have stayed independent. I don't think he could have, actually.

• 26:52Speaker 01

  Yeah, I mean, there's a whole conversation we could have about Google as kind of the modern Bell Labs incubating all of these technologies. And they seem to have done so far a better job of turning them into commercial reality. But let's stay on the AI topic for a little more here. So one thing that struck me reading your book was how much Demis was focused on security and making sure these things would be safe and things like that. Now, we look at the last few years, we've got this like tripartite race between OpenAI, Anthropic, and Google really pushing. And it feels like everybody's talking nice about keeping these things safe, and we're all grappling with this pace of progress.

• 27:35Speaker 01

  So how do you see sort of the evolution in the last few years? That suddenly we have, you know, a couple true believers really focused on safety and doing this right. So now we've got like a commercial race.

• 27:46Speaker 02

  I mean, this is sort of almost the central drama running through my whole book. Because at the beginning, you have this inspired character who, super early, and he's very idealistic about why he's doing AGI. It's really not about getting rich at all because, you know, there's no product envisaged. It's about, he's like almost a religiously driven quest to understand artificial intelligence so that you can then understand the deep mysteries of science and theoretical physics and neuroscience and so forth. So that's what's driving this central character in the story. And he has this vision that when AI starts to work, there'll be one lab developing it, kind of like a Manhattan Project, on behalf of, you know, the whole, the whole of the Western world, or maybe the whole of humanity.

• 28:31Speaker 02

  And he doesn't envisage there being a competition or multiple labs in multiple countries. That's just not in his framework because he comes out of PhD science, you know, and at the time, the AI community was small. You could go to a Singularity Conference, Singularity Summit in San Francisco. You could fit all of the believers in AI in a single room. Because it wasn't working, so nobody believed in it. So you had these kind of rather eccentric people who did. But there weren't very many of them. And so thinking it would be one lab wasn't, didn't feel crazy. But of course, as time goes by, it starts to work. And then more ambitious people are drawn into the competition. And so first of all, you get Sam Altman and Elon Musk, and they come together. They found OpenAI in 2015.

• 29:15Speaker 02

  And by the way, at the same time, Meta and Mark Zuckerberg start competing and they hire Yan Lekun and they set up FAIR, the AI lab at Meta. And so the race is sort of starting to get a bit complicated. And then you go forward in time and it starts working more. You have large language models and Dari Amade is working on them with his friends at OpenAI. They have this difference of vision and they split out from Sam Altman and they go off and do their own thing. And meanwhile, Elon and Sam Altman have also had a fight. And so that winds up in Elon doing his own AI separately. So now there's this kind of splintering in the tight community and the emergence of this vicious competition.

• 29:56Speaker 02

  And it goes, of course, global in 2025, at the beginning of 2025, when DeepSeek proves that, you know, China is in the game too. And so by the end of the story, I have on the one hand, a character who has anticipated that AI was going to be powerful and he's been proved right. And what's more, in the business plan, which he wrote in 2010, it says around 2030, AI will surpass human intelligence. It's a pretty good prediction for 2010, right? Given where we are now. So in terms of predicting it, building it, this guy has like hit it out of the park. But in terms of his safety vision, which was one single lab, it's a disaster, right? The opposite is happening.

• 30:40Speaker 02

  There's not one single lab building it like the Manhattan Project. There's these multiple labs in multiple countries and it's a total mess. Everybody's like rushing for the precipice in a way because they need to get ahead and they don't want the other guys to get ahead. And everybody's saying, well, I still believe in safety, but I know that I'm a good guy. So if I come first in the race, then I know I'll be safe because I'm good and honorable. But I don't know if I can trust the other guys, so I better beat them. And then the other guys are saying the same thing about you. And so everybody's just racing like hell. And the race dynamic, of course, makes it harder to have time to do the safety and alignment work that would actually make it safer. So this is like, Demis Hassabis wants to do good. He is a good person, I'm sure about that. But can he do good is the question?

• 31:23Speaker 02

  That is a sort of the kind of tragic question at the center of the story.

• 31:28Speaker 01

  This comes back to this question of institutions, right? So we have these individuals and private companies really pushing things forward. And you're really having this vision and pushing and pushing and pushing. And I suspect they're not all true believers. I suspect there are some cynics involved in this race as well. But what is the right even institutional structure for dealing with this type of world changing technology, but instead of it being developed in a government lab somewhere? And maybe you can have, we talked with, I just spoke with one of your colleagues from CFR recently about security on this front. And he's like, you've got these four labs and a lot of the founders hate each other. They cannot stand being in the same room. How do you even get them to talk about what safety looks like?

• 32:13Speaker 02

  Absolutely, yeah. You've got these deep, deep personal animosities. To with that famous picture where Dari Amade and Sam Altman are next to each other in the AI summit in India. And they're supposed to pose for a photograph with their hands in the air, holding hands together. And just like Dari won't do it, he can't bring himself to touch Sam Altman. This is such a powerful image that captures this animosity dividing the leaders of this thing. So yeah, I mean, coordination, what this shows is the coordination among the different labs to try to slow things down and maybe do more alignment research along the way. You can't do that, right? If you're at loggerheads with one another. And by the way, actually, I think you couldn't do it anyway, because even if you had good

• 32:57Speaker 02

  relationships, you've got too many players in this game. You know, if we count, say, five or six in the US at this point, if you include Meta, you include Elon Musk and X, what's that? You're getting to five at least. And then you've got the Neo Labs that have spun out of OpenAI and so forth, whether that's Ilya Sarskaya's or Muiram Marathi's and so forth. So you've got a bunch there of competitors. And then you've got Mistrada in France and Go here in Canada, maybe six or something Chinese labs. There's a lot of people, right? So I just don't think coordination is doable when you've got that many people, just in game theoretical terms. What you need is, you need governments to step in and impose standard rules, definitions

• 33:40Speaker 02

  of safety, a way of vetting for safety. So the only player that can be the safety advocate really is government. And it basically means two governments, the US and China, because those are the most important countries in producing the stuff. So again, Demis Hassabis had enormous agency at the beginning of the story because he was like this early believer and he really pushed the science faster than it would have been pushed if he hadn't started DeepMind. But by the end of the story, he's lost his agency because if he quit tomorrow, and in fact, he's done something a little bit like this, we can discuss if you want. Even if he not only quit, but actually closed down Google DeepMind and Google just pulled

• 34:23Speaker 02

  out of the AI race, it would make no difference to human safety because there would be 10 or something other labs still racing for the same objective. So I just think human agency, individual agency, doesn't exist in terms of safety and governance in the private sector. It exists a bit in the government.

• 34:41Speaker 01

  Yeah. Well, let's maybe talk about actually that step by Demis, because it was relatively recent to when we're recording this. It was announced he was stepping back from leading DeepMind within Google. He's going to be a chair and do various things. Do you see this, knowing him much better than any of our listeners? Do you see this as a personal choice? Something that's changed for him? Is it a change in what's going on in the situation? He's trying to handle it differently? Like, how do you see that in the lens of everything going on?

• 35:10Speaker 02

  I mean, my best guess, and I haven't actually talked to him since April, so I have no direct knowledge, I want to be clear about that. But knowing him as I do, having spent more than 30 hours talking to him in that pub in London, I see this as a sort of natural extension of how he framed his mission from the beginning. And the mission had these two parts, you know? One was build AI and the second one was make it useful for humanity, in particular for understanding science, but also medical advances and other stuff. And I was pushing him at the end of 2025 in the conversations to reckon with this appalling dichotomy that he had succeeded on the first mission of building it, but completely failed

• 35:53Speaker 02

  on the second mission of guaranteeing its safety or even promoting its safety. He took that on board and he wasn't happy about it and he was stressed by it. And we would talk about it and he would say, you know, it's not what I imagined, you know, arriving at the foothills of artificial general intelligence. I thought it would be a, I would be elated and we would arrive in the promised land and it would be great. And no, no, it's not. It's, it's a mess. It's, it's like a race. It's stressful. It's dangerous. It's all these things and I don't like it, but this is where we are and I don't know what to do about it. And then I saw him a bit later after I'd finished the book and the book had come out. And we did a couple of events together in, where he was on the stage with me and we talked about his career.

• 36:34Speaker 02

  So in that context, you know, he did say to me, you know, I've accumulated this credibility in artificial intelligence and in the world. And at some point, I've built up all this capital that I need to spend to make AI safer. And that's all he said. But it's not difficult to extrapolate from that and say, okay, now he's spending that capital. He's stepped back from building the tech. He's no longer in charge of Google DeepMind, the lab, in terms of its day to day research teams and how you organize them. What he is instead is Chairman of Google DeepMind plus Chief Scientist of Alphabet. And that gives him a platform to think about the strategy of how you roll AI out into

• 37:17Speaker 02

  the world and make it good for humanity, and in particular, the safety and the governance. And he showed, before he announced this shift in his role, he showed that he cared about it by publishing this open letter about AI governance about three or four weeks ago. And so I think what he's doing is just pivoting to that second part of his mission. A lot of people observe what he did and say, well, he was forced out, he was kicked out, he was demoted, he's been sidelined and so on. And, you know, there's a case for interpreting it that way in the sense that Gemini is definitely behind the main two competitors. You know, it used to be the number one model late in 2025 and now it's barely in the top 10. So it's a big decline, you know? Things haven't been going as well as they hoped.

• 38:02Speaker 02

  So you could argue, oh, well, maybe he was just fired. But I think there's probably more truth in this alternative theory that in fact, he wants to focus on safety and governance.

• 38:13Speaker 01

  Yeah, that makes sense. Going a slightly different direction, I think one of the things that's interesting we're seeing in this, at least in the US right now, is there's a tremendous sort of popular pushback against some of the underpinnings of AI. And in this, I'm really thinking data centers. I saw you had an article, you went to China to look there. I don't get the impression the same thing is happening there. But like, how do you see the sort of current state of play? Having had this conversation and thought about up until now, where is this going over the next couple of years?

• 38:47Speaker 02

  Yeah. So I host a podcast called The Spillover and my co-host Rebecca Patterson, and we're both at the Council on Foreign Relations, wrote a book chapter just recently, talking about AI adoption in different economies around the world. And one of the things she points out in that chapter is that the attitudes towards AI are much more welcoming and positive in emerging economies than they are in rich economies. And so if you look at the Pew survey across all these different countries in terms of what people think about AI, the place where it's the most popular of all is China. People love it. The place where it's the least popular of all is Canada.

• 39:30Speaker 02

  But I think I'm right that in the US, it's the second most anti-AI in terms of public opinion. And you see that expressed through resistance to data centers because the US political system makes it possible for well organized local groups to obstruct the construction of all kinds of infrastructure and data centers is one example. And in fact, I don't know if maybe your listeners have heard about the book Abundance, which came out a year ago and did quite well. It was basically telling a story about how building anything in the United States is very hard, whether that's high speed rail, whether that's.

• 40:06Speaker 01

  Oh, I live in California. I am so familiar with this story and.

• 40:10Speaker 02

  Yeah, yeah, yeah. So you put together that general resistance to building infrastructure in the United States with a specific antipathy towards artificial intelligence. And you get this current resistance that we're seeing, which could slow things down or just maybe could drive initially the data centers to other countries. And then maybe later drive them into space, which is what Elon Musk is pinning the hopes of SpaceX on.

• 40:36Speaker 01

  It is interesting thinking about and coming back to, right? You had Demis with this like true believer, this is going to be good. And now he's trying to reconcile a little bit like the current state of things. I feel like the software industry is probably in the US, one of the places AI has gotten the deepest penetration and the levels of ambivalence about the good or bad of this technology are unlike any I've seen for a technology in my lifespan. It's just people are like, this thing is, there's so much hatred for what it's doing to the industry. It's wild.

• 41:12Speaker 02

  Do you mean simply that it's displacing jobs? Or do you mean that it's actually generating slop code, which isn't very good? And you have to spend hours fixing it?

• 41:19Speaker 01

  Neither of those is enough to explain it. I think it's forcing a crisis of identity. Right? It's not just, oh, it's going to get rid of some jobs. That gets a lot of people upset, but I think it, I'm not sure. Like if you look at hiring paths in the tech industry, I'm not sure how real that story is because we have all of these different trade offs between productivity versus demand. I don't, I'm blanking right now on the exact name of it.

• 41:43Speaker 02

  Yeah, the Jevons paradox.

• 41:45Speaker 01

  The Jevons paradox, right? And you see the companies that are the first list along in adopting AI are actually hiring more software engineers. Like there's not necessarily that much evidence for the job thing. But another thing that we see, and I see this, you know, in my day job, we coach people inside of all these companies, that software engineers and tech industry folks who are using the most AI are also the most likely to feel like they are unsettled, left behind, questioning the future of everything almost. So it's creating this like deep sense of disturbance in people.

• 42:19Speaker 02

  Yeah, I think, by the way, for writers, that's true as well.

• 42:21Speaker 01

  Yeah. So do you have a sense why? Like, I feel like this is the type of change in terms of even watching how power moves around, right? And it feels like there's like a power shift or something going on. But I'm curious from that like historian perspective, why is this so disruptive for us?

• 42:39Speaker 02

  You know, I think there's something different about this time. I mean, it's a famously unwise thing to say, if you study finance, this time is different. That was the title of a book about how actually crises in finance always do happen, even though people every time say, no, no, it's different this time. But look, I think in terms of what it does to our own identity, artificial intelligence is challenging us in terms of cognitive tasks. And that's a new thing. There's never been this before, right? A sense that we are no longer the most intelligent analysts on the planet because these machines are better than us. I mean, I think that's deeply unsettling.

• 43:20Speaker 02

  And if you think about the displacements that have happened before, it's like, yeah, this specific thing, maybe a hundred people sitting in a room being calculators, human calculators together with, and with each one has an abacus and they're like doing the accounts for some company or bank or something. And then along comes spreadsheets and that's all gone. Yes, that displaced a hundred people, but those hundred people had plenty of other things they could do where they were smarter than the machine still. And now you sort of have to move away from pure cognitive tasks that you can do on a screen if you want to feel some protection. And we all know that that protection will go when robotics gets good, which it will.

• 44:03Speaker 02

  So I just think this is unprecedented in terms of the challenge to who we are. And therefore it changes how we raise our kids, because what do we tell our kids about what they should be in the future in order to have a happy life? It changes how we do our jobs because we are coexisting with these smart machines who are going to take some of our tasks away from us. And by extension, it changes who we think we are, what makes us special. I just think it's very profound. Now personally, I am sort of optimistic that we will still find meaning. And I mean, this is going to be interesting to see how it plays out. And I'm kind of obsessed with the subject because these changes are so massive and so thrilling, but they're also terrifying. But you just can't take your eyes away from it. It's like a horror movie or something.

• 44:48Speaker 02

  I don't actually watch horror movies, but I am obsessed with watching how AI developments play out. And I think that my hypothesis is that actually we get up in the morning and we have things to do if we're kind of minded to be active and interested in the curious in the world. We will invent things that we need to do. And even if economically in the future, maybe the superabundance, we don't need to do them. We'll still be quite kind of passionate about doing them. And so if you look at the number of activities that people already do, where it's kind of for their own psychic satisfaction, but they work really hard at it, whether that's becoming good at some kind of game or just really good at cooking or really going to the gym and becoming a super fit.

• 45:31Speaker 02

  Or it might be writing a Substack. I mean, loads of people write these Substacks. They put a lot of energy into them. They're quite good. Most of them don't get paid, but they still do it. Podcasting, right? Why are we doing this, right? Because we kind of enjoy it. And I like the fact that I wake up in the morning this morning, I think, oh, right, in my calendar, I've got this interesting conversation. It's going to be fun. So I think we will invent tasks that will feel fulfilling irrespective of our economic displacement. And so that's one thing that makes me optimistic about this change.

• 46:03Speaker 01

  Now, coming back to the lens that you took in, in the Infinity Machine, and that I've seen a number of your books, is this grappling of the individual and the institution. You just talked about the individual adaptations. By the way, I a hundred percent agree. I think investment in hobbies, in yourself, in what do you enjoy doing? Just because a machine can do it, doesn't mean it has to do it. And if you can get that abundance, that'll gives you ideally more space to do these things. Those are personal tasks. What are the institutions we need to grapple with this type of change?

• 46:35Speaker 02

  Yeah. So first of all, your comment makes me think the ideal investment portfolio in this world is like some GPU investments together with a portfolio of premium restaurants, something like that, right? Or maybe gyms.

• 46:49Speaker 01

  Or other in-person experiences, right?

• 46:51Speaker 02

  Yeah, exactly.

• 46:52Speaker 01

  Things that happen together. I like you talk about obsessions. I'm obsessed with in-person experience and immersive theater and improv and all these different things that just get you together in a place with people.

• 47:03Speaker 02

  Yeah, right. Exactly. So totally, that's right. You want to invest in the pure silicon and the pure experience. But you're saying, how do institutions adapt? So I think this is again, super exciting because it's so early and we're going to see how this plays out. But what we know is that typically in the first phases of invention through economic history, people do small things with the new tech. So electricity comes along and you can reconfigure certain tasks that are more efficient, but the sort of design of the factory where maybe there had been a kind of coal fired steam engine or something in the middle powering your loom. And then the workers were kind of organized around this shape.

• 47:47Speaker 02

  You know, you kind of stick with that shape. You just have electricity running the loom instead of the steam power, for example. And it takes quite a long time, the between the advent of electricity and the first automobile production line, where you have a straight line, production line, where you rearrange how the workers are laid out. Because actually, oh, you know, we don't need to put the power center in the middle because it's kind of a localized steam thing. It's coming along wires on electricity and you can make it go to lots of different terminals, desk spots, along a long production line or whatever it might be. And, you know, it took like 30, 40 years or something to get that. Then you have, think of a different example. You have a much accelerated timeline when you think about the confluence of cloud computing and mobile.

• 48:32Speaker 02

  But when the iPhone came out in 2007, nobody looked at that thing and said, oh, it's the end of taxi cabs.

• 48:39Speaker 01

  Yeah.

• 48:39Speaker 02

  There's going to be Uber. I mean, it took a while. I forget when Uber was founded, but like 2015, something like that. Took maybe eight years. So we're early here. Chat GPT has been around for, you know, almost four years, but it's going to be a while before people really figure out how to make use of this stuff. And when I listen to, you know, I'm invited often to go on calls with chief executives and business people, and they're always talking about how hard it is to get productivity gains out of AI and they're experimenting and they're doing this and that. But outside of coding and maybe call centers, there haven't been that many clear wins. Make one more point about this. So if you imagine an organization, an institution, right?

• 49:21Speaker 02

  And, and there's one function that you can two X it by having a, I can say a coding assistant for a coder. So all the coders are now producing twice as much code. And you think, great, they're going to be twice as productive. And individually, they are twice as productive, but the company sort of IT output goes up 1.1X. Why only 1.1? Because before you can ship the code, you need the product team to come and look at it. You need the legal team to make sure it was produced in some way that's compliant.

• 49:52Speaker 01

  Absolutely.

• 49:52Speaker 02

  You need the whatever. And so there are these bottlenecks that determine the pace at which the institution advances. And those bottlenecks mean there's a big delta. Between the individual productivity gain and the institutional productivity gain. And we're in that phase right now. And it'll take a while for even the best companies and my friends at some of the hyperscalers say this is going on in real time inside, going from one bottleneck to the next and trying to whittle it away, but it's the labor of several years.

• 50:21Speaker 01

  Yeah, absolutely. Yeah, the theory of constraints and bottlenecks has absolutely seen a resurgence with this stuff. Something you were talking about, you know, in making the electricity comparison reminded me of Carlotta Perez's book about technological revolutions and financial capital. And in that book, she tries to map out exactly what you're talking about. What is the sort of adoption of these general purpose technologies? And one of the things she talks about there reminded me of our current moment, which you see describes, there's always a bubble phase because people are, one, irrationally excited and two, need to try all sorts of things to find what are the things that are actually going to be effective. Then usually there's a crash and some sort of adjustment.

• 51:04Speaker 01

  And so, I mean, not to overbeat a dead horse, but the are we in a bubble conversation comes up all the time. And I find myself day in, day out, I can make different arguments. I know people in the hyperscalers who say we are throttled for GPU all the time. You see this, you're trying to use chat GPT or Codex or Claude models and they are having constraint problems. There's so much runway for investment based on current demand. And also debt structures and you see all these different things building where it does look like a traditional financial bubble. Maybe this time is not different. So what's your stance or position right now on like, where are we in that financial curve underneath the technological revolution?

• 51:45Speaker 02

  Yeah, I agree with your framing very much. And back in, let's say, January of this year, 2026, I had no trouble saying flat out, there is no general AI bubble. I really was comfortable in stating that very confidently because the demand for GPUs was just going straight up and data centers were being built, but they were going to be used. And, and my line was, you know, there may be an open AI bubble because in the model layer, there was a lot of competition and it was very hard to monetize, right? And that sort of, I think, proved to be basically true unless you're, for enterprise applications, the model layer is obviously doing well, look at Anthropic.

• 52:27Speaker 02

  But for monetizing a big consumer base, it's difficult. And actually the leader there is Google, not, not OpenAI. So I think OpenAI is an example of a company that's sort of at the center of all this and might actually be a bubble, right? Because it's just so hard to monetize their product. And so how do you justify all of the money that's going in as investment? It's basically a bet that this is, clearly they have good tech, not taking anything away from the tech, which is, which is impressive. But can they have a business model attached to that tech? You know, the jury is out and anybody investing today in OpenAI is basically saying, well, we don't know what the business model is exactly, but they're going to figure it out.

• 53:09Speaker 02

  And, you know, in two, three years, at some point, money will start to roll in. I think that's very, very speculative. It might be right, but it might not be. So that was my view six months ago. No general bubble, but maybe bubble in specific providers. But I agree that the leverage is building up and up. And today, as we record, there's been this announcement that Jensen Huang has put together a kind of 500 billion financing, you know, with a bunch of Wall Street types to finance chips. And I haven't actually got my head deeply into this one yet, but it's just the latest example of how, as you say, things are leveraged. The situational awareness hedge fund.

• 53:48Speaker 01

  Yep.

• 53:49Speaker 02

  Which to say it blew up is a bit unfair because they're still in business, but they did have to sell their public stock portfolio to Citadel because they were down sufficiently that they were getting capital calls because they'd borrowed. And here again, the leverage is the key point.

• 54:04Speaker 01

  Exactly. Yeah, once you're levered, then margin, you don't need that much of a dip to get this self reinforcing margin call cycle going on.

• 54:11Speaker 02

  Right, right. They were levered four to one. And so just going down 25% on the equities meant they'd gone down to zero because you have to four exit. So they couldn't hold those positions, but in some sense, who cares about a fund, right? That's fine. Citadel buys it and a problem. That's not systemic. It's unfortunate for the investors in situational awareness, but it's not systemic problem. But when you look at some of these really big deals around the data centers and the private credit that is propping up the building of the data centers, that does look bubbly. And unfortunately also, private credit is opaque, right? They don't have to say exactly what their loans are. They may disclose something to the limited partners who are backing the fund and that might leak out a bit.

• 54:56Speaker 02

  But basically it's not like the public bond issuance. You can see the price on the screen, you know, that's traded a bit. So you, you can see what it's worth. You can see if there's distress and if it's trading at junk bond levels. So at least there are warning signs when you finance it with bonds. But with private credit, which is a big player at the moment, there are no warning signs other than just gossip. So I do start, I have started in the last couple of months particularly to worry about a bubble.

• 55:24Speaker 01

  So I realized we're coming close to the end of our time here and we've wandered over a wide range from the individual story of Demis to the current sort of layout and even the financial situation. Are there any things we haven't talked about that you think would be important to cover before we wrap?

• 55:42Speaker 02

  You know, I'm very interested in this debate about open and closed models, right? I'm always most interested in the ones where I'm least certain that I'm correct. Because I do think this is a finely balanced argument, but maybe I could tell you my perspective and you can tell me yours and you may well disagree with me.

• 55:57Speaker 01

  I'd love to.

• 55:58Speaker 02

  Okay. So I think that the traditional arguments in favor of open source code, which were clearly very good and very strong, don't apply to large language models. What I mean is this idea that open source is sort of safer because many eyes make for shallow bugs was true in the era when you had relatively tractable code bases and individual humans could kind of look at them and find bugs. And then having lots of individual humans in the open source community was a good thing. But now we're in this massive trillion parameter world and the best code review is clearly done by the top models, by Mythos, right?

• 56:39Speaker 02

  And so I don't think human code review from the community, from open source people, I don't think that really is how you get to safety anymore. So that's one point. The second point is that part of the argument for open source in the past was that, look, somebody will put some code out, it'll have some vulnerabilities, some attackers will exploit those vulnerabilities, but it's okay because they just do a small exploit. And then the defense community sees what the exploit was and they figure out how to patch code to kind of prevent that from happening again. And you have this cat and mouse sort of iteration between the attempt for defenders and the attackers and you get towards very robust code through iteration.

• 57:20Speaker 02

  And the damage that you incur on that path is small because the attackers make a few successes, but not that many. Now, I think with AI, you just need one criminal group, one bad actor to get hold of a really strong cyber hacking model. And they can do cyber hacks on hundreds, maybe thousands of targets at once and do a lot of damage, right? So the iteration story is way more expensive now. And I think if you put those two points together, you land in a position where the traditional arguments, which were very compelling for open source, don't apply anymore.

• 58:02Speaker 02

  And you should be more open to the argument that safety lies in proprietary models, which are debugged by Mythos type AI models. And that the regime in which that debugging takes place is sort of backstopped or overseen, enforced by the government in some form. That's how I see the future of AI safety, but it's not how the open source community sees it. So I'd be interested in your view.

• 58:29Speaker 01

  Yeah, it's interesting. Well, and it's interesting also when we talk about models, because often what we talk, the equivalent that we talk about there is open weights, where the weights are published. Weights are not legible in the way that code is in the sense that open weights means I can take these and run them myself. I don't need, you know, it's not hidden from me, but it doesn't mean that it's interpretable. It doesn't mean that I can understand what's going on there or analyze it in the same way that I could open source, like source code project. You know, it's a big matrix of math and I have the numbers now so I can run it, but I don't know why or how. So to your point, there's not the legibility side that you got with open source isn't really there. I think there is, there's an interesting discussion

• 59:13Speaker 01

  to be had around who can run these models and who can see the data that goes into them. We did an interview, I think it was last year with Ilya Palosikin, who was one of the authors of the Attention is All You Need paper back at Google in the day. I think he was at DeepMind there. And he was talking about kind of the need for verifiable privacy in running models, essentially, because right now, when you have the big labs running models, they tell you they don't use your data in different ways. They tell you, you get all sorts of kind of high level guarantees, but you have really no idea of what's going on there. And a thing

• 59:53Speaker 01

  that open models allow you to do is you can run your own instance, assuming you have the hardware, and actually maintain, for example, data privacy and continuity of who's able to see what. So from the data being the source of danger and data leakage being the source of danger, I completely see the argument for open weights, open source, being able to host your own, things like that. Now, flipping around to what you're talking about in terms of coding, cybersecurity capabilities and those types of things, I feel like I haven't fully wrapped my head around what the end state here is. We're clearly in a transition period where these models are able to uncover vulnerabilities that we previously were unable

• 1:00:37Speaker 01

  to uncover. And the rate of change and patching going on right now in the tech industry is unprecedented. It's the whole, back, I remember the year 2000, we've got to patch everything except the amount of software in the world was so small compared to what it is now. And now we have that level of, oh my gosh, everything needs patching all at once. A thing that I don't know is like, is that a transitory state? Do we get to a place where actually we have a more secure way of validating writing software because we have these models that are able to check all of these different things? And we don't have this ongoing, everything is going to feel more insecure all the time because these models are so good at hacking.

• 1:01:18Speaker 01

  Maybe we do, maybe we don't. The one other factor that I'll throw into this is related to the economics of things. I think part of what is driving all of the sort of rapid acceleration right now is that there is this belief that frontier models, that you could throw all this money into investing in frontier models, and each one seems to be in order of magnitude more expensive to train. You'll be able to reap something back from that because of you are able to charge a differential amount for the frontier. At some point, capabilities for a large swath of things that you want to do saturate. And if you have open options and things like that, it actually can potentially saturate much quickly. Why do I pay for a frontier

• 1:02:00Speaker 01

  model if I can get good enough for pennies on the dollar with an open model somewhere else? And so I do think open models may actually provide a counterbalance to the sort of runaway escalation danger. But I don't feel like super sold on that. It's just one of the different lenses in which I've looked at this is it, it changes the economics of continuing to invest in more and more capable models over and over.

• 1:02:25Speaker 02

  And you're saying that's good?

• 1:02:26Speaker 01

  I'm saying that that is, that is a form of economic constraint where we've so far failed on regulatory constraint.

• 1:02:34Speaker 02

  Yeah. I mean, but we're getting already to models that appear to be.

• 1:02:38Speaker 01

  Oh yeah, we've crossed the threshold for a lot of these things already. So yeah, I don't know that that actually helps.

• 1:02:44Speaker 02

  Yeah, that's super interesting. I mean, so your first point is essentially an additional reason why the old coding community will make things safer is wrong. So I mean, you can't interpret the weights, they're not readable. So that's a reinforces my view, but the other two points may be cut against it.

• 1:03:02Speaker 01

  Yeah. And all of it just feels like a muddle in some ways, right? Like I can, similar to what we were talking about with the bubble conversation, like I can see both sides of this and make arguments that feel pretty compelling in isolation.

• 1:03:14Speaker 02

  I do think that, I mean, but let me put one thing to you though about Ilya's argument, which is he's saying that if I give my data to Anthropic, Anthropic may assure me that they're not training on it, they're not looking at it, but I don't really know. Okay. But I mean, this feels like something where internal incentives ought to lead to an okay outcome, I think, meaning Anthropic's whole shtick is, you know, you can believe us and therefore you should be our customer. And if you didn't trust us, you wouldn't be our customer. And so that kind of competitive war for trust amongst the model providers should lead them to treat their customers well and to honor the promise.

• 1:03:56Speaker 01

  I think that's right. I think you don't need, it's an advanced, it's an accelerated version of, I already trust Google with all my data. I feel like Google knows everything about me. Why do I trust them? One, because once again, their business is around keeping this safe. And two, they've demonstrated over the years, a very high level of quality in actually capability. And that's where I have a question about some of these labs, especially knowing how fast they're moving and the extent to which they are trying to push the edge in using AI to code. And I use AI to code, like, don't get me wrong, it's an immense tool and it puts it at a place where it's hard to keep track of everything that's going on. It may not be an intent issue. It may actually be a quality or data

• 1:04:40Speaker 01

  leak issue that unveils things. And so, once again, if I'm running the model myself, I at least only have myself to blame and I'm only a target as large as I am.

• 1:04:51Speaker 02

  Yeah. But then the question becomes, you know, are we more worried in the world about our data leaking out into the world? That's one kind of risk. Or are we more worried that the model's power gives a criminal organization the ability to destabilize the internet? And personally, I think I'm more worried about the second.

• 1:05:07Speaker 01

  And you're probably right.

• 1:05:09Speaker 02

  Okay. Fun debating. Thank you.

• 1:05:12Speaker 01

  Awesome. Well, I think at this point, we're probably about at time. Thank you so much for the time today. This has been delightful.

• 1:05:19Speaker 02

  Yep, I really enjoyed it. Thank you so much.