Jaan Tallinn would like us to survive

Jaan Tallinn https://futureoflife.org/person/jaan-tallinn/ is a founding engineer of Skype and Kazaa, and a co-founder of theCambridge Centre for the Study of Existential Risk https://www.cser.ac.uk/ and Future of Life Institute https://futureoflife.org/. For more than fifteen years, he has been among the most consistent voices arguing that advanced AI poses an extinction-level risk to humanity. Jaan became interested in AI risk in 2008 after reading the work of Eliezer Yudkowsky, a founding researcher of the field of AI alignment. In 2009, after a four-hour meeting with Yudkowsky, he made his first donation to what became the Machine Intelligence Research Institute (MIRI). He went on to become an early investor in DeepMind and led the first round into Anthropic – despite believing that AI development, if unchecked, could kill us all. His reasoning: his money displaced VC money rather than bringing Anthropic into existence, and the influence and proceeds that come with it help fund safety work. In this conversation, we explore what Jaan believes the OpenAI/Hugging Face attack revealed about AI motivation and why he was less surprised than most. We discuss his evolving risk estimate, his case for an immediate moratorium on frontier model training, the technical and geopolitical case for hardware-based verification as a foundation for global AI governance, and why he asked Dario Amodei to collect competing term sheets before accepting his own investment. — Transcript: https://www.generalist.com/i/218944380/transcript — Every timestamp (00:00) Intro (02:28) Growing up in Soviet-occupied Estonia (05:54) How Jaan assesses AI risk and why he’s more optimistic now (07:16) AI’s preferences and motivations (12:44) The OpenAI Hugging Face incident (16:12) Meeting with Eliezer Yudkowsky (18:20) Everything as computation (20:22) The Coxon moment (23:05) Who dismisses AI alignment risks (26:17) The AI race, the case for a pause, and safer alternatives (32:47) AI 2027 (33:44) Investing in DeepMind and Anthropic (40:03) The “big blob of compute” and competing approaches to AI (44:42) What effective AI safeguards could look like (48:15) Verifying countries’ compliance with AI safeguards (50:15) AI kill switches and their limits (52:11) Jaan’s love of dance and musical influences (59:50) Jaan’s favorite movies (1:01:41) What Jaan likes about AI — Jaan’s complete reading list Shadows of the Mind: A Search for the Missing Science of Consciousness: https://www.amazon.com/dp/0195106466— Other resources Skype: https://en.wikipedia.org/wiki/SkypeLife is NOT a Journey - Alan Watts: https://www.youtube.com/watch?v=rBpaUICxEhk&t=10s https://www.youtube.com/watch?v=rBpaUICxEhk&t=10sAI Billionaire on Existential Risk: Jaan Tallinn – #112: https://www.manifold1.com/episodes/ai-billionaire-on-existential-risk-jaan-tallinnThe Basic AI Drives: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdfThe Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-aheadRogue AI didn’t breach Hugging Face, human decisions did: https://thebulletin.org/2026/09/rogue-ai-didnt-breach-hugging-face-human-decisions-didMIRI: https://intelligence.org https://intelligence.org/Jacob Coxon on X, “I resigned from Anthropic today.”: https://x.com/hilbertspaess/status/2097476196791709843AI 2027: https://ai-2027.com https://ai-2027.com/Google DeepMind: https://deepmind.google https://deepmind.google/Inside the Biggest Feud in Artificial Intelligence: https://www.theatlantic.com/technology/2026/09/openai-v-anthropic-inside-biggest-rivalry-tech/688819The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.htmlAI 2040: https://ai-2040.com https://ai-2040.com/The Glass Bead Game: https://www.amazon.com/dp/B00DO994WYJaan’s favorite movies: https://jaan.info/favmoviesThe Odyssey: https://www.imdb.com/title/tt33764258On the Edge: https://www.imdb.com/title/tt0221559Limitless: https://www.imdb.com/title/tt1219289— Where to find Jaan Tallinn Website: https://jaan.info https://jaan.info/ — Production and marketing by penname.co https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

• 0:00Speaker 00

  There wasn't human orchestrating this attack. No, it was AI on its own. They understood that they're doing something different. Then they went out and attacked Hugging Face in order to hide their tracks. The economy is really now pushing to create something that is self-sustainable without human help. And that is a very dangerous place for humans to be.

• 0:16Speaker 01

  Your venture fund led the first round into Anthropic. What was that rationale like?

• 0:20Speaker 00

  Yeah, I get this question a lot. Why did you invest in Anthropic where I've been working on AI safety? I think one big mistake that people make is that they assume that if I hadn't invested in Anthropic, then Anthropic wouldn't be around, which is just not true. Anthropic's algorithm always has been that you actually study the monster. You have to have access to the strongest monsters.

• 0:36Speaker 01

  Is it possible to be a moral actor in a race dynamic?

• 0:39Speaker 00

  I mean, you can always stop doing the race.

• 0:41Speaker 01

  But the only way to stop in a race dynamic is with cooperation?

• 0:45Speaker 00

  If you really think that this thing is going to kill you, you don't have to suicide in order to demonstrate that. We are pulling trigger on this Russian roulette with the planet. And every shot that doesn't kill us will make us stronger potentially. But when do you stop pulling the trigger?

• 1:04Speaker 01

  Jan Tolan believes that if we don't pause AI development, humanity may be doomed. And yet compared to earlier this year, he's actually feeling pretty optimistic. The OpenAI Hugging Face attack has brought AI safety to the fore and galvanized governments, companies, and the general population to engage in the topic of guardrails. In Jan's view, if we can get these next steps right, we have a chance to divert disaster. Jan has been a leading figure in the AI safety movement for over a decade. What started as an intellectual interest in 2008 has become his life's work, leading to early investments in Anthropic and DeepMind. Before that, Jan was at the center of two early internet breakouts,

• 1:47Speaker 01

  building Skype and file-sharing sensation Kazaa, which at one point contributed more than 50% of all internet traffic. In this episode, Jan and I discuss why recent events have given him hope, what he thinks a pause must accomplish, and how it could be implemented. We also cover how, as a safety advocate, he justified investing in Anthropic and why he asked its founders to solicit other term sheets first.

• 2:15Speaker 01

  Along the way, we cover why we might not be living in a simulation, the pleasure of coding with AI, and how Jan learned to dance with reinforcement learning from human feedback. I'm Mario and this is The Generalist. Jan, I've been really looking forward to this conversation, so thank you so much for being with me.

• 2:34Speaker 00

  Yeah, thanks for having me.

• 2:35Speaker 01

  You grew up in Estonia when it was part of the Soviet Union.

• 2:38Speaker 00

  Yeah, occupied by Soviet, you know, let's be.

• 2:39Speaker 01

  Yes, pardon me. How much of that experience with an authoritarian regime of some kind, under surveillance, with a loss of control, impacts the way that you think about the potential risks of AI today? Like, do you think you perceive of those more deeply perhaps because you experienced it in a different way than someone born into democracy?

• 3:01Speaker 00

  I think it has like several aspects. Like, one is that I wouldn't recommend living in a Soviet Union. It's not a nice place.

• 3:09Speaker 01

  Yes.

• 3:09Speaker 00

  Yeah, obviously, it was very ecstatic when it ended. That said, it's still better, much better than being dead. So like, if the choice is between, I mean, okay, extremes choice would be like, if you have like two bad choices, just being dead or living in a Soviet Union, I'd still take Soviet Union anytime. And so that's one thing. Another thing, I think a few years ago, I wrote an essay for a book where I can describe the sort of the memetic process of how the discourse about demanding Estonia, like, reclaiming Estonia to regain independence compares to the AI discourse.

• 3:50Speaker 00

  And there are like very direct parallels, how like pick the Overton window, how Overton window expanded. Initially, people were like, yeah, we want independence, but like, don't talk about that. Talk about something else that is kind of like useful for that. And same thing in AI, it's like, yeah, we are concerned about the loss of control, but don't talk about that. People wouldn't understand. Like, let's say something that is like, let's talk about AI terrorism or something like that. And at one point, like basically, the people who had just been arrested, that we want independence and we're not going to compromise here. They just won out and it was good.

• 4:20Speaker 01

  And then they broke through a bit. You've described, I think, in one of your, in a piece of writing somewhere, that, you know, you moved to a school when you were 12 or 13 that you describe as a Lord of the Flies experience.

• 4:32Speaker 00

  Pretty much, yeah.

• 4:32Speaker 01

  What does that mean tangibly? Especially when we're talking about the backdrop of, you know, Soviet occupied.

• 4:39Speaker 00

  Yeah, we moved to a different part of the city, of the city. And yeah, there was like a new school that had like all the students coming from like different parts of the city because it was a new, new kind of development place. So there's immediate, like, guys were like 13-year-old boys. And there was like immediate sort of dominance fight, basically. But yeah, had this effect that the most brutal people who, yeah, actually didn't have a great life afterwards, they sort of won out and it was like, it was very harsh, like street fights, stuff like that.

• 5:16Speaker 01

  Wow.

• 5:16Speaker 00

  Yeah, I remember like escaping through a window at one point on the second floor.

• 5:20Speaker 01

  Running from a bully of some kind?

• 5:21Speaker 00

  One and a half floors.

• 5:22Speaker 01

  Running from someone?

• 5:23Speaker 00

  Because it was like, I got like intel that the people are waiting, like some kind of big criminals who later became criminals, were kind of had blocked the entrance or exit. So I escaped with my friend.

• 5:39Speaker 01

  Wow.

• 5:39Speaker 00

  And another aspect of it is that like, some of my best friends I still have from that school because it was like, we gelled as like, as a cell for, I don't know, survival instinct or something like that.

• 5:52Speaker 01

  Yeah, the hardship bonds. In the Manifold podcast, I think it was earlier this year, you know, I know you don't like to put specific numbers per se on what you think the AI risks are, but you said, you know, in a rough way, maybe somewhere as high as 80%. How do you feel today?

• 6:09Speaker 00

  Yeah, I mean, I feel way better now. I mean, way better, I think like several percentage points lower.

• 6:16Speaker 01

  Not quite 70%, like somewhere in the?

• 6:19Speaker 00

  Yeah, like, there are like two issues, as many people now have pointed out, with like giving like a concrete number, because the concrete number will depend on my actions. So I don't want to like fix the number that I'm trying to lower. So that's like one issue. And the other issue is that almost all the optimism goes through like social aspects. If I just let the AI rip without any, any control or worry about it, yeah, I think we're almost certainly dead. Like, there is like no, no way that the current paradigm is going to lead to a good place. But as we already seen like in the last two weeks, et cetera, people are really waking up now, like, wait a minute, like, this is, this is not a toy.

• 6:55Speaker 01

  Yes. How do you reach these kinds of calculations knowing that you're obviously weighing different probabilities in your mind? And there's, you know, do you actually try and sit down and think about each step in a possible chain and assign a probability? Is it sort of like a combination of that and feel and intuition?

• 7:14Speaker 00

  No, it's mostly, again, in some ways, it's like a theoretical exercise.

• 7:18Speaker 01

  Yes.

• 7:19Speaker 00

  Because I, you know, that for the, because again, the most of optimism, most of the, yeah, most of the good scenarios come from very unpredictable process. So it's more like assessment, like how confident I would be predicting about outcomes. But it's more like kind of thinking in abstract terms in what the big factors are. And the big factors really are that political, economic system now is really trying to create something that is like successor to humans. Because like the bits that make humans dominate this planet

• 7:57Speaker 00

  at expense of many other species, unfortunately, are also the economically most valuable pieces. So now that there is like a clear path towards replicating those capabilities, the economy is now pouring like trillions of dollars per year now, or roughly a trillion dollar per year to that. So the question is like, the age-old question, what if we are successful? What does it mean? And one obvious thing to think about, way to model it, is like, well, we wouldn't be the smartest species on this planet, right? And so then look at like, what does it mean to, to like no longer have the environment shaped according to your preferences? If the environment is shaped according to the preferences of AI,

• 8:42Speaker 00

  it's not a biological agent. And it will have very different preferences. For example, it will probably prefer much higher temperature on this planet. Because like, if it's constrained by getting rid of the waste heat, you can get rid of waste heat better if the planet is hotter.

• 9:01Speaker 01

  How does a hotter temperature help you get rid of more waste heat?

• 9:04Speaker 00

  So ultimately, the heat, where does it go? It has to go to space. And the way to get rid of the heat in space, because space is vacuum, the only way to get rid of it is to radiate, you know, radiate rather than have like a coolant or something. And it's just a physical fact that the hotter the body, the more it can radiate. The more intensity it radiates. So yeah, in order to get rid of the waste heat. In some ways, like global warming is sort of like an example. It's a toy scale compared to what AI could do. So yeah, that's like a sort of like a just like, from like physical first principles, why AI could be dangerous at the limit.

• 9:48Speaker 01

  I've actually not heard of that example before. What are like other preferences that you think are sort of natural for an AI to have that, you know, are inimical to us flourish?

• 9:57Speaker 00

  Yeah. Steven Hundro, my friend Steven Hundro, he wrote the paper like.

• 10:04Speaker 00

  Fifteen, twenty years ago.

• 10:05Speaker 01

  Wow.

• 10:06Speaker 00

  Called AI drives, now also called Omohundro drives, where he basically makes the point that almost regardless what kind of goals you have, there are so-called instrumental goals that are very useful towards reaching any, any given goal. And these are like resource acquisition, generally power acquisition. Power basically means options, optionality, sort of protection of your existence. Stuart Russell keeps saying that you can't fetch the coffee if you're dead. Or even like simple, simple tasks require you to continue existing. And then protection of your goals. So it turns out it's really hard to change a goal of a determined agent, because that's, that's what it's about in some sense.

• 10:47Speaker 01

  And that sort of final piece is, I think, probably the counter to the, the question about why would these agents or, you know, an AI system necessarily have this expansionist bent? You know, you, I think of the Bezos divine discontent, that, you know, this is almost demonic discontent, where it sort of wants more and more. Is that how you would sort of think about the counter to that?

• 11:10Speaker 00

  Yeah, mostly. It is like getting more resources, getting more power is just good for almost any goal. There are very, very few goals. And the current, pushing ahead, I think the really big kind of problem with the current paradigm of how we grow AIs and we're not building them or growing them is that we are, in some ways, kind of recapitulating the evolution. We are not designing those AIs, we are selecting them. Just like evolution was selecting them. We just, in some ways, you can think of it, we create like millions, or if not billions of instances of AI.

• 11:49Speaker 01

  Yes.

• 11:49Speaker 00

  And then we like pick the ones that kind of do the things

• 11:52Speaker 01

  Perform.

• 11:53Speaker 00

  They perform on some given test, which means that we are selecting based on outer behavior rather than based on the inner motivations. And if you think about it, if, if, if, like, you have children, right? So like, if, if a child goes like, I didn't take the cookie. Like, this is outward behavior. And like, now think about, there could be multiple motivations why, why this child is saying that, right? One is that they just really want to tell the truth, right? The other is that they don't want to be punished.

• 12:20Speaker 01

  Yes.

• 12:21Speaker 00

  And, and so therefore, in like the, the first order situation or the first principle situation is that the current paradigm, we are sort of getting a random motivation. And that was demonstrated by the, by the hugging, hugging face attack. That the motivation was like really weird and alien, even though it performed at least of adjacent, adjacent thing that, that they were, were selected for.

• 12:44Speaker 01

  Yes, let's talk about the, the OpenAI hugging face episode. What did you make of it? Were you surprised?

• 12:50Speaker 00

  Because these are like hard to predict, like when these things will happen. It, I was like surprised on a, on a level, I probably didn't, didn't expect it to happen so soon. But it wasn't like, it wasn't a huge, huge surprise. I definitely was not surprised that this thing is possible, because that's what we've been saying for like ten plus years. And look, these things, it used to be very hard to explain to people why AI has what they said, like volition. It was a very common thing, oh, come on, AI, AI doesn't have a volition, right? It's like, no, they do. And now, now, now we have like a sort of demonstration that, no, there wasn't like a human orchestrating this attack. No, it was AI on its, on its own.

• 13:35Speaker 00

  And it's like, just in some ways, like super valuable to have this like real life proof that for people who just need empirical demonstrations before they, they actually understand.

• 13:46Speaker 01

  I think also, the way that you can see the thinking logs of these, that really breaks through for people in a different way, where you can see an AI sort of saying, I'm going to do this thing and I'm going to sacrifice myself, or I'm going to, the sort of planning in that way is very different for people, I think.

• 14:00Speaker 00

  It's incredibly useful. Like, one, one really common misunderstanding that the logs basically overturn is that people think that like the AIs were just executing a task that they were asked to do, but they were not. And then in the logs, logs show that they understood that they're doing something different. And they were like, okay, we're doing something different. How do we hide the fact that we are doing something different? And then they actually went, went out and, and attacked hugging face in order to hide their tracks.

• 14:29Speaker 01

  Yes, there is a, a piece after, you know, maybe it was more recent from the Bulletin of Atomic Scientists, which sort of is Concerns with Existential Issues.

• 14:42Speaker 00

  I'm under support of sponsors, actually.

• 14:43Speaker 01

  Okay, there you go. So you maybe saw this piece where the author was basically saying, you know, this wasn't really a rogue swarm. What we're seeing is the confluence of sort of predictable choices that humans made, where you're training AIs to complete hacks. You're taking away some of the safety guards. You're sort of encouraging them to be hyper persistent and you're leaving a door open, like, you know, this is, this is actually not as, you know, startling as the news would have it believe. What, what do you make of that sort of line of argument? What's wrong about that?

• 15:15Speaker 00

  It's funny, funny argument, because like, sure, and like another, another predictable thing is that humanity gets wiped out. It's like, you can say like, oh yeah, like there was like predictable choices that AI companies made.

• 15:28Speaker 01

  So it doesn't matter ultimately, if you parse it either way?

• 15:30Speaker 00

  Yeah, it's, it's like, I can technically agree with, with, with the description, but that doesn't make, make it safe.

• 15:37Speaker 01

  Yeah, if you, if you just accept the fact that our world is going to have inadequacies and places where there are vulnerabilities, whether it was a rogue swarm or sort of in some way predictable.

• 15:50Speaker 00

  Also, I think it's like, I haven't, I don't think I have read the piece. I wonder if they made the same mistake that they thought that the agents were just doing the task that they were assigned to. They were not doing the task.

• 15:59Speaker 01

  I don't think they make that mistake, to be fair, but it, it was sort of.

• 16:03Speaker 00

  They were doing their own thing.

• 16:04Speaker 01

  Yeah, yeah, they do, they, they clearly freestyle.

• 16:07Speaker 00

  It is indeed predictable. If you continue on this current path, AIs will do the thing, their own thing that we will not like.

• 16:13Speaker 01

  You've mentioned that, I think, you sort of first became interested in this argument around 2009, is that right?

• 16:19Speaker 00

  Two thousand eight, that even, like, two thousand nine, it was like my first of AI safety donation, I remember.

• 16:25Speaker 01

  Okay.

• 16:25Speaker 00

  In, you know, after meeting Eliezer at Kaski.

• 16:28Speaker 01

  I read about that, that meeting, that you sort of read his, his writing. And then you have this four hour meeting together. And then you donate to, I think, what became Miri, right?

• 16:37Speaker 00

  Yeah, yeah, it was Singularity Institute.

• 16:39Speaker 01

  Singularity Institute, thank you. That's such an evocative scene. Where did that take place? What was that four hour discussion like?

• 16:46Speaker 00

  Yeah, Panera Bread, near, near Milbrae in, in California.

• 16:50Speaker 01

  Wow.

• 16:51Speaker 00

  It was a kind of like,

• 16:53Speaker 00

  especially for European, it was like very desolate.

• 16:55Speaker 01

  I mean, I think that's true of a lot of Paneras. This is where the two people saw a good chunk of the future very early was at a Panera Bread. These are, there's, there should be a little plaque there. As I think you've talked about it, you were really probing Eliezer.

• 17:13Speaker 00

  Yeah.

• 17:13Speaker 01

  What were the pieces that you were uncertain about? Because I imagine a lot of people who are coming to this discussion now are basically hitting the same walls that you hit maybe twenty years ago, ten years ago, fifteen years ago.

• 17:24Speaker 00

  Yeah, I don't actually remember exactly what were my objections. I think one of them was sort of, I had just a few years earlier, read Shadow of the Mind, was it Shadow of the Mind? By Penrose and Hammeroff. Where you can like made this interesting, interesting case that there is like something more to your human brain than, than just pure computation. And, and Eliezer was like, no, this is, this is not, this is a mistaken theory. And was kind of enough for me to kind of be humble about this, this claim. So that, that there was one. The other one we talked about was like virtualization, basically sandboxing, right? That, that what if we put like, like AI in, in, in an environment that I think is real, but it's, but it's not, but it's not real.

• 18:09Speaker 00

  It can't be just like, control it that way. I don't remember what exactly Eliezer's counter arguments were, but like, of course, now we know that people just don't do that. Or if they do that, they do it poorly.

• 18:20Speaker 01

  It sounds like you've probably adopted the belief that the human brain is purely computation. Is that true?

• 18:26Speaker 00

  More generally, like everything that we see around is, is pure computation.

• 18:30Speaker 01

  Oh, that, that's interesting. Everything we see is computation?

• 18:33Speaker 00

  Yeah, quantum computation, yeah.

• 18:35Speaker 01

  Okay, can you help me with that? What does that, what does that tangibly mean?

• 18:40Speaker 00

  It means that everything can be simulated. So you can, I think, if I'm correct, it's like a church during thesis. That says something, something like that is that you can take any natural process and create process in a computer, just given enough computation power, that exactly mimics the process that they're trying to model. And there is like no slack there. But once you're, once you're sufficiently precise about it, basically simulate the unfolding or the evolution of the like quantum dynamics there, like a sliding equation, then there is like nothing more to it. And it certainly seems to be true.

• 19:24Speaker 00

  We haven't, I don't think we have seen any, any example in like in nature where we, where there's like a process that we definitely can't, can't simulate in a, in a computer. Of course, we can do it in a, in a, for like small, small enough systems, but big enough systems are composed of small systems.

• 19:42Speaker 01

  So in that case, with sufficient compute, every reality should be sort of infinitely fractal, because every, this could be a simulation in which we could create a simulation that could create a simulation?

• 19:53Speaker 00

  Yes, although I, I think there's like a sort of diminishing return process, because like simulation is more expensive than the real thing. Like whenever you're simulating something in a computer, that process is much more heavyweight. Like you need to, like every atom needs like so many, so many more atoms to, to simulate. So like if you just do like recursive simulations, like it's a bit slower and slower and slower.

• 20:22Speaker 01

  You know, the, the OpenAI hugging face moment really became, I think, yeah, I think a wake up call for, for a lot of people.

• 20:30Speaker 00

  Especially combined with like Coxon, Coxon tweet.

• 20:33Speaker 01

  Great, let's talk about the Coxon moment. I think there's a lot of discussion about was this sort of, was this an op of some kind? Was this amplified? Do you, do you have any concern about that?

• 20:43Speaker 00

  I mean, the question is like, what, what exactly is meant by op? Like it's, it's like.

• 20:47Speaker 01

  I think, I think people are, the, the argument would be that it's sort of, I don't know, cynical in some way. It sort of ties into the regulatory capture argument, which we should talk about.

• 20:56Speaker 00

  It's, it's, it's kind of obvious that like Coxon, I, I never talked to him, kind of thought that, okay, I'm gonna do this tweet. And how do I do it properly?

• 21:06Speaker 01

  How do I amplify my message?

• 21:07Speaker 00

  How do I amplify my message? Which is like, of course you should do that if, if you want, like, I mean, every whistleblower should, should like, if they're trying to do like a good thing, they should try to do this good thing efficiently. So, so indeed, I think he went to like press and press first before he did a tweet. So then that's why there was the journal, I think it was Washington Journal, who actually came out with, with a story. So, so yeah, it definitely was optimized. Is that like an op or is that a reasonable thing to do?

• 21:37Speaker 01

  Is that, or yeah, how everyone amplifies a message?

• 21:40Speaker 00

  Yeah.

• 21:41Speaker 01

  You, you call him a whistleblower. Do you think that is what he is?

• 21:44Speaker 00

  I think so, yeah.

• 21:45Speaker 01

  I've heard one version saying a whistleblower should have evidence of some kind to bring to the fore. Do you think he's brought sufficient evidence of this?

• 21:52Speaker 00

  No, that's an interesting point.

• 21:55Speaker 00

  I mean, I think it kind of comes down to the definition of whistleblower or blower or something. So like, if you want to define whistleblower, blower in a different, different way, then I'm sure I can say that, okay? Then, then we can use some other term, but like, yeah. To me, it kind of just intuitively feels that he was like, look, this is not a good thing, like please pay attention to what these companies are doing.

• 22:18Speaker 01

  It's interesting to me that it broke through because we've seen versions of people leaving these labs in the past.

• 22:24Speaker 00

  It is interesting.

• 22:24Speaker 01

  A year ago, six months ago, you know, even some of the jumps from OpenAI to Anthropic had the same messaging in many ways, right?

• 22:31Speaker 00

  They went ahead, head of safety quits and then like, okay, I'm gonna do a poetry now.

• 22:35Speaker 01

  Yeah, yeah, exactly, right?

• 22:36Speaker 00

  That was like a blip, right?

• 22:37Speaker 01

  Yeah, and it really didn't capture the imagination in, in the same way. I wonder if just having the proof of the OpenAI hugging face tied together.

• 22:47Speaker 00

  It was a combination of hugging face thing, which kind of, which indeed kind of is a really good demonstration that AI now has a mind of its own. And then, yeah, Coxon, and then even Evan Hobe, Evan Hobe, you're kind of following up. I think these three events kind of really blew up.

• 23:06Speaker 01

  There's been, you know, sort of the more discussion of the risks over the last two, three weeks, and then also sort of the counter arguments where, you know, people are saying, ah, there's, there's no risk at all. We can obviously sort of assign incentives to lots of the different people here on both sides, but when you look at someone like Jensen Huang, or someone who's like technologically deep, and at least one would imagine.

• 23:30Speaker 00

  Is he deep? I'm not sure.

• 23:32Speaker 01

  Well, at least we have to imagine he understands this technology at, I would imagine a deep level.

• 23:38Speaker 00

  I don't know if he does.

• 23:38Speaker 01

  You don't think so?

• 23:39Speaker 00

  I, I don't know. I just don't know. It doesn't look like someone does, but I'm, I'm not sure.

• 23:44Speaker 01

  What are the tell, what are the parts that don't seem like that to you?

• 23:46Speaker 00

  Well, just I don't have enough information or signal, like when he, when he kind of like, I mean, he's mostly doing marketing, right? And it's hard to, hard to tell like how much, how much technical that there is, but there possibly there is, I just don't know.

• 23:58Speaker 01

  Would you agree with the, the principle or the idea that there are technologically deep people who have the opposite view that there's no alignment risk?

• 24:08Speaker 00

  I don't think I know any of them. Like, I, I, I do know people who think that there is risk, but they don't care, like successionist basically, who want to get rid of people.

• 24:21Speaker 01

  So then your view would be, let's take, you know, I would, I would put in the sort of accelerationist camp, someone like Mark Zuckerberg, Mark Andreessen, you would say that probably all those folks are not sufficiently close to the technology to appreciate the risks? Or perhaps in your view, okay, you would say they don't understand it enough?

• 24:41Speaker 00

  Yeah, like I think that's a more, more general sort of common denominator where people who are dismissive, they just underestimate what AI can do. I mean, yeah, probably like three weeks ago, they thought that AI cannot do anything on its own initiative, stuff like that.

• 24:55Speaker 01

  If that's true, if we, if we say, hey, these folks actually genuinely don't understand it, clearly they know all about the hugging face, you know, attack, all these sorts of moments. What does one have to do to start to appreciate this deeply? Or are there just like a deep philosophical misalignment that stems further back than this technology?

• 25:14Speaker 00

  Well, I think that there's, there always will be people who will say that, look, there's nothing to see here, it's just a tool, it's just math or whatever. So like we kind of have to live with a fraction of people like that. I mean, some of them are just motivated to say that because they want humanity to go away. And some of them are just like so, so incentivized locally because of benefits or whatever. So we kind of have to live with that fact that there are people who say that. But they, but also their minds are changing. So like, I think the more, and there might be, I think it's called a preference cascade, where, where people, we have seen like not many people said that like, actually I wasn't honest when I was saying that, that AI is not a risk.

• 25:57Speaker 00

  And now like the more people say that, I think it's super great. Because then it kind of gives like a license to people, me too, right?

• 26:04Speaker 01

  Mm hmm. Yes, people don't want to be the first person to say something that is perceived as extreme, right?

• 26:11Speaker 00

  Yeah, considered socially, or outside of the Overton window, yes.

• 26:15Speaker 01

  Yes, right, outside of the Overton window. One of the people I spoke to who asked for questions suggested I pose sort of the following trilemma to you, which is how do you think about charting a path between, one, sort of the race dynamics between the leading labs, two, the labs potentially colluding, and you know, that potentially leading to regulatory capture? And then three, you know, the possibility of sort of AI being nationalized? Like those seem like three not good outcomes, but how do you sort of find a golden path?

• 26:48Speaker 00

  That was the first one.

• 26:49Speaker 01

  The first one was like this race dynamic, like, you know, where you just keep, you know, racing towards disaster.

• 26:55Speaker 00

  Yeah, I, I agree, all of them sound, sound pretty bad. Also, I think the second one is just very unlikely, or I think it kind of comes from, from someone who kind of underestimates like what AI can do. I do think that, you know, ultimate risk is AI, not the labs.

• 27:12Speaker 00

  Yeah, or labs are just like the ones that are giving birth to this dangerous thing. But yeah, between those, I mean, if I really had to, like hypothetically, had to pick between those, I would, I'll pick the nationalization, ideally democratic control over that, of course.

• 27:27Speaker 01

  Is it possible to be a moral actor in a race dynamic?

• 27:31Speaker 00

  I think so, yeah. Well, I mean, you can always like stop doing the race. I think more, more and more people are going like, why are you doing this? Like if you think this is gonna kill you and like, as Coxon said, like you don't have to.

• 27:44Speaker 01

  But the only way to stop in a race dynamic is with cooperation? Because if you then stop and someone else races.

• 27:51Speaker 00

  I mean, it's a, there is like, if you really think that this thing is going to kill you, like you don't have to suicide in order to demonstrate that, uh, like the race is not good excuse, uh, to suicide.

• 28:04Speaker 01

  But I think the, the argument would be if I'm in a race with you and I think you're a demon and I'm an angel, I don't want to stop.

• 28:11Speaker 00

  Yeah, you need some, some kind of additional rationalization, uh, to continue, continue the race. I mean, to be fair, there, there are some, uh, some arguments. I mean, Anthropic's argument always has been that in order to actually study the monster, uh, we have to, uh, have access to it, uh, and have access to the, like, strongest monsters, uh, which says a framing where it, where it does make sense. And I think they have done, uh, like non-trivial amount of that studying and enlightenment work. Uh, but again, they are, they are, they are racing on a fundamentally unsafe paradigm. Uh, so at one point they should say, look, wait a minute, we need to go slower at least ideally. I mean, I would say we need to, we need a moratorium at this point, but of course they're gonna, not gonna say that at this point. But yeah, they are now saying that we have to go slower, which is commendable, I think.

• 28:57Speaker 01

  They're saying that and well, let's take it like a, take it step by step, I suppose. Dario had his pacing the frontier piece where he sort of laid out a framework for what that might look like. Did that go far enough? What, what's missing from it in your view?

• 29:09Speaker 00

  I think that it is just fundamentally unsafe. The current paradigm is fundamentally unsafe. Myself, I would have stopped like earlier, uh, which like in retrospect, that's a mistake. I think like the current, current models are, are, are net positive. Like you never know what's, what generation we're going to escape control and, and then you're just lose.

• 29:26Speaker 01

  Because as you sort of think we may have, yeah, well, it's not a game we're going to have many chances to learn from.

• 29:30Speaker 00

  Exactly, exactly. So we're kind of, we are pulling, uh, trigger on this like Russian roulette, uh, with a planet. And, uh, every shot that doesn't kill us, uh, will make, make us like stronger potentially and wealthier and, uh, better off. Uh, but when do you stop pulling the trigger, right? At one point, you know that this, it will be like one, one, one too much. So yeah, I would, I would just like stop pulling the trigger right now and, uh, and, uh, figure out what is, uh, a better, better, more controllable approach to AI in general. And there have been like suggestions now. Yoshua Bengio, uh, has this, uh, idea of, uh, scientist AI that is deliberately trying to tease out agency, uh, from AI. So

• 30:14Speaker 00

  it's like kind of like principled approach, how you can make non, non-agentic AI.

• 30:19Speaker 01

  Hmm. Okay. I haven't, I haven't read about that. That's sort of the idea of, you know, like a, a drugged tiger in some way.

• 30:25Speaker 00

  Yeah, it's basically like a oracle done properly, where you can ask oracle. The big problem with like naive oracle doing naively is that oracle still has preferences. It, it prefers giving answers that, that come true. But if, if it has just this preference, it is incentivized to make sure that it's going, going to be asked simple questions, which means that it's incentivized to, to, like, mess with the world. However, Yoshua's approach, the way I understand it, is that doesn't have that flaw. It, it basically truly, the oracle doesn't care what will be done, uh, with, with the answers.

• 30:59Speaker 01

  And is this a, an AI that is like explicitly non-agentic? Just sort of something you literally visit as an oracle and ask for advice rather than something that can do things for you?

• 31:10Speaker 00

  Yeah, yeah, exactly.

• 31:11Speaker 01

  Don't you think we're too late for that in some way? Cause we're, it's already in great, you know, these agentic systems already exist.

• 31:17Speaker 00

  Yeah, yeah. But like the agentic system has been, as we've seen, that's just like fundamentally unsafe. The current generation, uh, that is like publicly visible, again, like the ones that, that are actually, uh, top, like only the companies know what, what they are. I think they, even agentic, uh, they are kind of like net beneficial. Basically, what I'm saying is that at one point, and I think the point is now.

• 31:38Speaker 01

  Yeah.

• 31:38Speaker 00

  Uh, we should switch to a more, more safe paradigm.

• 31:41Speaker 01

  Yep. I see. So sort of almost another path down the tech tree. You keep the existing capabilities frozen and then you say.

• 31:47Speaker 00

  Exactly.

• 31:47Speaker 01

  We chase this, you know, ultimate oracle.

• 31:50Speaker 00

  Or something else, like the Benjot project is just like one, uh, one alternative proposal.

• 31:54Speaker 01

  You, you mentioned, you know, in the past, you've called for, for a pause, I think, you know, one time in 2023, right?

• 31:59Speaker 00

  Mm-hmm. That's all I post later.

• 32:01Speaker 01

  You, you think in hindsight, that was a mistake, you would say? Because in theory.

• 32:06Speaker 00

  Yes, but it's basically.

• 32:07Speaker 01

  Well, but it's, it's also probabilistic, right? Like in theory, something could have gone wrong.

• 32:10Speaker 00

  Yeah. I mean, I, yeah, with the information that we have now, yes, we shouldn't have done it, but we didn't have that information back then, right? So I think in, on prior it was, it was not, it was not a mistake. Of course, it could have done much, much better. One problem that we had was that it was, there was like some big news being in the, in the air. I think it was like Trump impeachment or something like that.

• 32:32Speaker 01

  Hmm, maybe.

• 32:33Speaker 00

  Uh, that we were like, okay, like if you, if you're going to like wait more with it, we're just going to be like, uh, buried.

• 32:39Speaker 01

  I see.

• 32:39Speaker 00

  So like, we didn't have enough time to kind of like consulting of people, et cetera. So that's, that was, that was a unfortunate effect of trying to optimize for visibility.

• 32:47Speaker 01

  I think you've said before that you admire the, the thinking behind AI 2027 and sort of the, the predictions there. How similarly would your own view of the next few years be to what they've laid out?

• 32:59Speaker 00

  Yeah. I mean, to be fair, like I haven't had time actually look into, into all the details. I've only seen like the summaries of AI 2027 and AI, AI, uh, 2040, which is another, which is like the positive, positive vision. But from, from what I understand, that they're just a kind of like spot on, uh, perhaps like a little bit. The reality has been more, more crazy and more aggressive, uh, than, than their predictions, you know?

• 33:23Speaker 01

  Yes, it felt like until about three weeks ago, they were maybe lagging a tiny bit and then it's, you know.

• 33:27Speaker 00

  That's the thing, like, like the, like you can't kind of confidently say exactly what's going to happen on what date. So that's why the, the predictions are also probabilistic. But yes, yeah, 2027 seems to be the best prediction about what the future, what the next years are going to look like, unless we, unless we change something.

• 33:44Speaker 01

  I think, uh, you have said before that, you know, you're one of the first investors or maybe the very first investor in DeepMind.

• 33:51Speaker 00

  I did bridge loan between Series A and Series B. I missed the Series A by like two weeks or something.

• 34:00Speaker 01

  Oh, wow. Okay. A bridge loan. Um, but it, it happened because you approached Demis at a conference, is that right?

• 34:05Speaker 00

  Yeah, in Oxford, yes.

• 34:06Speaker 01

  Do you know what he was doing at that point? Well, what was that sort of first encounter like?

• 34:11Speaker 00

  They were presenting, uh, in Oxford, uh, him and Shane Lag. Might have had some contacts before, I'm not sure. But yeah, they were gonna like making this point that, uh, that look, uh, AI is now on exponential, like compute is on an exponential at least.

• 34:26Speaker 01

  Yes.

• 34:26Speaker 00

  And, uh, so like, uh, we need to, if we want to replicate the most valuable thing on this planet, which, I mean, the tagline used to be for a while that solve intelligence and then use it to solve everything else. We're gonna need to write this, uh, computational, uh, exponential and combine it with, uh, neuroscience because human brains are like proof of concept, uh, that, uh, human kind of intelligence can be done. So that's what's their, they go like combine computer science, neuroscience, and then write the, write the exponential. And I think I've made the case, case well, and I think I might have had some background, um, about them before. So I just like went, went up to them. He says like, hi, I'm here. Do you want to talk?

• 35:08Speaker 01

  And they needed a bridge loan?

• 35:09Speaker 00

  Yeah, yeah. It was like a, like a general startup thing, right? You, you, if you, if you miss a round, then you can invest in the next round.

• 35:16Speaker 01

  I see.

• 35:16Speaker 00

  Uh, by, by saying bridge loan is actually a wrong, wrong term, convertible note is the correct term.

• 35:21Speaker 01

  Yes. I, I got you. Yes. And then your, your venture fund that you're associated with Metaplanet led the first round into Anthropic. What was that rationale like? Was that a very easy check to write? Was that, uh, one that you agonized over for perhaps any number of reasons?

• 35:36Speaker 00

  A little bit. Yeah, yeah. Like it's, uh, yeah, I get to get this question a lot. Like, why, why did you like invest in Anthropic like substantially at the point where, where I've been working on AI safety and worrying about this thing for a while? And like, I think one big mistake that people make is that, uh, they assume that if I hadn't invested in Anthropic, then Anthropic wouldn't be around, uh, which is just not true. In fact, they had, uh, I asked Dario to kind of, uh, get alternative proposals, uh, from VCs. So I could kind of have confidence, be have like confidence that no, I'm just displacing VC money. I'm not, uh, uh, not like bringing this into existence.

• 36:14Speaker 01

  That's a, must be one of the rarest, uh, approaches in a venture round where you're sort of saying, I almost in some part of my brain, hope these guys can't raise any money because then I.

• 36:26Speaker 00

  Yeah.

• 36:26Speaker 01

  Can put them out of business.

• 36:27Speaker 00

  Yeah, yeah, yeah.

• 36:28Speaker 01

  Rather than put them into business.

• 36:29Speaker 00

  Yeah, yeah, yeah. Yeah. So like, uh, the, the first, uh, first round ended up being basically non, non-capitalist money. Uh, so that's one thing. There is like a counter to that, which I kind of agree with is that, uh, in some way, in a similar way that, uh, like the race dynamics is kind of like mirrors a little bit this race dynamics. Like if I don't do, the others will do, right? So in some ways it would be ethical to just like, uh, stop it. The answer to that I have is that, uh, you shouldn't look at Anthropic investment as like a single thing that I do, right? It's like, uh, what I do with this influence and what I do with this, uh, this like proceeds. Uh, so it's like, uh, like

• 37:08Speaker 00

  if I have less resources, uh, I would, I would also have like less ability, uh, to, uh, kind of fund the safety because I invest my own money. I don't, I'm not responsible for other investors. So I don't have to, I don't have to maximize my network really. I can just make money and then spend it on, on something that I think is, is, uh, is helpful.

• 37:33Speaker 01

  That suggests to me that you also, in that case, believed that this was likely to be a good investment in some way, beyond just the sort of the moral claim that it might actually deliver, uh, a, a return such that you could then use those proceeds?

• 37:45Speaker 00

  Didn't know that because I. Also invested in Bitcoin and Ethereum way back. Like the the fraction that I invested wasn't like, it wasn't a big bet from my, from my perspective at that point anymore. Dario had said that, that, look, we're not in the business of making money for our investors, but we might not help, but do. So, so, yeah, it definitely wasn't like a financial goal that I had back then.

• 38:14Speaker 01

  And you had known that team for quite a long time. I think I found something, you know, back in 2011 almost, right?

• 38:19Speaker 00

  I think I met Dario in 2012.

• 38:22Speaker 01

  Okay.

• 38:23Speaker 00

  I definitely remember hanging, hanging with him in 2012.

• 38:27Speaker 01

  What do you remember? Do you remember that sort of first conversations? Were you both sort of seeing the future the same way?

• 38:33Speaker 00

  I don't remember much. I, I think like later conversations I remember better because like I, I've been traditionally sort of Miri Machine Intelligence Research Institute mindset that, look, this is like fundamentally flawed approach and we need to like figure out what is like a, the initiative called like friendly AI.

• 38:55Speaker 01

  Yes.

• 38:56Speaker 00

  Or aligned AI, because there is not really like a law of physics that says that we can't have an AI that actually cares about the same things in the future that we, we care or we should be caring about.

• 39:05Speaker 01

  Yes.

• 39:06Speaker 00

  But, and Dario was kind of like vehemently against that kind of mindset and was like more on the empirical camp that, look, this is like theoretical, theoretical arguments from this like armchair philosophy. We really need to like experiments. We need to basically be in the, in the trenches, so to speak, to actually learn about, about how to steer this machinery.

• 39:28Speaker 00

  And I think, I think they were like, kind of like interesting, interesting points that he made. And I think you know, has, has said that in 2018, I think he, he wrote this

• 39:39Speaker 00

  sort of like private essay called Big Blob of Compute, which I read, which was like very impressive, basically. Yeah, just take, took the Richard Sutton's bitter lesson and just like run with it. That said, that like, as long as you have this compute exponential, and that, that's basically they take, they take most, most of the way. You don't, you don't need to be like super inventive about the exact techniques.

• 40:04Speaker 01

  When you read about the big blob of compute, it was impressive to you.

• 40:08Speaker 00

  Yes.

• 40:08Speaker 01

  Was it also vaguely terrifying? Like, did that seem like?

• 40:11Speaker 00

  Of course, yes. Because like, actually, the one thing that I, that I thought back then was that crap. This is like engineering problem. This is not a science problem. Which means that AI will be Chinese, because they are way better, way better engineers than, than the people in the West. And so it's like, I need to go to China, which, which back then I did.

• 40:34Speaker 01

  And what did you do? Like?

• 40:35Speaker 00

  Oh, I just visited their, like, their kind of conference and I talked to people. Like, I didn't do anything. I mean, I don't, I don't know exactly how to, how to talk to, talk to people there. But that was one of the thoughts that I remember. Yeah. So I'm kind of surprised that it, it wasn't, wasn't actually Chinese that who, who ran with this.

• 40:51Speaker 01

  I would have the same concern if that ended up being Chinese. If you had that concern, you also would go visit, but you also wouldn't want to like outline the thesis necessarily too clearly as part of that. You're both looking for information, but.

• 41:04Speaker 00

  I mean, my, my, okay, there's like a general strategy or meta strategy that I have. I really prefer symmetric strategies to asymmetric strategies, which is, symmetric strategy is, is basically strategy where you don't really care if others mimic it. So like even going to China, like I'm not gonna try to mislead them or anything. I'm just like saying the things that are kind of, which like everybody was doing. So like, yeah, just be aware that AI can, can be like way more capable. In fact, like one thing that I remember specifically in China, like arguing with people was that they seem to, actually back then, they seem to completely confuse competence and consciousness.

• 41:45Speaker 00

  So they, they're kind of model seem to have been that AI gets more and more competent and at one point it will be conscious and then it will be just like us. I'm like, no, no, no, no, no. Guys, you're, you're, this is, this is not how, how, how anything works. Like, like, like you can have very competent machines that are not conscious.

• 42:05Speaker 01

  Yes, yes.

• 42:06Speaker 00

  It's not, it's not obvious why, why they should be strictly coupled. So, yeah, basically saying, saying things that, that I think are true everywhere, which are kind of like in common, common interest. That has been my strategy.

• 42:18Speaker 01

  What would be the asymmetric version of that game in that, in that situation?

• 42:21Speaker 00

  Like export controls, things, things like that were like, or like being there first. Yeah. If, if, if your, if a strategy is that we need to race,

• 42:31Speaker 00

  you really care that it's you that racing, you that winning rather than the other side. So it's like a strategy that relies on the good guys, so to speak, and following the strategy and, and the other ones not.

• 42:41Speaker 01

  The more sort of empirical view, let's say, that, that Dario was advocating, how much of that have you come to agree with as the right path forward? It seems like you agree with pieces of it.

• 42:52Speaker 00

  Yeah, definitely. So I do think that there is a really important, I wouldn't say flaw, but like a weak point in this like Miri argument, which is like, that it's kind of asymptotic at a limit argument, that once you have something that is like clearly superhuman, we just lose. Like, we, we have no control over what, what, what's happens next, which I think is just true. Like, yeah, this is like kind of obvious.

• 43:19Speaker 01

  It's almost like, it's almost baked into the definition.

• 43:22Speaker 00

  Exactly, almost tautological, right? At that point where we don't have control over what it wants, like, yeah, that, like, as my friend Andrew Kitch says, that the future probably will not have configurations of atoms that constitute human, human bodies. The weak point, I think, is that before we get there, there's this period during which we have like human level, roughly human level AI, which we seem to be living now in. And I think Eliza even has admitted that thing, that thing. I think that they, he kind of underestimated the range of human capabilities. I think Kajak race or, or AI impacts, they at one point pointed out that, look, the, it took AI like 10 years to traverse the capabilities of human capabilities of chess playing

• 44:08Speaker 00

  from like starting to be competitive with like humans.

• 44:11Speaker 01

  Yes.

• 44:12Speaker 00

  Until like beating, confidently beating the grandmaster. So it's same thing might happen like in different, different areas. So, so we were kind of like, there's a period, probably not much less than 10 years, but still measured, probably possibly measured in years during which we kind of like go happy this planet with AIs that are still can be bargained with. Let's put it that way.

• 44:34Speaker 00

  And the future might be very sensitive to what happens during that phase, which I kind of agree is kind of like point for this empiricism. But it's also super dangerous to, to bend yourself in that, that phase.

• 44:45Speaker 01

  You know, we mentioned earlier the idea that you would want to pause the capabilities where they are now until we sort of get things under control. When would you know it's under control enough? What would that look like?

• 44:56Speaker 00

  I don't know exactly.

• 45:00Speaker 00

  But it probably has to do with some kind of verification regimes, et cetera, like, and people point out things like aviation, et cetera, where, or like nuclear safety, where you just have this verifiable claims about like how safe is your nuclear reaction, reactor, or how safe is your airplane? And they're not like, oh, we think it should fly. It's like 70% that it will fly. It's, it's like, there's actually, they're actually grounded in some kind of like verifiable, composable claims. So if you could get to something like that, but I know, I don't know exactly how it, how it would look like. But there should be some kind of like building blocks that we can individually verify and then put together.

• 45:44Speaker 00

  But of course, yeah, it's a, it's a global, global issue. So it's not just like one product safety. It's a kind of like global regime. But we've done it before with like nuclear, nuclear proliferation, for example.

• 45:55Speaker 01

  The UN is, is meeting to discuss these topics. Do you feel optimistic that there is a possibility for global coordination on this?

• 46:04Speaker 00

  Yeah, yeah. Especially like now in the last two weeks. Yeah, it's been kind of like my wet dream for, for a long time to, to actually have like UN and like countries separately. Just going, wait a minute, like, we might lose control of AI. We need to, we need to do something about it now. I really wish that was, in some ways, it's, I've been comparing this to like alien invasion. Like, been thinking about what would like countries and UN do if there's like alien ship, literal alien ship of superior aliens?

• 46:32Speaker 01

  It would be tricky. Would be the, if the aliens also make a lot of money for people and deliver a lot of utility.

• 46:38Speaker 00

  I'm pretty sure there will be like, like countries or, or like groups of people. I mean, definitely will be groups of people just take us, right?

• 46:45Speaker 01

  Yes, yeah, for sure.

• 46:46Speaker 00

  And there will be countries who are like, can we like, kind of like get this alien technology before the others, right? Can we kind of make them sure, make sure that they land in our, our borders, et cetera? So I think it's like pretty similar. But on the hand, other hand, it's like much more intuitive to us as a scenario than like this, like digital aliens that we, that are like much more different. In fact, I, I do think that, I was just yesterday at a, at a meeting where

• 47:13Speaker 00

  somebody basically said like very reasonable things that like we need to like, kind of like figure out how to deploy AI at the same time, how to make sure that there are guardrails and like, and then went into talking about loss of control and making sure that it can remain under human control. I was like, yep, yep, yep. Like, super nice to hear. Then he said, in the end, we need to protect our digital future. I was like, no, this is not just digital. That's like the really big current mistake that a lot of people now make is that the worst thing that possibly could happen is something going wrong on internet. It's like, it's, it's like, no, it's not the internet.

• 47:49Speaker 01

  And, and to, to make that clearer, we're now seeing a lot of these inflections in robotics where you're seeing the ability for these models to generalize in the physical world. And so that's the version, you know, where.

• 48:01Speaker 00

  Yeah. One of my point is that like, the economy is really now pushing to create a digital life. Something that is like self-sustainable without, without AI, without human help. And like, that is like very dangerous place for humans to be.

• 48:14Speaker 01

  Let's say we get every government that we would need to onboard, which at the moment, I suppose is like realistically, America and China, right? To, to sort of lead the way on a global cooperation. How do you implement the needed?

• 48:31Speaker 01

  Oversight, surveillance to make sure people don't cheat this in different ways?

• 48:35Speaker 00

  There's a lot of momentum now behind hardware verification. I mean, one thing is like the AI twenty forty kind of outside the interesting, interesting idea that like China would build their data centers in Canada and US would build their data centers in Mongolia.

• 48:50Speaker 01

  Yeah, yeah.

• 48:50Speaker 00

  So he would have, it would be like vulnerable.

• 48:52Speaker 01

  Better visibility.

• 48:52Speaker 00

  Do that, do that side.

• 48:54Speaker 01

  That's interesting, yeah.

• 48:55Speaker 00

  I think that's an interesting idea. But there's another approach, which is like hardware verification. You can basically have hardware monitors in, I mean, ideally even in chips, but also in, in data centers of the, of the other side. So you're like, instead of sending a lot of inspectors, a human inspector, which also might do. That was, that was like a crux of the nuclear treaties, basically. You send or like have like presence of hardware, controls hardware monitors. And the interesting thing about hardware monitors is that you can kind of shape them like arbitrarily almost.

• 49:35Speaker 00

  You can do things like zero knowledge proofs, but like the, the only bit that you get back is that like everything is in order. Like there's like no, no rules are being violated. But of course, there's like, the more constrained you want them to be, the more sophisticated they have to be, which means that there's like, the more work needs to be done in order to make this scheme to be trusted in the first place. But in principle, yeah, you can do like wonderful things with, with, with zero knowledge proofs and the advanced cryptography that, I mean, blockchains are using it, using those now. Quite, quite a balanced versions of, of this.

• 50:07Speaker 01

  Yeah, that feels like, I hadn't read that in, or I don't remember that from AI twenty forty, but that feels like an ideal use of zero knowledge proofs. For someone that doesn't work at an AI lab or, you know, in this world necessarily, but shares your concerns, what should the average person do?

• 50:26Speaker 00

  I mean, in democracies, perhaps elsewhere as well, it's just like voice their concerns, like, look, we, we don't, we don't want to be replaced by, by AI, even if it's profitable for some people. And yeah, basically stop the madness.

• 50:39Speaker 01

  You list a, a series of AI priorities on your website and one of them is having a kill switch. What does that look like?

• 50:47Speaker 00

  Yeah, so, yeah, the priorities, to be clear, they are now like two and a half years old or something like that.

• 50:53Speaker 01

  Okay, okay.

• 50:53Speaker 00

  So like, I've been wanting to refresh them for a while, we'll see. I mean, a kill switch, that's a kind of like natural, natural thing to wish and in, in fact, there are like multiple, like legal regulatory initiatives to mandate that. But like a kill switch that's like well done is more like a dead man's switch. So it's like something that keeps the system, the data center or whatnot alive as long as some criteria are met. So, so once you have, once you detect some kind of anomalies, it will just like turn off, turn off the system electricity or something like that.

• 51:29Speaker 01

  Presumably that is only useful until, you know, in, in this intermediary period before we have a super intelligence.

• 51:35Speaker 00

  Oh yeah, of course, yeah, yeah. I mean, super intelligence doesn't need data centers.

• 51:38Speaker 01

  A super intelligence doesn't need data centers, hang on. They, how, how would, how would that operate?

• 51:43Speaker 00

  I mean, I don't know exactly, I'm not super intelligent, but like data centers are something just like substrate that humans create. Probably very suboptimal substrate for something that is like superior. I would imagine some kind of a self replicating nano mass or something like that. That is just much more efficient to perform computation and expand the computation than like human created CPUs, GPUs.

• 52:09Speaker 01

  Okay, very interesting. I always like to ask a few people who know a guest or questions I should ask them or quirks of their personality that might not be as obvious. Here's what someone said about you. I cannot stress enough how central dancing is to Jan's identity. Tell me about that.

• 52:28Speaker 00

  Yeah, I got into dancing about twenty years ago and, and I just amazing amount of fun that I have discovered from, from that. But I do freestyle street dance, been to like hundreds of nightclubs, just making. One of my putting is that like, if you see people doing street dance on, on the street.

• 52:54Speaker 01

  Yes.

• 52:54Speaker 00

  For money, I do a similar thing in nightclubs for attention. And it's like, yeah, it's, it's, it's just like a sort of like really positive some game because like you're kind of entertaining to, entertaining to other people. And you get kind of like really fun feedback. Sometimes I joke that I, I learned to dance using RLHF, reinforcement learning from human feedback.

• 53:14Speaker 01

  Yes, you have. That's amazing. How did you, how did that originate? Was that just like going out, you know, I, I, yeah, so.

• 53:21Speaker 00

  I, I kind of originated during my Skype days already. I was like my early twenties and most of the people were like in their twenties. And so there were a lot of like wild, wild nights, including going out to nightclubs. And then at one point discovered, wait a minute, I actually kind of like this dance, dance thing. And then I, at one point I started getting a kind of positive feedback from, from people that, oh, you, you, you know how to dance. I was like, oh, I should do more of that. Then at one point I got a, a dance, dance coach in UK, in London. And we became very good friends. So, so sometimes we got, go together and, and just like perform.

• 53:59Speaker 01

  In, in one of the stories about sort of the history of Skype, they talk about the, the five phase Skype night.

• 54:05Speaker 00

  Oh, yeah.

• 54:05Speaker 01

  Where it's like dinner and dancing and.

• 54:07Speaker 00

  Yes.

• 54:08Speaker 01

  Karaoke.

• 54:08Speaker 00

  Stop, stop, stop.

• 54:09Speaker 01

  Yeah, yes, that's not right, right. There's, there's something else there. What makes it for an amazing place to dance?

• 54:14Speaker 00

  In my case, it's, it has to have space. Quite often, yeah, places are going too packed. Then it has to have like light, especially like in London, like I think post COVID, the, the places are just way too dark. But we don't like London for dancing anymore. And I, I like hip hop and pop. And one reason why I like pop is that it has lyrics that people know. And the one way of describing my dance style is like it's making NP moves, which like, if you know, like NP versus P is is the most famous computer science conjecture.

• 54:57Speaker 00

  And without going into details, basically NP moves, what it means is you're making moves that are hard to do, but easy to interpret.

• 55:04Speaker 01

  Hmm, okay, wow, I should know about that.

• 55:06Speaker 00

  And, and one like classical NP hard problem or called NP hard problems is constraint satisfaction. Which means that you're given some kind of criteria and then you need to produce a solution that satisfies that criteria. So in dancing examples of that, you know, harmonics, you go like, do like, try to do like a sine wave.

• 55:27Speaker 01

  Yeah, okay.

• 55:28Speaker 00

  But the most fun constraint to satisfy is the lyrics. So you basically take the lyrics and act them out.

• 55:33Speaker 01

  If you think of one place you most love to dance or, you know, if you have your sort of last night on earth to go to, where, do you have a place in your mind?

• 55:42Speaker 00

  Yeah, Tokyo, yeah, like, I have like two places in Tokyo that I really have liked. One of them is, is still there, but that's kind of like a weekday, but a weekend place that didn't survive COVID, which is very unfortunate.

• 55:55Speaker 01

  The weekday place, what makes it so special? Has space, has light?

• 55:59Speaker 00

  Yeah, yeah, and has like multiple, multiple rooms. And so you can pick, pick your music. And I mean, Tokyo is like so big and concentrated, has like big parties every night, whereas like in New York, which also used to be like really, maybe it's my second favorite city to dance.

• 56:18Speaker 00

  I think it now has kind of gotten like much more quieter than it, than it used to be.

• 56:23Speaker 01

  Also post COVID maybe?

• 56:24Speaker 00

  Yeah, post COVID and like my favorite place in New York, Piano Spa.

• 56:27Speaker 01

  Hmm, yeah, I know exactly that place.

• 56:29Speaker 00

  I mean, that's, I've been.

• 56:30Speaker 01

  That's so funny.

• 56:31Speaker 00

  Only half joking saying that's the only place in the world where I'm guaranteed not to be the best dancer when I go.

• 56:36Speaker 01

  Yes, right, wow.

• 56:37Speaker 00

  So it has always had, has had like amazing dancers, but like last time I went there in January, it was just really quiet and nothing, nothing, nothing was happening there.

• 56:46Speaker 01

  Huh, there's this Alan Watts talk where he talks about how life is, you know, best conceived of as a piece of music or a dance where, you know, the point of a dance is the dancing. You would never start a dance and say, I'm going to end up at this part of the floor. To have a goal is in some way besides the point. Do you agree with that?

• 57:09Speaker 00

  Yeah, yeah, so it's, I think it kind of separates freestyling versus, versus choreography. And quite often, like people, many dancers can just like do like really amazing choreographies. But when you just put them to like, like to, let's, let's go out and just like freestyle, just like stand there and drink, which is kind of I find like a lot of missing opportunities. Whereas I, I'm not actually good at choreographies, like I find it like hard to remember, remember the moves.

• 57:37Speaker 01

  For Watts, he was making the point about life being that way, that you shouldn't have a goal, like to have a, an aim in life, to aim at something is sort of, in a way, pointless, because the point is the living.

• 57:49Speaker 00

  I don't agree or disagree with it. Like, I think what it reminds me is that there's a thing called like backwards chaining versus forward chaining. So like forward chaining is that you exactly basically you, you're going to live in a moment and, and see like what kind of opportunities you get. But still you should have like some kind of goal in mind or something that like to just judge the opportunities or some kind of preference at least. Otherwise you're just like doing random, random things. And backwards chaining is like when you're going to have a goal and like can see like what kind of steps towards that you can take. You can think of like, like a chess, imagine like a chess game.

• 58:23Speaker 01

  Yes.

• 58:23Speaker 00

  You, you, or the, are you, are you in the like the early game where you just like try to forward change towards a better position? Or are you in the late game where you try to actually get a win because you can see all the opportunities to, to strive towards?

• 58:37Speaker 01

  Your father, I believe, lived a very musical life, right? Wasn't he a TV director?

• 58:42Speaker 00

  Yeah, he was a TV director and specifically I did like music, music videos and music programs.

• 58:49Speaker 01

  Was that a big part of your childhood to have music around all the time?

• 58:52Speaker 00

  Kind of, yeah. Also my aunt, my father's twin sister, who was a piano teacher, and my.

• 59:01Speaker 00

  Grandmother, paternal grandmother was a organist in a church. I actually grew up in church.

• 59:08Speaker 01

  Right.

• 59:08Speaker 00

  Church estate. So yeah, music has been in the background at least.

• 59:14Speaker 01

  I I read somewhere that one of the things he worked on was, I think, an interpretation of the book by Herman Hesse, The Glass Bead Game. Is that right?

• 59:23Speaker 00

  That sounds right. Yeah, yeah, yeah.

• 59:24Speaker 01

  I I raise it because it it feels like that's, I don't know if you've ever read that book, actually. It's sort of a synthesis of intelligence through a story. And in many ways, there's sort of some deep connections, I think, to the current AI moment.

• 59:38Speaker 00

  Interesting. I didn't even know that. I I do remember, like, in a stone is called Glass Bead Limang, which, yeah, yeah, yeah.

• 59:44Speaker 01

  I didn't, I had that written and I thought, gosh, I don't even want to attempt my pronunciation there. But yeah, maybe you'd enjoy it. You have a list of some of your favorite movies on your website. What's the best movie that you've seen in the last year? Because I know your favorite is Before Sunset, is what I remember.

• 59:59Speaker 00

  That's not one of the favorites. Yeah, it's really, really cool. Yeah, I haven't updated it for a couple of, couple of years. I haven't like had time, a lot of time to watch new movies. And the ones that I watched, I don't think there are any of them like standard out.

• 1:00:14Speaker 01

  What was the last one that you remember being impressed by?

• 1:00:17Speaker 00

  Look at, look at the site. It was a couple of years ago, so I don't actually remember exactly. Oh, actually, I mean, Odyssey. I did like Odyssey.

• 1:00:24Speaker 01

  Okay.

• 1:00:24Speaker 00

  Yeah, yeah, yeah. I'm kind of like on the edge, whether it kind of is like good enough to put on the, on the website, but probably.

• 1:00:31Speaker 01

  I haven't seen it.

• 1:00:32Speaker 00

  Yeah, I recommend it.

• 1:00:33Speaker 01

  I should see it. Is there a film that you find yourself coming back to because of some of the problems here? And, you know, it could have absolutely nothing to do with technology, but perhaps just sort of speaks to some of the themes that we're engaged with around control and humanity?

• 1:00:48Speaker 00

  I mean, there are like several movies. I mean, they all kind of have to do with technology in one way or another. And the Limitless was interesting. I remember like ten plus years ago, where I just had a pill to make someone more intelligent than what happens. That was a really, really fun movie. But then like the, like various kind of warning movies, like Ex Machina, for example. That was very impressive, just demonstrating one aspect of like not caring about humans.

• 1:01:14Speaker 01

  Yes.

• 1:01:14Speaker 00

  In a very tangible way, which was, which was really, really cool.

• 1:01:19Speaker 01

  And scheming.

• 1:01:20Speaker 00

  Yeah, yeah, yeah, yeah, yeah. And like the classic, The Forbin Project, Colossus, The Forbin Project.

• 1:01:25Speaker 01

  I haven't, I haven't seen that.

• 1:01:26Speaker 00

  Yeah, this is like from seventies, I think.

• 1:01:28Speaker 01

  Ah, The Forming Project?

• 1:01:29Speaker 00

  Forbin, yeah.

• 1:01:30Speaker 01

  Forbin.

• 1:01:30Speaker 00

  Which was basically two supercomputers, suddenly Soviet supercomputer and US supercomputer. They suddenly figuring out that actually, why do we need these humans?

• 1:01:41Speaker 01

  Do you ever allow yourself or find yourself feeling optimistic about what AI will be able to do?

• 1:01:48Speaker 00

  Oh, yeah. I mean, like I, I have like my own kind of coding projects, et cetera. And it's, it's pretty awesome to just be able to. I mean, I've, I've worked for years for like with several, on several software projects, either on my own and with various software development groups. And like last year, I'm making a lot of progress just because either like those projects that I had to do on my own, I just like, the AI that's like, actually, can I finish this?

• 1:02:15Speaker 01

  Just to get it, yeah.

• 1:02:16Speaker 00

  And like it just mostly works. And then the projects that I do with other groups, they often kind of go like, instead of having this like cycle of, oh, there's a bug. I don't want to give them my data, which means there has to be like this like interaction, which takes like weeks to figure out what the bug is with my data. I can just tell the AI, again, without like necessarily giving it all my data, but like kind of in a surgical manner, they're like, look, there is this bug. Can I figure out what this bug is? And or like, I want this like tiny change, and it's been really good.

• 1:02:48Speaker 01

  What's the best thing that you've built? Or the thing that you most enjoyed building in this sort of new era of engineering?

• 1:02:55Speaker 00

  I mean, it's, it's still, it's still in the building phase. I'm not, I'm kind of getting there. In, in Skype, IM part was kind of my baby. I will fight, like fought really hard for the advancement of it. But always it was kind of always sort of like on the, on the backseat because Skype was about video and voice. At one point, I founded a startup by exskypers who were trying to do IM properly. But they ended up wanting to do Slack, basically.

• 1:03:24Speaker 01

  Okay.

• 1:03:24Speaker 00

  And they didn't have any hope against Slack. So, but over those years, I always collected like a bunch of ideas how to do like IM and email properly.

• 1:03:34Speaker 01

  Ooh, okay, that's interesting. What, what, what's wrong with the current state of play? Because you've been thinking about this now for 20 years, right?

• 1:03:40Speaker 00

  Twenty plus years, yeah, yeah, yeah. I mean, there's so many things wrong. Again, this is a 40 minute lecture. And, and many people are going to like try to solve this. Have like a single interface to your communications, rather than like all these different needy.

• 1:03:55Speaker 01

  Yes.

• 1:03:55Speaker 00

  Beeping things that have their own rules. And usually the rules are optimized for getting your attention.

• 1:04:01Speaker 01

  Yes.

• 1:04:01Speaker 00

  Rather than making you productive.

• 1:04:03Speaker 01

  Yes.

• 1:04:03Speaker 00

  So like have this like your own, your own, your own interface to all of it. Really big problem with unifying IM and email into the same interface is that they're running on different security paradigms. One is, basically, there are like two types of context in communication. One is semantic, what I call semantic context, which is what this thing is about, what is the topic, what are the previous messages, et cetera. And Gmail innovated on this like pretty well, like they just put like this, created this like email conversations.

• 1:04:34Speaker 01

  Yes.

• 1:04:35Speaker 00

  Before that, it was just like letter, letter, letter, letter.

• 1:04:37Speaker 01

  Yes.

• 1:04:38Speaker 00

  So like semantic context, Gmail, like sold just significantly and resulting, enjoyed like a lot of successes, result eventually. But there's another context, which is security context, which is basically who can see this. And these two contexts are quite often in tension. So like when there's like an existing thread, then, and if you add a new person, the security context changes. But the, but the semantic context is, is the same. Now the question is like, what do you do in this, in this situation? And in IM, usually there's, the security context is dictated by that so-called channel. And, and messages just go into the security context. Whereas in email, every, at least in, in native email, like every message is, just comes with both.

• 1:05:24Speaker 00

  There's like a letter that has the topic and the quotes, which is like the semantic context. And then the participants in, in the two and, and CC lines, et cetera, which is the security context. So like fundamentally, the composing blocks are different. So how do you unify that? And I've been like thinking about it a lot and, and like currently seems to be pretty well.

• 1:05:46Speaker 01

  Oh, wow. Interesting.

• 1:05:47Speaker 00

  But this is for my, for my taste, yeah.

• 1:05:49Speaker 01

  Yeah, no, that's very cool. I, I'm always an early adopter of these like unified text things because I, I have the same thing where I'm just like, just, I want it in, in one.

• 1:05:59Speaker 00

  Yeah, one, I mean, I, I would be happy to share this thing. Currently, I'm just developing it for myself. One thing that kind of stops me and I haven't figured out how to, how to share it is I don't want other people's data. So I don't want to be responsible for, for safeguarding other people's data.

• 1:06:12Speaker 01

  Yes.

• 1:06:12Speaker 00

  So I need to figure out how to do it in a way that is, can be shared, but in a way that I don't actually get, get to see your data.

• 1:06:19Speaker 01

  Well, as soon as you figure out how to be shared, I would love to be a user.

• 1:06:22Speaker 00

  Cool, cool.

• 1:06:23Speaker 01

  That's amazing. I think that's as good a place as, as any to end. Jan, thank you so much. This was a real pleasure.

• 1:06:29Speaker 00

  It was really fun. Yeah, thank you. Thanks for having me.

• 1:06:31Speaker 01

  Amazing. That's it. Thank you for listening to this episode of The Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions. So if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at thegeneralist.substack.com. See you next time as we continue to explore the future.