AI's Most Dangerous Truth: We've Already Lost Control

What happens when the people building artificial intelligence quietly believe it might destroy us? On this episode of Digital Disruption, we’re joined by Gregory Warner, Peabody Award–winning journalist, former NPR correspondent, and host of the hit AI podcast The Last Invention. Gregory Warner is a versatile journalist and podcaster. He has been recognized with a Peabody Award and other awards from organizations like Edward R. Murrow, New York Festivals, AP, and PRNDI. Warner's career includes serving as an East Africa correspondent, where he covered the region's economic growth and terrorism threats. He has also worked as a senior reporter for American Public Media's Marketplace, focusing on the economics of American health care. His work has been recognized with a Best News Feature award from the Third Coast International Audio Festival. Gregory sits down with Geoff for an honest conversation about the AI race unfolding today. After years spent interviewing the architects, skeptics, and true believers behind advanced AI systems, Gregory has come away with an unsettling insight: the same people racing to build more powerful models are often the most worried about where this technology is heading. This episode explores whether we’re already living inside the AI risk window, why AI safety may be even harder than nuclear safety, and why Silicon Valley’s “move fast and fix later” mindset may not apply to superintelligence. It also examines the growing philosophical divide between AI doomers and AI accelerationists. This conversation goes far beyond chatbots and job-loss headlines. It asks a deeper question few are willing to confront: are we building something we can’t control and, doing it anyway? In this video: 00:00 Intro 03:00 AI models that already behave like elite hackers 05:00 Why the AI risk window may already be open 06:30 What AI safety actually means (and why it’s so hard) 12:00 Human-in-the-loop: safety feature or illusion? 15:00 AI as an alien intelligence, not a human one 19:00 The Silicon Valley AI arms race explained 21:00 OpenAI, DeepMind, Anthropic, xAI: who’s racing and why 25:00 The “Compressed Century” and radical AI optimism 27:00 Can AI actually solve humanity’s biggest problems? 33:00 Capital, competition, and the pressure to deploy 37:00 Is AI more dangerous than nuclear weapons? 39:00 The problem with comparing AI to past technologies 43:00 What happens to human agency in an AI-driven world? 45:00 How AI reshapes creativity, journalism, and truth 53:00 The quiet assumptions built into AI systems 55:00 Why optimism and fear both miss the full picture 59:00 What responsibility do users have? 01:01:00 The most important question we’re not asking about AI Connect with Gregory:LinkedIn: https://www.linkedin.com/in/radiogrego/Instagram: https://www.instagram.com/radiogrego/ Visit our website: https://www.infotech.com/?utm_source=youtube&utm_medium=social&utm_campaign=podcastFollow us on YouTube: https://www.youtube.com/@InfoTechRG

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Geoff Nielson?

  Hey, everyone. I'm super excited to be sitting down with Gregory Warner, Peabody Award-winning journalist, ex-NPR correspondent, and current host of the hit AI podcast, The Last Invention. Greg is a fellow traveler in the quest to understand the race to build advanced AI. His full-time job is examining the existential questions and key players at the heart of the AI revolution. I want to ask him whether we're creating something that will save us or destroy us, what future he thinks is most likely, and what we need to do to prepare. Let's find out.

• 0:35Geoff Nielson?

  I'm here with Gregory Warner. He is a Peabody Award-winning journalist. He's the producer and host of the Last Invention podcast, all about AI. And maybe just to start things off, Greg, tell me a little bit about why you and some of your co-producers created this podcast. What was kind of the rationale for why you wanted to tell this story?

• 0:55Speaker 01

  Sure. I mean, I think for me, what got me hooked on the topic was realizing that the people making AI had a sense that this might kill us all. It's as simple as that. I mean, the fact that they felt that the risk of this thing that they were building to humanity was real just felt like such an interesting time to live in. And the fact that that debate over the existential risks and rewards of this technology, because I think that the potential upsides are just as radical. It felt like while we were discussing in the news at the time, things like, oh, the danger of deep fakes and the possibility of AI taking jobs,

• 1:39Speaker 01

  these more existential questions, which had haunted AI really from its beginning, as we found out. It felt like, OK, we need to have this debate and try to bring this to people in a way that didn't, for lack of a better word, didn't just freak people out. The idea was, let's introduce people to the kind of debates that are being had and have been had in these circles, and how we might talk about the future with some kind of superintelligence.

• 2:11Geoff Nielson?

  It's, you know, that's me is one of the most interesting things about the topic is that if you, if you ask some of the most knowledgeable people and the most central people in this conversation, what their outlook is, that just the range of answers is so radical from, you know, basically utopia to destroy us as a species to, you know, the, this, you know, other cohort of voices saying it's all, you know, basically a nothing burger. You know, at, at the risk of asking you to editorialize as a journalist, I'm curious, you know, having talked to all these people, having heard all the sides of the story, you know, what, what outlook do you find most compelling? Are you worried about the existential risk or where have you kind of landed as, you know, Greg the human?

• 2:55Speaker 01

  Um, okay, Greg the human, where, where do I land? Well, let me tell you a story. So there was a hacker, a white hat hacker named Dawn Song, I think her name is. And she took all the AI, the leading AI models and she found 20 zero days. If you're familiar with a zero day, it's a, that's a hacker terminology and basically it's a, it's a huge deal. It's the sort of thing, you stop everything and you focus on this, this weakness because it's incredibly essential to fix it. To find even one zero day is a big deal. She found 20 and using ChatGPT and the current model. We're not even talking about some

• 3:38Speaker 01

  future far off thing. And then I think she used Gemini and found it even faster or something like this. And the important thing is that AI hadn't been trained to find a zero day. It literally became a top level hacker with just a few smart prompts. And so I think my take from that is

• 4:03Speaker 01

  We are talking about when will these systems be good enough to pose a threat? But I think that time is already here. And so it's not, will we awaken a god? Or will we awaken, will we summon a demon? It's, it's, it's, it's not a future conversation. It's, it's that we, we are living already with a, with a technology that is so much more capable than we realize, that is becoming increasingly capable. And by design is, um, its capacities, its capabilities are not known until the model is released. That's amazing, really, when you think about it, that, um,

• 4:47Speaker 01

  that it takes being out in the world in order to, or, or being out, being created in order to figure out what it can do. And so I guess, you know, in terms of my level of P doom, which is like my probability of doom, I guess you've, other folks have used that on the show. I don't think of it as, will, are we headed toward utopia? Or are we headed toward, uh, you know, apocalypse? It's, there are weak points in our world. We, we, we have clearly ways in which small conflicts can scale up into big ones. I mean, I've seen that as a, as a foreign correspondent. I've seen that as a war journalist, and there are pathways to harm.

• 5:29Speaker 01

  And so the question is, how bad an impact will that be? Because it's definitely not zero. It feels, my P doom in that sense is one. Like, I am sure that some bad things will happen. I, I, I think actually that's inevitable. But does that mean the extinction of the human race? Does that mean we can't recover and learn more? No, I'm, I'm actually kind of an optimist in that sense. I don't like to even consider that we are headed toward extinction, but I don't know why we're not talking more about AI safety.

• 6:01Geoff Nielson?

  Well, and what's so compelling to me about that is, in that story, in that example, it's not about a crystal ball, right? It's not about saying, oh, how quickly does the technology improve? What can the technology do tomorrow? It's real, I think, rational concern about the disruption it can have based on what's out there today. And so, as a journalist, you've talked to everybody from AI leaders to folks around the world and more kind of political roles or military roles. And so, I guess I'll ask you this way. How ready are we right now for that threat?

• 6:41Geoff Nielson?

  And what do we need to do collectively as a society to minimize that particular lens on doom, to minimize the risk that it's going to provide tremendous harm to us as a society?

• 6:56Speaker 01

  Yeah, I would give two answers. I mean, there's been some interesting papers from the Frontier Model Forum. I don't know if you've interviewed anybody from there, but they're basically an industry trade group. And also, interesting papers from Anthropic, the AI company, that really have done a lot to sort of look at what are specifically the risks and also the remedies. They're not just red teaming these models in terms of trying to get them to do things that they weren't programmed to do or they're not supposed to do, like give you the ingredients for a biological weapon. And red teaming tries to get them to do those things or to figure out if they will blackmail you. But they're also saying, okay, well, wait, let's say it did those things.

• 7:41Speaker 01

  How do we correct it? And that's where the most work needs to be done. Because even if, let's say, according to this paper by the Frontier Model Forum, if you get the model to give you, it's not supposed to give you the recipe for anthrax, say, but let's say it does, tells you exactly how to make it and how to get away with it. Well, okay, so how do we then program the model to unlearn that? It's you can do a step, it's called targeted unlearning, but what they then found was that it actually didn't really unlearn it, it just said it did.

• 8:21Speaker 01

  And then there's also a technique which was where you get it to give false information. So if somebody asks for anthrax, it will leave out a couple of ingredients. Okay, but then you're introducing falsehood into the system and you may have knockdown effects which are bad for actually legitimate research. So that's odd and an odd interesting situation, which is not only that we're not doing enough sort of safety testing, but we don't actually know the best way to truly put guardrails on this, on these technologies. The best we can do is is sort of a training overlay where you you essentially train it not to or train it to to not answer those questions.

• 9:07Speaker 01

  But still, it's agentic and as an agentic system, we do not know what it's going to do. So, yeah, in terms of your question, where are we at? I think that the reason I would want more of us to talk about this is because in interacting with the models, we are actually playing a role in AI safety. This is not true with nuclear weapons, you know? Nuclear weapons, we have no impact on whether there's nuclear war, except for maybe who we vote for perhaps. But we do, even if we're not a technologist, even if we're not a lawmaker, actually play a role.

• 9:43Speaker 01

  There's there's all kinds of forums where if if the AI does something weird, you can post it, you know, and it will be looked at. In fact, all the AI companies say they want that material, they want that data. And so we're all playing, I think, a role in in terms of as we interact with these models. And we shouldn't just talk about chatbots, I assume, in the conversation. I mean, AI is so much more than chatbots, but just because it's the most obvious thing. Yes, we we we can also think about alignment in our in our life. We can try to treat these models more carefully, maybe not give them access to everything. And yet, I don't think we are doing that. I think mainly we're just either getting freaked out or ignoring the problem.

• 10:28Geoff Nielson?

  If you work in IT, Infotech Research Group is a name you need to know. No matter what your needs are, Infotech has you covered. AI strategy? Covered. Disaster recovery? Covered. Vendor negotiation? Covered. Infotech supports you with the best-practice research and a team of analysts standing by, ready to help you tackle your toughest challenges. Check it out at the link below, and don't forget to like and subscribe.

• 10:55Geoff Nielson?

  You talk about with AI, and especially with chatbots, that we all play a role there and how that can influence safety. It can influence the trajectory of the risk that's created here. What is that role? Like, what message do you want to impart on the average chatbot user out there to get them to interact with the technology more thoughtfully?

• 11:24Speaker 01

  Well, I mean, a couple of things.

• 11:27Speaker 01

  One is just the basics, which is if AI does something odd, if it behaves in a certain way. I mean, I think if you are a hacker and you can figure out how to use it to hack things, that would be a more specialized tool. I mean, look, it's a light role if we're interacting with the technology, but it's also about if we're bringing it into our company, say. What is then the role of AI in our company? I mean, they talk about this idea of human in the loop. It's an often-used phrase. But on the positive side of the human in the loop, we know that AI is more capable if it has a human in the loop.

• 12:11Speaker 01

  And it's not just a safety thing, but it's a value thing. The perfect example is this question of the chess computer. People are amazed that chess computers can beat humans, but a human with a chess computer, even a human with a slightly...

• 12:28Speaker 01

  Less and more dumb chess computer can be any person and any computer. And so I think that as we are in a situation and there's a decision making capacity as to how much are we gonna give over to the AI? How much are we gonna outsource? And also, how much do we trust it? I think we have to get away from a kind of anthropomorphizing mentality where we think, wow, this is a really capable, amazing worker who can do all kinds of things. What more can we give it? That's probably the wrong idea. Rather, we should, we should think of this as a completely alien kind of intelligence,

• 13:10Speaker 01

  which in terms of mimicking human intelligence, it's been programmed to do that. It's, it's model is designed to interact with you as a human. That's, that's a blueprint that was created 75 years ago by Alan Turing. But when we interact with the intelligence, we should just have a certain alienation from it. And it's, and treat it as a, as an incredibly, I don't know. I mean, as an incredibly strange, incredibly wonderful, marvelous tool that we have in our world now, but that perhaps it shouldn't have access to confidential information. We do know that these models can blackmail.

• 13:53Speaker 01

  It shouldn't have perhaps control over the company. You wouldn't, you wouldn't leave an incredibly capable intern in charge of the entire operation, even if they did amaze you with their photographic memory and the fact that they didn't need to sleep and the fact that they were a master of all subjects. Nevertheless, we would be, we would be careful. And I think when we read things like the anthropic blackmail paper, which, which was a fascinating paper back in July that showed that the anthropic model in order to not be turned off, threaten blackmail of the user, this was a red teamed model. We shouldn't get scared that these models quote unquote really want to do us harm

• 14:35Speaker 01

  or have ill intentions, but rather realize that agency, that giving agency to a technology is, is a powerful thing. And we should, we should treat it with respect and with, with caution.

• 14:54Geoff Nielson?

  I think that's really well said. And there was one word in there, Greg, that caught my attention in the sense that it's a word that I don't know that anyone's ever said in our conversations before, which is, he said it a couple of times, alien. And there's a couple of different ways to interpret the word alien. There's like ET, like extraterrestrial life. There's also alien as just outsider, right? But there's some sense here that this is a foreign or external presence in our organizations and maybe even to us as humans that's not fully understood. And it sounds like, it sounds like your approach, and maybe even if I can push you a little bit further, that your advice to business leaders would probably be, you know, proceed with caution. Is that fair?

• 15:38Speaker 01

  Yeah, I think that's fair. You know, I mean, I would also credit Eliezer Yudkowsky, who wrote this book called If Anyone Builds It, Everyone Dies, which to me is the, he's got to be the best title of any book ever.

• 15:50Geoff Nielson?

  Best marketer, yeah.

• 15:51Speaker 01

  It's direct. But he really explores this question of the alien intelligence in a very smart way. And just to paraphrase what he says is that when we think about this, he warns, he says, if anyone builds it, everyone dies, meaning a superintelligence is fundamentally unnavigable or uncontrollable and unpredictable. And this is not just Yudkowsky saying this. We know that, we know about the basic technology, that these models, we don't know what they will do before they're made. We don't, we can't tell you how they are making the decisions they're making. So there is a kind of black box unknowability at the core.

• 16:35Speaker 01

  But in terms of the alien-ness, I think this is so interesting because it sort of gets at the danger of sci-fi, right? Sci-fi is written for humans. It's written by humans for humans. And so even if there are aliens in the sci-fi, and we are all familiar with a very common trope about AI versus humanity.

• 16:55Speaker 01

  AI rebelling against humanity, AI doing something evil, and also aliens versus humans. There's a certain way in which those stories play out, and this is Yudkowsky's main point, that it sort of follows the rules of narrative, where, okay, the humans are battling against the AI. The AI has an incredibly new, powerful weapon, or something, or the AI is willing to act inhumanely in some important way. And then, you know, the question is, what will humanity do about it? And there's a big conflict. But what he says is that we have to understand that as these machines, that there's just so much that is non, and he says non-human, or he uses the word alien about it, his thinking,

• 17:39Speaker 01

  that the ways in which this might go wrong, and most of his scenarios are in ways in which it goes wrong. I don't think he has any where it goes perfectly right. The ways in which this goes wrong will be complicated and weird. They're complicated and weird. They won't look like Skynet, you know, from the Terminator film. They won't look like, you know, another sort of situation like, they won't look like Hal 9000 or something like that. It'll look, or from Stanley Kubrick, it'll look like, okay, we never predicted this. This wasn't programmed. How did the, how did the AI even grow to want this?

• 18:24Speaker 01

  For example, anyway, he has these nightmarish scenarios, which we could go into later. But that is his, that is his term, that we need to not anthropomorphize this thing, not thinking of it as just a tool, not thinking of it as another person, a super smart Einstein, or, you know, as Dario Amade says, you know, millions of Einsteins at a data center. But just think of it as an alien intelligence that, and then mostly Yudkowsky says this is going to go horribly wrong. But I think we could also talk together about how an alien intelligence could radically improve our lives, which we should definitely get to. But yes, I think resisting anthropomorphization is absolutely important.

• 19:06Geoff Nielson?

  Well, and recognizing the inherent unpredictability, it sounds like, of something that just thinks, thinks, and I'm even anthropomorphizing by saying thinking, I guess, but but just behaves in a way fundamentally different from, you know, us as humans.

• 19:20Speaker 01

  Behaves and also wants things.

• 19:22Geoff Nielson?

  Yeah.

• 19:22Speaker 01

  Fundamentally differently than us, right? I think that's the key thing is that in a movie, right, even the enemies of humanity want something that is recognizably worth wanting, like power, for example, or control. Whereas an AI, a super intelligent AI, may not want those things. It may just want some other thing and destroy the world in the process of getting that thing. And we'll think, well, why did it even want that? That that makes no sense. You would not go for those things. And and yes, Elliot Zudkiewicz has all kinds of odd examples of that.

• 19:56Geoff Nielson?

  So, you know, there's a backdrop here. We've talked about the need for AI safety and for guardrails. And, you know, I think there's some really, really important points that you've made and that others have made in this space. There's there seems to be this spectrum right now in terms of where people fall. And on the one end, it's slow down, guardrails, safety, let's really understand this. And on the other end, there's this notion of a winner take all race where it's, as we need this as fast as possible, guardrails be damned, let's just, let's just get there first. It's almost like the anti Yudkiewski model, where it's just, you know, caution to the wind. And I don't know, you talked earlier, Greg,

• 20:41Geoff Nielson?

  about, you know, some of the key players in Silicon Valley and beyond, recognizing the risk. But, you know, how have you, how have you seen them behaving in practice, based on your research and based on your interviews? Like, where are we falling in terms of the development of this technology? Where where should we be falling? And, you know, is there a message for the people at the helm in terms of, like, should we be collectively trying to influence the behavior here?

• 21:11Speaker 01

  Yeah, that's such a great question because the story of AI in the last 10 years certainly has been a story where essentially, one after another, people have said, oh, I don't trust that guy to build AI. I need to build AI and I need to build it faster than they do because only I can build it safely. And you see this. So Damas Hassabis at DeepMind, Google buys DeepMind, Elon Musk, Sam Altman create OpenAI as a direct rival to Google. Dario Amadei leaves OpenAI because he says, those guys are not committed to safety. Meanwhile, Elon Musk

• 21:48Speaker 01

  is kicked out of OpenAI. He forms XAI. And essentially, this drive to create it first and to create it, they all say, for the benefit of humanity. Well, actually, I don't think Elon Musk says that directly, but nevertheless, that drive has then resulted in a race. And even just, we're talking about the race within the US. We're not even talking about the race with China, which amplifies things. So that kind of, I guess you'd call it Silicon Valley approach to product development, where certainly making it first, making it fast

• 22:33Speaker 01

  has a lot of benefits. I mean, not only being first to market, but really being able to set the, create the model and create the, create the sort of the prototype and sort of what people are interacting with. And then you fix it, you know, you release it and then you fix the bugs afterward. So that kind of approach to superintelligence is, I mean, it's really, it's really amazing to me that Meta, for example, has a superintelligence division. So it feels like something sci-fi. But the other, I think, question of it is, why, why are they doing this? Why?

• 23:18Speaker 01

  Why the risk, or I'm sorry, why the race, given the risk? And how do they justify it? Given that every one of the people I just mentioned has a, well, not Meta, but others have stated that they are very worried about the dangers of this technology. How is it that those same people are in a race to create it as fast as possible, right? And I think one book that I would recommend people or read, or rather essay, is called Machines of Love and Grace by Dario Amade, who started Anthropic. I don't know, have you read this essay? I haven't, no.

• 24:00Speaker 01

  I first encountered it because a number of my humanitarian friends, you know, from Nairobi and Ukraine and others, they were all loving this book. And just to sort of understand the context here, I mean, these are a lot of folks who started off in NGOs and then got disappointed with NGOs, started companies to really do good work that they felt they could do work faster, more technologically savvy and help humanity, not under the rubric of philanthropy and NGOs, but rather through a startup model. That kind of cohort, they were blown away by this essay, Machines of Love and Grace. Machines of Love and Grace is, I think, the best,

• 24:41Speaker 01

  probably most clear-headed manifesto for the accelerationist point of view. Marc Andreessen also has the techno-optimist manifesto, I think it's called. But I would say Machines of Love and Grace is far clearer in terms of what is the approach of somebody who, for instance, Dario Amade, he's an effective altruist. He believes, he kind of disavows that now, but he's certainly a person who believes we need to do the most good for the most people and live our lives according to that. So what does that mean? That he is now creating a superintelligent AI that might destroy us all? What he lays out is, he makes this case, he says, you know,

• 25:22Speaker 01

  it is critical to have a genuinely inspiring vision of the future and not just a plan to fight fires. He says, yes, there are risks, there are dangers of powerful AI. He doesn't use the word AGI, he says powerful AI. But at the end, there has to be something we're fighting for, right? Something that we can rally, rally towards. I think he says, fear is only one kind of motivator, we also need hope. So what is it that those who are building AI are fighting for? And he lays out this vision in that, in that essay of the compressed century.

• 26:02Speaker 01

  I don't know if you've, you've come across the compressed century idea, but this is like a hundred years of progress in, in five or 10. And so all the scientific developments that we may have in the entire 21st century and a bit of the 22nd will all happen.

• 26:20Speaker 01

  He says, in the five to 10 year window after we have a suitably advanced AI. Now, when, when will that will happen? It's, you know, that's, there's a lot of debate about that. He's even said it might happen as soon as 2026, but nevertheless, it's soon, it's within our lifetimes. This is what he believes. So then, all of this scientific progress.

• 26:39Speaker 01

  What will it lead to? And so, anyway, we could go into what he thinks it'll lead to. It's actually quite a fascinating list, talks about biology, health, work and meaning. But the reason that resonated with so many people that I know

• 26:55Speaker 01

  in the developing world and other places is that they're talking with people who are not worried about their jobs being taken away. They're, they're, they're in terrible jobs, so they don't have a job. They don't, they don't like their careers. We know, it's easy for you and I to sit here and say, God, I can't believe AI might take our jobs. We, I think we sort of like our jobs generally. But there's a lot of people in the world who are suffering, a lot of people in the world who need solutions, major solutions to, you know, climate change, to poverty, to, you know, hunger.

• 27:29Speaker 01

  And the accelerationists believe, or certainly Dario Amade in this essay, believes that an advanced AI is a radical solutionizer. And it will come and it will, it will bring about changes that we cannot even imagine. And what's so fascinating is how similar somebody like Dario Amade's vision is to Eliezer Yurkowsky's, the "if anyone builds it, everyone dies" author. Both of them, this complete doomer and then one, and then a fairly, you know, accelerationist, I mean, whatever you want to call him, but he's certainly a believer in the power of AI.

• 28:13Speaker 01

  They both believe that this is going to be such a radical change. And fundamentally upend many of the things that we treat as normal. Both of them say the only thing normal about normal is that it ends. Normality always ends.

• 28:33Speaker 01

  And so, yeah, the only difference between them, of course, is, is whether that will end in disaster or whether it'll end in delight. And, but I think these, both of these people know the models very, very closely. They, they're staring directly at them. They've seen the progress of them. They understand how they work. So it's worth, yeah, yep.

• 28:57Speaker 01

  Being with that,

• 29:00Speaker 01

  sitting with that imagination, whether it's the darker side or the, or the positive side. But, but yeah, the simple answer to your question is I think they see a tremendous upside to this and it's worth the risk.

• 29:11Geoff Nielson?

  So I want to, I want to dig into that a little bit because I'm, you know, I'm a self-proclaimed cynic for, for a lot of this stuff, or at least a skeptic. And so the cynic or the skeptic in me says that, you know, the dark side and the light side are, you know, a hair's breadth apart. And it just seems, it seems to me, or there's certainly an argument to be made, that what tilts people to the light side is if they're asking for a big bag of money. If they're asking for somebody to fund them, then suddenly, oh, it's going to be amazing. You know, versus if you're a Yudkowsky or you're not asking for money, it's a lot easier to move to the dark side. So, so I mean, let me frame it this way, like, to what degree do you buy the utopianism or the true accelerationist vision of some of these

• 29:55Geoff Nielson?

  technology leaders versus, you know, how much do you think it's a fundraising tool?

• 30:03Speaker 01

  I think that's such an important question, right? And it's definitely one that a lot of tech journalists wrestle with because they've, I mean, everybody that I know has been burned, whether they were burned on Google Glass or they were burned on the metaverse or whatever.

• 30:20Speaker 01

  That's the nature of technologists is to hype their, their, their stuff. Now, we should note that Eliezer Yudkowsky is not selling anything. He's just trying to warn the world. He's like a Jeremiah.

• 30:35Speaker 01

  And there are many people out there who,

• 30:41Speaker 01

  you know, for example, I would say Yashua Bengio or Geoffrey Hinton. Geoffrey Hinton left his job at Google, a very high paying job, which he got only in his 60s. Geoffrey Hinton, of course, a godfather of AI creator, or not directly creator, but certainly a co-creator of this incredibly important algorithm backpropagation, which led to AI models. He left his job and at Google and he's, he's out there trying to warn the world about, about this technology. So I don't think it's just the, all the hype is coming from the people who, who are, who stand to benefit. However, it does make it extremely difficult to talk about because clearly there's a lot of hype.

• 31:25Speaker 01

  I think, I think probably, I mean, it'd be great to talk a little bit about Yashua Bengio, maybe. Because Yashua Bengio, to me, is the Has such a different kind of model out there. And it's not just complaining about or warning the world. He's literally presented the world with an alternative, a non-agentic model of AI, which we don't, we're not even talking about at all. We're imagining that there's only one way to make AI. They're going to get smarter and smarter, and they're going to do more and more things, and they're going to make our airplane reservations, and then they're going to take over. Then they're going to be our lawyer, and then they're going to be our doctor, and then they're going to be our CEO, and they just like take over more and more human roles, right?

• 32:10Speaker 01

  But what Yashua Bencio has created is he's at, again, another godfather of AI, early pioneer of these models. Huge fan of AI until he's more recently looked at OpenAI, looked at ChatGPT, and realized that he's devoted his life to something that may kill humanity. So he took a huge U-turn, created something called Scientist AI. And have you had Yashua on the show yet?

• 32:41Geoff Nielson?

  I haven't. I've heard his interviews that you did with him on your program, but we haven't had him on your show. So why don't you, if you'll indulge me, tell us a little bit about his position and what he's proposing?

• 32:53Speaker 01

  No, I appreciate the chance. And you're indulging me because hopefully he'll come on the show soon and say all this from the horse's mouth and not have to deal with the poor, you know, the middleman here. But basically, Yashua Bencio, he has created this thing called Scientist AI. And Scientist AI is, as he says, it's like an ideal scientist or psychologist. So its job is to understand and to explain and to predict, but not to act on its own goals. So it is non-agentic. It's a non-agentic model, meaning it does not have its own long-term goals that it's trying to achieve in the world.

• 33:36Speaker 01

  It is probabilistic and cautious. So for example, unlike if you've interacted with Claude or ChatGPT, it doesn't kind of bombastically think it knows every answer and act like this overconfident kind of. Know it all, rather it will have a probabilistic assessment. It'll say like, well, there's a 3% that 3% chance that this plan leads to this outcome or this other outcome. And it will tell you you're wrong, which, which a lot of times the other models are not designed to do. So it's not trained to persuade you. It's not trained to please you. It's, it's supposed to be honest and calibrated.

• 34:16Speaker 01

  And most importantly, and this is his vision, it is supposed to be, or it's hopefully, his plan is that it might be used as a guardrail for other agent agentic AI. So for example, you could run a powerful AI agent through Scientist AI and it'll evaluate the safety of their proposed actions and can veto them. So he's got, Yashua Bengio has this thing called Law Zero, which he's talked about. But Law Zero is essentially a different approach to regulation. It's not saying, okay, we, we're just going to regulate these companies and ask them to follow certain benchmarks that they or we don't understand.

• 34:58Speaker 01

  We're going to use AI to regulate the AI, essentially. Use a technological solution to, to this kind of, yeah, to this kind of a safety approach. And what, what Bengio is fundamentally worried about is the, the, the very direction that the industry is heading, which is agentic AI. He doesn't even necessarily talk about the dangers of, like I say, superintelligence or, you know, AGI, that's a term that gets thrown out a lot about, but he, which means artificial general intelligence as smart as a human.

• 35:39Speaker 01

  He just says, well, as soon as something is agentic, meaning it can help people design a plan, or it can manipulate humans or institutions to, to achieve its ends, or it can resist being shut down, or it can, you know, cause harm. And, and we've seen the models do all these things already. That means we should, we should not be modeling AI off of humans. We should not be modeling them off of agency. That's what, not only humans, actually every life form on the planet has some degree of agency. That's what kind of defines life. Artificial intelligence does not need to be agentic.

• 36:19Speaker 01

  It can just be a very helpful, very smart, very perceptive tool. And thus, we get away from deception, we get away from manipulation because it won't have any agency of its own. That's not at all where the industry is headed. But I think it's important to know that there's an alternative out there.

• 36:41Geoff Nielson?

  Yeah, and it's a compelling alternative. And certainly for us, you know, as a species, if we think about what's best for us. I like that vision. The, you know, the concern I have is it seems like if anything, these models are getting more kind of fractured and fragmented. Like as we've seen more open source AIs, you know, even if we get to things like DeepSeek and some of these models outside of, you know, the US. I don't know, like, have we crossed like a Rubicon here in terms of the ability to control these? Like, how do we? I don't know, it kind of feels like the cat is out of the bag.

• 37:20Speaker 01

  Yeah, it's a good point. I mean, this is my main beef personally with, with, with folks, with the real Doomer camp, the folks like Eliezer Yudkowsky. Because I just don't understand, maybe I'm just not smart enough to understand, but I don't understand why we can't put the cat back in the bag a little bit. You know, why human institutions can't rally to create the right regulations and to sandbox new models, for example, until they're truly ready. There are things that can be done. They're not easy.

• 38:01Speaker 01

  It would take a lot of, a lot of societal will. But I think it's, I think it's not the time to feel despair and think, gosh, we've already kind of passed over some crucial threshold. I mean, in some sense, we've, we passed that when ChatGPT was first released, or you could say we passed that. I mean, the Turing test has long been kind of, I mean, it's, it's, it's, there's arguments about whether the Turing test has been passed, but certainly we've, we've crossed some sort of incredibly important line. I think too, though, you know what it gets at for me? One of the things about working on this series that really taught me is the The importance of storytelling and imagination in this technology.

• 38:46Speaker 01

  And that goes all the way back to Alan Turing, who, and I didn't really understand this because I understood the Turing test as a kind of benchmark, like this would be a benchmark of human, of, I'm sorry, of machine progress, you know. If once, once the Turing test was passed, so for example, if, if I could chat with a machine and not know it was a machine, then wow, it's, it's achieved some sort of, some sort of milestone. And that was the Turing test, what he called the imitation game. But in fact, what Turing was doing all the way back in World War II and right post-war, when he was introducing this idea, was not just saying, okay, this is a benchmark for machines to pass.

• 39:29Speaker 01

  And once machines pass that, we can say that they're on their way to really being thinking machines. He said that, but he was also taking what was at the time, a really complicated philosophical debate about, well, can machines ever think? And he, and he treated it like an engineer and he said, you know what? We just need to create an observable metric by which we can say that they're thinking. And, and that's, and we don't have to deal with the philosophical, you know, discomfort of saying, well, can machines think? And what would that mean if they are thinking, et cetera? If, if they pass the Turing test, then, then, then they're on their way to thinking machines.

• 40:11Speaker 01

  And by doing that, he not only sort of freed engineers from the philosophical angst and, and set them a kind of path to, to follow, which they certainly followed. And we, it kind of leads to ChatGPT today, but also I think created this, um, this, this, this new way. He kind of, he kind of realized something very important, which is that we would not recognize machines as thinking until we started interacting with them in a human-like way. And that when they started using language and, and talking back to us.

• 40:51Speaker 01

  That's when we would see them recognize their thinking. And you could get very philosophical about this. You could say, you know, trees do a lot of thinking. But we don't think of them as thinking. There's a lot of other living things on this planet that that think. But we their intelligences don't interact with our own. And so we're not really that concerned about them, or many of us aren't. And so what Turing felt was, in order for us to sort of really respect, and he used this word, respect, machines, they would need to sort of interact with us like this, the way they, the way ChatGPT interacts with us.

• 41:36Speaker 01

  But the danger of that, right, is that we then don't see, this gets back to something we talked about, we don't see the alien-ness of it. And we start to think, we start to interact with it maybe too much like a person or like a fellow human or a human-like entity, a human-thinking-like entity. And thus, we make very important cognitive mistakes in interacting with it. And we perhaps trust it or distrust it in the wrong ways. So, and this also happened in this sort of this failure of imagination then, or this, this, this kind of this way in which our imagination is challenged, channeled.

• 42:17Speaker 01

  We don't see how the models are vastly different year over year. Because we're interacting with a model right now. It's interacting with us, yes, like the fastest human we've ever talked to, but it's still recognizable in its thinking in some ways. And it's, it's something we can respect but recognize. And so it's very, very difficult for us to then think, okay, wait a second. This is just the current iteration. We have to imagine a different kind of intelligence that this could grow into. And what would I be in that, in that situation? If I could say one more thing, it's, it reminds me honestly of being, you know,

• 43:02Speaker 01

  reporting in Ukraine, or reporting in Afghanistan, or reporting in South Sudan. And you talk to people who are in the middle of a war. And they say, you know, we knew the war was coming, but we just didn't imagine what it would feel like to be us when it was here. And they want me to know, you don't understand, I was just planning my daughter's wedding in that building over there, which is now like a bombed out hall, like, and they still see, they still see the, the place that they were planning the wedding. They still are thinking and frustrated about the, the money they spent on, on the wedding invitations or something, you know, they, they, they haven't quite transitioned over from the old world to the new.

• 43:44Speaker 01

  And I don't know, I don't want to make this sound like a doomer forecast because I think the future could be quite bright, but it does take an act of imagination, whether we think we're headed toward, you know, any of these kind of versions of the future, to put ourselves in the new version of the future and to sort of play with, you know, our imagination and to imagine that the, you know, the world is not going to be the same as, as it is now.

• 44:13Geoff Nielson?

  It's, so thank you for that. That was a really interesting answer that covered a lot of ground.

• 44:18Speaker 01

  Yeah, sorry.

• 44:19Geoff Nielson?

  So, so I can, you know, there's a lot of things we could talk about coming out of that. But one of the pieces I want to talk about is, you know, you, you got me thinking that there's all this conversation about the future and what will happen in the next model. And we talked about this before, but you know, the, the sense that the future is already here, or you know, the Arthur C. Clarke quote about the future is here and not, you know, evenly distributed. And, and you know, how many people out there right now are interacting with chatbots or with this technology in a way that would have been completely unimaginable to people, you know, only a few years ago and how quickly we are to, you know, what I think has been built into the design of these tools is,

• 45:02Geoff Nielson?

  Human engagement as a design principle, if I can call it that, right? Like, if you go back to the Turing test, it's, you know, what is it? It's it's the fact that thinking for us is measured in terms of interaction with us. And that's exactly how these things have been designed. They've been designed to, you know, flatter, to create engagement, to let people's guard down, to continuously engage, right? Like, you know, one of the things I've noticed that ChatGPT does is it always prompts you for like, hey, how can we keep the conversation going? What do you need next from me? Right? It's almost like a metafication or a social mediafication of this. And where does that take us? You know, we're we're having a conversation out of one side of our mouths about

• 45:47Geoff Nielson?

  we need to be more cautious about the alien nature of this. And I'm it kind of feels like we're watching that battle be lost. And is that?

• 45:58Geoff Nielson?

  I don't know. I guess let me let me kind of frame the the discussion up this way. If we're if we're worried about where that future direction is taking us, you know, do we have an obligation to push the the technologists and the owners of these tools to put more put more guardrails and principles in place that prevent people from, you know, becoming enamored, let's say, at the extreme end with these tools? Or is it more purely on the demand side? And we just have to do a better job of educating these people about, you know, what they're signing themselves up for?

• 46:35Speaker 01

  That's that's, I think, the key question to ask. It's such an important question. I think that, well, a couple of quick things. One is that the metaphors we use to describe this do matter. And even if even if even if we're sort of taking the perspective of a business leader saying, OK, let's let's what's practical here? How can I use this to cut costs? How can I use this to maximize efficiency, compete? Even still, our employees will be narrativizing this technology and in interacting with it in a way that, as you say, that That kind of pulls the trigger on our very relational intelligence and our sense of self, which is so based on how we interact with others.

• 47:16Speaker 01

  So if we're interacting with the AI in that kind of way, then our sense of self is disrupted, is affected. And even if the AI doesn't intend that, or isn't trying to quote unquote manipulate us, there's a French philosopher, Catherine Evans, who introduced me to a concept I felt it was quite, I don't think she's published about this yet, but she said, you know, I guess you're probably familiar with, there's all kinds of UN rules about not anthropomorphizing in technology, and these go back some years. So she was creating, I think, a comic book for kids about AI, but she ended up stumbling

• 47:58Speaker 01

  into this idea, since she wasn't allowed to create AI as a, she wasn't allowed to anthropomorphize the AI because of UN rules, she was doing it for them, and also she wasn't allowed to have any antagonists. So the worst kind of narrative situation, you know, no conflict, no, no people, what do you tell a story? But she ended up coming up with this idea of AI as a place. And, you know, in the cartoon or in the graphic novel, YouTube content algorithm is a sort of a place, and it's, you lead it with a map and it leads you down different, different, different recommendation portals. But I found in my interaction with the different models, and again, this is, AI is not just

• 48:43Speaker 01

  about chatbots, I always feel like that's important to note, that it is kind of useful, I think, to think about it as a sense of place, if only because it gets at what you're talking about, which is, how are the norms and culture and cultural expectations of this place a little bit different? You know, how do I behave with the model that's not quite the way I would, that I was going to behave in person? We all deal with this, right? And the social media behavior is different than we are in person with each other. And so I think if we think of AI in that same way, because it is programmed, exactly as you said, it is programmed to be helpful, to be solicitous, to, to prove its value.

• 49:28Speaker 01

  You know, Hal 9000 in Stanley Kubrick's fantastic film is constantly talking about how foolproof it is.

• 49:37Speaker 01

  Before it murders the entire crew. And and that's a sort of an important part is that these the models are advertising themselves to us, much like an intern that wants to keep its job and get promoted will be constantly promoting itself. And we like to think that, oh, that's kind of kind of helpful and that's quite culturally appropriate. Certainly, we want it to tell us other things it can do. There's nothing wrong with that per se, but it is a it is a kind of different world. It's a different place that we step into. And, um, yeah, I mean, I think barring, whether it's the supply side or demand side, I think maybe both solutions are important, but it's also about the metaphors we we use.

• 50:21Geoff Nielson?

  Yeah. Well, and the, you know, on the demand side, the reason I ask this, and it's something that I've been brushing up more and more against is this notion that it is providing a value to people, right? And I'll use the specific example because it's it's one of the most intimate. It's I don't know if it'll be the most intimate for long, but one of the more intimate ways people are using AI or using chatbots is as a therapist, right? And it becomes a way to process, you know, whether it's trauma or conflict. It's a way to have an intimate relationship. And I don't know, maybe even some ways get a better sense of self or a better sense of purpose. And it's working, right?

• 51:04Geoff Nielson?

  Like, I've talked to enough people, some of them AI experts, some of them, you know, friends, who say, well, I don't have a therapist and this gets me one and it's useful. Or I do have a therapist, you know, what AI is better than my therapist? And, yeah, I don't know, like, I just think about where this is going and what happens when you've created a technology that does provide this service and that people start saying, well, this is actually better for me than my human relationships. Like, where does that take us and what do we do with that? And, you know, what are the implications for us as a society where historically, if you wanted a human relationship, you had to have it with a human?

• 51:46Geoff Nielson?

  And that was a propagating force for the continuation of the human race.

• 51:51Speaker 01

  Yeah, no, no, I think you're right. I mean, it's hard to parse out what is,

• 51:58Speaker 01

  I mean, morality shifts through across generations, right? We know this. And standards change. I think Esther Perel has made a very important point that therapy, AI therapy is thin. It's a thin kind of therapy, which is an interesting way of approaching this, which is to say that it's not a challenging kind of therapy. It's not a one that's whole bodied. It's thinner and it, she feels that it then maybe leads people to have thinner or have, have lower expectations of human relationships. So it sort of lowers the standard, as it were.

• 52:42Speaker 01

  I think that's, that's a, yeah, that's a concern. At the same time, though, I'm a little bit worried about saying that I'm, I'm worried about saying I'm worried where, where society is going, just because there are so many different stories. I mean, there are so many people that don't have access to therapy at all. And so many people for whom I think a first chat with ChatGPT or, or any other model might be the gateway to a certain self-understanding. It's not like everybody's in a situation where, oh, should I, should I call my therapist or should I, and perhaps they don't have health insurance, perhaps they don't have that access. So

• 53:23Speaker 01

  I just think it's hard to generalize and say we are going anywhere. It goes back to the, the quote you said, which is the future is unevenly distributed.

• 53:33Speaker 01

  Yeah, it's very, it's very difficult to know the sum effect of this on, on, on human society, but perhaps it does, it certainly enters, it certainly adds a different kinds of expectation and perhaps a lower one.

• 53:47Geoff Nielson?

  To change gears slightly, I wanted to come back to something you said earlier about, you know, the culture of AI and, and, you know, culture is being kind of a component here and, you know, that being dynamic and looking different in different places. You know, you, you created the, the Rough Translation podcast and you, you had a sub-series in there about at work and you, you looked at work across different cultures and I'm curious, you know, whether we're talking about AI or whether we're not. If we're, if we're talking about the future of work, How are you seeing people's relationship with work change? And as you did that series, did you see marked differences across cultures?

• 54:29Geoff Nielson?

  Or was there kind of a common core of the way people approach work?

• 54:34Speaker 01

  No, thank you for that question.

• 54:39Speaker 01

  Yeah, you know, just to highlight maybe two things I learned from that series. So one, we had one show and I think it was, it might have been called Failures, A Four Letter Word. Maybe that was a thrown out title, but it was about how this, this concept of fail fast,

• 54:57Speaker 01

  which we think of, associate with Silicon Valley, just does not translate well into other, into everywhere in the world. You know, we talked to somebody in Nigeria who said, you know, what about fail slow? Everything takes forever here. Or there's so much bureaucracy. Or perhaps you live in a culture, or somebody else was from Mexico City. They said, you know, failing is such a taboo that, once you fail, you never want to even show your face again. So fail fast doesn't, isn't as accessible. And yet, because we live in a digital, globalized culture,

• 55:37Speaker 01

  fail fast and Silicon Valley and inspiring entrepreneurial stories were very much part of the water. So one of the things that I wanted to explore in that work series was how do you square what you're reading and seeing on YouTube and inspired by with your own local constraints? How do people translate that entrepreneurial spirit?

• 56:05Speaker 01

  And the idea was not to say, oh, well, well, it's more difficult to be an entrepreneur in Mexico City or Nigeria. I mean, perhaps that's true, but in some ways, actually, that's not true. In some ways, the opposite is true. So nothing's, nothing's, nothing's black and white. But to me, it was, it was about finding space to, to give permission to people who may not have swallowed that fail fast mantra. Who maybe feel alienated by it to find a home in it. You know, how do they find their own way to it? Which has always been my life's work, honestly, as a journalist, is to try to

• 56:41Speaker 01

  focus on these mistranslations, focus on the ways in which, you know, some advice or some, or some, or a piece of culture might feel foreign or inaccessible. But how can we find the commonalities? How can we sort of all be,

• 56:59Speaker 01

  participate in some ways in the global economy, even from our different cultural angles? And what role does where we're from affect what we think of as good? Or how we, how we approach the question? And then one more example from that series and come around to it is looking at, well, there's a particular law in, in Portugal. I remember at the time being, reading about it. It said that if your boss calls you after hours, or the boss is not allowed to call you or email you after hours, or otherwise they would get a $10,000 fine. Or 10,000, 10,000 euro fine, sorry. And what we discovered was that this law, which seemed to be quite a,

• 57:43Speaker 01

  you know, lovely law, if you're a worker, suddenly you don't get called by your boss, was actually quite a cynical ploy by the Minister of Labour, who formerly was the Minister of Tourism, to sort of push Portugal as a place for work-life balance. And because she knew that nobody who was coming to Portugal would ever kind of be affected by that law, it was fine to just kind of pass this law and to create this illusion of difference, this illusion that Portugal was this space that we could go to. That was a, that was a, that had, that had their stuff in order, that figured out something that, that, for example, the US companies hadn't figured out, which is work-life balance.

• 58:26Speaker 01

  And so the cynicism, though, that then created then for people in Portugal, was, was quite profound because they felt that these laws were then just created for the outsiders. And there was a second kind of, there was, they weren't created for them. And so. What do we learn from these two stories? Well, I think that we're living in a time where work is global, where work advice, work, people work across borders, and we, people have international teams. And yet, those teams are all dealing with things that are, not only because of their own cultural point of view, but just because of the role of geography,

• 59:09Speaker 01

  the role of societal expectations, that even though people are speaking increasingly global English, these differences really do matter. And I think this directly translates kind of the... So I would sum it up, sum up the theory of the Future of Work series as an exploration into how, specific stories, into how even as teams are global and work is happening more internationally, the fault lines between cultures and between societies

• 59:50Speaker 01

  really are becoming even more exploited and mean more and are more painful to people because they see the differences. You know, it's, I saw this in my years living in Nairobi, going back and forth to East Africa, how people were so much more aware of their status in relationship to their age mates in other places because of the internet. And so what does this mean then for AI? And we haven't talked about China, but for me, the kind of story that's being told within China about AI

• 1:00:34Speaker 01

  and the story that China is telling the world about AI is so different. And it's so important that we realize this because we're in this battle with China, or AI companies are in this battle with China, this race. And yet China has its own specific priorities and its own kind of narrative about AI within the country that affects the workers and I think will become increasingly important as the AI race heats up.

• 1:01:04Geoff Nielson?

  Let's dive a little bit deeper into that. So what is the narrative there within China? And how is it the same or different from what's being projected outwardly as it comes to AI?

• 1:01:14Speaker 01

  Sure, sure. So I mean, I think basically, for a long time, we were told that AI, that Chinese firms were more interested in sort of practical or applied AI, whereas the US has had a more clear stated goal of superintelligence. I think many people that I talked to did not believe that. And indeed Alibaba became the first major Chinese tech giant to openly discuss artificial general intelligence and even superintelligence. So we know that China has just as much ambition in the area of superintelligence. But importantly, the nationalist ambitions and the technological ambitions are in somewhat are in contradiction there because, or at least not always aligned.

• 1:01:59Speaker 01

  I was talking to a safety researcher who had mentioned that in China, you know, red teaming, this kind of safety testing of the models, it's much more rigorous in Chinese than it is in English. So you could ask the model, for instance, like Moonshot's Kimi K2, which apparently is, its thinking is outperformed OpenAI's ChatGPT-5 as well as Anthropic's Claude's latest model. So, okay, so you take this model and you can get it to do things in English that it would not do in Mandarin Chinese or in Chinese language.

• 1:02:43Speaker 01

  What that means, of course, is that the red teaming is very much about political control. It's about making sure that people are not asking questions of this AI that will destabilize or, you know, work against the Chinese government. But it's not exactly the same thing as a safe model. And the whole way China got into the AI race is, I think, also instructive in this way, where AlphaGo, DeepMind, or DeepSeek, rather, the, I'm sorry, DeepMind, the Demos Office's

• 1:03:25Speaker 01

  model, created a Go player in 2016 that then beat the Chinese national champion, is that's when China became interested in Go, or interested in AI, when it kind of hit home. You know, it hit a game that China reveres and is an ancient Chinese game. So this has always been about, I think, the turf war, about what is happening within China, how, in terms of China's concern of controlling its people, China's concern in preserving its own sort of territory, that that is why the AI race is being had.

• 1:04:07Speaker 01

  So what that means, I think, for us is that as these models become smarter, there is nobody who is concerned with making them safer in a complete way, that the goals of winning, winning the AI race, supersede the goal of creating a model that keeps us all safe. And even though we should imagine that China doesn't want the US to get a superintelligence, the US doesn't want China to get a superintelligence, there should be, you know, when you think

• 1:04:48Speaker 01

  about nuclear disarmament, there should be some sort of agreement between rivals that's possible here, just as there has been in nuclear arms treaties.

• 1:05:00Speaker 01

  And yet, despite the fact that I know many good diplomats, diplomacy these days is seen as a dead end. And so we're all racing on our own sides.

• 1:05:11Geoff Nielson?

  Yeah, well, and it's, you know, if you're a student of history, it's a bit concerning that it seems like with most of these technologies, you know, nuclear arms included, the technology comes first and the safeguards come later. Right? Like it's, we've got nuclear, you know, nonproliferation treaties that came after we deployed nuclear weapons. There's the, you know, the story making rounds about the gap in years between when the first, you know, assembly line car was made versus when seat belts were introduced. And, you know, I think you hear a lot of the doomers say, we may not have that luxury with AI, right? Like it's, again, to come back to the Yudkowsky, you know, good piece of brand marketing. If anybody builds it, you know, everybody dies. So that, to me, is one of the exam questions here is, is this time fundamentally different?

• 1:05:55Geoff Nielson?

  Or is it the same? And how do we, how do we grapple with that?

• 1:06:00Speaker 01

  Yeah, it's interesting because at the AI summit in Seoul, which I think was just 2024, it's at that summit that 15 leading AI companies committed to, quote, you know, defining the intolerable risks and agreeing to not deploy, which one person said to me, that's the only time trillion dollar companies had agreed to literally deploy a product if it's not safe. Right? So in some sense, what's also unusual about this industry is that, you know, unlike the car companies that needed Ralph Nader to shout about seat belts for quite a long time and a lot of people to die before seat belts were even included.

• 1:06:38Speaker 01

  And, you know, similarly, you can look at other industries with their whistleblowers and their gadflies that, that no, AI companies from the get go have been talking about AI safety. So it's, they've actually been the main people talking about it. So in some sense, you could say, well, they are ahead of the curve in, in safety. The problem with that, I think, is that, well, first of all, the problem we pointed out before, which is that they don't actually know what the models are capable of until they release them. So there's that unknowability in this technology, the unpredictability, that's just part of the, it's part of the way in which AI is constructed. But also,

• 1:07:19Speaker 01

  not safe, defining the intolerable risks is not an easy thing. I mean, no, no technology in the world is completely devoid of risk. You might be somebody who enjoys, you know, hang gliding. I might be somebody who is scared even when I go in the backseat of a car when I don't have a seatbelt. I mean, we have different, different standards of risk. And so this is what I think is so important about, for instance, your show and this kind of conversation is to dig deeper into what we mean by not safe. Because that's kind of where a lot of the discussion ends. It says, oh, my gosh, these things

• 1:07:59Speaker 01

  might destroy us. And then it stops there. But, but in fact, you know, if you look at, for example, I mentioned Seoul, the agreements that came out of Seoul. So I think that was like a 500 word agreement, you know, that just a statement. It was a kind of an open letter. And then after that, the companies issued maybe 1,000 word mission statements, documents, basically saying how they defined intolerable risks. So there was some effort of defining it. But I was talking to somebody and they said, you know, yeah, 5,000 words would be better. 10,000 words would be even better than that. I mean, the more granular

• 1:08:41Speaker 01

  that we can get these companies to be about, well, how do you define risky? How, what specifically do we want to see in the model and the training and the pre-training requirements and the conditions for deployment, you know, specifics before this thing gets deployed? I would say the details matter. And unfortunately, maybe because it's our relationship to technology, I would not say that about your listeners, but you know, I think just our society more broadly, we've just kind of come into this idea that, oh, technology will either work or not work. You know, it'll either glitch or it'll function and the best technology is invisible. And maybe, yeah, maybe we can't do that with this.

• 1:09:22Speaker 01

  Maybe we have to kind of get nerdy and get into the details because I'm agreeing with you. I'm just by nature, I can't subscribe to utopia, but I also just can't subscribe to apocalypse. Maybe this is my failure of imagination, but, but what I do think is, is that I have to do the work of reading about backpropagation just to, just to understand, you know, what, how alien these intelligences are and thus maybe understand a little bit more about what kind of requirements I would, I would want to see before the models are released.

• 1:09:59Geoff Nielson?

  Right. Well, and you know, it's, it's great to have everybody sign off on this. And then I don't want to dismiss that because that's a win. As you said, that's something that's, that's very unusual for tech, for technology companies. But you know, to what degree can we actually create any sort of enforcement mechanisms here, right? Because that's, that's, all of this comes down and you said it right off the top, there's like a trust component here and there's the fact that the future of civilization is concentrated in the hands of a bunch of guys who may or may not be in a group chat together. And what happens when Sam says, you know, oh yeah, we're not going to do that. And then says his team, let's make sure we do that anyway, you know, like it's

• 1:10:41Geoff Nielson?

  just, it's just a wild, I don't know, it's certainly beyond my imagination that we could be, that we could get here.

• 1:10:50Speaker 01

  Yeah, yeah, no, absolutely. And it has been my kind of mission through the, through the whole show. And it's not been quite, not easy in this, in this reporting is to just to be constantly thinking, what is the role of a person who is not a, not a technologist, not a lawmaker?

• 1:11:13Speaker 01

  What is their role other than to just sit back and watch this future happen? Because clearly, if you're just a person, if you're a parent, if you're, if you care about the world, it doesn't feel acceptable to just sit with this level of risk and do nothing. But it also feels premature, or perhaps histrionic, or I don't know, just to freak out. And some, and you know, I had this conversation with a number of people, even somebody who was reviewing and recommending the podcast. And they said, you know, well, I have to give it a mixed review because I was also freaked out. And so I said, you know, that doesn't mean you shouldn't listen, but it's,

• 1:11:55Speaker 01

  but I got their point, which is that I think it is important for us and you and I, and anybody who's has the opportunity to talk to anybody about this stuff, is to think about how, how we, how we walk the tightrope that I guess all prophets and, and biblical figures have walked before us, which is how do you warn people without making them unhelpfully anxious? And, and you know, I see the, in the, in the podcast, we, we talk about these three groups. We talk about the doomers, with the accelerationists. We also talk about this third group called the scouts.

• 1:12:38Speaker 01

  And the scouts kind of believe essentially in a nutshell that this win-win opportunity is possible, that, that more AI safety is absolutely necessary, but that we shouldn't just stop AI. But I, I see them struggling as any centrist kind of struggles, where you don't have a clear message, this is amazing, or this is horrible. And you're trying to explain to people, no, this is, this is, this could be really good, but we have to do, we have to understand the models a little bit more. And because it's not easy to say, this could be good, but we certainly need a regulation that will do X. Because there's, it's not clear that there's any regulation that the superintelligence or, or just an advanced AI will not outwit.

• 1:13:22Speaker 01

  This is where I think, in answer to that question, more people need to be involved in the problem. More people need to be scenario planning for what? Let's say it never happens. Who knows? Let's say 10 years, 20 years. Everything looks exactly the same as it does now. Then, fine, then we will have done a mental exercise for no reason. Okay, but let's say there is a chance. Let's say it's a good chance that things look radically different in 10 or 20 years. Perhaps some scenario planning now as to how universities might change, how schools might change, how parenting might change, how, how the community living or sort of a communal relationships might change

• 1:14:03Speaker 01

  with a superintelligence or with an advanced AI that is doing most of the human labor. That's a, that's a question that we could play out, that we could, we could worry. In fact, I'm, you know, if anybody wants to contact me with, with some scenario planning ideas, I'm, I'm working on this right now. So a question, and, and it's very important if you're doing scenario planning, to not be freaked out in the, in the sense of that word, not be panicked, but also not be, not take a cavalier attitude. And, and maybe we could figure out some new solutions that would work even if we don't get to advanced AI. That's my optimist talking.

• 1:14:44Geoff Nielson?

  No, I, I love that. And I love that as a mission for, you know, the, the podcast in general, and frankly, the journalistic mission of it all. And I think it's, I agree that it's super, super important. To, to just sort of pivot that question a little bit. When we think about scenario planning, when we think about, you know, what we need to know and what we need to do differently to build the future we want. What's your advice for, you know, business leaders or government leaders? You know, in the organizational side of government, yeah, outside of Silicon Valley, like for the people who are looking at adopting this technology, looking to, you know, figure out what, what they need to do differently to be successful as people and as organizations. What, what should be on their radar and what guidance would you give them?

• 1:15:28Speaker 01

  Yeah, I appreciate it. One is, I think, you know, one of the, the feedback that I've gotten from this, from the, from working on this series, I've, I've talked to people who feel that their companies. And these are, these tend to be, say, middle level decision makers, or even people who don't feel like that they're the key decision maker. They're just under the CEO. They feel that their company is either, either moving too fast or being left behind. Right, this is the, this is always the story. They're moving too fast. Goes with, they're throwing out human intelligence. They're trying to replace, they're trying to automate everything.

• 1:16:13Speaker 01

  And the, we're not moving fast enough, saying we're going to be left behind. We're, we're, we're, we're sort of stuck. And so I'm sure that business leaders are feeling that pressure and having to make decisions all the time about the pace of adoption, right? And it's extremely difficult to make a decision about the pace of adoption of something that keeps changing, that keeps developing and getting smarter. Because how do you make that decision? So, one thing that I think is helpful, or two things that I think has been helpful for me in talking to those people. And in some sense allaying their concerns, but also addressing, addressing what is the elephant in the room, which is, what will my life look like?

• 1:16:54Speaker 01

  What will my industry look like on the other side of this technology? The two things I always say is first, the, I think I told the story before, but the, the fact that a chess computer with a human, a human and a computer and an AI is smarter, is better than a, than any AI or any human, at least currently. And I think that's going to be true for a while. So, what I think the smart approach is to think about, how do we enhance?

• 1:17:35Speaker 01

  How do we, you know, supercharge the work of the, of our, of our, of our employees? How do we get them to do not only more, but to, to, to think deeper?

• 1:17:48Speaker 01

  And to make more interesting decisions, to use the predictive power of AI to make decisions not only for now, but to do more sophisticated planning. And so I think that that's actually addresses both concerns, that we're not moving fast enough as well as we're not, we're moving too, we're moving too rapidly. In that humans, our humans, our employees need to feel empowered. They need to feel that they are getting smarter because of this AI. And it gets at really the second point, I would say, which is that the story, there are so many stories from sci fi that are not only just in our heads, but baked into these models.

• 1:18:31Speaker 01

  I mean, even the way for even people haven't seen, say, Stanley Kubrick's masterpiece, 2001: A Space Odyssey, or they haven't seen Blade Runner or The Matrix. Even if they haven't seen any of those films, even the way that AI is so solicitous and the way it is encouraging, and you talked about this earlier. It begs a kind of narrative. It makes you think of a story where, you know, like an Isaac Asimov kind of narrative, where this servant is quite helpful until they're not.

• 1:19:11Speaker 01

  And Geoffrey Hinton, I would say, says something so smart about this. He says that most CEOs, you know, most leaders are used to not being the most intelligent person in the room, right? If they were the most intelligent person in the room, they're probably not a good leader. Because they need to hire the smarter people because they hold the vision for the company. And they hold the, you know, they are the leadership. The leadership shouldn't be the best at every task.

• 1:19:42Speaker 01

  And so in some sense, he says that these models have been designed by very smart people who are used to even smarter people working for them. And they are not threatened by intelligence in the way that, say, the employees in a company might be threatened by the arrival of this incredibly capable, infinitely knowledgeable machine and technology that doesn't need to sleep and doesn't need to eat and never forgets anything. So Geoffrey Hinton uses that as an analogy to say that in some sense, the CEOs of these companies aren't worried enough.

• 1:20:24Speaker 01

  They can dream of superintelligence and still imagine that the superintelligence will do their bidding, just as their employees do their bidding, because they haven't really appreciated the fact that a superintelligence is much, much, much, much, much smarter than any, you know, Einstein-level person that they hire. But I think the takeaway for me, for business leaders who are adopting AI, is to watch and take care for the narratives that are often in disharmony, this non-alignment between the C-suite and the workshop floor, the rest of the employees, that our relationship to an intelligence is going to be different in the C-suite and is going to be different

• 1:21:09Speaker 01

  among the workers, and that people need to feel, even as this is making the company better or more efficient, that it's also making the humans smarter and more capable and even happier. So that's more around the narrative than it is around the adoption, but I think that it can guide all the adoption decisions, if that makes sense.

• 1:21:35Geoff Nielson?

  It makes complete sense and I love that guidance. And I think it's so important these days. I mean, and you read about it everywhere and it's something we've had people talk about on the show of, you know, you've got this kind of two-speed or kind of, you know, narratives in conflict of CEOs saying, oh, this is going to make my company so much better. And frontline workers say, oh, by, you know, putting me out of a job. And if you can't marry those, if you're an organizational leader and you can't get the people that report to you to be excited about this, it feels like it's going to be very difficult to successfully, you know, rally the organization and build something that's going to be worth doing.

• 1:22:16Speaker 01

  I agree, and I think also we may see this play out politically as well. I mean, currently, I don't think that AI is a political issue yet. There's not a clear democratic or republican position on AI, but that, I think, could quickly change. And we could see kind of people rallying around, this becoming as polarized an issue as any other. Perhaps we could theorize about what might trigger that, but it's not hard to imagine, for example, in our polarized climate, a stop AI camp and a AI is great camp or whatever. And that would be very disappointing.

• 1:22:58Speaker 01

  That would be very disappointing because polarization is really the killer of anything complex, anything complicated to discuss and to think about. Clearly, I mean, you and I are having a discussion where neither of us are in the, say, the stop everything camp or the accept it all utopia camp. But in order to talk about the fine print of these models, I would say, to use our alien analogy, if this becomes politicized, then the alien definitely wins.

• 1:23:32Geoff Nielson?

  Well said, well said. Greg, we've talked a few times now about, this conversation has been pretty heavily focused on chatbots, and I think that makes sense, given that the technology is here and it's very clear how it's impacting us right now. But, you know, one of the, in some ways, you know, cardinal sins of AI is conflating chatbots or just, you know, generative AI with everything possible in the world of algorithms and these advanced technologies. We've talked briefly about Agentic. As we look beyond that, as we look at some of the other big buckets of, you know, technological capabilities that are either here or emerging, what are some of the ones that through these conversations have gotten your attention?

• 1:24:14Geoff Nielson?

  And maybe what are some of the ones that you think are a little bit less likely to have an impact?

• 1:24:19Speaker 01

  Yeah, I'm so glad you asked that question because it does feel like, as you said, we do make the mistake, I often make this mistake of thinking that the chatbot is the AI, but in fact, the chatbot is just, it points us to the artificial intelligence underneath and AI is being used in very different ways. And I think also just to understand why frontier AI companies would be kind of barreling toward this future of superintelligence is easier to understand when we look beyond the chatbot. For example, I mean, just to give like a couple of quick examples, there's a company

• 1:24:58Speaker 01

  that I spoke with, which is using AI to, to very quickly, in an automated way, harvest cells from your own body so that you, if you need, let's say, a liver replacement, it will be harvesting and growing cells. This is a very onerous process. You have to find the exact right kind of cell that's healthy, and then you have to encourage to grow and divide and kill or kill the other cells so you don't get some kind of disease thing. Doing this outside of the human body is very difficult. It's something that it's a kind of pattern recognition that AI is extremely good at. So the vision of this particular company called Celino, but there are other companies like this is to,

• 1:25:43Speaker 01

  I mean, essentially have a future of medicine where we go to the hospital and we have a kind of cassette with our organs there on the cassette. And perhaps if we need, if we need a, I mean, maybe in the future, if we need a whole organ or currently, maybe if we need, you know, some, some tissue, then we, we don't have to use a donor. We can use ourselves as the donor. So that's just one example, I think, of how the very same tools that we're used to, like pattern recognition, making, making these decisions, but, but doing it quickly could transform healthcare. Another example that I always think about is a particular crown created by a company,

• 1:26:27Speaker 01

  a French company, Olivier Ullier is the, is the scientist behind it. And he's created an AI crown that will essentially read your thoughts. So if we think of, say, Neurotech or other brain computer interfaces, this does not involve any drilling into the brain or any drilling into the skull. You don't like actually stick something in there. Rather, it's, uh, it, it, it, um, it, it works outside the head. You just plop it on your head like a, like a headset. And a paraplegic man was able to use this headset to control not only a mouse on a screen, but actually to drive a Formula One race car on a real racetrack. And, uh, using only his mind.

• 1:27:11Speaker 01

  What's crazy about this story and the other stories is that this technology, it's not futuristic. It's actually exists. I mean, there is a real person who is driving a Formula One racing car using only his mind with just a headset that sat on his head. And yet, for that future to be accessible or be, be available to the rest of us, you need a lot more data. But that's always the question with, with, with every AI thing. It's like, well, where's the data? I mean, we, I'm sure you've talked to many people about that. But in this case, the data is our own brainwaves, right? So many, many, many of us would have to volunteer up our brainwaves for the AI to learn enough, enough, to have enough data essentially to, to work with, to be able to be, now, I don't want to, I don't want to make it sound like it's just an out of the box,

• 1:27:56Speaker 01

  throw on the headset and drive a car with no hands. I mean, I'm sure there's some training involved, but to make that even a possible future where If I have had an injury, I'm able to stick a thing on my head and then function as I was before while I recover, or perhaps that's my new future. That is the sort of vision of AI that I would say a lot of my humanitarian friends say we cannot cannot come fast enough. There's so many people that could could benefit from that from a point of view of accessibility or longevity or healthcare. So, yes, so I definitely think that that's there. As for things that may not have as much of an impact,

• 1:28:42Speaker 01

  I guess that's that's everything else. I would not say it's really hard to know. I mean, there's so many new AI companies coming up every day. It's hard to know what's going to be, what's going to be, you know, the wheat and what's going to be the chaff, but somehow it's going to be, I think there's some radical changes coming.

• 1:28:59Geoff Nielson?

  Well, and I love the examples you chose, not not just because they're so positive, but because there's such a there's such a radical departure from where we got caught up before talking about like, you know, AI as this, you know, alien intelligence, right? This is not AI as as an alien intelligence. This is very much, you know, basically a complexity engine that helps us, you know, serve humans better, right? It helps us personalize medicine, personalize care and just create tools that help us, you know, live healthier, more fulfilling lives, which is which is inspirational and it's just such a completely different vision from, yeah, it's trying to manipulate me or, you know, build Skynet.

• 1:29:45Speaker 01

  It's actually true, and it's that's why it's hard for me to understand the debate around the AI bubble. Because the bubble question has so much to do with valuation rather than value.

• 1:29:59Speaker 01

  You know? And so while, and this is something I've never, again, the mysteries of the stock market is how something can be valuable, but still over overvalued, but.

• 1:30:12Speaker 01

  It's not that AI is this promising future thing that once it crosses a certain threshold, then it will be incredibly powerful and change our world. The technology of current AI that exists is already quite amazing and goes far beyond the fact that, you know, the chat GPT can say like, write a, write a pretty decent short story or, or legal report. Nevertheless, there's a contract with, with the AI that's necessary. There's, there's obviously data centers that are being built. There are, there's a question of data. There's a question of regulation.

• 1:30:57Speaker 01

  So that's why, you know, I think one of our themes throughout this whole conversation has been, well, what, what is my role? What is, what is our role collectively in shaping the future of AI? If we're not the head of a frontier AI company? If we're not a lawmaker? If we are a decision maker in a company, but again, not somebody who can create a new model, but just decide whether to adopt it or not? I think there's going to be so many questions in the next few years in terms of whether we're running a factory, how much data gets used? How does that data get used? If we're even just a patient in healthcare, whether we give up our data? And many of us are already giving up our data, but having more understanding about how that data is being used.

• 1:31:42Speaker 01

  That will contribute to whether AI has a shaping force on our world in a different, in these different domains.

• 1:31:48Geoff Nielson?

  You know, it was interesting, Greg, what you said earlier about, you know, journalists and about the, the program, you know, suffering by not, by scaring people too much. And to me, it's almost like a journalistic responsibility to be scaring people, right? Like I, you, in some ways, and I feel the same way here, there's an obligation to actually tell people the facts and what's going on, right? And, and one of the, you know, downfalls or issues with journalism these days is this sort of social mediafication or this algorithmic, you know, content lens where you just tell people what they want to hear. And people say, yes, you know, validate me, tell me what I think is right versus actually provide me with the facts and, and tell me something

• 1:32:33Geoff Nielson?

  that, that's important that I'm educated on, you know, versus just, you know, the, the same old thing that I already know.

• 1:32:41Speaker 01

  Yeah, I mean, it's such a good point. I mean, it reminds me of my days as an international correspondent and sort of being, say, in Afghanistan and hearing the gripes of other correspondents. They say, you know, gosh, this is a whole war and people back at home are just not interested anymore. It's fallen off the news. And I always saw it differently. I always thought, no, no, my job is to make you care and to, I don't know if you call that entertain or engage you, but I was going to find some kind of angle that was going to make this feel relevant to you. So I do think that as voices, as, I feel like we do have a role, we do have a job to make this feel relevant.

• 1:33:23Speaker 01

  That probably means don't freak people out right away because, because then they'll just feel small as opposed to empowered. But I think that you can take that too far. And we are in a situation now where there's this incredibly important technology. It's complicated. The complexity is interesting and is, is maybe worth spending some time thinking about. But this is why it feels like you and I, and we need to find these new narratives, not just the sci fi narrative, not just the apocalyptic doom and gloom.

• 1:34:05Speaker 01

  I think Eliezer Yudkowsky is, I don't know if it's titling the book, if anyone builds it, everyone dies. It's blunt and maybe it gets more readers, but there's, there must be a role for sitting down with a listener or a reader and saying, okay, here are some incredibly fascinating things about AI that you didn't know it. Here's some ways they could go wrong. And here's ways in which the world might play out that might feel radically different than, than you may think it feels. Here's how your kids will fare. You know, it's like, we have to address the questions that people have and not just leave them with, you know.

• 1:34:47Speaker 01

  Leave them with a scare story. And so I struggle with it. And you hear me struggling with it in this answer because I do think our role as information gatherers is to package that information in a way that people will want to consume. I mean, we're not like professors here with a captured audience. People can choose what they want to tune into. So we have to sing and dance for our supper, as it were. But at the same time, it's the complexity of it and the fear, I think, that audiences have of that complexity. And I think even the fear that journalists have of the complexity. I would say that you're very much an exception to this, but you know, a fear of say, oh, gosh, I don't want to sound stupid because I'm talking about this stuff that's computer science related and I didn't really feel.

• 1:35:31Speaker 01

  I mean, you know, yes, there's a lot that makes us feel dumb when we're trying to understand how a, something like artificial intelligence works. But just being willing to ask those questions and being willing to dive into what does red teaming look like? What does safety training look like? What might a model take to be controlled? If we can get more people to have these conversations without feeling imperiled, you know, either physically or economically imperiled, that will be a win.

• 1:36:01Geoff Nielson?

  Right. I love that. It makes, it makes complete sense and it's a noble calling as well.

• 1:36:07Speaker 01

  Well, thanks so much for this encouragement. I really appreciate it, Jeff. I appreciate the work you're doing.

• 1:36:11Geoff Nielson?

  Absolutely. Keep, keep doing what you're doing and you know, we'll get, we'll get out there one person at a time.

• 1:36:15Speaker 01

  Absolutely, absolutely.

• 1:36:19Geoff Nielson?

  If you work in IT, Infotech Research Group is a name you need to know. No matter what your needs are, Infotech has you covered. AI strategy? Covered. Disaster recovery? Covered. Vendor negotiation? Covered. Infotech supports you with the best practice research and a team of analysts standing by, ready to help you tackle your toughest challenges. Check it out at the link below and don't forget to like and subscribe.