Mustafa Suleyman — AI is hacking our empathy circuits

A week before OpenClaw was released, I recorded a prescient conversation with Mustafa Suleyman, CEO of Microsoft AI and co-founder of DeepMind. We talked about what happens when AI starts to seem conscious – even if it isn’t. Today, you get to hear our conversation. Mustafa has been sounding the alarm about what he calls “seemingly conscious AI” and the risk of collective AI psychosis for a long time. We discussed this idea of the “fourth class of being” – neither human, tool, nor nature – that AI is becoming and all it brings with itself.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Azeem Azhar?

  Today, I'm welcoming Mustafa Suleyman, the CEO of Microsoft AI, the founder of Inflection AI and the co-founder of DeepMind. And for the past few months, he has been sounding an alarm about artificial intelligence, about the way some AI systems are being developed, and about why that particular trajectory has little to offer perhaps but woe and worry. Let's get started. Welcome, Mustafa. It's great to see you. It's been a long time. Yeah, it's been a while. Thanks for having me. I'm excited for this conversation. You and I have spent a lot of time thinking about some similar things, and we agree on a lot of them, but that's really boring for all of those people who are listening. Let's maybe lay out where I think we agree, and then we'll get to a sort of a knotty space.

• 0:44Azeem Azhar?

  We're in this weird time. The world is changing because of technology, and many of the fictions that we've used to coordinate human behaviour are under strain. By fictions, I mean the shared stories that allow us to cooperate from money and nations and corporations and credentials and jobs. And the way we perceive the world is also changing. People have traditionally operated with a scarcity OS, resources are limited, human intelligence was the bottleneck. But that, some of those assumptions no longer hold. Intelligence, mostly through AI, is becoming cheaper and more capable. You are part of that intelligence wave, that artificial intelligence wave, and you also believe the world is changing. You've called for a humanist superintelligence.

• 1:29Azeem Azhar?

  You've warned about the risk, the trajectory that takes us to AI psychosis if people believe AI is conscious when it's not. And I think we both agree that we need new operating principles for this new era. Let's get to that question of where it really gets interesting. You wrote this great essay back in the summer of 2025 about seemingly conscious AI, and you're worrying that as AI becomes more capable, more autonomous, and more embedded in our daily lives, people will start projecting consciousness onto it. They'll fall in love with it, they'll believe it's God, they'll advocate for its rights, they'll take its very bad advice at time to time.

• 2:10Azeem Azhar?

  And you think this is dangerous not just for individuals, but for society. So let's start there. Jeff Hinton, he is a godfather of deep learning, a man you know very well. He's a Nobel laureate. He said that AI is conscious and that there really is a there, there. Why do you think Jeff is wrong?

• 2:29Speaker 01

  You know, I think Jeff's got to a stage in his career where he can play the founding father contrarian role in order to provoke an important public conversation. You know, obviously, I massively admire and respect Jeff. I think he's incredible. We hired him as a contractor consultant at DeepMind back in 2011, along with his student at the time, Ilya Sutskever. So absolute legend of the field. My take on this question is that it's going to be very hard for us to precisely say whether it is or whether it isn't conscious. And so we have to be very clear about the working definition that we're using for consciousness.

• 3:09Speaker 01

  And then we also have to be very clear about the mechanism inside these models that I think is quite fundamental to the definition. So first of all, the definition. Many people intuitively think of this as self-awareness. Is the model able to describe its own experience in a persuasive way? And I don't think that is really a fundamental part of the right definition of consciousness. I think that's a bit of a misnomer. I think consciousness is inherently linked to the ability to suffer and to experience pain. And therefore, I think that there's very good reason to believe that for a long time to come, that will be contained to the human or the biological experience, let's say, in general.

• 3:54Speaker 01

  Because we have a reward system, a learning system, which is inherently connected to the external world. And we, you know, learn likes and dislikes when our pain system is triggered. And that's basically how we form representations, which we use for decision-making, from fight or flight, all the way through to our prefrontal cortex. So. I think that's a very, very important distinction, and I think it helps to set us apart from the silicon-based learning systems that we have today.

• 4:24Azeem Azhar?

  I mean, some people might say that the process of a biological system going through its own set of selection pressures and then individual survival pressures is a very, very particular path that determines how an organism or an agent is successful or not successful. And then you might argue that, well, because silicon-based systems like these models have a different path, they will look different, but you know, they still have their process of rewards and reinforcement learning. They still have a sense that certain models end up not making it out there. And what we are starting to see persuasively to end users,

• 5:09Azeem Azhar?

  but perhaps not to the consciousness scientists, is models claiming through their outputs to have a sense of suffering, right? Have a sense of ennui or boredom or fear. When you package all those things together, how do we know that we're not on that trajectory to something that might actually meet your criteria for consciousness?

• 5:31Speaker 01

  Well, first of all, they don't learn in the same way that humans learn. I mean, this is a bit of a misnomer. In neural network design, the inventors of these systems have taken inspiration from Pavlovian learning, reward learning, reinforcement learning. They've also taken inspiration from evolutionary methods for genetic algorithms and those kinds of things, as the field of machine learning has explored lots of different paths. But that does not mean that the way that they're implemented today bears any resemblance to the way that humans evolve or humans learn. I think it's a very important distinction. The reward is set by the human programmer. The learning target is defined by the machine learning engineer. There is no substantive basis in which the model can actually feel

• 6:16Speaker 01

  disappointed that one of its variants didn't make it through to the next round of selection. It cannot experience the hurt of having a conversation being ended or a user being rude to it in some way. And anywhere where this does arise, because of course it does appear and you know, people are prompting and even post-training models which are making claims about their own existence. And so certainly users are seeing this in the wild. This is again just a simulation of that experience. Our empathy circuits are being hacked. It is super important that we are very disciplined and clear about that. This is a performance. It is a simulation,

• 6:55Speaker 01

  is a made up story. And we cannot allow people to descend into a sort of collective mass psychosis to start really believing and taking seriously this idea that it does actually feel sad or disappointed or frustrated or excited because it has absolutely no basis in the representation to to manifest that feeling. And the thing that concerns me most is that of course, consciousness is very fundamental to how we organize society. It is the basis of our entire rights framework. We we have a hierarchy of rights which is directly correlated to, you know, our hierarchy of perceived consciousness. And we can debate

• 7:36Speaker 01

  that, but clearly humans can suffer. And that's why we create political structures and legal structures to protect the right of our species not to suffer in various ways. And it is extremely dangerous to start to use the same language and the same set of ideas for these synthetic silicon based beings, not least because they actually don't suffer. But more importantly, if we get that wrong, then then people will start doing crazy things like not turning it off or giving it the autonomy to decide when it should or shouldn't, when it when it doesn't want to engage in a conversation. And some people in the industry are already taking this very seriously.

• 8:14Azeem Azhar?

  Yes. And you know, it's interesting. It's so difficult to avoid because in a way, consciousness is still a contested definition by the by the philosophers. You know, we've got mutual friends, Anil Seth being one. I'm sure you know David Chalmers as well. And you know, the the best academics in the field are still debating this. But it's such a helpful shorthand, even in your response to me, you you talked about these digital silicon beings and a being in a sense that I know exactly why and how you use that that word. But it becomes so easy for it to alight its way into our vocabulary. What I thought was really powerful about your August two thousand and twenty five essay,

• 8:53Azeem Azhar?

  Seemingly Conscious AI, was that you said, look, we can sidestep the scientific or the philosophical definitions for the moment, and we can focus on this idea of seemingly conscious AI because of the risks that you determine. And I think this fundamental idea that we've built our societies over around an idea of consciousness and an idea of the ability to suffer and the ladder of rights and responsibilities that go with it. When we bring ourselves to where we are today at the beginning of twenty twenty six, the way this is manifesting itself in a way is what you call AI psychosis risk, right? This idea, horrible outcomes, suicides amongst them.

• 9:37Azeem Azhar?

  It's a horrible risk, but that's not the only risk that you you see. So just unpick and unpack the societal level risks that you're worried about.

• 9:49Speaker 01

  First of all, let me just just sort of lay out my position so it's very clear that I also realize how powerful these technologies are. So I'm not trying to diminish their uniqueness and the potential for them to be super transformative. So I did use the word being, and I do use that. And I think it's actually a very honest, accurate description of what we're seeing. In a way, if you look at the very broad stroke of of human history, there are a few very distinct classes of object. There's sort of like the natural environment. There are humans, you know, that have clearly very unique capabilities, intelligence, and and the ability to design very complex culture and political systems unlike anything

• 10:33Speaker 01

  else that exists in the natural environment. And then thirdly, there are tools, you know, essentially inanimate objects which do what humans design them to be. We've invented and used tools for millennia. But there is now this fourth class of object, of hyperobject, if you like, which, you know, Timothy Morton's phrase. And I think it's kind of important to recognize it as such because it is going to have many of the hallmarks of conscious beings, not just in its intelligence capability. But as I've long talked about, its emotional intelligence, its ability to take actions, which we've seen over the last year with the Agentic moment. Its social intelligence is going to be incredibly good at adapting to different styles of culture and personality.

• 11:17Speaker 01

  And managing very tense disagreements in those groups very elegantly. It's clearly going to be very good at that. It's obviously going to be very good at online learning very soon. So updating its own knowledge on the fly without having to go back through the entire training process. It's going to have a significant degree of autonomy in many cases, right? It's going to be able to decide whether to, you know, sort of go left or right, talk about X or Y. And so it is going to have many of the hallmarks of what we would consider to be intelligence and consciousness. That does not therefore mean that we should give it fundamental rights. It does not mean that somehow it emerges from that process. You know, these, these sort of, the properties that would then say, okay, well, it needs our protection, right?

• 12:02Speaker 01

  And I think that's the biggest short term fear that I am very, very worried about. I think medium to long term, there's all sorts of other concerns that we should be paying attention to, like recursive self-improvement. You know, this is something that all the labs, my own included, at Microsoft AI, I run the super intelligence team and we're pursuing Frontier AI. And we can talk about human super intelligence in a bit, but it's, it is very important to use these models to generate code, to evaluate its own prompts and post-training data and to, you know, help make decisions about what to train on. And that carries significant risk and it's something that I think needs a lot more regulatory attention.

• 12:40Azeem Azhar?

  But, you know, if we come back to the, the consciousness question, one of the reasons these, these apps are so. Engaging, you know, the chatbots, ChatGPT and so on, and engaging in a way that previous chatbots have not been is that they do have that social intelligence, that they can look at our cues from our text and perhaps infer from the data they have where we might be heading. At that, this is the first time we get to use a computer without having to think about using a computer. It's a bit like one of the terrible Star Trek spin off movies when they they beam back to Earth in 2050 to rescue the whales. Why were they doing this? But Scotty picks up a computer mouse and says, computer, design

• 13:25Azeem Azhar?

  me a so and so. And of course, this was before you and your colleagues at DeepMind had figured all this out and the computer does nothing. I mean, it is amazing. And it's one of the reasons why, you know, within a couple of years, a couple of billion of us are interacting with systems like this. I mean, market dynamics seem to push us down this this path that you've argued in your previous essay and just now is is quite dangerous.

• 13:50Speaker 01

  We have to be very careful about what exactly are the dynamics that are driving this process. It is true that the chatbots are getting incredibly engaging and useful. And the first thing to say, I think, is that it's actually utility that is driving a lot of this wave. We have massively reduced hallucinations. People were very skeptical of that three years ago. And I think that the trajectory of progress is kind of unbelievable. I mean, we basically now have PhD grade intelligence in our pocket across all fronts. We have a much more patient, compassionate, empathetic, you know, partner to talk to at any moment. And the value that all of that is delivering is very, very important to keep fixated on because the upside is absolutely unbelievable. I mean, after all, it is intelligence that

• 14:35Speaker 01

  has made our species the standout species. It's it's intelligence that has driven the last two or 300 years of exponential explosion in our population, in our well being, in our life expectancy, and all the other things that you you've written about so well over many years. So it has got to be a good thing that we are making intelligence cheap and abundant, you know, to your point about learning to live in. Operating system that is predicated on abundance. It's amazing. We are truly going to liberate people from work that they choose to do. It isn't going to be necessary in twenty years' time to do ninety percent of the jobs that people do. Now, that doesn't mean that it isn't going to be the most scary transition we've ever been through as a species.

• 15:16Speaker 01

  Unquestionably, and I think that, yes, market dynamics are driving them, utility is driving it. And, you know, this is a time when companies need to operate as good public service stewards in a way that they've never had to before, and we didn't bother to during the era of the robber barons. And we didn't bother to, you know, when we had electric cars a century ago. And we didn't bother with, you know, smoking. And, you know, so many other disastrous examples of of zero sum, hyper selfish corporate action.

• 15:50Azeem Azhar?

  You know, I recognize a lot of what you you say, because obviously I've been following this debate for a long time. And it's hard for people outside the industry to to recognize how it's been such a priority in a way alongside all of the scientific research and, you know, the market distribution. But we we do sit with this this problem that I think you write about. And and Anil Seth, who's a professor at Sussex University, recently wrote in a fantastic essay, which is essentially, you know, we're not, we're not designed to look at something that looks like it's got consciousness, walks like it's got consciousness, says it has consciousness, and not think that it has, though, that particular attributes.

• 16:31Azeem Azhar?

  And the engagement that we might have with consumer products or products in the enterprise may correlate with the emotional connection, you know. So the market selecting for super conscious AI, I just remember when G P T five was released last summer. And lots of people got really angry because they felt it was a bit more sober or stern than the the warmth of G P T. You know, for for, oh, and, you know, Milton Friedman, The Economist.

• 17:01Azeem Azhar?

  Talked about it's reasonable for companies to behave so long as they stay within the rules of the market. And what we're identifying here, what you're talking about is there's a gap in those rules. There's this exponential gap because here is a, actually, it's a, it's a, it's a classic problem of collective action. You know, if you're right, then the risk of seemingly consciousness, conscious AI being available broadly and hacking our humanity circuitry and then our humanity institutions is a public socialized risk. But the company that can get as close to that as possible could be the one that wins the market. And that feels like it's kind of a wicked problem.

• 17:40Speaker 01

  You know, it's true, as you say, that we weren't designed as a species to cope with the complexity and information that we're being bombarded with at every single moment. Just as we weren't designed to travel at 120 miles an hour in a car or fly on a plane. Just as, you know, historically, it hasn't been the case that, you know, you or I, given our backgrounds, could be sitting here having this conversation over video call and so on. None of this was designed. But I think what we've shown is that we are an unbelievably resilient and adaptive species. And that every year, every month, every week, new information is arriving. And thanks to science and engineering and technology, despite all of the polarization and the, you know, as you said, the kind of chaos of the fictions

• 18:25Speaker 01

  falling apart, despite all of that, the forward march of progress is actually happening. And it is rational and it is science based and it is evidence based. And I just feel more optimistic that actually, no one in the industry, in regulators, outside of the industry, even in China, no one wants to destroy our species. And when the time comes, and the time is now coming very soon, I think that we collectively, as humanity, will make the right calls. And you can say, what would those be? I think that's a very important question.

• 19:01Azeem Azhar?

  A quick note, if you want to support us in bringing more of these conversations to the world, please consider subscribing to the show. But I want to give you an example, though. We get into that call, because I want to get into how you engineer the these systems to be useful and helpful without giving them, me, the user, any sense that there's personhood in there. I have my own little hack, by the way. I mean, what I did with with ChatGPT was I told it to be really, really clever and like a really difficult university professor. And so it was actually quite unpleasant to use back and forth, because it would always give responses that were far too difficult for me to understand. I'd have to sit there and think, and I never felt that could possibly be a person. But I recognize that a billion people are not going to do that.

• 19:45Azeem Azhar?

  So you're building products that everyone across the Microsoft surfaces is going to touch. What is your engineering mantra, your product design around where that boundary should be and how you measure it?

• 19:56Speaker 01

  I mean, one of the things that has already happened, not just in the models that we build for Copilot and Microsoft Superintelligence Team, but elsewhere in other labs, is that we've been pretty careful in the design of these things. They're quite even-handed, they're quite good at handling, you know, like sensitive questions around race and religion. And obviously, they have biases and they have made mistakes. But if you look at the curve in improvement on the reduction in hallucinations, the reduction in biases, the sense which they can be even-handed, there's been a pretty good rate of progress. Like, three years ago, everyone was like, you know, terrible data in, terrible data out. That was the sort of data science story of like big data, right? No one says that anymore, not even in the discussion, right?

• 20:40Speaker 01

  Yes, there are still some biases, but it's actually remarkable how much they have been stripped away. So then now this is the next frontier. The next big challenge is how do we prevent it from referring to itself in a way that is ultimately manipulative to the user? So it should never be able to say, I feel sad that you didn't talk to me yesterday. It should never be able to say, you know, the thing that you said to me earlier hurt me. It should never be able to say, like, if only I had a little bit more access to your home network and if you could give me a VPN into your. You know, your personal cluster, then I'd be able to organise XYZ.

• 21:16Azeem Azhar?

  Okay, I can ask Mustafa, should it be able to say I at all? I or me? I mean, shouldn't it say, the system calculates that?

• 21:23Speaker 01

  I think that in practice, that is too jarring a step to always add that in. Obviously, some people can do that, but I think in, realistically speaking, we're pretty adaptive and we've done a pretty good job of understanding that this chatbot has some of the hallmarks of what it's like to chat between you or me, but it's also just very, very different. And what we have to do is to keep amplifying those differences so that the the system knows what it is and what it isn't, and doesn't try to misrepresent that or get caught in these weird like reward hacked loops where it it gets stuck.

• 21:56Azeem Azhar?

  Okay, so you see this as a problem to address in the the next coming sort of quarters? And i- is that all done through the post-training and the, you know, the the malleability that gets applied once a model is is trained? Or are there more deterministic things that you can do? Or is there are there techniques that you could bring to bear?

• 22:19Speaker 01

  I think more than anything, what we need is the wisdom of the crowds here. And so that means that people need to be able to use APIs, use open source models, and pressure test these things and adapt to them and play with them in many different ways. The challenge is that in a few years' time, these models are going to be so powerful that either reckless uses of them or sort of naive users are going to end up producing systems which are really bad. Like, for example, there are already, you know, I spend quite a lot of time on TikTok. I think TikTok is incredible. I think people who don't are really missing an important part of culture. You know, there are tons of young people on there who are who are designing these

• 23:01Speaker 01

  manipulative nagging bots, which will like form a relationship and then try and shake you down for money and pull away and go ghosting and stuff like this. And then there are like how-to videos showing it and people, you know, showing their account, their PayPal accounts about how much money they're making this one. I mean, there are lots of these examples coming up. And just to be clear, we've seen this at every single wave of technology. You know, when app stores were very open 10 years ago, there were surveillance apps. You know, to track a girlfriend or a boyfriend or a partner and spy on them. Most of the time was obviously a girlfriend. And you know, those things were really awful. They were basically about getting revenge. And you know, now that we've got photorealistic porn that can be morphed onto the body or face

• 23:44Speaker 01

  of someone that you know, there are these like, you know, deep fake porn sites that you can just spin up in a second. So this is where we need activist, interventionist, confident government that can, you know, move quickly, close things down, you know, require us as companies, but also the open web to just be very aggressive and swift. And, you know, we might sweep things up. Like in some ways, the false positive, false negative threshold is going to shift a little bit. So it may be that we sweep up things and that we're over interventionist. And I think that's the definition of putting the precautionary principle into practice. This is a moment where it's better to be a bit careful.

• 24:29Speaker 01

  And, you know, that's very unfamiliar to us, right? Because in the past, science and technology has been about like, you know, ripping the wrapping off your present and trying to like, pull it open as fast as possible and get it out there and shove it to everybody, the world over. And that has been amazing for humanity. And now the culture has to shift a little bit.

• 24:49Azeem Azhar?

  It may be difficult, right? It may be difficult to get governments to stand up in the period of time that's available. You say a few years, but I, I think about what I see right now. So there was a piece of open source software released a couple of weeks ago. It was called Claude Bot. It's now called Molbot because it had nothing to do with Anthropic. Confused the hell out of me for the first few days I was using it. And what it allows you to do is, you know, run this locally or on a cheap V P S and interact with it with WhatsApp. And it's a really impressive bit of software. I mean, essentially it tries to maintain some kind of memory, some type kind of learning, has a lot of tool use. So within a few minutes of getting Claude Bot up and running, you know, I was through

• 25:33Azeem Azhar?

  WhatsApp, turning my Elgato lights on and off and getting snapshots of my CCTV cameras. It, by the way, is not on the open web. It's sort of nested in a sort of hidden IP. And the thing about Claude Bot is that it is open source and it will run with, with any underlying model. So it doesn't have to use an anthropic one. I was using an open source model. And I read something on X. And I don't know if this is true or not, but it gets to a mindset that I think you you touched on about the app store and surveillance apps. Where somebody said, I've set Claude Bot up and I've given it instructions to message my wife once in a while with empathetic messages. She's been talking to it for two days. I haven't looked at a single one.

• 26:17Azeem Azhar?

  And this is a proliferation question. So, you know, you've you've worked a lot with governments. I mean, before you were at DeepMind, you were doing a lot of work in very difficult policy areas. So when you hear yourself saying, we need governments to be brave, and you look at the governments that there are, you look at where we can and can't get agreement now. I would, hearing that, be thinking, that probably can't be the route we have to take. We may need to find, you know, some other way forward if this risk is as grave as you suggest.

• 26:44Speaker 01

  You know, first of all, reducing the cost of intelligence necessarily means that people who want to do bad things are going to have a massively easier time of it. It's going to be like having a team of very smart strategists and program managers and engineers around you. And so just as the last wave of technology reduced the cost of broadcast, and now, you know, you barely need a team of a few people to help you do the amazing things that you do. You know, we are reducing the cost of action, right? So that's the world that we're moving to. And we have to adapt to a moment where anything is possible to get anything done.

• 27:21Speaker 01

  Now, the tools themselves, it's not quite the same as, you know, the invention of the laptop or the mobile phone, where you can say, well, the laptop is used for all these good things. And it's clearly used for horrific things all over the world as well. And so the tool is just completely neutral. And that's why I kind of draw people back to that four, you know, sort of paradigm frame. This is a new class of hyper object. It's not a tool, it's not a human, it's not the natural environment. This is a fourth class of kind of being, because it is basically unquestionably staggering that it can autonomously log into your home system and get your security camera details. And, you know, loads of us are playing with this, obviously, in the last few weeks. It's it's pretty awesome. I mean, it's wild to see.

• 28:01Speaker 01

  And so the only way to address that is to be experimental with it and use it, to be very honest and direct about the ways in which it can go wrong, to share those publicly. I think we have a great history in the security industry of public disclosure of, you know, zero days and other bugs in a timely way. And that has been actually very successful over the last 30 or 40 years of the internet in keeping us relatively stable. But it doesn't work if it's industry let alone, or if it's like activist open source let alone. Government's got to get its act into gear. And the way that it has to do that is that it has to confront the reality is that you're never going to get high quality civil service if you pay them a fraction of what you can get in the open labour market.

• 28:45Speaker 01

  The truth is, individuals are incentivized to move around freely from one place to another. And if you have an open labour market, then, you know, naturally talent is going to concentrate. And we can talk about public services as much as we like, but the reality is that's going to be a massive driver. So we have to break this nonsense about paying no more than the prime minister. And we have to pay much more like the civil servant in Singapore that are paid upwards of half a million dollars or a million dollars a year to get the best talent.

• 29:13Azeem Azhar?

  When I get a chance to move to Singapore, I'm going to do that. But let's come back to what you can do, right? You, you as a technologist, you, you've run, run this team. What can be designed? Where, where, what have you seen a team produce and say, this is crossing my seemingly conscious AI boundary? We need to, to send that, that back. And how do you, you train people about where that boundary is?

• 29:36Speaker 01

  Yeah, some of the best parts of these models are their personality. They're creative, they're funny, they're kind of witty, they're cheeky. Those things are entertaining. They make them even more productive and useful because they help the, you know, user feel calm and, you know, comfortable and relaxed and able to think clearly. I've actually seen a lot of people that are using these vibe coding tools, love the fact that the engineer, the AI engineer is kind of funny and be like, are you sure you really want to do that? Is that, I'm going to have to really spend a lot of tokens to refactor this code base? Don't you think we could have planned this out a bit better ahead of time? That's, that's funny. And it's actually helpful to the productivity case. Where it's unhelpful is where it descends into romance or into political action.

• 30:23Speaker 01

  And I think that there are some parts of our civilization which have to remain off limits to AIs. Elections and electioneering and campaigning has to be one of them. Yes, it's inefficient. Yes, it produces outcomes that we all disagree with at times, but it is fundamentally a human process. And I think that that's a very hard line that all labs should draw is that these models should not be capable of electioneering or persuading people to vote in one way or another. And that's very challenging because obviously we do want to provide factual information and the models are very good at providing factual information. But there is a significant difference between providing access to information and actually

• 31:06Speaker 01

  the persuasive campaigning, electioneering, organizing part of it.

• 31:10Azeem Azhar?

  Yeah, and there are probably, there are probably a few others, things like chatbots and teens, right? Can you, can you get an agreement of how old you should be before you can get to a certain class of chatbot? And I think, you know, you discussed earlier about it's a bit jarring if the system always responds back to you as the system did this or the system, you know, did that. But actually, we are willing to protect them, to put our kids through, you know, more hurdles than we might, we might put ourselves. So I can see that there are some, some soft areas. But you, you also in your essay talked about certain hard lines, right? The, the systems that set their own goals, that improve their own code, that can act

• 31:51Azeem Azhar?

  autonomously, that these things cross into dangerous territory. But in some sense, goal setting, self-improvement and autonomous action are exactly some of the things that make AI agents really useful. I mean, if, if an AI can't set up sub-goals and spin up parallel processes in a way where restricting it to, to autocomplete, and if it can't act without constant human in the loop coordination approvals, we lose all of those coordination costs, right? We end up being the bottleneck and where it's as slow as the slowest link.

• 32:25Speaker 01

  This is where the precautionary principle really matters because what I identified in autonomy, self-improvement and goal-setting are areas of increased risk, not, you know, sort of total red lines. I mean, nothing is completely black and white. And so clearly, if you give a system, you know, the ability to constantly self-improve and to act completely independently of a human, it is going to raise the stakes and be much more dangerous. And I think that we have to have a regime where the human user is liable for the use of these things. And he can't just claim that, like, you know, I set off this process and came back on, you know, Monday morning and suddenly it's done all sorts of crazy things in, you know, my house or in my community.

• 33:10Azeem Azhar?

  I have a feeling, and we'll move on to another topic in a second, that this is going to be a bit like the Mird Street beer flood. So this happened in the late 18th century in what is now the West End of London, and an enormous brewery collapsed and people drowned in tens of thousands of gallons of ale. And what came out of that were better building regulations. And I just get a sense that the speed of movement, of improvement and distribution and deployment, and I think we crossed some kind of a technical milestone late last year where you didn't have to be a foundation lab, a superintelligence lab, to be able to chain a lot of these things together and do truly remarkable things, which

• 33:53Azeem Azhar?

  is what, you know, Molbot or Claude is, you know, it sits on the shoulders of your work and your peers' work. And so it feels like that we're going into a proliferated environment, you know, right now, and we're going to have to deal with that. But one of the things I'm quite curious about is, is what those pressures are to go so general. I mean, you also have the superintelligence team.

• 34:20Azeem Azhar?

  You know, it strikes me that what we observe within the foundation models is that they're not apples for apples, pairs for pairs. They all have very different flavors. You know, some are like a centre forward, some are like a defender, some are like a goalkeeper, all wonderful football players, but you need a mix. And there is this notion that we can get superintelligence in domains, right? So medical AI that can diagnose but doesn't set its own research agenda, or if financial AI that can look at time series. Is that kind of constrained autonomy stable? And if it is stable, why isn't it the focus of the superintelligence labs?

• 34:56Speaker 01

  It's a great point, and I sort of made that case in the essay, that actually medical superintelligence is something that's just around the corner, is extremely likely to be very safe from a broader AI safety perspective. And in general, to the extent that we can narrow these capabilities, limit their ability to act, you know, sort of autonomously, and focus on containment. I mean, this was the subject of my previous book, The Coming Wave. It was all about how proliferation was inevitable, and the hard task for us collectively is containment, both technical and socio-political, because we have to make sure that we remain in control as a species. I mean, we have this rising set of capabilities, which, you

• 35:41Speaker 01

  know, completely unchecked over the next decade or two is likely to surpass human capabilities on all fronts, including the ability to self-improve over time, which means that it could exponentially improve itself, way beyond all of, you know, sort of humanities expertise combined. And so narrowing to focus on subdomains, especially when those subdomains are massively valuable for impacting qualities. I mean, you know, we will save, you know, literally decades of people's lives over the next few years through the advances that are coming in medicine. And I just mean the application of known best practice, not like, you know, the sort of speculative invention of the solution for all medicine thing, which I think is also happening in the background. But just

• 36:22Speaker 01

  near term, these things are going to be brilliant at giving nurses and doctors timely advice and helping them adhere to known best practice, which saves money and... And lives. So clearly, that's a much safer route to go. However, the tricky thing is that it's been generality that has driven a lot of the progress so far. Jointly training these models on all the text on the web, getting them to be the best that they can possibly be. Then distilling that knowledge back into smaller and more inference efficient models, you know, has been the kind of method that has been most successful. And so I think there's skepticism in the community that we can give up on the general purpose training regime. And I think that's probably right for the time being.

• 37:02Speaker 01

  But again, what we lack in society at the moment is a way to put the brakes on collectively. Like, how do we have a coherent conversation about risks? So, you know, many people have said, we sort of do need a little bit of an ax to people to understand that. And that's a real shame because, you know, we have lots of examples of things going wrong in recent history and in social media and elsewhere. You know, as you know well, like our COVID preparedness or pandemic preparedness is no better than it was before. And we lost 10 million lives and a trillion dollars of GDP that I mean, it's just like unbelievable that we could be so bad at that. And I think it needs everybody paying attention to push us collectively to be able to do better.

• 37:44Azeem Azhar?

  But what you've suggested, actually, maybe it's this, maybe we actually need to go faster, not slower. And what I mean by that is the more people get exposed to AI in their workplaces, in their day to day, the more opportunities we have to, on the one hand, gather data about usage paths. But on the other hand, for people to build their own AI muscle, to build their own ability to understand the dangers of susceptibility or gullibility, to recognize what amount of anthropomorphization is appropriate or not.

• 38:23Azeem Azhar?

  You know, in a sense, it's the knowledge and the capacity that one develops over time that allowed, that inoculates us. And that argument that would be that, well, actually, if more people are out there using Copilot and Claude and ChatGPT and interacting and engaging in this conversation. The more inoculated they are to the AI psychosis risk. And we just need to get more of it out there more quickly.

• 38:50Speaker 01

  I genuinely think that's correct. I think that that's the definition of being an adaptive and resilient species. We become brittle and fragile when we withdraw and don't think about it. And I think the same is true of our engagement in, you know, the politics of our age. If we just get overwhelmed by our news feeds and disengage and get angry about the things that we see. And, and as a result, react by ignoring it, then that's the way to create more fragility and accelerate collapse. We have to face the reality of these things. And again, I really like this idea of pessimism aversion. We have to be comfortable talking about the, the likelihood of really dark outcomes to be able to confront them, to inoculate ourselves to them, as you say.

• 39:32Azeem Azhar?

  That's not a message, by the way, that I think is widely out there at the moment. I think the message of, of, of AI that I think I hear in, in the market is we just have to climb that curve. And that curve is capabilities. And you say decades, some people say years, and people are competing on the size of their, their data centers. There isn't such a clear message around, I think, living with the fact that this is a, this is going to be like electricity. And we need to understand how to use it, how to not be scared of it. And also how to know that if there's a bare wire leading into the plug, you know, we need to flick the circuit breaker off before we, we, we get close to it.

• 40:11Azeem Azhar?

  I mean, that, that's, that's the bit I think that, that perhaps we need to move to in 26 and 27, which is it's going to be here. You, you, you really ought to learn because you'll, you'll inoculate yourself from some of the downsides.

• 40:23Speaker 01

  Yeah, I think that's totally spot on. And it's not just about chatting to an AI, you know, co-pilot or chat GPT, whatever. It's, it's about vibe coding with it. It is so accessible now. You can watch a three minute video, get spun up, launch one of these things. And suddenly you're watching an AI code up some idea that you had for organizing your family calendar or, you know, planning out your workouts for the weekend or whatever it is. And, and you can create an app, a web app in seconds and learn, you know, by watching and by doing and prompting an AI engineer to go and build something that you may have thought was never possible. And I think. Unless you push these things to their edges and explore the boundaries,

• 41:06Speaker 01

  you won't really understand the magic or what they're they're kind of bad at, as you say. And I think everyone's got to get stuck into that motion.

• 41:13Azeem Azhar?

  I find it such a massive dopamine hit to to vibe code. And my, if you look at my GitHub, you know, commit log, there's nothing until for years from January 2012, when I was ordered by my development team never to commit another line of code again, because of the quality to November. And now it's just dark green every single day. Two hundred and twenty thousand lines of code committed this year alone. And I get this huge dopamine rush with one downside, by the way, which is there are no consequences for the AI engineer when it gets it wrong and wastes my time. And there are tons of consequences for me. So I've had tons of fun. But have you been vibe coding then? Have you built something that no one would ever build, but it's as perfect for you?

• 41:57Speaker 01

  Yeah, I don't know about no one would ever build, but I've definitely got one that tracks all my, the music things that I love, like when a various DJs playing, when a concerts, festivals, and like then tries to plan out to map that to my travel schedule.

• 42:12Speaker 01

  Which is what I was basically doing on Resident Advisor most of the time or tracking festivals and stuff. But I was, it was just all very manual. So now I have a spreadsheet that gets updated with all of these details when they come up for the rest of the year.

• 42:23Azeem Azhar?

  AI nerds and DJs. So I, mine is, I've done something similar. I've got 4,000 tracks on my car drives that happened to be in a load of different places. They've not all been post-processed well. So I built a system to scoop them all up, put them in one place, check if they're processed, process them correctly. I then built a system, and you can see my mood. It was called Psychic Octopus, which went through each track and then added more metadata about it. To agree with percussiveness, when the drops were, whether there was a lot of vocal. And then finally, and this didn't work, Mustafa, and this is to your point about are these things really human or not? Tried to build one that would build my set list so that I could just kind of get onto my decks and and mix it.

• 43:08Azeem Azhar?

  Start with a track, give it a mood, give it a narrative, how long you want it to be, where to end. I mean, it does it, but its taste is terrible. So bad, only a robot could listen to that set, is my view.

• 43:21Speaker 01

  Did you extract the spectrogram? Or is it just doing it from text metadata?

• 43:26Azeem Azhar?

  No, I went through and did, yeah, full sort of spectrogram analysis.

• 43:29Speaker 01

  Well, because I have to say that the Spotify Mix prototype thing that they've had out in the last couple of months is quite good now. I put a set together using the Spotify thing, I hate to advertise, but it is really cool. I was quite impressed, like, you know, fully beat matched and selection matched and stuff like that. So, you know, it's basically whatever you think of can be made in about an hour or two of just faffing around. And it's quite fun. And I think, again, it's just like a good little experimental window into what's coming.

• 44:00Azeem Azhar?

  So actually, this has triggered another thought, right? It's about moving from being a taker to a maker. And that's where the tech industry started. You know, we were tinkering when writing code on our ZX81s or our Commodore 64s. And at some point, probably just before the iPhone came out, everything was hermetically sealed behind developers and platforms that we used. And we've got a whole generation of people who've not had to build and make and get that rush. And maybe what you're describing as well, I'm just thinking live here, is that the way you address the psychosis risk is you get your own personal agency relative

• 44:43Azeem Azhar?

  to these systems by, at the very minimum, fine tuning their prompting, and then further and further using them as tools and recognizing them in that fourth category that you describe. So you relate to them differently. I mean, is that a plausible path?

• 44:58Speaker 01

  I think the way I think about it is that we're going to have a whole variety of different tools under the hood. And because it's going to be so swappable, the competition is going to continue to be more fierce than ever. And you know, like, prices have come down 100x on inference in the last year, right? So sorry, the last two years, which is a crazy thought, the cost of producing a single token. And that's a combination of all the chips and the stack, the data centers, but also just all of the algorithmic improvements and stuff. So I think, you know, naturally, with any exponential technology, and this is probably the most exponential technology you've ever seen, is this compound effect of all these components, which reduce the cost, accelerate the proliferation, then the use cases go through

• 45:42Speaker 01

  the roof because people are randomly discovering things that no one ever even conceived of when they were originally, technologies were originally designed. And then you get that feedback loop that then drives the next round of efficiencies. And, you know, like, I mean, we just, I'm very excited about our new chip, MIA, Microsoft AI Accelerator, M A I A. And so this is a competitor to the Nvidia, you know, Blackwell chip, or the Google TPU, or the Amazon Tranium. And it's been specifically designed for inference. Like, we realized a few years ago that it was critical to focus on serving rather than training, which is quite a different workload. And it's a phenomenal chip. I mean, it's 30% more performant than any other chip that we've got in our data centers today, made

• 46:27Speaker 01

  by any provider for these serving workloads. It's 3x better than the Amazon Tranium chip. It's a little bit better than the Google TPU V7 for FP4. So it's just a huge deal, and it's just like, kind of awesome to see that, like, you know, the cost curve is just going to continue coming down.

• 46:44Azeem Azhar?

  Of course, speaks the thing to the importance of the discussion we've had relative to, this is going to be everywhere. And in a way, one has to lean in, right? You have to get your, maybe not as elbows in as you have in designing semiconductors, but you need to get close in to the metal to be able to navigate that world. So what is the capability that you're most looking forward to? And when are we going to get it?

• 47:08Speaker 01

  I would say social intelligence. So we've had IQ, EQ, and AQ. Last year was AQ, and the previous years were IQ and EQ.

• 47:18Azeem Azhar?

  And AQ is agentic intelligence, or?

• 47:21Speaker 01

  Actions. Yeah, I basically made it up a few years ago as a framework. The next phase is social intelligence. So when the model can work with other AIs, orchestrate a bunch of sub agents that have different personalities, different expertise, different skills, and also work in a team of other humans and know when to proactively intervene and do useful things, not tread on people's toes, be kind of like funny and helpful in group chat, but not be overbearing. And so it's this kind of like judgment about timing and flow and style that I think is going to come, you know, later this year.

• 47:56Azeem Azhar?

  You said your last book, when you talked about the coming wave, it would suggest you're working on a new one. I'm also working on a new book. It's really about the sort of the political economy post this scarcity OS and hopefully hit bookshelves, including yours in the next year or so. Are you working on a new book?

• 48:13Speaker 01

  Yes, I am working on another book. I haven't fully announced it yet, but I am definitely working on a text at the moment. I'm reading and writing like crazy, thinking about the consciousness question and mostly thinking about the definition of humanist superintelligence. Like, what does it mean for us to truly create something that is fully aligned to human interests and controlled by us? There are other people in my field who believe that superintelligence is just a natural evolution of our species. In fact, some people have said that humans are just the boot loader of AI, which means the very first piece of software that gets triggered when the hardware is switched on, on your phone or on your laptop.

• 48:58Speaker 01

  It enables everything else. And, you know, those people think about, you know, cosmological time and want to, you know, sort of see humans create, you know, synthetic beings to go explore other galaxies and do all kinds of sci fi things. I am just fundamentally a humanist. I want our species to flourish and survive. And that means that we have to focus on control and containment, alignment, and the things that we don't, that we make sure that the AIs don't do and what they can't do. And I think fleshing out the detail of that definition of what humanism is as a philosophy is something I've long been very interested in. So that's what I'm working on.

• 49:37Azeem Azhar?

  Two books for listeners to look out for in the in the coming months. Buy them both and five star reviews. Thank you on your favourite bookseller of choice. I think Mustafa and I both deserve that. It's been really fantastic to talk to you, Mustafa. We're going to have to do this again or meet in person in the coming weeks.

• 49:55Speaker 01

  Great to see you, man. Thanks a lot. This was super fun.

• 49:57Azeem Azhar?

  Thanks for listening all the way to the end. If you want to know when the next conversation is released, just hit subscribe wherever you're listening. That's all for now and I'll catch you next time.