AI's Rising Risks: Hacking, Virology, Loss of Control — With Dan Hendrycks

Dan Hendrycks is the Director and co-founder of the Center for AI Safety, and an advisor to Scale AI and xAI. He joins Big Technology Podcast for a discussion of AI's growing risk profile, and what to do about it. Tune in to hear Hendricks explain why virology expertise in AI models is an immediate concern and how these systems might soon enable devastating hacks. We also cover intelligence explosion scenarios, the geopolitical implications of AI development, and why an international AI arms race could lead to faster development than the world can handle. Hit play for an insider's perspective on how governments and AI labs are wrestling with unprecedented technological power that could reshape global security. Connect with Dan: https://www.nationalsecurity.ai/ https://x.com/DanHendrycks Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  Now that artificial intelligence has tried to break out of a training environment, cheat at chess, and deceive safety evaluators, is it finally time to start worrying about the risks that artificial intelligence poses to us all? Here to speak with us about it is Dan Hendricks. He's the Director of the Center for AI. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis Chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then, I sat down with Adam Mitchell, Head of Enterprise Business Solutions at Voya Financial,

• 0:42Alex Kantrowitz?

  to go inside a large-scale Agentic CX transformation, including the business decisions, trade-offs, and lessons along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get Frontier Compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation.

• 1:21Alex Kantrowitz?

  AI policy and also an advisor to Elon Musk's XAI and Scale AI. Dan, it's so great to see you. Welcome to the show. Glad to be here. It's an opportune moment to have you on the show because I'm recently doom-curious, and I'll explain what that means. So I had long been skeptical of this idea that AI could potentially break out of its training set or out of the computers and start to potentially even harm humans. I still think I'm on that path, but I'm starting to question it. We've recently seen research of AI starting to try to export its weights and scenarios where it thinks it might be rewritten, trying to fool evaluators and even trying to break a game of chess

• 2:06Alex Kantrowitz?

  by rewriting the rules because it's so interested in winning the game. So I'm just going to put this to you right away. Is what I'm seeing in these early moments of AIs trying to deceive evaluators or trying to change the rules that it's been given, is that the early signs of us having AI as an adversary and not as a front?

• 2:29Speaker 01

  I think the easier way to see that it could be adversarial is just if people maliciously use these AI systems against us. So if we have an adversarial state trying to weaponize it against us, that's an easier way in which it could cause a lot of damage to us. Now, there is a, an additional risk that the AI itself could have an adversarial relation to us and be a threat in itself, not just the threat of humans in the forms of terrorists or in the forms of state actors, but the AIs themselves potentially working against us. I think those risks would potentially grow in time. I don't think they're as substantial now compared to just the malicious use sorts

• 3:14Speaker 01

  of risks. But yeah, I think that as time goes on and as they're more capable, if, if some small fraction of them do decide to deceive us or try to self-exfiltrate themselves or develop an adversarial posture toward us, then that could be extraordinarily dangerous. So it depends. So I want to distinguish between what things are particularly concerning in the next year versus somewhat more in the future. And I think in the shorter term, it is more of this malicious, but that's not to downplay the fact that AIs could be threats later on.

• 3:46Alex Kantrowitz?

  Of course, and we're going to cover the short term and the immediate term as we go on through this conversation, but I don't want to lose this thread here because what I'm trying to figure out is what, is what I've seen with, again, AI trying to rewrite the rules of chess, AI trying to exfiltrate its weights or copy itself. Now, it runs a code saying, if I run this code, I copy myself to another server. It was a fake code. It was tricked by its evaluators into thinking it could do that, but it did run the code. And then also manipulating, showing evaluators that it was actually modeling one behavior, whereas when I thought they weren't watching, it was doing something else. This is all stuff that's happened. Are these, are these the early signs of what could go wrong with AI? Or, or is this just benign activity that we shouldn't read too much into?

• 4:30Speaker 01

  So I, I think that this, it is suggestive of some loss of control scenarios. However, it is not the type of thing that I'm most concerned about with loss of control. So that's because we still can control these AI systems somewhat reasonably right now. And maybe we'll get better methods for doing so. However, The loss of control mechanism that I'm most concerned about is when we have automated AI research and development. So imagine at some point in the future, it could do what AI researchers do. And if you have one AI that can do that, and that would be involve automating a lot of software engineering, if you could have one AI do that, then you could make, you

• 5:15Speaker 01

  know, 100,000 copies of these and have them perform research simultaneously. So that could lead to some very substantial acceleration in the rate of development. You might get a decade's worth of AI developments within the course of a year. And this would be highly automated, where there's very little human oversight. And I think in those situations, or in that scenario, I think a loss of control is much more likely. The sort of AIs that we have right now, kind of, you know, being a little nefarious here and there, that is a concern, but that seems more tractable as a thing to research and improve and reduce the risks of.

• 5:58Speaker 01

  Meanwhile, some automated AI research and development loop that goes extremely quickly with very little human oversight, that seems hard to de-risk and get the risks to be at a negligible level, just because it's so fast and so, and there's so little human involvement. You couldn't involve humans that much because of competitive pressures. The larger geopolitics of this would make it harder to, you know, slow down in such a scenario. So I think that is a, that's more the type of loss of control risk that I'm concerned about. And I think that those other papers are suggestive, but there's a lot more hope for it being empirically tractable to counteract it.

• 6:38Alex Kantrowitz?

  Now, from what I understand it from your first answer, you are concerned both the way that humans use AI and AI itself, sort of taking its own actions, loss, our loss of control of artificial intelligence. So can you just rank sort of where you see the problems in terms of most serious to least serious and what we should be focusing on?

• 7:02Speaker 01

  Um, good. That's a really good question. So I think the the risks and their severity sort of depend on time. Um, some become much more severe later. So I don't think AI poses a risk of extinction like today. Okay, I don't think that they're powerful enough to do that. They because they they can't make PowerPoints yet, right? They they don't have agential skills. They can't accomplish tasks that require many hours to complete. And so as as since they lack that, this puts a severe limit on the amount of damage that they could do

• 7:41Speaker 01

  or their ability to operate autonomously. So I think there's a variety of risks. I think there's in malicious use in the shorter term. When AIs get more agential, I'd be concerned about AIs causing cyber attacks on critical infrastructure, possibly by as directed by a rogue actor. There'd also be the risk of AIs facilitating the development of bioweapons, in particular pandemic-causing ones, not smaller-scale ones like Anthrax. And those are, I think, the two malicious use risks that we'll need to be getting on top of in

• 8:26Speaker 01

  the next year or two. At the same time, there's loss of control risks, which I think primarily stem from people and AI company trying to automate all of AI research and development. And they can't have humans check in on that process because that would slow them down too much. If you have a human do a week review every month of what's been going on and trying to interpret what's happening, this is this would slow them down substantially. And the competitor that doesn't do that will end up getting ahead. What that would mean is that you'd have basically AI development going very rapidly where there's nobody really checking what's going on or hardly checking. And I think a loss of control in that scenario

• 9:10Speaker 01

  is is more likely. So but that comes a bit later. So it it it depends. And it'll depend on what we do. So these risks aren't aren't.

• 9:23Speaker 01

  These risks aren't something that exists out there and are immutable. Like, maybe we can do more research to make the malicious use risks go down substantially. Maybe there can be, um, maybe states can deter each other from pursuing this automated AI R and D loop, um, so that we don't have this risk of loss of control. So it, it depends not just on technical research, it depends on the politics and geopolitics as well, and those will keep changing, and so the risk sources will keep changing.

• 9:56Alex Kantrowitz?

  Right, and with the Center for AI Safety, we're going to talk today about risks, but we're also going to talk about solutions. And with the Center for AI Safety, what you're doing is basically pointing out the risks and trying to get to solutions to these problems. You told me you were just at the White House yesterday, the day before we were talking. So, uh, this stuff is something that you're actually working towards mitigating, and I think we're going to get to that in a bit. But first, let's talk a little bit through some of the risks that you see with AI and how serious they actually are.

• 10:21Speaker 01

  Mm-hmm.

• 10:22Alex Kantrowitz?

  Uh, one of them that just jumped out for me right away was bio.

• 10:26Speaker 01

  Mm-hmm.

• 10:27Alex Kantrowitz?

  Creating bioweapons. Uh, let me run you through what I think the scenario could be in my head, and you tell me what I'm missing. With bioweapons, uh, you'd basically be prompting an LLM to help you come up with, uh, new biological agents effectively that you could go unleash against an enemy. And I think, wouldn't that be predicated on the AI actually being able to come up with biological discoveries of its own? Uh, right now, current LLMs, they don't really extend beyond the training set. Maybe there's an emergent property here or there, but they haven't made any discoveries, and it's sort of been the big knock on them to this point. So I am curious, if you're talking about immediate risks and one of them being, okay,

• 11:12Alex Kantrowitz?

  there could be bioweapons that are created with AI, um, doesn't that suppose that there's going to be something, uh, much more advanced than the LLMs that we have today? Because with current LLMs... To me, it's basically like Google. It's a search for what's on the web and it can produce what's on the web. Uh, but it's not coming up with new uh compounds on its own.

• 11:34Speaker 01

  Yeah, so um I I think that uh for cyber, that's more in the future. But I think uh virology, expert level virology capabilities um are much more plausible in the short term. So uh for instance, we have a paper that'll be out um maybe in some months. We'll we'll see, um but most of the work's before it's been done. And in it, we we have um Harvard and MIT expert level virologists sort of taking pictures of themselves in the wet lab. Um and asking, what steps should I do next? So can the AI, um given this image and given this background context, help guide through step by step these various wet lab procedures in making viruses and manipulating their properties?

• 12:19Speaker 01

  And um we are finding that with the most recent reasoning models, um quite unlike the models from two years ago, like the initial GPT four, the most recent reasoning models are getting around ninetieth percentile compared to these um expert level virologists in their area of expertise. Uh, so um uh this suggests uh that they have some of these wet lab type of skills. And so if they can guide somebody through it step by step, that could be um that could be uh very dangerous. Now, there is an ideation step, um uh but that seems like a capability, them doing brainstorming to come up with ways to make viruses more dangerous.

• 13:01Speaker 01

  I think that's a capability that they've had um for uh over a year, the the brainstorming part. But the implementation part seems to be fairly different. So I think in bio, actually, the um uh I would not be surprised if in a few months, there's a consensus that they're uh expert level in many relevant ways and that we need to be doing something about that.

• 13:21Alex Kantrowitz?

  Wow, that's that's crazy to me because I would think it would be the opposite, right? That cyber would be the thing that we need to be worried about because these things code. So, well, not virology. So I just want to ask you.

• 13:32Speaker 01

  But on that, biology has been such an interesting subject because they just know the literature really well. They they know the ins and outs of it. They got a fantastic memory.

• 13:42Speaker 01

  And they have so much background experience. It's been, for some reason, their easiest subject historically, biology and virology. In in earlier forms of measurements, like if you see how they do on exams. But now we're looking at their practical wet lab skills and they have those increasingly as well.

• 14:02Alex Kantrowitz?

  So what about the evolution of the technology? Because this is all with large language models, right? Reasoning is just something that's taking place within a large language model, like the GPT, which powers ChatGPT. So what is it about the current capabilities that have increased to the point where they're now able to guide somebody through the creation or manipulation of a virus? That seems to be like a step change in capability.

• 14:27Speaker 01

  Well, now they have this image understanding, image understanding skills. So that's a problem that that they didn't use to have. That that makes it a lot easier for them to do guidance or sort of, you know, be an apprentice or sort of a guide on one shoulder saying, now do this, now do that.

• 14:45Speaker 01

  But I don't know where that came from, that that skill. They've just trained on the internet and maybe they read enough papers and saw enough pictures of things inside those papers to have a sense of the the protocols and how to troubleshoot appropriately. So since they've read basically every academic paper written, maybe maybe that's the the cause of it. But it's a surprise. I mean, I I was thinking that this practical tacit knowledge or something wouldn't be something that they would pick up on necessarily. It'd make a lot more sense for them to have, you know, academic knowledge about, you know, knowledge of vocab words and things like that. But so I don't know where it came from. It's there.

• 15:29Alex Kantrowitz?

  Right. But this is still all stuff that is known to people. Like, it's not like the AI is coming up with. Nude viruses on its own.

• 15:37Speaker 01

  Well, so.

• 15:38Alex Kantrowitz?

  You can't like prompt whatever GPT it is and say.

• 15:42Speaker 01

  Well, so.

• 15:42Alex Kantrowitz?

  Create a new coronavirus.

• 15:43Speaker 01

  So if you're saying, I'm trying to modify this property of the virus so that it has more transmissibility or a longer stealth period. Then I think it could, with some pretty easy brainstorming, make some suggestions. And then if it can guide you through the intermediate steps, that's something that can make it be much more lethal. I don't think it needs a, you don't need breakthroughs for doing some bioterrorism generally. The main limitation for risks generally, risks will be capability and intent. And historically, our bio risks have been fairly low because the people with these capabilities has been a very small number, maybe a few hundred top virology PhDs. And then a lot of them just don't intend to do this sort of thing.

• 16:28Speaker 01

  However, if these capabilities are out there without any sorts of restrictions and extremely accessible, then as it happens, then your risk surface is blown up by several orders of magnitude. A solution for this, for to let people keep access to these expert level virology capabilities is that they can just speak to sales or ask for permission to have some of these guardrails taken off. Like if they're a real researcher at Genentech or what have you, wanting these expert level virology capabilities, then they could just ask and then like, oh, you're a trusted user. Sure, here's access to these capabilities. But if somebody just made an account a second ago, then by default, they wouldn't have access to it.

• 17:10Speaker 01

  So it, so for safety, a lot of people think that the way you go about safety is, you know, slowing down all of AI development or something like that. But I think there are very surgical things you can do where you just have it refuse to talk about topics such as reverse genetics or guide you through practical intermediate steps for some virology methods. And.

• 17:33Alex Kantrowitz?

  Wait, those safeguards don't exist today?

• 17:35Speaker 01

  Um, at XCI they do.

• 17:38Alex Kantrowitz?

  You're an advisor at XAI?

• 17:39Speaker 01

  Yeah, yeah, yeah.

• 17:40Alex Kantrowitz?

  But like, what were the models that you were testing to try to find out whether they would help with the enhancement?

• 17:45Speaker 01

  So, so, um, uh, we tested pretty much all of the leading ones that have these sort of multimodal capabilities and they'll have some sort of safeguards, but there are various holes. And so uh, those are, those are being uh, um, patched or we've communicated that, hey, you know, there are various issues here. And so I'm hopeful that uh, very quickly, uh, some of these vulnerabilities will be patched with it. And then if people want access to those capabilities, then it could possibly be a trusted third-party tester or something like that, or work at a biotech company. And then those restrictions could be lifted for those use cases. But random users, we don't know who they are, asking how to make some virus more lethal or something, sorry, uh, animal affecting virus.

• 18:28Speaker 01

  It's just, just punt or have the model refuse on that. That seems fine.

• 18:32Alex Kantrowitz?

  Yeah, we do see the benchmarks come in through each model release and it's like, oh, now it scored eighty-fourth or ninetieth percentile or ninety-seven percentile on this math test or on this bio test. And for us, it's like, oh, that's the model doing it. But what you're trying to say is, um, and correct me if I'm wrong, if it's getting ninety percent of the way that an expert virologist might get, then it could take a crafty user, you know, a number of prompts effectively to find their way towards that one hundred percent, because if they tried enough times, they might accidentally get to the, not accidentally, but they might end up getting the bad virus that we're trying not to have the public create.

• 19:09Speaker 01

  Yeah, yeah. So this, this is, this is what concerns me like quite a bit. And I'm being more quiet about this just to, you know.

• 19:15Alex Kantrowitz?

  Well, you're talking about.

• 19:16Speaker 01

  Yeah, I guess what I'm talking about now, but I'm not, you know, I'm not, there's this orders of magnitude with it. This is, we're, it's, it's being taken care of at, at XAI. And this is sort of in our risk management framework there. Um, and uh, um, other labs are taking this sort of stuff more seriously or finding some vulnerabilities and then they're patching them. So I'm being, you know, nonspecific about some of the, um, vulnerabilities here. Um, but hopefully can, uh, provide more precision, um, once they have that taken care of. But yeah.

• 19:44Alex Kantrowitz?

  Okay. I look forward to reading the paper. You're an advisor to Scaly AI.

• 19:50Alex Kantrowitz?

  They are a company that will give a lot of PhD level information to models in post-training, right? So you've trained up the model on all of the Internet. It is pretty good at predicting the next word, and then it needs some domain-specific knowledge. Scale, from my understanding, has PhDs and really smart people writing their knowledge down and then feeding it into the model to make these models smarter. How does a company like Scale AI approach this? Do they like have to say, all right, if you're a virology PhD, we shouldn't be fine-tuning the model with your information? Like, what's going on there and how are you advising them?

• 20:24Speaker 01

  So I've largely been advising on measuring capabilities and risks in these models. So we did, for instance, a paper on the weapons of mass destruction-related knowledge that models would have together last year. And for that, we were finding a lot of the academic knowledge or knowledge that you would find in the literature. Like, does it really understand the literature at the quite well? And we were seeing that in biology and for bioweapons-related papers that they did. However, this just tested their knowledge, not their know-how. So that's

• 21:09Speaker 01

  why we did the follow-up paper to see what's their actual wet lab know-how skills. And those were lower, but now they're higher. And so now those vulnerabilities need to be patched, or and those patches are, I gather, underway. So we've also worked on other sorts of things together, like in measuring the capabilities of these models, because I think it's important that the public have some sense of how quickly is AI improving? What level is it at currently? So a recent paper we did together was Humanities Last Exam, where we put together various professors and post-docs and PhDs from from all over the world, and they could join in on the paper

• 21:54Speaker 01

  if they submit some good questions that stump the AI systems. And I think this is a fairly difficult test, so it was think of something really difficult that you encountered in your research and try and turn that into a question. And I think each person, each researcher probably has one or two of these sorts of questions. So it's a compilation of that, and I think when there's very high performance on that benchmark, that would be suggestive of something that has, say, in the ballpark of superhuman mathematician capabilities. And so I think that would revolutionize the academy quite substantially.

• 22:33Speaker 01

  Because all the theoretical sciences that are so dependent on mathematics would be a lot more automatable. You could just give it the math problem and it could probably crack it, or crack it better than nearly anybody on earth could. So that's an example capability measurement that we're looking at. We excluded in humanities last exam, no virology related skills.

• 22:56Alex Kantrowitz?

  Okay.

• 22:57Speaker 01

  So we were not collecting data for that because we didn't want to incentivize the models getting better at that particular skill through this benchmark.

• 23:04Alex Kantrowitz?

  And how's the AI doing today on that exam?

• 23:06Speaker 01

  Um, they're in the ballpark of like ten to twenty percent overall, the very best models. So, you know, it'll take a while for it to get to eighty plus percent. But I think once it is eighty plus percent, that's basically a superhuman mathematician is one way of thinking of it.

• 23:23Alex Kantrowitz?

  But the thing is, they're at ten to twenty percent now.

• 23:25Speaker 01

  Mm hmm.

• 23:25Alex Kantrowitz?

  And many experts within the AI field, the practitioners, we had Jan on a couple of weeks ago, talking about how we're getting to the point of diminishing returns with scaling, right? That the current growth trajectory of, or the current trajectory of generative AI in particular,

• 23:43Alex Kantrowitz?

  is limited because basically the labs are maximizing their ability to increase its capabilities. So I'm curious what you think, whether you think that's right, because you're obviously working with these companies, working with XAI, you're working with scale. If we are getting to this data wall or some wall or some moment of diminishing marginal return on the technology, is it possible that all this fear is somewhat misplaced? Because if the AI is not going to get much better than it is right now, at least with the current methods, you know, we may not be a year or two away from AGI, right? We may not be getting AGI at the end of twenty twenty five, like some people are suggesting. And so then maybe we shouldn't be as afraid because again, the stuff is limited.

• 24:27Speaker 01

  Yeah, so if if we were trapped at around the capability levels that we're at now, then that would definitely reduce urgency and, you know, means one could chill out a bit more and take it easy. But I'm not really seeing that. I think maybe what he's referring to is the sort of pre-training paradigm, sort of running out of steam. So if you train, take an AI, train on a big blob of data and have it just sort of predict the next token to do what basically gave rise to older models like GPT-4, that sort of paradigm

• 25:08Speaker 01

  does seem like it's running out of steam. It has held for many, many orders of magnitude, but the returns on doing that are lower. That is separate from the new reasoning paradigm that has emerged in the past year, which is where you train models to on math and coding types of questions with reinforcement learning, and that has a very steep slope. And I don't see any signs of that slowing down. That seems to have a faster rate of improvement than the pre-training paradigm, the previous paradigm had. And there's still a lot of reasoning

• 25:53Speaker 01

  data left to go through and do reinforcement learning on. So I think we have quite a number of months or potentially years of being able to do that. And so personally, I'm not even thinking too specifically about what AIs will be looking like in a few months. They'll be, I think, quite a bit better at math and coding, but I don't know how much better. So I'm largely just waiting because the rate of improvement is so high and we're so early on in this in this new paradigm that I don't find it useful to try and speculate here. I'm just gonna wait a little while to see. But but I would expect it to be quite better in each of these domains, in these STEM domains.

• 26:37Alex Kantrowitz?

  Right. I guess reasoning does make it better at the areas that you're mostly concerned about.

• 26:41Speaker 01

  Yeah.

• 26:41Alex Kantrowitz?

  Doing math.

• 26:42Speaker 01

  Yeah.

• 26:42Alex Kantrowitz?

  Doing science.

• 26:43Speaker 01

  Yeah, that's right, yeah.

• 26:44Alex Kantrowitz?

  Because when it goes, tell me again if I'm wrong, when it goes step by step. It's much better at executing and working on these problems than if it's just printing answers.

• 26:52Speaker 01

  Yeah, and there is a possibility, and this is sort of a hope in the field, I don't know whether it will happen, is that these reasoning capabilities might also give these agent type of capabilities, where it can do other sorts of things, like make a PowerPoint for you, and do things that would require operating over a very long time horizon. Potentially, that skill set would fall out of this paradigm, but it's not clear. There has been a fair amount of generalization from training on coding and mathematics to other sorts of domains like law, for instance.

• 27:27Speaker 01

  And maybe, if those skills get high enough, maybe it will be able to sort of reason its way through things step by step and act in a more coherent, goal-directed way across longer time spans.

• 27:37Alex Kantrowitz?

  I'm gonna try to channel Yan here a little bit. I think he would say that this is still gonna be constrained by the fact that AI has no real understanding of the real world.

• 27:48Speaker 01

  Well, I don't know, that sounds like almost a no true Scotsman type of thing. Like, it's like, what's real understanding?

• 27:53Alex Kantrowitz?

  Right, okay.

• 27:56Speaker 01

  If it has predictive ability, if it's, they're sort of like, if it can do the stuff, that's what I care about. But if it doesn't satisfy some strict philosophical sense of something, some people might find that compelling, but I don't.

• 28:08Alex Kantrowitz?

  I'll give you an example, like with the video generators. Like, if AI really understood physics, then, you know, when you say, "Give me a video of a car driving through a haystack", it will actually be a car driving through a haystack. As opposed to what I've done is give it that prompt, and it's just hay exploding onto the front of a car with perfectly intact hay bales in the background.

• 28:31Speaker 01

  I think that for a lot of these sorts of queries, at least with images, for instance, we'd see a lot of nonsensical arrangements of things, and things that don't make much sense if you look at it more closely. But then, as you just scale up the models, then they tend to just kind of get it increasingly so. So we might just see the same for images, or, excuse me, for video. I think as well, they have like some good world model stuff, like they'll have like vanishing points being more coherent. And like, if I were drawing or anything like that, I'd probably be lacking, you know, lacking in understanding of the physics and geometry of the situation, and making things internally coherent relative to them, so.

• 29:15Speaker 01

  I don't know. Yeah, they seem pretty compelling and have a lot of the the details right, including some of the more uh structural details, but uh there's there'll be gaps uh that one can keep zooming into and but I just think that that set will keep decreasing as was sort of the case with uh images and text before it. I mean, text back in the day, the same argument. It doesn't have a real understanding of causality. It's just sort of mixing together words and whatnot. And it's when it was barely able to um construct sentences coherently.

• 29:42Alex Kantrowitz?

  Now it can.

• 29:42Speaker 01

  And then, yeah, I know it can. Um uh so I don't know if it like then got a real understanding in in the sort of philosophical sense that he's thinking for for language, but it was good enough. And that might be the case with uh video as well.

• 29:56Alex Kantrowitz?

  There were points where I was like, oh, but it is getting the guy sitting on the chair when I say, you know, do a video of a guy sitting on a chair and kicking his legs and those legs are kicking and they are bending at the joints.

• 30:07Speaker 01

  Yeah.

• 30:07Alex Kantrowitz?

  So there must be some understanding there.

• 30:09Speaker 01

  Yeah, in some ways, and that but if you ask them to do like gymnastics and I'll just have lips flailing all over.

• 30:13Alex Kantrowitz?

  The person just disappears into the floor. So, okay, like you said at the beginning, we're ChatGPT isn't gonna kill us.

• 30:19Speaker 01

  Yeah, yeah.

• 30:20Alex Kantrowitz?

  Yet.

• 30:22Alex Kantrowitz?

  Uh let's talk about hacking.

• 30:23Speaker 01

  Uh huh.

• 30:23Alex Kantrowitz?

  Uh I do think that we glanced over a a little bit before, but in terms of we're we're now going through, I think, the humans plus AI problem.

• 30:32Speaker 01

  Uh huh.

• 30:32Alex Kantrowitz?

  Right? Um and hacking to me is is one that I think we should definitely focus on. You mentioned that we're we're still not quite there, but it does seem to me, again, I'm just gonna go back to the point I made earlier. You can really code stuff up with these things and they enable like pretty impressive code already. You you could think that uh ChatGPT could produce pretty good phishing emails. If you just kind of creatively and not just ChatGPT, but all of these GPT models. If you creatively prompt it, right, it will give you an email that you can send and try to fish somebody.

• 31:02Speaker 01

  Mm hmm.

• 31:02Alex Kantrowitz?

  Um or even, let's say, you just take an open source model like DeepSeek, download it and then run it without safeguards.

• 31:08Speaker 01

  Mm hmm.

• 31:10Alex Kantrowitz?

  So where's the risk with hacking? I know you said it's a little bit further off. Why is it further off? And what should people be afraid of? Or what should people be concerned of?

• 31:18Speaker 01

  Yeah, yeah. So the the risk from it, more of the risk comes from when they're able to autonomously do the hacking themselves. So trying to break into a system, finding an exploit, escalating privileges, causing damage from there, things like that. And that requires multiple different steps and these agential skills that I keep referring to, that they currently don't have. So although they could facilitate in like ransomware development and other forms of other forms of malware, for them to autonomously execute and infiltrate systems, that at or

• 31:59Speaker 01

  that is something that will require the new agential skills. And I don't see, it's very unclear when those arrive. Could be a few months from now, could be a year from now. It's a little less, I'm a little more suspicious, and maybe it would even take two years for that. So so that's something to, for us to get prepared for, figure out how we're going to deal with that, try and make safeguards increasingly robust to people trying to maliciously use it in those ways. But yeah, I think much of the risk source comes from being able to take one of these AIs, let's say, one of these DeepSeek AIs, let's say it's DeepSeek agent version, and it's able to actually do these cyber attacks. Then you could just run 10,000 of them simultaneously.

• 32:45Speaker 01

  And then you, you know, some rogue actor could have it target critical infrastructure. Then this is causing quite severe damage. So for like critical infrastructure, you know, this could be like have it reduce the detector, the or the filter in, you know, a water plant or something like that. Then the water supply is like ruined. Or you could target these thermostats, these in in various homes and because they're, you know, often some of the more advanced ones are, you know, connected to Wi-Fi. And then you sort of turn them up and down simultaneously. And this can just like ruin like transformers and like blow them. And then, you know, they take multiple years to replace. Things like that. And

• 33:28Speaker 01

  But they aren't capable of doing that sort of thing currently. Uh, so it's, it's more of a on the horizon type of thing, but uh, but I'm, you know, I'm not like feeling the urgency with that uh currently. Um, yeah, I'm, I'm more concerned about, I, I think there's more, um,

• 33:44Speaker 01

  uh, the geopolitics of this, like, you know, making sure that, uh, um, uh, um, uh, states are, you know, aware of what's going on in AI, like they're, uh, at least able to follow the news and things like that in, in some capacity. Um, I, I think that's, the, the things like that feel somewhat more urgent to me than, than trying to address cyber risks. There are things to do though, and I think we should create incentives beforehand, but, you know.

• 34:13Alex Kantrowitz?

  Maybe I'm too much of an optimist for my own good, but when I hear you talk about this, I also get a little bit excited about the capabilities of these

• 34:20Speaker 01

  Mm-hmm.

• 34:20Alex Kantrowitz?

  programs because, for instance, if AI can enhance the function of a virus, AI can probably create a vaccine. Uh, make, make medical discoveries. If AI can hack into the infrastructure of some country, right? Find exploits and uh, turn the thermostats up and down, then AI could probably do incredible amounts of very beneficial coding and computer work for humanity. So if we do get to that point, it seems to me like there's going to be these, these maybe two poles here, right? One is the potentially scary and destructive stuff that you can mitigate.

• 34:51Speaker 01

  Mm-hmm.

• 34:52Alex Kantrowitz?

  Right, with some of the controls that you talked about, but also amazing opportunity.

• 34:55Speaker 01

  Mm-hmm. Yeah, so it's, it's, uh, in, in the thermostat thing was for messing with the electricity and that

• 35:01Alex Kantrowitz?

  Oh, yes.

• 35:01Speaker 01

  causing strain on the power grid and, um, destroying transformers. The, uh, just, just for, uh, clarification in case it, uh, but, um, uh, yeah, I think you're pointing at that it's dual use. So, um, uh, I'm not saying AI is bad in every single way and, uh, it's, it's like other dual use technologies. Bio is a dual use technology. Can be used for bioweapons, can be used for healthcare. Um, uh, uh, nuclear technology is dual use. There's civilian applications for it as well and chemicals too. And we have managed all of those other ones by selectively trying to, you know, limit some particular types of usage and restricting the capabilities of rogue actors to some of these technologies and making sure there are good safeguards for the civilian applications.

• 35:46Speaker 01

  And then we can actually capture the benefits. So it's not an all or nothing type of thing with AI. It's what are surgical restrictions one can place so that we can keep capturing the benefits. And so, for instance, with virology, that's a matter of you have the safeguards and then the researchers who want access to those can speak to sales. That's basically a resolution of that problem, provided that you have the models kept behind APIs. And so now on this dual use part, though, there's an offense-defense balance. So for some applications, it can help, it can hurt.

• 36:27Speaker 01

  And maybe it helps more than it hurts, or maybe it will hurt more than it will help. So in bio, I think that is offense dominant. If somebody creates a virus, there's not necessarily a cure that it will immediately find for it. If it would help a rogue actor make a somewhat compelling virus, now that could be enough to cause many millions to die. And it may take months or years to find a cure. There are many viruses for which we have not found cures yet. And for cyber, in most contexts, there's a balance between offense and defense, where if somebody can find a vulnerability with one of these hacking guys, then they could also use that to patch the vulnerability.

• 37:09Speaker 01

  There is an exception, though, where in the context of critical infrastructure, there the software is not updated rapidly. So even if you identify various vulnerabilities, there will not necessarily be a patch because the system needs to always be on, or there are interoperability constraints, or the company that made the software is no longer in business. These sorts of things. So our critical infrastructure is a sitting duck. And so in that context, cyber is offense dominant. But in normal contexts, it's roughly. There's roughly a duality and for virology, I think that's largely offense dominant.

• 37:44Alex Kantrowitz?

  So before we go to the nation state element of this, I need to ask you a question about the actual research houses themselves. Every research house says they're a concern with safety, from OpenAI to XAI, everything in the middle. Maybe not DeepSeek, we'll get to DeepSeek. Yet, they're the ones that are building this technology. And I find it a little strange that you have companies that are saying, we're built, it's weird. We have to build this and advance this technology so we can keep people safe? I never really understood that message.

• 38:17Speaker 01

  Yeah, I don't know if it's to say that we need to keep people safe. I think it's more that the main organizations that have power in the world now are largely companies. And so if one's trying to influence the outcomes, one basically needs to be a company, is how many of them will reason. They'll think that, yeah, you could be in civil society, or you could protest, but this will not determine the course of events as much. So there's sort of, many of them are buying themselves the option to hopefully influence things in a more positive direction, but most of the effort will be to stay competitive and stay in this arena. So I think over, you know, 90% of the intellectual energies that they're

• 39:00Speaker 01

  going to spend is actually, how can we afford the 10x larger supercomputer? And that means being very competitive, speeding this up, and making safety be some priority, but not necessarily a substantial one. So I do think there is sort of a, an interest in contradiction, or something that looks like a contradiction there. But I think if we think back to nuclear weapons, nuclear weapons, nobody wants nuclear weapons. If there'd be zero on Earth, fantastic, you know, that would, that would be a nice thing to have, if that would be a stable state. But it's not a stable state. One actor may then develop nuclear weapons, and then they could destroy the other. So this encourages states to do an arms race, and it makes everybody

• 39:45Speaker 01

  all collectively less secure, but that's just how the game theory ends up working. So you get a classic, what's called a security dilemma. Everybody's worse off collectively, but, and even if you took it seriously, you'd say, yes, nuclear technology is dual use and potentially catastrophic. And we need to be very risk conscious about it. You can agree with all those things, but you still might want nuclear weapons because other parties will also have nuclear weapons. And unilateral disarmament in many cases are just didn't make, uh, didn't make game theoretic sense. So, um, uh, in the way that like an individual company pausing, um, their development while others race ahead doesn't make game theoretic sense.

• 40:27Speaker 01

  So I think this just points to the fact that there's some, um, game theory is kind of confusing. And so you're getting some things that are seeming contradictions that if you use a nuclear analogy, go, yeah, I suppose that makes sense. And it's just kind of an ugly reality to internalize.

• 40:42Alex Kantrowitz?

  Doesn't that discount the fact that like these companies, if they want to influence like the way things are going, um, they are gonna be, it's like you're one in the same. Yes, you're influencing, but without you, this wouldn't be moving as fast as it is. Like it is interesting, for instance, thinking about Elon Musk, right? Um, obviously he has you in two days a week to work on safety inside XAI, but he's also putting together what million GPU data centers to build the biggest, baddest L M ever.

• 41:08Speaker 01

  Um, well, if he didn't, then, then he would be having less influence over it. So it's, um, uh, there's, it's not something that I would envision everybody would just sort of voluntarily pause. So subject to companies not sort of voluntarily rolling over and dying, then what's the best you can do subject to those constraints? But the competitive pressures are quite intense such that they do end up prioritizing focusing on competitiveness, um, and other priorities like, um, what's the budget for safety research? Um, it will be generally lower than would be, you know, nice to have if, um, this were a less competitive environment.

• 41:53Alex Kantrowitz?

  Do you think Elon is more interested in like restoring this original vision that he had for OpenAI, making everything open source, uh, making it safe? I would imagine like he founded OpenAI with Sam Altman, a sort of a B chat against Google because he was afraid of what Google was gonna do with this technology. So I'm curious if you think. Think that XAI is along that mission? Or is he more interested in the sort of soft cultural power that comes with having the world's best AI? For instance, like you can change the way that it speaks about certain sensitive political issues. It can be anti-woke, which we all know is sort of where Elon stands.

• 42:35Alex Kantrowitz?

  So what what do you think his true interest lies in building XAI?

• 42:39Speaker 01

  Well, I think the and I won't, you know, position myself as sort of speaking on behalf of them here.

• 42:44Alex Kantrowitz?

  We won't put you as Elon spokesperson, but you are in there.

• 42:47Speaker 01

  Yeah, so I think that the mission is to understand the the universe. And so this means having AIs that are honest and accurate and truthful

• 43:00Speaker 01

  to improve the public's understanding of the world. So we will be getting in a very fast-moving, trying situation with AI if it keeps accelerating. And so good decision-making will be very important. And us understanding the world around us will be very important. So if there are more features that enable truth-seeking and honesty and good forecasts and good judgment and institutional decision-making, those would be great to have with. The the hope is that Grok could help enable enable some of that. So that civilization is steered steering itself more prudently in this potentially more turbulent period that's upcoming. That's that's one read on the the mission statement.

• 43:45Speaker 01

  But I think that that's the objective of it is understand the universe and their different sub-objectives that that would give rise to. And I think it's I think it's ability to

• 43:58Speaker 01

  help culture process events without censorship

• 44:04Speaker 01

  or by political bias one way or the other is is a stated objective. And I think that would be indispensable in the years going forward.

• 44:14Alex Kantrowitz?

  Do you buy that they're that's what they're doing? Because we also heard the same thing from Elon when it came to buying Twitter, now X.

• 44:22Alex Kantrowitz?

  But but.

• 44:23Speaker 01

  I think community notes has been, you know, quite.

• 44:25Alex Kantrowitz?

  But that was something that was built under Jack Dorsey. I'm not gonna take sides here. I'm gonna just observe empirically what I've seen. I mean, we know that Substack links have been deprioritized because it was seen as a competitor with Twitter. We know that Musk, I think, according to reporting, changed the algorithm to have his tweets show up more often. And his tweets took a strong stance towards supporting Donald Trump in the election. So to me, the idea that, like, hearing, again, from Elon, and again, look, I respect what Elon's done as a businessperson. But hearing, again, that he has a plan to make a culturally relevant product that's free of censorship and politically unbiased. I don't know if I believe that anymore.

• 45:07Speaker 01

  So I don't know about some of the specific things, such as the, you know, waiting thing or something like that. Profile things, for instance. I think that, overall, in terms of cultural influence and people being more disagreeable and doing less self-censoring has been, has been successful. I think that was the main objective of it. And so I think, I think that X had a large role to play there. So, I don't know. I think, like, I think in terms of shaping discourse norms in the US, and that seems to have been successful, in my view.

• 45:47Alex Kantrowitz?

  Yeah, I'm not saying pre-Elon, Twitter didn't censor, which is the wrong, probably the wrong word, because that's usually from the government. Didn't sort of shape the definition of speech to its own liking. It obviously had a progressive approach and moderated speech on a progressive approach. I just don't think Elon is not using his own influence when it comes to how he runs X. But you and I could speak about this.

• 46:11Speaker 01

  Yeah, and I, this isn't even my sort of wheelhouse as much, but yeah, I mean, it's sort of like, since, you know, doing.

• 46:18Alex Kantrowitz?

  You brought it up, so.

• 46:19Speaker 01

  Oh, okay, all right, sure.

• 46:21Alex Kantrowitz?

  I mean, just the non-biased and truthful things, so it's worth talking about.

• 46:25Speaker 01

  I mean, it is, if there are, like, ways in which it's, like, extremely biased, one way or the other, that's useful to know. This is a, a thing that is continually trying to be improved, at least for, for XAI's Grok. So, and I think that all of the sort of product offering could get quite better at this.

• 46:52Alex Kantrowitz?

  Okay.

• 46:52Speaker 01

  But I'm not speaking as a representative there or anything like that, but I guess maybe in my, I guess right now, in my personal capacity, I think that there's things to improve on for all these models in terms of their bias.

• 47:06Alex Kantrowitz?

  All right, we agree on that.

• 47:09Alex Kantrowitz?

  You hinted at it previously, but you talk a little bit about how companies, basically how you don't think it's a good idea for there to be an arms race here. And certainly, there is one between the US and China. We know the US has put export controls on China. China has in some ways gotten around them through like very creative procurement processes that go through Singapore. Right, we can probably say that with a pretty good degree of confidence. Then, of course, we see the release of DeepSeek and some other AI applications from China. And everyone's trying to build the better AI so that they have the soft power like we spoke about to effectively, you know, A, control, like to influence culture across the world. But also, it's an offensive capability and defensive, like you're saying.

• 47:55Alex Kantrowitz?

  If your country has the ability to manipulate viruses or to do cyber hacks, you become more powerful and you get to sort of, you know, potentially put your view of the world, implant your view of the world on the way that it operates. You have a paper out that's sort of arguing against this arms race. It's called Superintelligent Strategy. It's with you, Eric Schmidt, who we all know, former CEO of Google. I think he just started. He's taking over a drone company, so you can tell me a little bit about that. And Alexander Wang, the former, no, not the former, the current CEO of Scale AI, who's been formerly on this show. Talk a little bit about why you don't think it's a good idea for countries to pursue this arms race. You say it might be leading us to mutually assured AI malfunction, not mutually assured like nuclear destruction.

• 48:40Alex Kantrowitz?

  I think that's what you get that from.

• 48:42Speaker 01

  Yeah, so the strategy has three parts, one of which is competitiveness. But we're saying that some forms of competition could be destabilizing and that you may be irrational to pursue it because you couldn't get away with it. So in particular, this. I'm making a bid for superintelligence through some automated AI research and development loop is could potentially lead to one state having some capabilities that are vastly beyond another state's. If you have, if one state gets to experience a decade of development in a year and the

• 49:24Speaker 01

  other one is a year behind, then this results in a very substantial difference in the state's capabilities. So this could be quite destabilizing if one state might then start to get an insurmountable lead relative to the other. So I think that form of competition would be very dangerous and because there's a risk of loss of control and because it might incentivize states to engage in preventive sabotage or preemptive sabotage to disable these sorts of projects. So I think states may want to deter each other from pursuing superintelligence through this means.

• 50:06Speaker 01

  And this then means that AI competition gets channeled into other sorts of realms such as in military, in the military realm of having more secure supply chains for robotics, for instance, and for, for AI chips, having reduced sole source supply chain dependence on Taiwan for making AI chips. So states can compete in other dimensions, but them trying to defeat, compete to develop superintelligence first, I think that's, that seems like a, a very risky idea and I would not suggest that because there's too much of a risk of loss of control and there's too much of a risk that one state, if they do control it, uses it to disempower

• 50:47Speaker 01

  others and affects the balance of power far too much and destabilizes things. So, but the, the strategy overall, think of, think of the Cold War. Before you go on the strategy, my reaction to that is good luck telling that to China. So I think it's totally so for the for deterrence, I think if the US were pulling ahead, both Russia and China may have a substantial interest in saying, hey, cut this out, pulling ahead to develop superintelligence, which could give it a huge advantage and an ability to crush crush them. They'd say, you don't get to do that. We are going that we are making a conditional threat that if you keep going forward in this because you're on the cusp of building this, then we will, you know, disable your data center or the

• 51:32Speaker 01

  surrounding power infrastructure so that you cannot continue building this. I think they could make that conditional threat to deter it and we might do the same or the US might do the same to to China or other states that would do that. So I don't see why China wouldn't do that later on. Right now they're not just thinking about, you know, superintelligence and advanced AI. So this is

• 51:53Alex Kantrowitz?

  This episode is brought to you by App Point. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what App Point does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. App Point, the unifying trust layer for AI. Head to

• 52:38Alex Kantrowitz?

  apppoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast.

• 52:54Alex Kantrowitz?

  Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a wide-glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/bigtechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are

• 53:39Alex Kantrowitz?

  and how to lock it down before a threat reaches their front door. That's ironwell.com/bigtechnology. Learn how to stop online threats before they become real-world attacks. The link's in the show notes.

• 53:53Speaker 01

  This is more of a description of what the dynamics later on, when AI is more salient. But it would be surprising to me if China were saying, "Yes, the United States. Go ahead, do your Manhattan Project to build superintelligence. Come back to us in a few years, and then tell us you can boss us around because now we're in a complete position of weakness, and we'll be at your mercy, and we'll accept whatever you say or tell us to do." I don't see that happening. I think they would just say move to preempt or deter that type of development so that they don't get put in that fragile position.

• 54:29Alex Kantrowitz?

  Are you in like the Eliezer Yudkowsky camp of bombing the data centers if we get to superintelligence?

• 54:36Speaker 01

  Well, so I think I'm advocating or pointing out that it becomes rational for states to deter each other by making conditional threats and by means that are less escalatory, such as cyber sabotage on data centers or surrounding power plants.

• 54:56Speaker 01

  I don't think one needs to get kinetic for this. And I think that if discussions start earlier, I don't see any reason things need to be escalating in that. That way or unilaterally actually doing that. We didn't need to, you know, get in nuclear exchange with Russia to sort of express that we have a preference against nuclear war. When so I think.

• 55:17Alex Kantrowitz?

  Thank goodness.

• 55:17Speaker 01

  So so so indicating or making conditional threats through deterrence seems like a much smarter move than a. Hey, wait a second. What are you doing there? And then bomb that that that seems needless.

• 55:34Alex Kantrowitz?

  Yeah, I'm not into that.

• 55:35Speaker 01

  Yeah.

• 55:36Alex Kantrowitz?

  Uh, but what you're talking about is sort of assuming that there will be a lead that will be protectable for a while. But everything we've seen with AI is that no one protects a lead, right?

• 55:48Speaker 01

  Well, um, if there's so one difference is that when you get to a different paradigm like automated AI R and D. The slope might be extremely high such that if the competitor starts to um.

• 56:04Speaker 01

  Do automated AI R and D a year later, they may never catch up just because you're so far ahead and your gains are compounding on your gains. Sort of like in social media companies, Eric will use this analogy where if one of them starts blowing up and growing before you started. It's often the case that you won't be able to catch up and they'll have a winner take all type of dynamic. So um.

• 56:30Speaker 01

  Right now, the rate of improvement is not um that high or there's less of a path for a winner take all dynamic currently. Uh, but but later on, um, when you have, you know, the ability to to run a hundred thousand AI researchers simultaneously. Uh, this this really accelerates things. Maybe OpenAI has got a few hundred, maybe it'll say three hundred AI researchers. So going from three hundred AI researchers to or is a magnitude more world class ones to create quite quite substantial developments. This is something that isn't, you know, new. This is something that like Alan Turing and and the founders of computer science. Uh, had pointed out that this is a natural property of when you get AIs at this level of capability, then this creates this sort of.

• 57:15Speaker 01

  Um, recursive dynamic where um things start accelerating um extremely quickly and quite explosively.

• 57:23Alex Kantrowitz?

  Okay, we managed to spend most of our conversation today talking about present risks or like risks in the near future. We should focus a little bit more on intelligence explosion and loss of control, and we're going to do that right after the break. And we're back here on Big Technology Podcast with Dan Hendricks. He is the director and co-founder of the Center for AI Safety. Dan, it's great speaking with you about this stuff. Let's talk a little bit. You've been sort of talking about it in the first half, but I want to zero in here on this idea of intelligence explosion or what you talk about as basically having AI autonomously improve itself. Just talk through a little bit about how that might happen and whether you see that being something that is actually probable in our future.

• 58:07Speaker 01

  Yeah, I mean, the the basic idea is just imagine automating one AI researcher, one world-class one. Then there's a fun property with with computers, which is there's copy and paste. So you can then have a whole fleet of these. It's well, you know, with humans, you know, if you just have one of them, you know, it's it's maybe they'll be able to train up somebody else who has a similar level of ability. So this this adds a very interesting dynamic to the mix. And then you can get so many of them proceeding forward at once. So and, you know, AIs also operate quite quickly. They can code a lot faster than than than people.

• 58:47Speaker 01

  So maybe it's maybe we've got 100,000 of these things operating at 100x the speed of a human. How fast will that go? Maybe conservatively, let's say it's just overall 10x-ing research. But 10x-ing research would mean, say, like a decade's worth of developments in a year. So that telescoping of of all these developments makes things pretty wild and means that one player could possibly Get AIs that go from like very good, you know, world-class to being vastly better than everybody at everything and a superintelligence. Something that towers far beyond any living person or collective of people.

• 59:32Speaker 01

  So if we get an AI like that, this could be destabilizing because it could be, you know, used to develop a superweapon potentially. Maybe it could find some breakthrough for anti-ballistic missile systems, which would make nuclear deterrence no longer work or other types of ways of weaponizing it. So that, that's why it's destabilizing. And so states then, if they're seeing, oh, they're, you know, don't, don't run this many AI researchers simultaneously in these data centers, working on to build a next generation or a superintelligence, because if you do so, then that will put us in, that will make our survival be threatened.

• 1:00:16Speaker 01

  So them saying, them deterring that would, would help them secure themselves and they can make those threats very credible currently, and I think we'll continue to be able to have these threats be credible going forward. So this is why I think it might take a while for superintelligence to be developed, because there'll be deterrence around it later on. And then maybe in the farther future, there could be something multilateral, but that's speaking, you know, quite far out in very different economic conditions. In the meantime, with the AIs that, that we'd have in the future, those could still automate various things and increase prosperity and all of that. So we'd still have explosive economic growth if you had something that was just at the

• 1:01:00Speaker 01

  average human level ability, running for very cheap. So I think that those are some of the later-stage strategic dynamics, and I don't think we can get away with, or I don't think any state could get away with trying to build a superintelligence, go build a, you know, big data center out in the middle of the desert, trillion-dollar cluster, bring all the researchers there and just not invite the other states to go, what do you, what do you think you're doing here?

• 1:01:29Alex Kantrowitz?

  You were at the White House yesterday.

• 1:01:31Speaker 01

  Well, this is largely just sort of speaking about some of these, you know, strategic implications.

• 1:01:38Alex Kantrowitz?

  Are they receptive?

• 1:01:39Speaker 01

  Yeah, I mean, it's a, it's a, this isn't a.

• 1:01:46Speaker 01

  There's always interest in, you know, thinking what are some of the later term dynamics? What things should happen now and whatnot? But this is, um, yeah, I think, I think when people think White House, it's, you know, sounds, you know.

• 1:02:02Alex Kantrowitz?

  Where the president lives.

• 1:02:04Speaker 01

  So there's the, well, yeah, so there's, there's the Eisenhower Building, which is, you know, part of the White House, kind of not, but you know, that's where everybody's works and whatnot. And I think, um, you know, some of the things we're speaking about here, like virology advancements, things like that, there's just a lot of, you know, things to, um, uh, speak about and, um, think what, what things make sense or what things to keep in mind going forward. So, yeah.

• 1:02:29Alex Kantrowitz?

  Yeah, I guess I'd rather an executive branch paying attention to this stuff than not.

• 1:02:33Speaker 01

  Yeah, yeah, that's right. Yeah. Yeah. And what, what are the sort of ways that help, you know, um, maintain competitiveness? Because, you know, how people normally think about this, they'll think it's all or nothing and good or bad thing. And instead we're saying, you know, it's dual use. So that means there are some particular applications that are concerning and there are other applications that are good. And you want to stem the particularly harmful applications and what are ways of doing that? Um, while capturing the upside.

• 1:02:59Alex Kantrowitz?

  Right. Okay. So the intelligence explosion part of this conversation, Neveley brings up the loss of control part.

• 1:03:06Speaker 01

  Mm-hmm.

• 1:03:07Alex Kantrowitz?

  Where to me, I think the thing that when people think about AI harm, they're always worried that AI is going to escape, you know, the simulation or whatever it is and act on, act on its own and try to basically ensure that it preserves itself. Uh, we've seen it recently, I think I brought this up at the beginning of the show, where Anthropic has done some experiments where the AI has run code to try to copy itself over onto a server if it thinks that its values are at risk of being changed. Um, is this, so it's, it's fun to think about, but it's also like probably just probability? Like if you run it enough times because it's a probabilistic engine?

• 1:03:44Speaker 01

  Yeah, well, that's concerning though. If, if, if it was like, oh, it's only one in a thousand of them intend to do this. Well, if you're running a million of them, then you're basically certain to get many of them to try and, you know, self-exfiltrate.

• 1:03:55Alex Kantrowitz?

  And so are you worried that this self-exfiltration is going to be a thing?

• 1:03:58Speaker 01

  Um. I think from a, you know, a recursive automated AR and D thing, I think that has really substantial probability behind it of a loss of control in that situation.

• 1:04:11Alex Kantrowitz?

  So you're worried about this?

• 1:04:13Speaker 01

  So there's that, but I would distinguish between that and these sort of things that are not super intelligences, or things that are not coming from that sort of really rapid loop, and like the currently existing systems. I think that the currently existing systems are relatively controllable, or if there is some very concerning failure mode, we have been able to find ways to make them more controllable. For instance, for bioweapons refusal. We used not to be able to make robust safeguards for that two years ago. But we've done research, like with methods such as like called circuit breakers and things like that, and those seem to improve the situation quite a bit and make it actually

• 1:04:56Speaker 01

  prohibitively difficult to do that jailbreaking. And so maybe we'll find something similar with self-exfiltration. So I think people generally want to claim that like, oh, current AIs are not controllable. And I think that they're not highly reliably controllable, they're reasonably controllable. Maybe we could get some, or it seems plausible that we'll get to have increasing levels of reliability. And so I'm sort of reserving judgment. It'll depend more on the empirical phenomena. So I think everybody should research this more, and we'll sort of see what the risks actually are. But there are some that seem less empirically tractable, or things that can't be empirically

• 1:05:40Speaker 01

  solved, like this loop thing, like how are you gonna, you can't run this experiment a hundred times or something like that and make it, you know, go well. You're making a huge attempt at building a superintelligence and it has destabilizing consequences. Like, this isn't something that, that's totally unprecedented, and for that you have more of like a one chance to get it right type of thing. But with the current systems, we can continually adjust them and retrain them and come up with better methods and iterate. So it is concerning, it would not surprise me.

• 1:06:11Speaker 01

  If this would really start to make AI development itself extremely hazardous, instead of just the deployment, but instead inside the lab, like you need to be worried about the AI trying to be breaking out sometimes, that's totally in the realm of possibility. But yeah, I could see it going either way.

• 1:06:29Alex Kantrowitz?

  Yeah, I mean, this personally freaks me out because, yeah, if you see the AI trying to deceive evaluators, for instance, or you see the AI trying to break out, you really can't trust anything it's telling you. And we had Demis Hassabis on the show a little while ago, and he's basically like, listen, if you see deceptive behavior from AI, if you see alignment faking, you really can't trust anything in the safety training because it's lying to you.

• 1:06:53Speaker 01

  There is truth to that.

• 1:06:58Alex Kantrowitz?

  Are you seeing deceptiveness at Grok, by the way?

• 1:07:00Speaker 01

  Oh, yeah, yeah. So we have a paper out. Last week, we're just measuring the extent to which they're deceptive. And in the scenarios we have, like, all the models were in these sorts of scenarios under, you know, slight pressure to lie, not being told to lie, but just some slight pressure. Then some of them will, like, lie like 20% of the time, some of them like 60% of the time. So they don't really have this sort of virtue sort of baked into them, the virtue of honesty. So I think they'll, we'll need to do more work and we'll need to do it quickly. So I, I'm sort of, you know, speaking in a more nonchalant way about this, but I can't, like, you know, get worked up about every single risk, because they're also just, you know, be at 11 all the time. So there are some that I'm, you

• 1:07:41Speaker 01

  know, putting in different tiers, different tiers than, than other risks. And this is a more speculative one. We've seen these sometimes get surprisingly handleable. But yeah, it could end up making things really, really bad. We'll see. We'll, we'll do things about it to make that not be the case.

• 1:08:02Alex Kantrowitz?

  Okay. Thank you. Two more topics for you, then we'll get out of here. Center for AI Safety, who's funding it?

• 1:08:10Speaker 01

  Well, there's not sort of one funder. It's largely just various philanthropists. The main funder would be, um, uh, Yon Talan and, um, uh, Yon Talan, others who's a Skype co-founder.

• 1:08:23Alex Kantrowitz?

  Founder of Skype, right.

• 1:08:24Speaker 01

  Um, there's a variety of, of other philanthropies or philanthropists. The generally for, um, uh, generally for, so for instance, Elon doesn't, I've never asked him to, to fund the center. So that is it to say I don't get any money from Elon. My appointment at XAI, I get a dollar a year. Um, uh, um, at scale, I've, uh, at scale AI, I've increased my salary exponentially to where I get $12 a year, a dollar per month from scale. Uh, but I'll, I'll try not to, um, or I'll try to avoid, you know, getting complicated, having some complicated relations with them just so that I can, uh, you know, um,

• 1:09:09Alex Kantrowitz?

  You're doing the work.

• 1:09:10Speaker 01

  Not feel on behalf of, of any of them in particular.

• 1:09:12Alex Kantrowitz?

  So you're basically doing the work for them for free?

• 1:09:14Speaker 01

  Well, but it's useful.

• 1:09:16Alex Kantrowitz?

  Right.

• 1:09:17Speaker 01

  It's useful to do. Um, and, uh, I mean, yeah, I mean, I think the main objective is, yeah, just try, try and, try and generate some value here and have as best as one can. So, um, um, um, by reducing these sorts of risks. And yeah, I think it's, it's, it's a good arrangement because it enables me to like, you know, it's do have a choose your own, your own venture type of thing.

• 1:09:40Alex Kantrowitz?

  Right.

• 1:09:41Speaker 01

  Of like, oh, now I think the, the politics or geopolitics, this is more relevant. So now I can go off and learn about this for some months and then, uh, work, work on a paper there. And compared to if it's like, no, you gotta be coding 80 hours a week. That's, that's your job.

• 1:09:53Alex Kantrowitz?

  Yeah.

• 1:09:54Speaker 01

  That, that would be quite restrictive. And I couldn't be speaking with you. So.

• 1:09:57Alex Kantrowitz?

  I'm glad you're here. So thank you, Alex Wayne. Um, so let's talk a little bit about, about this funding because, uh, I think that after Sam Altman was fired and then rehired at OpenAI.

• 1:10:10Speaker 01

  Mm-hmm.

• 1:10:11Alex Kantrowitz?

  There was a sort of skepticism around effective altruism's impact on the AI field.

• 1:10:17Speaker 01

  Mm-hmm.

• 1:10:18Alex Kantrowitz?

  Even Yon Talan, I'm reading, uh, from his statements, right? After the OpenAI governance crisis highlights the fragility of voluntary AI motivated governance schemes.

• 1:10:25Speaker 01

  Oh, yeah.

• 1:10:26Alex Kantrowitz?

  So the world should not rely on such governance working as intended. Uh, now Yon is of course associated with EA. EA is like basically leading the conversation around AI safety. Um, is that good?

• 1:10:38Speaker 01

  So I think that in terms of beyond, I think these funded organizations that are EA affiliate, I don't know if he'd call himself that, but whatever, you know, people can, you know, ascribe labels how they'd like. I think that the, I mean, I've tweeted that EA is not equal to AI safety. I think that EA community generally is in slur on these. So I lived in Berkeley for a long time when I was doing my PhD. And there's sort of a school, a sort of AI risk school that was, had very particular views about what things are important. So malicious use, for instance, when I was, when I was talking about malicious use in the beginning of this thing, that, you know, that.

• 1:11:22Speaker 01

  They're not, they're, they're the long term, the long term. Yeah, yeah, the only loss of control, don't talk about malicious use. The other, that's a distraction. Okay. And so that was annoying because I'd always been working on robustness as a PhD student, where the main thing was malicious use.

• 1:11:40Speaker 01

  So, yeah, I ended up leaving Berkeley even before graduating, just because of the sort of relatively suffocating atmosphere and the sort of central focus on, there'd be some new fad and you'd have to, you know, get interested in that. It's elk eliciting late knowledge, this is the important thing that you have to focus on, or you have to focus on interoptimizers, or there's just lots of these speculative, empirically fragile things. So for instance, this alignment faking stuff that you're seeing, like, there's some concern there, but, you know, I'm not totally sold that this is like a top tier type of priority. But in these communities, this is all that matters currently, roughly speaking. This involuntary commitments from AIs, from AI companies.

• 1:12:24Speaker 01

  I think voluntary commitments from AI companies are also a distraction.

• 1:12:29Speaker 01

  Because the companies will, you should expect, most of them by default to just break those sorts of commitments if they end up going up against economic competitiveness. Okay. So I think it's a distraction, relatively. And so I think it's, I think there are many people who think that EA's broadly, their influence on this sort of thing has not been overall positive. I think at least for me and making, and other sorts of researchers in this space who've been interested in AI risks. The the amount of pressure to adopt some particular positions though on this be extraordinarily high and I think quite quite destructive. So I'm very pleased now that in only the

• 1:13:12Speaker 01

  most in the past year or so, there's been a lot more diversity of opinion, which has been quite quite important. So and I think this is just because the broader world is getting more interested in AI. So a lot of these, a lot of these, you know, fixation on this is the one particular risk. This is the most important risk and everything else is distraction. That just doesn't work when you're speaking with the or interfacing with the real world. There's a lot of complications. And AI is so multifaceted. So you can't in your risk management approach, can't just be focusing on one of them.

• 1:13:46Alex Kantrowitz?

  Right. So you're not an effective altruist?

• 1:13:48Speaker 01

  I don't I don't think of myself as that. I don't particularly get along with the this this school of thought, this sort of Berkeley AI alignment monolith. And I'm pleased that people can be more independently operating in this space now, which I don't think was the case for many, many years, including basically the entire time I was doing my PhD. And there'll be many people like Dylan Hadfield-Minell, a professor at MIT, who was also at Berkeley at the time, very suffocating Rohan Shah, researcher at DeepMind, very suffocating, that all feel this way, yeah.

• 1:14:29Alex Kantrowitz?

  Okay. Let's let's bring it home. We've been talking for more than an hour about AI safety as if it's controllable. But open source is like really putting up a pretty valiant effort in this field, keeping pace with the proprietary labs. And of course, open source is not controllable. What do you think about that? I mean, we just saw DeepSeek, not to, you know, go back to it all the time, but it effectively equaled the cutting edge at the proprietary labs and

• 1:14:59Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 1:15:28Speaker 03

  In this life, you can't win. When the world gets you down. No! Escape to Pluto TV.

• 1:15:34Speaker 01

  This is objectively fantastic.

• 1:15:36Speaker 03

  Relax into the movies and shows you already know and love, all for free. Schwing! Even our news is uplifting.

• 1:15:41Alex Kantrowitz?

  You stay classy, San Diego.

• 1:15:43Speaker 03

  Watch School of Rock, Crazy Ex Girlfriend, Anchorman: The Legend of Ron Burgundy, The X-Files, Wayne's World, and so much more.

• 1:15:49Alex Kantrowitz?

  We're not worthy!

• 1:15:51Speaker 03

  The best things on TV really are free. You just have to know where to look. Escape to Pluto TV. Stream now, pay never.

• 1:15:58Alex Kantrowitz?

  You know, put the weights on its website. So how can we possibly have a relationship of safety with AI if open source is out there exposing everything that's been done?

• 1:16:09Speaker 01

  So I've been, I haven't been endorsing open source historically, but I've thought that releasing the weights of models didn't seem robustly good or bad. So I sort of was like, it's fine. Seems to have complicated effects. There's an advantage to it, which it helped with diffusion of the technology so that more people would have access to it and sort of get a sense of AI and this increased literacy on this topic and just increased public awareness and get the world more prepared for more advanced versions of AI. So that's been my historical position. But this depends on, it should always proceed by a cost-benefit analysis.

• 1:16:52Speaker 01

  So if the, if for instance, they have these cyber capabilities later on, yeah, I think that, or I think that would be a potential place to be drawing the line on, on open weight releases personally.

• 1:17:11Speaker 01

  In particular, the ones that could cause damage to critical infrastructure. You could still capture the benefits by having the models be available through APIs. And if they're like software developers, they have access to these, you know, some more cyber offensive capabilities. But if they're a random user, they don't. If they're a random faceless user, they don't. And likewise for virology, once there's consensus, once the capabilities are so high that there's consensus about it being expert level in virology, I think that would be a very natural place to be saying, having an international norm, not saying a treaty because those take forever to write and ratify, but to a norm against open weights if they are expert level virologists.

• 1:17:54Speaker 01

  For the same reasons that we had the Biological Weapons Convention. Russia and, or the Soviet Union and the US got together for the Biological Weapons Convention. The US and China did as well. We also coordinated on chemical weapons with the Chemical Weapons Convention and the Nuclear Nonproliferation Treaty. States find it in their incentive to work together to make sure that rogue actors do not have extremely hazardous, potentially catastrophic capabilities like chem, bio, and nuclear inputs. So I think something similar might be reasonable for AI when they get at that capability threshold.

• 1:18:35Alex Kantrowitz?

  Dan, I am at once kind of reassured that people are thinking about this stuff, but also more freaked out than I was when we sat down. But I do appreciate you coming in and giving us the full rundown of what to be concerned about and what maybe not to be as concerned about as we think about where AI is moving next. So thank you so much for coming on the show.

• 1:18:56Speaker 01

  Yep, yep. Thank you for having me. This has been fun.

• 1:18:58Alex Kantrowitz?

  Super fun. If people want to learn more about your work or get in touch, how do they do that?

• 1:19:02Speaker 01

  I guess this paper or strategy we've been speaking about is at nationalsecurity.ai and then I'm also on Twitter or X or whatever it's called.

• 1:19:11Alex Kantrowitz?

  You should know your word with Elon.

• 1:19:13Speaker 01

  X.com, there, X.com slash Dan Hedrickson would be another way of following the goings ons and as the situation evolves, we'll keep trying to put out work and, you know, seeing what's going on with these risks and if we come with, you know, technical interventions to make them less, then we'll also put that out too. So, yeah. That's where you can find me.

• 1:19:34Alex Kantrowitz?

  Well, Godspeed, Dan, and we'll have to have you back.

• 1:19:36Speaker 01

  Yep, thank you.

• 1:19:36Alex Kantrowitz?

  Thanks again. All right, everybody, thank you for listening, and we'll see you next time on Big Technology Podcast.