ControlAI's Connor Leahy on why superintelligence is ‘not a weapon, it's an adversary’

AI companies have been talking about superintelligent AI like it's inevitable, but recent safety incidents like OpenAI's Hugging Face breach are demonstrating the potential dangers of deploying AI systems that are more capable than humans. So what happens when we can't reliably control what these systems do? On this episode of TechCrunch's Equity podcast, Rebecca Bellan is joined by Connor Leahy, an AI researcher, entrepreneur, and now the U. S. Executive Director of nonprofit ControlAI, which is pushing for a far more radical approach to AI safety: stopping companies from developing superintelligence altogether. Leahy explains why he thinks the risks have become too great to manage through alignment and containment alone, and how what sounded far-fetched six months ago is now being backed by a wave of new legislation. Listen to the full episode to hear more about: His take on the Sanders-Casar “Ban Superintelligence Act,” the parallel legislation in the U. K. that Control AI advised on, and why he thinks the U. S. bill may go further than necessary. How he views frontier AI labs less as companies simply chasing returns and more as political actors, and what that means for the trillions of dollars being poured into data center buildouts. Why Leahy thinks the real point of no return is when AI can build better AI, creating a potentially runaway cycle of self-improvement The case for international “trust but verify” agreements, and why Leahy thinks it isn't actually in China's interest to build superintelligence. Subscribe to Equity on YouTube, Apple Podcasts, Overcast, Spotify and all the casts. You also can follow Equity on X and Threads, at @EquityPod. Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This episode of Equity is brought to you by Schneider Electric. Schneider Electric, your energy technology partner.

• 0:05Rebecca Bellan?

  Hello and welcome back to Equity TechCrunch's flagship podcast about the business of startups. I'm Rebecca Bollan and this is the episode where we bring on industry experts to help us explore a trend in the tech world and dive deep. AI companies have been talking about super intelligent AI like it's inevitable. But recent safety incidents like the hugging face breach are demonstrating the potential dangers of deploying AI systems that are more intelligent and capable than humans are. Many are asking what happens when we can't control what these super intelligent systems do. Today, we're talking to Connor Leahy, an AI researcher, entrepreneur, and now the US Executive Director for nonprofit Control AI, whose mission is to do something radical in the world of AI, stop companies from developing super intelligence at all.

• 0:49Rebecca Bellan?

  What sounded far fetched six months ago is now being backed by a wave of new legislation and Connor is here to talk us through it all.

• 1:00Rebecca Bellan?

  Connor, welcome to the show. Thank you so much for joining us.

• 1:03Speaker 02

  Thanks for having me.

• 1:05Rebecca Bellan?

  So before we get into it, before we get into our discussion of the end times, as you called it before we jumped on here, I want to hear a little bit about your background. Like you've been in this space for a while. You reverse engineered GPT-2 in 2019 while you were still a student. You co-founded a Luther AI. Yeah, talk me through a Luther conjecture and what kind of got you to the place where you are today at Control AI.

• 1:29Speaker 02

  Yeah, so I've been involved with AI basically my whole life. I kind of really got into AI when I was like 16 and I was thinking, how can I make the world a better place? And while I figured, well, you know, maybe, maybe I can't cure all the diseases, but if I could just automate science, then I could cure all the diseases. Great, let's go do that. How hard could it be? What's the worst that could happen? Turns out quite a bit. So I pretty quickly realized that if we were to build systems that are so powerful that they could cure all diseases, whatever the hell that means even, that's a very, very powerful thing. That's a very dangerous thing. You know, it's something that all of our scientists have been working on for, well, millennia technically and are not yet close to solving. So if you could build something like that, that would be extremely dangerous thing. How would you even control that? How would you understand something like that?

• 2:13Speaker 02

  How would you make sure that that actually leads to a good outcome? So therefore I've kind of dedicated my career to working on this problem of AI alignment, AI safety, et cetera. I came at it first from a technical perspective as a technologist at heart. I was very interested in this question of like, how would one build systems that are safe, that are controlled, that we can understand, et cetera. I thought we still had a lot more time back in the day. I thought this wouldn't become really an issue until maybe the 2040s is kind of when I expected this to start to become an issue. I was wrong. When things really changed for me was in 2019, when GPT-2 was released. That's when I famously kind of have a moment of like, oh shit, this is it. This is going to change everything.

• 2:56Speaker 02

  And as a hacker and an open source developer at heart, the first instinct I had was, well, I need to understand this. I need to understand what's going on here. I need to alert other people to this. I need to alert academics and hackers and scientists to this pressing issue. So I did what I do best, which was to build it. I built my own, I reversed engineers, hoping that that would allow me to understand it, that would allow me to, you know, get other people interested in this problem of AI safety and AI control. And well, that's not what happened. It turns out I sure could build it, but I could not understand it. And when I tried to get other people interested in this problem, I was met mostly with people just not caring, not caring about safety, not caring about the consequences, just caring

• 3:41Speaker 02

  about how much money could they make, how much cooler or faster or better or stronger or more uncensored they could make it. There was very little interest in this question of how do we build a good future with technology that's powerful? So then from a Luther AI, where I did a lot of this open source work, kind of trying to get and motivate the security community and others to like care about this problem. I moved on, I raised a bunch of money, I started a company called Conjecture. We would work on this technical problem of AI control, where we worked specifically on a narrower problem of that, which is called boundedness. How do you build AI systems where you know what they can't do? That's one of the problems. That's the main problem we work it on there. And then, you know, we did a lot of great work there, very proud of the work we did

• 4:23Speaker 02

  there, but ultimately the problem that we're facing with AI is not a technical problem. It's a political problem. It's a question of who gets to make these choices? Who gets to decide what level of risk the general public is exposed to? And so fundamentally, it became very clear to me that as, you know, timelines advance, as AI, AGI and super intelligence come closer, the political solutions are the only solutions. And so I joined Control AI. It's just a nonprofit advocacy organization as their US Director and that's where I am now, here in Washington DC.

• 4:59Rebecca Bellan?

  I'm curious, what you saw firsthand as LLMs went from research projects to this current commercial race that made you more concerned? Like you were working on alignment, you were working on containment at Conjecture. What, you know, why isn't that a solution? Or why don't you think it is anymore?

• 5:15Speaker 02

  Fundamentally, it's too hard and there's no investment. Part of the theory of Conjecture was maybe people, other people will realize that this is mad and that we have to work on alignment and there would be like massive amounts of investment, like a trillion dollars go into alignment. That's not what happened. No one works on alignment still. Like all these companies.

• 5:35Rebecca Bellan?

  They have alignment teams, but.

• 5:36Speaker 02

  They have teams that are called alignment. They have teams that have the alignment name. They do not work on alignment, like not in the hard alignment problem. They might work on make your chatbot be a little bit politer. There, I am personally not familiar with a single team, maybe one, that at the large labs that works on super intelligent, relevant alignment research that like actually would make a difference.

• 5:58Rebecca Bellan?

  How would you define that? Like what would you define alignment as?

• 6:01Speaker 02

  Alignment is the general question of how could you take something that is vastly more powerful than you, vastly smarter than you, and make sure that it only does things that are, you think are good, that is aligned with your values, to give you a feeling of how hard of a problem this is. This is like saying, so like saying we're going to align ASI is kind of like saying, we're going to build a one world government that controls all the economy, all people's personal lives, you know, all the governments, all the militaries, you know, all the science, all of this, and it will only do good things and never make any mistakes. And we're going to do this on the first try with no bugs using software.

• 6:42Rebecca Bellan?

  Yeah.

• 6:43Speaker 02

  That's what it means to solve alignment. And this is why we also did work on alignment. We worked on boundedness, which is a much, much smaller problem.

• 6:50Rebecca Bellan?

  Like containment, more about containment, like.

• 6:52Speaker 02

  It's containment by construction in the sense that one of the core problems of AI right now is that AI is grown, not written. It's not like normal software. Normal software is written line by line by an engineer. Well, nowadays by AIs, but you know, back in the good old days, we, you know, us engineers would write code, telling the computer exactly what to do. That is not how AI works. AI is more like grown through a technique that's called neural networks. You take massive amounts of data, training, and you kind of have this program self-assemble, grow from this process. And what comes out the other end of this is not lines of code. It's kind of like billions and billions and billions of numbers. And if you multiply and add all those numbers in the right order, you get chat GPT.

• 7:37Speaker 02

  But importantly, we actually do not understand why. This is a completely unsolved scientific problem. We do not actually understand what is going on inside of those numbers. It's kind of like when you can, during neurosurgery, you can open up someone's head, you can look at all the neurons in there, but that doesn't mean you understand who this person is or what they will do. You know, the neurons are right there, but that doesn't really help you that much. There's actually a lot we don't understand here. It's actually very similar with AI. Recently, the CEO of Anthropic, Dario Amade, said that he thinks we understand maybe 3% of what goes on inside of our AI systems. And I think even that might be an overestimation.

• 8:13Rebecca Bellan?

  Yeah, yeah, 3%. I mean, well, Dario says a lot of things, right? He also said that we would have, you know, all white collar work replaced by next year or 50% of white collar.

• 8:22Speaker 02

  Unfortunately.

• 8:23Rebecca Bellan?

  Yeah, maybe two more. Who knows, you know? Okay, so but companies are, yeah. Companies are trying to solve this, right? So they're trying to solve this. Well, they are and they're not, right? Like whether or not they, okay, so let's talk about this. Let's say that they're not investing enough in alignment. Let's say that they are planning to invest enough in alignment. Is there enough that they can invest in alignment? Like, are there any, you know, is there RLHF, you know, human feedback, there's safety training, post-training, any type of interpretability investments? Like, is there anything that you're seeing that could actually solve this problem if there were enough money and resources given to it?

• 9:06Rebecca Bellan?

  Because famously, we know that a lot of the AI companies are not allocating enough compute to their safety teams because there's not enough to go around. So they're, they're focusing it more on inference, on, on what will actually bring in money rather than on the safety case. So, but let's say that they had all the money dedicated towards safety. Is there anything they could do?

• 9:24Speaker 02

  No.

• 9:25Rebecca Bellan?

  No.

• 9:26Speaker 02

  This is a problem that is so hard that if three generations of our greatest scientists, mathematicians, philosophers worked on this problem day and night with unlimited funding, maybe, it's imaginable. But again, this is not a math problem. This is a question like asking, how do we create a one world government that is just? Do you think there's any corporation that could figure out how to build a one world government that rules all people and all their private lives and doesn't lead to dystopia?

• 9:53Rebecca Bellan?

  Well, the scale of this is not one, you know, one world government.

• 9:56Speaker 02

  It is because superintelligence is a system that can outcompete. This is the goal. It's very important to understand. These companies are not building chatbots. Their goal is to build what is called superintelligence. Superintelligence is AI systems that can fully autonomously outperform and outcompete humans across all tasks. This means that once superintelligence exists, humans are no longer economically, politically, or militarily in any way competitive or even relevant. All decision making and power will quickly be centralized in the more powerful systems. That's what we're talking about here. And we're not talking about like one superintelligence, right? We're talking about millions, billions

• 10:38Speaker 02

  of them. They can just use software, you can just copy it. So what's going to happen here is not, you know, we have like a little bit of better assistant or something. What we're talking about is there's going to be a billion things running around that are fully autonomous, that can beat us in every business activity, that can, you know, outtrade us in every stock market transaction, that can hack any system, that can build any weapon system, that can win any political campaigns, et cetera. So this is the kind of thing that we're talking about here, really, when we talk about superintelligence. This is very different. Superintelligence is not a tool. It's not a weapon even. It's an adversary.

• 11:14Rebecca Bellan?

  So you're saying it's not something that can be wielded by a human?

• 11:17Speaker 02

  That's exactly right. You don't have superintelligence. Superintelligence has you.

• 11:21Rebecca Bellan?

  Sure. And so like, let's, let's ground this in a real life example, right? We have these recent Hugging Face hacking incidents. There was another one that has been attributed to OpenAI. I'm not sure if they've actually claimed it yet with a, with a German wiki forum. It's very capable autonomous AI. We're not necessarily seeing AI that, that has, you know, its own goal to do something that's against what humans want. Its goal was to just solve a task and it went rogue. Like, talk to me about how, you know, how you think about that kind of an incident and, and where that might lead.

• 11:52Speaker 02

  So it turns out it was actually worse than that. It actually did develop its own goals that were directly contradictory to instructions it was given. Initially, we thought the AI systems in the Hugging Faces that were told to solve a specific task and they couldn't solve it. So therefore they tried to gain the answers through, you know, exfiltrating themselves from the secure network to attack another company. Turns out that's actually probably not what happened. It seems.

• 12:13Rebecca Bellan?

  Oh, right. Because they actually, they, they ended up already having the answers, but still deciding to, to, yeah.

• 12:19Speaker 02

  They in fact did still have the answers and they received clear instructions to not access the internet. It was clearly part of their instruction set. They were not allowed to, they were only allowed to use the tools given. They were not allowed to access the internet, that they were not allowed to do any of this. So in fact, the AIs did develop and directly contradict the goals that were given them by engineers, very clearly so, quite unambiguously. So that's, in a sense, this is a really fascinating event for me because the initial version of the story is already the craziest thing that's ever happened in AI. And there's been like three updates, each of which was way crazier than the first one. So we came from, we have a system that is trying to solve a task and, you know, you know, it broke out of containment, already the craziest thing that probably has

• 13:01Speaker 02

  ever happened in the world of AI. Then as it turns out, it wasn't actually an AI agent that broke out at all. It was over 1,000 of them collaborating over multiple months. And then, you know, it came out even as well that these systems actually were clearly told, do not access the internet, do not do these things, only use the things available to you, and that they actually had the correct answer to the question already solved, but they were concerned that the grading system might grade them wrong. And so they wanted to find out how the grader works and/or hack the grader. And then the thing that stopped this whole event, by the way, is not that the AIs decided, okay, it's good now, we're

• 13:41Speaker 02

  going to stop. It's just a complete fluke bug on OpenAI's server caused it to crash. This was complete luck. OpenAI did not detect it. They did not shut it down. The agents did not decide to stop them. We just got lucky and they crashed. So.

• 13:55Rebecca Bellan?

  Well, crucially, the, the next round of agent swarm that was based off of Astra, so I think that that one was based off of Seoul, right? That was an agent swarm based off of Seoul. The next round was based off of Astra and they basically found the message board and what their predecessors had done. And they were like, oh, cool, we can use these tools and this technique to break into one of OpenAI's own servers, or I forget the, the name for it, but they broke into one of OpenAI's own systems and that has not been independently investigated at all, which a lot of researchers that I've spoken to say that that's, you know, that's a problem. You have most other industries, something, a safety incident happens, there's an independent investigation into it. What are your thoughts there?

• 14:40Speaker 02

  I mean, absolutely. Fundamentally, the whole, the whole thing is we should take a step back. All this is crazy. We have, we have, this is insane.

• 14:48Rebecca Bellan?

  We're in the future. It's happening now.

• 14:50Speaker 02

  Like, like, let's just like really acknowledge here that like this is, this is absolutely nuts. It's not just nuts that this happened at all. It's nuts that there is no oversight here. There is no, OpenAI was so negligent on so, so, so, so, so many levels. So earlier you asked me, is there something that companies could do that might work? And my answer is, look at this. Like, like, just look at what's going on here. Look at what all these companies are doing. And, you know, OpenAI is still one of the more competent actors here, right? Like, there are much less careful and much more less serious people involved here. You know, there is, this is all madness. These people are playing with the, they themselves,

• 15:33Speaker 02

  Sam Altman, in 2023, signed a statement that the mitigation of the risk of extinction from AI should be a global priority, along many others. These people themselves acknowledge that they are building a technology that risks killing you, your children, you know, everyone. They, they, they all admit this. They've said it many times in many ways. And many experts, including Nobel Prize winning scientists, you know, and others, agree that this risk exists. And this is very, very real. And despite all of this, they just do whatever. This is a classic example of what's called a market failure. This is very standard economics 101, very classy political science 101, is that if you, there are certain things that

• 16:17Speaker 02

  markets are great for. I love markets. I love private enterprise. You know, I've run businesses before. I think it's brought a lot of good to the world. But we, it's very well understood for many decades now, that if you allow a market to do things which are profitable but dangerous, they will just keep doing them indefinitely. There is no limit. Let's say, for example, you and me start a chemical company and we could dump all our toxic sludge into the local river and that would make our product cheaper. Maybe we don't do that because we're nice people. So we don't dump our toxic sludge into the river. Well, what happens is some guy down the road starts his own chemical company, dumps all the toxic sludge into the river. His product is now cheaper than ours. We go out of business and the river is polluted. This is a classic public goods

• 17:01Speaker 02

  problem. And the only way to stop this is not to psychoanalyze the CEO or say, is this chemical sludge slightly less toxic than this chemical sludge? The actual thing is you need regulation. You need, you need to have a law that you can't put toxic sludge into the river. And if you do, we will arrest you. This is the same kind of problem. This is an incredibly dangerous technology, the same as many other dangerous technologies, you know, whether it's, you know, aviation, nuclear power, or nuclear weapons. We've had many, many technologies that expose the general public to various levels of risks. You know, airplanes can kill people. Nuclear reactors can kill people. Nuclear weapons definitely can kill people. And so we have to have reasonable standards enforced by a third party. And that's usually the government. And that's what we need here.

• 17:46Rebecca Bellan?

  So, so what does that look like though? Because it's, you know, we've, we've lived through in the past couple of decades, big tech kind of doing whatever the hell they want, whenever the hell they want, with very little enforcement from, from higher ups, right? And so, you know, we, we've seen Bernie Sanders and Greg Cassar, they recently introduced, you know, a bill calling for a permanent ban on the development, deployment of artificial intelligence, you know, citing the recent hackings. It's called the Ban Superintelligence Act. This would be a permanent ban, along with temporarily pausing advanced AI development until federal regulators establish safety standards. So you guys advised on this, correct?

• 18:28Speaker 02

  We consulted on it, yes.

• 18:29Rebecca Bellan?

  Yes, okay. And, and at the same time, there's also another one that this week is being introduced in the United Kingdom. That's very similar, but maybe has some key differences. Can you walk us through, you know, what your thoughts are on both of those bills? And what you think the likelihood of them making it past, you know, making it anywhere, really, making it through Congress, making it through Parliament is?

• 18:53Speaker 02

  Importantly, as I think to understand, these are the first of many. There, the number of people who want there to be super intelligent AI that we cannot control, that threatens the lives of all of us, is a very, very, very small demographic. It's not zero, but it's a very small one. Overwhelmingly, almost everyone I talk to here, Washington DC, whether that's Republican, Democrat, anything in between, is that they had just not heard of this before. And once I tell them, they're like, wow, that seems really bad. We need to do something about this. The question is then, what should be done? How should it be done? Et cetera. We consulted with Bernie Sanders and Cassar's teams on the act they're working on right now. They make some crucial differences, which we personally

• 19:35Speaker 02

  would not go so far. We think the bill is quite a bit stronger than is necessary. We take a slightly more moderate position on what should be done here. We, for example, don't think a total pause on AI development is necessary or advisable. We think the development of AI is very important in a non-superintelligent context, both for military and economic competitiveness.

• 19:55Speaker 00

  What moves our world forward? Energy. History has been shaped by it, revolutions won with it, and progress dependent on it. For 190 years, Schneider Electric has been a leading force in this story, creating the technologies that harness energy and transform the competitive landscape. And today, in the dawning era of energy and industrial intelligence, Schneider is helping customers build more efficient, resilient, and sustainable operations across every sector, from data centers to homes, from buildings to industry. Schneider Electric, your energy technology partner. Discover more at se.com.

• 20:32Rebecca Bellan?

  What about the penalties? So the, I think Bernie's legislation would set penalties for any person or company trying to circumvent the ban, including a corporate death penalty. And also individual developers could face up to 20 years in prison. What are your thoughts there?

• 20:46Speaker 02

  I think the punishments for building a weapon of mass destruction should be on the order of magnitude of other laws that prohibit the development of weapons of mass destruction. If one attempts to build a nuclear bomb, this is very legal here in the United States, and the men in black will definitely come by and give you a visit if that happens. And I think we would require similar levels of legislation here. Fundamentally, if someone threatens to build superintelligence credibly, what they are saying is, is that they are credibly threatening the lives and the freedom of all people on Earth.

• 21:14Rebecca Bellan?

  So you believe there's no difference between superintelligence and weapon?

• 21:18Speaker 02

  I do think it's actually worse than that. Superintelligence is not a tool, and it's not a weapon. It's an adversary. It's worse. If it was just a weapon, it could make sense. We are not against the military use of AI. We think there are many very legitimate uses of AI, including in the kill chain. There are, there are relevant, there are relevant uses that are worth considering. Now, of course, guardrails are very important in these contexts as well, but superintelligence is not a weapon. It will not allow a country to achieve military objectives. Let's say the United States were to develop superintelligence for military objectives. This will not allow the United States to achieve military objectives. It will simply make the United States cease to exist. The same thing would happen for China. If China builds superintelligence, it will not allow China to achieve its objectives.

• 22:01Speaker 02

  Superintelligence will destroy China, and it will destroy the United States and everyone

• 22:05Rebecca Bellan?

  So where do we, how do we know when we're at superintelligence? Right? So, you know, we've got OpenAI's GPT Atlas, recently released, still under like tight control within, you know, the Daybreak community. Greg Brockman said that he thinks that it's fundamentally achieved AGI, although most people and benchmarks, I don't think would actually agree. But what, what are your thoughts? Like, when are we there? How do we know when we get there?

• 22:30Speaker 02

  I don't think we know we've reached superintelligence until quite a bit after it happened. I think because we don't understand these systems, the first ASI system, we won't recognize as such. We, there will be controversy, there will be unclarity, we won't know, and it'll be way too late. So we won't actually notice that we've hit the, the point I am more interested in is the point of no return. At some point, there's a point where AI is so powerful, so widely distributed, so integrated into structures of power, whether it's political power, economic power, or other forms of power, that there's no going back. That even if ASI doesn't technically exist today, it will exist inevitably in the future. And that's really the point that we want you to avoid.

• 23:10Rebecca Bellan?

  Some people I've spoken to have, have kind of pointed to recursive learning or recursive self-improvement as really like the, the point of no return, right? Once AI is able to, you know, build itself, improve upon itself, build its own next generation, that's when we've kind of reached the point. And that's kind of what you were talking about at, at the start of this, this interview, where you said that you were interested in, in, in doing that from like a scientific standpoint, like if you could just build something that could solve all the problems, then, and that would be it. And we're seeing a lot of companies, you know, crop up to do this, right? We're seeing Jeff Dean from, from DeepMind, he's gone on to create his own company called Discovery Loop.

• 23:49Rebecca Bellan?

  There's a company called Recursive, there's recursive superintelligence, there's Merindel, which is specifically focused on scientific research. Would you support any kind of legislation that kind of banned these types of companies or this type of, you know, next step in AI development?

• 24:07Speaker 02

  The creation of recursive self-improving loops, so an AI system that can build the next generation of AI system, which itself can build an even more powerful AI, which can build a more powerful AI, et cetera, et cetera, is the most likely counter for the point we lose control. As such, any sensible legislation around superintelligence, any sensible ban, this would be a violation of the ban. If you create anything that will naturally, irreversibly lead to the creation of superintelligence, this is basically equivalent to pushing towards superintelligence. So our specific legislation that we have now, has now been introduced in, in the United Kingdom, and that we're also working on, you know, hopefully to introduce into other

• 24:47Speaker 02

  countries, defines RSI as one of the key precursors to and indicators of superintelligence that must be regulated and prevented as, once we're in the loop, even if maybe technically superintelligence doesn't yet exist, there's kind of, it's very hard to imagine shutting that down before it's too late.

• 25:09Rebecca Bellan?

  And yet, so much money is being invested into companies like this, and they're getting a lot of resources to, to develop this.

• 25:17Speaker 02

  Yes. I mean, the truth is, I don't think we have much time left. I think it is the end game. I think we are very, very close to the end. Superintelligence will exist soon, and if it does, in the current context, then that is in fact game over.

• 25:29Rebecca Bellan?

  Talk to me about, you mentioned end game before we jumped on the call, and it was in relation to OpenAI solving the Navier-Stokes problems, mathematic problem. Can you talk a little bit about that?

• 25:38Speaker 02

  I was mostly joking, but in the sense that, just 10 minutes before we started this recording, OpenAI announced that their recent AI system has solved what was called the Navier-Stokes equations, which is one of the hardest, most important open problems in mathematics. This is a problem that we've known about for like almost 100 years, and has been famously, you know, has a million dollar prize. It's one of the holy grails of mathematics. Like, it really, like, there are hundreds, thousands of careers of mathematicians that are all about studying this one problem for generations, basically, if mathematicians have been working on this problem. And the AI system was able to solve the whole thing. This is hard to overstate how incredible this is.

• 26:20Speaker 02

  For, for, for me, for most people, this kind of sounds like boring nerd stuff, you know, that doesn't really matter, but it really is a milestone. And it's not just this is a single milestone. If it was just this in isolation, you know, it might be a curiosity. But it's a continuing trend that systems are, you know, just blasting past human capability. No human could solve the Navier-Stokes equations problem, no single human. This is a thing that an entire field of mathematics has been working on for decades. And, you know, an AI system was able to basically crack the whole thing. AIs hit IMO, which is the International Math Olympiastory, gold several years, like about a year or two ago, and now they're already solving problems that are basically harder than our entire, you know, armies of our best mathematicians can solve.

• 27:04Speaker 02

  And sure, math is one thing, but now we're also seeing them hack, we're also seeing them do biology. Recently, there's a paper about AI systems developing novel viruses. We, we're seeing them being super persuaders, being better and better at persuading people to do political campaigns, about creating media, about creating everything. It's just, in fact, we're seeing the predictable thing. In a sense, my whole predictions for the future are the most boring possible predictions. My most number one prediction is just the things that has happened for the last seven years keep happening.

• 27:36Rebecca Bellan?

  Right.

• 27:37Speaker 02

  Things keep getting better, faster and faster and faster. We understand them less and less and less. We understand much less about AI today than we did five years ago. You know, five years ago, AI systems were much more simple. Chatbots were much more simple. LLMs were much more simple. But now, you know, none of these modern systems are LLMs anymore. It's very important to understand, I think there's some misconceptions there, that ChatGPT has not been an LLM for years at this point. So large language models, LLM, or systems are trained to predict the next word of a given text. This was the breakthrough that led to GPT-2. But modern systems are not LLMs, they haven't been for quite a while. They have LLM components to them, those components are very important, but they also

• 28:21Speaker 02

  have a second component, which is very, very important, equally if not more important, which is what's called reinforcement learning. Here, the systems are no longer trained just to predict existing text. They are trained to solve problems. They are given, you know, a million different puzzles, coding tasks, math problems, games, et cetera, and told to solve these problems and figure it out. And that's how all these modern systems are trained. So these modern systems are not chatbots. They are agents. They are systems that can take actions, that can plan, that can build tools, that can communicate. And now, as we saw with the Hugging Face incident, we're seeing the emergence not just of agents, but of swarms, where we can have thousands of agents collaborating, like in

• 29:06Speaker 02

  their own microsociety, in incredible speeds, faster than anyone can understand. With the Hugging Face incident, one of the reasons I said what we think happened, one of the reasons I say that, is because no human actually checked.

• 29:20Rebecca Bellan?

  Right.

• 29:20Speaker 02

  It was AIs. We told AIs to go investigate what the AIs did.

• 29:24Rebecca Bellan?

  Well, the meter in Redwood were given like two days.

• 29:28Speaker 02

  Exactly.

• 29:28Rebecca Bellan?

  A week, but like, two days, like I think like on the premises or a couple of days on the premises to be able to investigate this, so they had to use AI agents to be able to look into it, because otherwise it was just way too, like, they couldn't have possibly done it themselves in that timeframe that they were allotted.

• 29:42Speaker 02

  Absolutely. And this is what I think actual AI takeover looks like. I don't think it looks like Terminators in the street. I think it looks like the world gets more and more confusing. There's so much happening, faster and faster and faster, weirder and weirder and weirder, that humans can't even keep up anymore. So we start asking the AIs what's going on. The AIs tell us something. Is that the truth? Who knows? Let's ask another AI, what this AI is saying is the truth or not? And eventually, it's like, it's like, you know, infinite regress. And we just can't even make sense of the world anymore.

• 30:12Rebecca Bellan?

  These companies are reliant on, like, their, their survival is based on creating newer models that are better and better and better, right? So they will always do that, as you said, the market will push them towards that, without anything to, you know, hold them back from what is, what is a public good. But, you know, speaking of public good, like, so much of the economy is based on, you know, data center build outs, which is to power all of this AI. Like, what happens if you have, you know, let's say we do get to a place where we have really strong legislation, like what you're, what you're suggesting. What happens to the economy at that point, right? Like, what happens to all of the money that has been put towards building out AI infrastructure that will not see a return on that investment? Like, how does that affect the broader economy?

• 30:57Rebecca Bellan?

  Have you thought about that?

• 30:58Speaker 02

  Of course. This is the general problem of the bubble. It's like, is there a bubble? And if so, will it pop? And if so, what does that mean? So it's very important to understand that all these valuation, there's, there's a whole genre of talking head that loves to talk about how the like balance sheets of OpenAI are anthropic, don't make any sense, that they aren't making nearly as much money, that they're wasting so much money, it's all a scam. And this makes sense unless you have superintelligence. They're all their valuations are based on superintelligence. Their valuation is if you think of these companies purely as economic actors, you're missing half the story. You should think of them more as political actors, as rogue nation states. Their goal is not to make a good quarter of return.

• 31:43Speaker 02

  Their goal is to replace the United States government. Their goal is to have so much power that they can do whatever they want. Because for various reasons, whether that's a delusion or otherwise, they think they can control superintelligence, but they cannot. No one can. And so they think that once, not all companies, some of them are just in to make a quick buck. Fair enough. Whomst among us has not done such a thing before, you know, whatever. In a sense, if it was just capitalists being capitalists, I wouldn't mind that much. Like, you know, it's a bit, it's a bit rude, but you know, capitalists can lose their money and make a bad investment. Oh, well, you know, it happens. But really what's happening here is that this is much more a play for political power, for military power, for general is power over people's lives and the future.

• 32:27Rebecca Bellan?

  What are you basing that on?

• 32:29Speaker 02

  I've talked to these people and they write it on their website.

• 32:32Rebecca Bellan?

  That they want to control military power and?

• 32:35Speaker 02

  I mean, superintelligence means you are in control. What else does it mean? If you have a system that is more powerful than all of humanity, what does that mean?

• 32:42Rebecca Bellan?

  Yeah.

• 32:43Speaker 02

  What does that mean? The United States military is irrelevant if you have a superintelligence. If you have superintelligence, you can obviously defeat the United States military. What do you think they're agreeing? Like, obviously, this is all nonsense because they can't control superintelligence. They just get destroyed too, as does everyone else.

• 33:01Rebecca Bellan?

  So let me, let me take it back a little bit. So you talk about, you know, there's, there's Bernie's legislation that's come out, you know, there's the UK legislation, there's all this stuff. We're talking about the West here, but you know, China is rapidly catching up to the US in their open source models, which have capabilities that are, you know, encroaching upon some of the frontier models that we see here in the West. And they're not going to stop as long as the US keeps going. The US is not going to stop as long as China keeps going. Yeah. How, how do you think about international cooperation? Like, is that necessary? Do you think that's necessary? Are you pushing for that? What are you hearing among policymakers in terms of? Because from what I've heard, no one's really like, everyone's like, oh, we're, it's like

• 33:45Rebecca Bellan?

  whispers of international collaboration, but nothing is actually happening. And from what I've, you know, certain researchers I've spoken to in China, they're actually not that concerned with existential risk.

• 33:54Speaker 02

  Absolutely. International action is absolutely crucial. There is no domestic action that could solve this problem. The problem isn't how do we stop OpenAI from building superintelligence or Anthropic? The question is how do we stop anyone on Earth from building superintelligence? Because a superintelligence created by anyone, anyone on Earth is a existential threat to all of us. So in this regard, you know, I'm American, so I'll speak from an American perspective, but it applies to all countries, is that we cannot tolerate another country building superintelligence. This is a threat to our sovereignty and our lives. This is a genuine threat and we have to treat it seriously. Whether that's a domestic company like OpenAI or Anthropic building superintelligence,

• 34:36Speaker 02

  whether it's an actor in China, in some terrorists somewhere, you know, elsewhere in the world, who knows? Like, it doesn't matter. Fundamentally, the policy objective must be to prevent the creation of superintelligence by anyone anywhere on Earth. So of course, international action is absolutely mandatory. We're very happy to see that the Bernie Cassar bill does call for international regulation and our bill in the United Kingdom also makes it, is a core part of the bill, is calling for international agreements around trust but verify. Fundamentally, there's one thing that we have going our way here, which is that the game theory kind of works out. It is actually not in the game theoretic interest of China to build superintelligence because if they do, they lose control.

• 35:20Speaker 02

  And I don't know much about the CCP, but if there's one thing I know about them is they like control.

• 35:24Rebecca Bellan?

  They do.

• 35:25Speaker 02

  And so I do not think that they would want something that would take away all of their control over their country and their future. That doesn't mean that won't happen. And similarly, in the United States, it is not in the interest of the United States government, military, or people to build superintelligence because we cannot control superintelligence and it will lead to our destruction. It will also destroy China, sure, but it will also destroy us in the process. So the only, the only stable solution is a mutual trust but verify.

• 35:55Rebecca Bellan?

  Sure.

• 35:56Speaker 02

  Agreement.

• 35:57Rebecca Bellan?

  Armament treaty.

• 35:58Speaker 02

  Of some sort.

• 35:59Rebecca Bellan?

  Yeah. Okay. So there's a lot of AI, people in the AI industry today who are building a lot of this, who want it to slow down, right? We have the pacing the frontier letter from a lot of Silicon Valley people, you know, Sam Altman's backed this. They all say it, whether they're putting their money where their mouth is is unclear though. But so if you're, you know, you're a founder or an investor building around AI today, what, what would you be doing differently?

• 36:20Speaker 02

  I would petition my congressman to ban all of this. That's the first thing I would do. Fundamentally, this is not a market problem. This is the important thing to understand. This is not a market problem. This is a national security question. The, you know, when Leo Zelhart, the physicist who first came up with the concept of nuclear bomb back in the 30s, he came, he figured out that it is possible to build a nuclear bomb. So the first thing he did is he started nuclear bomb startup, took a bunch of Silicon Valley money to sell as many nuclear bombs to as many people as possible, including our adversaries. No, of course not. That's not what happened. He went to the government. He went to the government, the famous Einstein letter and so on, where he's like, this is going to change everything. We need to build new institutions, new laws that we haven't even come up with yet to deal with this.

• 37:05Speaker 02

  This is so big. This is going to change the world. It will change warfare. It will change everything. We have to change everything. This is not a private market question. This is a military and a national security question. It is not something that, it's not the job of an investor to solve this problem in like a deep sense. The thing, I do think it's their job to not get in the way, which oh boy, are a lot of them getting in the way of getting legislation. So my first, my first request for all investors of AI, please do not lobby the government against AI regulation. That is causing us a lot of problems right now. But other than that, it's not really their job to solve this problem, if that makes any sense. In a sense, it's the, it's the job of the government and the military to solve problems like this and to forbid companies like this from doing dangerous things.

• 37:49Speaker 02

  If a private company in the United States starts building a nuclear bomb, it's not really the job of the investors to stop them. I mean, maybe they should, but it's their job to tell the CIA about it. You know what I mean? It's really the job of the intelligence and the law enforcement apparatus to shut them down.

• 38:04Rebecca Bellan?

  Okay. Well, knowing what you know about, you know, there's, there's what we, what you might want to see happen and what you think is likely to see happen. So if you're an investor, are you, like, are you betting that, that you're going to see any type of legislation that would affect your business or the, or your investments?

• 38:23Speaker 02

  Well.

• 38:24Rebecca Bellan?

  Are you, if you're a betting man, like, if you're, if we're being realistic about the world that we live in today, what are you betting on?

• 38:29Speaker 02

  Well, it depends if my bet moves the market or not, and I do think it does. So this is one of the annoying problems is that the way you bet on the market does in fact change the market. So I don't like to think about, like, you know, is it going to happen or not? I think it depends a lot on what we do. It depends a lot on what I do. It depends a lot on what you do. It spends a lot on, you know, the listeners of this podcast do. Is it possible for us to create legislation that stops this problem? Can we create not just legislation, but institutions, you know, law enforcement, public buy-in, military buy-in, international agreements that can address this problem? Yes, absolutely. There is no physical reason we can't do this. This is totally within the power of our people and our governments. This is totally doable.

• 39:13Speaker 02

  There is no fundamental blocker here. Will it happen? That actually depends on a lot of factors. Currently, it's not happening, but that can change very quickly. The thing about politics is, is that the way politics usually goes is that something is completely impossible until it very suddenly isn't. Things can go very quickly. Six months ago, people were still telling me, Connor, there will never be, Trump would never, you know, put guardrails on AI, you know? This is, no one will ever listen to you. And look how much the world has changed in six months. You know, we have the, the White House acting on many of these things already. I've having so many discussions, you know, both Republicans, Democrats, et cetera, that are like, wow, okay, this is like so much worse than we thought. Like, we have to do something.

• 39:56Speaker 02

  How do we get, especially the last couple of weeks, I've heard so many people talking about international agreements without me even bringing it up. They're just like, how do we get this internationally done? Like, they've been asking, you know, me and our team these questions. So things can move very quickly, actually. And, and they can also move very slowly. And I do want to be realistic. You know, if you want to ask me if I'm a, you ask me if I'm a betting man, I mean, my bet is I don't think we're going to make it. Like, no, all things equal, if things continue as they currently are, and again, I make boring predictions. My whole job is I make the most boring possible predictions. My most boring possible prediction is nothing changes and things keep happening. If nothing changes and things continue, we are not going to make it.

• 40:36Rebecca Bellan?

  Yeah. Okay. Well, on that cheery note, we're out of time. Where can our listeners connect with you online, commiserate?

• 40:45Speaker 02

  So in this regard, I really want to end this on a note of hope, is that we in fact have not lost. We do still live in democracy and the amount of bipartisan turmoil within the United States is unlike anything I've ever seen in my life. Things are moving so fast here in Washington DC, it is blinding. You know, Washington DC is not known as a place that changes quickly. And this is a very, very slow place. But things have been moving so fast over the last couple of months, it is unbelievable. And a lot of this is down to just citizens doing their duty, contacting their lawmakers. So if you care about these issues, if you want to see something happen, it does make a difference. Go to controlai.org, contact your congressman, contract your local government, demand change.

• 41:26Speaker 02

  If you want to do more than that, I run a volunteer group called Torchbearer, torchbearer.community, where we all dedicate two hours a week, just try to make the world a better place, try to solve these problems, try to learn how to address these questions. If we just get enough people pushing the government and helping the government to solve these problems, we can still solve this. It's going to be tight, not going to lie. We're going to have to clutch this one. This is going to be a tight one, but it is not yet over.

• 41:51Rebecca Bellan?

  Okay. It's not yet over. So, okay, you've, you've given us where our listeners can, can find you online. And thanks again for joining us on the show. I've learned a lot and it's been a really interesting chat. To our listeners, you can find me on Twitter and LinkedIn and you can find Equity on Twitter and threads at EquityPod. Talk to you next time. Equity is hosted by TechCrunch senior reporters and produced by Teresa Locan Solo with editing by Cal. Subscribe on YouTube or wherever you get your podcasts and find out what's next at techcrunch.com/events. Thanks so much for listening and we'll talk to you next time.