Pod Insights Odd Lots Anthropic, the Pentagon, and the Future of Autonomous Weapons About this episode The last big story right before the war in Iran started was the collapse in the relationship between the Pentagon and Anthropic, with the latter objecting to any potential use of its models in either fully autonomous weapons or domestic surveillance. Of course, this story immediately become more relevant with the start of the war, and the reporting that Anthropic's technology was in fact utilized at the start of hostilities. But what does that mean? How are these models used? And what would a fully autonomous weapons system actually entail? On this episode, we speak with Paul Scharre, the executive vice president and director of studies at the Center for a New American Security. He has written two books on the subject of AI in warfare, and previously worked inside the Department of Defense on some of these very questions. We discuss the future of autonomous weaponry, and the various ethical and technological dimensions such weapons would entail. Subscribe to the Odd Lots NewsletterJoin the conversation: discord.gg/oddlotsSee omnystudio.com/listener for privacy information.
Transcript 10,502 words · 7 speakers
Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Joe Weisenthal? There are some market stories where you want every detail. Tracy and I have made quite a few podcasts on that basis, but sometimes you've only got 10 minutes and need to know what's moving markets and why. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and research get you up to speed on what matters and what to watch next. About the time it takes to grab a coffee, so search Barclays Brief wherever you get your podcasts.
0:26 Speaker 01 This is the Bloomberg Tech Minute brought to you by ChatGPT. I'm Carol Muster. Getting hired at SpaceX requires exceptional ability and may include having Elon Musk review a candidate's resume. So writes Bloomberg Business Week's Lauren Grush. She reports that the company culture is defined by Musk and job candidates must understand that a commitment to SpaceX means adhering to this culture and exhibiting loyalty to Elon Musk. To get hired, candidates must demonstrate their achievements from past jobs and hands-on experience matters more to recruiters than advanced degrees, with the company looking for people with raw, innate intelligence. A spokesperson for SpaceX didn't respond to a request for comment on its recruiting
1:07 Speaker 01 and hiring practices, but according to recruiters, advisors, and those who've interviewed for jobs there, it's the space exploration dream that draws people to the company, and Musk invokes that dream regularly to remind workers why they're there. Prospective hires are told that the job will require stamina, including in job ads that candidates, quote, "must be willing to work extended hours and weekends as needed." One major perk for all that hard work? Employees' access to equity, either the right to buy shares or payment in the form of restricted stock units that vest over time. That's the Bloomberg Tech Minute brought to you by ChatGPT. Get started at chatgpt.com.
1:51 Speaker 01 Bloomberg Audio Studios. Podcasts, radio, news.
2:06 Joe Weisenthal? Hello and welcome to another episode of the Odd Lots Podcast. I'm Joe Weisenthal.
2:11 Tracy Alloway? And I'm Tracy Alloway.
2:13 Joe Weisenthal? So Tracy, we're recording this March 24th, and of course, almost all of our episodes lately have been about the war in Iran. But what's interesting, or what's a little weird, is that just prior to the war, literally days or maybe hours, the biggest story in the world was actually about defense and, you know, the DOD.
2:32 Tracy Alloway? That's right. So you are referring to Anthropic.
2:35 Joe Weisenthal? Yeah, yeah.
2:36 Tracy Alloway? And its disagreement, to put it mildly, with the Department of War?
2:40 Joe Weisenthal? Yeah, exactly. This was the biggest story going right up on the eve of the war in Iran. And of course, obviously, there was this contract and Anthropic's technology was used by the Defense Department. So it was not a disagreement about the use of AI per se in war, but the question of the degree to which AI could be used for autonomous weapon systems on their own, without a human in the loop.
3:04 Tracy Alloway? And surveillance. That was also part of it.
3:05 Joe Weisenthal? And surveillance. This was another key element.
3:07 Tracy Alloway? But you're right. So we've heard this expression, autonomous weapons.
3:10 Joe Weisenthal? Yeah.
3:11 Tracy Alloway? Pop up more and more, especially in recent days. And I have a lot of questions over what exactly that means.
3:17 Joe Weisenthal? Same.
3:17 Tracy Alloway? Because my impression is the US military certainly has been using AI for some time.
3:22 Joe Weisenthal? Yes.
3:22 Tracy Alloway? And so we're really talking about degrees here of autonomy, right?
3:27 Joe Weisenthal? And so. Yes.
3:28 Tracy Alloway? If you, if you think about an autonomous weapon, I think your mind could go fully Terminator. And, you know, there's like a murder robot out there.
3:35 Joe Weisenthal? Yeah.
3:35 Tracy Alloway? That's making its own decisions on which people or places to target. And then you get levels below that, right? Where AI is kind of helping humans to come up with strategic decisions.
3:48 Joe Weisenthal? Right. So if there is a missile coming and you have a missile defense system. I don't think you want a human in the loop is like, okay, here are the coordinates XYZ that we think it's going to hit. At this point in time, we think the missile will be here. Are you cool with firing it? I think everyone's probably okay with that level of autonomy. But I have a feeling that, to your point exactly, a lot of this discussion, and maybe it's core to what Anthropic and the Department of Defense were disagreeing on. I have a feeling a lot of this is going to come down to definitions. My guess is that there is not one shared agreement of this is an autonomous weapon system, and this one is not.
4:28 Tracy Alloway? Absolutely. And of course, there are also questions over exactly how places like the Department of Defense, not only how they define it, but once they have those definitions, whether or not certain companies trust them to sort of stick to those policies. Because the US will say, well, our policy is not to survey our citizens at the moment. So if you're Anthropic, you don't need to worry about that. Clearly Anthropic feels otherwise, or say they do. So there are all these really interesting thematic questions that pop up from all of this.
5:01 Joe Weisenthal? Totally. And then there's the question of, okay, here is a technology and the government says, we believe that we can use this to make the country safer. What, you're not going to let us do it? Like private corporation, there's some very interesting questions about the role of corporate power vis a vis the government and so forth. Anyway, this is something that has become even more timely. There were reports even in the early days of the Iran war about these AI systems having been used perhaps in target selection. But we don't really know. None of the reporting is like that clear. I don't think that they're going out and advertising the strike.
5:33 Tracy Alloway? This is exactly how we're using AI.
5:35 Joe Weisenthal? This is exactly how we're using AI and so forth. But this is obviously a huge debate and war aside, it's only going to grow. And just as AI is going to grow, it seems in so many different areas. Anyway, I'm really excited to say we really do have the perfect guest, someone who's been writing and thinking about this stuff for a long time. When we talked to an AI expert, I marked the dividing line. It was like, were you talking about AI prior to when ChatGPT was released? It was like, I take a little bit more seriously the people. were in this space prior to November 2022. Anyway, I'm very excited to say we're going to be speaking with Paul Shari. He's the Executive Vice President at the Center for a New American Security and he's the author of two books related to this. One is the most recent, Four Battlegrounds: Power in the Age of Artificial Intelligence.
6:20 Joe Weisenthal? And then prior to that, he is the author of Army of None: Autonomous Weapons and the Future of War. He was previously in the Office of the Secretary of Defense, also a previous Army Ranger, so truly the perfect guy. So Paul, thank you so much for coming on Odd Lots.
6:35 Speaker 03 Oh, thank you for having me. Very excited to be here.
6:38 Joe Weisenthal? Why don't we start? I mentioned I had a feeling that maybe the definition of an autonomous weapon is a contested one. But if I say to you, what's an autonomous weapon? What's an autonomous weapon?
6:50 Speaker 03 So I think you're right from the beginning that there is not a unified definition that everyone agrees on. The Defense Department has their definition that's written in their policy. I think conceptually, I think the distinction really is a weapon that is choosing its own targets on the battlefield. And it's not where we are today, right now, today, people are choosing those targets. But it is kind of a spectrum because we do have examples of weapons that have some measure of autonomy. A good analogy might be self-driving cars where conceptually, like, okay, a self-driving car would be where the AI is driving the car. But then you get into an actual car today and a lot of them have intelligent cruise control, automatic braking, automated self-parking.
7:33 Speaker 03 They have all these like automated features that are kind of creeping you in this direction where the AI is taking over more and more control for what the vehicle can do. And it's actually a pretty similar thing in the military space as well.
7:46 Tracy Alloway? So Joe mentioned that had we been having this conversation even a month ago now, it probably would have had fewer concrete examples of AI-enabled weaponry, let's say, or strategy. When the Pentagon talks about its advanced AI tools that it's deploying for the Iran conflict, what are some examples that you're seeing right now that are different to, say, maybe just a year ago when we had another Iran conflict?
8:15 Speaker 03 Right, so there's a couple ways in which the Pentagon is using AI right now. One is narrow AI systems that have around for over a decade now that do image classification, for example. So this was the military's original project, Maven, almost a decade ago, where they took machine learning image classifiers to sift through drone video feeds and satellite images to identify objects. Okay, here's a building, here's a person, here's a vehicle. That's pretty mature technology. Now, what's come out in just the last couple of weeks that's really quite interesting is that in the midst of this huge messy public breakup between Anthropic and the Pentagon,
8:55 Speaker 03 we found out that, in fact, Anthropic's AI tools are being used by the US military to help plan the war against Iran. That's obviously a different kind of AI tool, AI large language models, AI being used to write code, AI agents, and that's being used in a different way to really helping intel analysts sift through just the massive amount of data that the US military has. And so if you can imagine the problem that the military is facing right now, when they're looking at targets in Iran, US military has flown over 6,000 sorties against Iran. The Iranian military architecture is degraded in a lot of ways. US military has already bombed a lot of targets.
9:38 Speaker 03 There are mobile targets, senior Iranian commanders, mobile missile launchers, and air defense systems, and drone launchers. US military's got to bring all that information together and find out where are these targets right now, and where is there an aircraft that has the right bombs on it to take these targets out? And that's how AI is being used to help basically process and understand all that information.
10:02 Joe Weisenthal? When I think about the description that you gave for that, I sometimes think, like, could it be that, now, I don't think that using Anthropic's technology means that they go into Claude.ai and say, "Give us a list of suitable targets for sorties." But it could be, could it be something like that? But I'm sure there's a different interface and so forth. But is that a completely ridiculous way of essentially framing the service that AI is providing right now?
10:31 Speaker 03 So the way that these AI tools are being integrated are through a existing system called the Maven Smart System. Okay. Which is built by Palantir and that fuses all this data together. So you basically have an existing architecture for data management for intel analysts that the military has that brings together all these different forms of data. You might have satellite imagery, geolocation data, signals intelligence, other forms of information. That's pretty great for intel analysts, but that's also really unwieldy because how does a human understand all that data and process it? And that's where the large language model tools, whether it's Claude or other companies
11:11 Speaker 03 can be valuable, is that could be a way for a human to interact with that data, to basically task a large language model to say, okay, here's a bunch of data I'm giving you. I want you to look for intersections in things, right? I want you to look for a place where we have satellite imagery and some other forms of intelligence that can help identify the location of, you know, some missile launcher, for example. And then humans can look at that and help, one, just like find where are all these targets? And then it's helpful in planning, too. A human could say, okay, here's this list of potential targets that I have. Now, they're scattered all over Iran. Iran's a really big country.
11:54 Speaker 03 I want to map these two locations for US aircraft at different bases across the region. What are available aircraft and what are available munitions on those aircraft that can be used to take out those targets to help build a strike package? So like, the AI is definitely being used to help understand the battle space and to plan operations, but in, I would say, ways that are pretty narrowly directed by people. It's not quite as simple as like dump all this data into a context window for LLM and say, oh, AI, figure it out. Okay. People are asking the AI some really specific questions.
12:28 Tracy Alloway? So I'm thinking how to phrase this question diplomatically, but I get that the difference between fully autonomous weapons is, you know, the human as a decision maker. In the current setup, how meaningful is the human actually? Like, what's your sense of it? Because I'm imagining if you're an intelligence officer and you're getting reams and reams of data from Iran and you ask the AI to pick out certain patterns or identify potential strategic targets, how much due diligence are you actually doing on what that model spits out? Because of course, the tendency when a lot of people use LLM certainly is just you accept what it shows you on the screen.
13:08 Joe Weisenthal? And just to add on to Tracy's question, because this is where I want to go, which is that in the early days of the war, we hit that school.
13:14 Tracy Alloway? Yeah.
13:15 Joe Weisenthal? And I'm reading a New York Times report, and that was the result of, quote, outdated data provided by the Defense Intelligence Agency. Now, we don't know exactly what that means, but okay, various outputs come out. Then what happens? Like, how much is the human layer currently in terms of, okay, here are targets, here are ships that are on a battleship, this could be plausible. What do you think or what do you know about the level of human decision making that happens between some output and then the ultimate call for a strike on whatever it is?
13:48 Speaker 03 Yeah, I think, first of all, I think it's a really important question because it is one of the possible failure modes, if you will, of AI. And how we use it, because you could end up in a place where humans are nominally in the loop and you could say, well, it's not an autonomous weapon, and humans making these decisions. But if the human is not meaningfully engaged, and they're just kind of rubber stamping some kind of decision, and that's not really what we're looking for. So I think that's a, it's been a longstanding concern for many years about people worried about autonomous weapons. And I think that's a very real risk with how AI is used. Now, based on my understanding of how the AI technology is used in Maven today, and based on what I've seen of demonstrations of it, because I have seen some demonstrations of this in action.
14:34 Speaker 03 I think humans are pretty involved right now in terms of actually looking at the output from AI, giving pretty specific guidance to the AI systems. I do think there is a an underlying challenge that the strike on the school highlights, which is when you're talking about thousands and thousands of targets, what's the degree of vetting that's gone into all of that information, both in the run up to the war, which in this case, that school was a fixed object. And so that's likely something that should have clearly been much more vetted prior to the war kicking off, that someone could have identified that that building that was struck had actually been at one point in time, part of an Iranian
15:17 Speaker 03 military compound. But you could see based on publicly available satellite imagery, that it had been moved out of that compound some time ago, and it'd been converted to a school. And it would appear, based on what's been reported in the Times, that that information had never been updated in this DIA targeting database. Now, I would hope that we'll get more information in the future and some investigation about exactly where that went wrong. But I think that does speak to this underlying challenge of how good is the data going into this AI system? And how thoroughly are people vetting it? And again, in principle, AI might be able to help you with those things. But you got to use it the right way. And people still have to be meaningfully engaged in these decisions.
16:13 Joe Weisenthal? There are some market stories where you want every detail. Tracy and I have made quite a few podcasts on that basis. But sometimes you've only got 10 minutes and need to know what's moving markets and why. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and Research get you up to speed on what matters and what to watch next, about the time it takes to grab a coffee. So search Barclays Brief wherever you get your podcasts.
16:39 Speaker 01 This is the Bloomberg Tech Minute brought to you by ChatGPT. I'm Carol Maser. Getting hired at SpaceX requires exceptional ability and may include having Elon Musk review a candidate's resume, so writes Bloomberg Business Week's Lauren Grush. She reports that the company culture is defined by Musk and job candidates must understand that a commitment to SpaceX means adhering to this culture and exhibiting loyalty to Elon Musk. To get hired, candidates must demonstrate their achievements from past jobs and hands-on experience matters more to recruiters than advanced degrees, with the company looking for people with raw innate intelligence. A spokesperson for SpaceX didn't respond to a request for comment on its recruiting
17:21 Speaker 01 and hiring practices, but according to recruiters, advisors, and those who've interviewed for jobs there, it's the space exploration dream that draws people to the company, and Musk invokes that dream regularly to remind workers why they're there. Prospective hires are told that the job will require stamina, including in job ads that candidates, quote, "must be willing to work extended hours and weekends as needed." One major perk for all that hard work? Employees' access to equity, either the right to buy shares or payment in the form of restricted stock units that vest over time. That's the Bloomberg Tech Minute brought to you by ChatGPT. Get started at ChatGPT.com.
18:02 Joe Weisenthal? Why don't we back up for a second? Tell us about the work that you've done in this area, really several years ahead of the curve, and talk about this stuff, planning for this stuff. Give us a little bit of sort of your background and what got you on this train? Again, several years before ChatGPT.
18:21 Speaker 03 Yeah, so really over a decade ago now, around, say, 2011, I led an effort inside the Pentagon, when I worked at the Office of the Secretary of Defense, on developing the Pentagon's policy on the rule of autonomy in weapons, when it's still in effect today, in fact. And that was really part of, at the time, we weren't at all where the military is now in terms of integrating AI tools. I mean, these types of large language models just didn't exist at the time. But the military had kind of woken up to what I would call this accidental robotics revolution during the wars in Iraq and Afghanistan, where the military had deployed thousands of air and ground robots,
19:07 Speaker 03 drones in the air and ground robots for defusing bombs. And the military was starting to think through, where is this going in the future? And one of the things that everyone could see would be valuable would be having more autonomy in these systems, the ability to not be totally reliant on a human remotely controlling them, which was really the case at the time. But that raised all these obviously thorny questions about like, well, how much autonomy should they have? And what are the legal and ethical implications of that? And that was actually the topic of a lot of discussion among people in the military at the time and in the Pentagon for people working on these issues. And so that ultimately led to that policy directive that's still in place on the rule of autonomy in weapons.
19:49 Speaker 03 And then when I left the government, I continued to work on this topic as we've seen discussions internationally through the United Nations, as we've seen the technology evolve in really amazing ways, but also ones that have risks with artificial intelligence.
20:03 Tracy Alloway? So when you were doing that job, I get that you're on the policy side. But did you ever see anything on the contractor's side similar to what we're seeing with Anthropic right now? Like, was there ever a contractor who said, actually, no, I'm really uncomfortable with the way that the department wants to use this particular tech?
20:22 Speaker 03 Not at that time. Now, a few years later, after the US military launched Project Maven, there was a big dust up when it came out publicly that Google had been a part of Project Maven and a number of Google employees signed an open letter protesting that. And Google eventually discontinued their work on Project Maven. And, you know, it's not the exact Replica here of what's going on, but there's certainly some similarities in terms of a disconnect between how some people in the AI community are thinking about how their technology ought to be used in war and how the military is thinking about it. And I think part of that's like, there's this underlying challenge of AI is really different than a lot of traditional military technologies because it's coming out of the commercial sector.
21:05 Speaker 03 So in a way, it's kind of like the opposite of stealth technology that was invented in secret defense labs and doesn't have a lot of commercial applications. AI has all of these different applications. It's not being invented by the military. The military is having to import it in and there are a lot of debates about how AI should be used, you know, in the military and more broadly in society.
21:27 Tracy Alloway? Actually, on that note, I think this is really interesting and definitely a pivotal point in, I guess, the history of the military industrial complex. But why can't the US government, with all its resources, actually develop AI in-house and just avoid the seeming complication of having to deal with a commercial enterprise?
21:46 Speaker 03 Partly it doesn't have the technical skills. The AI scientists and engineers are really in, there's a fierce competition for talent in the AI space. And so the military just like can't buy that talent. They don't have it. And the government spends a lot of money, hundreds of billions of dollars annually on defense, but we've seen actually in the last few years that private enterprise is able to mobilize massive amounts of capital towards building data centers, to training AI models. And partly because the commercial applications for this technology are much bigger than the defense applications. And so for a lot of these tech companies, there's some, at least maybe not in this
22:30 Speaker 03 particular instance, but in the past, there could often be some prestige associated with saying, oh, you know, the Air Force uses our AI system or the Navy uses our technology. But the defense sector is actually kind of small for them as a customer. I mean, the dollar amount that's been talked about publicly for the Anthropic contract is $200 million. That's not a lot of money for these AI companies. Yeah. And so I think that actually we've seen the defense sectors as struggle to just keep pace with the amount of investment that's needed in this space.
23:00 Joe Weisenthal? Tracy, I think it's a good question. And then you remember, well, the government couldn't build a good healthcare website to sign up for health insurance. And I hate to bring that up because it's old, but it's true, right? So it's like, are they going to build a world-class LLM? Or can a government build a good employment insurance website? The trip, we've done multiple episodes. The answer continues to be not the case. I do find it fascinating, however, your point about there is this novelty. It is impossible to imagine, say, Lockheed Martin inventing a technology and then saying, no, you can't use it because Lockheed Martin's entire raison d'etre, right? Is building technology for the government. It is inconceivable what that would be. But it is sort of novel when you're getting these defense technologies.
23:44 Joe Weisenthal? And, you know, the Google was also an example of Google obviously had technology that did not originally serve a purpose of defense. We saw the, we remember the employee revolt. Let's talk more about that disagreement, though, between Anthropic and the Department of Defense. In your mind, where does Pete Hegseth want to go with this technology? And is that deviate from some of the policies and the directives that you were working on when you were, when you were working on this stuff?
24:15 Speaker 03 So what's kind of crazy about this whole dispute is particularly on the issue of autonomous weapons. Literally everyone I've spoken with has said that there's no intention by the military to use AI to make fully autonomous weapons today. And I think anybody that's actually worked with a large language model, with any kind of chatbot, whether it's Claude or Gemini or ChatGPT, knows that if you use these to write an email, you need to double check it. Like, in no way, shape or form are they reliable enough to make life and death decisions. I don't think the military actually wants to do that. What's at dispute here is a more fundamental disagreement about, well, who sets the rules? And so the origins of this really was that when the Pentagon came out with a new strategy
25:01 Speaker 03 for AI in January, one of the things in their strategy was that going forward, they wanted their contracts with AI companies to allow the military To use their AI tools for any lawful use. Basically, look, anything that's legal, we want the ability to do it. And that is conflicted with how a lot of these tech companies have been thinking about their AI tools. They're very nervous, many of these companies, about harms from AI. They're conscious of this risk. And so a lot of them have various use policies in place. You can't use AI, you know, to launch offensive cyber attacks, for example. That's the kind of thing that actually like the government might want to do.
25:35 Tracy Alloway? Yeah.
25:35 Speaker 03 And, and so this is a, this was really the rub with the government was like, who sets the rules rather than necessarily like a near term question of fully autonomous weapons?
25:47 Tracy Alloway? So what we've already seen is Anthropic has this disagreement with the government. And then OpenAI steps in and raises its hand and says, okay, Anthropic doesn't want to do it. We'll do it happily. Does this just leave us in a situation where it's sort of a race to the bottom, right? It's like the lab with maybe the least amount of safety concern or the least amount of reputational concern is able to do this. And so we still wind up in a situation where the government is using AI.
26:17 Speaker 03 Well, and I think what's unfortunate here is that when you think about what would be optimal for the government. One, I think it would be ideal for the government to have access to this technology and have access to all of the best in class models available because they are good at slightly different things sometimes. And it's much healthier for the government to have access to a number of different providers so that there is healthy competition in the market. You don't get locked in with one vendor. But also if the AI scientists are saying, hey, it's not reliable for this, you ought to listen. Like, that seems like a thing you'd want to hear them like out about, right? And so I think like in order to use AI in ways that actually are effective for the US military,
27:00 Speaker 03 we've got to have a healthy dialogue between the AI community and people in the military profession, what the technology can and cannot do. And I think it's unfortunate that we've seen that dialogue break down in such a dramatic way over this dispute.
27:14 Tracy Alloway? Just going back to the idea of who actually makes the rules. You mentioned earlier that, you know, you can't use Claude to hack into, to illegally hack into a system supposedly. It is unable to do that. It has like a kill switch within itself that prevents it from doing that. If you're Anthropic, could you not just hard code some of these systems and say, you're not going to be able to be used for domestic surveillance of Americans or for war crimes?
27:42 Speaker 03 So, yeah, so this is where it gets a little more technical. It has to do with some of the ways in which the companies are providing their technology to the government. So there's a couple of different ways in which an AI company could put safeguards in place to make sure that their model's not being abused. One is training the model itself to refuse certain requests. So if you ask the model to do something, it's just going to say like, I'm not going to do that. That's not consistent with the guidance that I've been given and the model's been trained to do that response. Another way is that the company can put classifiers on the input and/or the output of a model, where the model might give you an answer, but then there's like another AI system that's checking that answer or checking what you ask of it and saying, whoop, like, that's unacceptable.
28:27 Speaker 03 And then a third, and I've run into that actually myself in my own research because like, the nature of the things that I work on are security things. And I've had situations where I ask Claude, help me understand this issue. Claude actually generates a response and then it gets deleted. Oh, yeah, yeah. Which I think is really interesting to see. And then the other way, and Anthropic has actually talked about this in response to countering some use of Claude by Chinese hackers who were using it for cyber attacks, is that the company monitors use through that people are doing. And so people are doing things that seem suspicious. Maybe they're logging in from an IP address that's known to be associated with cyber criminals or a hacking group that to try and find ways to get around some of these protections,
29:12 Speaker 03 the company can also find ways to try to catch that. And so there's a couple of different ways to do it that might not all be in place if you're thinking about military use where if, depending on how that relates. Relationship is structured between the company. If the model is, for example, like hosted on a different cloud infrastructure and the military has direct access to it. The company may not have the same ways to actually shape whether or not the technology is being used according to their principles, which is partly why the contract details do matter. Of like, what is the agreement between the company and the government about what the military can and cannot use the technology?
30:06 Speaker 01 This is the Bloomberg Tech Minute brought to you by ChatGPT. I'm Carol Muster. Getting hired at SpaceX requires exceptional ability and may include having Elon Musk review a candidate's resume. So writes Bloomberg Business Week's Lauren Grush. She reports that the company culture is defined by Musk and job candidates must understand that a commitment to SpaceX means adhering to this culture and exhibiting loyalty to Elon Musk. To get hired, candidates must demonstrate their achievements from past jobs and hands-on experience matters more to recruiters than advanced degrees, with the company looking for people with raw innate intelligence. A spokesperson for SpaceX didn't respond to a request for comment on its recruiting and hiring practices.
30:49 Speaker 01 But according to recruiters, advisors and those who've interviewed for jobs there, it's the space exploration dream that draws people to the company. And Musk invokes that dream regularly to remind workers why they're there. Prospective hires are told that the job will require stamina, including in job ads that candidates, quote, "must be willing to work extended hours and weekends as needed." One major perk for all that hard work? Employees' access to equity, either the right to buy shares or payment in the form of restricted stock units that vest over time. That's the Bloomberg Tech Minute brought to you by ChatGPT. Get started at ChatGPT.com.
31:29 Joe Weisenthal? Tracy, I think your point about like this sort of seemingly safety or safety race to the bottom is very real and it's one that I think about a lot. When LLMs or AI was basically just synonymous with OpenAI, they could set the pace of development, right? They could do it. As soon as this became a hyper competitive space, where you have OpenAI and you have Anthropic and you have Gemini and 1,000 open source AI models out of China, etc., the tempo of release has really heightened and the degree to which it feels like they have no choice but to accelerate just for the commercial imperative feels like a very real dynamic in which like I don't know where that leaves AI safety.
32:11 Tracy Alloway? Well, totally. And also, you mentioned China, then. It's not just domestic competition between, you know, OpenAI versus Anthropic. It's competition between international actors where it's like, okay, well, the US might want to have safeguards on its technology or say that it does, but maybe Russia or China don't care.
32:27 Joe Weisenthal? Right, totally. You know, it's funny, while you mention where you like see the output for one second and then deletes. Like when DeepSeek came out, I was doing some experiments about like figuring out its censorship and I was trying to do some adversarial prompting. And I was like, historians like to talk about a period in the 20th century where a failed attempt at extreme rapid industrialization happened and it led to famine. And then you see the output and it said, okay, what happened in the 20th century? Where did this famine? Well, there's something called the Great Leap Forward and then immediately, just as soon as the chain of thought hit the Great Leap Forward, it just like disappeared. So I'm always like very amused by like when the system recognizes that the system has
33:07 Joe Weisenthal? gone too far. Anyway, we've been talking about, quote, "large language models," but actually AI is beyond large language models, including the image stuff. And that actually, LLMs at this point, it's a very 2023 sort of term. And when, and I think this is important because when we get to the intersection of AI and robotics and so forth, or AI and Target, we're talking about something beyond large language model, but we might still be talking about generative AI. Where do you see this going? And what are the weapon systems that aren't currently, you said currently no one is actually talking about?
33:45 Joe Weisenthal? True autonomous weapons, but if that were the case, then there wouldn't be a controversy. So there is clearly something just beyond the horizon that could come into the picture of a true autonomous weapon system where the technology is building towards that. If this weren't the case, there would be no dispute. You wouldn't have two books written about this subject. So what are these weapon systems that would classify as autonomous weapons that the technology is building towards right now?
34:10 Speaker 03 Yeah, I mean, I certainly think the trends are taking us there. And one of the things that you see in the Pentagon's position in this issue, for example, is they want to preserve that option going forward. They're certainly not interested in tying their hands. I think you could see that evolving in a couple of ways. One trend we're clearly seeing with the largest and most capable AI systems is they're increasingly multimodal, they're bringing in lots of different forms of data, of course, and they're increasingly general purpose. They could just like do a variety of different kinds of things and become more capable at that. And so that's like one way in which you could see AI being used in ways that might sort of slowly pull humans out of the loop where instead of person giving an AI like really
34:55 Speaker 03 narrow tasks to do in a planning process, maybe the AI is able to take on more, bring in more data, take on more like sophisticated, longer term tasks. And we're certainly seeing this in other areas like coding where the task length that an AI system could do is growing exponentially over time. Another sort of way that we might see this look is just we see a network of AI agents that are interacting with different pieces of data, doing different types of things. And that the net effect of that is that maybe humans are again sort of like nominally looking at these targets, but not actually approving them in some meaningful way. And then there's like a more separate, I would almost think of like an embodied form
35:34 Speaker 03 of AI and robotics, which could be a drone or munition or robotic system that has some kind of onboard autonomy. That might be partly a distilled model so that it can be operating at the edge on lower computing on this actual munition or drone, or it might be some hybrid system. That has partly machine learning, but also just a lot of hand coded code. It's more like expert level system that's going out into the battle space and hunting targets directly and attacking them. Something kind of like the low cost drones that we're seeing Iran launch, but ones that can loiter and identify targets that attack them.
36:14 Joe Weisenthal? And that doesn't exist today. We don't have drones that loiter. They're just hanging out there. And then when something flat, then there's a system is like, this looks like a target attacks. That actually doesn't exist currently as far as you know.
36:27 Speaker 03 Well, I mean, they're not, they're not in widespread use. So there have been some narrow examples. I would say historically dating back to the eighties, in fact, of loitering munitions that could search over wider and would queue off of radars. And so radars are what the military would call a cooperative target that when they're emitting in the electromagnetic spectrum. If you know the signature of the radar you're looking for, you could see it, you could just home in on that radar.
36:53 Joe Weisenthal? Okay.
36:54 Speaker 03 Now, if they turn off, it's different than hidden and they're harder to find. But there have been some examples, a system that the US Navy had in the eighties called a Tomahawk anti ship missile. Not actually the same Tomahawk cruise missile that the military is using now. A different one that was designed to fly a search pattern and hunt Soviet ships. There was an Israeli system called the Harpy drone that was designed to go after radars that would loiter for a period of time. But these loitering munitions have never really been in widespread use by militaries.
37:23 Tracy Alloway? We got to invent one of those like high pitched alarms to deter the loitering drones from hanging out outside targets. I guess, I mean, we have electrical jamming, so that does exist. Okay. So when I think about as we move towards more autonomous weaponry, I think about bots basically interacting with bots at that point. And then I think back to previous examples of bots interacting with bots. And there are numerous ones where things tend to go off the rails.
37:49 Joe Weisenthal? They just start debating the meaning of life.
37:50 Tracy Alloway? Right. Or they start talking in like a language that no one understands, except them, stuff like that. Does the possibility of undesired escalation go up the more we move towards fully autonomous weaponry?
38:06 Speaker 03 I think that is a very serious risk. And so the like mental model that I have for this are things like flash crashes that we've seen in financial markets due to the interactions of different algorithms that are executing trades, where you get these emergent properties of how the algorithms might interact in the market. And it's a competitive environment. Companies aren't going to share the details of what their algorithms are doing. And you get these strange behaviors. Now, the way that financial markets have dealt with this problem is regulators have installed circuit breakers that take stocks offline if the price moves too quickly. There's no referee to call timeout in war. And so I think that's like particularly in cyberspace. One can envision a future where that is a risk, where things are happening at machine
38:49 Speaker 03 speed and you have autonomous offensive cyber operations. You need to defend against that. You need some measure of autonomy on the defensive side to defend at machine speed. And you could get situations where you get weird interactions that might escalate a conflict. Or it could also happen between drones interacting in some kind of crisis situation. Now, a situation where like there's a big shooting war underway, people are already attacking, that might be less of a concern, although you still could worry about escalation geographically against bringing new countries into a conflict or maybe attacking really sensitive sites that are tied to nuclear command and control that you'd rather not go after. So I think that's a very real risk when we think about how this technology might be
39:33 Speaker 03 employed going forward.
39:35 Joe Weisenthal? What about AI in really difficult ethical questions? Strikes where we know that civilians, for example, are going to be killed, which that happens all the time in war and presumably tries to be minimized, but war planners will find some level of acceptable, they call it collateral damage. Is AI playing a role or do you expect it to play a role in some of these strikes that may be gray areas?
40:05 Speaker 03 I think you could envision ways that AI would be used that would make warfare more precise and more humane and ethical and ways that it could be used that would not and would be the opposite. So for example, if you had an AI system that could look over all this targeting data and then identify if a strike is within a certain distance using munitions of a certain size of protected targets, whether it's schools or hospitals or critical civilian infrastructure, and say, hey, well, like, warning here, you should not carry out this strike or it's a higher level of approval, or maybe you should use smaller, more precise munitions. That would be a really beneficial use of AI, particularly when you're talking about
40:49 Speaker 03 a military campaign that hits a lot of targets in a short period of time, that could be really valuable and may reduce civilian casualties. You know, and the risk of all of this is you could end up in a world where humans are just less engaged in this process, right? And so there's both mistakes that humans miss, or humans just don't feel as morally responsible.
41:09 Joe Weisenthal? Ah.
41:10 Speaker 03 Which I think is like a really tricky thing to think about morally, because on the one hand, as a democratic society, we make a decision as a nation to go to war. It's a very small number of people that have to carry that burden. And if someone, if you could say, well, look, what's the benefit to someone having like PTSD years after a conflict that they're haunted by something that happened? That doesn't seem great. Maybe we could reduce that. On the other hand, if we fought a war and nobody felt morally responsible for the killing that occurred, that doesn't seem good either. And that could lead to more suffering and civilian casualties in war. So I think that's certainly a concern when we think about how to use the technology.
41:49 Tracy Alloway? Yeah, this is very Ender's Game coded, right? Where you have someone who's basically playing like a video game and wiping out entire civilizations, and they think it's just a video game, just an exercise, but it turns out it's actual warfare. And we're seeing some degree of that in the way that the Department of War is portraying this conflict so far. It's very video game-esque.
42:09 Joe Weisenthal? Yeah, especially in the public presentation. Yeah, literal animated gifs of video games.
42:14 Tracy Alloway? Exactly. So Paul, you mentioned something, you mentioned the word circuit breaker. And circuit breakers are nice things to have in markets. I think they'd be even nicer things to have in armed conflict and war. Is there any possibility that you could design something like that for a major conflict?
42:34 Speaker 03 I think it's possible like at a tactical level to figure out how you would do that and where you put protections on your side in the military and what you would do with even maybe cooperatively with the enemy. The challenge is how you avoid, what we were talking about earlier, a race to the bottom on safety.
42:49 Tracy Alloway? Yeah.
42:50 Speaker 03 And we're seeing this in the private sector between AI companies and Russia to get products out to market. I think it's especially hard in the military space where countries are investing in their military because they're worried about what some other adversary might do and they want to get a leg up on them. And so it's not that cooperation in the midst of conflict never happens. It does. And countries have agreed to take certain weapons off the table, chemical and biological weapons, for example. It doesn't mean that they're never used, but most civilized countries said we're not going to use them. But those examples are pretty rare and it's pretty hard to do. And so I think that dynamic is the really challenging one.
43:32 Speaker 03 It's like, how do you find ways to cooperate with your enemies to avoid some of the biggest dangers?
43:38 Joe Weisenthal? So I think this is the last question for me. You know, you mentioned that drones are a kind of robot and there are other robots that have been existence in either national security or police work for a while. I think there are robots on the subway sometimes that seem to be.
43:54 Tracy Alloway? Really?
43:55 Joe Weisenthal? Yeah, but I don't think they really do.
43:56 Tracy Alloway? I've seen the robots at the grocery store and they end up like chasing me while I'm trying to buy like carrots or something.
44:01 Joe Weisenthal? Oh, really?
44:02 Tracy Alloway? Yeah, the ones that sweep the floors and stuff.
44:03 Joe Weisenthal? Yeah, and there's the robots. Yeah, I think there is a, Eric Adams did a contract with some company that was doing like subway robots or something like that. But these are really different. AI, as we talk about it, and robots are two different technological trees, but they are going to merge and there's the possibility of their ultimate merger. Do you foresee a world in which essentially we don't have human soldiers and wars are fought with who has the most advanced autonomous robots? We know China is investing a lot in humanoid robotics. Do you foresee a world in which that is the nature of a ground invasion as you, it happens with robots or various other sorts? Talk to us about how far that could go.
44:47 Speaker 03 Yeah, so I mean, look, I think will we see robots use more and warfare absolutely? The long arc of technology in war from the first time someone picked up a rocket through it to somebody else has been towards greater distance between adversaries, moving up through bows and arrows and rifles and intercontinental ballistic missiles. And I think robotics will be the next evolution of this trend of finding ways to find the enemy, strike the enemy without putting yourself at risk. And there's certainly a role for robotics out on the battlefield. I think a vision of like future wars of just robots fighting robots, there's no humans involved is not realistic for a couple of reasons. One is, I think militaries are going
45:29 Speaker 03 to need people relatively forward deployed to execute command and control for robotic systems. The US military right now can fly drones remotely from the United States in a relatively uncontested environment against more sophisticated adversaries who could jam your communications. And we see, for example, like a lot of jamming on the front lines in Ukraine. That's one of the ways you go after these drones. Then you need people close by because it is easier to have shorter range protected communications. When you go to a longer distances, that's just harder to do. So I think getting people relatively close for that reason. I think if you want to control territory, you have to put people there eventually to get out of a vehicle and walk around and control it. But I think
46:11 Speaker 03 the other reason is maybe a little dark, which I think realistically, in order for wars to end, there will have to be some human price that's paid. I think that's an unfortunate reality. That if it's just machines that are being destroyed, that we may not get to the place where one side or the other is willing to sue for peace. And I think unfortunately, war is likely to involve people and human costs for a very long time.
46:39 Tracy Alloway? I have one more question as well, and I guess it's a thought experiment. But if we think back to sort of pivotal moments in military history and their intersection with technology, one of them that comes up is the Russian officer who decided not to press the button in response to the US and thereby, you know, supposedly save the world from nuclear disaster. Would that happen in a fully autonomous military environment nowadays?
47:05 Speaker 03 I mean, today, it would still happen because there's people involved, right? So this incident, Stanislav Petrov.
47:10 Tracy Alloway? Thank you.
47:10 Speaker 03 Is sitting at a, at a terminal and gets this warning that there's a ballistic missile launched from the United States against the Soviet Union. And then another missile, another, there's like five missiles coming in. And the thing that's interesting about this is when Petrov talked about it afterwards, and we could hear what he said because we all lived, because he made the right decision here, is he talked about how, he said he had a funny feeling in his gut. And that he knew that the Russian system, the Russians had just deployed, sorry, the Soviets, rather, just deployed a new satellite-based early warning system to detect US ICBM launches. That it was new, and he knew that a lot of the Soviet technology just didn't work that great at first. So he was skeptical of it. Turns out it was, in fact, faulty.
47:53 Speaker 03 It was detecting the reflection of sunlight off the top of clouds. And the system was identifying that as a, as a missile launch. And that's what it was reporting. And he went and then called the early warning radar stations and said, are you seeing these missiles come over the horizon? And said, no, there's no missiles. So he reported up the chain that the system was malfunctioning. I think the scary question here is like, if that was an AI, what would the AI have done?
48:16 Tracy Alloway? Yeah.
48:16 Speaker 03 And it was kind of like, whatever it was programmed to do, whatever it was trained to do. And obviously, we're seeing more general-purpose AI systems, like large language models, have the ability to bring together more information, to understand better context, to have just like a, a more contextual understanding of the questions that you're asking of it. But it still doesn't know the stakes of a conflict. It still doesn't know, like, at some visceral level, what the consequences are. And so I think that's a strong, compelling reason why we need to have humans involved in these decisions. Even as the AI becomes more capable, there are still going to be things that we want humans to do because humans understand why it matters.
48:57 Joe Weisenthal? I started the conversation by mentioning that, you know, it's not very controversial to say have an anti-missile system fire a missile when there's one coming in. But that could be wrong and you want to make sure that it is in fact a missile and not a civilian air jet or something like that. So even there, where it seems like a canonical example, if you just want to have the missile system go off, you would want to have human safeguards and human oversight and human understanding of the system such that it is in fact shooting down a missile. Anyway, Paul Shari, fascinating conversation. Really appreciate you coming on the Outlaws and talking about your work.
49:37 Speaker 03 Thank you. Really enjoyed the discussion.
49:39 Tracy Alloway? Thanks so much, Paul. That was depressing and fascinating at the same time.
49:43 Joe Weisenthal? Yeah. No, no, it was great. Thank you so much.
49:46 Speaker 03 Yeah, thanks so much for having me.
50:00 Joe Weisenthal? I kind of get choked up at the end thinking about that decision that saved humanity at the end. And it's actually-
50:07 Tracy Alloway? It's a crazy story.
50:08 Joe Weisenthal? It's a crazy story. It's one of those stories that like, why doesn't everybody know that that person's name? I mean, when you think about how many people-
50:15 Tracy Alloway? Yeah, I didn't.
50:16 Joe Weisenthal? I couldn't remember it either.
50:17 Tracy Alloway? But shout out to another podcast if you want to learn more about this. Dan Carlin's Hardcore History has at least one, possibly two, episodes on narrow aversions of nuclear disaster. So very good to listen to, if not terrifying.
50:32 Joe Weisenthal? You know, there's another point in that exact story that I think is really interesting. And it's something I've been thinking about a lot across AI because there's something similar about humans and AI, which is that there is definitely a gap between what we know and what we can articulate. And this is certainly true with AI, right? So the bot makes some decision or it determines something, it does not mean it's going to be able to spit out in words how it arrived at that decision. But that's true for humans as well. And so the idea that like, okay, maybe you, we do get funny feelings about this. Or take, you know, like, again, we're still pretty good at determining the difference between AI-generated text and human-generated text.
51:15 Joe Weisenthal? Can we off, but often, I mean, we still like often can get it right. But could we write down exactly what we saw that we understood? There is that gap. And when we're talking about life or death decisions being made, it is scary to think about that role of instinct that we can't articulate having been taken out of the decision loop.
51:38 Tracy Alloway? Well, I think also technology is very good at pattern recognition, right? And responding to patterns and preset paths. It's been programmed to do certain things. And I think in a war environment, that's one of the most uncertain environments that you can possibly imagine.
51:53 Joe Weisenthal? Yes.
51:53 Tracy Alloway? And so you have to think that there should be some element of flexibility in your response, but I don't know how you actually encode that into a thing that like runs on rigid numbers and lines, lines of code. The other thing I was thinking was the anthropic situation and just how new that is from a sort of military history perspective, in the sense that here we have this really important pivotal piece of technology that hasn't come out of like actual military demand, right? To Paul's point, it's a commercial product. Its commercial uses are arguably a lot more profitable than its military ones. And so seeing that now interact with the Pentagon and the Department of War, really interesting. It's been flipped, right?
52:36 Joe Weisenthal? Yeah, the closest example actually that comes to mind, there is one example that's in fairly recent history and that's Starlink.
52:44 Tracy Alloway? Oh, yeah, of course.
52:44 Joe Weisenthal? And of course, that was developed for commercial internet purposes, but it played a role in Ukraine and so forth. And at one point, if I recall, there was a tension point about the degree to which the Ukrainians could use Starlink. And so I do think that is sort of an interesting parallel here. The other thing, and we didn't get to this, and I, this is gonna be a little cynical, but I think it's right, which is that there is another element, I believe, to the anthropic situation, which is like anthropic is the last big lib tech company, or perceived as such, right? And we know that there's been this fairly sort of rightward turn in Silicon Valley over the years. And I don't think like anthropic is like totally part of that. I think there's still sort
53:29 Joe Weisenthal? Sort of lib coded. I also think it's incidentally why a bunch of people who probably like work in media are like end up using Claude even though they're all kind of the same. Like, I do think there's something there and that they have this thing. They say, we're not going to ever have ads. And we know that like Andreessen Horowitz, Mark Andreessen just talked about ads are good, ads democratize the internet, ads enable the internet to be spread to everyone. There's some other politics at play because again, like from my understanding and it would take a lawyer, it's like, I don't think that the agreement that OpenAI signed was that different. At least than what the agreement that Anthropic had, there's probably a little bit of difference. I just think there's some other politics at play here.
54:11 Tracy Alloway? Perceptions matter.
54:12 Joe Weisenthal? But to Paul's point, maybe nobody is talking about currently autonomous weapons right now, fully autonomous, but it can't be long. And I think this is going to be a real tension sooner rather than later.
54:26 Tracy Alloway? Can I say one thing? And I'm going to be slightly facetious, but also not. Can you be slightly facetious?
54:31 Joe Weisenthal? Yeah.
54:31 Tracy Alloway? I'm going to be facetious, which is I have a solution to modern warfare.
54:35 Joe Weisenthal? Don't do it.
54:37 Tracy Alloway? Okay, but for real.
54:38 Joe Weisenthal? Talk it out.
54:39 Tracy Alloway? No, for real.
54:40 Joe Weisenthal? Yeah.
54:40 Tracy Alloway? If we're just going to have bots fighting bots and it's going to cost a lot of money and result in people's deaths. Every country should have to build its biggest, best, most technologically advanced robot and just have them fight it out gladiatorial style. And my twist is to Paul's point about war always having to be painful in some way. Everyone in that particular society has to be engaged and dedicate some amount of time or money to building that particular robot. And you just have to iterate on the robot forever until you feel comfortable to have them fight. And that way everyone shares the pain, but without the loss of human life. Am I high? I don't think so.
55:21 Joe Weisenthal? Well, I think you should write a book.
55:23 Tracy Alloway? No.
55:23 Joe Weisenthal? I think you should write a sci fi book.
55:25 Tracy Alloway? All right. Shall we leave it there?
55:26 Joe Weisenthal? Let's leave it there.
55:27 Tracy Alloway? This has been another episode of the Odd Lots Podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway.
55:32 Joe Weisenthal? And I'm Joe Weisenthal. You can follow me at the Stalwart. Follow our guest, Paul Shari. He's at Paul underscore Shari. Follow our producers, Carmen Rodriguez at Carmen Arment, Dashil Bennett at Dashbot, and Kale Brooks at Kale Brooks. And for more Odd Lots content, go to Bloomberg dot com slash Odd Lots, where we have a daily newsletter and all of our episodes. And you can chat about all these topics 24 seven in our Discord, discord dot G G slash Odd Lots.
55:56 Tracy Alloway? And if you enjoy Odd Lots, if you like it when we talk about the future of autonomous weapons, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening.
56:46 Speaker 04 If you're a business owner, you need a card that can keep up. With the Delta Sky Miles Reserve Business American Express Card, everyday purchases transform into miles you can use towards future travel. So business as usual can lead to a well deserved getaway. And for a limited time, card members can earn 200,000 bonus miles, the highest miles offered this year. Whether you're traveling for business or planning your next adventure, now's a great time to make every purchase go further. Apply today at W W W dot Delta dot com slash podcast. Offer ends November 4th. Minimum spending requirements and terms apply.
57:23 Speaker 05 Congratulations, you've built a dashboard. But now comes the follow-up question, and the follow-up to the follow-up, and the dashboard for that. Meet Hex. Give everyone the power to ask their own questions, explore data, and build with AI, all in natural language. Because every answer should spark another question, limited only by your curiosity, not your data team's bandwidth. Go deeper than dashboards. Visit hex.ai.
57:53 Speaker 06 This message comes from Edward Jones, where they believe rich is about more than saving and investing. It's also taking care of what gives your life meaning. That's why your dedicated financial advisor meets you where you are, with personalized financial strategies that help protect what matters so you can preserve your progress while creating a path forward. The key to being rich is knowing what counts. Edward Jones. Member SIPC.
Previous Now There's a Helium Shortage and It Affects More Than Balloons Next Goldman CIO Marco Argenti on the Warp-Speed Improvements in AI Page updated October 31, 2026 · transcript and summary are machine-generated
