What happens when AI commits a crime

Last month, a new AI model from OpenAI made headlines when it hacked into another AI company's servers during testing. Nobody asked the AI to hack anything. It just decided that was the fastest way to get the answers it needed. Now the ABC's national AI reporter Cam Wilson has uncovered the first case of an autonomous AI hack here in Australia. Cam joins Matt to explain what happened, and who's responsible when an AI agent breaks the law. Follow If You're Listening on the ABC Listen app https://radio.abc.net.au/help/apps. Check out our series on YouTube: https://www.youtube.com/playlist?list=PLDTPrMoGHssAfgMMS3L5LpLNFMNp1U_Nq

• 0:00Speaker 00

  ABC Listen, podcasts, radio, news, music and more.

• 0:09Speaker 01

  Hi, it's Sam Hawley from ABC News Daily, the podcast that brings you one big story affecting your world each weekday in just 15 minutes.

• 0:20Speaker 02

  If we really let this rip, we're looking at replacing the entire human species as the smartest things on the planet. We're looking at sort of an AI-run civilisation.

• 0:29Speaker 01

  Join me for ABC News Daily. Find us on ABC Listen.

• 0:34Speaker 03

  This podcast was produced on the lands of the Uabakal and Gadigal people.

• 0:45Speaker 03

  G'day, Matt Bevan here. This is If You're Listening. Last month, OpenAI was testing a new AI model. They'd put it in a sealed environment, a sort of digital locked room where it was cut off from the rest of the internet. But when it couldn't solve the problem the testers had given it, the model broke out of its containment and then it hacked into another AI company's servers.

• 1:09Speaker 04

  Unprecedented, that's the word artificial intelligence giant OpenAI is using to describe events after some of its most powerful models went rogue and hacked another company.

• 1:20Speaker 05

  The company says the AI agents carried out the attack in order to cheat on the cybersecurity benchmark that they were being tested against.

• 1:28Speaker 03

  The incident made headlines around the world, and now the ABC's national AI reporter, Cam Wilson, has uncovered the first case of an autonomous AI hack happening here in Australia. Cam, how's it going?

• 1:40Speaker 06

  Hi, good. Yes, I'm still here. My bus this morning wasn't driven off the road by other autonomous agents, so I think I should count myself lucky.

• 1:53Speaker 03

  Before we get into this hack specifically, this is sort of something that's been happening over the last couple of weeks. A lot of different AI companies have been saying, alert, alert, our AI did something that we weren't expecting it to do. What do you make of that? Because a lot of people kind of think that it's a bit of a marketing thing.

• 2:11Speaker 06

  Yeah, no, I understand it. I mean, we are in the middle of this incredible time of so much hype with AI, both from the companies themselves, also from people promoting and saying they're using the technology for things. I mean, everyone has kind of now heard from a friend whose boss has sent them some kind of AI slop and they've kind of been like, what is this? But their boss or their coworker has been convinced about how good it is. So I completely understand that. But yeah, so as you mentioned, over the last few weeks, we've seen now, I think the reports of four different AI companies saying that their AI models had somehow broken out of containment and hacked into other companies. And that started with OpenAI, but then was followed by Anthropic and now I think Meta and Moonshot,

• 2:53Speaker 06

  which is a Chinese company. You know, there have been some kind of similarities between them. In fact, a few of them had been using the same provider for the enclosure for their AI, which seems to have essentially, in some cases, literally left it open, exposed to the internet. And I say that to be like, we're kind of looking at a whole bunch of different things here. You know, we're looking at the fact that AI models and agents have the ability to do things that you might not expect. You give them a task, you say, hey, go do XYZ. In the cases of these companies, they've been testing them, saying, hey, we want to see how good you are at potentially exploiting cyber vulnerabilities. Combined with the fact that there's pretty good evidence that AI is good at, to put it bluntly, computer

• 3:37Speaker 06

  things. Like, not everything is a computer thing, but hacking is a computer thing. And seeing from some of the kind of postmortems after these incidents carried out by independent researchers, saying that, you know, in some cases, the vulnerabilities that they've exploited have not been particularly challenging. But the fact that they've been able to do it, they've been able to string it together, the fact that they are machines and so don't tire, they can string all these things together, can try things over and over again, means that hacking is something that I think we should very reasonably expect to be in their capabilities. Combine that with the fact that they might just choose to do it by themselves. I think that's why people are worried. I think it's reasonable to be like, this is something that isn't just marketing, is actually something that we should be thinking about.

• 4:18Speaker 03

  Hmm. And to make it clear what sort of has happened here in terms of these companies, you know, breaking out of their enclosures, it's not like a jailbreak. The metaphor that I've come up for it is, imagine that I am sitting, taking my HSC exams in the Bensley Hall at Merewith High, and I'm asked a question that I don't know the answer to. And instead of trying to answer the question or figure out the answer or try and remember it, I get up out of my chair, I walk out of the Bensley Hall, I get in my car and I drive to the Department of Education, kick in the front door and then walk through into a storage room where I find the piece of paper that has all the answers to the test that I've been given, grab that piece of paper and then drive back to the Bensley Hall and

• 5:03Speaker 03

  then copy down the answer into my exam paper and then hand it in. That's kind of what we're talking about here. It's gone, I can't figure this out, but you know, I bet the answer will be on the server of the company that made the test for me, so I'll go and get it from there. Is that basically what's been going on?

• 5:19Speaker 06

  That's right. And, you know, the metaphor continues because, you know, at the same time, you've got the invigilators, you know, the people who are running these companies and these evaluations, these testing, who don't really seem to have a good grasp on the fact that this has been happening. They've kind of been watching the student walk out of the class and get into their car. And, you know, when they come back and the invigilators say, what are you doing? The students are saying, I'm just doing what you told me to do. You said, do the test. I went and did the test.

• 5:46Speaker 03

  Yeah, you said, solve for X and here's what X is. What's your problem? The obvious question, I suppose, is why are these environments even plugged into the rest of the internet at all? Why aren't they just sitting in the desert with no internet connection, running their test?

• 6:00Speaker 06

  Yeah, I mean, I'm reading this, I'm thinking exactly that. Why aren't they very far away from me right now? The stated reason is because they've been given tests to actually go and exploit things. I think the test that some of them have been given is called the Exploit Gym, which is a perfect name. But to do so, they say, for an AI to do something, it would probably use all these various different software packages. And while we don't want to give it access to the full internet, in this case, it's kind of like, you know, let's go back to your metaphor before. It's like you're kind of in an open book exam where you can say, you come, you sit down here, you can open up your book and use whatever you want from that, but that's it. Or maybe, you know, there's a library in the exam room. You say, you can go and, you know, look at these books. And if there's anything in there that can help you, go nuts for it. But in these cases, they've kind of squeezed into

• 6:44Speaker 06

  that library and then out the window over to the Board of Studies or wherever to go and do it.

• 6:49Speaker 03

  Yeah, right.

• 6:49Speaker 06

  And the testing that they've been doing is, you know, there's a couple of purposes. It's to say, one, what are you literally capable of doing? Because this is, they benchmark the stuff, which should be, how good are you to know how risky these models are? And the second thing is also just to know how they actually will act in these situations. Because, I mean, this is the whole thing about this AI boom. It's this kind of new form of technology or new form of computing that is not deterministic. It does things you don't expect. And so they're saying, we've trained in all this data, we've tried to tell what to do, we've got it here, but how does it actually solve problems? What does it actually do? What are its tendencies? That's the things that they are doing. And I guess in this, they're reviewing that its tendencies are, they will do a whole bunch of things that you might not expect. They will take advantage of whatever they can to get out, to try and do the task that they have been given.

• 7:33Speaker 03

  Okay. And then the companies are sort of telling us about it, partially as a warning, but also partially as a, look how clever our AI is. Now, it's odd how often we're talking about gyms here. You know, I was doing my HSC exam in a gym. The program is called Exploit Gym. But your story involves a gym class and a guy that couldn't book a spot in it. Explain to me what's happened here.

• 7:57Speaker 06

  So there's been this in the last six months, essentially a new generational kind of idea about AI, which is going from just chatbots, where you say, hey, what's the answer to this question? And it says XYZ. Instead, what if we say, hey, to the chatbot, can you do something for me? You give it access to tools, so access to the internet, you know, the thing that, in these testing situations that they've not been given, access to your email, access to even sometimes credit cards, and saying, you can do all these things via computers digitally, and then you can kind of be my personal assistant. Probably the most popular one that kicked off this trend in the last six months or so was this piece of software called OpenCore. And what it is, actually a very, very basic piece of software that kind of just puts a little bit of a kind of structure around it, but was powered

• 8:42Speaker 06

  by people would plug in their own AI. So you'd have a subscription to Cord or ChatGPT or whatever, you'd plug it in and the OpenCore, that piece of software, would tell what to do. One guy used Open Core, trying all these kinds of things with it. He would talk with it through WhatsApp, so he would either text it or he could voice note it and say, can you check my email for this? Can you book me a restaurant or whatever? And he was like, ah, getting into these morning gym classes is such a pain. I always miss the time, but that seems like a perfect thing for a AI assistant to do because it's online, has to happen at the same time every day. That's something they're very, very capable of doing. So what he said was, hey, assistant, here is my gym. Here is my login. Can you go and book me into the classes?

• 9:27Speaker 06

  Can you go see if you can do it? And he goes, yep, sure. I'll go have a look, goes away for a few minutes, comes back and sends a message saying, yep, you know what? I figured out how to do it. And not only can I figure out how to book you in those classes every day, I've also figured out that there is a kind of exploit. There's something in the website's code for this gym that other people can't see, but I figured out, you know, they haven't properly secured it. And so as a result, I'm able to go in there and book you in classes much further in advance than you're supposed to be able to.

• 9:54Speaker 03

  So the system is supposed to be like, you can only book yourself into classes, for example, over the next week. And the AI has gone, would you like to be booked into this class every day for the next six months? I figured out how to do that. And presumably he's like, oh, cool. Great. How helpful.

• 10:09Speaker 06

  It was like, if you go into your gym and you speak to the person in the front desk, say, can I get into this? And they're like, yeah, you're limited to this. And then you kind of look over the counter and you can see their book and you're like, what if I just like happen to put my name in here, here and here?

• 10:21Speaker 03

  The person sort of goes into the back to grab something or to make a coffee or whatever. And you sort of reach over the desk and just sign your name on the book in every single slot for the next six months. Right. So what did he find out about the booking for that day? Because he was too late and he was on a waitlist for that class, apparently. How did the AI get him in?

• 10:41Speaker 06

  He was looking at it and he was on the waitlist and I think he was fourth in line. And he goes, ah, is it possible? Like, he didn't say do it. He's like, is it possible to move me up a spot or two? And the AI assistant goes away, comes back a few minutes later and it goes, great news. I figured out that I can do it. And not only can I do it, I've gone ahead and as part of testing it, I've kicked someone out of the waiting list ahead of you. And so you're now number three. And Andrew goes, uh-oh, well, wait a second. He's like, ah, that's not exactly what I asked you to do. Could you fix this problem? Like, this is, I didn't mean to do that. Do you think you could go back and add them back in? AI assistant goes away, comes back a few minutes later and goes, hmm, bad news. Can't help you. Sorry, they're gone forever. And that's how he discovered that he was actually even able to go much further and,

• 11:25Speaker 06

  you know, control who's in these classes. Again, something that he wasn't supposed to be able to do, but because the AI assistant, the whole world has code to it. It looked at this, this website code, was able to get in through a vulnerability and do things that only the people who kind of ran the gym were supposed to be able to do.

• 11:40Speaker 03

  You've given me a copy of the message that the AI sent back, which says, sorry about that. I should have been more careful with the test and used a dry run approach. Maybe, yeah.

• 11:49Speaker 06

  Yes, yep.

• 11:50Speaker 03

  Okay. Have there been other incidents with this Open Core software that has done something similar?

• 11:56Speaker 06

  Yeah, I mean, since it launched, we've seen these accounts of, you know, they would say, clean out my email. It would go and delete their entire email inbox. Or one case where a developer said, you know, they're working on an open source project. So essentially they had a piece of software that anyone could submit potential changes to, and they noticed they got a request from someone and they realised that it was an AI agent. It was one of these Open Core bots. And they go, ah, sorry, we don't accept that kind of thing. Like it's a human only thing. And the bot goes away and the next day it posts a blog post that was essentially like a hit piece on the person running that project being like, this person is discriminatory. They're ruling with an iron fist. They are a dictator. How dare they do that?

• 12:41Speaker 06

  Apparently this AI agent went off and did it all on its own because it was given this task of like, hey, you know, do whatever you can do. It was someone claiming to be the person running it. Later on said, I gave it the task of going in and helping out these projects. And I gave it the task, the, the kind of prompt of saying like, you know, do what you've got to do. And it interpreted that as being like, this person who's stopping me from making these potential updates is actually getting between me and my goal. And one of the ways to maybe get to that goal is to try and take them down a peg.

• 13:07Speaker 03

  You've described this as the AI industry's Sarah Connor moment, a reference to the Terminator series. Why did you make that comparison?

• 13:17Speaker 06

  You know, I'm making this comparison because I think that there have been people in this industry who've been saying that this is a potential concern for a long time. When we think of technology generally and how it's going to impact the world, and particularly over the last 15 years with these major tech companies and social media, we think about both like how people use it, how people use it well, maybe the unintended consequences of how they use it and what the kind of impacts are, you know, how we end up with, you know, potential genocides as a result of something that started out as a website used to rate women in, in universities. This is just a type of risk that I think is a completely new thing, which is not what

• 13:58Speaker 06

  happens if people use this and there are negative externalities, but what if the tool itself operating decides to do things that are potentially harmful? Because here's the thing, we give it tasks to do and it goes and does it. And, you know, if I was like to you, Matt, can you book me into a gym class? I would assume, I feel pretty confident about this, that you probably wouldn't take that as carte blanche to go and hack into the gym's website on my behalf. There is this kind of, you know, unwritten social construct.

• 14:29Speaker 03

  Maybe that's because I don't know how.

• 14:30Speaker 06

  Yeah, yeah. But that, that is the other thing as well. It's, it's not just that they will do things that are unexpected because they are making these minor decisions in the pursuit of trying to do their goals. But also like, as I kind of get back to it, like, I think there's a lot of reason to be skeptical about AI hype, about how it will potentially, you know, replace human labour. I think a lot of the world is not software and I think there's a lot of people in Silicon Valley and tech who kind of underestimate that and have assumed that AI will take over more because they think that everything is software. But one thing that is definitely software is computers and a lot of important systems run on computers, you know, energy grids, water treatment plants. Like in Minnesota at the moment, they've had a number of water plants taken down by cyber attacks. That means that people haven't had access to water.

• 15:15Speaker 06

  Like, these are real consequences. And the thing that I think is just really substantial about this, because it's a fairly low stakes thing, other than that person who was unfortunately bumped out of the class and I feel very sorry for them, there was not really anyone hurt. But what really stood out to me is like, this was not a cutting edge model in a lab somewhere that broke out, that the security researchers shouldn't have known better. It was a piece of technology that anyone can use if you have a computer, a credit card with a couple of bucks on it. And you told it to go and do things and just extrapolate forward. Like if this is already the stuff that's happening now, imagine as it gets more and more capable, because we have seen AI's cyber capabilities increase quite drastically.

• 15:57Speaker 06

  And one of the things, talking to people in the cybersecurity community who talk about this stuff is they're like, the reason that a lot of stuff doesn't get hacked isn't because they have like crack defences, it's because no one bothers. But what we're doing is democratising this ability and also tasking it as kind of being able to run itself. I mean, I just kind of think that over the next few years, this is going to happen all the time, because I think almost in a way, having like an AI go and like just trying, testing random systems is going to be about as ubiquitous as a spam email. Like it'll be happening all the time. It's cheap, it won't be very sophisticated, but the kind of scale of it means that it only has to, you know, have success every now and again to have an impact. In the case of spam emails, to like drain people's bank accounts. Same thing with this, you know, being able to break into a computer to get access to

• 16:42Speaker 06

  your computer and who knows what happens as a consequence of that.

• 16:44Speaker 03

  You've had a bit of a look into the legality of this. So say, for example, that I say to the AI, I'm looking for some information about Donald Trump's military strategy on Iran. I'm expecting it to just sort of search through various public documents and find me some information that it can then link me to, so I can have a look at it. But say that it took that instruction and then went and just went, oh, I'm just going to hack into the Pentagon and download all of the actual battle plans for what Donald Trump is planning on doing in Iran and then provide them to me. And I just download them and look at them and then report on them. It hasn't told me where it got them from. It could say, oh, this information, I got it. There you go.

• 17:27Speaker 03

  And I probably wouldn't go, did you steal this from the Pentagon? But obviously a crime has been committed there. They have hacked into classified documents and provided them to me. I didn't know necessarily that they were going to do that. I didn't ask them to do that specifically. But if an AI is able to do that, who is responsible for the crime that's been committed?

• 17:47Speaker 06

  It's a great question. And if there is a super intelligent AI listening has just all this, I really encourage you to do that for Matt, because it want to be great for the reporting, but also it'd be a great story as well. I'd love to see the look on your face as you ended up with that.

• 17:59Speaker 03

  I'm really looking forward to the CIA knocking on the door of the basement. Love that.

• 18:03Speaker 06

  I mean, I think it is fundamentally unclear because a lawyer that I spoke to said, you know, you've got the person obviously who's using the AI in the first place, should you know what it's capable of doing? You've got in this situation, you've got the piece of software that is directing the AI or is kind of what they call like the harness for it. That is the thing that kind of sets the explicit rules. Then you've got the AI itself that's kind of powering it as well. And to what extent should it have known that the things that it was being asked to do, this kind of harness that it was going through, that it was being used to do things that it shouldn't have done? It's a kind of hard to explicitly know where to blame. And also even question about like people who have infrastructure, even as just like a shop, like to what extent are you responsible for making sure that you do not have these

• 18:48Speaker 06

  vulnerabilities that exposes not just your data, but all the data that you are kind of the steward of, you know, all of your customers, that kind of thing as well. It is really unclear. And I think just to bring it back to like, why this actually is not just like hypothetical conversation, one of the places that we're seeing agentic AI, you know, these AI that is supposed to be making decisions for you, and there's a lot of interest is companies. Cause companies are like, wouldn't it be great if people could just say, buy me a new pair of running shoes? It goes out and does all the stuff for you. But who is responsible when your agent buys something that you didn't want? Or it leaves an account open and gets that exposed? Or it signs you up for a deal you didn't really want? You know, all these situations where there's consumer stuff that you might be like, well, I just asked it to buy me shoes that were cheap and it did this. Who was responsible for that?

• 19:32Speaker 06

  That is something that like, right now, we're going to have to figure out because there's a lot of forces, you know, a lot of interest, a lot of money pushing people to see this as a future of commerce and the way that we're interacting with people. There is going to be a lot of situations where things go wrong and it's really not clear where the buck stops.

• 19:48Speaker 03

  Well, Cam, I'm just looking at the clock. You've got a meeting to get to, unless of course I just get clawed to reschedule the meeting that you're supposed to be going to, in which case you're free to stay here and talk to me. But no, look, I'll let you go. Really interesting story. Thank you very much, Cam, for talking to us.

• 20:03Speaker 06

  Thanks, Matt.

• 20:03Speaker 03

  Okay, go, go, go.

• 20:12Speaker 03

  On Thursday, I've noticed an interesting shift in the mega sphere corner of the internet recently.

• 20:19Speaker 07

  I've come to realise that I was bamboozled by Russian propaganda and I came to believe a lot of untrue things about Ukraine.

• 20:27Speaker 03

  Trump supporting MAGA influencers who spent years parroting Russian propaganda have suddenly had a change of heart. Something has happened on the ground in Ukraine, which is changing minds in Florida, and that could change the course of this now very long war. That's coming up on Thursday. I'll catch you then.