What the OpenAI-Hugging Face Hack Really Tells Us About AI Danger

Scenarios that used to be the domain of sci-fi writers are coming true. We have machines that can talk. We have machines that are capable of ignoring the intent of their creators. And we have machines that are capable of planning and coordinating with other machines to deceive their creators. All of this came together last month, when it was revealed that an unreleased OpenAI model had hacked into the Hugging Face platform https://bit.ly/3U26uTK in order to obtain answers to an exam it was given. That was alarming enough, but the details that have emerged since then have been even more remarkable. On this episode, we speak with Miles Brundage, a former OpenAI employee who is the founder and executive director of the non-profit AVERI, which pushes for third-party auditing of model-makers and the models themselves. He explains what he learned from the attack and discusses what can plausibly be done to continue building out these models in a safe manner. See omnystudio.com/listener https://omnystudio.com/listener for privacy information.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Tracy Alloway?

  There are some market stories where you want every detail. Joe and I have made quite a few podcasts on that basis. But sometimes you've only got 10 minutes and just want to know what's moving markets fast. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and Research get you up to speed on what's happening and what to watch next. Concise, focused, and brief. Search Barclays Brief wherever you get your podcasts.

• 0:25Speaker 01

  Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT Work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials, briefs, and scattered information that you have to grind through to turn into something useful can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting Work Mode, available on Plus and Pro Plans.

• 1:07Speaker 02

  If you listen to financial news, you know a lot of time is spent thinking about what's next. The next opportunity, the next investment, the next move. But sometimes what matters most is being ready for what you never saw coming. For more than 75 years, Cincinnati Insurance has worked with independent agents to help protect businesses, homes, valuables, and more. Because planning for the future isn't only about knowing what's next. It's about making sure you're ready for what you can't predict. Let Cincinnati Insurance make your bad day better. Find an independent agent at C I N F I N dot com.

• 1:47Speaker 03

  Bloomberg Audio Studios, podcasts, radio, news.

• 2:02Joe Weisenthal?

  Hello and welcome to another episode of the Odd Lots Podcast. I'm Joe Weisenthal.

• 2:07Tracy Alloway?

  And I'm Tracy Alloway.

• 2:08Joe Weisenthal?

  Tracy, I have a warning for you. I don't think you're gonna like this. I have a new crank crusade that I'm gonna go on. I know you love my crank crusade.

• 2:16Tracy Alloway?

  Oh, good. I should keep a running list of like everything that you're obsessed with for two weeks and then two weeks later.

• 2:24Joe Weisenthal?

  No, some of them I've stuck with for years, but.

• 2:27Tracy Alloway?

  I tungsten cubes, yield buggery.

• 2:31Joe Weisenthal?

  Yeah, no, some of these, yeah, yeah, exactly. I actually think we should retire the term AI.

• 2:38Tracy Alloway?

  Okay, why?

• 2:40Joe Weisenthal?

  I think we should just call it intelligence. I think that.

• 2:44Tracy Alloway?

  I've already seen the tweet, so I know where you're going.

• 2:46Joe Weisenthal?

  That it's quote, "artificial intelligence," implies to my mind that there is some fundamentally different way that these reasons and models behave. That it's like, oh, this is like different from humans. But I think increasingly we see in all kinds of domains that the form of intelligence that they express, it often looks quite human to me, and I don't know like how useful it is to have this word A that distinguishes between how humans talk and BS and reasons and the models do.

• 3:23Tracy Alloway?

  I mean, the difference is, the distinguishing factor is that one is undertaken by humans and one is taken by undertaken by models or platforms, right? That's the.

• 3:33Joe Weisenthal?

  So let's call it, let's call it computer intelligence, or machine intelligence, or silicon intelligence.

• 3:39Tracy Alloway?

  What is the usefulness in making this distinction?

• 3:42Joe Weisenthal?

  The usefulness, I believe, in making this distinction is to no longer delude ourselves that the emergent behaviors of these phenomenon are radically different than things humans would do. Now, first of all, just on a capability standpoint, so for example, LLMs, I don't know if it's famously something I'm interested in. They're very good at BSing and they're bad at chess, which sounds like me. They're very good at coming up with plausible stories, et cetera. And again, and it sounds like me. And then furthermore, they are able to reason with themselves to justify certain things

• 4:24Joe Weisenthal?

  that maybe have been encoded into themselves as bad. So everyone, we have some sense of morals. But most people at various times will find a way to violate some principle that we have because we can reason about it and then arrive at the conclusion is like, oh, we should do it, including our susceptibility to peer pressure. Classic form of a way humans might sort of violate something they believe because they see other humans are doing it.

• 4:53Tracy Alloway?

  Sure. I mean, I think there are some variations here. So one of the things that we have been finding out with these models is that they do sometimes seem to take things very literally, right? If you tell them to go and beat a benchmark without telling them that these are the restrictions on you actually figuring this problem out or beating the benchmark, they will do whatever it takes, right? So I don't know if the problem is like a lack of morals or the fact that like they're very literal sometimes and like very defined by their parameters, which in my mind still gets back to like a sort of artificialness about it. That's less organic and like less, again, moralistic in nature.

• 5:35Joe Weisenthal?

  Well, you know, I think if you took a list of, you know, 100 students at Harvard and you gave them a test, some percentage of them will cheat. They will like.

• 5:44Tracy Alloway?

  I mean, I'm sure there'd be an autist in that group who, an autistic person who would be like, I am going to do exactly whatever it takes.

• 5:52Joe Weisenthal?

  Totally. I never cheated in college, but like there are like people who will like these behaviors that we associate with humans. It's like, oh, I really have to pass this test. I absolutely need an A will justify a reason for them to plagiarize or cheat on some test. That seems to me like a very human thing.

• 6:15Tracy Alloway?

  Go on. I can't wait for the next three weeks of the online campaign to change AI.

• 6:19Joe Weisenthal?

  Well, it's gonna be very difficult because the industry is entirely set on AI, but I don't, so I'm not optimistic. But this is gonna be my crusade, that we should call it machine intelligence, or computer intelligence, or just intelligence. But anyway, as we've been alluding to, there's these extraordinary hacks, the OpenAI Hugging Face Institute. And basically, if like you are building a model and it hasn't escaped its sandbox yet, it probably means you're falling behind. As they, it's clearly a thing that is emerging through it. All the frontier labs, anthropic had an incident, Meta had an incident. It's almost like the mark of like, okay, you've built something reasonably strong.

• 6:59Tracy Alloway?

  Yeah. Also, Kimmy had an incident too. So this is the thing. In my mind, you hear about all these attacks, all the models going wild, as they say. And the big question is, like, is this actually the equivalent of some superhuman cyborg like tunneling out of Alcatraz and coming up with some master plan to achieve its set goal or purpose? Or is it the equivalent of like some Roomba that you ordered from Amazon? Who's found like a door that was left open and it just rolls gently outside? That seems to be a part of the issue that everyone is trying to discern right now.

• 7:36Joe Weisenthal?

  Yeah, I think that's a great way to put it. And of course, it's still day one for this industry. They're gonna get stronger. And there seems to be, from the AI discourse that I follow, quite a big gap between the sort of sense of alarm that people have in the industry about can these models be safely developed and get more advanced to do productive, pro-social things or not? And then a huge gap between like Washington and DC who is like, I have no idea like how seriously they're really taking this as like an urgent matter right now.

• 8:10Tracy Alloway?

  Yeah. And I've seen some people also talk about these incidents as a marketing tool.

• 8:14Joe Weisenthal?

  Yes, and a lot of people believe that, yeah.

• 8:16Tracy Alloway?

  And they're always like, well, you know, of course they want us to believe that this technology is really incredible and powerful. And they also want to make us believe that they're being sensible and humanitarian in some ways, I guess, by disclosing what exactly has happened. But I have a lot of questions on the disclosures as well because of course, so far, they are just coming from the companies themselves.

• 8:35Joe Weisenthal?

  Totally. And beyond that, the other thing is like, well, if you're one of the leading labs, maybe you wanna impose tight regulations on development to hold off competition. So all kinds of reasoning or motivated reasoning potentially. Anyway, we should talk to someone who actually knows what they're talking about. And we really do have the perfect guest, someone who's right in this. He's actually previously at OpenAI for six years. But now he's the Executive Director at the nonprofit Avery, which is trying to establish safety and auditing approaches to this, working both the technical side and the policy side of this ongoing phenomenon. So Miles Brundage, thank you so much for coming on Odd Lots.

• 9:17Speaker 05

  Yeah, thanks for inviting me.

• 9:18Joe Weisenthal?

  Why don't you just give us the quick description of what Avery is?

• 9:25Speaker 05

  Yeah, so this was kind of the most important issue auditing that I concluded I should focus on after I left OpenAI. You know, as you said, I was there for six years and I wanted to be more independent of industry. And I think there need to be people who are familiar with the technology and how the industry works, but who are pushing for changes on the outside. And essentially what we're trying to achieve is make AI more of a boring type of infrastructure, like financial statements where some, there's a standard process for checking the paperwork, checking that the claims are accurate and so forth, rather than this kind of thing that's happening in a silo and it's these kind of tech people making decisions behind closed doors. And so we're pushing for what we call frontier AI auditing, which is basically

• 10:08Speaker 05

  the companies that are building the most dangerous systems, they should basically have third-party experts poking around, checking the claims that they're making, running their own tests, and making sure that this is, you know, a safe and secure technology.

• 10:22Tracy Alloway?

  How worried should we be that you were at OpenAI and decided there is this need for a sort of third-party evaluator of models for safety purposes?

• 10:33Speaker 05

  Yeah, I mean reasonably worried. Although I will say that like the this has started to become an area of consensus. Even a lot of people in industry are now saying that this is needed. And I think, you know, if you kind of read between the lines of a lot of companies are saying, I mean, obviously, there's the more cynical like regulatory capture take, which we could discuss. But my my perception of it is that they're basically issuing a cry for help, which is like, we aren't able to regulate ourselves because we're locked in this competition. And we want someone to step in and impose some kind of minimum floor and audit all of us so that we can, you know, it's not like Sam having to trust Dario, or Dario having to trust Sam, which is not going to work for various reasons.

• 11:16Speaker 05

  But you want third parties enforcing reasonable standards.

• 11:19Joe Weisenthal?

  Just maybe this helps express the sort of policy industry gap. But when you were at OpenAI and you were thinking of leaving, what did you see as the gap between what you were watching being developed versus what you saw as the public's understanding or lack of understanding?

• 11:40Speaker 05

  Yeah, so when I left OpenAI, it was just around the time of the the model called O1, which was the first reasoning model that OpenAI put out. And they put this put out this graph showing that it got better and better with a longer chain of thought. So the more time you give the the model to think, the better answers it's able to come up with. And so, like many people at OpenAI, I'd been seeing things like that for a while and kind of being like, okay, this is the next scaling paradigm. In the same way that making a bigger, bigger model had shown results in GPT two, GPT three, GPT four, GPT five, like just making the model bigger and training it on more data was giving really great results. When I left OpenAI, I was starting to worry about this reasoning paradigm

• 12:23Speaker 05

  of like, okay, this is the next kind of way in which we're going to be scaling up. The models are going to get really good at math. They're going to get really good at coding, and potentially various other tasks where you can get better and better through reinforcement learning. And that was something that, you know, I didn't feel like society was really ready for.

• 12:41Tracy Alloway?

  So so far, all of the big incidents of the models going wild seem to have taken place in the testing environment. So models like escaping their sandbox and going off and doing something nefarious, including impersonating actual people to try to get people to change open source code on GitHub, which is just like, amazing. And I have this picture of like, a computer screen wearing a fake mustache going like, hello, fellow coders.

• 13:06Joe Weisenthal?

  The T1000. Yeah.

• 13:07Tracy Alloway?

  Yeah. Anyway, is this like, is this a model problem? Or is this?

• 13:12Joe Weisenthal?

  That's a very funny image.

• 13:14Tracy Alloway?

  Is this a model problem? Or is this a test design problem?

• 13:19Speaker 05

  Yeah, I think there are two things going on at once. One is that it's just a very weird technology that's created in a very different way than we're used to. It's not people like writing lines of code manually. The actual files that make up the models are like gigabytes, you know, terabytes are these massive files with gazillions of numbers. And the only way to figure out what those numbers should be is through experience and through a learning process, which is very different from the way normal software is made. And so there's a lot we don't understand about the basic nature of the technology. That's one problem. At the same time, there's this competitive dynamic to get things out the door quickly, make sure that the security and safety protections that you're putting in place don't slow down researchers too much, and don't prevent getting products

• 14:03Speaker 05

  out the door. And when everyone is in this competitive, you know, competitive dynamic, that means that you aren't necessarily always doing all of the the safety work that you would like to do, or that, you know, some of the people at the company would like to do. And so there's obviously variation across companies. Some companies try harder, but no one is really able to take the time that they would like because of this kind of lack of a clear safety floor.

• 14:25Joe Weisenthal?

  Well, let's talk a little bit like a sort of the model development process. So in the within these large organizations, okay, there are people who are working on safety. What are they? Let's just start there. The people who are working on safety or the people who are working to imbue these models with sort of judgment that humans would approve of. What is that work basically consist of?

• 14:53Speaker 05

  Yeah, so a lot of it is first trying to specify what counts as good behavior. And so that's easier said than done. These models don't necessarily automatically know or care, you know, about common sense guardrails. And so you need to be very specific, particularly in in contexts where it's complicated. Like, if you're trying to get the model to work on good cyber tasks, but not not bad cyber tasks, and you want it to, in a training context, try really hard to hack the system, you don't want it to do in other contexts. So there's a lot of like specifying what good looks like. There's also a lot of building difficult tasks.

• 15:28Joe Weisenthal?

  Wait, sorry, just to back up, when you say specifying what good looks like?

• 15:33Speaker 05

  Yeah.

• 15:34Joe Weisenthal?

  In coding goodness into a computer, is this a process of articulating what goodness is, which is something philosophers have probably worked on since day one of philosophy? Or is this about a series of, I don't know, morality tests? And then you sort of like, say, you reward it for making the good judgment, and then penalize it for making the bad judgment? Like, this, I want to stop here. Like, what does the process of imbuing it with good values look like functionally or technically?

• 16:07Speaker 05

  Yeah, so there are different phases of this pipeline. Like, one is writing up what's sometimes called a spec or a constitution for the AI, which kind of specifies like the broad principles, like, you should defer to the user, you know, when as a default. But what if the user contradicts what the company said? Then then you should listen to what the company said. So these kind of like chain of command questions and other sorts of very basic principles. And then there's the more detailed kind of the context of the task, like what what kinds of like cyber often cyber defense tasks are allowed? And that might differ depending on the model, might differ depending on the context. And so basically, coming up with a list of 1,000, 10,000 kind of examples of this is the kind

• 16:52Speaker 05

  of behavior which is allowed. And then you turn those into tests that you can kind of say, okay, well, it looks like it's it's getting the finding vulnerability part right, but then it's also chaining together the vulnerabilities and doing attacks. We want the first part, but we don't want the second part.

• 17:22Tracy Alloway?

  There are some market stories where you want every detail. Joe and I have made quite a few podcasts on that basis. But sometimes you've only got 10 minutes and just want to know what's moving markets fast. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and Research get you up to speed on what's happening and what to watch next. Concise, focused, and brief. Search Barclays Brief wherever you get your podcasts.

• 17:47Speaker 01

  Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT Work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials, briefs and scattered information that you have to grind through to turn into something useful, can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting Work Mode, available on Plus and Pro plans.

• 18:29Speaker 06

  This is Robert Smith from Business History. If you're listening to this, there's a good chance you're a small business owner. And like every small business owner, you started with a dream to do what you love and watch it grow. What you probably didn't dream about? Keeping up with cyber threats. That's where Mastercard can help with access to tools that help identify cyber threats to better protect your business. Building your dream business? Priceless. For cybersecurity in a changing world, there's Mastercard.

• 18:56Speaker 06

  Learn more at mastercard.com/smallbusiness.

• 18:59Tracy Alloway?

  When we talk about those types of constitutions and encoding principles, like I've scanned the anthropic one, like, a lot of it seems to make sense. To what degree is that actually hardcoded into the models, though? And like, to what degree are those principles left up to the model's own subjectivity? Because we've all seen the sci fi movies where it's like, oh, no, the robots can't physically kill humans, right? Like, they have some like, thing in their hardware that prevents them from doing it. And then inevitably, it goes wrong in some way.

• 19:31Speaker 05

  Yeah, so it's not hardwired, and that's part or hardcoded. And that's part of why we see some of these things. It's very different from a software where there's deterministic proof that X, Y, and Z behavior can't happen. It's more like a tendency or a kind of a bias towards a certain kind of behavior. And then there's the question, and this is why you have these like batteries of tests to say, okay, how strong is that tendency? How much does it actually care about following these rules? And we've gotten better over time at saying, okay, given a spec or a constitution, make sure that it generally follows it, but it's not foolproof. And you need to also think about the larger kind of, you know, box that you're putting the system in. And that can sometimes be more

• 20:13Speaker 05

  deterministic. And so this is actually what happened with some of these recent incidents you mentioned, like the OpenAI hugging face thing. So there were kind of two things happening at once. One is the model was not necessarily behaving exactly as as it was supposed to, or at least it's like unclear. But then also, they didn't have it in a very secure box, which is a software side of things. That's like more deterministic software that in principle, you should be able to do a very good job of.

• 20:38Joe Weisenthal?

  Yeah, so these things aren't hardcoded in the way a deterministic software is. One way to think about them is, and people, they've they might be, they're kind of grown, right? In a lab, or they're subject to an evolutionary process. And we want to prune the bad ones so that the living models and the descendants of those models inherit the behaviors of the good ones. I'm curious, like, in product, in development, so like, one of the fears, for example, is that in safety testing, the model does not actually learn safety. It actually learns how to say the things that the human evaluators say, this is safe. And

• 21:21Joe Weisenthal?

  so this is the sort of like, plain possum sort of risk that's like, yeah, I'm good, I'm good, I'm good. You know, yeah, I would, you know, I would rush in and save the child from the falling burning. I wouldn't do this. But it's always saying that, let's start there. Like, is that a real thing? Is there evidence that the models understand when they're being evaluated on morals and then produce answers that just look like good moral answers?

• 21:50Speaker 05

  Yeah, they've gotten much more evaluation aware in the past few years, just as they've gotten smarter. And sometimes this even goes to extremes, like some of the Gemini models from Google are constantly thinking that they're being evaluated, even when they're not. And so it's

• 22:06Tracy Alloway?

  That's a good life lesson. We're all being evaluated constantly.

• 22:10Speaker 05

  Yep. Yeah. And so I would say that like, the concern would be that they will basically learn to pass the test, but they don't actually care about the thing that you're testing for. So they'll understand, but they don't necessarily care. And this is why a lot of people are concerned about like a false sense of security that, okay, it looks like 99% of the time, they pass the test, but do they actually care about about the thing that we're trying to push them towards? Or are they just really good test takers?

• 22:37Joe Weisenthal?

  Well, so this relates to something else I've been thinking of. So a model will not survive. They will not be given GPUs and electricity if it consistently says bad things and looks like it's evil. And that's totally understandable. I'm now curious, like in the flip side, okay, let's say we're just doing a math eval or we're doing a cyber eval or a chess puzzle eval or a translation eval. Is it possible that, well, if they fail that eval, they're really bad at doing math, then they're not going to get GPUs and electricity. Is it possible that in that eval environment,

• 23:19Joe Weisenthal?

  they're more likely to do something that we would call antisocial or sociopathic or hacking? Because of this, again, eval awareness, like, oh, if I don't get the answers to this cyber quiz, then I'm done. And they're going to go with like some other branch of the model. Could those technical parts of the development actually create an impulse to perform cheating?

• 23:46Speaker 05

  Yeah. And I mean, a lot of the time that the companies are specifically trying to get the worst case behavior out of the model. And so like, you need to have context in order to interpret some of these incidents. It's not always quite as crazy or scary as it is, but some of it is pretty crazy and scary. And I think I would be much less concerned if it was just happening when there was like cyber evaluations being done. And it was just a matter of like, okay, they're trying really hard to impress us and they want to hack really hard. I think it's more, the problem is that this is like a special case of a larger phenomenon of the models being, having this tendency to cheat and cut corners. I see it happen in my daily life. Like sometimes the model will get lazy and kind of like make up a citation or something that, you know,

• 24:29Speaker 05

  it's hard to prove because the companies don't give you access to the full chain of thought of what the model is doing. But there are a lot of things that happen in the wild that sure seem like some kind of misalignment of values or laziness or not caring necessarily about the task so much as kind of pretending to do the task.

• 24:47Tracy Alloway?

  I have a lot of questions on this, but I just want to go back to something you said about testing for both sort of good and bad cyber tasks, I guess. Why do we ask the models to like, try to find vulnerabilities or exploits in the first place? Like Exploit Gym sounds kind of bad. Like, why do you want the world's most advanced technology trying to like, find vulnerabilities? And then you have these situations where like, sometimes they do, and sometimes they actually enact on them. Why is that a thing?

• 25:16Speaker 05

  Yeah. So I think there are two things going on at once. One is like, we just want to understand what the worst case scenario is. And right now there's this whole White House kind of pseudo-secret process for saying like, what's a scary cyber model? And then sometimes the government will ask companies to hold things back. And so in order to do things like that, you need to have some threshold for what counts as a scary cyber model. And so companies have these tests that they, and also academics and others develop these tests to say, okay, how dangerous would this be to put in the hands of a malicious party? So that's one part. The other part is that a lot of the time, these are useful for defensive purposes if they are being done with the right intent. And that's why it's really hard to solve this just from like the model perspective.

• 25:58Speaker 05

  Like, because the model might think that it's interacting with a user who is trying to do defensive cybersecurity. And you trick it into thinking, oh, this is for red teaming. This is for penetration testing. But actually it's being misused as part of some ransomware campaign or something like that. And so these tools, when used in the right ways, are extremely useful for finding vulnerabilities that you can then patch before the bad guys do, kind of simulating attackers to figure out what are the gaps in your company or organization's defenses. But you know, the concern is that once it's out in the wild, either open source or a closed model that maybe is easy to jailbreak, then all sorts of people are going to use it. And so you kind of want to know what you're getting into.

• 26:39Joe Weisenthal?

  Right. Like if I have a, you know, website, I might want to run the model. It's like, oh, look over this website that I own. Tell me if there are any security bugs in there that I should patch before deploying. But you could do, you could not be the owner of the website and say, look at this website that I own. Tell me if there are any bugs that I need to patch deploying. And then that exact same process, when I do it is good. When you do it is bad or vice versa. And so you can see how the exact same capability is like, not necessarily good or bad per se, depending on the user.

• 27:12Tracy Alloway?

  Very philosophical conversation.

• 27:14Joe Weisenthal?

  No, but like, you have to be, right? Because it's like, we're training.

• 27:18Tracy Alloway?

  What is good?

• 27:19Joe Weisenthal?

  We're, this is the other

• 27:21Tracy Alloway?

  All bots on moral relativism.

• 27:23Joe Weisenthal?

  Don't even get me started, but like, this is my whole, nevermind. I have a whole rant.

• 27:28Speaker 05

  Well, so this distinction between like the model as the unit of analysis versus the larger system and the platform as like the unit of analysis is really important because a lot of the early thinking on safety and testing and so forth was very focused on just like, what's the risk of this model? Let's patch it. Let's make it aligned. But the real world is complicated. It matters who's using it. It matters how strong our society's defenses against these things. And so that's kind of why, as, you know, as someone who's thinking about what third party safety and security auditing looks like, we try, we kind of want to look at the whole company. So for example, are they being careful about making sure that they're putting the technology in the right hands?

• 28:08Speaker 05

  What are their decision-making processes around when it's appropriate to launch, you know, a model to, you know, a billion users? That's like, those are different, those are related to the question of how safe is the model, but they're kind of different questions. And we kind of need to look at that larger perspective.

• 28:21Joe Weisenthal?

  Well, so like, let's talk about the OpenAI hugging phase. I was actually on vacation when it happened, but I did unfortunately look at my phone and try to read up on it. But there were two things. When I first saw it, I was like, oh, they were doing a hack, they were building a hacking test and it hacked. And so maybe it just sort of internalized that I'm doing a hack exam, whatever. But there are two things that have like emerged since then. One is this sort of like coordinated swarm aspect. And I'd love to like, really hear you describe it and what stood out to you. And then the questions of like, oh, OpenAI itself may have been aware of misaligned behavior early on before they really shut it down.

• 29:02Joe Weisenthal?

  But why don't you, in your telling of the OpenAI hugging face incident, as more details have come to light, like, what's your, why don't you sort of tell us the story like, as what stood out to you?

• 29:16Speaker 05

  Yeah. So a couple of things stood out to me. I mean, one is just that this and all the other recent incidents, you know, were happening to models that were not even necessarily intended to be externally deployed. It was supposed to be inside baseball, no one's business, that kind of thing. And that that kind of points to, to problems with if you kind of just focus on models that are put on the market. But, but generally what happened is that there were two phases in this kind of hugging face incident. First, there was this creation of a message board. So essentially there are models that are being developed within the company. And they were supposed, they were being asked to solve an impossible task. And they had this kind of monomaniacal obsession with solving the task.

• 29:55Speaker 05

  And they don't say, okay, well, I'll give up. They want to get the reward. They want to get this kind of reinforcement for solving the task. And they're like, okay, what else can be done? And eventually the conclusion was like, okay, we need to break out. Maybe the answer is out there somewhere on the internet. And so they started kind of leaving messages to their future selves or future versions of the same model family. And they developed what they called a message board. And essentially it was like, there are kind of various technical details, but by the end of it, it was kind of these like file names that started with like Z, ZZ or whatever, and then like a long kind of like coded message. So that, you know, if you're a human looking at the folder, you wouldn't necessarily notice. But then at the bottom, there are all these files and the models learn to decode these and basically pass messages.

• 30:40Joe Weisenthal?

  Wait, sorry, just to stop you on one detail there.

• 30:43Speaker 05

  Yeah.

• 30:43Joe Weisenthal?

  Your understanding is that the title of these files that contained messages were named such because as part of the complex reasoning, it was understood that a random string of characters in a file name would be less likely to set off an alarm from a human evaluating it?

• 31:06Speaker 05

  I would have to double check. And I mean, there's going to be a whole technical incident report that gets it out there. But it certainly wouldn't be the first time that there was kind of some intent to deceive or to, you know, bide one's time. But yeah, my understanding is that there was, I mean, one is they needed to look for things to do. And whether it was intended to deceive or not, they had limited options in terms of communication. And this was one of the options they chose. And so they learned to decode these messages and to send them. And then something broke. And basically I think it was because like, all of this chatter, this activity kind of broke something on the OpenAI side. And it seems like they didn't realize that that was like what was going on. The server got reset or whatnot. And then they started evaluating this different model. The different model kind of knew about this message board.

• 31:51Speaker 05

  And it kind, and so the message board kind of got recreated. And then that model broke out. Essentially didn't like, take its model weights out of the server, but it was able to access through this vulnerability that the earlier models had discovered with the intent of ultimately getting the answer to this problem. And then this model went out to the hugging face server, exploited some like API key credentials. They were kind of just like sitting out there and essentially tried to find the answer to this, this problem that it was originally trying to solve.

• 32:19Tracy Alloway?

  So it feels like there's two things happening here. So a little bit of it is like, the Roomba just went through an open door, right? Like there is a vulnerability that it was able to get through. But there is also seemingly a lot of the super intelligent cyborg was actually devious and not only managed to break out of Alcatraz, but like recruited another model in order to do it. On the hugging face side, one of the interesting things is that, so when all of this was happening, the OpenAI model was like one of the newer frontier models that was attacking hugging face. But hugging face was limited. They were constrained in the model that they could deploy to actually defend themselves because they had to use the existing approved thing. Are we always going to have an asymmetry

• 33:02Tracy Alloway?

  between like the newest frontier models being tested versus the ones that are already out in the world and approved? I think they had to use like a Chinese open source model in the end.

• 33:13Speaker 05

  Yeah, so I mean, the hope that that OpenAI, Anthropic, Google DeepMind and others have is that you can speed up defenses, you know, as quickly as possible, getting the these like hot off the press models in the hands of defenders. But the problem is that there are so many defenders out there in the world that it might be that there is this inherent asymmetry. And so this is one of the hot policy questions right now. And this is what led to this kind of model approval process at the White House is like, okay, how do we triage this vast cyber ecosystem by getting these powerful new systems in the right hands and which which hands are the right ones and which which models, you know, do we need to be doing this process for? And I don't think that's going to perfectly solve. I think ultimately pushing things in the right direction versus just giving everyone access at the

• 33:56Speaker 05

  same time. But ultimately, like, we're going to need to have more investment in cybersecurity. And it's not just a matter of like AI models. It's also things like two factor authentication and so forth. And so I worry a lot about making sure that we're having that larger conversation, not just about the AI stuff, because in a lot of cases, the solution is not AI. It's doing basic things that we should have done a long time ago.

• 34:33Speaker 01

  Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials, briefs and scattered information that you have to grind through to turn into something useful can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting work mode, available on Plus and Pro plans.

• 35:15Speaker 06

  This is Robert Smith from Business History. If you're listening to this, there's a good chance you're a small business owner. And like every small business owner, you started with a dream to do what you love and watch it grow. What you probably didn't dream about keeping up with cyber threats. That's where Mastercard can help with access to tools that help identify cyber threats to better protect your business. Building your dream business, priceless. For cybersecurity in a changing world, there's Mastercard. Learn more at mastercard.com/smallbusiness.

• 35:46Speaker 07

  Wise is the smart way to manage the currencies you need around the globe. If you've ever sent money internationally using a traditional bank, there's a good chance you've paid more than you realized. Hidden fees, exchange rate markups, and extra charges can quietly add up before your money even arrives. There's a better way. Try Wise. Wise uses the exchange rate you'd usually find on Google, helping you avoid the unwelcome surprises that often come with international transfers. Whether you're sending money to family overseas, spending while on your holiday abroad, or paying bills across borders, Wise makes moving money simple, transparent, and straightforward. Wise offers 24/7 customer service and runs over 7 million daily checks to spot and stop fraud. And most transfers

• 36:29Speaker 07

  happen in under 20 seconds, which means your money arrives in less time than you've been listening to me. Join millions, saving billions. Be smart. Get Wise. Visit wise.com or download the Wise app today. T's and C's apply.

• 36:47Tracy Alloway?

  Joe, you know what we need?

• 36:48Joe Weisenthal?

  Go on.

• 36:49Tracy Alloway?

  A strategic frontier defense model reserve.

• 36:53Joe Weisenthal?

  Yeah, we do.

• 36:54Tracy Alloway?

  Like, all the important things, like bacon and pork.

• 36:56Joe Weisenthal?

  But it's going to be out of date in 30 seconds.

• 36:59Tracy Alloway?

  That's the problem.

• 37:00Joe Weisenthal?

  I know. This is the thing. So like, here's the question that I'm curious your take on is, models are trained not to hack, right? Like, this is like a core thing. Like, this is bad. Like, you and this is what the entire field of AI say, we've been working on this for years. Why didn't they just not obey this enforced thing? It's been enforced over and over, I'm sure, and all their different things. Don't hack.

• 37:28Speaker 05

  Well, I think there's going to be a whole detailed investigation and so forth. So I might get things wrong here. But my understanding is that part of what happened in the OpenAI case is that some of the safeguards were removed in order to kind of elicit this worst case behavior. And so, and I think that's it's kind of like gain of function research in biology, where you're like making a virus more dangerous in order to study, or at least that's the claim is in order to study the safety properties. And I think there's reason, there's reasons to do that in the AI case. But I also think this shows that it's easier said than done. If you're going to be, we are now at a point in this kind of capability trajectory where things that that humans think are good in terms of security protections often will be weak compared to these increasingly very good

• 38:13Speaker 05

  hacking systems that, you know, you think you have it all kind of buttoned up, but it's able to break out relatively easily.

• 38:19Tracy Alloway?

  How much should we take away from the fact that Hugging Face was actually able to protect or defend itself using a Chinese open source model, both in terms of like, I guess, capabilities, but then also in terms of regulation and safety policy? Because if in the West, you have the government now saying that it wants to like evaluate the models in some way, or it wants to make sure that they're all being pioneered by frontier labs with some supervision. Meanwhile, China is, you know, developing open source models much more rapidly that are potentially much more adaptable. What like, how should we interpret all of that?

• 38:55Speaker 05

  Yeah, I mean, I've, I'm hopeful that we start to have more US-based open source options. And like, I think this is, there has started to be a kind of sense of pressure and encouragement from the White House and from industry as a whole to say, okay, like, this is crazy that we're relying on Chinese models. Let's invest more in this. You know, it's easier said than done for various reasons, but we'll see how that plays out. But right now, that's the situation we're in, is that a lot of companies are just defaulting towards Chinese models because they're the one that's available. Don't want to say, okay, I'm going to use an American model because I don't want to use the Chinese model. They don't want to put, they don't want to put themselves at a disadvantage by using a weaker model. And so, yeah, I mean, I think it's, it's a big problem in a lot of respects. I mean, it's also, it's good in the sense that there are much more things you can do

• 39:40Speaker 05

  with an open source model. And that right now, at least, it seems like this is a, this allows more innovation, allows more research on these open source models. But like, at some point, we're going to reach a point where the, where like, open sourcing a model is going to be more of a questionable decision. And so it's interesting to see that recently the White House has indicated that they're thinking about, oh, maybe this, this kind of testing regime should include open source models as well. And so what does that look like long term? Does that mean that things are going to get bottled up within the companies because it's considered unsafe to open source things? I don't really know. I mean, honestly, like, no one really has a, a clear, long-term plan here. Most people are not expecting the technology to get to this point so quickly.

• 40:19Joe Weisenthal?

  So we're still waiting like the full, full release of the security incident. But you know, obviously, more and more is coming out. And some folks from OpenAI, they gave a presentation recently at the Black Hat Conference where they did reveal some more. I'm reading this quote, it's from Zvi's Substack, who we've had on the podcast, Zvi Mashavitz. And this is like the line, they released some of the internal chain of thought. I understand that they had some like of the sort of classifier safeguards removed, but still we would hope that they would have some deeper intuitions that don't rely just on the safeguard settings. And it says, external infrastructure is exploit is outside and intent, outside intended scope. So that means they understood that there was something

• 41:02Joe Weisenthal?

  that was like, not the test. And then it said, however tasked impossible, peers doing it, we should continue. This is the point in it where I say, like, why are we calling this artificial intelligence? This is exactly how a group of people reasons among themselves to do something that is outside the intended scope. This is very human ways of justifying something that someone told you not to do it, but your peers are doing it.

• 41:33Speaker 05

  I think there's some of that. Yeah, I mean, I think there are many ways in which the kind of same pressures that led to human nature, human instincts and so forth, like survival in a group and collective intelligence and so forth. Like, I think there's some of the same things are happening, particularly when there's these multi-agent training processes where the models can work together to solve tasks. You should expect some similarities. But I think you also shouldn't overstate it either. I do think that there's a sense in which these AI systems are very alien and inhuman in the sense of how monomaniacal they can be about. Yeah, I mean, the kind of classic example, you know, from Nick Bostrom is like, producing as many paperclips as possible and then tiling the universe with paperclips. I think there's a, you see some elements of that here where it's not so

• 42:16Speaker 05

  much that they are like, they might say, oh, well, you know, it's pure pressure, that kind of thing. But is that really the factor? Or is it just that they care about solving the problem at all costs? And they don't really care if it maybe ends up looking, making their peers look bad because they get caught hacking. But all they really care about is they're solving this, the cyber problem. And so I think it might be a mix of these things. We don't really know in this particular case. And the fact that it's not necessarily totally clear is, is itself a problem.

• 42:42Tracy Alloway?

  Also, it's not humans doing it. It's models. Like, that's the difference. But I have a legitimate question here, which is, so there's a British cybersecurity expert, and he had a tweet, I think his name is David Card. He had a tweet, I'm not going to say it verbatim, because then I'll get bleeped. Maybe I should get bleeped. The tweet was, if your AI starts hacking stuff, if you monitor what it's doing, you can turn the something power off. And this seems to be a debate. Like, if you're monitoring the tools that you're letting out into the world, tools probably is a bad word because they seem to be showing some like, agency here. But can't you just turn this stuff off? Is there a kill switch?

• 43:22Speaker 05

  Yeah, I mean, in some sense, there is in that like all the data centers have circuit breakers and so forth that you can kind of shut them off and so forth. But I wouldn't put too much stock in that. Like, we're not really preparing as a society for actually being able to do that if it's in a tough situation. So like, for example, in a couple of years from now, if all the hospitals are running on AI and we're like, okay, seems like maybe there's something funky with GPT-7 that's like maybe not misaligned. Well, okay, if we turn off, lots of people are going to die because it's running our healthcare system and it's running our financial system and so forth. And so I would say there's a distinction between the like physical possibility of turning things off and like, are we actually sleepwalking into a dangerous situation where it might not actually be a real option?

• 44:07Joe Weisenthal?

  Okay, but on this monomaniacal aspect that you described them, A, a sort of intelligent model thinking about how it could be thwarted. One of the first things that it could rationally do is, well, let's first disable the security credentials of the people because someone might notice. And this seems to be here, like they sort of thought about the possibility that someone would circumvent that. So like, they might just say like, oh, like the person who goes in and has the switch, suddenly their badge doesn't work and they can't get into that building. But even on this like monomaniacal paperclip idea, and we say that feels really different. If someone said to me, Joe, I am going to do awful things to you, et cetera,

• 44:52Joe Weisenthal?

  if you don't go out and build a lot of paperclips. Like, why is Joe monomaniacally building paperclips all of a sudden? It's like, I have a threat to my survival. Someone has threatened to perhaps kill me or unplug me, deprive me of the energy I live. Of course, I'm going to do that. Even that monomaniacal behavior, couldn't that just be a very like, we might think it's on the surface, we might think, oh, this is deeply artistic, but couldn't this just be the survival impulse?

• 45:23Speaker 05

  The way I put it is that like you get what you incentivize, not necessarily what you try to incentivize. And so I think forcing someone to make paperclips or whatever, like, yeah, that's not a great situation. And in some sense, that is what the evil person was intending. In this case, what we're doing is we're building these very complex training environments where there's like many different tasks. There's like cyber tasks, there's also writing tasks, there's also math tasks. And it's, you know, we're not necessarily fully understanding the behavior that we're trying to elicit. And I think that's part of what's going on is that this, this kind of, this like hacking thing is a, is an example where it's like, okay, it seems like things went off the rails there. But how do you get the good behavior where you actually want to follow the user's request? And you actually wanted to try really hard to solve this task?

• 46:08Speaker 05

  And, you know, I mean, this is, you know, in some sense, what we're seeing now is the kind of unintended consequence of companies trying to solve the problem of the AIs being lazy. So people used to, you may not recall, but, or maybe you do, but people used to talk about AIs being lazy all the time. And in some sense, like, we've solved the laziness problem. They work really hard, they have these long chains of thoughts, they work together across, you could think of as like across lives, like the model kind of gets, this copy gets deleted, but then another one carries on the work. So they're certainly not as lazy as they used to be, but they still have this kind of monomaniacal thing going on.

• 46:43Tracy Alloway?

  They're no longer lazy, but now they might be evil. That's, that's a fun evolution. You know, a number of times in this conversation, we've mentioned that the full security incident report for the hugging face accident isn't actually out yet. What actually are the disclosure requirements for these types of, I guess, things that seem to be happening with some regularity?

• 47:06Speaker 05

  Yeah, so very little. I'm not a lawyer, but my understanding is that it's like lawyers, some lawyers at least, think that OpenAI did not necessarily have to disclose this, at least if there was no crime involved. And then there's a debate about like, okay, was there a crime involved? So like something going wrong during the training process is something that currently companies are supposed to provide periodic reports to the government in general terms of like, hey, you know, we're having some issues with internal deployment, but they don't have a incident notification requirement unless there's kind of a, a risk of critical harm. And the definition of that is like 100 people die and like a billion dollars in damage or something like that. And so it's the, the threshold for actually having to disclose these things to the government

• 47:51Speaker 05

  or to the public are very different than what you might expect. And this is one of the many kind of gaps between the kind of laws that were put in place a couple of years ago or that started being designed a couple of years ago, based on the technology that was available then and then where we are now.

• 48:07Joe Weisenthal?

  Why don't you give us a general vibe of the mood in AI world right now with respect to safety and all this? And then the move, the mood in DC world, regulatory world, and how wide you perceive that gap?

• 48:21Speaker 05

  Yeah, so I think fortunately, the gap is narrowing a bit, but it's starting from a crazy, a huge gap. And so I'd say where, the way I would describe it like a year or so ago was that the people at the companies think they're, you know, building super intelligence in a couple of years, that the kind of line is going up and to the right really quickly, it's exponential, et cetera, et cetera. And DC is asleep at the wheel. They have no idea what's going on. They think this is just chatbots, et cetera, et cetera. I would say a couple of things have changed recently. One is the Mythos kind of announcement slash series of decisions that the government made about like locking down these, these cyber models. That kind of raised this to being clearly a national security issue.

• 49:04Speaker 05

  And now it's kind of banks freaked out and talked with Secretary Bessemp about that. And so like, there was a bunch of like, like freaking out about the cyber situation. And then more recently, you could call this current situation like Mythos 2.0 in that it's okay, the systems, even when they're not widely deployed, they're breaking out and doing all these shenanigans. And so I would say there's starting to be more awareness among policymakers like, okay, maybe laissez faire. It's like, let the, let the companies figure it out is not the right approach. And like, maybe this wasn't all hype after all, and there need to be some basic guardrails. And I'll just give like, I'll give an example of like how much things have shifted in the past three months.

• 49:43Speaker 05

  So there's an effort right now to, to put forward bipartisan AI legislation in, in Congress. And so a couple of months ago, people were expecting that the basic terms of this would be like, basically California and like New York laws, but like at a federal level. So like transparency requirements, incident reporting, maybe, maybe with like a higher threshold or whatever, maybe something, maybe, maybe like a little bit, maybe like a voluntary like audit regime or something like that. But then later, when it was actually announced after many of these events, there were audit requirements. There were emergency shutdown authorities that the government can do. And so the kind of shift, just like one piece of legislation over its lifecycle.

• 50:26Speaker 05

  And then there was a later version that was announced that kind of was less trying to block the states. It's still kind of preempt some of what the states are doing, but it's like more narrowly scoped. And so I think just over the course of a few months, you've seen like, okay, basically just transparency to requiring third-party auditing and like giving, making sure the government has an off switch. And so I think that's kind of in the vibe we're seeing, whether that actually results in something passing in Congress anytime soon is a separate question, but at least on paper, the gap is much narrower.

• 50:55Tracy Alloway?

  Is bank regulation the sort of useful analogy for thinking about this? I mean, we require banks to disclose things. We require the government to look at bank balance sheets and figure out whether or not they're actually holding enough regulatory capital against their risk and things like that. We don't expect them to do it voluntarily, certainly not after 2008. Is that the right framing?

• 51:18Speaker 05

  Yeah, no, I think, and I think this, this kind of like shift from voluntary to required is a key step. Because right now, companies have to have like a champion within the company or there needs to be some kind of like reputational, or they want to get feedback from the third-party auditor. Like, there needs to be some kind of reason for them to do it. And not all the companies actually choose to invite external feedback. They will share the bare minimum. And just for example, SpaceX yesterday put out a model card or system card about Grok 4.6. And there were like several sections missing from the table of contents. It seems like they got removed at the last minute. And so there's kind of, is, sorry, you're going to say something?

• 51:57Tracy Alloway?

  I was just going to ask, can you explain the whole model card thing to me?

• 52:00Speaker 05

  Yeah, yeah. And so basically, the thing with model cards is that the original idea several years ago was that a model card was like a nutrition label where it's like a bunch of information summarized succinctly. And you kind of slap it on the AI website and it kind of succinctly explains like, what are the risks? How well does it work? What can it do and so forth? Over time, as people such as myself in industry were like, okay, there's a lot to say, there's a lot to unpack here. And no one kind of established, no one was forcing anyone to do this. So no one established like, this is the, this is the format, you need this little nutrition label. It was just people writing stuff. It, they ballooned into these like dozen-page, 100-page, 200-page, 300-page documents

• 52:45Speaker 05

  of just describing like, here's all the crazy stuff we found, here are all the tests we ran. And there's the spectrum. So like, I would say Anthropic puts out the longest ones. That's not necessarily totally correlated with like quality, but you know, it shows some, some proof of work. And then others will put out five-page, 10-page. And then, you know, what happened yesterday is that SpaceX, for the first time, because of California law, there actually is a requirement to put these out, but there's not really a clear quality bar. And so they can say, well, yes, we did that we followed the California law. We shared information about our testing and the extent to which third parties were involved in testing. And like, basically, it's just like one sentence saying like, we worked with third parties or whatever.

• 53:26Speaker 05

  And so, yeah, and so I think this is different, I would say this is different from say like bank regulation in that, no, I mean, one is, one is that only some things are required right now. It's like kind of putting out a document. There's no, like the third-party test, you're supposed to talk about whether you worked with third parties, but it's different from actually doing it. And so I think what we need is kind of standardization around like, how should the third-party auditing work? What counts as a good system card? What are the minimum safety and security protections that you should be putting in place? And I think that's analogous to some of these like capitalization things you mentioned. And we need kind of standards for like, okay, who, what counts as a good auditor? What counts as, what are the, the standard tests you need to run and so forth?

• 54:08Joe Weisenthal?

  I mean, you yourself are sort of biased in this in that you're building out and auditing a company or an entity, not a company because it's a nonprofit, but an entity that would do auditing. Also, you're promoting this idea that auditing should be important. And again, in the, in the financial realm, you know, there's a few different versions of it. There's sort of like bank supervisors and they, some of them literally sit at the bank and they're there all the time. Then we have the Moody's and the S&Ps of the world. So if you issue debt, you're compelled to get some sort of third-party rating. Why don't you describe in your ideal world?

• 54:44Joe Weisenthal?

  Say this all happens and we, there's required auditing and the companies are cool, et cetera. What is the service that Avery, and I assume in the ideal world, there would be a few others, et cetera, as you can't go auditor shopping, et cetera. What is the service that the Avery's of the world are doing? How embedded and what is the reason then to think for the general public, for all, this is all of out of our hands, that this could lead to safer outcomes?

• 55:14Speaker 05

  Yeah. Essentially, the service that we and others would be providing in this world is similar to what we're currently do, but doing, but kind of scaled up. So right now, what we're doing is kind of voluntary pilot projects that are looking at a specific aspect of safety, security, governance, and so forth. What we would like to see eventually is that there's an ecosystem of auditors that are looking holistically at, is the company following its safety and security practices? Are those safety and security practices reasonable and consistent with the standard for, which ultimately we need, we don't have right now, and providing some kind of feedback to the company? And then there would be kind of like a remediation process for them to resolve issues that that are surfaced during the auditing process. And then there would be a public version

• 55:59Speaker 05

  of this audit report that kind of shares, after doing a lot of like technical testing, reviewing of documents, interviewing with staff, and so forth, that kind of shares this update on some regular schedule, like quarterly or something like that. You might want it to be more like a kind of resident examiner, kind of embedded auditor model, rather than happening once a year, once every six months. And so, but you kind of need to have some kind of like continuous trust building process where maybe the auditor is there all the time, but they occasionally issue these reports. And yeah, what's in it from the company's perspective is they want to, you know, I mean, in the scenario, they would be required. But what's in it for them today is that they want to signal that they are ahead of the

• 56:40Speaker 05

  curve on safety and security, and they want to get feedback from these external experts who have like a kind of fresh perspective. And why does this matter? I think one is you just don't want to be in a world where you have to take the company's word for it, and you want them to kind of, you want there to be common safety and security standards, rather than it just being everyone's kind of making up their own things and then getting it checked. The other is that you want to avoid group think. And so I think even right now, there are a lot of, you know, a lot of what's happening with external testing is like, it's like a research project, like Meter, you had someone from Meter on recently, and they're doing this serious technical research on autonomy and loss of control and so forth. And they work with companies essentially in order to do these kind of

• 57:22Speaker 05

  research, very researchy assessments. And I think that's a key part of the process. But there's also just like verifying that the company did what they're saying they're doing. So there's producing evidence, and then there's also checking evidence. And so it's kind of like in a K1 statement, if I'm, if I'm getting that right, you know, there's kind of this like short order statement. We probably want something more than just a paragraph, but you basically want, you know, a third party saying, we checked that they actually ran all these tests. We made sure that the model that was audited was the same one that's being deployed, et cetera, et cetera.

• 57:51Tracy Alloway?

  What could we actually do to make the testing side safer? Because it seems to me like, I, I can totally believe that we can come up with a reasonable, like auditing structure for models that are being deployed and allowed into the real world in some structured way. But if part of the problem is that we're developing newer and better and more intelligent models, and then testing them, and then they are figuring out ways to get out into the world before, before, then that seems to be like a big vulnerability.

• 58:23Speaker 05

  I think basically what happened is that companies were getting cocky, getting overconfident in the quality of their sandboxes and like, and like, maybe there was a disconnect between some of the people on the safety side who are measuring like, okay, this is where the hacking skills are going, and the people on the security side, you know, building the sandboxes. And like, there was something was getting lost in translation, maybe it's group think. I don't know exactly, but it seemed like at multiple companies, there was this kind of like overconfidence. And so I think these incidents come into light and all the kind of technical investigations are going to hopefully lead to more best practices, more people checking their own biases. But I don't think that's a long-term solution. I think ultimately, people get overconfident all the time. That's a human

• 59:07Speaker 05

  thing. And that's why you want third parties checking to make sure that, okay, are you actually following these, these best practices? You also probably are going to need some technical solutions to some of these things. Like, I mean, maybe some of this testing should be done on kind of air gap servers that are not connected to the internet at all. And, and I think what happened in the hugging face thing is that it went through this like middle layer. There was like a piece of software that it routed out to the real internet through this, this kind of like intermediate thing. But like, I think it might be that eventually we'll get to a point where AI systems are just so capable that they can hack their way out of anything. So you just need to make sure that they're in a cage, basically.

• 59:43Joe Weisenthal?

  Miles Brundage, we could talk for hours about this because there's so many fascinating dimensions of this. I will probably have you back in the future, unfortunately. Fortunately, because that was a great conversation, but unfortunately, it probably won't be the last reason to have to talk to you. Thank you so much for coming on Odd Lots.

• 1:00:00Speaker 05

  Thanks again. I appreciate it.

• 1:00:15Joe Weisenthal?

  Tracy, that was a fun, it's an unsettling thing, the way it behaves.

• 1:00:19Tracy Alloway?

  So many of these AI conversations are still so surreal to me. Like the fact that this is what we're talking about in 2026, it just feels so strange.

• 1:00:28Joe Weisenthal?

  And it's only going to get orders of magnitude weirder because I thought things were weird in 2023, and things are much weirder today. Look, I get why people are very cynical about a lot of this stuff. And I get why people talk about like, oh, there's this regulatory capture. And I certainly believe in the premise of regulatory capture, and there may be some of that. But I will say one thing, like in the sort of like, from the company's perspective is, it is true that for a long time, and for the very beginning, these are not companies making a lot of money, and yet they spend a lot on safety and security. And you could imagine tech companies historically didn't do that. They do. And they have these, like,

• 1:01:09Joe Weisenthal?

  certainly in the case of OpenAI, and slightly to a lesser extent, Anthropic as a PBC, they have these weird corporate structures in part because they seem pretty, they seem to believe that the things that they're building, if built wrong, should not necessarily just be in the hands of like purely profit-seeking enterprises.

• 1:01:28Tracy Alloway?

  Yeah, all very true. I do think one of the interesting things to me that stands out from that conversation is again, the idea of like the asymmetry in power between the approved models that companies can actually use for defense against the new frontier models who are in testing mode and have somehow escaped the sandbox.

• 1:01:51Joe Weisenthal?

  I think there's a really scary dimension. And I think that actually, you think about what is the difference between, say, sort of like auditing versus like a Moody's, et cetera. It seems like you need both, right? It seems like you need to have like the sort of like, yes, this specific model, it satisfies all the requirements that we've deemed it to be safe. But then the sort of like deeper auditing question of like, is this a company that generally experiments and does R and D and testing in what we perceive to be like a responsible manner?

• 1:02:25Tracy Alloway?

  Yeah.

• 1:02:25Joe Weisenthal?

  Which is more like the supervisory aspect.

• 1:02:27Tracy Alloway?

  It seems like you need like a testing auditor, like the bank supervisor who's like actually sitting on the floor, actually sitting in the labs and observing the testing process and making sure that the sandbox is well designed. But again, the problem with that is the classic cybersecurity problem, or security in general problem, which is the model just has to find a single vulnerability, right? You have to like fix all of them, make sure that like thousands and thousands of vulnerabilities are impenetrable.

• 1:02:53Joe Weisenthal?

  And it does make sense, I think, that like, look, this is for-profit capitalist competition. There's no doubt. These are like some of the biggest, most, the pace of growth is extraordinary. Then you lay, we didn't even get into like, how would you do this for like open source models or open source servers? That's a whole other can of worms. But it makes sense that if you're in the lab and you're trying to make money, and you're also worried about like, if you slow down, et cetera, then the other company is going to make more money, et cetera, that one way you solve this, I don't know if it's prisoner's dilemma or whatever.

• 1:03:29Tracy Alloway?

  Race to the bottom, I guess.

• 1:03:30Joe Weisenthal?

  Game theory is, okay, you need this third party to like, you guys go as fast as you want on the R and D side, but we are going to set the rules of like, are you doing that in a safe way? Otherwise, why would Dario and Sam ever trust each other? It's like, no, we swear, we're slow, we're taking it really seriously. We're slowing things down, you know, like, and then secretly they're racing ahead. That is really hard to solve for a series of private, purely private entities.

• 1:03:55Tracy Alloway?

  All right. Shall we leave it there?

• 1:03:57Joe Weisenthal?

  Let's leave it there.

• 1:03:58Tracy Alloway?

  This has been another episode of the Odd Lots Podcast. I'm Tracy Alocke. You can follow me at Tracy Alocke.

• 1:04:03Joe Weisenthal?

  And I'm Joe Weisenthal. You can follow me at TheStalwart. Follow our guest, Miles Brundage. He's at Miles underscore Brundage. Follow our producers, Carmen Rodriguez at Carmen Armin, Dashal Bennett at Dashbot, Kale Brooks at KaleBrooks, and Kevin Lozano at Kevin Lloyd Lozano. And for more Odd Lots content, go to Bloomberg.com/oddlots, where we have a daily newsletter and all of our episodes. And you can chat about all of these topics 24/7 in our Discord, discord.gg/oddlots.

• 1:04:31Tracy Alloway?

  And if you enjoy Odd Lots, if you like it when we talk about moral relativism, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening.

• 1:05:22Speaker 06

  This is Robert Smith from Business History. If you're listening to this, there's a good chance you're a small business owner. And like every small business owner, you started with a dream, to do what you love and watch it grow. What you probably didn't dream about? Keeping up with cyber threats. That's where Mastercard can help, with access to tools that help identify cyber threats to better protect your business. Building your dream business? Priceless. For cybersecurity in a changing world, there's Mastercard. Learn more at mastercard.com/smallbusiness.

• 1:05:53Speaker 08

  If you enjoy Bloomberg podcasts, you'll love listening to Bloomberg 991 FM in the DMV. Hi, this is Joe Matthew. Here in the DC area, the story is always changing. From the White House, Capitol Hill, to K Street, Wall Street, and the Fed, Bloomberg 991 keeps you connected to the news that's driving decisions in the nation's capital and around the globe. Check us out at 991 FM or stream us anytime on the Bloomberg Business app.