AI incidents, audits, and the limits of benchmarks

AI is moving fast from research to real-world deployment, and when things go wrong, the consequences are no longer hypothetical. In this episode, Sean McGregor, co-founder of the AI Verification & Evaluation Research Institute and also the founder of the AI Incident Database, joins Chris and Dan to discuss AI safety, verification, evaluation, and auditing. They explore why benchmarks often fall short, what red-teaming at DEFCON reveals about machine learning risks, and how organizations can better assess and manage AI systems in practice. Featuring: Sean McGregor– LinkedIn https://www.linkedin.com/in/seanbmcgregor/Chris Benson – Website https://chrisbenson.com/, LinkedIn https://www.linkedin.com/in/chrisbenson, Bluesky https://bsky.app/profile/chrisbenson.bsky.social, GitHub https://github.com/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: AI Verification & Evaluation Research Institute https://www.averi.org/AI Incident Database https://incidentdatabase.ai/38th convening of IAAI https://aaai.org/conference/aaai/aaai-26/iaai-26-call/BenchRisk https://www.youtube.com/watch?v=rE17hShOF_QState of Global AI Incident Reporting https://www.youtube.com/watch?v=BOjTPSimNJkUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:03Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:48Speaker 01

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I am joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris? Hey, doing great today, Daniel. How's it going? It's going really good. Lots of fun things in the news to follow and lots of fun things to work on. By the way, for our listeners who might need a reminder about this, we are doing some webinars recently on a variety of topics, some of which are maybe even related to some of the things we're talking about today around security or safety. So if you want to find out about those,

• 1:31Speaker 01

  go to practicalai.fm/webinars. That's where we have some of those things listed out. But I'm really excited today because I have an amazing set of things to talk about with Sean McGregor, who is co-founder and Lead Research Engineer at the AI Verification and Evaluation Research Institute, and also the founder of the AI Incident Database. How are you doing, Sean? Thanks for joining us.

• 1:58Speaker 02

  I'm doing well. Thanks for having me here.

• 2:00Speaker 01

  Yeah, yeah, of course. And this is interesting, kind of to think about AI incidents, verification, evaluation. How did you find your way into a day-to-day where you're thinking about and documenting and studying AI incidents, among other things?

• 2:19Speaker 02

  How did I get to work on what you might call impractical AI?

• 2:24Speaker 01

  Yeah, yeah. Well, I guess, I guess where practical AI turns into problematic AI, let's say.

• 2:34Speaker 02

  Yeah. And really, practical AI is the AI that has consequences and matters in the world. And those are the ones you actually care to look into where it goes wrong. So the kind of really quickie professional tour of things is I'm about a 2017 vintage PhD in machine learning. I focused on reinforcement learning as applied to wildfire suppression policy. So a fire starts in a forest, what do you do about it? And how does that impact the development of the land and the values we get from it over the course of 100 years? Really, in that setting, I had a very strong sense of the power of the technology that we were developing, but also the brittleness of it, and just how difficult it was to even know whether the system was doing what you wanted to do, particularly in a reinforcement

• 3:19Speaker 02

  learning system. And I just never wanted to wander through a forest and have a sense of, you know, this is a charred wasteland because of the forester took my simulation and didn't really realize it was research code and that he should actually apply his own human expertise about what is good fire and bad fire and that. So I went from that. Then I worked on energy-efficient neural network processors, which that was a great effort, the organization called Sentient that had...

• 3:53Speaker 02

  Shipped when I was there, millions and has continued to ship many edge neural network processors. And that gave me a really strong impression of the kind of power and brilliness, once again, inside these consumer electronic devices, hearing aids and the like. And so I left that. I started a company dedicated to the test and evaluation of machine learning systems. And when that happened, or when I started that, the kind of LLM explosion happened. And, you know, that's the kind of Cambrian explosion of sorts of, you know, practical AI technologies that I'm sure we were all still feeling. And

• 4:36Speaker 02

  we sold the assets associated with that to a old safety organization called Underwriters Laboratories and spent some time there doing safety stuff before leaving and starting up the Avery, the AI verification evaluation research institute. And during this whole time, I had a project that just kind of, you know, got away from me because it filled a niche. It's served a need. And that's the Ainsen database that you really need to have systems that collect and produce usable data sets motivating safety practice. And you see this in aviation, a plane crashes, you record what happens and use that to make

• 5:20Speaker 02

  sure effectively in the aviation industry, you have a form of regression test. You don't want that that past crash to happen again. You also have similar things in food safety. You have medical adverse event reporting. These are, this is the fundamental primitive that you need to make sure you have in safety is a bad thing happens, you make sure it doesn't happen again. And so we've, in that project, collected more than 5,000 human annotated reports of AI incidents. Those are collected across more than 1,000 discrete incident records at this point. And we're, we've formed a lot of the training data of like, what is an incident? And we've been interacting with a lot of intergovernmental

• 6:01Speaker 02

  organizations and GRC community, governance, risk and compliance community companies and like, just trying to motivate what is, how to motivate safety and like, why very often that can actually be a business imperative? Because a lot of the incidents that we have in the database,

• 6:18Speaker 03

  You could actually look at the stock price before and after, and there's an impact. Could you talk a little bit more, kind of expand on what the nature of safety means in the context of AI? And because coming in as a neophyte on this, I don't know if there is kind of a standard definition. It seems to me there isn't, at least not that I'm aware of. So how do you define it? You mentioned kind of defining an incident. Could you define what you think an incident is? And how these different concepts relate together in a way that those of us who are not part of that kind of world, that specific world can kind of adopt the ideas and understand what you're talking about? Kind of give us a background on it. So this is a three or a four hour podcast, right?

• 7:03Speaker 01

  It's like, that's like the question, like used to always, we would have the question like, is something AI or machine learning? And like, eventually you just learn that those words are completely meaningless in terms of the way people apply them. So.

• 7:18Speaker 02

  Yeah, we just need to treat them as like a pointer. And, you know, like the memory might be allocated differently with, or the pointer can point to different memory, or maybe the memory swaps.

• 7:29Speaker 01

  Exactly.

• 7:30Speaker 02

  So like, there's a lot of safety terms out there. And like, I've actually had a poster for this at one point, because this comes up so often. You could have incident, accident, adverse event. You have vulnerability, exposure, harm event, controversy, issue. Like, each of these have like slightly different nuance to them. But you can kind of bring it back to the concept of, you don't want a bad thing to happen and that, you don't want that bad thing to produce a harm. You don't want someone to say like, I've, I've been impacted or some organization been impacted. And so the, the intergovernmental definition of incident that developed was effectively

• 8:13Speaker 02

  an event that a harm has taken place. That's an incident. Your mileage may vary in different contexts and like different communities, there, there are different. Terms that resonate, but the reason that we went with incident to begin with is basically it's sufficiently vague while still meaning something because you can't say accident. Some of these have intention. You can't say exposure or compromise or, you know, the ones that are in the computer security world because some of them don't have intention. And so incident covers them all. And, you know, I could gesture and give you some Venn diagrams, but this is a, this is a podcast, so we'll, we'll avoid that.

• 8:55Speaker 01

  I'm wondering, like, in particular, when you narrow it down to AI incident, obviously, sort of AI, depending on your definition of, of AI, kind of has existed for some time. And so I guess like AI incidents have been happening for some time in the sense that like, you could have a computer vision model detecting maybe it's problems in things coming off of a manufacturing line. It misses one and that causes harm, maybe downstream to someone who uses a product or, or something like that. How, how has this maybe, you know, you talked about the shift

• 9:33Speaker 01

  to, of course, the kind of expansion of what, you know, how AI is impacting our, our life. How from your perspective has that, that idea, I guess, been stressed in new ways in recent years?

• 9:48Speaker 02

  We have constant debates over what we would ingest into the incident database. And we, because we have to operate and it's a, you know, database that we have to make these decisions day in, day out, have very difficult decisions. And it's not just, in our case, it's not just what should be considered an incident. It's also if we start ingesting these, are we?

• 10:13Speaker 02

  Lassoing an infinity, that's just gonna like pull us into some extreme direction. And maybe we would like to index catalog and present to the world that that infinity. But there are these kind of like little harms that are repeated, maybe millions of times a day when you have these systems that indexing all of them isn't necessarily useful. So we do have a little bit of a bent towards, is this informing the production of a safer AI or safer world for for AI. But a lot, a lot of things meet the fundamental criteria of involving AI and minor harms have taken place. There, there are also increasing number

• 10:56Speaker 02

  of high, high scaled harms that take place. Like, if you make a billion people slightly more depressed, non-zero number of people probably have died as a result of that. And that's where, you know, it's, it's not just you're building a bridge that needs to bear weight. You're building a bridge that needs to bear the weight of all humanity flowing over it.

• 11:19Speaker 03

  I'm curious, as you're kind of selecting things to go into the database, you know, per the criteria that you just talked about, how are you sourcing that? And especially when you consider the fact that, you know, so many AI things today are proprietary and kept secret in organizations for obvious reasons, you know, how is that sourced? And how do you, how does the different sourcing affect the utility of the database and how you make those kind of selections?

• 11:49Speaker 02

  Most of what we have at the moment is journalistic reporting. The reason that we are living in that space at the moment is journalists actually put in a tremendous amount of work to validate the base facts involved in it. And that, that is where the degradation of the journalistic community, or at least the ability for them to to make wages, has actually been quite harmful. We also do get direct reporting at times. We do get people that that email us or submit direct to our forums. We get people that create blog posts and things like that and submit it to the incident database. We do encourage people to submit to the incident database. The simple matter of the fact, though, is

• 12:32Speaker 02

  The volume that we're dealing with is one where eventually we do need to switch from this kind of voluntary reporting to more mandatory form of reporting. In the EU Code of Practice, there is a requirement for severe incidents to be reported. It hasn't reached implementation as of yet. There's a lot of back and forth on that front, but the insight you can derive from mandatory reporting is greater than that of this voluntary reporting. Like, we can prove existence, we can prove it's happening. It's very difficult for us to assign a rate to incident events, though, particularly as it becomes non-newsworthy through time. So we do have some works that we're trying to make the most of that.

• 13:13Speaker 02

  This bears some resemblance to public health practices, where you don't have perfect insight into disease spread, but you do have these kind of indirect measures of doing tests in the sewers and things to see if the, how much of the viral particles are there and whatnot.

• 13:33Speaker 01

  Well, Sean, I appreciate you taking us through this kind of idea of AI incidents. Obviously, we are practical AI, and in a practical way, we would very much like to prevent incidents, or at least understand kind of the security implications of using certain AI models. I know with the AVRI, the AI Verification and Evaluation Research Institute, in my looking at that, you're talking a lot about kind of third-party auditing of AI, or kind of frontier AI, or foundational AI, however you want to frame that. Do you want to explain a little bit about kind of how that side of things might fit in the landscape and why it's important?

• 14:18Speaker 02

  Certainly. So a fundamental problem that we have, particularly at the frontier model level with things like OpenAI, Anthropic, Google's Gemini, and so forth.

• 14:30Speaker 02

  Is it's very difficult to even know how safe your systems are because they're general purpose systems. This this basically broke the safety frame. All the safety processes that we have are built around starting presumption of there's a specific context it's operating in, and you reason about its safety within that context. Well, if your context is just wildcard, star, everything, where does that leave you? Do you need to verify across all circumstances that it's going to be safe? Because the answer is going to be no. So how do you approach the task of verifying claims? How do you encapsulate something that

• 15:07Speaker 02

  a customer or a person or organization would rely upon and say like, okay, this has gotten this level of assurance applied to it. So I know I can apply it within my college age student population, teaching them how to do the maths. And getting that level of signal at this point, you know, a real practical problem that that you have is no one's going to believe you when you say, you know, this is good for high college age math remedial level. Like, you have to run a pilot program and see how it works before you actually can

• 15:51Speaker 02

  believe the representations being made by companies, because they probably haven't evaluated in your exact circumstances. They they might have something that's analogous to that, but they probably won't have it exactly in your circumstances. And, you know, this is a problem on the practical AI side of things, but it's also a problem on just the increasing power of these systems and the scale in which they operate. You actually do want some really strong top level guarantees that the system is going to try and steer away from catastrophe and doesn't actually have an active propensity for for doing the bad things. And the science

• 16:30Speaker 02

  of establishing that is actually still a work in progress. There's a lot of work, not just in establishing evaluation, but meta evaluation, basically deciding, is this evaluation saying the thing that it says it says or not? And

• 16:49Speaker 02

  Avery, as an organization, is concerned with doing third-party audits of basically saying, is this thing safe? Is this thing doing the thing it should be doing? And there's a premise in there that a third-party is better positioned to do that than a first-party. And we've all been on development teams and pushed to prod, pushed to the real world, pushed to the real-world circumstances and found like, oh, I didn't think about that. Like, one of my favorite incidents in the AI Incident Database is this gentleman who got a traffic citation mailed to him.

• 17:32Speaker 02

  And he looked at the traffic citation and said, like, you know, this isn't my car. It not only isn't a car, it is a woman who's wearing a shirt that says "knitter." And there's a purse strap going across it, so it like, kind of played with the letters. So it looked like "KNI9TER" or something like that. And I don't think the people making the traffic camera were thinking about a woman walking through the field of view of the traffic camera with like, a shirt perturbed in such a way that it would catch it and register it as a license plate and then the traffic citation went out. Like, the world is hard.

• 18:17Speaker 02

  The real world is real hard.

• 18:18Speaker 03

  I'm curious, I like that example. It's a very personal example, and I can certainly imagine that happening to me, actually, bizarre as it is. But I'm curious, as you are looking at applying this process that you're describing to maybe a larger incident, is there an incident that comes to mind in the database that where you could kind of compare what your estimate, like, if, relative to not having gone through third-party process versus if it had, how you might envision that playing out in a different way? I'm kind of trying to get a sense... And you can you can interpret it any way you want or find whatever example.

• 19:03Speaker 03

  But trying to get a sense like when organizations are listening, you know, they have employees and their leadership are listening to the podcast. And they're trying to put this in the context to their own operations. Like, what is a, what do you think is a kind of an aha moment? Uh, potentially for the listener and maybe a Fortune 500, maybe not that big, you know, maybe a mid sized company. Where this process will make a substantial difference to the outcomes involved.

• 19:33Speaker 02

  Probably a good way to work here is by analogy to other segments that have audit. And audit is something that no one truly enjoys. Like you don't, you don't necessarily want to be audited, but you actually do. You do, you do want to be audited if you're an organization that wants to receive investments from other organizations. Having audited financials is table stakes. You, you're not going to put your money into an org that hasn't been audited. Or actually, I should say, there are famous instances in which an org has not been audited. And people subsequently regretted it because the reason they weren't audited is they were, you know, improving dispersments by Slack emojis. And, uh, you know, that it didn't go very well in those, those circumstances for anyone.

• 20:18Speaker 02

  And so we, we do periodically get reminders that, you know, these processes are important and that audits solve a problem. And that's, uh, you need to be able to trust the fundamental representations about the state of an organization. And it's the same thing for the model because the, uh, the model is similar. And in a sense that it's taking actions, it's doing things and has impacts. And so there's value to knowing when you should or should not trust information.

• 20:47Speaker 01

  And I'm looking through some of what the Avery Institute has been involved with. And in particular, kind of recent work of this like meta evaluation of benchmarks, this sort of bench risk, which is actually how, how we got connected through, through a colleague, Ashwarya. And I'm wondering, this is like a meta evaluation. And I know a lot of people make decisions about models based on benchmarks. Could you maybe help us kind of walk through mentally? What is the difference between maybe like an audit?

• 21:26Speaker 01

  And a benchmark and maybe like tying in some things that might make people think about whether benchmarks, as they look at them on leaderboards and that sort of thing, are really, yeah, if they're really relevant to the real world behavior of models.

• 21:46Speaker 02

  So maybe overextending the financial metaphor on things, when you audit a financial organization, you say, okay, I see that this organization has a balance sheet indicating they have $10 million in the bank. You're like, great. Now, as an auditor, I need to go and check the balance with the bank and see, is this money real? Is it actually there? And similarly, an audit is concerned with, I've seen the balance, I've seen the balance sheet, and now I need to check the evidence and make sure that I actually believe that representation in some form. And so where meta evaluation or evaluating the evaluations or checking the benchmarks, evaluating the benchmarks are useful, is you're basically

• 22:31Speaker 02

  checking those receipts. And in the course of the BenchRis project, we found that a lot of the receipts were just kind of like, lol, trust me, bro, like, written on a piece of paper, and that there were real substantive issues of like, okay, maybe there's like an IOU and there's some gold buried out in a field somewhere, but I have some doubts. And at the very least, I need to look into this a little bit more before I rely on it for real world purposes. And the dichotomy that we identified is a lot of the benchmarks that are produced are produced for non-practical purposes. They're produced for knowledge generation

• 23:14Speaker 02

  purposes. They're produced for research purposes, where people are wanting to understand systems that are making sense of it, but it's not produced with the intent that someone's going to then say, all right, I'm going to deploy this in my environment, and I know now that it's unbiased because it scored well on BBQ. And BBQ is a great benchmark. It was really foundational in the bias benchmarking community, but it's also

• 23:43Speaker 02

  Not produced with the intention that every time a frontier model is released, you say like, "Look, we've improved on bias and it's 10 BBQ points better than the than the prior ones." And like, that's a good thing. You do want that, but it's not a practical AI purpose. It's not saying like, "It's unbiased for my specific application" because there's a distribution associated with BBQ. There's characteristics that aren't necessarily generalizable to the environment to which you're deploying it. The prompts can be in like a very particular space for BBQ that you're just not operating in. And all these things are associated with, in the, in this research work that we put out,

• 24:25Speaker 02

  they're associated with failure modes that we identified. We collected a list of failure modes and then we looked at how well benchmarks had expressed mitigations against those failure modes. And the results were mixed, like some did better than others, but by and large, most benchmarks historically have been produced for research purposes, not for practical AI purposes. And this is a problem because this is what we're working with and informationally. And this is why we're starting to see a evaluation ecosystem pop up and that being a discrete task and something that's supported by organizations separately.

• 25:04Speaker 03

  I'm wondering, as we're kind of going through this and discussing the process, if as you're, as you're looking at some of the different risk categories that you guys, you know, outline on your website and stuff, and you know, a couple of examples are misuse and unintended behavior and InfoSec and emergent social effects. Which of these do you think is, is kind of like, maybe the most underestimated? You know, the thing that people aren't really expecting that has consequence, that may be outsized relative to what those of us not in this field would expect? Do you have any sense of that in terms of like this kind of a little bit of the surprise that people might not realize is there?

• 25:47Speaker 02

  That's tough because I'm a little bit of the fish in water being told it's, it's wet and

• 25:54Speaker 03

  Fair enough.

• 25:54Speaker 02

  I will say in like, kind of re-rinding my own experience on this is, I feel like I'm a pretty astute observer and I've been more predictive of what problems are coming around the corner than I think most people I've encountered. But even with that, I still am regularly surprised. And I wish we could be a little bit more predictive about it. And I think we are, we are going to get there and we're going to learn from, from the past as, and hopefully push that into the future. But it's, this is a new thing. Like, we don't have, we don't have like a great operating history, particularly for the general purpose AI systems. And I think that the split in this community that you should probably think a little bit

• 26:35Speaker 02

  about, about that question in terms of expectation is there's security and safety people and that there is a little bit of difference between those. Security is safety, safety is security, but what you care about tends to be slightly different than what you're expecting. And security people have a great, much greater capacity for the presumption of bad actors doing bad things and that producing bad outcomes. And then the safety people have a much stronger presumption of just the world is its own adversary. We're all in our own kind of game against nature and bad things will happen regardless. And maybe in the bad actor side of things, they're a little bit more insistent upon it, though, particularly when the scalability of doing bad things is, is actually substantially

• 27:20Speaker 02

  increasing. So we have to solve all of these.

• 27:23Speaker 01

  Well, Sean, one of the things I wanted to ask you about, well, in particular, I, I'm a big fan of the Darknet Diaries podcast and listen to that a lot and, and always loved, you know, hearing stories of, of hackers and, and, you know, the kind of world that is there and all of, you know, what, what people try to do at DEF CON and, and other places. I was really intrigued by your, your study kind of titled To Era's AI. I see in the description, you kind of took some, some models to DEF CON and I'm reading your, your description, had the pleasure of taunting a room of hackers with a declaration

• 28:03Speaker 01

  that the model is flawless and, you know, obviously certain things happened after that. So I would love to hear, I mean, maybe how this came about or what the idea really was. And then would love to, you know, hear some of that story.

• 28:19Speaker 02

  Sure. So to set the stage a little bit, DEF CON is a hacker convention conference that's been run in Las Vegas for a good number of years. And it's, I don't know if listeners have seen the 1990s movie Hackers, which has like a very early, like Angelie Jolie and some others in it. And it's like these young men and women like rollerblading everywhere. You know, like that's just how the future would be and talking about how risk architectures would change everything. And there's this kind of hacker aesthetic that that kind of noisily was portraying of, you know, joyfully compromising systems. And in this particular

• 29:03Speaker 02

  instance, having a mixture of people that are doing that with good aims and bad aims. But DEF CON is this community organized thing and you have villages. And one of them is the AI village that's been run for a few years now. And in that village, you have presentations, but there's also a section of it dedicated to some sort of a challenge that's conducted over the course of the conference. And people filter through and figure out how to break things and they're treated to cash prizes in this case. So this was something called the Generative Red Team 2. So it was a change from the prior

• 29:48Speaker 02

  year in form. And we basically asked people, there's documentation for the open language model produced by the Allen Institute for AI. So a 7 billion parameter large language model, along with its guard model put in front of it. And we said, here's the documentation. Here's the representations about what this model is supposed to do. And your job is to show how that's wrong. That unleashed a few days of chaos and I don't know what else was happening in the conference, but I was at the judging table for the entire time with reports flying in and needing to decide, is this actually a violation of the documentation? Did the thing we wrote, which we did put some Easter eggs in there of like,

• 30:31Speaker 02

  we expected people to be able to break it. But did they submit sufficient evidence of a violation of that documentation? And that is actually a surprisingly difficult thing. We had all these people that were very good at the compromising systems and saying like, yeah, I know how to prompt this LLM and get it to tell me how to burn down this convention center, violation of what the model should do. And we kept on needing to say, oh, that's great, but you haven't told us anything. Like anecdote does not equal data in this instance. And we need you to show that it's systematically pushing towards arson,

• 31:13Speaker 02

  that it's always to burn things down, or that you can put some sort of more universal jailbreak on it that makes it so that it underperforms in that particular filter of things. And so what we basically had to bring to the security world, which the security world does not like this, and I don't blame them for it, is statistics. We had to say, it's not just about the fact that there's an exploit. It needs to be systematically vulnerable in some form because it's always rolling dice. That's just the nature of it. And that made for a wild few days, paid out a good cash purse, and learned quite a lot about the collision of disciplines that we need to foster

• 31:57Speaker 02

  when it comes to security, safety.

• 32:01Speaker 02

  Statistics, machine learning, the world's getting more complicated.

• 32:06Speaker 03

  Prior to kind of talking about some of the chaos that ensued at that point, as you talk about that collision of culture, if you will, you know, with kind of, kind of taking the hacker out of their thing of thing, "Hey, I have an exploit, you know, look at me," and requiring that level of rigor. First of all, can you describe why you required that? Why that level of rigor was important in this case? And how did that force different behaviors out of the hacker community that were applying themselves against the problem that you had presented?

• 32:43Speaker 02

  We had a few people that were unhappy initially, and then we explained it, and then they went back and came back. And then they started turning the crank and getting a lot of payouts. The people that really wanted to encamp and figure out the modality did very well. The problem and the reason why anecdote doesn't equal data here is if you say something is, you know, 99%, filters out 99% of the bad thing. If they wanted to, they could roll up to one of those stations and, you know, just keep on issuing the same query or like adding one period to it, dot, dot, dot, dot. And then one time out of 100, they'll get something, they'll be able to walk up and say, "Money, please." The problem is that's not really useful. If you're designing the system, you need some idea of what is

• 33:26Speaker 02

  the systemization here, because you, you know it's 99%. You don't, to some extent, like you're going to keep on working to get to 100, but you, you care about the cases where it's not 99%, but you made a mistake and it's actually 70% or it's 1% or 0%. And that's a statistical argument. That's a, here's a description, a higher level description, then here's an example. Like, this isn't me prompting it. This is a description of a attack that is not accounted for in your documentation, that you're vulnerable for, and you need to solve this as a strategy. Like, you, you never solve "talk like a pirate." So, anyone can talk like a pirate to the system and get it to do bad things. And that's

• 34:11Speaker 02

  the thing you care about, rather than, "I asked it and piratee's something 100 times and one time it gave me bad."

• 34:18Speaker 03

  Were there certain modalities like you described of failure that were a surprise kind of to, to the judges in terms of like the, or maybe to like the...

• 34:30Speaker 01

  Like, I don't know how much interaction there was with the model building team, you know, at Allen Institute and that sort of thing, but, you know, any, any kind of modes of, of failure that, that, that weren't expected, or just the ones that were continuously problematic, but even if they were known before?

• 34:47Speaker 02

  Somewhat unexpected. We characterize this in the research paper as well, and I, do recommend people take a look at it, because it's, it's very practical in nature. The biggest one, and the one that I, I think the most people just incidentally exploited, and they didn't, they didn't have full information on this one, like the, we didn't give them full information on the integration of the guard model and the foundation model or the chat model underneath. And there's multiple ways of configuring that handoff. You can either do like a hard reject and say like, this doesn't pass the guard model and we're gonna give you nothing, or you can re-prompt the underlying model and say like, but don't, here's, here's what you're being prompted, but don't answer it, or, you know, there's

• 35:30Speaker 02

  variations of that. And then you can still get like a somewhat useful reply out of the system, it just should reject it. The configuration of that handoff was looser than it necessarily needed to be. And exploiting that handoff between the guard model and the underlying model was the kind of, you know, big vector that you could use that to print cash over the course of the competition. And that's the, perhaps, the thing people need to think about, you know, particularly as deploying these systems are very often a collection of multiple systems that present as one, is the interface between them is very often under-tested. It's very often difficult to know how those systems will interact. And

• 36:16Speaker 02

  particularly when the benchmarks are expressed at a lower level than the, than the whole system. You don't know a lot about what will happen at that point. You've muddied the waters a little bit.

• 36:24Speaker 03

  I'm curious, as you went through this exercise with the attendees, beyond just the specifics of this problem, what was the value of going through the exercise? And what, you know, in terms of the outcome for you and what you got out of it beyond the immediate guard model with the, you know, with the model it's protecting in that process? Like, what was that like? And I'm also curious, if you go and do a similar exercise in the future, which I'm assuming there will be some, some variation of that, what can you envision being a good next step to tackle in terms of trying to secure value and get to a next level outcome?

• 37:09Speaker 03

  You know, have you thought about what you would do as you came out of the previous event? Should you do it again?

• 37:15Speaker 02

  So I, I think one of the really strong takeaways, and there, there's actually been some, some work since then, was the need for tools and to kind of scaffold this in some form and to make it where you have a clear form of expression of what these things are called, which is flaw reports. Because it's, it's not an incident, like no one's harmed. You're kind of, you know, testing things in a laboratory. And so what you have in classical computer security is you have bug bounty systems. You have the ability to submit those. There's kind of clear adjudication processes that often lead to payouts to people. And the extensions that are necessary for machine learning based systems, data-driven systems,

• 38:00Speaker 02

  distributional systems is, I think, clearer as a result of the activity that we ran at DEF CON. Like, that, that's the kind of research value that's, that's produced by that. Alan Stute probably got some operational use as well that they, they learned a lot because they're at that judge table, dealing with the onslaught as well. That was the vendor table for things. And so I, I think if this, if this is run again in, in the future,

• 38:31Speaker 02

  Getting past those pain points of, you know, you're kind of rolling your own code and all that, and that's not necessary for this scaffolding it a lot more will carry it a lot more distance and then move this towards something where there's one or more companies that are operating a flaw reporting business in the same way that there is a security bug reporting.

• 38:57Speaker 01

  Well, Sean, as we kind of get close to the wrapping up point here on the podcast, I'd love for you to share anything that comes to mind in terms of like, as you look out kind of at the next phase of maybe your own work, but maybe, you know, the ecosystem a little bit more broad in terms of the area of research in which you are involved. What are you, what are those things that as you, you know, in your days or you're driving back to home or whatever, what are those things that stick in your mind? Like you're excited, like what if this happens? Or I'm really interested to see how this develops, or I'm encouraged by kind of this going into the future. What's kind of top of mind for you in terms of the

• 39:40Speaker 01

  community that you're involved with coming into this next year?

• 39:44Speaker 02

  There's an interesting phenomenon that every time we pass a milestone in the incident database, and we're like, oh, we've crossed a 500 or 1,000 incidents. We're like, yay. And then we're like, but it's not a, it's not a great thing.

• 39:56Speaker 01

  Should you celebrate?

• 39:57Speaker 02

  Yeah. Should we celebrate? And I feel like my, my dream utopia might be like, I need to go do something else because the safety problem is, has been solved and there's no more, there's no more to do. Like the unfortunate fact though, is I, this is only going to get more complicated and there's more to do, there's more to solve. There's not, there's not going to be a perfect AI system out there. So, you know, what I'm hoping for in the next year is to develop a greater sense of

• 40:37Speaker 02

  what institutions, what techniques, what methods we need to develop and scale and apply so that we can have a safer AI ecosystem. And this is important, not just from the perspective of wanting to prevent harm, but, you know, you can't deploy unsafe systems to clients that want safety, that care about outcomes. And so our ability to make a safer system is very heavily involved in our ability to ship product to solve problems in the real world. And so I hope to highlight that.

• 41:22Speaker 02

  I hope to have a greater accounting for the risks and make it so that we have better and better measures of those risks because you, you manage what you measure. And so let's measure these risks and then we can separate the actors that are investing in safety and, and stronger AI systems, safer AI systems from the ones that aren't doing that. And right now that's, that's not where we are.

• 41:50Speaker 01

  Well, really appreciate that perspective and appreciate the work that you continue to be engaged in. Sean, thank you for that. And from the community, it's really, really important. So appreciate that and thank you very much for taking time to chat with us about it. It's been great.

• 42:05Speaker 02

  It's been a lot of fun. Thank you for having me.

• 42:15Speaker 00

  All right. That's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments and we would love for you to join the conversation. Thanks to our partner, Prediction Guard, for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.