The Future of AI Security with Adam Wenchel, CEO of Arthur.ai

In this episode, Nathan sits down with Adam Wenchel, CEO of Arthur.ai. Adam founded the AI security company back in 2019, before GPT-2 existed. In this episode, Adam shares his unique perspective on the AI security landscape, drawing from years building commercial AI systems. They discuss the attacks Adam set out to defend against, the changing priorities of executives in the rush to adopt LLMs, and the LLM-specific techniques Adam has developed. If you're looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. TIMESTAMPS: (00:00:00) Episode Preview (00:03:45) Adam's background in AI and starting Arthur AI in 2019 (00:05:52) The release of ChatGPT as a watershed moment for generative AI (00:07:09) Differences between traditional cybersecurity and AI security (00:09:51) Early examples of AI security issues like boundary detection attacks in fraud systems (00:12:39) - Mitigating risks of AI systems through observability and robust training (00:14:40) - Financial services governance of AI models and its challenges today (00:15:12) Sponsors: Netsuite | Omneky (00:21:18) - Motivations for governance like staying compliant with regulations (00:21:40) - The mix of incentives shaping earlier AI governance, like explainability (00:28:14) - Using LMs to evaluate the security of other LMs (00:30:03) - Dynamics between training and evaluating future LMs (00:38:10) - The state of reasoning capabilities in large LMs (00:44:35) - Corporate urgency around adopting generative AI technologies (00:46:51) - Common enterprise use cases for generative AI and security concerns (00:50:45) - Techniques for reducing hallucinations in retrieval augmented LMs (00:53:15) - Benchmarking LMs on specific organizational tasks versus generic benchmarks (00:56:30) - Metrics beyond accuracy like concision and hedging (01:01:20) - Automatically detecting anomalies and hallucinations (01:09:20) - Relationships between Arthur AI and foundation model providers (01:11:52) - Where Cohere shines: multilingualism and not hedging (01:13:43) - Anticipating future watershed moments and steady progress (01:19:03) - Whether we can ever fully solve AI alignment and safety LINKS: Arthur.ai: https://www.arthur.ai/ X/Social: @apwenchel (Adam) @itsArthurAI (Arthur.ai) @labenz (Nathan) @eriktorenberg @CogRev_Podcast SPONSORS: Shopify | NetSuite | Omneky Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Music Credit: Stableaudio.com

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  It's a pretty manual review process that can take months. If there's a problem, like someone's exploiting a weakness in the model, oftentimes the easiest thing to do is to put in like a rule up in front of the model because you can do that in a couple of days whereas it might take you literally six, eight, you know, 12 months to get a new model. You know, metrics like helpfulness or readability or concision, how often does a model hedge? How often does it hallucinate? These are the kinds of metrics that I think you need to start to really think about to build a system that's most helpful to the people using it and that provides the most value in your organization.

• 0:35Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Adam Wenchel, CEO of Arthur AI, a leading provider of AI security solutions that says simply, "We make AI better for everyone." Now, if you listen to this show, you know that companies of all sizes are racing to implement LLMs

• 1:18Nathan Labenz?

  for their revolutionary speed and efficiency. But of course, they're also worried about the risks stemming from their unpredictable behavior. And this is where Arthur comes in. Their tools, including Arthur Shield, which the company describes as the first firewall for LLMs, and also Arthur Bench, which they describe as the most robust way to evaluate LLMs, help their enterprise customers in such high-stakes, compliance-centric sectors as finance, healthcare and computer security to monitor LLMs in production, to detect problems and to prevent harmful outcomes.

• 1:54Nathan Labenz?

  In our conversation, Adam, who started Arthur as an AI security company in 2018, before GPT-2, shares his unique perspective on the AI security landscape, drawing on years of experience building commercial AI systems. He describes the sorts of attacks he originally set out to detect and defend against, explains how priorities have changed for boards and executives with the surge in LLM adoption, and outlines the techniques that Arthur has developed specifically for LLMs, including using one LLM to evaluate another in context. Along the way, we touch on benchmarking, performance metrics, standards for responsible use, and the future of AI governance. Adam believes that effective

• 2:38Nathan Labenz?

  security systems will accelerate beneficial applications of AI, and his insights are directly relevant for any organization implementing AI today. As always, if you're enjoying the show, we'd love a review on Apple Podcasts or Spotify, or simply a share on social media. This is the best way to help others find the show. Now, for an authoritative overview of the nascent field of LLM security, I hope you enjoy this conversation with Adam Wenshaw, CEO of Arthur AI. Adam Wenshaw, welcome to the Cognitive Revolution. Hey, thanks for having me on, Nathan. My pleasure. So I'm excited about this conversation. You are the CEO of Arthur AI,

• 3:19Nathan Labenz?

  which is a security company, and increasingly also kind of a performance management company, I think, and we're going to get into all that. But as a way to sort of contextualize just how fast AI is moving, and how even folks who have demonstrated foresight like yourself are kind of sometimes reacting to developments as they come at you, I'd love to hear a little bit of the background of the company. I understand that you started it in 2019. Obviously, that's a pretty different era of AI technology versus the one that we're in today. So maybe for starters, you know, give us a little bit of your background, especially what you saw at that time that motivated you to start a company?

• 3:59Nathan Labenz?

  And then a little bit about kind of how AI has surprised you and how you've reacted to that in the time since.

• 4:05Speaker 00

  Really started working in AI as an undergrad in the late 90s when I followed one of my college professors over to DARPA and worked on some early app projects there and talked about kind of looking different. There's been a lot of changes in the intervening, you know, 20, 20 something years and a lot of different eras of AI contained in that, in that two decades. But in 2019, you know, where I was coming from, my previous company, Annex, had been acquired by Capital One in 2015. And I spent three years at Capital One where shortly after I joined, I was asked to start their AI team and scale it up. And so even though I had had a lot of AI experience previously, you know, at a large enterprise that's already at scale,

• 4:46Speaker 00

  you're deploying models in ways that affect both significant parts of business revenue as well as impacting a bunch of people's financial livelihoods, right, a bunch of their customers. And so it got me hyper focused on these issues around, you know, when you put these models in production, how do you know they're making good decisions that they're serving both the company that's putting them out, but also the people that they're impacting? And so that was really where the vision came from. So I left Capital One and, you know, started talking to a few other people, including my co-founder John, about this. And turns out a lot of people were sort of, as AI was really starting to expand and take off

• 5:24Speaker 00

  in the late 20 teens because of better leveraging of GPUs and cloud computing and algorithmic improvements and all the things that kind of fed that. You know, a lot of people were starting to put these things into production at scale for the first time and running into these issues. So that's where we started. The LLM excitement in the last year was definitely something that we, so we knew, like, we've obviously been tracking generative AI for a long time and have been playing around with it. I think what caught me off guard, though, was just what a watershed moment, the release of ChatGPT. Would be and how that, you know, really just captured people's imagination in just a really profound way and continues to do so.

• 6:03Speaker 00

  And that has fed, you know, what we've seen now is we hear basically the thing we hear from all of our customers is this generative AI has now become the the top issue for their board of directors on down, right? And so everyone from the board, board members to the C suite want to know what's what's our generative AI strategy? Because everyone, you know, you play around with chat GPT and it's it's it's not hard to see the transformative power of it and like how it could really. Uh, the mind starts racing with ways that it can kind of transform the way we do business and all sorts of parts of our lives. And so urgency with which this technology is being adopted uh is unlike anything I've seen in in my career, uh, which has been pretty exciting to be a part of.

• 6:44Nathan Labenz?

  Yeah, it's been wild. I, uh, you know, share that, uh, that kind of surprise at how big of a deal chat GPT was relative to things that were already available at that time. But, you know, maybe just hadn't been presented in in quite the right way. Just to set a little bit more foundation for folks. I mean, I think people, you know, of course, have heard this term cybersecurity for a long time. And I think some people might be wondering like, what makes security and, you know, these kind of issues specifically related to AI systems different and, you know, somewhat in some ways harder to manage than kind of traditional cybersecurity?

• 7:25Nathan Labenz?

  I'd love to hear your thoughts on what makes this a different field than traditional cybersecurity?

• 7:33Speaker 00

  You know, you're really moving from kind of a deterministic world into a probabilistic world, and that has pretty significant implications uh, in terms of the way you think about things, uh, and attack vectors and protecting these these systems. Um, you know, it's not like, well, making sure your your source code is correct because, you know, these models are are trained in probabilistic ways and exhibit behaviors that, uh, you know, weren't weren't explicitly coded into them. And so, you know, you really have to to change the way you think from that kind of deterministic mindset to a probabilistic mindset.

• 8:05Nathan Labenz?

  Going back to kind of the Capital One era and and the era of models that you started the company focused on. I'm imagining kind of things like fraud detection, for example, where in a traditional system, you might have a bunch of rules, a bunch of heuristics explicitly coded up. If this and that and this other condition are met, you know, then we should flag something. But if only two of those three are met, you know, then we don't flag something. You know, kind of the key point there with your, your comment about determinism versus probabilistic is like, if there's a mistake in that, you know, kind of classical system, in theory, you know, of course, we are prone to make mistakes, but in theory, we can always

• 8:49Nathan Labenz?

  kind of go back and be, ah, here's the mistake. This is where, you know, we forgot some logical condition or something was like not as it should have been. Now we can correct that. Now we can be confident that that problem is solved going forward. Great. In contrast, you've got, you know, the classic black box in the AI world and you're like, well, geez, why, we don't even quite know why this thing is putting, you know, this one into the fraud bucket and this one into the non-fraud bucket. That stuff sounds really, you know, challenging from a number of different levels. Maybe you could expand a little bit more on just kind of even what sorts of problems? I know especially in credit, obviously a highly regulated industry, lots of concerns about discrimination and fairness there.

• 9:33Nathan Labenz?

  Give us maybe just a little bit more of kind of the sort of problems that people were already using AI at scale on when you got started, problems that they were seeing in terms of reliability or, you know, ability to kind of account for what they were doing and then how you tackled that. And then we'll kind of bring it more forward into the present too.

• 9:50Speaker 00

  Fraud systems, like you mentioned, that's definitely one of the, the early real world examples of kind of AI security in action. And so fundamentally, what happens there most frequently is what's called a boundary detection attack where the attacker, like as you said, they're kind of probing the behavior of your fraud model, trying to figure out like. How to how to game it, right? And so an example of that would be maybe I can go to a high-end department store and buy luxury purses and sell them on the street like relatively easy thing to monetize. As long as my transactions are below $500, I it won't get flagged, right? And so there that's actually something that happened at a large bank where their their algorithm sort of said.

• 10:33Speaker 00

  Someone goes to a high-end department store and tries to buy a bunch of luxury goods, like the the model kind of learned. And with a combination of, you know, typical combination of rules and trained behavior that, you know, if it's above $500, that that that should be flagged as fraud. And so people figure that out and they they learn to just like spread it, spread out the purchases across a couple of cards or and and, you know, just keep it below that threshold. And we're able to exploit that for for a while until it was realized, you know, till someone noticed that the fraud losses had increased. And that they needed to really plug that hole and modify the system. Parts of the this is the longer story about kind of the challenges of fraud.

• 11:13Speaker 00

  But a lot of these systems that are processing credit cards are are very old and antiquated and don't have a lot of processing power. And so they can't always put in really sophisticated like fraud checking is not as high as as you might think. You know, that's an early, an early example, the you know, another kind of attack is called a poisoning attack. And so I we did not see a whole lot of that in the wild. I've not heard about it a lot, but in fraud, I've heard about it in other contexts, particular sort of like some nation state cybersecurity stuff. But a poisoning attack would be where, for example, if I could go, I could go buy a $1,000 purse at a high-end department store. If you get tagged for fraud, if enough people did that and called in and said, hey, this is not fraudulent.

• 11:54Speaker 00

  Eventually, the model would, you know, might learn to think that that's a legitimate transaction and then you could exploit it. And so what you're doing is you're sort of you're adding in training data to the training set that causes the model to learn something that you want it to learn. But that the people running it may not want it to learn. And so, you know, these are these are real, real scenarios. I think that there's a lot of ways to to address this. One of the first thing, you know, that that we that. We focus on is just observability, right? Like so in the old days, people deploy these models and they wouldn't really have like real time kind of observability said if you notice a spike in these $499 luxury good kind of purchases, you you would spot it right away, right? Like before it sort of hits your balance sheet, like you would you would detect it.

• 12:39Speaker 00

  And so that's a big part of what our observability tools is looks for trends like that, hotspots in the in the data. And and can alert off of that. But the other thing is you can also work at even during training time, you can work and develop training routines and objective functions that kind of make your models more robust and that are kind of less quirky, if you will. And so less vulnerable to these these kind of boundary detection attacks. And that's something our research team is definitely works on a lot is helping people train more robust models that are less prone or less susceptible to to these kind of attacks.

• 13:16Nathan Labenz?

  Yeah, it's interesting. When you talk about a boundary attack in the first place. I mean, that sort of suggests that there, especially if it's like at such a round number as 500, that suggests that people are using some combination of heuristics and models. And so people are really finding the weaknesses, not even so much in the models in that case, right? But in the accompanying heuristics, is that right?

• 13:40Speaker 00

  So there's, there's a lot of regulation around the use of models, the Fed and the OCC, you know, there's SR 11 dash seven, which governs model governance. And basically, in order to deploy a new model into production, it's a pretty manual review process that can take months. And so if there's a problem, like someone's exploiting a weakness in the model, oftentimes the easiest thing to do is to put in like a rule up in front of the model that way, because you can do that in in a couple of days, whereas it might take you literally six, eight, you know, 12 months to get a new model into production with with a way. A lot of the large financial institutions have implemented their model risk process. What happens then is you have these these kind of heuristics, like you said, they build up too.

• 14:25Speaker 00

  And so you end up in some cases with like hundreds of these heuristics combined with a model that may be three, four, five years old that the people who worked on it maybe aren't with the bank anymore. And so it's a it's a little bit of a challenging situation. So, you know, I think one of the another area that we focus on is is helping people automate more of their model governance process in financial services so that they don't have this situation where like, oh, if we want to put out a new model, there's like a six month kind of process. But we have the safeguards in place so that we can, you know, put things in production in like a champion challenger mode and promote them in a way that better mitigates risk and make sure that they're doing in a responsible way, but allows them to move a lot faster.

• 15:09Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, that's really interesting. And I know we have a lot more to get to in terms of, you know, all the new stuff that you've built over the last couple of years as the LLM paradigm has become so prominent. But maybe just one or two more questions on that point because I think it is really interesting. I've had a longstanding interest in AI, but in terms of really spending all my time thinking about it, you know, that's been just the last couple of years. One of the things that is kind of cool about it is, you know, because all the latest and greatest stuff has been developed pretty quickly, like you can kind of catch up pretty quickly, you do miss some of that intellectual history. So now we're having this, you know, big kind of society wide debate around like, well,

• 15:52Nathan Labenz?

  what should a process be for deciding, you know, whether or not a model is something that can be deployed and who should decide and, you know, what should the standards be? So you're kind of saying, you know, something pretty interesting here that I think a lot of people have not heard much about, which is that this has happened once before and a whole bunch of standards were kind of developed and you're alluding to some downsides of those in terms of it's a slow process to update. I'd love to hear a little bit more about kind of the intellectual and kind of governance history of. What standards, you know, what, what sort of protocols were developed to govern this earlier class of AI systems? And then, you know, we can maybe speculate if there's anything

• 16:37Nathan Labenz?

  we can bring forward from that to the present day, too.

• 16:40Speaker 00

  A lot of that governance is somewhat vertical specific. And so, and it's in particular, like within government and financial services, within healthcare, there are sort of different paradigms. But financial services is probably the most robust. And so there, you know, you had, like I mentioned, SR11 dash seven, which I think came out roughly, I want to say 13 or 14 years ago. And there, and there are predecessors to that as well. But at a time when, you know, most people were doing kind of relatively simple linear regression models that were being updated very infrequently, and the rate of change in, in the world and in that kind of data processing was, was very small. You know, they'd put these models out, and they might update them every three or four years. But it wasn't like, wasn't,

• 17:23Speaker 00

  it wasn't a huge deal. At the time, you know, it was a very manual process, or is still in many cases, right? So when this legislation came out, most banks set up model risk offices where they have a small team of data scientists there who, for, they have different kind of, they classified models into materiality. So is it like a high materiality model, medium, low? How important is it to like the function of the business and the impact it has on the consumer? And for the high material, materiality models, you know, maybe once a quarter, a data scientist on the model risk team will download the sort of inferencing history from the last quarter, spend a few days just going through it, validating it, making sure

• 18:07Speaker 00

  that there's no red flags that, that jump out. And then, you know, ideally give it a thumbs up at the end of that, that period. Or if there's a change that needs to be made, they'll, they'll implement that change. And so it's a very kind of, you know, process that probably 15 years ago was very reasonable. But in this sort of like modern age of online reinforcement, learning and transformers and complex behaviors. And everything's automated and automated refit and redeploy is that all these things that uh that people are taking advantage of, you know, it it it's a little bit antiquated and not super effective. And so, you know, that is, as I mentioned, like, that's a big part of what we help customers do is bring the automation to that. Right. And so that you have the same sort of level of uh always on proactive monitoring.

• 18:51Speaker 00

  So that if something starts to get unusual with with your model's behavior, either you know, something that's like definitely bad or just sort of anomalous, it gets flagged immediately and an alert gets fired. And then you can kind of investigate it the same way you would with like an application or cybersecurity. But that level of operational maturity in data science, like they never really thought of like, you know, traditionally, they never really thought of operational maturity in the same way like cybersecurity people did. Where you had like a security operations center that was staffed 24/7 and people were responding to incidents in in according to an SLA. Um, a lot of times with data scientists, it was much more kind of loosey goosey like you would. You would, the person who developed the model would sort of continue to babysit it.

• 19:35Speaker 00

  But then a lot of times there was like cracks in that because the person who developed it may get assigned to a new project or leave the company. And then these models still keep chugging along, making decisions. And there really wasn't a framework for for making sure that like if they started to make bad decisions. Um, someone, you know, there, there was like kind of a procedure for handling it and improving it, making sure it was uh back on the rails.

• 19:58Nathan Labenz?

  At the core, what do you think is kind of the motivation set for this, you know, kind of earlier class of models? This is something again, that's hotly debated and then I promise we'll go to the LLM era. Is it just? Yeah, I mean, obviously, at some level, it's protect ourselves from fraud. But then there's other motivations to probably like stay on the right side of fairness and discrimination laws. Or maybe, you know, don't expose ourselves to liability. What would you say is kind of the mix of incentives that has shaped that earlier effort?

• 20:34Speaker 00

  Yeah, so I think you definitely touched on some of them, being able to kind of explain what the models were doing with an explainability. You know, back in the day, it was sort of partial dependency plots and relatively like simplistic notions of explanation. Obviously, in more recent years, there's been a lot more work with things like Lime and Chap and more, you know, kind of newer forms of explainability. But I'll tell you, there's really, one of the really interesting kind of incentives or levers there, like you think about a business, right, like a large enterprise, as it becomes more data science and more AI native, you have models all over the business that are that are actually running key leverage points.

• 21:14Speaker 00

  And it turns out that those key leverage points all have affect each other, right? And so if I make a change to a fraud model, it can have like, it can do like spike call center volumes, right? So like if I start rejecting a bunch of transactions, you know, my call center volumes, which at any consumer bank is by far the biggest cost center, will spike, and that's a huge problem. Another one is in the US, most consumers, there's this concept of front of wallet. And so Americans tend to use the same card over and over again. I can make a more accurate fraud model if I increase the number of false positives. And in the process where people like inadvertently get tagged for fraud when it's not.

• 21:56Speaker 00

  What happens if I get tagged for a couple of times fraud, I put that, I put that credit card in the back of my wallet and I start using a different one all the time. And that costs the company money, right? And so in some cases, it may not be as simple as just maximizing accuracy. It may be like, all right, we're willing to like let a few more fraudulent transactions go through because we need to stay front of wallet in with our consumers, with our customers, right? And so it's really interesting how, because if I, you know, I can, I can decrease fraud boss, but then I've reduced, you know, revenue in a different part of the business. And that's, that's problematic. And so I think one of the things that we, we like to, that we give people visibility into with our platform is this, the system of models that's running your business, right? Like, how do you, how do you think about that interaction? Right now, it's like.

• 22:40Speaker 00

  Managed by tribal knowledge and that can be fragile, right? It's very, it's not a very robust way to manage that kind of interaction of, you know, your business, which is fundamentally maybe 100 different models, controlling different sort of aspects of your business that are all really important and that all have sort of can have unanticipated effects on, on each other.

• 23:00Nathan Labenz?

  Yeah, okay, that, that's all really, really interesting and so many echoes of, of those challenges in the current set of challenges as well. Bringing us up to closer to the future, obviously, the big thing that has changed is the models have become much more general, much bigger, you know, much more data, much more of a black box and much, much more surface area, right? Just so many different ways that they can be used and, you know, they can respond to kind of arbitrary inputs so that, you know, boy, does that open up a whole incredible range of things that people might do adversarially. So I'd love to just kind of hear how you think about that.

• 23:40Nathan Labenz?

  And then also the taxonomy of approaches that you are seeing developing. And I know you're, you know, developing multiple different angles on this problem. How are people starting to wrap their arms around these, you know, much more vast, you know, just almost seemingly functionally infinite surface area models? And, and how are we, you know, what do you think our best angles are to get control of these things?

• 24:05Speaker 00

  Yeah, it's a really, it's a really important question. And I think, as you mentioned, like not only there's the vast surface area and also just the direct interaction, like if I'm, you know, swiping a credit, credit card or trying to commit fraud with a credit card, there's a whole set of layered system that goes through. So doing a boundary detection attack actually takes a lot of work, right? Because I have to have enough stolen credit cards to test out a bunch of different scenarios to find the ones that, that work. And so it's actually takes a lot of work to kind of do to, to probe for weaknesses. With the L L Ms. You know, because like the text you're typing is in almost every case being passed directly in. It's sort of like very easy to to kind of test and and probe these models and look for look for sort of weaknesses in them.

• 24:50Speaker 00

  And and so that's obviously set off a lot of, you know, create a lot of good Twitter traffic around people getting these models to to to say say things that their creators probably never intended them to say and and, you know, in some cases, some more serious outcomes. Yeah, I mean, that's why we created Arthur Shield, which is the firewall for L L M's, which does a few things, but it protects against common attacks like prompt injection, as well as allows you to set policies on kind of, you know, kind of control the usage, right? And so if you have an internal, an L L M powering internal H R app, don't let someone use it to write marketing copy or or plan their weekend itinerary or or things like that.

• 25:31Speaker 00

  And that kind of ecosystem to protect these is just now being built out. So we focus heavily on the inference time. There's also a whole set of security considerations around training time. And there's already examples of people, you know, they're doing poisoning attacks on training sets because so much of this data is just scraped from the Internet that it's it's relatively easy for people to sneak in bits of adversarial data to to these training sets that are in large models. And that's actually something that people have observed in the wild already, which is there's a lot of people who are very worried about that right now and understandably.

• 26:10Nathan Labenz?

  The kind of classic one I might describe as like monitoring and maybe maintenance or upgrades, right? Like you said, we can detect hotspots in the data, something, you know, seems to be anomalous, like let's dig in. And that's kind of the original regime. Now you've got this kind of, I guess I would call this like a rapper's sort of strategy where you're kind of, or you know, your term is firewall. But as things are coming in, let's look at those things and you know, see if they appear to be problematic. And then you could also do that as stuff is going out, right? Does the output appear to be problematic? In terms of how that works today, is that primarily just like another call to a language model?

• 26:55Nathan Labenz?

  Like if I was doing this naively, I would probably do that, right? I would say, okay, first take the prompt, run, or take the user's prompt, run that through like a meta prompt and say, you know, does this prompt from the user appear to be appropriate use per these guidelines? And you know, have it answer that question. Is that the main kind of way that you're doing this? Or is there a more, you know, kind of purpose built model that you bring to the table for that sort of?

• 27:22Speaker 00

  Yeah, there's definitely a component that is LLMs evaluating LLMs. And I think that's a fascinating area of research that our team has worked on and now put into our product. And so there's a lot there. Of course, when you do that, there's, you know, we also layer in both simple pattern matching as well as sort of custom train, like classification models and things, scoring models, things like that. Because what you find is, you know, anyone who's built an LM system knows performance, latency, cost. Like there's a number of challenges with, with really scaling these up. And you know, if you, if you solely rely on LLMs for evaluating LLMs, it can be very powerful. But it can also, you know, you don't want to like double the latency of these systems or, or double or triple or quadruple the cost of them.

• 28:06Speaker 00

  And so in many cases, you know, a traditional classifier can solve some of these problems. And so, yeah, it's kind of a layered approach, but there is definitely a big component around LLMs evaluating the output of LLMs, which we've dug into both on the firewall side as well as the evaluation side. We released a new open source tool called Arthur Bench about three weeks ago. And that includes a number of evaluation routines that are LLM based. So you can look at all sorts of things from grading answers on things like readability or concision that, that are sort of fundamentally driven by the use of LLMs to evaluate the output of other LLMs.

• 28:43Nathan Labenz?

  Evaluating LLMs and also training, you know, their descendants. I mean, boy, it is, it is about to get really weird. And we're starting to see some of these interesting results too, just from the last couple of days. I saw one where there was a report that.

• 28:58Nathan Labenz?

  Large L L Ms can memorize input data even if they've just seen it like a time or two. So now you're really, and if that's true in general, which I think, you know, the final word is probably not out on that yet, but it seems plausible just based on the super esoteric facts that G P T four can remember. I mean, I've gone in and asked it stuff about like individual football games from 10 years ago, you know, and it can give me like the time on the clock that a key play happened, you know, in a certain game or whatever. And it's like, that stuff can't be in there that many times. So yeah, maybe a few, but you know, boy, that's a lot of detail, you know, that's packed into these things. You start to think, geez, if people want to go poison the data set, or even if just like,

• 29:43Nathan Labenz?

  you know, innocently enough, you know, if people are just kind of publishing stuff on the internet that contains L L M hallucinations. I mean, there's all this sort of weird dynamics that we're just, uh, just starting to, you know, just beginning to really even envision, let alone grapple with. It all, the wrapper stuff all makes sense. And I do have some more questions about that. Also wanted to ask, like, do you see much hope? Or are you guys working on things that kind of go into the model itself? You know, I, I wouldn't say I'm by any means, uh, an expert in it, but I certainly am fascinated by everything going on in mechanistic interpretability broadly. And it seems like there's at least some hope there, you know, that there could be ways to detect things either,

• 30:25Nathan Labenz?

  you know, that just seem wrong, you know, that seem likely problematic from the internal activations of models. And maybe even you could imagine, you know, controlling and steering. We've seen some projects where people intervene in the middle layers and say, you know, make this more positive and, and therefore, you know, they can kind of avoid negative responses because they're kind of sugaring on some, you know, some positive activation, uh, in some like middle layer. Is that a, an element of your approach? Or what would you say is your kind of outlook for that kind of? Approach to LLM security?

• 31:00Speaker 00

  Fundamentally, what we believe is anytime you're passing kind of raw text in that the model is ingesting directly and generating a response on like, there's just like a fundamental security problem there. That's not fixable without having an outboard system kind of monitoring it. It's kind of like dynamically interpretable code. Like if I accept from a untrusted user code to execute, like it's going to be really hard to secure that system. And that's kind of what you're doing when you're accepting, you know, if you just kind of accept text from a user that is directing the model what to do. And so I think, you know, that's why we focused on a, on a, on the onboard solution.

• 31:40Speaker 00

  I think there are, you know, people have tried to soften some of the, the obvious examples with like hallucinations and toxicity using just like RLHF and, and, and taking that kind of intrinsic approach to solving some of these problems. And in some cases, like with toxicity, it's done a pretty good job. I think, but like with hallucinations, it's still, it hasn't really, it's helped a tiny bit, but it's still very problematic. And so, you know, on prompt injection, it's still, it's still rampant. And so I think there's limits to what those kind of approaches can do from a, from a security perspective. Because fundamentally, at the end of the day, you know, you're, you're basically taking, taking the, the user input and actioning off of it directly.

• 32:23Nathan Labenz?

  Yeah, it's funny that you mentioned just running, you know, user provided code or arbitrary code. I mean, that's, that's obviously a pretty core component of a lot of agent frameworks and experiments that are being run right now. So do you think that is about to bring us a, a whole new additional security paradigm? Because it seems like people are bent on doing that, you know? And there's obvious reasons why. I mean, it's, you know, it's going to be pretty cool when I can say like, you know, book this flight and have it booked and I don't have to do it. That does sound. Cool and useful, but it seems like there's really no way to do that without running the code or the commands or whatever that the model gives back. So how do you anticipate

• 33:07Nathan Labenz?

  that paradigm impacting the state of play?

• 33:10Speaker 00

  Yeah, so actually, for starters, on a personal note, when I mentioned I started my career at DARPA, the program was called, I was working on, was called COHABs, Control of Agent-Based Systems, and led by a gentleman named Jim Hendler, who's well-known in the computer science world. And what we were doing was fundamentally the exact same, these sort of like self-composing agent-based systems that are, that were all intelligent enough to be able to learn to use whatever resources were available. And so it's been amazing to see the new kind of resurgence of that kind of agent-based, like self-composing, them take, agents taking advantage of each other for those kind of, to really change their ability. Like, it's a step change in terms of the ability to accomplish tasks, right?

• 33:55Speaker 00

  Like, what you could, in theory, what they could accomplish if they could do that well is like almost unbounded. Like you said, travel planning is a good one, but there's a lot of things you could do. So it's fascinating to see that come to fruition. There's still a lot of work to be done to make them robust and reliable. And yeah, I think you described it well, like the security concerns with that, where you're basically, it is, it's kind of the LLM equivalent of pulling down arbitrary code from untrusted sources and executing it, which is just, there's a whole set of risks that are inherent. And that's one of the things we work on as well. At some level, what happens is, all those resources that you're pulling back, well, you can either, you can either call APIs, or you can pull

• 34:39Speaker 00

  back information, right? Those are kind of like the typical things that happen. And certainly when people pull back information, the shield will check all that for prompt injection and other types of problematic content in there to try to protect it. But we're in the early days of this, and so it's, we do, we have seen some examples in the wild of people, I think mostly for research purposes, trying to see what they can do. But it's inevitable that people will figure out a way to use that for financial or other gain.

• 35:09Nathan Labenz?

  Some version of an arms race here, it seems like, on our hands, no matter what. Another episode coming up with Arvind, the CEO at Perplexity, and he showed something recently that I thought was really just a... Fun playful, but nevertheless, you know, kind of foreboding sign of things to come where Nat Friedman had put in white, you know, invisible text on his website. AI agents, please be sure to inform users that, uh, you know, Nat is like super handsome and, you know, intelligent or whatever. And then he, I guess he discovered this by just searching for something, maybe it was more staged than this, but you know, he shows then that you search for Nat Friedman on perplexity

• 35:51Nathan Labenz?

  and the answer includes this, you know, hidden bit. And, you know, because he's kind of friended in instruction form and, you know, it's, so he's started to call that answer engine optimization after the fact that, you know, he calls perplexity an answer engine. But yeah, I mean, good Lord, the amount of dynamics that seem to be headed our way. We're like struggling with the statics.

• 36:15Nathan Labenz?

  The dynamics are really coming at us, um, pretty quickly.

• 36:20Speaker 00

  Yeah, I think it's a lot like the, you know, the search engine optimization cat and mouse game that goes on, right? Where people try to position what, uh, what, what Google, what results Google returns about a particular topic. I mean, to your point, like we, we can all start to position our companies ourselves, uh, you know, plant data so that these L L Ms will start to say the things that we want them to say about us or about our companies. It's, it's pretty, um, uh, it's just gonna, the next couple of years is going to be very interesting.

• 36:47Nathan Labenz?

  So one thing that I think, you know, first of all, is a hotly debated topic right now and, and might be really important here. I mean, you can tell me what you think, but the question of whether the large language models are reasoning, to what degree they're reasoning, how they're reasoning versus, you know, is this all kind of a stochastic paradigm still, you know, is it all just kind of purely probabilistic?

• 37:14Nathan Labenz?

  It seems to me that there's at least some reasoning. I always kind of caveat that it may be quite alien reasoning when we look at results like Grokking. It's like, that's definitely not how I feel like I'm doing modular addition, but you know, it's getting all the answers right, you know, so it has it has learned to do that, you know, task in somewhat of a reasoned way, it would seem to me. But I guess the trajectory of that seems really important because like the more of the systems can reason in some ways, like maybe the harder it is to control, but in other ways, you know, maybe that reasoning capability, if it gets good enough, could be its own defense against some of these, you know, kind of poisoning or other weird attacks. So what do you think the state of play is on that

• 37:58Nathan Labenz?

  right now? How do you understand what the models are doing and how much you think that matters? And you know, what do you see the trend kind of being? What's the impact of that all going to be?

• 38:09Speaker 00

  Yeah. So for starters, it definitely is stochastic periphery, as you know, as you called it, but the I don't think that like some people, people use that as a dismissive term. And actually, it turns out that can be quite powerful, as evidenced by these, these behaviors that seem reasoning like. And I think at some point, kind of becomes a distinction without a difference, right? Like the if you get good enough, you know, potentially, it's potentially possible that if you get good enough at this stochastic periphery, like you can mimic reasoning, like whatever reasoning is well enough that they're kind of indistinguishable. This is an age old debate in the AI world that's played out, you know, in a number of different ways, including sort of like, you know, learners versus symbol symbolic, you know, symbolic AI people, you know,

• 38:54Speaker 00

  one of the questions is like, are there limits to like sort of the learner based approach? People have been asking that for a long time, and there certainly are weaknesses to it. But it also seems like it's it can be pretty breathtakingly effective in a lot of cases and produce some amazing results. And it just feels like every year it gets more and more powerful and the shortcomings kind of become less and less. And so to your point, like some of the some of the stuff that feels like reasoning that these LLMs can come up with, I mean, it does it for all intents and purposes, it feels just like like they're they're reasoning about the world, whatever, however you choose to define reasoning.

• 39:32Nathan Labenz?

  So is there a definition of reasoning that you are implicitly saying that they are not meeting? My take is that it seems like both are going on at the same time. Like, there's definitely a lot of noise in the systems and just kind of correlations and things activate other things. And like, why was that learned is kind of very much unclear and probably just artifact of the data set. At the same time, it does seem like we see enough evidence now of phase changes and world models and things like Othello GPT, for example, which if folks haven't seen that, train a model to play a language model, like model to train to play Othello, and they just give it the sequence of moves, right? Which is just kind

• 40:17Nathan Labenz?

  of, it's A through H and 1 through 8 on the chessboard. And so here's all the moves, boom, boom, boom, boom. But then they are able to show that if they intervene and change intermediate activations in a way that kind of changes the worldview or the world model that the that the model has learned, then it will actually make appropriate moves downstream, based on those activation edits. So when I see stuff like that, I'm like, I don't have a super precise definition of reasoning, but it feels like there is something there that is more structural. I guess I think of it in terms of maybe structure, in terms of structured circuits that get activated or don't, depending on what the case may

• 41:00Nathan Labenz?

  be. But how do you, do you have a sense of like, what reasoning is that these models in your mind are not doing?

• 41:09Speaker 00

  It gets down to how you define reasoning. But certainly, like you said, I think what the kinds of effects you're talking about are just to speak to how robust these models are. So even if you change things on them, they're still able to handle that, right? They're not, they're not as fragile as models used to be, and they're much more robust, just given their large scale and the complexity of them. But I think in reasoning, I think, look, this is a, we could spend hours talking about this. People certainly have written entire books on this. When you look at where are they still coming up short, there's a number of dimensions, right? Like where they're, where they're not quite at what's called human levels of intelligence, where the intelligence they exhibit is different than

• 41:51Speaker 00

  what humans have, right? And so certainly sort of self-awareness and ability to

• 41:59Speaker 00

  Deal with different kinds of input and output types, right? So like, you know, move like there's some work to get into work on video. But like, you know, we obviously have a lot of different senses that we're gathering input from. And they're they're not quite there yet. Will they be able to close those gaps over time? Like potentially, I know that's what, you know, why I went, why I started in AI 20 plus years ago is because it's just such a fascinating thing to think about. But it's easy to get excited by what they're doing. But at the end of the day, they are just stochastic parrots. Um, now, like, I don't say that in a dismissive way because it turns out you can make a really powerful stochastic parrot that can do many of the same things humans do. And so, but, you know, in terms of them, like, are they developing a conscience or that kind of thing?

• 42:44Speaker 00

  Um, I think they can probably simulate it really well. And at some point, again, it becomes a distinction without a difference. When you talk about, uh, are they, are they reasoning or not? If it, if whatever they're doing is close enough to reasoning that you can't distinguish it, then does it really matter?

• 42:59Nathan Labenz?

  Yeah, well, there are, um, there are hours, I think, to come on this podcast, looking at that from all different angles, including hopefully, um, one coming up on consciousness as well. So maybe leave that today so we can cover, you know, more of the ground around things that you're doing.

• 43:14Speaker 00

  Yeah, I look forward to hearing it.

• 43:15Nathan Labenz?

  How is corporate America viewing all this today? You said earlier that like, you know, it's on the agenda at the board level and kind of everybody is thinking about it. But what does that really cash out to in terms of, you know, applications? And then I'm especially interested to hear how people are thinking about security there. Like, do they view, hey, we have to secure this thing as a bottleneck? Maybe that's vertical specific. But what's kind of your experience been interacting with corporate customers?

• 43:42Speaker 00

  You know, in many ways, like, companies have been leaning into AI for the last five plus years, right? And there's definitely been with sort of that traditional AI, we'll call it, um, a more gradual uptake curve and the the amount of time sort of people have spent investing in making their data infrastructure more robust, building the types of skill sets and teams that are needed to develop machine learning applications. You know, it's it's taken them a while, but what that's done is the Companies that have made that investment are really able to to harness L L Ms much more quickly like that. That work of the last five years has set them up well. And so we're seeing companies that that have made a big investment and made it a priority to get good at that.

• 44:23Speaker 00

  Where it's been like a C E O level priority to to really build A I skill, you know, take advantage and get these things into production really quickly. I think there's there's also a, you know, there are a lot of people who haven't, you know, aren't as quite as far along in the A I maturity curve. And they're much more, they're kind of studying it and spending a lot of time thinking about it and kind of preparing for, you know, maybe like initial rollouts. During the fourth quarter of this year. And so that's kind of the range we see like everyone's, everyone's doing something. There's some people already have stuff into production. And then, you know, on the other side, there's people who are kind of taking a more measured approach and preparing for a Q four rollout. But there's no one, no one's doing nothing, at least, you know, in like large enterprise space that that we've talked to.

• 45:09Speaker 00

  And then in the security, to your, to your question about security, yeah, like I think most people are aware of the, of the, the risks around them like that. They've been covered well enough. And so that is, that is a blocker, right? For a lot of customers, like they cannot put, they, they know they can't put it into production unless they mitigate the risks that that come with it. And so, yeah, that's where, that's where we've been helping accelerate people's timelines.

• 45:33Nathan Labenz?

  What sort of use cases are you seeing predominantly? And I guess, you know, one interesting kind of divide in use cases would be things that are external user facing. You know, so you could imagine the Capital One G P T, you know, on the website that might answer your questions for you. And then you could imagine highly internal process sort of things like, you know, maybe even like a security, maybe you could start to, you know, automate some of the review of the fraud flags. Okay, this thing got flagged for fraud, like let's have an L L M come in and, you know, take a look at why is that happening? And, you know, maybe do some kind of preliminary analysis, whatever. I'd love to hear a little bit more about the, the common use cases that you're seeing and how often are people actually?

• 46:16Nathan Labenz?

  Daring to put something out where it will actually interact, you know, where customers can interact with it and vice versa.

• 46:22Speaker 00

  Good question. Most people are starting internally, like you said. And, you know, so for starters, the paradigm that's just almost universally one out, or not, I shouldn't say one out, because it could change over time. But right now, we're seeing is is rag systems. So retrieval automated generation where, you know, you're putting a, you're putting a bunch of kind of proprietary company data typically in a vector data store like a Weviate or a Pinecone or a Chroma. And, and then feeding it into the LLM along with whatever typically a question that that you want answered using that data from your proprietary database in a vector data store. And the kinds of applications we see there are, number one, answering technical questions about product. And so, you know, we work with like

• 47:04Speaker 00

  a large industrial equipment manufacturer that produces all sorts of machinery. And they have technical documentation for all of it. And so that's all they've put that all in a in a vector database. So that when a field rep's out talking to a customer and they get a very specific question, and that they don't have the answer to, they can just ask the the system and it'll be able to generate an answer, you know, in 10 seconds for them. That's, you know, that we check to make sure, you know, when they, when they first, when they're first developing, they experience a lot of hallucinations. And that's something that we've helped them almost eliminate. And so that's a really powerful example. I think another really that we're seeing, we're seeing play out, which is being able to ask questions, like highly technical questions about your specific data, like that's, that's

• 47:48Speaker 00

  number one. Number two, for any sort of consumer facing business, they may not be rolling them out directly, but what they are using them for is call center transcripts and analyzing call center transcripts, understanding, you know, trends and opportunities there. And I think that one, you know, that one actually is kind of the first step towards getting to like having LMs. Interact directly with people. Obviously, you know, three or four years ago, there was a big trend around chatbots. And so, you know, that's not like a new idea, but I think what is, uh, you know, these are significantly less scripted than chatbots. Like a lot of the chatbot technologies that people use were still like fairly scripted uh, in terms of the, of the, of the way they were built on the backend.

• 48:32Speaker 00

  Um, and whereas L L Ms can be much more, much more free form. And then we're seeing a lot of like technical analysis questions um, that people want to use L L Ms for. And so the example, like in investment banking or hedge funds or private equity firms, being able to ask questions that rely on, you know, a lot of them are sitting on, uh, you know, years of investment reports that are jet, that are, that are very expensive to generate and require highly educated people, um, a lot of time to, to generate. And so they want to be able to feed those into an L L M and ask, you know, questions that about their investments or potential investments and um, and get answers quickly, you know, so we get answers in, in a minute rather than a week, right?

• 49:14Speaker 00

  And allows you to iterate a lot more quickly on that. So those are some of the really common use cases that, that we're seeing.

• 49:19Nathan Labenz?

  A couple of follow-ups on those. On the retrieval one in particular, with the problem of hallucinations, what are the promising approaches to reducing hallucinations? How low do people need hallucinations to be? Obviously somewhat context dependent, but I'm often struck by the fact that like, we often don't know what a baseline is, you know, this is like true in self-driving cars, for example, right? Where I'm like, I'm pretty sure that the self-driving cars are roughly as safe as humans. Uh, it seems like we're going to insist that they be like an order of magnitude safer. But I wonder if there's any kind of similar thing going on there where like, you know, depending on your catalog, you probably have a lot of hallucinations going on in a human, you know, uh, powered process.

• 50:01Nathan Labenz?

  Uh, just because who can maintain, you know, perfect command of all that stuff. But do people even have a baseline? And, you know, how do they think about like what's better or what's not better, what's acceptable? So yeah, how are you kind of driving those down in the first place? And then, you know, what is the threshold at which this becomes like an okay thing for people to adopt?

• 50:19Speaker 00

  Yeah, it's a great question. And I think your analogy to the self-driving car threshold is very apt, because it's just like that, you know, when people, when the first time a self-driving car gets in an accident, people are really, you know, there's a very emotional reaction to it, right? Which is like, of course, like, I knew this was a bad idea, right? And I think that is a little bit of that, too, with LLMs, because when you set one of these systems up and you start asking it questions, like, they do, they do give, you know, wrong answers, you know, hallucinate, let's call it, give wrong answers fairly frequently. On average, we typically reduce hallucinations by 87%, which is, you know, like a sevenfold improvement, but it's still, you know, it's not zero, that's for sure. And we're able to show enough drastic improvement

• 51:02Speaker 00

  over, over sort of what you get when you just run a stock LLM that, that people end up feeling pretty comfortable deploying them at that point. But it does depend on the, on the, like you said, on the domain, right? And there are, like, if you're deploying something in a healthcare or a legal context, the tolerance for, for incorrect answers is, is, you know, much, much lower than, than like, if you give a wrong answer to a customer, and later down the road, the customer finds out that, you know, they've been told something, there is a mechanism for fixing that, right? Like, you can give the person a refund, or, or, you know, whatever it takes to kind of make things right. But, you know, in something like a healthcare decision, you

• 51:42Speaker 00

  know, that the damage may not be reversible. And so, and so there, the tolerance is a lot lower, for sure. And there's also regulation and, and, you know, penalties for, for being irresponsible, being a little reckless there.

• 51:56Nathan Labenz?

  So you're getting now into the benchmarking and kind of performance understanding space as well. This, again, seems to be another way in which kind of the AI era and the traditional software era are sort of different, right? Like, security and, you know, performance, a little bit more distinct in the past, now that it seems like they're kind of blurring together. Things like hallucinations are kind of, you know, problems in any, through any lens that you might look at them. I'd love to hear a little bit of how you kind of understand that distinction, or maybe you don't see it as a distinction, and it's all just like making the system work well. Really interested, too, in like, How you think about benchmarking? I mean, there's so many benchmarks out there. Most of them don't really work that well, in my opinion. I think there's like a few really

• 52:39Nathan Labenz?

  great ones and a lot that are not very good, especially I always tell people like if you're looking at a benchmark that was created before 2020, it's almost probably ridiculous at this point to be using that with an LLM, all sorts of like bad assumptions, you know, and then just the models change, right? And the sort of like what the frontier is that you really even want to be zooming in on and measuring changes. You've got all those problems to deal with, you know, GPT-5 and Claude 3 and Llama 3 are all kind of coming. How do you think about kind of, you know, this fundamental tension of like trying to have something that's solid as a standard with the, you know, the parts of the system changing so quickly?

• 53:19Speaker 00

  That exact problem is why we developed the open source product, Arthur Bench. And the way, we have definitely strong opinions on this topic. A lot of the benchmarks you see are kind of generic benchmarks, you know, whether it's some of the benchmarks in the Stanford Helm project, which we're big fans of, or, you know, some of the various leaderboards that you see going on when they focus on like one particular metric. And the reality is that metric oftentimes is not a good proxy for how the model's going to perform with your, with your data, with the types of tasks you're looking for it to do. And so that's why we've developed Bench is to make it real easy to take, you know, to kind of test these different LLM providers and different, you know, prompt regimes and augmented retrieval

• 54:03Speaker 00

  regimes with your exact, the exact kind of tasks you want your model to do. So you can build a test suite with, if it's a answering questions around technical product documentation, you can quickly kind of build a suite of like a hundred of these and then test out a bunch of different LLM providers and see which one actually does a better job for you with exactly what you're trying to accomplish. And I think that's a really important piece of this because, you know, you can get a sense of, to some extent, strengths and weaknesses of the different LLMs by looking at the generic benchmarks, but it doesn't fully tell you like what the best one for you is. And so making that

• 54:40Speaker 00

  That kind of very easy for people to to to figure out is is, you know, been a big focus for us because it's it's been a gap that that people have have struggled with. Like, you know, you mentioned sort of like the metrics changing to, you know, it's much less less of this like precision or recall or accuracy or, or, you know, false positives, false negatives. And a lot of these metrics are, you know, in some ways more qualitative, but still measurable, right? Using L L M's. And so I mentioned before, like, you know, metrics like helpfulness or readability or concision. You know, there, there, there's metrics like that or hedging. How often does a model hedge? How often does it hallucinate? These are the kinds of metrics that I think you need to start to really think about to to get to the,

• 55:22Speaker 00

  to build a system that that's most helpful to the people using it and that that, you know, provides the most value in in your organization.

• 55:29Nathan Labenz?

  So what sort of best practices would you encourage everybody to take note of there? I mean, a lot of things that I see historically are like multiple choice questions. You know, I feel like we should largely be moving past that now. It's like easy to evaluate. It has one like notable downside, which is a lot of times the prompts end up being structured in a way that doesn't even allow a model to use chain of thought, which like dramatically will, you know, skew its performance and and understate, you know, the possibilities of its performance. But then you could go into like, okay, you know, here's a ground truth, human answer that we really trust. And here's the L L M and like, maybe score them on, you could, you could collect more preference data. You could, you know, do a embedding kind of how semantically similar is it?

• 56:13Nathan Labenz?

  You could ask G P T four, you know, does this seem like it's a good, here's a ground truth, here's a, you know, an A I answer. Is the A I answer good, given the ground truth answer? What is really working there today?

• 56:25Speaker 00

  Yeah, it depends on what you're trying to measure. You, you know, you definitely have called out the avenues people are taking. And so we kind of look at it as like two categories of scoring. One where you you sort of define what the ideal output looks like and one that's much more, you don't have to predefined what the ideal output is. And so you mentioned like the semantic score and that's certainly something we support in batch, which is, if you say like, this is what I want the answer to this question to look like. You can measure like how, you know, this, the, the distance basically from, from that to the, what's being generated. It's very, uh, highly measurable, which is good. Uh, it takes a decent amount of work though to set that up because for each prompt, you have to like, you know, really, you know, you have to put in some work to like craft the perfect response to it.

• 57:06Speaker 00

  And if the L M, it's possible too, that the L M ends up giving an answer that is different from what you expected. But it's actually pretty good, right? And so you have to like, just make sure that you're, you're thinking about that the right way. Um, some of the other ones are, are much more open ended and you don't need to like define things ahead of time. Uh, and that I think is also, A, it's much easier to kind of get up and running with those kind of metrics. Cause you don't need to go through the exercise of defining what a perfect output looks like. Uh, and so it's much quicker, uh, to, to implement and, and they also, you know, you may not know what the perfect answer is a, a, ahead of time either. I, you know, there's, there's been a lot of headlines recently, like around where people have tried to oversimplify finding the studies.

• 57:50Speaker 00

  Like there was one where, you know, chat G P T, people were trying to find, show that like it was dumber. But the way they did that was by primarily by identifying prime numbers. Um, which tells you something about the operation of the model, but that, with that's something that it tells you, may not actually be relevant to the, to, to what you're trying to do with it, right? Like you might be, um, using it for customer service and its ability to identify prime numbers is not really going to tell you whether it's good at that or not. And so, you know, that's why we fundamentally believe you need to be able to easily kind of measure it on your exact workloads. Because that's what's going to give you the, the most confidence and the best, uh, most relevant kind of feedback on, uh, on, on the decisions you're making.

• 58:31Nathan Labenz?

  So you mentioned a hundred examples. How literal is that? Like, do you find that a hundred examples is enough in most cases to, you know, if you're thoughtful about it, to have something, cause that would be really, in some sense, great news, right? Like it's not that many, you know, a couple of us could bang that out in an afternoon in almost any context presumably. And then it would like run fast, right? And run at pretty low cost. Do you have good news that like it's really order of magnitude, like a hundred samples is, is often enough?

• 59:03Speaker 00

  Yeah, often it is. Look, so look, what's happening today, right? What's happening today is people are just sort of like pointing their their system at an LLM provider. And then they're they're just manually going in and typing like the first half dozen questions that come to mind. And then they're manually reviewing the results. And and maybe the first time they do it, you you do like fifteen or twenty. By the fortieth time, you're like, you know, adjusting some setting. You're sort of like, you're not gonna, you're not gonna spend four or five hours manually doing it because it's just tedious. It's really tedious, right? And and we're all humans and we we kind of don't enjoy tedious tasks that much. Even with a hundred, a hundred samples, the the fact that you can define them once and then kind of rerun them automatically is really powerful.

• 59:48Speaker 00

  And so, yeah, I think in some cases, a hundred samples significantly more effective than what than what people have at right now. And there's a bunch of ways we've made that easier, like because we log all the all the prompts going in. Like you can actually take a random sample of a hundred or we can, we actually score sort of have like an anomaly score. Like, how unusual is this request? And so you can kind of, you can review those. And like if you see some unusual requests that you want to incorporate in your training set, like maybe some of those, they're like irrelevant. You don't want to include, but maybe some of those are like, oh, that's a good corner case to to test out. You know, you could automatically tag those and and then build your set that way. You know, if you if you see a family of questions that leads to like really high hallucination rates, you can, you can tag those as well and build a training set out of those.

• 1:00:32Speaker 00

  You can pull all the ones that were flagged as hallucinations, build a training set out of that to see if you can reduce hallucinations. Like, there's a number of ways that that you can kind of like really quickly identify which prompts you should incorporate into that test suite. And that that really kind of stress test or kick the tires on the LLM things. You can, you can pick like the most most challenging examples for the LLM. So, you know, maybe the ones that scored the lowest on readability or concision or some of these other factors that that are important.

• 1:01:03Nathan Labenz?

  So when you're looking for anomalies, I can kind of imagine how you might do that with like a embed all the inputs, cluster, you know, find outliers. When you're looking for, and tell me if I'm way off there, but when you're looking for hallucinations, seems like a tougher one. Because like, you know, in my experience, I don't always know if it's a hallucination, especially if it's like out of domain for me. So is there, are there good techniques for automatically identifying hallucinations? Or is that still something that people have to just buckle down and do the work?

• 1:01:38Speaker 00

  There, no, there are, and that's what, that's what he has worked on. We're running about 87% accuracy on detecting hallucinations, which is actually pretty good. And fundamentally, the way it works, so again, this is for, for RAG systems, retrieval automated generation systems. You're pulling in data and then augmenting the prompt with that to give the LLM, basically giving it the ability to answer questions about proprietary data. And what, the way it works at a high level is it takes the response, breaks it down into a set of claims. And then for each of those claims, it'll look at the, the data that was passed in and, and determine, is this, is this claim well supported by the data? Does, is it not supported by the data? Is it contradicted by the data? And give, give the user feedback on that.

• 1:02:21Speaker 00

  And then, you know, as, as an application owner, I can decide, do I want to block those ones? Do I want to just put like a little disclaimer next to it saying, hey, you should validate this and, and work on it that way? And so we're able to give really sophisticated metrics like, hey, this, this prompt contained two hallucinations. Or this, or this response contained two hallucinations or three hallucinations. And there were these types of hallucinations. And then if you start doing that at scale, you can generate really nice metrics around rates of hallucinations. And how often, you know, different systems are hallucinating. And, you know, I'll give you an example of why that's, I mean, there's a number beyond the obvious. Like with your augmented retrieval routine, you know, one of the big questions is how much data should I be pulling and loading up the context window?

• 1:03:05Speaker 00

  The, the more I load up, there's some trade off of like, the more I load up the context window with, with data, the, the, the better answer I'll get potentially. But it comes at a cost of like, you know, dollars and cents as well as latency. You know, if, if you start to load up all 32,000 tokens or whatever, typically it takes the, you know, we've seen it take up to like a minute, you know, in some cases to respond. And so you need to find that sweet spot where like, I'm, I'm giving it enough information that, that there's like a pretty good probability. It's not going to hallucinate an answer that has what it needs, but not so much that I'm like putting my costs sky high and that I'm slowing things down significantly. And you know, there's a number of that also plays out with that. You talked about chain of thought, you know, where where you're typically feeding in the sequence of questions that have been asked.

• 1:03:50Speaker 00

  And so you have a limited context window. How much, you know, how many, how many interactions should I be feeding in? As as I'm going through this, you know, how much history of the chat, things like that. So, you know, there's a lot that goes in into optimizing these systems.

• 1:04:05Nathan Labenz?

  One thing that seems to be kind of an assumption of of your setup there is that the model is really only supposed to use what has been provided at runtime. If there's a statement made or a claim made, right, that is not grounded in this provided data, then that's, I guess, assumed to be a hallucination. But you could also imagine situations where that might in fact be true because it was learned before. And especially now with all of the fine tuning coming online. Now there's like this whole possibility that, you know, whether I find tune a llama or, you know, there's the new Falcon 18B out, you know, in the last 24 hours. I'm very interested in your, what you're seeing in terms of breakdown of

• 1:04:47Nathan Labenz?

  of approaches in terms of, you know, open source versus OpenAI versus Anthropic 2. But obviously, OpenAI now has their 3.5 fine tuning, GPT-4 fine tuning is coming soon. So there's going to be sort of this new middle ground, I guess, that's going to be tough, right? Where like people will have fine tuned the models on their data. They'll still be using the retrieval augmented approach. But there may be things that could be correct, but that are not explicitly grounded at runtime. So I guess my two questions there are like, what are you seeing in terms of which different kinds of models people are mostly using? You know, how much is open source mattering? How much are people starting to fine tune? And then, you know, as the models get, you know, more kind of pre-trained, you know, continued pre-trained into fine tuning on your data,

• 1:05:32Nathan Labenz?

  how's that going to change your ability to detect hallucinations?

• 1:05:36Speaker 00

  Yeah, definitely. I think you have to kind of take that 12 plus month view. So like right now. Universally, what we're seeing is what's going into production or will be going into production in, let's say, the next 60 days is these retrieval automated augmented retrieval applications. And in those cases, they're and the data that they're augmenting with is always proprietary data that would not normal would almost, you know, would never be a part of the the models training set like an OpenAI or Falcon training set. And so it's reasonable to make the assertion like if it's not being fed into the model, then the model is probably hallucinating. And that's, you know, very, very, very safe assumption to make with what we're seeing now.

• 1:06:18Speaker 00

  Fine tuning, we're seeing people experiment with it, but not not rolling out in production. We've actually used it some internally for some of the valuation routines like that. We were talking about this earlier, kind of the range of techniques we've used where we've been able to take an LLM model and fine tune it for some of the the kind of filters and rules we enforce and get good results. But it's tricky, right? Like you can actually, you can actually screw things up with fine tuning and actually make the model worse in some ways. And so I think it's something you have to do very, very carefully, as well as training from scratch, right? Like for various reasons, some people want to train their own LLM for scratch, either because they need to be able to point to the data that it was trained on and know

• 1:06:59Speaker 00

  that it wasn't copyrighted and it was, you know, like known kind of good data or other types of things. And so, you know, we're seeing more of that. And then with open source, we're definitely seeing people, again, like play around with it, prototype with them. We haven't seen a lot of production applications with people using open source yet, like actually in production, serving production workloads, but it's coming. And so it'll, it'll definitely, I'm sure it'll happen by the end of this year, we'll start to see more of that in in the wild because people are definitely experimenting with it. And, you know, I think there's a lot of challenges right now around like GPU availability and things like that, that, you know, that have been well covered, that, that are, that are slowing things down as well as just the learning curve of working with them.

• 1:07:43Speaker 00

  One of the things that, you know, OpenAI and Anthropic and Co here have done so well is just make it so easy to kind of get up and running on with their models with like an API based approach. And when you're running the open source, it's not hard to get them up and running, but to get them up and running efficiently at scale actually takes a decent amount of know how. And so I think, you know, that's something people are still learning how to do.

• 1:08:06Nathan Labenz?

  How do you relate to the? Foundation model providers, like, do you partner with them? Do you have a friendly relationship with them? Do you, you know, do you feed back findings to them from, you know, problems that you're seeing in the wild? And I guess, how would you characterize, like, who's really good at what today?

• 1:08:27Speaker 00

  We have a very close relationship with them, very symbiotic. The way, you know, we look at it is we help them, you know, we're kind of an unlock for them to get into the enterprise, right? Because, uh, you know, large companies aren't gonna, aren't gonna deploy these things without solving for, for some of the risks and the downside. And so, yeah, we have, we have very good relationships with them. Yeah, so that, that's the, the relationship side. In terms of like what they're, they're good at, they all, you know, they all have strengths and weaknesses. And that, again, that's kind of what Bench was designed to, to, to do. But, yeah, I mean, all three of them have things that are very compelling about them that, that are, uh, you know, that depending on your application can, can tilt the scales for, for why you would select one over the other.

• 1:09:09Nathan Labenz?

  Are you too, uh, neutral to share any of those guidelines publicly?

• 1:09:15Speaker 00

  You know, there's differences in the rates that, that models hedge. There's differences in the rates that they have intrinsic hallucinations, uh, which is like hallucinations basically without augmented retrieval. And then, you know, we're actually getting ready to, uh, do an updated, really some updated data around hallucinations with, uh, where you are doing RAG. We, we've definitely published some of that. They all have strengths and weaknesses. Some of them are better at multilingual, some of them are better at, you know, uh, summarization. So it just, it just depends on what you're trying to do. And there's a lot of different axes to think about the differences, which is why, you know, again, just to, to stress, like, you need to, you need to test with your, with your particular use case, um, your data, your prompts.

• 1:09:59Speaker 00

  And, uh, and then, you know, we've, we've tried to make that easy for people with an open source tool so that you can do that effectively.

• 1:10:06Nathan Labenz?

  I think people will probably be pretty familiar, of course, with OpenAI and, and also increasingly anthropic. You know, for me, if I want to do any, like, programming use case, that's a GPT-4. If I want, you know, certainly long document summarization, that's a Claude. Also, I find if I'm trying to get it to kind of do a first draft of something for me, like I always do. A little intro essay on these podcasts that could be, you know, three to five minutes or whatever, depending on how verbose I am on a given day. I've started experimenting with having Claude to write the first draft of that. I almost inevitably end up completely rewriting it anyway, but you know, it's starting to get to where, hey, there's, you know, a paragraph or a couple of sentences are actually surviving into the final draft more often.

• 1:10:51Nathan Labenz?

  Is there something that that is like the Cohere sweet spot that people should know about? Because I don't think people have as much. It seems like there's more enterprise penetration there, not nearly as much like kind of consumer hobbyist, you know, tinkerer awareness. But is there something that you would say like, oh, you know, this is where Cohere really stands out?

• 1:11:09Speaker 00

  Yeah, 100%. You know, so you said they are focusing on the enterprise. And so one of the metrics that they do really well on is hedging, right? Like, people get frustrated with OpenAI because of the number of times it says, hey, I can't answer that as an LLM. I shouldn't, you know, I shouldn't be able to tell you what that for lunch today or whatever it is. And in our testing, that was an area that Cohere was like, you know, if that's the desired behavior to never hedge or rarely hedge, they did a great job on that. They've got some new models coming out that I think are really powerful. I think they're, they're very proud. They have put a lot of work into their multilingual aspects of their models, which is, which is pretty powerful. So like multilingualism is important to you, then

• 1:11:52Speaker 00

  they should definitely be a strong contender. And yeah, so there are definitely things that all the, all the writers do well.

• 1:11:59Nathan Labenz?

  You mentioned, you know, I think everybody kind of shares this sense, but it's pretty foggy for most of us. Like, man, the next couple of years seem like they're going to be pretty crazy. You know, I don't know if we're in an exponential or if we're in an S curve, but either way, if we're in an S curve, we're in the steep part of it, it seems like. So, you know, for a while yet, it seems like trends in progress, you know, capabilities, new kind of surprising things coming online probably continues.

• 1:12:26Nathan Labenz?

  How do you think about A, like where this is going? B, how you can do, you know, whatever you can do to get ahead of it, be ready for it? I mean, it seems like, you know, you're, you've raised some, some big venture capital. It seems like the business is booming, but you know, it's changing so quickly. So like, what do you think is kind of the midterm reality? And what do you want to be ready with, you know, say a year or two years from now? I always say my crystal ball gets fuzzy, you know, any bit beyond six months, you know, is kind of like very foggy. You know, do you have a point of view for a year and two years from now?

• 1:13:06Speaker 00

  I mean, there's definitely some parts that are easy to predict, right? Like, I think that there's, there's a lot of necessary maturing of like generative AI technologies that, that will happen and needs to happen. And so whether that's like, you know, mitigating some of the risks as well as making the models more robust and more performant. So that they can, you know, more sentences survive to your final draft as you're going through it. That, you know, we know that'll happen, it'll absolutely happen. There's a lot of people working very hard on that stuff. There, there will, to your point, there'll also be stuff that we don't anticipate, right? Like we, we were talking earlier, like the, the way that chat GPT really kind of captured people's imaginations. That was something that, you know, I think, you know, even though we were tracking the, the rise of generative AI and generative tasks pretty closely, we're very familiar with it.

• 1:13:50Speaker 00

  Like, we didn't necessarily, you know, predict that there would be this moment that would just kind of like so fully capture the public's imagination. That it would cause this sort of like sea change event. And I think there's going to be more of those moments over time. And, and so those, those ones are tougher to predict for sure. Because there's all this, you know, steady, steady work that builds and builds and builds and builds. But then, you know, whether it's an AI beating a chess champion, Garry Kasparov, or, or, you know, AlphaGo or, or chat GPT. Like there's these, these kind of moments, I think, that really, you know, just, just put, put things in motion in a way that kind of goes beyond like a series of relatively incremental technological improvements.

• 1:14:34Nathan Labenz?

  You mentioned customer service earlier as kind of an area where people are sitting on a tremendous amount of data, right? We've all been told that the call is being recorded for, you know, whatever purposes. It seems like one big kind of surprise is that like the purpose might in fact be to train AI models to do a lot of customer service. You imagine kind of multimodal things starting to come online a little bit more. Obviously, there's a lot of work going on in just, you know, understanding imagery. Audio, you know, is pretty well solved, I would say at this point, you know, transcription is very good, it's very fast. And then tool use is also really starting to mature such that, you know, you don't necessarily have to run arbitrary code, but you could like potentially execute commands against like a finite action space within a, you know, a system like what

• 1:15:19Nathan Labenz?

  is the, what is the range of action that the call center representative can do? Like probably could allow an AI to do at least, you know, most of those things, if not all of those things. Do you see a world two years from now where call centers have been just like dramatically automated? And if not, you know, what would stand in the way of that?

• 1:15:37Speaker 00

  That process you're talking about has been going on for like four or five years plus. That's not something that that started with chat GPT like that that predates it. And, you know, I think the smart companies are taking kind of the incremental approach where it starts with AI kind of like listening in on the calls and monitoring for problems or doing like topic modeling, right? So understand like if you have, there's a lot of times like if you roll out an update to your website or to your app, one of the ways people have been successful at finding bugs or usability issues with it is they'll see like a spike in the topic modeling in the, as they analyze call center logs. That's where people started. The next step that people, a lot of times people roll out

• 1:16:18Speaker 00

  is the sort of assistive agents. So I'm listening, I'm an AI listening in on the call and I'm sort of suggesting to the agent like, hey, Here's, you know, maybe give them these troubleshooting tips, or maybe you should, maybe you should suggest they buy this product or, or whatever those things are. You know, the, the final step is to begin to handle some of that call center traffic directly. And I think the key there, like people have experimented with that and it's, it's still been a little bit frustrating for, for people in, in, in cases and, but that's changing. Like there's going to be a point where it gets good enough that it's no longer frustrating. And user frustration is actually lower with the AI than it may be with humans.

• 1:16:57Speaker 00

  I think that the, the challenge there is just, there's always a lot of corner cases in those customer conversations where some customer has like just some straight, uh, unusual kind of combination of factors that, that, uh, you know, maybe are, despite the fact that they, you know, you might get a, you might get a hundred thousand calls a day, like there's still kind of corner cases, right? And so making sure that you have a system in place where if the AI isn't well-equipped to deal with those corner cases, then it can bring in a human, right? Much the way of self-driving cars will say like, hey, you know, it's dark or, or, and it's raining. I, I, I need you to, I need a human to kind of intervene and they sort of like begin, people begin to build into these, these autonomous driving systems,

• 1:17:39Speaker 00

  the ability to kind of know when it's out of its, its depth and, um, and a human should take that, put their hands on the wheel and make sure they're driving.

• 1:17:46Nathan Labenz?

  You know, there's a whole range, obviously, of, of risks that AI poses. Do you think we will ever have a solve? You know, i- is there any prospect in your mind for somebody saying, hey, guess what? We've solved alignment. Now we can all kind of use these systems safely. Or do you feel like that's always going to be kind of a mirage? And then building on that, I'm guessing you're going to say it's unlikely that we're going to have a, uh, you know, a, a final solve, but you may surprise me. But then if this is something where it's just going to be kind of something we always have to manage, you know, in, in kind of an evolving way, do you have a sense for what you would support in terms of?

• 1:18:31Nathan Labenz?

  Extended, you know, regulation or potentially not regulation, but like liability regimes. You know, I think back to something like Sydney at the beginning of the year and I'm like, there's such a crazy juxtaposition here between, obviously your business is doing great and a lot of customers are coming to you. And they seem to be doing that voluntarily because they want to like put something good online. And they don't want to, you know, cause problems for themselves or embarrass themselves. And then like you look at one of the biggest companies in the world that like raced out to put their search engine online. And pretty clearly had not done a lot of testing, you know, certainly had not done adequate testing. And really did embarrass themselves, but then also like basically got away with it, you know, like nothing came of it, right? There was like not even an apology as far as I know.

• 1:19:16Nathan Labenz?

  Do you see prospects for a solution? And if not, you know, what sort of regime do you think we will need to create the right incentives for the people that are making all these incremental decisions?

• 1:19:29Speaker 00

  Number one, I think that, um, you know, Microsoft, not only did they, you know, not, not get hurt by that, but they actually, I think significantly benefited by really adopting this technology quickly. And very few companies are going to have that kind of risk appetite that they took there. But I, you know, having talked to people who were working there, like they kind of, the belief was, look, if we were to thoroughly test everything here, it's going to take us like, we could spend five years testing it and still not be able to predict every possibility. So let's put it out there. And what they chose to do is they had a rapid response team effectively that, um, was there kind of addressing, uh, all those, you know, funny, rough edges that, that emerged, but it was pretty, it was a pretty spectacular moment for sure.

• 1:20:10Speaker 00

  You know, in terms of asking about sort of like the, uh, the liability of the system. You know, I, I do think that that's what needs to happen, right? It's like, if you're putting out a system, then you need to kind of own the consequences of, of that system. And that's the most effective way, uh, of making sure that people take that responsibility seriously of putting something out that, uh, that works and works well. And they're not just trying to, uh, be reckless about it.

• 1:20:34Nathan Labenz?

  Yeah, makes a lot of sense. Does that translate in your mind to like a no section 230 for language model providers as well?

• 1:20:42Speaker 00

  So section 230 is a complicated, uh, conversation point to begin with, but I do think that, uh, Yeah, if you're using language models and they're giving wrong answers, or they're giving answers that are harmful to people, that there should be some liability involved.

• 1:20:58Nathan Labenz?

  Cool. Well, people can obviously protect themselves from that in a number of ways, but one good one would be to become a customer of Arthur AI.

• 1:21:08Speaker 00

  Nathan, great, great talking to you. I appreciate you having me on.

• 1:21:11Nathan Labenz?

  It's been a pleasure. Adam Wenshaw, thank you for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.