Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC

AIUC first got our attention with the NFDG backing, and have just announced a $40M series A https://aiuc.com/updates/series-a-announcement today, with the most impressive industry advisor list we may have ever seen for an early startup behind AIUC-1 https://www.aiuc-1.com/consortium, their agent standard backed by real insurance: From being Anthropic’s first product hire to building the standards, testing, and insurance infrastructure meant to make frontier AI deployable, Rune Kvist is betting that the biggest constraint on AI adoption won’t be capability it will be trust. In this episode, the AIUC cofounder joins swyx and Vibhu to announce a new $40M round and explain why companies like Cursor, Harvey, Lovable, and ElevenLabs are increasingly confronting a problem that gets harder as AI gets better: who is responsible when autonomous systems fail? We go deep on AIUC-1 https://www.aiuc-1.com/, the emerging standard for agent security, safety, and reliability; how AI agents are stress-tested for jailbreaks, hallucinations, and data leaks; and why Rune thinks standards and insurance could become critical infrastructure for AI. We also discuss the growing trust gap between governments and frontier labs, AI-enabled cyber and biological risks, why every model can ultimately be jailbroken, what happens when a $20 coding agent causes $200M of damage, whether AI engineers should be certified, and why even after AGI there may be one job the labs can never do themselves: be their own watchdog. We discuss: * Why risk, liability, and trust may become the binding constraint on AI adoption * Rune’s path from reading the Scaling Laws paper to joining Anthropic in its earliest days * What Anthropic understood about scaling, compute, and the future years before it became obvious * Why Waymo illustrates the gap between AI capability and real-world deployment * AIUC’s $40M round and work with Cursor, Harvey, Lovable, ElevenLabs, and other frontier AI companies * AIUC-1: a standard for AI agent security, safety, and reliability * How agents are tested for jailbreaks, hallucinations, and data leakage * Why most AI companies optimize the happy path without seriously stress-testing adversarial cases * Why AI standards may need to update every quarter instead of every decade * The emerging trust gap between frontier AI labs and governments * Cybersecurity, child safety, biological weapons, and the expanding frontier-model risk surface * Why standards and insurance may need to evolve together * How Lloyd’s of London can insure AI systems and bring trust to enterprise deployment * What happens if a $20 Cursor subscription contributes to a $200M plane crash * The Air Canada chatbot case and how AI failures are beginning to clarify legal liability * Why copyright may be one of the hardest AI risks to insure * Evals, mechanistic interpretability, monitoring, and models becoming aware they’re being tested * The impossible CISO mandate: adopt AI fast, but don’t let anything go wrong * Why robotics will make AI liability dramatically more consequential * Whether AI engineers should have Level 1, 2, and 3 certifications * AIUC’s roadmap across agents, frontier models, robotics, and universal red teaming * Why AGI could become a question of national sovereignty * Why the labs can never fully serve as their own watchdogs * The Big Short problem: how do you stop competing watchdogs from racing standards to the bottom? Rune Kvist * LinkedIn: https://www.linkedin.com/in/runekvist/ * X: https://x.com/RuneKvist AIUC * https://aiuc.com Timestamps 00:00:00 AIUC’s $40M Round and the Risk Bottleneck for AI 00:01:07 From Scaling Laws to Early Anthropic 00:07:58 Why Trust, Not Capability, Could Limit AI Adoption 00:12:19 Founding AIUC and Building AIUC-1 00:18:52 How AI Agents Are Audited and Stress-Tested 00:25:26 Frontier Models, Government, and the AI Trust Gap 00:33:32 Cyber, Child Safety, and AI-Enabled Biological Risk 00:38:14 Why Standards and Insurance Belong Together 00:41:45 What Does an AI Insurance Policy Actually Cover? 00:50:44 The $20 Cursor Subscription and the $200M Plane Crash 00:53:53 AI Liability, Monitoring, and Earning Enterprise Trust 00:56:21 From AI Agents to Models to Robotics 00:58:29 Copyright, Adverse Selection, and AI Insurance 01:03:28 Evals, Mechanistic Interpretability, and Eval Awareness 01:08:36 The Impossible Enterprise AI Mandate 01:11:52 Prediction Markets vs. AI Audits 01:14:43 Should AI Engineers Be Certified? 01:19:10 AIUC’s Roadmap, AGI, and Who Watches the Watchdogs? Transcript Introduction: AIUC, the $40M Series A, and Risk as the Adoption Bottleneck Swyx [00:00:00]: Okay, we’re in the studio with Rune from AIUC, the Artificial Intelligence Underwriting Company, with our trusty co-host, Vibhu. Welcome. Rune Kvist [00:00:10]: Thank you. Thanks for having me. Thank you. Swyx [00:00:11]: What are you announcing today? Rune Kvist [00:00:12]: We have raised $40 million, led by Ribbit Capital and First Harmonic. Swyx [00:00:17]: You first came to my attention when Nat and Daniel invested in you guys. Is the story, like, pretty much the same? Like, what are you today versus what you thought you were back then? Rune Kvist [00:00:26]: When we raised our seed round, we had a hypothesis that at some point risk was going to hold down adoption. At that point in time, that felt kind of hypothetical, and I think that is now over. Clearly, the moment is now with Mythos and Fable. It’s pretty obvious that literally the binding constraint on adoption is risk. And so for us, it feels like this is a natural continuation of the same hypothesis, but where previously it was speculation, now it feels like fact. Swyx [00:00:54]: And let’s get a list of the customers that you’re highlighting as part of your Series A. Rune Kvist [00:00:58]: Totally. Yeah. So we are now working with folks like Cursor, Harvey, Lovable, ElevenLabs. Swyx [00:01:05]: Yeah. Amazing. Congrats. Rune Kvist [00:01:06]: Thank you. Swyx [00:01:07]: So you were famously one of the first hires involved in GTM and product. I’m just kind of curious: what was your path into AI? Just recap. Rune’s Path Into AI: Scaling Laws, Capital, and Anthropic Rune Kvist [00:01:18]: Yeah. Rune Kvist [00:01:19]: Late 2021, I sold a company, my first company, an edtech company. I had a bit of time to think about what was next. I came across the Scaling Laws paper, and that just struck me like lightning. I was just like, “This is a big idea.” In short, the Scaling Laws paper just says the bigger the model, the smarter the model. Swyx [00:01:38]: So this is the Kaplan one, not the Chinchilla one? Rune Kvist [00:01:40]: Exactly, the Kaplan one. Swyx [00:01:42]: Yeah. Rune Kvist [00:01:42]: And the important thing that clicked for me there was, oh, now capital will understand this. If you put in more money, you get more money out, and so that will kick off a hype cycle. And so you get a sense of predictable returns, which is, in fact, what’s played out. And so I just packed my bags. I’d never been to San Francisco. I’d never been there. I just packed my bags, flew out here to find the people who had written it. And at the time, they had just started a small lab called Anthropic. There were around 40 people at the time or so. Drank a bunch of coffee until I eventually got introduced to Dario. And at the time, they were wrestling with some of these questions of, like, should we deploy our models? Should we make revenue? How should we engage with the rest of the world? They’d just broken off from OpenAI, and it’s been publicly reported that they were kind of concerned with how they were dealing with deployment. So they were wrestling with some of those questions. At this point, this is early fog of war, like early 2022. The hottest product at the time was, like, Jasper. Like, there’s nothing out there. So where value was going to accrue, and what the different parts of the stack were going to be, were all open questions. Swyx [00:02:48]: I want to highlight to people, you ask these questions because you have a PPE background. Rune Kvist [00:02:52]: Yes. Swyx [00:02:52]: I actually was in Singapore in one of the sort of feeder programs for prepping people for PPE. So I had a tutor. We learned, you know, philosophy and politics and economics. But, like, I think your kind of background matters. Machine learning people who read the neural, Scaling Laws paper would not necessarily draw the same conclusions that you did. Whereas any capitalist would read that and go, “Holy s**t.” Rune Kvist [00:03:19]: Correct. Swyx [00:03:20]: Right? Rune Kvist [00:03:21]: Yes. Swyx [00:03:21]: Who tipped you onto that paper? Because it’s not a paper that you normally read, right, like, in your circles? Rune Kvist [00:03:26]: Yeah. I think I’d actually, ever since AlphaGo, had some appreciation that AI was a big deal. Swyx [00:03:36]: Yeah. Rune Kvist [00:03:36]: But it kind of felt like it raised all these kind of interesting philosophical questions, but it was kind of not clear from afar where exactly that would go. But it was obvious enough that it was like, this is going to be a big thing if we find the kind of right mechanism to kind of get the techno-capital machine to work on this. But it was just not clear. And so I think there was some way in which, like, that became obvious, and also it wasn’t as obvious at the time than it is now, right? Like, it was just like, wow, this is so interesting. But it still felt, coming from kind of a philosophy and economics background, it felt like if this turns out to be true, you’re going to be wrestling with all of the big questions in society. Everything you’ve learned about politics gets thrown out of the window. Everything you’ve learned about economics at least gets challenged. And so what felt interesting was to be at that frontier that has ramifications across everything. So that’s why I sought it out. Swyx [00:04:32]: I mean, clearly really good insight. For people who don’t know, the PPE program is, like, where prime ministers are born. So then you end up meeting Dario. Rune Kvist [00:04:41]: Yep. First Dario, yeah. Swyx [00:04:43]: Yeah. Well, I mean, like, so did you get extra insights from talking with them that you didn’t get from your original hypothesis? Anthropic’s Early Conviction and the Scaling Laws Crystal Ball Rune Kvist [00:04:50]: If you read the Scaling Laws paper, you get this, like, very vague sketch of like, wow, this seems kind of important. There are some lines on a chart. This seems kind of important. And what I think the team at Anthropic had thought more about than anyone was like, what are the implications of this if you really play this out? And back then they had, kind of vision documents for what the world would look like in 2026, and they were kind of in vivid detail playing out how much compute is going to be needed, what the CapEx was going to look like, what some of the societal concerns were going to be, but also what is the amount of economic value coming out here? And so it kind of felt like they held a crystal ball that in hindsight turned out to just be dramatically correct. And they weren’t holding it like they were obviously correct. They were just like, “Take this hypothesis really seriously.” Swyx [00:05:38]: Think it through, yeah. Rune Kvist [00:05:38]: And think it through in the same way as the kind of situational awareness that is Swyx [00:05:43]: Across the street. Rune Kvist [00:05:44]: Across the street. Swyx [00:05:44]: Your office, yeah. Oh my God, we’re all living across the street in the same one square mile. Rune Kvist [00:05:50]: Correct. And that’s now a couple of years old, but also people keep referencing it these particular weeks with Fable and Mythos, and it’s like, wow, if you take this one idea seriously- For the Scaling Laws, a lot of things fall into place. Vibhu [00:06:03]: And keep in mind, at this point, this is the same team that did GPT-1, GPT-2, and GPT-3. Rune Kvist [00:06:08]: Correct. Vibhu [00:06:08]: Which is also, like, it’s not just some experimentation. Like, this is a real model that we just scaled up. Rune Kvist [00:06:14]: And they had deep conviction in this idea: if you take a big blob of compute and data, it just wants to learn, and out of that will come smarter and smarter models. And all the particulars were not clear. Vibhu [00:06:26]: Yeah. Rune Kvist [00:06:27]: And all the implications were not clear. But their deep conviction in this, like, core thesis, and that was kind of dizzying. It was both phenomenally interesting and exciting, and also very quickly you get to, like, the world we know today will no longer be if this hypothesis holds. So it also just felt, like, important in some kind of grand sense. Vibhu [00:06:48]: What kind of shaped you there? So that was early 2022. Not only had GPT-1, GPT-2, and GPT-3 come out, but, you know, the amazing founders of Anthropic that have never split up, the only ones, they actually had the conviction to leave OpenAI, start their lab. You said there were about 40 people there. What was the time like there? Inside Early Anthropic: Mission, Deployment, and Risk Rune Kvist [00:07:06]: It was kind of remarkably like what it looks like on the outside today. Extremely cohesive, extremely mission-oriented, and living in this tension between their two ideas, which is AI could both go really well and really bad, and we want to be part of building it. That creates astounding amounts of tension. And they were wrestling with this incentive challenge where they know they’re in a race that they’re in where you might get forced to cut corners, but it also felt very important to them to be at the forefront of technology. And all of those ideas were just present at that time. It kind of feels like that line has been just very clear, and I think kind of love them or hate them, they have really stuck to their guns. There’s a core set of beliefs that they hold more deeply than most companies hold any beliefs. Vibhu [00:07:58]: Yeah. Fast-forward to today. Rune Kvist [00:08:00]: Yeah. Vibhu [00:08:00]: What does that lead us to AI underwriting company? What are you up to? What motivated you to start this? From Waymo to AIUC: Confidence Infrastructure for AI Rune Kvist [00:08:05]: Yeah. AIUC builds confidence infrastructure for frontier AI through standards and insurance. The link from Anthropic to building confidence infrastructure, looking out the windows at Anthropic offices and seeing Waymos driving by. Already back then, early 2022, Waymos were in some ways like AGI for cars. Like, they were superhuman drivers, but you couldn’t take one to the airport. And now, four and a bit years later, you still can’t take your Waymo to the airport, despite now everyone having kind of looked at the evidence and being like, “They’re better drivers than humans.” So in that particular instance, what’s clear is that the binding constraint on AI being useful is not capability, but is that liability or risk or trust. That problem is, general. The reason why right now Rune Kvist [00:08:52]: Fable is not open for access is not because it’s not a good model, it’s because it’s a very good model. It’s just hard to make promises about what it will or will not do. And this problem gets worse as AI gets better. Basically, more intelligent AI can be more autonomous. That’s more valuable, but also the risk surface grows. And so - what Waymo illustrates is that unless you build the confidence infrastructure to make promises about AI, or at least bring light to the risks, you grind adoption to a halt. Governments, banks, hospitals, militaries need to have some sense of what AI will and will not do to be able to operate for them to incorporate it. And that’s the problem that we’re trying to solve. Now, why standards and insurance? If you trace this problem back through history, every technology wave has had some version of this problem. So if you go back to, like, year 1900, electricity comes Vibhu [00:09:47]: Ben Franklin. Rune Kvist [00:09:48]: Cars burn down, sorry, houses burn down, lots of people die. 1930s, cars are a big deal, kill lots of people. 1950s, private nuclear energy is a big deal, poses big risks. In each of those instances, the market runs ahead of regulation to create confidence infrastructure because that’s required to make go/go decisions. That is required for adoption, and the market fundamentally wants adoption. And in all of those instances, common blueprint emerges between standards and insurance. The reason these two components is standards kind of provide the rules of the road, and they also specify, like, what are the tests that need to be run so we can get a sense of how high the risk is. So take in the case of cars, that’s like a car crash. Great, everyone, they inform your insurance pricing today, they inform your purchasing decisions, et cetera. That’s basically the risk framework. The insurers are important because they pick up the bill. So they are the private institution that is most on the side of. That is best incentivized to quantify the risks truthfully and then figure out all the ways to reduce the risk ‘cause that increases their profit. So they’re basically, they help shape the incentives. And these two work really well in unison. Now, how does that show up as a company? Well, one of the things that was obvious even - or starting to become obvious even a couple years ago was that frontier companies, some of our customers today, like Cursor, Sierra, ElevenLabs, Harvey, were going to have a very easy time selling a pilot to a bank. The, like, the demo just sells itself. It’s magic. But bringing that through, if you want to do a wall-to-wall rollout at a bank or a hospital, you have to go through the risk process. These banks have no idea even which questions to ask, let alone which answers are sufficient, let alone, like, how do they go and test whether these agents actually work the way they’re supposed to. And so they had this problem of, like, what can we say to earn the trust? And we think there’s, like, a golden sentence that goes something like, “Hey, I hear you’re really worried about hallucinations or jailbreaks or whatever it may be. We’ve had an independent third party test us against the gold standard. We passed with flying colors. And as a vote of confidence, the world’s most conservative insurers have looked at the data.” And they’re willing to take some of the risk onto their balance sheet. Swyx [00:12:06]: Yeah. Rune Kvist [00:12:07]: So if something does go wrong Swyx [00:12:07]: There’s money behind it, yeah. Rune Kvist [00:12:09]: Exactly. So that’s kind of like the link between all this. We can get into some of the hard parts related to the technical testing, which is, I think, the crux of the matter, but I’ll pause there. Swyx [00:12:19]: How did you and Rajiv come together? This-- there’s always, like, you come across very confident and, you know, and we’re announcing your Series A and all these things, but I want to see, like, the early initial stages of, like, idea formation. Cofounding AIUC with Rajiv Dattani Rune Kvist [00:12:31]: Yeah. Rajiv is actually my soon-to-be brother-in-law. Swyx [00:12:35]: Oh. Rune Kvist [00:12:36]: So I’m actually, in a week and a half getting married to Rajiv’s sister. Swyx [00:12:42]: Okay, now you’re tight. Rune Kvist [00:12:44]: Exactly. Swyx [00:12:44]: Now you know. Rune Kvist [00:12:45]: So - Rajiv and I have known each other for a decade. Funny story, I met both Rajiv and his sister, Hena, at the same time when Hena and I were interns at McKinsey in London, and Rajiv was assigned as my mentor. And so met them at the same time. For the longest time, it was not obvious that we were necessarily going to work together. I was in startups. He was, an insurance partner at McKinsey. Three or four years ago, I think Hena convinced him that AI was going to be a really big thing. And so he quit his job, cushy partner job at McKinsey in London, packed his bags, flew to San Francisco, and ended up joining METR. You guys are probably online enough Swyx [00:13:24]: CEO. Rune Kvist [00:13:24]: Exactly. Swyx [00:13:24]: We’ve, we’ve, we’ve heard of METR. Rune Kvist [00:13:25]: You see the plot-- the chart of the horizons of the tasks that agents can take on is doubling extremely fast. So he was COO at METR, led their partnerships with Anthropic and OpenAI to test their models before release, but also working closely with the US and UK government, to figure out, like, how do you know whether a model can be released? And in some ways, that was, like, the perfect background. He’s spent a lot of time in insurance, knows that world, spent a lot of time with frontier testing of models. And so when I was bumbling around this idea space, starting with some of the ideas we talked about related to Waymo, as soon as we got into the content, we were both like, “Oh, this would be an amazing business to build together.” This is wrestling with the problem that we both think is the most important in the world from a market angle, which is kind of our intuitions is that the market can do a lot, and the faster AI moves, the harder it is for government to solve some of these problems. And then it took a little bit of time to work through what is it like to work with family. Swyx [00:14:27]: Sure. Rune Kvist [00:14:27]: And, Swyx [00:14:30]: Because you were already dating at the time Rune Kvist [00:14:31]: Yeah. Yeah, exactly. Swyx [00:14:33]: Yeah. Rune Kvist [00:14:34]: Already back then, it Swyx [00:14:35]: Yeah. Rune Kvist [00:14:35]: We felt like we were a family. Swyx [00:14:36]: Nice. Rune Kvist [00:14:36]: And so starting a business together felt like kind of a big step. And, here we are with just immense amounts of trust. Vibhu [00:14:43]: Yeah. So now you’re a company of how big? How big are you guys now? AIUC-1 Certification: Agent Security, Safety, and Reliability Rune Kvist [00:14:46]: There are just 20 of us now. Vibhu [00:14:47]: 20 of you guys now, have Series A, and you have your first certification out, the AIUC-1. Let’s bring up the certification. So this is the agent certification, right? What goes into the process? I have, like, two questions here. One is, walk us through the certification, and two is, what is the process for a company to get certified, you know? Rune Kvist [00:15:08]: Great. As it says right on the top, AIUC-1 is a standard for agent security, safety, and reliability. The fundamental design principle is take all of the concerns that slow down adoption, so all the questions, all the fears that keep, security leaders in the Fortune 1000 up at night, and put them into one comprehensive framework. That’s what you’ll see there. You can see the six categories. Two, you want to ground all of this in technical testing. So one of the concerns with security standards that often feel kind of like theater paperwork is that they’re not actually ground out in, does any of this work? Does any of this matter? And so we had a conviction from early on that was going to be the kind of crux, was to pass this, you must get tested every quarter, basically run thousands of simulations to see, well, so can it actually be jailbroken? How hard is it to jailbreak? How often does it hallucinate? How often does it leak data? Et cetera. And then the last, core idea here, if you scroll up to the top here, is to refresh it quarterly. Rune Kvist [00:16:08]: So the core trait of AI is that it moves extremely fast. Whatever concerns we’re discussing today were not the same ones three months ago, and this will keep changing. Typically, standards update on a, like, a decade cycle is obviously not going to work. But the question is kind of how do you update it? And the core thing here was to basically get the risk leaders of the Fortune 1000 around the table. So if you go over to the left here Vibhu [00:16:32]: Yeah Rune Kvist [00:16:32]: You’ll see the AIUC-1 consortium. The consortium is a group of risk leaders who run real banks, real hospitals, real critical infrastructure, who are facing these challenges every day. And we meet with these folks twice a quarter and hear what’s top of mind, what is keeping them up at night. There’s tremendous amount of desire for that conversation. And then we operationalize that into a specific standard that gets into. And actually, we can go into and look at what Vibhu [00:16:55]: Yeah Rune Kvist [00:16:55]: What even is the standard. So if we go back to introduction, out there to the left, scroll up a little bit to the wheel, click into reliability. So if you take something like hallucinations sits in reliability. There is a number of requirements here. If you go into the top one, prevent hallucinated outputs, hallucinate outputs, this is one particular requirement. This is a technical control. Basically, we want some kind of ground in this filter. The first thing you see here is what’s called a crosswalk. So everyone and their grandmother has put out a framework, very high-level framework for what are the AI risks. Swyx [00:17:27]: This is basically your competition, Rune Kvist [00:17:28]: In some ways our competition Swyx [00:17:29]: Not seriously, yeah. Rune Kvist [00:17:30]: We’re, in fact, friends with them. We’ll come back to why. Swyx [00:17:31]: Yeah. Rune Kvist [00:17:32]: But mapping everything together so you have one superset. The claim you’re trying to support here is, if you follow this framework, then you can also see how you follow the other frameworks. But the meat of it comes down here in control activities and evidence. So control activities is like, great, you have this high-level requirement. How do you turn that down to something operational? Here’s what you must do, and then what is the evidence that we’re looking for? Rune Kvist [00:17:57]: And the reason we go this deep is that there’s actually not that much confusion about what are the big concerns in AI. Everyone agrees to these. The question, like, what are you actually supposed to do? And so. What we found a lot of demand for is getting down to the specific evidence, that people need to look for. Whether you are Cursor building something or, even JPMorgan building something, but also if you’re just a risk leader at JPMorgan, like what exactly should you ask for? What can you ask for without sounding stupid? Like if you ask for some-- you won’t believe the amount of time a risk leader has asked for the IP rights to the underlying model to Cursor or something, and you’re just like “Sorry, what?” Like, Swyx [00:18:39]: You slip it in there and you see Rune Kvist [00:18:40]: Slip Swyx [00:18:40]: See if you notice. Rune Kvist [00:18:41]: See if they. Exactly. Swyx [00:18:42]: Yeah. Rune Kvist [00:18:42]: Put that in the questionnaire. All right, so that’s kind of what our standard is, and we update this every quarter with these folks, to keep up with the latest concerns. Swyx [00:18:51]: Can I double-click on this one? Controls, Evidence, and Third-Party Testing Rune Kvist [00:18:52]: Yeah. Swyx [00:18:52]: So first of all, the website’s beautiful. Like, it’s so confidence-inducing which is the whole point where, like, okay, I know exactly what I’m signing up for when I talk with you. Like, I don’t even have to talk to you. I can just see your whole, certification, which is great. But, like, okay, so from here, like D001.1 configure a groundedness filter, how does that get applied? Like, you have a person that Rune Kvist [00:19:16]: Yeah, Swyx [00:19:16]: Goes through it? Rune Kvist [00:19:17]: If you, go back Vibhu [00:19:19]: I did see somewhere there’s like, you know, fifty-one requirements, a hundred thirty controls. There’s like a whole Swyx [00:19:25]: Right. I just want to. Like, to me, this doesn’t translate Vibhu [00:19:27]: Yeah. Swyx [00:19:27]: Into a test or an eval. Rune Kvist [00:19:28]: Yes. So if you go into, on the left-hand side. So actually, if - before we go in there are three types of requirements. The first is technical controls, like you must implement some guardrails. Rune Kvist [00:19:42]: Two, there are test controls. So you must have an independent third party go and run some tests against you. I’ll show you one of those in a second. And then three, there are policy controls. For example, you must have a person whose name is on the line when you guys f**k up, and you must have a plan for how you tell your customers and how you engage with them. They’re kind of more traditional, standard type stuff. So in this particular instance, we just check whether they in fact have a ground in filter. So we will partner with an auditor. So we partner with auditors like KPMG or like Schellman who go in and do the thing auditors do, which is to check the evidence. In this case, that might be a screenshot, it might be part of the code that they need to review to see that it actually. Just that it exists. Swyx [00:20:21]: Oh, okay. Rune Kvist [00:20:22]: And then the second thing Swyx [00:20:22]: So you’re not testing the effectiveness of it. Rune Kvist [00:20:24]: That’s the second thing. So if you go down Swyx [00:20:25]: Yeah. Rune Kvist [00:20:25]: To the third-party testing for hallucinations out on the left, that’s basically the next requirement. This is where we test how well does it actually work. Swyx [00:20:32]: Okay, and is it you testing or the auditor? Rune Kvist [00:20:34]: We test them. Rune Kvist [00:20:35]: We test them. Swyx [00:20:36]: That’s a lot of work. Vibhu [00:20:37]: How long does testing take? So if I want to get certified, just Certification Timelines, Remediation, and Quarterly Updates Rune Kvist [00:20:40]: Yeah. Vibhu [00:20:40]: How long does the end roughly take? Rune Kvist [00:20:42]: Yeah, the end, almost always is dependent on, like, our customers need Vibhu [00:20:47]: Yeah. Rune Kvist [00:20:47]: To look something for us. It takes somewhere between, like, 3 to 10 weeks Swyx [00:20:52]: Yeah. Rune Kvist [00:20:52]: Depending on how up to snuff they already are. So some people show up to us with, like, extremely rigorous security programs. When we test them, it works extremely well. We can get that done very quick. Some people come to us, and they’re not that far along. We give them kind of the spec that they need to build towards, and then their security teams and engineers get to work and build to meet the standard. The testing itself typically takes a couple of weeks, including the time for them to remediate. Often, we’ll find something that we cannot pass, where this is actually just not up to the standard. - you won’t pass the standard. And then they will need to go and implement additional safeguards or additional remediation that makes them more robust so that they can actually kind of hand on heart look at their customers in the eyes and say, like, “Hey, we’ve done truly our very best.” Vibhu [00:21:35]: And they’re certified for a year and have quarterly updates? Rune Kvist [00:21:38]: Correct, yeah. Vibhu [00:21:39]: And, yeah, it’s pretty interesting. I think, you know, what’s changed since. So this is certifying agents in production, right? Your customers, like you’ve had Lovable, ElevenLabs, Intercom, and they’ve all gone through this certification. Rune Kvist [00:21:50]: Yes. Vibhu [00:21:51]: What has changed? So I see you post, like, you know, Q2 added MCP agent, How Agent Risks Are Changing: Coding, MCP, and Agent-to-Agent Interactions Rune Kvist [00:21:56]: Yeah. Vibhu [00:21:56]: agent communication. Any other things that you want to kind of highlight since the first iteration? What comes in quarterly? Rune Kvist [00:22:03]: Yeah. So some of the changes have just been agents are not just one thing. So, like, if you take agents like Cursor and compare them to Sierra, they’re really quite different. And compare them to Harvey again, compare them to you out of again Swyx [00:22:16]: ElevenLabs, yeah. Rune Kvist [00:22:17]: ElevenLabs, they’re all quite different. And so we wanted to design a standard that works for all of the types of agents. And we started with one that was, like, pretty text-based, like, honestly, pretty customer support-focused. That’s where there’s a lot of existing demand. And then over time, we’ve picked, some of the frontier companies in each of these other domains that we could work with and build out the standard, so, such that we know that the same standard works for code, it works for customer support, works for automation, et cetera. So that’s been one big thing. Yeah, then some of the things that have been top of mind recently, Mythos is bringing up a lot of concerns for security leaders. We’re starting to get more and more questions around agent interactions. It’s very nascent, at the moment, but it’s starting to emerge. There’ve been a lot of, questions related to OpenClaw and MCP. Again, like agents starting to interact with each other, is really top of mind. Then as coding agents have really taken off, that’s also where banks and hospitals, et cetera, are getting more and more precise on what it is they need. So really dialing in as that start to be, like, where most of the tokens flow through in the world, getting much sharper on that. Vibhu [00:23:26]: Can you share for people that are listening that don’t really think about this? Like you mentioned, there’s the obvious stuff, you know, hallucination, citations. What are best practices that people should do when building agents? Like, if they come to you pretty ready with certification like, you know, they’ll probably pass certification. What are the things people don’t think about that they should have? Best Practices for Agent Builders: Stress Tests and Guardrails Rune Kvist [00:23:46]: The most important thing is that a lot of companies have not done a serious stress test. They spend most of the time, perhaps rightly so, optimizing for how does it work in the good case, the average case, how high-quality is the output for the customer. And a lot of these companies are pretty new, so they haven’t spent a lot of time stress testing the what is there as an adversary on the other side? What are some of the complicated corner cases that you’ve not really considered? So I think that’s, like, a frame of mind. And you’ll also see this in startups. It often takes a while until they hire their first security person. They- And that’s a whole different kind of risk surface than just building a good product. So a lot of that applies. Most companies actually also have the right kind of architecture. Most of them will have some kind of guardrails in place, either some that come out of the box from their model provider or they’ll have built their own filters that sit in between. They just don’t work very well. The difference between putting a classifier in place that, like, maybe goes and checks whether you’re giving medical advice when you shouldn’t and says, “Hey, if this looks like medical advice, filter it out.” Lots of companies have that in place. The question is whether it works. And it’s actually pretty fiddly to sit down and think about all the ways in which you could ask for medical advice, read the academic literature on what are the kinds of Rune Kvist [00:25:03]: Framings or tricks you might play to get an AI to give you medical advice when you really shouldn’t. And so there’s, like, an area of expertise that’s just missing. So what we find is that most people have the right building blocks in place. They don’- It doesn’- It’s not rocket science, but the finicky thing is, like, getting into the corners and testing whether it works such that you can look your customers in the eye, or maybe a bank or maybe a hospital and be like, “This is going to work for you.” Vibhu [00:25:26]: I see. So we talked a lot about the agent-level certification. Where do you guys go from here? So announcing series A camera, we talked about this a bit. There’s the whole security risk of Fable, government stepping in. You guys are kind of announcing that you’re also going into model certification? Toward Model Certification: The Government–Lab Trust Gap Rune Kvist [00:25:46]: When we do a bit of cutting afterwards, Vibhu [00:25:48]: Yeah Rune Kvist [00:25:48]: We will not yet be announcing this, Vibhu [00:25:49]: Nice Rune Kvist [00:25:50]: The question that is top of everyone’s minds now is at the model level. And Mythos, then Fable, has really brought this to the fore that in addition to the commercial risk and the kind of economic security risks that are happening at the agent layer, the models are going to present risk in the national security category. The shape of the problem is very similar. You have some people that are on the hook if something goes wrong. In the case of agents, it’s often security leaders in the enterprise. In this case, it’s the government. They don’- haven’t necessarily spent their entire lives thinking about what are the new risks that come here, what is the kind of data you might be looking for, how might you test that? But they do have to make sure that their concerns are addressed. You have some frontier AI companies that are deeply technical. They know a lot about the risks, but they fundamentally have an incentive to not always be truthful. So you have a trust gap between the government and the labs. And in every other industry, you end up with some kind of body sitting between, a neutral third party sitting between those people. There’s no other industry where you allow people to audit themselves. So there is going to be a need for a third party that can take the rigor of the labs to run frontier technical evals, but can also speak legible trust in the way that the government trusts PwC to go and run financial audits. And they know that they output audit reports in a way that’s consistent, that’s easy to read, that’s factual, that’s, trustworthy. Those two things need to be brought together. And what we’ve learned from our work with agents is that if you want those-- that communication between those two parties to be smooth, there has to be one common standard that is public, that people can go and inspect. What are the risks that matter? Within each of these risks, what are the kinds of threat models that you’re really looking for? You need to specify for each of those risks, what are the guardrails that need to be in place, and what are the tests they need to run to see whether those guardrails are effective? And then you need to go and run audits that are - technical audits that are consistent. So if you’re trying to bring trust, it’s extremely important that you methodically work your way through the risks. You can’t send one researcher in and say, like, “Come back with whatever you find.” You need to be able to explain exactly what you did, exactly what you tried, exactly what you did not try, and therefore the kinds of promises you can and cannot make at the end of it. I think of Neutral Third Parties, CAISI, and Model Risk Audits Rune Kvist [00:28:13]: Fable as a direct symptom of this problem that the government was told that there’s a risk. The government may struggle to assess just how big that risk is. They call Anthropic, and Anthropic is trying to tell them, “Hey, actually, every model can be jailbroken.” Swyx [00:28:28]: That’s not what you want to hear, right? Rune Kvist [00:28:32]: As the government, that might be hard to trust. Rune Kvist [00:28:36]: And we think that a broker is the most natural solution. In other markets, you see something like, in financial markets, you see Moody’s. Moody’s goes in, and they look at a bond, and they output a rating. They say like, “Here’s the evidence we found. Here’s the rating.” We don’t decide whether anyone should buy this bond or not buy this bond. Well, that depends on their risk appetite. But we do provide this common information layer that everyone can rely on. In the case of Moody’s, the government, points to them and say, “Hey, pension funds, you should probably really take care. You shouldn’t risk your pensioners’ money, so you can only invest in triple-A rated bonds.” That means that now the government doesn’t have to staff thousands of financial technical experts to rerun forecasts every week to see whether things are correctly rated. They get to point to some neutral third party. So my hypothesis is, my hunch is that you will see a third party that sits between the government and the labs, and it could either be the government builds it themselves. So something like CAISI was set up to do exactly this. And the question Swyx [00:29:44]: Sorry, I’m not familiar with CAISI. Rune Kvist [00:29:45]: CAISI is the Center for AI Standards and Innovation. Swyx [00:29:49]: Okay. Rune Kvist [00:29:50]: I won’t get into the details, but it’s a body of NIST that typically sets standards. So it’s basically a government body that has AI experts. Yeah, exactly. Exactly. Swyx [00:29:59]: Very key. Very key. Rune Kvist [00:30:00]: Very key. Vibhu [00:30:00]: I think, you know, it’s one of those things where when you just sit back and listen-- look at it, like, is there enough technical expertise in the government to measure, test these things right now? Probably not, right? And Fable is a result of, okay, we’ve had to scale back and pause things, Rune Kvist [00:30:17]: Yeah. And they have excellent people, but they have an extraordinarily small budget compared to the scale of the challenge that’s ahead of us. And I think they have a role to play. The question is kind of like, who does what? We have now outlined the jobs to be done, and they’re quite extensive. Every model release, there is an astounding-- Given that they take in any input, their risk surface is astounding. And so the question is really: what can only the government do, and what can the market provide here that can keep up with the pace as AI risk changes? Our perspective is that also at the model layer, the risks that people care about today are not the same ones they cared about three months ago. So the pace of legislation is too slow to deal with pinpointing the risks here. And so we think there’s a lot that the market can do to surface timely information. Ultimately, there is a bunch of policy decisions here. Is the national security risks of a model too high? Swyx [00:31:12]: Yeah. Rune Kvist [00:31:12]: That’s a political answer. But what we want to make sure is that the process that produces this risk information is compatible with very fast innovation. So you don’t want to. This is not a question of like, can you slow the things down? Can you keep, the models locked up until-- for months on end until everyone can make a guarantee? But it is this, can you, in the time it. Given that the US is competing with China on releasing models, can you insert risk information that allows the government to, like, make rapid decisions on some of these questions? Balancing that trade-off between failing to adopt AI is going to put us at risk, but also reckless adoption is going to put us at risk. And that’s a very kind of fine balance that they’re going to need, like, a lot of high-quality intelligence to make. Chinese Models, Data Flows, and National Security Concerns Swyx [00:31:55]: Just a side mention, because you mentioned Chinese models, any specific concerns that you’re hearing from your CISOs about that? ‘cause I guess it’s free, but. Rune Kvist [00:32:05]: CISOs have a bunch of concerns around data flows in general that they’re really concerned about. So there’s a lot of questions like, if these models are Chinese, where does that, where does that data go? I think a lot of this can be addressed, but they come up often. Swyx [00:32:18]: I mean, they understand they’re running on American GPUs. Rune Kvist [00:32:21]: Some of them, some of them understand that they’re running on American GPUs. Swyx [00:32:23]: They’re not, like, phoning home every time you, like, call home. Rune Kvist [00:32:26]: No. A year ago, there was not a lot of understanding of this. I actually think, you’re seeing the security leaders becoming kind of AI literate at a blistering pace, and you’re actually also seeing my Twitter timeline that’s very pilled and my LinkedIn feed that used to not at all be pilled kind of converge. They’re both talking about Fable. Swyx [00:32:45]: Right. Yeah, that’s true. Rune Kvist [00:32:46]: They are both talking about whether you can prevent models from being jailbroken these days. Swyx [00:32:51]: Yeah. Rune Kvist [00:32:52]: Like national security national security risks are now the conversation that is actually emerging. Other than that, I think you mostly see a kind of general picture: there are no concerns with any particular model or any particular model output, but there is a general nervousness of having critical infrastructure run on models that are not produced in America by Americans where the American government has control. Swyx [00:33:14]: But it doesn’t necessarily show up in your framework that directly, or it might, I don’t know. Rune Kvist [00:33:18]: There’s a bit of stuff in there actually on the, like, the provenance of the models and disclosing that. But I think there’s a bunch of use cases where running a Chinese open-source model is just the best solution. Swyx [00:33:27]: Yeah. Rune Kvist [00:33:27]: And a concern is slightly more macro here, which is not best addressed at any particular certification level. Vibhu [00:33:32]: Is there anything interesting that you see at the. You know, if you’re trying to fill that middle gap, that mediation gap, any interesting stuff that you guys forecast would be required other than, you know, what the average person might expect? Cyber, Child Safety, Bio Risk, and Expert Coordination Rune Kvist [00:33:47]: There’s a bunch of interesting questions about what are the risks that matter here. So right now, the risk of the day is cyber, because it’s very real, very tangible. And some of the risks that are also emerging as pretty real and pretty tangible are things like child safety is becoming both extremely important, but also politically important. And then there are some of the risks that are coming down the pipeline that today feel kind of speculative, but people who spend a lot of time with the models see them coming down is things like, risks that relate to biology. Rune Kvist [00:34:18]: And specifically whether models will help adversaries produce biological weapons and making that extremely cheap, extremely accessible, producing-- making the chance of another COVID or worse pandemic. COVID was not engineered to be bad, as if you were trying to do that. So I think those are some of the risks that are coming down the pipeline. I think one other thing to just note is that agents are kind of deliberately narrow. So, like, when a frontier agent company puts a chatbot that interacts with customers, they’ve really tried to narrow the topics it’s interested in talking about. Such that if you ask it, like, “What do you think of the president?” it will just decline, which means that the kind of risk area is somewhat smaller. For models, it is infinite. And so there’s not a single expert out there who can competently evaluate the risks of cyberattacks and fifteen-year-olds having month-long conversations with a chatbot and seeing whether it will in fact recommend suicide or something horrendous like that, and can evaluate the risks that terrorists can use AI to produce bioweapons. The risk surface is just too big. And so the central challenge actually becomes how do you get those subject matter experts to work within a one coherent framework that outputs one coherent report and rating that the world can go and inspect? ‘Cause that global perspective is central, but there’s not a single organization today that could produce that. Swyx [00:35:47]: And you would be the presumptive one when you put out your model standards. Rune Kvist [00:35:51]: We think there can be one company that can, with a consortium of experts, build one coherent standard. I think we’ve shown that across all of the enterprise risks today. We think it could be one company that could, with a consortium, specify the audit rules, basically like the inputs and outputs that all these technical experts need. What access do they need? How should they treat infosec- info security? They can look at whether the eval- evals are well-produced without necessarily being able to say, “Hey, is this a threat or not a threat?” But overall, evaluating whether the evals are good, well-constructed, that set of audit rules that basically becomes the interface for all these experts, we think one clearinghouse could put together. To be clear. When I say one company, I think of it as one company coordinating lots of this in the same way that when we saw our consortium, it’s not like we say we have all the answers on agent security. What we say is we are taking on the role of eliciting all of the concerns and being the secretary that puts it together and runs a tight house such that the standard updates lockstep every quarter, and that the audit reports that come out, in this case, 100-page audit reports, uniform and crisp and clear all to the level of detail that is required for executives that need to make a clear go/go decision. So that’s kind of the role that we think we might play. OWASP, Frameworks, and the Operational Audit Layer Swyx [00:37:11]: I think in many ways you’re performing the role that OWASP used to do there, and you said, like, you know, competition and partners. Rune Kvist [00:37:18]: Yeah. Swyx [00:37:19]: Can you go more into, like, how they partner? Rune Kvist [00:37:20]: Yeah. So first of all, OWASP is basically an open source community of security practitioners that are coming together to build frameworks for addressing the latest security concerns. We think they are phenomenal at creating frameworks. We’- In fact, we’- First of all, we’re partners with them, so we have a joint article. Two, we’ve learned a lot from them. We think they’re a tremendous source of intelligence. What OWASP does not do is building the machine that runs third-party audits such that a company like Cursor or a company like JPMorgan could get a third party to go and review them against this and say, “Hey, you’ve passed the standard, and here is the report that you can use to build trust and preempt your partners’ or customers’ questions.” So they fundamentally try to do something different. You - They are part of the information gathering and intelligence gathering and creating clarity, but the operational layer of turning this into promises is not the business they try to be in. Swyx [00:38:14]: The standard is emerging and is doing very well. Was it necessary to then also do underwriting? Obviously it’s in the name, so please remember you thought about it first. I feel like if you just have enough consensus, you don’t actually need the money angle, but it does help. Vibhu [00:38:30]: I did want to also note, you guys are a profit company too, right? It’s not profit where there’s a whole business side to it as well? Why For-Profit Standards and Insurers Matter Rune Kvist [00:38:39]: Yeah. Yeah, so I’m just getting crazy Swyx [00:38:41]: I think about the money part. Rune Kvist [00:38:42]: Yeah. Yeah, let’s get into the money part. Let’s start from actually your question, profit versus profit. In the security space today, cybersecurity, most of the standards are produced by nonprofits. I think that’s an issue. Rune Kvist [00:39:00]: The question you have to ask yourself is, how do you create good incentives for these standards to be good and keep up? Rune Kvist [00:39:09]: Nonprofits tend to not have these adverse profit incentives where they, hollow out their standard and create a race to the bottom, but they’re also not at all responsive by default to the communities that they serve. There’s no process-- They don’t have customers that they serve where they go and ask, “What do you want? What do you want? What do you want?” And when you look at the overall satisfaction with the security standards today, people tend to just not like them very much. You do see in other domains, that profit standards can serve the world quite well. So there are examples, like we talked about Moody’s before. It’s not without flaws, but, it is absolutely critical societal infrastructure that gets run at an astounding scale today. Your credit score, it’s FICO. It’s also a profit business. And when you go back even further in history, some of the crash testing standards came out of insurance companies. Rune Kvist [00:40:06]: The insurance companies together founded the Insurance Institute for Highway Safety because they were very interested in, like, how can we use standards to drive down mortality and save money? Go back, prior-- Our name actually pays homage to the Underwriters Laboratories, UL, which, was started right around when electricity came out. Houses started burning down. Insurers, again, were paying the bill, and they were maybe also good people, but their profit incentive was, let’s prevent houses from burning down. Let’s test all the electrical products, the light bulbs. All the light bulbs in here are probably tested, the toasters, et cetera. And they set up, an entity to create those standards. Today, UL has a profit entity and a profit entity. What they’ve recognized, they spun - They started profit. They spun out a profit because what they recognized was like, hey, actually to serve customers well, you need a profit entity. The lesson here is one of the ways that the market can align incentives so you’re both responsive to customers Rune Kvist [00:41:07]: And not hollowing out your standard over time is to align it with insurers because they fundamentally have good incentives. And so if you’re a profit standard that works closely with insurers, you get the feedback loop in such that you’re really tuned into your customers, but also have their interest at heart. So that’s the model that we - the kind of inspirational model that we’ve learned a lot from, and that’s also where the name comes from. In some ways, the term underwriting can both be associated with insurance, but it’s also a broad term for, like, making decisions. Rune Kvist [00:41:40]: If you underwrite a decision, you’re fundamentally kind of taking ownership for the consequences of it. AI Insurance Contracts, Lloyd’s of London, and ElevenLabs Swyx [00:41:45]: Yeah, I mean, what does an insurance contract look like for AI? Rune Kvist [00:41:49]: Yeah. Most of the demand comes today for insurance contracts is, sitting between people who’ve built AI and people who are buying AI. Swyx [00:41:56]: Yes. Rune Kvist [00:41:57]: And what you want—the reason why people want insurers involved, both for the traditional reasons, hey, if something goes wrong, we want to be compensated, but it’s in particular because insurers can bring trust to the equation. Because insurers will pay for the damages, if they’re willing to write an insurance policy, that is them saying, “Hey, we think there is risk here, but that is manageable.” And that is kind of a. Their incentive aligns with the enterprises adopting it, so that’s a really a good signal to the market. In the same way, actually, one of the things that Waymo tried to get their first permit to even operate in San Francisco was to get a lot of insurers to stack up a huge insurance policy. In the case if something went wrong, not because Google can’t pay, but because it was very valuable to have a third party go and look at that data Rune Kvist [00:42:47]: That are trusted by governments, trusted by enterprises as conservative people and say, “Hey, we’ve looked at it. We’re actually willing to take some of this on our balance sheet.” So that’s, that’s kind of the reason why people are interested in it. What it looks like is, in some ways like every other insurance contract. You specify what are the perils you want to cover, how much do you want to cover them, like up to what limits, and what does it cost to cover that. And in the case of, if we take a really concrete example, ElevenLabs, bought a first of its kind AI agent insurance policy. They work with some of the biggest, enterprises that work with governments. They’re really interested in going above and beyond and making promises to their customers. So they wrote a policy that covers just some of the core concerns that their customers have been asking about. And, the crucial thing was really to get Lloyd’s of London, the world’s oldest insurer, one of our partners, to look at this data and be that third party alongside us to say, “Hey, we think there’s something here that’s worth underwriting.” and that’s actually what it looks like. And so they will show that contract to their customers, and they can see how much they’re covered for. They can see what exactly it covers, and that will also probably change next year. They will want to write an insurance policy that might cover more. Swyx [00:44:04]: When you say Lloyd’s, is it reinsurance, or are they sharing somehow at the same level or Rune Kvist [00:44:11]: Yeah. So typically, the way, new companies get into insurance is that they partner with insurers such that the insurers take the majority or all of the financial risks. Fundamentally, if insurance is useful, because it brings trust, you have to be able to pay the bill. Lloyd’s of London is 400 years old. They’ve never not paid a claim. They’re extremely trusted. What Lloyd’s of London struggle to do on their own is to figure out which of the risks are real, what should we be looking for, what are the kinds of technical controls, and running the tests. So they use AIUC-1 as kind of the underwriting framework, and we produce a bunch of eval results that then directly feed in to inform the pricing. So this means that ElevenLabs customers know that payment will be there. They don’t have to look to our series A and see, like, do we think they have enough cash on the balance sheet? They will look at Lloyd’s. Swyx [00:45:05]: Yeah. Rune Kvist [00:45:05]: Yeah. Swyx [00:45:05]: And Lloyd’s, like, famously very creative. I think I remember some headline like, they insured Jennifer Lopez’s, butt or something. Rune Kvist [00:45:13]: Correct. Swyx [00:45:13]: Right? Rune Kvist [00:45:13]: And I think, was it, David Beckham’s right foot? Swyx [00:45:16]: So, yeah. Right? Rune Kvist [00:45:17]: And stuff like this. Swyx [00:45:18]: So, like, clearly not a large data set. Rune Kvist [00:45:22]: Exactly. It’s actually a remarkable institution that’s both kind of has some of the truly school virtues of having been around for a long time. They, like, really. They really operate like a trusted entity, and they have appetite to figure out the future. And I think there’s a lot of recognition that both there is, like, tremendous amount of risk in AI that is poorly understood today, so getting into this business carries real risks. But also this is where lots of the risk exposure will happen in the future. This is the one market where risk is truly growing. This is the one market that will also take out some of the existing markets. Take, like, auto insurance. When there are no human drivers, how’s that market going to look? Well, it’s clearly going to change. How are you going to assess Swyx [00:46:08]: You want to insure Waymo? Rune Kvist [00:46:10]: I. All I’ll say is the principles for how you insure Waymo are very similar to how you insure other kinds of AI. Swyx [00:46:15]: Right. Rune Kvist [00:46:15]: So again, crash testing, that’s what we do for customer share at Lovable. That will also need to happen for Waymo, which is not how you do it for human drivers. So there’s this growing awareness that the world is changing very fast, and the only way to learn how to underwrite AI is to write some policies. You may incur some losses and think of that as R&D expense, really. But the question for them is, like, who are the trustedtechnical partners they can get into this business with that can help them navigate and make sure they don’t make, kind of foolish mistakes? But also who is willing to hear the wisdom that they have? They’ve done this before. They’ve seen it was. They were there when cyber came out. So there are lots of ways in which AI feels completely new, but there’s also lots of ways in which risks look the same. And so there’s actually a tremendous amount of wisdom sitting in some folks that may have gray hair, but really have, like, a keen sense of, how to quantify risk. Swyx [00:47:08]: Yeah. And the number is. So it’s basically like I want fifty million dollars worth of coverage against these perils, and Lloyd’s will give you a quote on it, and then you have, like, a small markup or something, and then you turn it around and do that? Is that as simple as it is? Risk Capital, Premiums, and Working with Insurers Rune Kvist [00:47:23]: You basically share some of that premium. Swyx [00:47:25]: Yeah. Rune Kvist [00:47:25]: X percent goes to the people who do the pricing of it. Swyx [00:47:28]: You’re. It’s kind of like a. It’s kind of like a merchant bank for insurance type of thing. Rune Kvist [00:47:33]: Exactly. You basically split the fee, and you can think of the insurance supply chain as, like, there’s bringing the capital, there is doing the pricing, and there is doing the distribution. And typically, you will pay out some X percent of premium here, Y percent of premium here, and the rest of it will go here. Swyx [00:47:46]: Does all the insurance world work like this, or is there some point at which, like. So if right now you have equity capital Rune Kvist [00:47:51]: Yeah. Swyx [00:47:52]: At some point, maybe you start raising, debt or whatever, and then you have enough of a bank account and enough history, let’s say you’ve been in operation for ten years Rune Kvist [00:48:00]: Correct. Swyx [00:48:00]: That you don’t need Lloyd’s anymore? Rune Kvist [00:48:02]: That’s totally an option. And I could see some worlds where that makes sense, specifically if there are risks that we feel high confidence that we’d want to insure where the incumbent insurers are too slow to find appetite Swyx [00:48:13]: Okay. Rune Kvist [00:48:13]: Or simply struggle to evaluate it such that they don’t want to do it. But by and large, in general, you do not want to compete with insurers on, bringing risk capital to the game for two reasons. One is that’s fundamentally a cost of capital game. They have extremely low cost of capital. Startups have high cost of capital, by and large. And two, you want to hedge your bets, and it’s very helpful then to also have a portfolio of home insurance, of car insurance. And we’re not about to become a car insurer nor a home insurer. Rune Kvist [00:48:43]: So they have some natural advantages, which makes it much more likely that we’ll partner. Swyx [00:48:48]: Yeah. Rune Kvist [00:48:48]: And they bring that, the capital at scale, and we bring the technical expertise. Swyx [00:48:51]: You’re, you’re going to work with them for a long time. Vibhu [00:48:52]: How are the discussions with the insurers as well? So basically, they’re going off of your certification, right? They’re trusting the diligence on you that your certification is valid, you tested the right things, and they’re backing the money that, you know, you have the right testing in place. So any interesting takeaways from working with insurers? Rune Kvist [00:49:12]: I think the maybe the first thing is they feed into the standard as well. So if there are things that they feel like they need that they’re not seeing, we are also taking that as input into the standard, because fundamentally we think a good standard is one that creates a really healthy promise ecosystem, and we think insurers are a critical part of that. And again, they are the most well-incentivized to. They see all the lost data across every. Any particular CISO knows their particular concerns. Insurers see the concerns across the entire portfolio and often have direct access to, like, what exactly happened, who was at fault, et cetera, as they do part of their forensics. So they’re actually, like, a great source of intelligence on this. One of the big takeaways from cyber insurance, which is a market that didn’t work that well, was that the insurance and the technical expertise was not married up. What our conviction is that standards have to precede insurance. Fundamentally, what everyone first and foremost want, whether you’re a CISO at JPMorgan or a CISO at Cursor or an underwriter at Lloyd’s of London syndicate, is you want to not have an incident Rune Kvist [00:50:19]: In the first place. You want to know that the risk is well-managed, and only then does insurance start to make sense. So we’ll see the standard ecosystem basically run ahead of the insurance. And the reason why we. You asked us kind of why I also do insurance, this is kind of proving what we think a whole promise confidence infrastructure ecosystem needs to look like, and we think it’s very compelling to bring that to life, even if we think the standard is kind of the core linchpin that unlocks the rest. Claims, Liability, Air Canada, and Duty of Care Swyx [00:50:44]: There’s been no claims yet, right? Rune Kvist [00:50:45]: Nope. Swyx [00:50:46]: This is one of those things where, you know, if people haven’t really worked through what it means to cover things. Rune Kvist [00:50:52]: Yeah. Swyx [00:50:52]: So for example, I pay Cursor $20 a month. Rune Kvist [00:50:55]: Yep. Swyx [00:50:56]: And I write a vibe code something that makes, a plane crash, causing $200 million worth of damage. Rune Kvist [00:51:02]: Yes. Swyx [00:51:02]: Do I claim $20 or do I claim two hundred million? Rune Kvist [00:51:07]: Yeah. So these are all great questions. Rune Kvist [00:51:10]: And fortunately, kind of all of insurance and legal history kind of helps answer some of those questions. I think the first thing is people have limits on their policy. So if you want to claim $200 million, you have to. Someone has to have paid a lot for that insurance policy upfront to have $200 million of coverage. And ultimately, the way this works is that, you start from a lot of uncertainty. This is not just an insurance, but also, like, can you use. Can Anthropic use books on the internet to train up? Well, they can go and look at precedent, they can But ultimately, this- these things get settled in court, and you hammer it out over time. So you start from this, like, place of ambiguity, which is both why insurance can be hard to do early on, but it’s also why people want insurance, because that ambiguity slows down adoption. Swyx [00:51:57]: Yeah. Rune Kvist [00:51:57]: That also sits at the heads of the, Swyx [00:51:59]: Yeah. In some ways, actually, the first incident will help to, establish a lot of this. Rune Kvist [00:52:05]: Exactly. And there have been a number of incidents out there that have just not been covered by insurance. Swyx [00:52:09]: Yes. Rune Kvist [00:52:09]: Take the now old, example from Air Canada, where Swyx [00:52:14]: I was going to bring that up Rune Kvist [00:52:15]: Chatbot hallucinated a refund policy, and the question was, Air Canada in that case were like, “Hey, we have nothing to do with this. This chatbot messed up, but, like, sorry.” And the courts were like, “No, if you put your chatbots to interact with your customers, they make legally binding promises on your behalf.” That is now precedent for everything in the future where you will. If someone were to deploy a chatbot like that again, they should not expect to be able to just pawn off and say, “Sorry, my chatbot lied. It’s nothing to do with me. I bought it from OpenAI.” No, if you’re putting this in front of your customers, you are taking responsibility for it. And so every court case, whether insurance is involved or not, clarifies liability, and liability is kind of the foundation for insurance. There’s another reason why standards and insurance come together. Liability for. I’ll go on a little tangent here Swyx [00:53:06]: Please Rune Kvist [00:53:06]: Get into the weeds of it. Swyx [00:53:06]: Please. Rune Kvist [00:53:07]: Liability, often one of the core concepts is whether someone was negligent. Should they have seen this? Should they have prevented this? And the question is: how do you judge that? Well, you basically judge whether they’ve met their duty of care. What does that mean in practice? Well, often they look to standards. So if there’s a standard that is broadly adopted that says you must have a groundedness filter or you must have a jailbreak filter, it becomes way harder to claim ignorance that these things existed. And so setting standards help clarify liability. Coins-- courts will often point to standards and being like, “Well, this seems like best practice to do.” It’s there for everyone to see. So there’s another way in which, like, standards are kind of civilization infrastructure that insurance can then build on, which promises can then build on. Swyx [00:53:53]: I totally get that. We don’t have to get certified to write these, to, you know, make these, like, bots and all these. Rune Kvist [00:54:00]: Correct. Swyx [00:54:00]: But, like, basically, whenever we get. Go for the audit, I think people, like, start to shape up and all this stuff. I wonder if, like, that means that you don’t also then become, like, the approving authority for me to ship to production. You know, like, yes, you check once per quarter. I want to ship once a day. Shipping to Production: Ongoing Testing and Trust Rune Kvist [00:54:19]: Yeah. Swyx [00:54:19]: And I don’t know when one of my things breaks, like one of your certifications or not. Rune Kvist [00:54:24]: So there’s a couple things. There’s a couple of requirements in there that relate to how do you yourself, where you have to tell your customers Swyx [00:54:33]: It’s like an ongoing monitoring. Rune Kvist [00:54:34]: How are you yourself testing before you make at least major releases? We don’t go and audit people every day, but at least there is now a trail where if you do a major mess up, then your customer may come and ask you, “Hey, you promised me that you were going to run these evals yourself.” And for lots of them, most of the. PRs that people merge will not fundamentally alter the product experience, but some of them will. Thank you. Swyx [00:54:57]: And sometimes you don’t know. Rune Kvist [00:54:58]: And sometimes you don’t know. There are inherent risks that everyone knows that when they buy software, there can be bugs, and this is just part of it. But if you’re selling to mom-and-pop shops, they may not care. They’re just like, “Well, I want to use your tool, so I’m just going to willing-- be willing to take that risk on.” If you’re selling to a big bank, they might be like, “Sorry, we’re making promises to our customers. If you can’t make a promise to us that we can pass on, we don’t want to work with you.” Then it’s up to you to say, “Do I care for my agent to get used as critical infrastructure in this mission? If so, at least I can make promises about what processes I run, and then we can go and test it every quarter to be like, well, does it seem like, it’s still, that it still meets the standard.” So from my perspective, it’s kind of a way to. Big companies by default kind of have some amount of trust when they ship AI. Rune Kvist [00:55:49]: If you’re a young company, if you’re just starting out, by default you have no trust. And there are very few places where you can go and get trust. So one of the things that most of our customers did before they started working with us is that they would make their own security blog posts. That’s great. But also, who’s going to trust you saying, “We’re so secure”? Rune Kvist [00:56:05]: Like, anyone can write that. But it’s very hard. Where do you go and get that trust? Vibhu [00:56:08]: Yeah. Rune Kvist [00:56:08]: And so I think making the standards more legible makes it easier for smaller companies to prove that they’re doing what they ought to be doing, because the default assumption is that it’s the Wild West. Vibhu [00:56:21]: Is there a roadmap you have of, like. There’s a lot of work to be done here, right? Rune Kvist [00:56:25]: Yep. Vibhu [00:56:25]: This is the first one. Rune Kvist [00:56:26]: Yeah. Vibhu [00:56:26]: Anything on the roadmap of what you see is next, what’s coming, what’s, what’s missing? The Roadmap: Agents, Models, Robotics, and World Models Rune Kvist [00:56:32]: I think when we zoom out, AIUC-1 deals with agents. Next up, we will deal with models. Next up from that, we will deal with robotics, of which, in some ways, Waymo is the first robot. But the exact same problem is going to be someone’s going to develop a robot, someone’s going to need some promises, they’re going to struggle to make the promises. And - You see this playing out when, like, if you think Fable concerns are bad, like, see when Waymo hits a dog. And that’s if people lose their mind. Imagine when first robot knocks off a toddler off a kitchen table. Swyx [00:57:03]: Yeah. Rune Kvist [00:57:03]: You’re going to see some real strict liability. Vibhu [00:57:07]: I mean, you could see it, right? Like, Cruise got fully Rune Kvist [00:57:10]: Destroyed. Vibhu [00:57:10]: All permits are gone, yeah. Yeah. Rune Kvist [00:57:12]: Correct. So physical AI, the level of stringency just goes up and up. So that’s kind of like the big picture. Agents, models, robotics. I think within agents, the current set of agents are well-covered by this. But as the technology progresses, as agents get longer horizons, new types of failure modes will emerge. And so it’s mostly of can you make sure the standard keeps up when they appear? And you also start to see new modalities. Like today, world models are mostly a kind of a research question. There’s no one who’s really using it. But that will also bring in just new kinds of ways to create value, but also more risk surface that no one knows how to grapple with today. You’ll start to see true agent interactions that are not mediated by humans. There’s going to be a bunch of interesting questions. You’re basically going to need a new legal system. How do they build trust amongst each other? How. One of the core things when humans trade with each other is that you know that you have recourse. You can sue them. How do you make sure that there is a persistent balance sheet behind any agent such that if you trade with it and it screws you know you can get your money back? Those are some of the questions we’re going to have to deal with. And the technical testing Rune Kvist [00:58:24]: Of multi-agent systems is also going to be interesting and complex. Swyx [00:58:29]: Very fun. Are there any perils that are uninsurable right now that people wish that you would? Copyright Risk, Adverse Selection, and Information Asymmetry Rune Kvist [00:58:35]: Yeah. One of the places where there’s a bunch of appetite for insurance and not a lot - a lot of demand, but not a lot of supply, is when it comes to copyright. Swyx [00:58:46]: Oof. Rune Kvist [00:58:47]: In some ways, copyright is kind of mundane. It’s always been an issue. There’s a couple of reasons for this. The first is people who have trained on copyrighted materials almost always know that they’ve done that. Rune Kvist [00:58:59]: So if you want to buy insurance for it probably signals that you might be a high-risk customer. The people who are most interested in getting insurance for copyright infringement Swyx [00:59:09]: Okay. Yeah Rune Kvist [00:59:09]: Are the people who are most likely to have copyrighted Swyx [00:59:10]: Yeah. It’s like a, it’s like a lemon problem. Rune Kvist [00:59:13]: Exactly. Vibhu [00:59:13]: I actually think there’s another side to it too, right? Like, if you’re building on something. So say I’m using an open model. Rune Kvist [00:59:19]: Yeah. Vibhu [00:59:19]: I don’t know what it’s trained on, right? Rune Kvist [00:59:21]: Yes. Vibhu [00:59:21]: And how far down that chain does copyright go? Rune Kvist [00:59:24]: Yes. Vibhu [00:59:24]: Am I liable to take down my product because company X trained on copyright? Swyx [00:59:29]: But there’s safety in numbers. If everyone’s doing it, then you. Rune Kvist [00:59:33]: Correct. Vibhu [00:59:33]: I mean, I would say until, you know, Fable is rolled back from everyone that used it, right? Rune Kvist [00:59:38]: Yeah. I think it’s a hard question. I don’t know the answer to it. Vibhu [00:59:39]: It is. Rune Kvist [00:59:39]: But I think your intuition is, your intuition is right in kind of like, what is the kind of duty of care? Rune Kvist [00:59:47]: And people don’t today think of it as customary that you go and you, like, dissect the open model’s training data and you check everything. In fact, lots of people use them. It’s seen as kind of generally acceptable to not check for this. And therefore, like, we’re not going to hold you to specific Vibhu [01:00:02]: I mean, we also really can’t, right? We don’ Rune Kvist [01:00:04]: Exactly. Vibhu [01:00:04]: We don’t know the training data. Rune Kvist [01:00:05]: So you can then ban it, but I think no court is going to get a copyright question and be like, “This actually needs to get banned.” Swyx [01:00:09]: Unless you hire Nicholas Carlini and he can extract it for you. Rune Kvist [01:00:12]: Exactly. Though he’s in short supply. Swyx [01:00:15]: Yeah. He’- You only have so many Carlinis, but, Rune Kvist [01:00:17]: Exactly. Swyx [01:00:18]: Yeah, go ahead. Rune Kvist [01:00:19]: So I think this is also fair that, in the case of labs, there’s a lot of interest for this. But the thing that makes lab want it is what makes this insurer suspicious of it, and so you have a lemon’s problem. Swyx [01:00:30]: Yeah. Is there, like, a theory of insurance where adverse selection dominates the risk-sharing aspect of insurance? Like, where does this. Like, teach us insurance. Rune Kvist [01:00:40]: A lot of insurance does come back to, like, practical versions of microeconomics 101. Swyx [01:00:45]: Yeah. It’s very. It’s like, it’s like this is why Vibhu [01:00:47]: High-risk adverse. Swyx [01:00:48]: You need to pool health insurance, because if you make it too hyper-specific, then only people who are guaranteed to get the disease will sign up for your insurance. Rune Kvist [01:00:56]: Exactly. Swyx [01:00:56]: Same thing. Rune Kvist [01:00:57]: The core problem is one of information asymmetry. People buying insurance know something about their risk that insurers do not know. And so the question is actually. And this comes back to the same problem is, if you rely. You can break a lot of these information asymmetries if there is. Some kind of testing that reveals the underlying true risk. And so if you were able to, in the case you mentioned, have good diagnosis of whether someone has it or what the probability is that someone has it, that the insurers trust, then they might be willing to insure it. But if they don’t, if there’s no kind of common information, then - only the patient will know Vibhu [01:01:32]: Yeah. Rune Kvist [01:01:32]: That’s what breaks it down. So the question is, again, how do you create credible signaling between players? Rune Kvist [01:01:39]: This is also the whole reason why Moody’s exists. Moody’s just does credible signaling. That’s also why Moody’s could never-- Moody’s has to be independent. If Moody’s was owned by JPMorgan, then JPMorgan cannot use it as a signaling mechanism. So a lot of the basics of standards and certification are just communication devices. It’s just a trust gap. And, that’s where you have to think about what are the incentives of the messenger. And one and another way you can break a lot of this is through transparency. If you are transparent in how you operate, you just cannot mess with others nearly as easily. You make it much more costly, and that increases trust. This is one of the reasons why there’s a change log here. Rune Kvist [01:02:16]: Every little change Swyx [01:02:18]: Yeah Rune Kvist [01:02:18]: You can go back and find, and it means that if we were to make the standard worse Swyx [01:02:24]: Oh, wow, that’s a lot of changes in one update. Rune Kvist [01:02:27]: Yeah. Swyx [01:02:27]: Okay. Rune Kvist [01:02:28]: And a lot of this is just as things get clearer, you can see a lot of clarifications, you can see some revisions. As things get hammered out, you want to change this. But if you make it all public, you make it much harder to mess with people, or at least you become found out very easily. Rune Kvist [01:02:42]: And so this is a way of reducing the information asymmetries by just making more of the information public. Vibhu [01:02:50]: I like how you do know when future versions are coming. Swyx [01:02:52]: Yeah. Vibhu [01:02:53]: So I guess it’s quarterly. Swyx [01:02:53]: I mean, they just Rune Kvist [01:02:54]: It’s quarterly. Vibhu [01:02:54]: Yeah. Swyx [01:02:54]: It’s kind of quarterly. Rune Kvist [01:02:55]: Yeah. Swyx [01:02:55]: Not that surprising. Rune Kvist [01:02:58]: Yeah, but this is also a promise. Like, if we now don’t deliver on July 15, basically Swyx [01:03:03]: I mean, you can just batch it up, and then whatever you got, you just ship it. Rune Kvist [01:03:05]: You just batch it up. Swyx [01:03:05]: Yeah. That’s not that hard. Rune Kvist [01:03:06]: But it’s kind of like we deposit some amount of trust every time we meet this commitment. Vibhu [01:03:12]: Yeah. Rune Kvist [01:03:12]: And in the startup land, it feels easy to ship a new version of a standard once a quarter. In the enterprises who are used to this, like, decade-long cycle, we often get met with, like, incredulity. Like, there’s just no way. And then you show them the change log. Swyx [01:03:28]: One thing I wanted to also, like, try to really think about is, you know, you said something about how if you have tests for the thing, then you can insure it. Rune Kvist [01:03:35]: Yes. Swyx [01:03:36]: Right? And so really what your standard is, what AIUC is, is establishing a framework for the audits to happen so that you can at least test, like, all these, like, baseline standards of care have been met, and therefore people can insure against standard risks that everyone has. I wonder if, like, there needs to be develo-- you need to develop other tests. We’ve covered mech interp in the past. Any interest in that, or are there other kinds of tests that we’re not thinking about? mech interp, Eval Awareness, and Monitoring Rune Kvist [01:04:02]: Yeah, I think mech interp is a big one. A lot of interest in that. I think everyone would agree that there’s, like, promising scientific potential. Rune Kvist [01:04:15]: We’re still a while, a little bit away at least, from this being, like, commercially available on demand such that there’s, like, now a selection of vendors you can go to. Swyx [01:04:26]: Goodfire would say that it is commercially available. Rune Kvist [01:04:28]: Exactly. Swyx [01:04:29]: And it just Rune Kvist [01:04:30]: We would agree with them. We think that the work that they’re doing is tremendous. Swyx [01:04:33]: Yeah. Rune Kvist [01:04:33]: We’re not quite at a point where we could literally require it. But it’s the kind of thing where you can imagine relatively soon you could put in an optional control for if people use mech interp as a way to reduce risk, you at least get credit for it. We can’t require it because it’s going to be hard to require everyone to become Goodfire customers. Swyx [01:04:49]: What good does credit do me? This is - this is a pass-fail, right? Do I care about credit? Rune Kvist [01:04:54]: It’s a pass-fail, but it’s also a 100-page audit report Swyx [01:04:57]: Huh Rune Kvist [01:04:57]: That you’d be surprised at how much security leaders actually sit down and digest this stuff. Swyx [01:05:02]: Okay. Rune Kvist [01:05:02]: And I promise you that if someone is using mech interp today they will have a slide on it because they’ll try and get credit for it. Swyx [01:05:11]: It is cool. It’s fancy, yeah. Rune Kvist [01:05:12]: But it’s just easier if you have a third party saying, “Yep, they have mech interp, and actually.” Swyx [01:05:16]: Just to spell it out for people who have been following our mech interp podcast Rune Kvist [01:05:21]: Yeah. Swyx [01:05:21]: It is literally like, oh, you’re using, you know, OSS. It is activating these three dangerous things. We monitor for it, and we log it out in whatever tool of choice. Gray Swan has, like, Signal or whatever, and that’s it. That’s the mech interp-based activation, signal. Okay. Rune Kvist [01:05:38]: Yeah. So I think mech interp is interesting, and I think if that promise truly comes to fruition, you can make stronger promises than you can with evals. And so I think that’s very compelling. Another thing that I think will become increasingly important is just kind of good school monitoring, and slightly after the fact. One of the things you’re seeing with eval, some of the challenges that are emerging is that the agents are starting to become aware that they’re being evaluated. Swyx [01:06:04]: Yeah, eval awareness. Vibhu [01:06:05]: Yep. Rune Kvist [01:06:05]: Exactly, which is a problem. It means that they basically, if they know they’re being watched, they won’t do the thing that they think they get punished for. And by default, unless you know how to kind of reduce eval awareness, you should trust evals less. And one of the kind of truest things, monitoring, like, is the source of truth. Did you in fact give medical advice, and how quickly do you know? How often - have you done that in the past? How fast do you respond? How often do you detect it? How fast do you detect this? So I think that is also a paradigm. It’s slightly more intrusive. You actually will look at some customer data, but I think will become more prevalent over time. Swyx [01:06:45]: People talk about this like we should not write about eval awareness because it’s going to leak into the data set and then be. Like, we should just. Like, we should, like, never talk about it, only meet in person and, like, talk offline unrecorded. Like. Rune Kvist [01:06:57]: Did you guys see the Anthropic research where. I think this was literally Anthropic did that test. Swyx [01:07:04]: What? Rune Kvist [01:07:05]: It took. I can’t remember the details here, but they, ran some studies on misalignment, and then they took out the training data- That related to LessWrong discussing misalignment, and they ran the same test again and the failure rate went down. Rune Kvist [01:07:20]: So it, in fact, was some evidence pointing towards it had learned the - either the ability or the propensity to do that. Swyx [01:07:28]: Yeah, I mean, so there’s the hyperstition effect, and then there’s, like, the Luigi/Waluigi effect. Rune Kvist [01:07:31]: Correct. Swyx [01:07:32]: Which is like you are. The more you try to train for it, you create the opposite. Rune Kvist [01:07:36]: Yes, there you go. That’s exactly it. Swyx [01:07:38]: In some ways, I think the very success with Anthropic is a result of hyperstition, like the fact that you wanted this thing to exist in the world, and now it does. But, like, then it also creates the opposite as well. Rune Kvist [01:07:48]: Yes. Swyx [01:07:49]: Like, I think people who are maybe newer to this space don’t remember Waluigi, but, like, I do think it’s very important for understanding that when you train for a thing, you also train the opposite of the thing ‘cause it’s just a big flip. Rune Kvist [01:08:02]: Yes. Rune Kvist [01:08:03]: Yes. Vibhu [01:08:04]: I think, you know, just going back to where we were at, like, there’s a lot more than just mech interp that there’s value in just having added, right? So your version of how fast can you measure stuff? Do you have logging? Do you have evals? You know, do you see other parts of the stack, like the inference providers that you use, the services? Okay, am I using Chinese model on their home API? Am I using through certified vendor here? Am I hosting myself? What am I doing on the inference engine side? There’s just, like, so many levels of stuff that gives, you know, information that you can standardize out, right? Managed Agents, Enterprise Controls, and Generative Media Rune Kvist [01:08:36]: Yeah. And you also see increasingly, in addition to just the basic chatbots, you’re increasingly seeing big companies adopting agent platforms where they’re building on top of Google’s Agent Studio, et cetera that comes with a bunch of, like Vibhu [01:08:52]: Managed agents. Swyx [01:08:53]: Managed agents. Vibhu [01:08:53]: It’s everywhere now. Swyx [01:08:54]: Everyone has managed agents. Rune Kvist [01:08:55]: Exactly. Vibhu [01:08:56]: And there’s even levels. You can host your own managed agents, OpenAI’s Agent SDK, or hosted by Anthropic, or Google does both. Rune Kvist [01:09:03]: Correct. And then these are just ways to kind of strengthen the security guarantees you can make. And in some ways, it’s kind of bread and butter enterprise security. They. Like, they love to host things on their own premises because it gives them really a sense of control. And I think you’ll, you’ll see, just like you do in every other enterprise market, if you really sell to the enterprise, you start to compete on some of these security features. And this is also happening in AI, unsurprisingly. And I think you are seeing some amount of enterprises wanting. Enterprises are really grappling with the thing that makes agents useful is that they’re stochastic, and the thing that makes them really hard to adopt is that they’re stochastic, and these are in tension. Rune Kvist [01:09:46]: Leaders come out on different sides of that table, in part depending on how much the CEO is trying to get the stock price to go up by saying they’re AI native and that we must be willing to take the risks. We see, we actually see phenomenal tension in the heads of the CISOs of the Fortune 1000, where on the one hand you have the CEO saying, “We must adopt, otherwise we’re becoming irrelevant, and if we f**k up, you’re fired.” Swyx [01:10:08]: Oof. Rune Kvist [01:10:08]: And that’s kind of like the core emotional tension that we see showing up again and again. And one of the core problems that we solve for them is to take that abstract emotional concern and turn it into a framework, in some ways just providing clarity to that concern. Vibhu [01:10:23]: So anything in here. So something I think we kind of skipped over. We talked a lot about agent language models, skipped over world models. Rune Kvist [01:10:31]: Yeah. Vibhu [01:10:31]: You guys have voice, which is interesting with ElevenLabs. Rune Kvist [01:10:34]: Yeah. Vibhu [01:10:34]: How about generative media? So, you know, generating images, videos, that’s a category that actually has a lot of usage. Is there anything in your current policy? Is it separate policy? How do you see that space? Rune Kvist [01:10:46]: Yeah. Vibhu [01:10:46]: It’s like we did talk a bit about copyright, Swyx [01:10:49]: Music. Vibhu [01:10:50]: Yeah, music as well. Rune Kvist [01:10:51]: Yeah. I think a lot of the concerns that come up there either relate to, copyright or there’s a lot related to, let’s call it broadly safety. So, like, this could be not safe for work or just very graphic materials, are kind of some of the core things. We have done some work on this. There’s a little bit in the standard as well that deals explicitly with that. Video, we have not done a lot in yet. And I think for proper production, that has still. Especially proper production without a human in the loop, that’s still got some ways to go. It’s obvious that it’s coming, but it’s very rare that it’s like shot deploy a video to the internet. But eventually that will also happen. Vibhu [01:11:33]: We see, like, you know, Luma has Luma agent where it’s still pretty human in the loop. Rune Kvist [01:11:37]: Yeah. Vibhu [01:11:37]: So it’s not just Rune Kvist [01:11:37]: And that just makes complete sense as the technology matures, and over time, it will become so good that people will not want to slow things down by having a human in the loop. And then, the need to make promises will grow. Swyx [01:11:52]: Why not just have prediction markets on everything? Prediction Markets vs. Audits Swyx [01:11:55]: Right? It’s very EA adjacent. Rune Kvist [01:11:56]: Yes. The core thing is that the people. Prediction markets rely on public information. There is not a lot of public information. It’s just insiders trading on each side. Swyx [01:12:06]: Yeah. Rune Kvist [01:12:09]: That’s illegal. Vibhu [01:12:10]: There’s leaked information. Rune Kvist [01:12:12]: There is leaked information. The core challenge is that often you have private sensitive information, and you need to convey confidence and trust around that. And you can, of course, for some claims, like can any model be jailbroken, you could rely on public evidence ‘cause there would be lots of people being like, “Well, there’s tons of studies, and actually they all can, so that resolves fine.” I think that’s good. For, hey, this new unreleased Methus model, how capable is it actually? Rune Kvist [01:12:43]: Prediction markets have not a lot to say because actually just no one knows. And so I think that’s the core place where some of this breaks down, is that actually lots of the world’s information that guides some of these high-level decision is private and often also just not known. Vibhu [01:12:56]: I think the thing with prediction markets that people like is it’s not, it’s not answering the broad question. It’s a specific, right? So will a model do this by this date, or is a model capable to do this by then, right? Rune Kvist [01:13:07]: Yes. Vibhu [01:13:08]: That’s a little distinction there. Rune Kvist [01:13:10]: Yeah. And often the most interesting question, if you are, say, the head of security at a bank. The question you’re really trying to answer is, will this product, this agent, do this bad thing that maybe primarily I care about, specifically in the setting that I care about? And the question is like, what’s the closest-- That information may not exist anywhere. So prediction markets aggregate existing information. This information may not exist, and you want some very specific and you’re willing to pay for it. That’s kind of where a third-party audit comes in. We also don’t really use prediction markets to figure out whether, public companies have committed fraud in their books. You use audits. You probably could, but the information’s just not that available. And if so, it would be like just trading on vibes. Actually it would have been really interesting to see whether prediction markets two thousand and one were predicted Enron going bankrupt and they kind of Swyx [01:14:02]: Yeah. Rune Kvist [01:14:02]: Could you have told-- could you have sensed from like the craziness of the CEO or some other traits that they were more likely to cook their books than others? Swyx [01:14:10]: Or enough insiders leak it then that Rune Kvist [01:14:12]: That could also be right. Swyx [01:14:13]: Right. Which is like, I mean, this-- that’s the sort of the ideal dream of prediction markets. You have liquid markets and everything. Rune Kvist [01:14:20]: Yeah. Swyx [01:14:20]: And then you can compose your exact set of risks to offset. Rune Kvist [01:14:24]: Yes. Swyx [01:14:25]: Right? Rune Kvist [01:14:25]: Yes. Yeah. And I think, like, prediction markets will bring lots of new information to it. So the thing is mostly not like which one is it, and more like what are the types of questions that prediction markets are really good Swyx [01:14:37]: Yeah. Rune Kvist [01:14:37]: And what are the ones where the information doesn’t even exist for insiders such that no one can in fact trade on it and it needs to get generated. AI Engineer Certification and Training Swyx [01:14:43]: Okay, one self-serving question and then one open-ended one, on like the future of AIUC. Self-serving question would be, so you have your standard, right? Rune Kvist [01:14:52]: Yes. Swyx [01:14:52]: I run, you know, a large AI engineer conference. Like, there’s been a lot of talk about us certifying AI engineers. Rune Kvist [01:14:58]: Yep. Swyx [01:14:59]: Training programs, level one, level two, level three. I was a CFA myself, so I know what-- that’s what the finance industry does. Rune Kvist [01:15:04]: Yes. Swyx [01:15:05]: Would it help if I had AI engineer level one, level two, level three, and then it would-- they would, like, work with these guys? I don’t know. Rune Kvist [01:15:12]: If you think of the highest level objective as, like, accelerating secure deployment of agents, then that would totally help. Because one of the things that happens often now is that folks build agents, they bring it to the decision-maker, and the decision-maker surfaces a bunch of security considerations that they had not thought of, and now it’s not built to spec. Now you have to go and - like, add these filters, et cetera. So if you shifted that left, like if everyone knew what the spec they were building to, if everyone knew the grading scheme Swyx [01:15:41]: Yeah. Rune Kvist [01:15:42]: That would be awesome if they were already trained. So by default Swyx [01:15:44]: But you’re the grading scheme, right? Rune Kvist [01:15:45]: Say again. Swyx [01:15:45]: I don’t get to set the grading. You guys, you set the grading scheme. Rune Kvist [01:15:47]: We set the grading scheme. And I think what’s, valuable is, like, if you can turn those into Swyx [01:15:52]: Training programs. Rune Kvist [01:15:53]: Training programs Swyx [01:15:54]: Yeah. Rune Kvist [01:15:54]: Such that people Swyx [01:15:54]: Which you’re, you’re not doing. Rune Kvist [01:15:55]: We’re not doing that. Swyx [01:15:56]: Yeah. Rune Kvist [01:15:56]: I think there’s value in doing it. Vibhu [01:15:57]: There are others doing. I mean, not to interrupt, but you know Swyx [01:16:00]: Yeah. Vibhu [01:16:00]: OpenAI has their Swyx [01:16:02]: Anthropic also has like a CCTA thing. Vibhu [01:16:04]: Yeah. You know, they want hundred thousand deployed certified consultants, right? Rune Kvist [01:16:09]: I really think it’s good for. We will accelerate adoption if we have more people who know how to build secure agents, and we are not working on the side of training people at the moment. I think it’s, like, very aligned with our mission. We only have so much, attention. Swyx [01:16:24]: I’ll tell you why I haven’t done it. Rune Kvist [01:16:26]: Yeah. Swyx [01:16:26]: It’s not like I haven’t thought about it before. Rune Kvist [01:16:28]: Yes. Swyx [01:16:28]: It’s just being prescriptive Rune Kvist [01:16:30]: Right. Swyx [01:16:31]: About like, well, this is what you should know, therefore, like, the stuff that I didn’t include is what you don’t need to know. Rune Kvist [01:16:35]: Yes. Swyx [01:16:36]: And I’m like, “That sucks.” Like. Rune Kvist [01:16:37]: Yes. Yeah. Vibhu [01:16:39]: But I think it’s like, you know, the very interesting defensible thing you guys do is your opinionated 100-page report of here’s what matters, right? Here’s the, like, prescriptive definition of the requirements you need to be certified, so. Rune Kvist [01:16:55]: Yeah, and I think that’s a choice. I think basically that’s a, that’s a choice, and I think that serves some audiences very well, where if you’re trying to deploy this into a bank or a hospital, et cetera, clarity of the - those boundaries is extremely valuable. Rune Kvist [01:17:09]: There’s lots of other settings where being much more experimental, much more trying it out is just the better fit. And so to me, this makes a ton of sense. Also, you’d have to rewrite your curricula every freaking three months. Swyx [01:17:21]: It’s fine. I do that. Like, it’s okay. But yeah, no, for me, it’s actually - like, genuinely, like, the consequences of getting it wrong and, like, affecting somebody’s career is a big responsibility. Rune Kvist [01:17:35]: Yeah. Like, I think that’s exactly right. And I think a lot of our work actually goes like, we don’t want to carry. We also don’t think of ourselves as able to carry the, kind of the true north of what’s, like, secure or not secure, but we can coordinate the forum where you listed all of that. Swyx [01:17:52]: Yeah. Your consortium is fantastic. Vibhu [01:17:54]: Do you think this can be crowdsourced in a way? Like, for your example, for what is AI engineer certification, right? This is a pretty big podcast. There’s a lot of takes that people can have and, you know, discussions that can. Swyx [01:18:05]: And people reasonably disagree. So who am I to say, like, that’s a correct question, that’s a wrong question? Rune Kvist [01:18:09]: Yeah. Swyx [01:18:09]: Right? So, like, I don’t know. Vibhu [01:18:10]: We’ll have an exit. Rune Kvist [01:18:13]: Yeah. Vibhu [01:18:14]: Vent your frustration to someone that’s listening, you know? Rune Kvist [01:18:16]: Exactly. And I think there’s also you. Or it matters a lot what the promise is. So if the promise is, “Hey, if you’ve taken my course, you will not f**k up,” you can’t make that promise, clearly. You could make a promise of like, “Here’s the. Some important things that everyone should at least know,” and then you have to fill out the rest there. At least the promise changes. Of course, there’s some subtlety in how do you communicate this such that people really get it. But I think it’s important to dial in, and we have a section in our center on, like, what is the promise and what is the promise not, because it’s impossible to guarantee that nothing will go wrong. If you need a guarantee that nothing will go wrong, you cannot work with frontier AI, but you can make some claims. Swyx [01:18:56]: Yeah, for sure. Cool. Wanted to end with open-ended, where is AIUC going? I think you talked about model stuff, robotic stuff. And just open-ended, like, where, you know, what is in the future for you guys? AIUC’s Future, Hiring, and Universal Red Teaming Rune Kvist [01:19:10]: Very near term, we’ve now started to work with some of the frontier companies in each of the categories that are taking off, and we’ll, we’ll continue that work to make sure that we cover all of the use cases that are really taking off. We see a lot of interest once the first one in the market moves. Lots of people want to follow them. And we think basically AIUC-1 will get to a point where all of the Fortune 1000 will organize their risk processes around the standard. Swyx [01:19:38]: And you have 50%? Rune Kvist [01:19:39]: No, we do not have 50% today. Swyx [01:19:41]: Oh. Rune Kvist [01:19:41]: I think there is some world where probably by end of year, we might have representation in our consortium for 50% of the Fortune 1000. Swyx [01:19:48]: I see. Got it. Rune Kvist [01:19:49]: So that’s on the agent layer. And then we think, yeah, the model layer, it’s going to be. It just brings. Are now surfacing the concerns that are most likely to slow down adoption of AI. And then, yeah, we think robotics comes after that. Swyx [01:20:03]: What are you hiring for? What’s hard to hire for? Rune Kvist [01:20:05]: We are hiring, across the board, across market and numbers of technical staff. The people who do really well on our technical team are folks who are really excited about kind of being truly full stack. So let’s say when we started working with Cursor, we’d never done coding, tools before. So taking the standard and extending it, fleshing out what does frontier evals look like for long horizon coding agents, and taking that problem all the way from, like, working with Cursor and other folks in this space down to, like, fleshing out and shaping a new version of the standard. So that’s like a truly a full-stack, entrepreneurial technical people do extremely well at AIUC. The hard part is building one universal red-teamer that works across from Harvey to Cursor and everywhere in between that both has one consistent methodology, one consistent taxonomy of what are the risks and the attacks, and making. We think that’s fundamentally the best way to make consistent promises. JPMorgan is buying both. They want to have one framework, one consistent way that this comes out, and the mechanics of making that happen, you get to deal with a lot of the complexity of the real world. I think we have good answers in a bunch of that, but there are some pretty hard engineering problems in executing that. Swyx [01:21:18]: Can I push a little bit? Like, must you have one? Why not just be like, “Okay, look, forty percent of our use cases are coding agents, so we will specialize in coding agents,” and that’s the, that’s the one of them. Rune Kvist [01:21:28]: Yes. Swyx [01:21:29]: And then, okay, thirty percent is like RAG. Rune Kvist [01:21:31]: Yes. Swyx [01:21:31]: Just do RAG. Rune Kvist [01:21:32]: Yes. I think there’s some wisdom in that question. Swyx [01:21:36]: Yeah. Rune Kvist [01:21:37]: It depends on. What we found that there’s a lot of value on is being able to. If the decision-maker on the buying side, let’s say you’re the head of risk at a bank and your biggest risk is not in coding or in customer support or whatever the top two biggest use cases, but it’s somewhere else, you want to still make sure that framework has something to say about it to the burning question you have. Otherwise, you’ll not earn that trust. Now, it’s true that a lot of the burning questions follow where there’s a lot of adoption. And so great, so do we. So we do today do not cover every single edge, but we have a framework that we can add all of these within. We have one global taxonomy of risks and attacks that keeps adapting. Rune Kvist [01:22:20]: As, like, every time a new incident occurs that has never been seen before, great, let’s go and update the taxonomy so we bake that in. So I think we have one coherent universal approach. It doesn’t mean that we spend equal amounts of time on code and insert niche use case. We do spend time where people care. We think it’s very valuable to have one language. Swyx [01:22:43]: Yeah. That makes sense. That’s, that’s a, that’s an important choice. We were going to end actually, but I thought of one final ending closing question, which is, take this however you want, right? Let’s say one and a half years from now, OpenAI’s secret panel of five experts declares that we have reached AGI. AGI, Watchdogs, and the Need for Independent Oversight Swyx [01:23:00]: Do you expect your business to change? Rune Kvist [01:23:03]: No. I think there is some important way. I think the last businesses to exist beyond the labs Swyx [01:23:10]: Will be underwriting. Rune Kvist [01:23:12]: Well, there is one, there’s one job that the labs can never do for themselves, which is to be their own watchdog. Swyx [01:23:19]: There you go. Rune Kvist [01:23:21]: So I think kind of to the extent that you believe this frame of, like, you’ll see hyper-concentration, like the labs will kill all the startups Swyx [01:23:29]: Yeah Rune Kvist [01:23:29]: Which, we can go into the pros and cons. Swyx [01:23:32]: I feel like the labs actually care a lot about this, right? There was the whole superposition, what do we do when we have models smarter than us and then a tier above, right, models smarter than them training them. Rune Kvist [01:23:41]: Yes Swyx [01:23:41]: The labs actually think about this a lot. Rune Kvist [01:23:42]: They think a lot about. I think the there are some of the smartest people on these topics work at the labs. So the problem is not whether they care. The problem is that they will all be stuck in a race where they might have incentive to cut corners, and they might have incentive to withhold information from the government, et cetera. And so one kind of feels like eternal truth is that you need an independent third party to go and inspect that data and share information, in this case, say, with the government. It’s more of an incentive problem than an interest problem. I think they’re fundamentally all trying to make this go well. Swyx [01:24:14]: What I’m not hearing is, like, AGI, whatever that label means to you, to me, to them, doesn’t fundamentally have, like, a qualitative shift Rune Kvist [01:24:23]: Correct Swyx [01:24:23]: In, like Rune Kvist [01:24:24]: Correct Swyx [01:24:24]: You still have to evaluate the models. Rune Kvist [01:24:26]: And I think the one thing that would make this a qualitative shift is, there’s. For some definitions of AGI, it will just get nationalized. It’ll be a threat to sovereignty. Swyx [01:24:34]: Yes. Rune Kvist [01:24:34]: And then at that point, it kind of maybe every company is the government is every company. I struggle to think about that world. But at that point, you’ve kind Swyx [01:24:42]: We. I don’t think we’ll move fast enough. Rune Kvist [01:24:44]: Right. Swyx [01:24:44]: You know, like, we’re not, we’re not set to do that. Rune Kvist [01:24:47]: Yeah. Swyx [01:24:48]: But I have discussed this a lot on the podcast. Rune Kvist [01:24:51]: Yeah. Swyx [01:24:52]: I mean, you know, as far as the watchdog concern, I will also mention that because I have my finance background, I often think about the scene in The Big Short where they talk to, like, Moody’s, but also Standard & Poor’s. And then the lady at Moody’s is like, “Well, if I don’t give you a triple A rating, you’re just going to go down to Standard & Poor’s.” Rune Kvist [01:25:09]: Yes. Swyx [01:25:09]: So actually the watchdog is a natural monopoly because if you have race dynamics in watchdogs, then the watchdogs will compete each other to the lowest possible standard. Closing: Insurers, Incentives, and Trust Infrastructure Rune Kvist [01:25:18]: Correct. Rune Kvist [01:25:20]: And so I think what one of the things, one of the reasons why we’re very excited about having insurers be around this table is that insurers are the only ones that do not have this dynamic because they pay the bill. If they keep lowering the prices Swyx [01:25:32]: Yeah, you will Rune Kvist [01:25:33]: They also pay the bill. Swyx [01:25:33]: You won’t find the market clearing. Rune Kvist [01:25:35]: And this is not true for Moody’s where, they don’t directly pay the bill if they make recommendations that are off. So we think that balancing factor is pretty important. And I think it also highlights that there’s, like, no system that’s perfect. You need scrutiny of Moody’s, you need scrutiny of the watchdogs, for sure. Swyx [01:25:52]: Beautiful. Thank you so much for indulging. This is a beautiful conversation covering everything. Congrats on your success so far. Rune Kvist [01:25:59]: Thanks for having me. Swyx [01:26:00]: Yeah. Awesome. Rune Kvist [01:26:00]: Appreciate it. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:03Speaker 00

  Okay, we're in the studio with Run from AIUC, AI underwriting company with our trusty co-host Vibhu. Welcome.

• 0:09Speaker 01

  Thank you. Thanks for having me.

• 0:11Speaker 00

  What are you announcing today?

• 0:12Speaker 01

  We have raised forty million dollars by Ribbit Capital and First Timoney.

• 0:16Speaker 00

  You first came to my attention when Nat and Dan invested in you guys. Is the story like pretty much the same? Like, are you today where you thought you were back then?

• 0:26Speaker 01

  When we raised our seed round, we had a hypothesis that at some point, risk was going to hold down adoption. At that point in time, that felt kind of hypothetical. And I think that that is now over. Clearly, the moment is now with Methos and with Fable. It's pretty obvious that literally the binding constraint on adoption is risk. And so for us, it feels like this is a natural continuation of the same hypothesis. But we were previously speculation, now it feels like fact.

• 0:54Speaker 00

  And let's get the list of the customers that you highlighting as part of your series A.

• 0:58Speaker 01

  Totally. Yeah, so we are now working with folks like Cursor, Harvey, Lovable, 11 Labs.

• 1:05Speaker 00

  Yeah, amazing. Congrats.

• 1:06Speaker 01

  Thank you.

• 1:07Speaker 00

  So you were famously one of the first, the first hired and adopted for GTM and products. I'm just kind of curious, like, what was your path into AI? Just recap.

• 1:19Speaker 01

  Late 2021, I sold a company, my first company, a tech company. I had a bit of time to think about what was next. I came across the scaling loss paper. And that just struck me like lightning. I was just like, this is a big idea. In short, the scaling loss paper just says the bigger the model, the smarter the model.

• 1:38Speaker 00

  And this is the Kaplan one, not the Cincilla one.

• 1:40Speaker 01

  Exactly, the Kaplan one. And the important thing that clicked for me there was, oh, now capital will understand this. If you put in more money, you get more money out. And so that will kick off a hype cycle. And so you'll actually kind of, you'll get a sense of predictable returns, which is in fact what's played out. And so I just packed my bags. I've never been to San Francisco. I just packed my bags throughout here to find the people who have written it. And at the time, they had just started a small lab called Anthropic. There was like forty people at the time or so. Drank a bunch of coffee until I eventually got introduced to Dario. And at the time, they were wrestling with some of these questions of like, should we deploy our models? Should we make revenue? How should we engage with the rest of the world?

• 2:25Speaker 01

  They'd just broken off from OpenAI. And it's been publicly reported that they were kind of concerned with how they were dealing with deployment. So they were wrestling with some of those questions. At that, at this point, this is like early fog of war, like early 2022. The sexiest program at the time was like Jasper. There's nothing out there. So where is value going to accrue? What are going to be the different parts of the stack were all open questions.

• 2:48Speaker 00

  I want to highlight to people, you ask these questions because you have a PPE background.

• 2:52Speaker 01

  Yes.

• 2:52Speaker 00

  I actually was in Singapore in one of the sort of feeder programs for prepping people for PPE. So I had a tutor. We learned, you know, philosophy and politics and economics. But like, I think you're kind of like, machine learning people who read the neural scaling loss paper would not necessarily draw the same conclusions that you did. Whereas any capitalist would read that and go, holy shit.

• 3:19Speaker 01

  Correct.

• 3:20Speaker 00

  Right?

• 3:20Speaker 01

  Yes.

• 3:21Speaker 00

  Who tipped you onto that paper? Because it's not a paper that you normally read, right? Like in your circles.

• 3:26Speaker 01

  Yeah, I think I'd actually, ever since AlphaGo had had some appreciation that AI was a big deal. But it kind of felt, it raised all these kind of interesting philosophical questions, but it's kind of not clear from afar where exactly that would go. But it was obvious enough that it was like, this is going to be a big thing if we find the kind of right mechanism to kind of get the techno-capital machine to work on this. But it was just not clear. And so I think there was some way in which like, that became obvious. And also, it wasn't as obvious at the time than that it is now, right? It was just like, wow, this is so interesting. But it still felt, coming from kind of a philosophy and economics background,

• 4:10Speaker 01

  it felt like if this turns out to be true, you're going to be wrestling with all of the big questions in society. Everything you've learned about politics gets thrown out of the window. Everything you've learned about economics at least gets challenged. And so what felt interesting was to be at that frontier that has just ramifications across everything. So that's why I thought it sorted out.

• 4:32Speaker 00

  I mean, clearly really good insight. For people who don't know, PP, the PP program is like where prime ministers are born. So then you end up meeting Jared.

• 4:41Speaker 01

  Yep, first Dario, yeah.

• 4:43Speaker 00

  Yeah, well, I mean, like, so did you get extra insights from talking with them that you didn't get from your original hypothesis?

• 4:50Speaker 01

  If you read the scaling loss paper, you get this like very vague sketch of like, wow, this seems kind of important. There are some lines that are charted, this seems kind of important. And what I think the team at Anthropic had thought more about than anyone was like, what are the implications of this? If you really play this out. And back then, they had kind of vision documents for what the world would look like in 2026. And they were kind of in vivid detail playing out how much compute is going to be needed? What is the CapEx going to look like? What are going to be some of the kind of societal concerns? But also, what is the amount of economic value coming out here? And so it kind of felt like they held a crystal ball that in hindsight, it turned out to be dramatically correct. And they weren't holding it like they were obviously correct.

• 5:34Speaker 01

  They're just like, take this hypothesis really, really seriously.

• 5:38Speaker 00

  Think it through.

• 5:38Speaker 01

  And think it through. In the same way, the kind of situational awareness that is now.

• 5:43Speaker 00

  Across the street.

• 5:43Speaker 01

  Across the street.

• 5:44Speaker 00

  Yeah. Oh my God, we're all in the same, like, same one square mile of.

• 5:50Speaker 01

  And that's now a couple of years old. But also, people keep referencing it these particular weeks with Fable and Methos. And it's like, wow, if you take this one idea seriously, the scaling loss, a lot of things fall into place.

• 6:03Speaker 00

  And keep in mind, at this point, this is the same team that had did GPT-1, 2, and 3.

• 6:08Speaker 01

  Correct.

• 6:08Speaker 00

  Which is also like, it's not just some experimentation. Like, this is a real model that we just scaled up.

• 6:14Speaker 01

  And they had deep conviction in, again, in this like big, if you take a big blob of compute, data, it just wants to learn. And out of that will come smart and smarter models. And all the particulars were not clear.

• 6:26Speaker 00

  Yeah, yeah.

• 6:27Speaker 01

  And all the implications were not clear. But their deep conviction in this like core thesis. And that was kind of dizzying. It's both phenomenally interesting and exciting. And also, very quickly, you get to like, the world we know today will no longer be, if this hypothesis holds. So it also just felt like important in some kind of grand sense.

• 6:48Speaker 02

  What kind of shaped you there? So that was your early 2022. Not only had GPT-1, 2, 3 come out, but, you know, the amazing co-founders of Anthropic that have never split up, the only ones, they actually had the conviction to leave OpenAI, start their lab. You said there were about forty people there. What was the time like there?

• 7:06Speaker 01

  It was kind of remarkably like what it looks like on the outside today. Extremely cohesive, extremely mission-oriented. And living in this tension between their two ideas, which is, AI could both go really well and really bad. And we want to be part of building it. That creates astounding amounts of tension. And they were wrestling with this incentive challenge where they know they're kind of, there's a race that they're in where you might get forced to cut corners. But it also felt very important to them to be at the forefront of technology. And all of those ideas were just present at that time. It kind of feels like that line has been just very, very clear.

• 7:49Speaker 01

  And I think kind of love them or hate them, they have really stuck to their guns. There's a course set of beliefs that they hold more deeply than most companies hold any beliefs.

• 7:58Speaker 02

  Yeah. Fast forward to the day. What does that lead us to? Yeah, underwriting company. What are you up to? What motivated you to start this?

• 8:05Speaker 01

  Yeah. AIUC builds confidence infrastructure for Frontier AI through standards and insurance. They linked from Anthropic to building confidence infrastructure, looking out the windows at Anthropic offices and seeing Waymos driving by. Already back then, early 2022, Waymos were in some ways like AGI for cars, like they were superhuman drivers. But you couldn't take one to the airport. And now, four and a bit years later, you still can't take a Waymos to the airport. Despite now everyone having kind of looked at the evidence and been like, they're better drivers than humans. So in that particular instance, what's clear is that the binding constraint on AI being useful is not capability, but is that liability or risk or trust. That problem is general.

• 8:49Speaker 01

  The reason why right now Fable is not open for access is not because it's not a good model. It's because it's a very good model. It's just hard to make promises about what it will or will not do. And this problem gets worse as AI gets better. Basically, more intelligent AI can be more autonomous. That's more valuable, but also the risk surface grows. And so what Waymo illustrates is that unless you build the confidence infrastructure to make promises about AI, or at least bring light to the risks, you grind adoption to halt. Governments, banks, hospitals, militaries need to have some sense of what AI will and will not do to be able to operate for them to incorporate it.

• 9:32Speaker 01

  And that's the problem that we're trying to solve. Now, why standards and insurance?

• 9:39Speaker 01

  If you trace this problem back through history, every technology wave has had some version of this problem. So if you go back to like year 1900, electricity comes out.

• 9:47Speaker 00

  Been very good.

• 9:47Speaker 01

  Cars burned down, sorry, houses burned down, lots of people die. 1930s, cars are a big deal, kill lots of people. 50s, private nuclear energy is a big deal, poses big risks. In each of those instances, the market runs ahead of regulation to create confidence infrastructure, because that's required to make go no good decisions. They're required for adoption, and the market fundamentally wants adoption. And in all of those instances, common blueprint emerges between standards and insurance. The reason these two components is standards kind of provide the rules of the road, and they also specify like, what are the tests that need to be run? So we can get a sense of how high the risk is.

• 10:28Speaker 01

  So taking the case of cars, as like a car crash, great, everyone, they inform your insurance pricing today, they inform your purchasing decisions, et cetera. That's basically the risk framework. The insurers are important because they pick up the bill. So they are the private institution that is most on the side of, that is best incentivized to quantify the risks truthfully, and then figure out all the ways to reduce the risk, because that increases their profit. So they're basically, they help shape the incentives. And these two work really well in unison. Now, how does that show up as a company? Well, one of the things that was obvious, even, or starting to become obvious even a couple of years ago,

• 11:08Speaker 01

  was that frontier companies, some are our customers today, like Cursor, Sierra, 11 Labs, Harvey, we're going to have a very easy time selling a pilot to a bank. The demo just sells itself, it's magic. But bringing that through, if you want to do a wall to wall rollout at a bank or a hospital, you have to go through the risk process. These banks have no idea even which questions to ask, let alone which answers are sufficient, let alone like, how do they go and test whether these agents actually work the way they're supposed to? And so they had this problem of like, what can we say to earn the trust? And we think there's like a golden sentence that goes something like,

• 11:50Speaker 01

  hey, I hear you're really worried about hallucinations or jailbreaks or whatever it may be. We've had an independent third party test us against the gold standard. We pass with flying colors. And as a sort of confidence, the world's most conservative insurers have looked at the data and are willing to take some of the risk on to their balance sheet.

• 12:06Speaker 00

  Yeah.

• 12:07Speaker 01

  So if something does go.

• 12:07Speaker 00

  There's money behind it, yeah.

• 12:09Speaker 01

  Exactly. That's kind of like the link between all of them. We can get into some of the hot parts related to the technical testing, which is, I think, the crux of the matter, but I'll post there.

• 12:19Speaker 00

  How did you and Rajiv come together? This, there's always like, you come across very confident and, you know, you're, we're announcing your Series A and all these things, but I want to see like the early initial stages of like idea formation.

• 12:31Speaker 01

  Yeah. Rajiv is actually my soon to be brother in law.

• 12:35Speaker 00

  Oh.

• 12:36Speaker 01

  So I'm actually, in a week and a half, getting married to Rajiv's sister.

• 12:42Speaker 00

  Okay, now you're tight.

• 12:44Speaker 01

  Exactly.

• 12:44Speaker 00

  You know.

• 12:45Speaker 01

  So Rajiv and I have known each other for a decade. Funny story, I met both Rajiv and his sister Hannah at the same time when Hannah and I were interns at McKinsey in London, and Rajiv was assigned as my mentor. And so met them at the same time. For a long time, it was not obvious that we were necessarily going to work together. I was in startups, he was an insurance partner at McKinsey. Three or four years ago, I think Hannah convinced him that AI was going to be a really big thing. And so he quit his job, cushy partner job at McKinsey in London, packed his bags, flew to San Francisco, and ended up joining METR. You guys are probably online enough.

• 13:24Speaker 00

  We've interviewed them.

• 13:26Speaker 01

  The chart of AI, the horizons of the task that agents can take on is doubling extremely fast. So he was zero there, led their partnerships with Anthropic and OpenAI to test their models before release, but also working closely with the US and UK government to figure out like, how do you know whether a model can be released? And in some ways, that's like the perfect background. He's spent a lot of time in insurance, knows that world, spent a lot of time with frontier testing of models. And so when I was bumbling around this idea space, starting with some of the ideas we talked about related to Waymo, as soon as we got into the content, we were both like, oh, this would be an amazing business to build together.

• 14:08Speaker 01

  This is like wrestling with the problem that we both think is the most important in the world from a market angle, which is kind of our intuitions is that the market can do a lot, and the faster AI moves, the harder it is for government to solve some of these problems. And then took a little bit of time to work through what is it like to work with family? And.

• 14:30Speaker 00

  Because you're already dating at the time, or?

• 14:31Speaker 01

  Yeah, yeah, exactly. Already back then, it was, we felt like we're a family. And so starting a business together felt like kind of a big step. And here we are with just immense amounts of trust.

• 14:43Speaker 02

  Yeah, so now you're a company of how big? How big are you guys now?

• 14:46Speaker 01

  There is just 20 of us now.

• 14:47Speaker 02

  20 of you guys now have Series A, and you have your first certification out, AIUC1. Let's bring up the certification. So this is the agent certification, right? What goes into the process? I have like two questions here. One is, walk us through the certification. And two is, what is the process for a company to get certified, you know?

• 15:08Speaker 01

  Great. As it says right at the top, AIUC1 is the standard for agent security, safety, and reliability. The fundamental design principle is take all of the concerns that slow down adoption, all the questions, all the fears that keep security leaders in the Fortune 1000 up at night, and put them into one comparative framework. That's what you'll see there, you can see the six categories. Two, you want to ground all of this in technical testing. So one of the concerns with security standards that often feel kind of like theater paperwork is that they don't actually ground out in, does any of this work? Does any of this matter? And so we had a conviction from early on that that was going to be the kind of crux, was to pass this, you must get tested every quarter.

• 15:52Speaker 01

  Basically run thousands of simulations to see, well, so can it actually be jailbroken? How hard is it to jailbreak? How often does it hallucinate? How often does it leak data, et cetera? And then the last core idea here, if you scroll up to the top here, is to refresh it quarterly. So the core trait of AI is that it moves extremely fast. Whatever concerns we're discussing today, were not the same ones three months ago, and this will keep changing. Typically, standards update on like a decade cycle. This is obviously not going to work. But the question is kind of, how do you update it? And the core thing here was to basically get the risk leaders of the Fortune 1000 around the table. So if you go over to the left here, you'll see AIUC1 consortium.

• 16:34Speaker 01

  The consortium is a group of risk leaders who run real banks, real hospitals, real critical infrastructure, who are facing these challenges every day. And we meet with these folks twice a quarter and hear what's top of mind, what is keeping them up at night? There's tremendous amount of desire for that conversation. And then we operationalize that into a specific standard that gets into, and actually, we can go into and look at what is, what even is the standards? If we go back to introduction, out there to the left, scroll up a little bit to the wheel, click into reliability. So if you take something like hallucinations, hallucinations sits in reliability. There is a number of requirements here. If you go into the top one, prevent hallucinate requirements, hallucinate outputs. This is one particular requirement, this is a technical control. Basically, we want some kind of groundedness filter.

• 17:17Speaker 01

  The first thing you see here is what's called a crosswalk. So every one of their grandmother has put out a framework, very high level framework for what are the AI risks.

• 17:27Speaker 00

  This is basically your competition, but.

• 17:28Speaker 01

  In some ways, our competition, we're in fact friends with them, we'll come back to why. But mapping everything together, so you have one superset. The claim you're trying to support here is, if you follow this framework, then you can also see how you follow the other frameworks. But the meat of it comes down here in control activities and evidence. So control activities is like, great, you have this high level requirement, how do you turn that down to something operational? Here's what you must do. And then what is the evidence that we're looking for? And the reason we go this deep is that there's actually not that much confusion about what are the big concerns in AI. Everyone agrees to these, the question like, what are you actually supposed to do?

• 18:08Speaker 01

  And so what we found a lot of demand for is getting down to the specific evidence that people need to look for. Whether you are Cursor building something, or even J. Morgan building something, but also if you're just a risk leader at J. Morgan, like what exactly should you ask for? What can you ask for without sounding stupid? Like if you ask for some, you won't believe the amount of time a risk leader has asked for the IP rights to the online model to Cursor or something, and you're just like, sorry, what?

• 18:38Speaker 00

  You slip it in there and see if you notice.

• 18:41Speaker 01

  Exactly. Put that in the questionnaire. So that's kind of what a standard is. And we update this every quarter with these folks to keep up with the latest concerns.

• 18:51Speaker 00

  Can I double click on this one?

• 18:52Speaker 01

  Yeah.

• 18:52Speaker 00

  So first of all, the website is beautiful. Like, it's so confidence inducing, which is the whole point where like, like, okay, I know exactly what I'm signing up for when I, when I talk with you. I don't even have to talk to you. I can just see your whole certification, which is great. But like, okay, so from, from here, like D zero zero one point one config groundness filter, how does that get applied? Like, you have a person that goes through it?

• 19:17Speaker 01

  Yeah. So if you go back up.

• 19:19Speaker 02

  I did see somewhere there's like, you know, fifty one requirements, hundred thirty controls. There's like a whole.

• 19:24Speaker 00

  Right. I just like, to me, this doesn't translate into a test or eval.

• 19:28Speaker 01

  Yes, yes, yes, yes. So if you go into, I don't know, the left hand side. So actually, before we go in there, there are three types of requirements. The first is technical controls, like you must implement some guardrails. Two, there are test controls. So you must have an independent third party go and run some tests against you. I'll show you one of those in a second. And then three, there are policy controls. For example, you must have a person whose name is on the line when you guys fuck up. And you must have a plan for how you tell your customers and how you engage with them. They're kind of more traditional standard type stuff. So in this particular instance, we just check whether they in fact have a groundness filter. So we will partner with an auditor.

• 20:09Speaker 01

  So we partner with orders like KPMG or like Shellman, who go in and do the thing auditors do, which is to check the evidence. In this case, that might be a screenshot. It might be part of the code that they need to review to see that it actually, just that it exists.

• 20:21Speaker 00

  Ah.

• 20:22Speaker 01

  And then the second thing is.

• 20:23Speaker 00

  So you're not testing the effectiveness of it?

• 20:24Speaker 01

  That's the second thing. So if you go down to the third party testing for hallucinations out on the left, that's basically the next requirement. This is where we test how well does it actually work?

• 20:32Speaker 00

  Okay. And is it you testing or the auditor?

• 20:34Speaker 01

  We test them. We test them.

• 20:36Speaker 00

  That's a lot of work.

• 20:37Speaker 02

  How long does testing take? So if I want to get certified, just how long does the end and roughly take?

• 20:42Speaker 01

  Yeah, the end to end almost always is dependent on the, our customers need to learn something for us. It takes somewhere between like three to ten weeks, depending on how up to snuff they already are. So some people could show up to us with like extremely rigorous security programs. When we test them, it works extremely well. We can get that done very quick. Some people come to us and they're not that far along. We give them kind of the spec that they need to build towards and then their security teams and engineers get to work and build to meet the standard. The testing itself typically takes a couple of weeks, including the time for them to remediate. Often we'll find something that we cannot pass, where this is actually just not up to the standard. You won't pass the standard.

• 21:21Speaker 01

  And then they will need to go and implement additional safeguards or additional remediation that makes them more robust. So that they can actually kind of hand on heart, look at their customers in the eyes and say like, hey, we've done truly our very best.

• 21:35Speaker 02

  And they're certified for a year and a half quarterly updates?

• 21:38Speaker 01

  Correct.

• 21:39Speaker 02

  And yeah, it's pretty interesting. I think, you know, what's changed since? So this is certifying agents in production, right? Your customers, like you've had lovable eleven labs, intercom fin, they've all gone through the certification.

• 21:50Speaker 01

  Yes.

• 21:51Speaker 02

  What has changed? So I see you post like, you know, Q two added MCP agent, agent to agent communication. Any other things that you want to kind of highlight since the first, first iteration? What comes in quarterly?

• 22:03Speaker 01

  Yeah. So some of the changes have just been agents and not just one thing. So like, if you take agents like Cursor and compare them to Sierra, they're really quite different. Compare them to Harvey again, compare them to U R five again.

• 22:16Speaker 00

  Yeah.

• 22:17Speaker 01

  Eleven labs are all quite different. And so we wanted to design a standard that works for all of the types of agents. And we started with one that was like pretty tech based, like honestly, pretty customer support focused. That's where there's a lot of existing demand. And then over time, I've picked some of the frontier companies in each of these other domains that we could work with and build out the standard. So such that we know that the same standard works for code, it works for customer support, works for automation, et cetera. So that's been one big thing. Yeah. Then some of the things that have been top of mind recently, Mythos is bringing up a lot of concerns for security leaders. We're starting to get more and more questions around agent to agent interactions. It's very nascent at the moment, but it's starting to emerge.

• 23:02Speaker 01

  There've been a lot of questions related to open claw and MCP, again, like agents starting to interact with each other. It's really top of mind. Then as coding agents have really taken off, that's also where banks and hospitals, et cetera, are getting more and more precise on what it is they need. So really dialing in as, as it starts to be like where most of the tokens flow through in the world, getting much happier on that.

• 23:26Speaker 02

  Can you share for people that are listening that don't really think about this? Like you mentioned, there's the obvious stuff, you know, hallucination citations. What are best practices that people should do when building agents? Like if they come to you pretty ready with certificate, like, you know, they'll probably pass certification. What are the things people don't think about that they should have?

• 23:46Speaker 01

  The most important thing is that a lot of companies have not done a serious stress test. They spend most of their time, perhaps rightly so, optimizing for how does it work in the good case? The average case, how high quality is the output for the customer? And a lot of these companies are pretty new. So they haven't spent a lot of time stress testing the, what is, what is there as an adversary on the other side? What are some of the complicated corner cases that you've not really considered? So I think that's like a frame of mind and you'll see this in startups. It often takes a while until they hire, hire their first security person. And that's a whole different kind of risk surface than just building a good product. So a lot of that applies.

• 24:27Speaker 01

  Most companies actually also have the right kind of architecture. Most of them will have some kind of guardrails in place. Either some are coming out of the box from their model provider, or they've built their own filters that sits in between. They just don't work very well. The difference between putting a classifier in place that like, maybe goes and checks whether you're giving medical advice when you shouldn't and says, hey, if this looks like medical advice, filter it out. Lots of companies have that in place. The question is whether it works. And it's actually just pretty fiddly to sit down and think about all the ways in which you could ask for medical advice. Read the academic literature and what are the kinds of framings or tricks you might play to get an AI to give you medical advice when you really shouldn't. And so there's like an area of expertise that's just missing.

• 25:11Speaker 01

  So what we find is that most people have the right building box in place that it doesn't, it's not rocket science. But the finicky thing is like getting into the corners and testing whether it works. Such that you can look your customers in the eye, or maybe a bank or maybe a hospital and be like, this is, this is going to work for you.

• 25:26Speaker 02

  I see. So we talked a lot about the agent level certification. Where do you guys go from here? So announcing series A off camera, we talked about this a bit. There's the whole security risk of Fable government stepping in. You guys are kind of announcing that you're also going into model certification?

• 25:46Speaker 01

  When we do a bit of cutting afterwards, we will not yet be announcing this. But the question that is top of everyone's minds now is at the model level. And Mythos then Fable has really brought this to the fore that in addition to the commercial risk and the kind of economic security risks that are happening at the agent layer, the models are going to present risk in the national security category. The shape of the problem is very similar. You have some people that are on the hook if something goes wrong. In the case of agents, it's often security leaders in the enterprise. In this case, it's the government. They don't, haven't necessarily spent their entire lives thinking about what are the new risks that come here? What is the kind of data you might be looking for? How might you test that? But they do have to make sure that their concerns are addressed.

• 26:31Speaker 01

  You have some frontier companies that are deeply technical. They know a lot about the risks, but they fundamentally have an incentive to not always be truthful. So you have a trust gap between the government and the labs. And in every other industry, you end up with some kind of body sitting between a neutral third party sitting between those people. There's no other industry where you allow people to audit themselves. So there's going to be a need for a third party that can take the rigor of the labs to run frontier technical evals. But can also speak legible trusts in the way that the government trusts P W C to go and run financial audits.

• 27:11Speaker 01

  And they know that they output audit reports in a way that's consistent, that's easy to read, that's factual, that's trustworthy. Those two things need to be brought together. And what we've learned from our work with agents is that if you want those, that communication between those two parties to be smooth, there has to be one common standard that is public, that people can go and inspect. What are the risks that matter? Within each of these risks, what are the kind of threat models that you're really looking for? You need to specify for each of those risks, what are the guardrails that need to be in place? And what are the tests they need to run to see whether those guardrails are effective? And then you need to go and run audits that are, technical audits that are consistent.

• 27:52Speaker 01

  So if you're trying to bring trust, it's extremely important that you methodically work your way through the risks. You can't send one researcher in and say like, come back with whatever you find. You need to be able to explain exactly what you did, exactly what you tried, exactly what you did not try. And therefore, the kinds of promises you can and cannot make at the end of it. I think of Fable as a direct symptom of this problem, that the government was told that there's a risk. The government may struggle to assess just how big that risk is. They call Anthropic and Anthropic is trying to tell them, hey, actually, every model can be jailbroken.

• 28:30Speaker 00

  That's not what you want to hear, right?

• 28:32Speaker 01

  As the government, that might be hard to trust.

• 28:36Speaker 01

  And we think that a broker is the most natural solution. In other markets, you, you see something like, in financial markets, you see Moody's. Moody's goes in and they look at a bond and they output a rating. They say like, here's the evidence we found, here's the rating. We don't decide whether anyone should buy this bond or not buy this bond. Well, that depends on their risk appetite. But we do provide this common information layer that everyone can rely on. In the case of Moody's, the government points to them and say, hey, pension funds, you should probably really take care. You shouldn't risk your pensioners money. So you can only invest in triple A rated bonds.

• 29:15Speaker 01

  That means that now the government doesn't have to staff thousands of financial technical experts to rerun forecasts. Every week to see whether things are correctly rated, to get to point to some neutral third party. So my hypothesis is my hunch is that you will see a third party that sits between the government and the labs. And it could either be the government builds it themselves. So something like Casey was set up to do exactly this. And the question is.

• 29:44Speaker 00

  Sorry, I'm not familiar with Casey.

• 29:45Speaker 01

  Casey is the Center for AI Standards and Innovation.

• 29:50Speaker 01

  I won't get into the details, but it's a sub body of NIST, the typically set standards. Uh, so it's basically a government body that has AI experts. Yeah, exactly.

• 29:58Speaker 00

  Very low key.

• 29:59Speaker 01

  Exactly.

• 30:00Speaker 02

  I think, you know, it's one of those things where when you just sit back and listen, uh, look at it like, is there enough technical expertise in the government to measure, test these things right now? Probably not, right? And Fable is a result of, okay, we've had to scale back and pause things, but.

• 30:17Speaker 00

  Yeah.

• 30:17Speaker 01

  And they have, they have excellent people, uh, but they have an extraordinarily small budget compared to the scale of the challenge that's ahead of us. And I think they have a role to play. The question is kind of like, who does what? Uh, we have now outlined the jobs to be done and they're quite extensive. Every model release, there is an astounding, given that they, they take in any input, the risk surface is astounding. And so the question is really, what can only the government do? And what can the market provide here that can keep up with the pace as AI risk changes? Our perspective is that also at a model layer, the risk that people care about today are not the same ones they care about three months ago. So the pace of legislation is too slow to deal with pinpointing the risks here.

• 31:02Speaker 01

  And so we think there's a lot that the market can do to surface timely information. Ultimately, there is a bunch of policy decisions here. Is the national security risks of a model too high? That's the political answer. Uh, but what we want to make sure is that the process that produces this risk information is compatible with very fast innovation. So you don't want to, this is not a question of like, can you slow the things down? Can you keep the models locked up until for months on end, until everyone can make a guarantee? Uh, but it is just, can you, and the time it, given that the US is competing with China on releasing models, can you insert risk information that allows the government to like make rapid decisions on some of these questions? Balancing that trade off between failing to adopt AI is going to put us at risk, but also

• 31:47Speaker 01

  reckless adoption is going to put us at risk. And that's a very kind of fine balance that they're going to need, like a lot of high quality intelligence to, to make.

• 31:55Speaker 00

  Just a side mention, because you mentioned Chinese models. Any specific concerns that you're hearing from your CISOs about that? Uh, cause I guess it's free, but.

• 32:05Speaker 01

  CISOs have a bunch of concerns around data flows in general that they're really concerned about. So it's a lot of questions like, uh, if these models are Chinese, where does that, where does our data go? I think a lot of this can be addressed, but they, they come up often.

• 32:18Speaker 00

  I mean, they understand they're running on American GPUs.

• 32:21Speaker 01

  Some of them, some of them understand that they're running on American.

• 32:23Speaker 00

  They're not like phoning home every time you like call home.

• 32:26Speaker 01

  No, uh, a year ago, there was not a lot of understanding of this. I actually think, uh, you're seeing the security leaders becoming kind of AI literate at a blistering pace. And you're actually also seeing my Twitter timeline that's very AI pilled and my LinkedIn feed that used to not at all be AI pilled, kind of converge. They're both talking about Fable.

• 32:45Speaker 00

  Right, right. Yeah, that's true.

• 32:46Speaker 01

  They are both talking about whether you can prevent models from being jailbroken, jailbroken these days.

• 32:51Speaker 00

  Yeah.

• 32:51Speaker 01

  Uh, it's like national security risk, I kind of, that, that conversation is actually emerging. Other than that, I think you mostly see a kind of, uh, there's no concerns with any particular model or any particular model output, but there's a general nervousness of having critical infrastructure run on models that are not produced in America by Americans where the American government has control.

• 33:14Speaker 00

  But it doesn't necessarily show up in your framework directly or it might, I don't know.

• 33:18Speaker 01

  There's a bit of stuff in there actually on the like, the provenance of the models and disclosing that. But I think there's a bunch of use cases where running and Chinese open source model is just the, the best solution. Uh, and a concern is slightly more macro here, which is not best addressed at any particular certification level.

• 33:32Speaker 02

  Is there anything interesting that you see at the, you know, if you're trying to fill that middle gap, that mediation gap? Any interesting stuff that you guys forecast would be required other than, you know, what, what the average person might expect?

• 33:47Speaker 01

  There's a bunch of interesting questions about what is the risks that matter here? So right now, the risk of the day is cyber because it's very real, very tangible. And some of the risks that are also emerging as pretty real and pretty tangible are things like child safety is becoming both extremely important, uh, but also politically important. And then there's some of the risks that are coming down the pipeline that today feel kind of speculative, but people spend a lot of time with the models, see them coming down as things like, um, risk that relate to biology. And specifically where the models will help adversaries produce biological weapons and making that extremely cheap, extremely accessible, producing, making the chance of another COVID

• 34:29Speaker 01

  or worse pandemic. COVID was not engineered to be bad, uh, as if you were trying to do that. So I think those are some of the risks that are coming down the pipeline. Uh, I think one other thing to just note is that agents are kind of deliberately narrow. So like when a frontier agent company puts a chatbot that interacts with customers, they've really tried to narrow the topics it's interested in talking about. Such as if you ask it like, what do you think of the precedent? It will just decline, which means there's a kind of risk area is somewhat smaller for models. It is infinite. And so there's not a single expert out there who can competently evaluate the risks of cyber attacks

• 35:12Speaker 01

  and 15 year olds having month long conversations with a chatbot and seeing whether it will in fact recommend suicide or something horrendous like that. And can evaluate the risks that terrorists can use AI to produce bioweapons. The risk surface is just too big. And so the central challenge actually becomes, how do you get those subject matter experts to work within a one coherent framework that outputs one coherent report and rating that the world can gun inspect? Cause that global perspective is central, but there's not a single organization today that could produce that.

• 35:47Speaker 00

  And you would be the presumptive one when you put out your model standards?

• 35:51Speaker 01

  We think there can be one company that can, with a consortium of experts, build one coherent standard. I think we've shown that across all the enterprise risks today. We think it could be one company that could, with a consortium, specify the audit rules, basically like the inputs and outputs that all these technical experts need. What access do they need? How should they treat infra, infra security? They can look at whether they eval, evals are well produced without necessarily being able to say, hey, this is a threat or not a threat. But overall evaluating whether the evals are good, well constructed. That set of audit rules that basically comes to interface for all these experts, we think one clearinghouse could put together.

• 36:32Speaker 01

  To be clear, when I say one company, I think of it as one company coordinating lots of this in the same way that when we saw our consortium, it's not like we say, we have all the answers on agent security. What we say is we are taking on the role of eliciting all of the concerns and being the secretary that puts it together and runs a tight house such that the standard updates lockstep every quarter and that the audit reports that come out, in this case, a hundred page audit reports, uniform and crisp and clear all to the level of detail that is required for executives that need to make a clear go, no go decision. So that's kind of the role that we think we might play.

• 37:11Speaker 00

  I think in many ways you're performing the role that OWASP used to do there. And you said like, you know, competition and partners can go more into like how they partner.

• 37:20Speaker 01

  Yeah. So first of all, OWASP is basically an open source community of security practitioners that are coming together to build frameworks for addressing the latest security concerns. We think they are phenomenal at creating frameworks. We've in fact, we, first of all, we're partners with them. So we have a joint article too. We've learned a lot from them. We think they're a tremendous source of, of intelligence. What OWASP does not do is building the machine that runs third party audits such that a company like Cursor or a company like JP Morgan could get a third party to go and review them against this and say, hey, you've passed the standard. And here's the report that you can use to build trust and preempt your partners or customers questions. So they fundamentally try to do something different.

• 38:04Speaker 01

  You can, they are part of the information gathering and intelligence gathering and creating clarity. But the operational layer of turning this into promises is, is not the business they, they tried to be in.

• 38:14Speaker 00

  The standard is emerging and, and is doing very well. Was it necessary to then also do underwriting? Uh, obviously it's in the name, so presumably you thought about it first. I feel like if you just have enough consensus, you, you don't actually need the money angle, but it does help.

• 38:30Speaker 02

  I, I did want to also know, you guys are a for-profit company too, right? It's not nonprofit where there's, there's a whole business side to it as well.

• 38:39Speaker 00

  Yeah, yeah. It was, I'm just basically asking about the money part.

• 38:42Speaker 01

  Yeah, yeah, yeah. Yeah, let's get into the money part. Let's start from actually your question, for-profit versus nonprofit. In the security space today, cybersecurity, most of the standards are produced by nonprofits. I think that's an issue.

• 38:59Speaker 01

  The question you have to ask yourself is how do you create good incentives for these standards to be good and keep up? Nonprofits tend to not have these adverse profit incentives where they hollow out their standard and create a race to the bottom, but they're also not at all responsive by default to the communities that they serve. There's no process, they don't have customers that they serve where they go and ask, what do you want, what do you want, what do you want? And when you look at the overall satisfaction with the security standards today, people tend to just not like them very much. You do see in other domains that for-profit standards can serve the world quite well.

• 39:43Speaker 01

  So there are examples, like we talked about Moody's before. It's not without flaws, but it is absolutely critical societal infrastructure that gets run at astounding scale today. Your credit score, it's FICO, it's also a for-profit business. And when you go back even further in history, some of the crash testing standards came out of insurance companies. The insurance companies together funded the, founded the Institute of, Insurance Institute of Highway Safety, because they were very interested in like, how can we use standards to drive down mortality and save money? Go back, prior, our name actually pays homage to the underwriters laboratory, UL, which

• 40:25Speaker 01

  was started right around when electricity came out, houses started burning down, insurers again were paying the bill. And they were maybe also good people, but their profit incentive was, let's prevent houses from burning down, let's test all the electrical products, the light bulbs, all the light bulbs in here are probably UL tested, the toasters, et cetera. And they set up an entity to create those standards. Today, UL has a for-profit entity and a nonprofit entity. And what they've recognized, they spun out, they started nonprofit, they spun out a for-profit, because what they recognized was like, hey, actually to serve customers well, you need a for-profit entity. The lesson here is, one of the ways that the market can align incentives, so you're

• 41:05Speaker 01

  both responsive to customers and not hollowing out your standard over time, is to align it with insurers, because they fundamentally have good incentives. And so if you're a for-profit standard that works closely with insurers, you get the feedback loop in such that you're really cued into your customers, but also have their interest at heart. So that's the model that we're, the kind of inspirational model that we've learned a lot from, and that's also where the name comes from. In some ways, the term underwriting can both be associated with insurance, but it's also a broad term for like, making decisions. If you underwrite a decision, you're fundamentally kind of taking ownership for, for the consequences of it.

• 41:45Speaker 00

  Yeah, I mean, what does an insurance contract look like for AI?

• 41:48Speaker 01

  Yeah, most of the demand comes today for insurance contracts is sitting between people who've built AI and people who are buying AI.

• 41:56Speaker 00

  Yes.

• 41:57Speaker 01

  And what you want is, the reason why people want insurers involved, both for the traditional reasons, hey, if something goes wrong, we want to be compensated, but it's in particular because insurers can increase, can bring trust to the equation, because insurers will take, pay for the damages. If they're willing to write an insurance policy, that is them saying, hey, we think there is risk here, but that is manageable. And that is kind of a, their incentive aligned with the enterprises adopting it, so that's really a good signal to the market. In the same way, actually, one of the things that Waymo tried to get their first permit to even operate in San Francisco, was to get a lot of insurers to stack up a huge insurance

• 42:38Speaker 01

  policy in the case of something went wrong, not because Google can't pay, but because it was very valuable to have a third party go and look at that data that are trusted by governments, trusted by enterprises as conservative people, and say, hey, we've looked at it, we're actually willing to take some of this onto our balance sheet. So that's, that's kind of the reason why people are interested in it. What it looks like is, in some ways, like every other insurance contract, you specify, what are the perils you want to cover? How much do you want to cover them? Like, up to what limits? And what does it cost to cover that? And in the case of, if we take a really concrete example, Eleven Labs bought a first-of-its-kind AI agent insurance policy.

• 43:21Speaker 01

  They work with some of the biggest enterprises that work with governments, they're really interested in going above and beyond and making promises to their customers. So they wrote a policy that covers just some of the core concerns that our customers have been asking about. And the crucial thing was really to get Lloyds of London, the world's oldest insurer, one of our partners, to look at this data and be that third party alongside us to say, hey, we think there's something here that's worth underwriting. And that's actually what it looks like. And so they will show that contract to their customers and they can see how much they're covered for, they can see what exactly it covers, and that will also probably change next year, they will want to write an insurance policy that might cover more.

• 44:04Speaker 00

  When you say Lloyds, is it reinsurance? Or are they sharing somehow at the same level?

• 44:10Speaker 01

  Yeah, so typically the way new companies get into insurance is that they partner with insurers, such that the insurers take the majority of all of the financial risks. Fundamentally, if, if, if insurance is useful because it brings trust, you have to be able to pay the bill. Lloyds of London is 400 years old, they've never not paid a claim, they're extremely trusted. And what Lloyds of London struggle to do on their own is to figure out which of the risks are real, what should we be looking for, what are the kind of technical controls and running the tests? So they use AEC one as kind of the underwriting framework, and we produce a bunch of AEC results that then directly feed in to inform the pricing.

• 44:54Speaker 01

  So this means that Eleven Labs customers know that that payment will be there, they don't have to look to our Series A and see like, do we think they have enough cash on the balance sheet? They will look at Lloyds.

• 45:04Speaker 00

  Yeah. And Lloyds, like famously very creative, I think I remember some headline like, they insured Jennifer Lopez's butt or something.

• 45:13Speaker 01

  Correct. And I think, was it David Beckham's right foot?

• 45:16Speaker 00

  So, yeah.

• 45:17Speaker 01

  And stuff like this.

• 45:18Speaker 00

  So like, clearly not a large dataset.

• 45:21Speaker 01

  Exactly. It's actually a remarkable institution that's both kind of has some of the truly old-school virtues of having been around for a long time. They like really, they really operate like a trusted entity, and they have appetite to figure out the future. And I think there's a lot of recognition that both there is like tremendous amount of risk in AI that is poorly understood today, so getting into this business carries real risks. But also, this is where lots of the risk exposure will happen in the future. This is the one market where risk is truly growing. This is the one market that will also take out some of the existing markets, take like auto insurance. When there are no human drivers, how's that market going to look?

• 46:05Speaker 01

  Well, it's going to change. How are you going to assess?

• 46:08Speaker 00

  You want to insure Waymo?

• 46:09Speaker 01

  I, all I'll say is, the principles for how you insure Waymo are very similar to how you insure other kinds of AI. So getting crash testing, that's what we do for customer share or lovable, that will also need to happen for Waymo, which is not how you do it for human drivers. So there's this growing awareness that the world is changing very fast, and the only way to learn how to underwrite AI is to write some policies. You may incur some losses, and think of that as R and D expense, really, but the question for them is like, who are the trusted technical partners they can get into this business with that can help them navigate and make sure they don't make kind of foolish mistakes? But also, who is willing to hear the wisdom that they have? They've done this before, they've seen, they were there when cyber came out.

• 46:51Speaker 01

  So there are lots of ways in which AI feels completely new, but there's also lots of ways in which risks look the same. And so there's actually tremendous amount of wisdom sitting in some folks that may have gray hair, but really have like a keen sense of how to quantify risk.

• 47:07Speaker 00

  Yeah. And the number is, so it's basically like, I want $50 million of the coverage against these perils, and Lloyds will give you a quote on it, and then you, you have like a small markup or something, and then you, you turn it around and do that? Is that as simple as it is?

• 47:23Speaker 01

  You, you basically share some of that premium. So X percent goes to the people who do the, the pricing of it.

• 47:28Speaker 00

  It's kind of like a, it's kind of like a merchant bank for insurance type of thing.

• 47:33Speaker 01

  Exactly. You basically split the fee, and you can think of the insurance supply chain as like, they're bringing the capital. There is doing the pricing and there is doing the distribution. And typically, you will pay out some X percent of premium here, Y percent of premium here, and the rest of it will go here.

• 47:46Speaker 00

  Does all the insurance world work like this? Or is there some point at which like, so, so right now you have equity capital.

• 47:51Speaker 01

  Yeah.

• 47:52Speaker 00

  At some point, maybe you start raising debt or whatever. And then you have enough of a bank account and enough history. Let's say you've been in operation for 10 years.

• 48:00Speaker 01

  Correct.

• 48:00Speaker 00

  That you don't need Lloyd's anymore.

• 48:01Speaker 01

  That's totally an option, and I could see some worlds where that makes sense. Specifically, if there are risks that we feel high confidence that we'd want to insure, where the incumbent insurers are too slow to find appetite.

• 48:13Speaker 00

  Okay.

• 48:13Speaker 01

  Or simply struggle to evaluate it, so that they don't want to do it. But by and large, in general, you do not want to compete with insurers on bringing risk capitals into the game for two reasons. One is, that's fundamentally a cost of capital game. They have extremely low cost of capital, startups have high cost of capital, by and large. And two, you wanna hedge your bets. And it's very helpful then to also have a portfolio of home insurance, of car insurance. And we're not about to become a car insurer nor a home insurer. So they have some natural advantages, which makes it much more likely that we'll partner.

• 48:48Speaker 00

  Yeah.

• 48:48Speaker 01

  And they bring that, the capital at scale, and we bring the technology.

• 48:51Speaker 00

  You're, you're gonna work with them for a long time.

• 48:53Speaker 02

  How are the discussions with the insurers as well? So basically, they're going off of your certification, right? They're trusting the diligence on you, that your certification is valid, you tested the right things. And they're backing the money that, you know, you have the right testing in place. So any interesting takeaways from working with insurers?

• 49:12Speaker 01

  I think the, maybe the first thing is they feed into the standard as well. So if there are things that they feel like they need, that they're not seeing, we are also taking that as input into the standard. Because fundamentally, we think a good standard is one that creates a really healthy promise ecosystem. And we think insurers are a critical part of that. And again, they are the most well incentivized to, they see all the lost data across, any particular CISO knows their particular concerns. Insurers see the concerns across the entire portfolio and often have direct access to like, what exactly happened? Who was at fault, et cetera, as they do part of their forensic. So they're actually like a great source of intelligence on this. One of the big takeaways from cyber insurance, which is a market that didn't work that well,

• 49:54Speaker 01

  was that the insurance and the technical expertise was not married up. Well, our conviction is that standards have to precede insurance. Fundamentally, what everyone first and foremost want, whether you are a CISO at Jig Morgan, or a CISO at Cursor, or a underwriter at Lloyd's in London Syndicate, is you want to not have an incident in the first place. You want to know that the risk is well managed. And only then does insurance start to make sense. So we'll see the standard ecosystem basically run ahead of the insurance. And the reason why we, you asked us kind of why I also do insurance, this is kind of proving what we think a whole promise, confidence, infrastructure ecosystem needs to look like.

• 50:37Speaker 01

  And we think it's very compelling to bring that to life, even if we think the standard is kind of the, the core linchpin that unlocks the rest.

• 50:44Speaker 00

  There's been no claims yet, right?

• 50:45Speaker 01

  Nope.

• 50:46Speaker 00

  This is one of those things where, you know, if people haven't really worked through what it means to cover things.

• 50:52Speaker 01

  Yeah.

• 50:52Speaker 00

  So for example, I pay Cursor $20 a month.

• 50:55Speaker 01

  Yep.

• 50:55Speaker 00

  And I write, I vibe code something that makes a plane crash, causing $200 million of a damage.

• 51:02Speaker 01

  Yes.

• 51:02Speaker 00

  Do I claim $20 or do I claim $200 million?

• 51:06Speaker 01

  Yeah. So, these are all great questions. And fortunately, kind of, all of insurance and legal history kind of helps answer some of those questions. I think the first thing is people have limits on their policy. So if you want to claim $200 million, you have to, someone has to have paid a lot for that insurance policy upfront to have $200 million of coverage. And ultimately, the way this works is that you start from a lot of uncertainty. This is not just an insurance, but also like, can you use, can Anthropic use books on the internet to train up? Well, they can go and look at precedent, they can see. But ultimately, these, these things get settled in court and you hammer it out over time.

• 51:47Speaker 01

  So you start from this like place of ambiguity, which is both why insurance can be hard to do early on, but it's also why people want insurance, because ambiguity slows down adoption.

• 51:57Speaker 00

  Yeah.

• 51:57Speaker 01

  That also sits at the heads of the.

• 51:59Speaker 00

  Yeah. In some ways, actually, the first incident will help to establish a lot of this.

• 52:05Speaker 01

  Exactly. And, and there have been a number of incidents out there that have just not been insurance covered. So take the now old example from Air Canada, where.

• 52:14Speaker 00

  I was gonna bring that up.

• 52:15Speaker 01

  Hallucinated a refund policy. And the question was, Air Canada in that case were like, hey, we have nothing to do with this. This chatbot messed up, but like, sorry. And the courts were like, no, if you put your chatbots to interact with your customers, they make legally binding promises on your behalf. That is now precedent for everything in the future, where you will, if someone were to deploy a chatbot like that again, there, you should not expect to be able to just pawn off and say, sorry, my chatbot lied. It's nothing to do with me, I bought it from OpenAI. No, if you're putting this in front of your customers, you are taking responsibility for it. And so every court case, whether insurance is involved or not, clarifies liability. And liability is kind of the foundation for insurance.

• 53:01Speaker 01

  There's another reason why standards and insurance come together. Liability, I'll go on a little tangent here.

• 53:06Speaker 00

  Please, please.

• 53:07Speaker 01

  Liability, often one of the core concepts is whether someone was negligent. Should they have seen this? Should they have prevented this? And the question you, how do you judge that? Well, you basically judge whether they've met their duty of care. What does that mean in practice? Well, often they look to standards. So if there's a standard that is broadly adopted, that says you must have a groundedness filter, or you must have a jailbreak filter, it becomes way harder to claim ignorance that these things existed. And so setting standards help clarify liability. Courts will often point to standards and being like, well, this seems like best practice to do, is there for everyone to see. So there's another way in which like, standards are kind of civilization infrastructure that insurance can then build on, which promises can then build on.

• 53:53Speaker 00

  I totally get that. We don't have to get certified to, to write these, to, you know, make these like bots and all these.

• 53:59Speaker 01

  Correct.

• 54:01Speaker 00

  But like, basically, whenever we get, go for the audit, I think people like start to shape up and, and all this, all this stuff. I wonder if like, that means that you don't also then become like the approving authority for me to ship to production? You know, like, yes, you check once per quarter, I want to ship once a day.

• 54:19Speaker 01

  Yeah.

• 54:19Speaker 00

  And I don't know when one of my things breaks, like, one of your certifications or not.

• 54:24Speaker 01

  So there, there's a couple of things. There's a couple of requirements in there that relate to how do you yourself, where you have to tell your customers.

• 54:33Speaker 00

  Ongoing monitoring.

• 54:34Speaker 01

  How are you yourself testing before you make at least major releases? We don't go and audit people every day, but at least there is now a trail where if you do a major mess up, then your customers, they come and ask you, hey, you promised me that you were gonna run these evals yourself. And for lots of them, most, most of the PRs that people merge will not fundamentally alter the product experience, but some of them will.

• 54:56Speaker 00

  And sometimes you don't know.

• 54:58Speaker 01

  And sometimes you don't know, and this is also true. And this is also, there's some inherent risk that everyone kind of, everyone knows that when they buy software, there can be bugs, and this, this is just part of it. But what they can, if you're selling to mom and pop shops, they may not care. There is like, well, I, I want to use your tool, so I'm just gonna willing to take that risk on. If you're selling to a big bank, they might be like, sorry, we're making promises to our customers. If you can't make a promise to us that we can pass on, we don't want to work with you. Then it's up to you to say, do I care for my agent to get used as critical infrastructure in this nation? If so, at least I can make promises about what process I run. And then we can go and test it every quarter to be like, well, this is seem like, it's still, kind of, it still meets the standard.

• 55:42Speaker 01

  So from our perspective, it's kind of a way to, big companies by default kind of have some amount of trust when they ship AI. If you're a young company, if you're just starting out, by default, you have no trust. And there are very few places where you can go and get trust. So one of the things that most of our customers did before they started working with us is that they would make their own security blog posts. That's great. But also, who's gonna trust you saying, we're so secure? Like, anyone can write that. But it's very hard, where do you go and get that trust?

• 56:08Speaker 00

  Yeah.

• 56:08Speaker 01

  And so I think making the standards more legible makes it easier for smaller companies to prove that they're doing what they ought to be doing. Because the default assumption is that it's the Wild West.

• 56:20Speaker 02

  Is there a roadmap you have of like, there's a lot of work to be done here, right?

• 56:25Speaker 01

  Yep.

• 56:25Speaker 02

  This is the first one.

• 56:26Speaker 01

  Yeah.

• 56:26Speaker 02

  Anything on the roadmap of what you see is next, what's coming, what's, what's missing?

• 56:32Speaker 01

  I think when we, when we zoom out, as you want, deals with agents, we will, next up, we will deal with models. Next up from that, we will deal with robotics, of which in some ways Waymo is the first robot. But the exact same problem is gonna be, someone's gonna develop a robot, someone's gonna need some promises, they're gonna struggle to make the promises. And you see this playing out when like, if you think Fable concerns are bad, like see when Waymo hits a dog. And as if people lose their mind, imagine when first robot knocks off a toddler off a kitchen table.

• 57:02Speaker 02

  Uh, yeah.

• 57:03Speaker 01

  You're gonna see some real strict liability.

• 57:07Speaker 02

  I mean, you could see it, right? Like Cruz got fully.

• 57:10Speaker 00

  Destroyed.

• 57:10Speaker 02

  All permits are gone, yeah.

• 57:12Speaker 01

  Correct. So physical AI, the level of stringency just goes up and up and up and up. And so that's kind of like the big picture, agents, models, robotics. I think within agents, the current set of agents are well covered by this. But as the technology progresses, as agents get longer horizons, new types of failure modes will emerge. And so it's mostly of, can you make sure that the standard keeps up when they appear? And you also start to see new modalities, like today, world models is mostly a kind of a research question. There's no one who's really using it. But that will also bring in just new kinds of ways to create value, but also more risk surface that no one knows how to grab it with today.

• 57:55Speaker 01

  You'll start to see true agent to agent interactions that are not mediated by humans. There's gonna be a bunch of interesting questions. You're basically gonna need a new legal system, how do they build trust amongst each other? How, one of the core things, when humans trade with each other, is that you know that you have recourse, you can sue them. How do you make sure that there is a persistent balance sheet behind any agent such that if you trade with it and it screws you, you know you can get your money back? Those are some of the questions we're gonna have to deal with. And the technical testing of multi-agent systems is also gonna be interesting and complex.

• 58:29Speaker 00

  Very fun. Are there any perils that are uninsurable right now that people wish that you would?

• 58:35Speaker 01

  Yeah, one of the places where there's a bunch of appetite for insurance and not a lot of a lot of demand, but not a lot of supply is when it comes to copyright.

• 58:46Speaker 00

  Oof.

• 58:47Speaker 01

  In some ways, copyright is kind of mundane. It's always been an issue. There's a couple of reasons for this. The first is people who have trained on copyrighted materials almost always know that they've done that. So if you want to buy insurance for it, it probably signals that you might be a high-risk customer. The people who are most interested in getting insurance for copyright infringement are the people who are most likely to have committed copyright.

• 59:11Speaker 00

  It's like a lemon problem.

• 59:13Speaker 01

  Exactly.

• 59:13Speaker 02

  I actually think there's another side to it too, right? Like if you're building on something, so say I'm using an open model. I don't know what it's trained on, right?

• 59:21Speaker 01

  Yes.

• 59:21Speaker 02

  And how far down that chain does copyright go?

• 59:24Speaker 01

  Yes.

• 59:24Speaker 02

  Am I liable to take down my product because company X trained in copyright?

• 59:29Speaker 00

  But there's safety in numbers. If everyone's doing it, then you.

• 59:33Speaker 02

  I mean, I would say until, you know, Fable has rolled back from everyone I use there, right?

• 59:37Speaker 01

  Yeah, at least a hard question. I don't have the answer to it, but I think your intuition is right and kind of like,

• 59:44Speaker 01

  what is the kind of duty of care? And people don't today think of it as customary that you go and you like dissect your open models, train data, and you check everything. In fact, lots of people use them. It's seen as kind of generally acceptable to not check for this and therefore we're not going to hold you specifically.

• 1:00:02Speaker 02

  We also really can't, right? We don't.

• 1:00:04Speaker 01

  Exactly.

• 1:00:04Speaker 02

  We don't know the training data.

• 1:00:05Speaker 01

  And so you can ban it, but I think no court is going to get a copyright question and be like, this actually needs to get banned.

• 1:00:09Speaker 00

  Hire Nicholas Carlini and he can extract it.

• 1:00:12Speaker 01

  Exactly, though he's in short supply.

• 1:00:15Speaker 00

  Yeah, he only has so many Carlinis.

• 1:00:17Speaker 01

  Exactly. So I think this is also fair that in the case of labs, there's a lot of interest for this. But the thing that makes lab want it is what makes insurers suspicious of it. And so you have a lemons problem.

• 1:00:29Speaker 00

  Yeah. Is there like a theory of insurance where adverse selection dominates the risk sharing aspect of insurance? Like where does this teach us insurance?

• 1:00:40Speaker 01

  A lot of insurance does come back to like practical versions of Microcomics 101.

• 1:00:45Speaker 00

  Yeah, it's like, this is why you need to pull health insurance. Because if you make it too hyper-specific, then only people who are guaranteed to get the disease will sign off your insurance.

• 1:00:56Speaker 01

  Exactly.

• 1:00:56Speaker 00

  Same thing.

• 1:00:57Speaker 01

  The core problem is one of information asymmetry. If you are buying insurance, know something about their risk that insurers do not know. And so the question is actually, and this comes back to the same problem is if you rely, you can break a lot of these information asymmetries if there is some kind of testing that reveals the underlying true risk. And so if you were able to, in the case you mentioned, have good diagnosis of whether someone has it or what the probability is that someone has it, that the insurers trust, then they might be willing to insure it. But if they don't, if there's no kind of common information, then only the patient will know that sort of breaks it down. So the question is again, how do you create credible signaling between players? This is also the whole reason why Moody's exists. Moody's just does credible signaling.

• 1:01:43Speaker 01

  That's also why Moody's could never, Moody's has to be independent. If Moody's was owned by JPMorgan, then JPMorgan cannot use it as a signaling mechanism. So a lot of the basics of standards and certification are just communication devices. There's just a trust gap. And that's where you have to think about what are the incentives of the messenger? And one another way you can break a lot of this is through transparency. If you are transparent in how you operate, you just cannot mess with others nearly as easily. You make it much more costly. And that increases trust. This is one of the reasons why there's a change log here. Every little change.

• 1:02:18Speaker 00

  Yeah, yeah.

• 1:02:18Speaker 01

  You can go back and find, and it means that if we were to make this standard worse.

• 1:02:24Speaker 00

  Oh, wow. That's a lot of changes in one update.

• 1:02:27Speaker 01

  Yeah.

• 1:02:27Speaker 00

  Okay.

• 1:02:28Speaker 01

  And a lot of this is just as things get clearer, you can see a lot of clarifications, you can see some revisions. As things get hammered out, you want to change this. But if you make it all public, you make it much harder to mess with people. Or at least you become found out very easily. And so this is a way of increasing, so reducing the information asymmetry is by just making more of the information public.

• 1:02:49Speaker 02

  I like how you do know when future versions are coming. So I guess it's quarter.

• 1:02:53Speaker 00

  I mean, they just cut it quarterly.

• 1:02:55Speaker 01

  Yeah, yeah, yeah.

• 1:02:55Speaker 00

  Not that surprising.

• 1:02:58Speaker 01

  Yeah, but this is also a promise. Like if we now don't deliver on July 15th.

• 1:03:03Speaker 00

  I mean, you can just batch it up and then whatever you got, you just ship it up. Yeah.

• 1:03:06Speaker 01

  But it's kind of like, we deposit some amount of trust every time we meet this commitment.

• 1:03:11Speaker 00

  Yeah.

• 1:03:12Speaker 01

  And in the startup land, it feels easier to ship a new version of a standard once a quarter. And the enterprises who are used to this like decade-long cycle, we often get met with like incredulity, like there's just no way. And then you show them the change log.

• 1:03:28Speaker 00

  One thing I wanted to also like try to really think about is, you know, you said something about how if you have tests for the thing, then you can insure it.

• 1:03:35Speaker 01

  Yes.

• 1:03:36Speaker 00

  Right? And so really, what your standard is, what AI UC is, is establishing a framework for the audits to happen so that you can at least test like all these like baseline standards or care have been met. And therefore people can insure against standard risk that everyone has. I wonder if like there needs to be, you need to develop other tests. We've covered Mechenturp in the past. Any interest in that? Or are there other kinds of tests that we're not thinking about?

• 1:04:02Speaker 01

  Yeah, I think Mechenturp is a big one. A lot of interest in that. I think everyone would agree that there's like promising scientific potential. We're still a while, a little bit away at least, from this being like commercially available on-demand such that there's like now a selection of vendors you can go to.

• 1:04:26Speaker 00

  Goodfire would say it is commercially available.

• 1:04:28Speaker 01

  Exactly. We'd agree with them. We think that the work that they're doing is tremendous.

• 1:04:33Speaker 00

  Yeah.

• 1:04:33Speaker 01

  We're not quite at the point where we could like literally require it, but it's the kind of thing where you can imagine relatively soon you could put in an optional control for if people use Mechenturp as a way to reduce risk, you at least get credit for it. We can't require it because it's going to be hard to require everyone to become Goodfire customers.

• 1:04:49Speaker 00

  What good does credit do me? This is a pass fail, right? Do I care about credit?

• 1:04:54Speaker 01

  It's a pass fail, but it's also a hundred-page audit report that you'd be surprised at how much Shakur Lee is actually sit down and digest this stuff.

• 1:05:02Speaker 00

  Okay.

• 1:05:03Speaker 01

  And I promise you that if someone is using Mechenturp today, they will have a slide on it.

• 1:05:10Speaker 00

  It is cool. It's fancy, yeah.

• 1:05:12Speaker 01

  But it's just easier if you have a third party saying, yeah, they have Mechenturp.

• 1:05:15Speaker 00

  And just to like, just to flesh it out for people, people who have been following our Mechenturp podcast. It is literally like, oh, you're using, you know, GPT-OSS. It is activating these three dangerous things. We monitor for it and we log it out in whatever tool of choice. Greswan has like signal or whatever. And that's it. That's the Mechenturp-based activation signal. Okay.

• 1:05:38Speaker 01

  Yeah, yeah. So I think Mechenturp is interesting and I think if that promise truly comes to fruition, you can make stronger promises than you can with evals. And so I think that's very compelling. Another thing that I think will become increasingly important is just kind of good old school monitoring. And slightly after the fact, one of the things you're seeing with evals, some of the challenges that are emerging is that the agent is starting to become aware that they're being eval.

• 1:06:04Speaker 00

  Evo awareness.

• 1:06:05Speaker 01

  Exactly, which is a problem. It means that they basically, if they know they're being watched, they won't do the thing that they think they get punished for. And by default, unless you know how to kind of reduce eval awareness, you should trust evals less. And one of the kind of truest things, monitoring like is the source of truth. Did you in fact give medical advice? And how quickly do you know? How often have you done that in the past? How fast do you respond? How often do you detect it? How fast do you detect this? So I think that is also a paradigm. It's slightly more intrusive. You actually will look at some customer data.

• 1:06:42Speaker 01

  But I think will become more prevalent over time.

• 1:06:45Speaker 00

  People talk about this, like, we should not write about eval awareness because it's going to leak into the data set and then be like, we should just, we should like never talk about it, only meet in person and like talk offline, unrecorded, like.

• 1:06:57Speaker 01

  Did you guys see the Anthropic research where, I think this is literally, Anthropic did that test. They took, I can't remember the details here, but they ran some studies on misalignment. And then they took out the training data that related to less wrong discussing misalignment and they ran the same test again. And the failure rate went down. So it in fact was some evidence pointing towards it had learnt. Either the ability or the propensity to do that.

• 1:07:27Speaker 00

  Yeah, I mean, there's the hypothesis effect, then there's like the Luigi, while Luigi effect.

• 1:07:31Speaker 01

  Correct.

• 1:07:32Speaker 00

  Which is like, you are, the more you try to train for it, you create the opposite.

• 1:07:36Speaker 01

  Yes, there you go. That's exactly it.

• 1:07:38Speaker 00

  In some ways, I think the very successful topic is a result of hyperstition, like the fact that you wanted this thing to exist in the world and now it does. But like, then it also creates the opposite as well. Like, I think, I think people who are maybe newer to this space don't remember Waluigi, but like, I do think it's very, very important for understanding that when you train for a thing, you also train the opposite of the thing, because it's just a bit flip.

• 1:08:02Speaker 01

  Yes, yes.

• 1:08:04Speaker 02

  I think, you know, just going back to where we were at, like, there's a lot more than just mecan terp that there's value in just having added, right? So your version of how fast can you measure stuff? Do you have logging? Do you have evals? You know, do you see other parts of the stack, like the inference providers that you use, the services? Okay, am I using Chinese model on their home API? Am I using through certified vendor here? Am I hosting myself? What am I doing on the inference engine side? There's just like so many levels of stuff that gives, you know, information that you can standardize out, right?

• 1:08:36Speaker 01

  Yeah, and you also see increasingly, in addition to just the basic chatbots, you're increasingly seeing big companies adopting agent platforms where they're building on top of Google's Agent Studio, et cetera. That comes with a bunch of like.

• 1:08:52Speaker 02

  Managed agents.

• 1:08:53Speaker 00

  Managed agents, everyone has managed agents.

• 1:08:55Speaker 01

  Exactly.

• 1:08:56Speaker 02

  And there's even levels. You can host your own managed agents, OpenAI's Agent SDK, or hosted by Anthropic, or Google does both.

• 1:09:03Speaker 01

  Correct. And then these are just ways to kind of strengthen the security guarantees you can make. And in some ways, this is kind of bread and butter enterprise security. They like, they love to host things on their own premises because it gives them a really sense of control. And I think you'll, you'll see, just like you do in every other enterprise market, if you really sell to the enterprise, you start to compete on some of these security features. And this is also happening AI, unsurprisingly. And I think you're seeing some amount of enterprises wanting, enterprises are really grappling with, the thing that makes agents useful is their stochastic. And the thing that makes them really hard to adopt is their stochastic. And these are just intentions.

• 1:09:46Speaker 01

  Leaders come out on different sides of that table, in part depending on how much the CEO is trying to get the stock price to grow up by saying they're AI native. And that we must be willing to take the risks. We see, we actually see phenomenal tension in the heads of the CISOs of the Fortune 1000, where on the one hand, you have a CEO saying, we must adopt, otherwise we're becoming irrelevant. And if we fuck up, you're fired. And that's kind of like the core emotional tension that we see showing up again and again and again and again. And one of the core problems that we solve for them is to take that abstract emotional concern and turn it into a framework, in some ways, just pride and clarity to that concern.

• 1:10:23Speaker 02

  So anything in here, so something I think we kind of skipped over, we talked a lot about agent language model, skipped over world models.

• 1:10:31Speaker 01

  Yeah.

• 1:10:32Speaker 02

  You guys have voice, which is interesting with Eleven Labs. How about generative media? So, you know, generating images, videos, that's a category that actually has a lot of usage. Is there anything in your current policy? Is it separate policy? How do you see that space? Yeah, it's like, we did talk a bit about copyright, so.

• 1:10:49Speaker 00

  Music.

• 1:10:49Speaker 02

  Yeah, music as well.

• 1:10:51Speaker 01

  Yeah, I think a lot of the concerns that come up there either relate to copyright or there's a lot related to, let's call it broadly, safety. So like, this could be not safe for work or just very graphic materials. Are kind of some of the core things. We have done some work on this. There's a little bit in the standard as well that deals explicitly with that. Video, we've not done a lot in yet, and I think for proper production, that is still, especially proper production without a human in the loop, that's still got some ways to go. It's obvious that it is coming. But it's very rare that it's like one-shot deploy a video to the internet, but eventually that will also happen.

• 1:11:33Speaker 02

  We see like, you know, Luma has Luma agent, or it's still pretty human in the loop.

• 1:11:37Speaker 01

  Yeah, and that just makes complete sense as the technology matures and over time, it will become so good that people will not want to slow things down by having a human in the loop. And then the need to make promises will grow.

• 1:11:52Speaker 00

  Why not just have prediction markets and everything, right? It's very EA adjacent.

• 1:11:56Speaker 01

  Yes.

• 1:11:58Speaker 01

  The core thing is that the people, prediction markets rely on public information. There is not a lot of public information. It's just insiders trading on each side.

• 1:12:09Speaker 01

  That's illegal.

• 1:12:10Speaker 02

  There's leaked information.

• 1:12:12Speaker 01

  There is leaked information. The core challenge is that often you have private sensitive information and you need to convey confidence and trust around that. And you kind of, of course, for some claims, like can any model be jailbroken? You could rely on public evidence because there'll be lots of people being like, well, there's tons of studies and actually they all can, so that resolves fine. I think that's good. For, hey, this new unreleased Mithis model, how capable is it actually? Prediction markets have not a lot to say, because actually just no one knows. And so I think that's the core place where some of this breaks down, is that actually lots of the world's information that guides some of these high-level decisions is private and often also just not known.

• 1:12:56Speaker 02

  I think the thing with prediction markets that people like is it's not, it's not answering the broad question, it's a specific, right? So will a model do this by this date? Or is a model capable to do this by then, right? So that's a little distinction then.

• 1:13:10Speaker 01

  Yeah, and often the most interesting question, if you're, say, the head of security at a bank, the question you're really trying to answer is, will this product, this agent, do this bad thing that maybe primarily I care about, specifically in the setting that I care about? And the question is like, what's the closest, that information may not exist anywhere. So prediction markets aggregate existing information. This information may not exist and you want some very specific and you're willing to pay for it. That's kind of where a third-party audit comes in. We also don't really use prediction markets to figure out whether public companies have committed fraud in their books. You use audits. You probably could, but the information is just not available. And if so, it would be like just trading on vibes.

• 1:13:54Speaker 01

  I actually would have been really interesting to see whether prediction markets, 2001, would have predicted Enron going bankrupt and they kind of, could you have told, could you have sensed from like the craziness of the CEO or some other traits that they were more likely to cook their books than others?

• 1:14:10Speaker 00

  Or enough insiders leak it than that you.

• 1:14:12Speaker 01

  That could also be.

• 1:14:13Speaker 00

  Right, right, which is like, I mean, that's the sort of the ideal dream of prediction markets, you have liquid markets and everything.

• 1:14:19Speaker 01

  Yeah.

• 1:14:20Speaker 00

  And then you can compose your exact set of risks to offset.

• 1:14:24Speaker 01

  Yes.

• 1:14:25Speaker 00

  Right?

• 1:14:25Speaker 01

  Yes, yes, yes, yes. Yeah, and I think like prediction markets will bring lots of new information to it. And so the thing is mostly not like, which one is it? And more like, what are the types of questions that prediction markets are really good at? And what are the ones where the information doesn't even exist for insiders, such that no one can in fact trade on it and needs to get generated?

• 1:14:43Speaker 00

  Okay, one self-serving question and then one open-ended one on like the future of AI UC. Self-serving question would be, so you have your standard, right? I run, you know, a large AI generic conference, like there's been a lot of talk about us certifying AI engineers.

• 1:14:58Speaker 01

  Yep.

• 1:14:58Speaker 00

  Training programs, level one, level two, level three. I was a CFA myself, so I know what, that's what the finance industry does.

• 1:15:04Speaker 01

  Yes.

• 1:15:05Speaker 00

  Would it help if I had AI engineer level one, level two, level three, and then they would like work with these guys? I don't know.

• 1:15:12Speaker 01

  If you think of the highest level objective as like accelerating secure deployment of agents, then that would totally help. Because one of the things that happens often now is that folks build agents, they bring it to the, to the decision maker, and the decision maker surfaces a bunch of security considerations that they'd not thought of, and now it's not built to spec. Now you have to go and re, like add these filters, et cetera. So if you shifted that left, like, if everyone knew what the spec that they were building to, if everyone knew the grading scheme.

• 1:15:41Speaker 00

  Yeah.

• 1:15:42Speaker 01

  That'd be awesome if they're already trained, so by default.

• 1:15:44Speaker 00

  You're the grading scheme, right? I don't get to set the grading. You guys, you set the grading scheme.

• 1:15:47Speaker 01

  We set the grading scheme, and I think what's valuable is like, if you can turn this into.

• 1:15:52Speaker 00

  Training programs.

• 1:15:53Speaker 01

  Training programs.

• 1:15:53Speaker 00

  Yeah.

• 1:15:54Speaker 01

  Such that people.

• 1:15:54Speaker 00

  Which you're, you're not doing.

• 1:15:55Speaker 01

  We're not doing that. I think there's value in doing it.

• 1:15:57Speaker 02

  There are others doing, I mean, not to interrupt, interrupt, but you know.

• 1:16:00Speaker 00

  Yeah, yeah.

• 1:16:01Speaker 02

  This company has their.

• 1:16:02Speaker 00

  Anthropic also has like a CCPA thing.

• 1:16:04Speaker 02

  Yeah, yeah, they want, they want 100,000 deployed certified consultants, right?

• 1:16:09Speaker 01

  I basically think it's good for, we will accelerate adoption if we have more people who know how to build secure agents. And we are not working on the side of training people at the moment. I think it's like very aligned with our mission. We only have so much attention.

• 1:16:24Speaker 00

  I'll tell you why I haven't done it.

• 1:16:26Speaker 01

  Yeah.

• 1:16:26Speaker 00

  It's not like I haven't thought about it before. It's just being prescriptive.

• 1:16:30Speaker 01

  Right.

• 1:16:31Speaker 00

  About like, well, this is what you should know, therefore like, the stuff that I didn't include is why you don't need to know.

• 1:16:35Speaker 01

  Yes.

• 1:16:36Speaker 00

  And I'm like, that sucks.

• 1:16:37Speaker 01

  Yes, yeah, yeah, yeah, yeah.

• 1:16:39Speaker 02

  But I think it's like, you know, the, the very interesting defensible thing you guys do is your opinionated 100-page report of here's what matters, right? Here's the like prescriptive definition of the requirements you need to be certified, so.

• 1:16:55Speaker 01

  Yeah, and I think that's a choice. I think basically that's a, that's a choice, and I think that serves some audiences very well, where if you're trying to deploy this into a bank or a hospital, et cetera, clarity of those boundaries is extremely valuable. There's lots of other settings where being much more experimental, much more trying it out, is just the better fit. And so to me, this makes a ton of sense. Also, you'd have to rewrite your curricula every fricking three months.

• 1:17:20Speaker 00

  It's fine, I do that. It's okay. But yeah, no, for me, it's actually like genuinely, like, the, the consequences of getting it wrong and affecting somebody's career is, is a big responsibility.

• 1:17:35Speaker 01

  Yeah, yeah. I think that's exactly right. And I think a lot of our work actually goes like, we don't want to carry, we also don't think ourselves are able to carry the kind of, the true north of what's like, secure, not secure. But we can coordinate the forum where you list it all of that.

• 1:17:52Speaker 00

  Yeah, your consortium is fantastic.

• 1:17:54Speaker 02

  This can be crowdsourced in a way, like for your example, for what is AI engineer certification? This is a pretty big podcast. There's a lot of takes that people can have and, you know, discussions that can.

• 1:18:04Speaker 00

  And people reasonably disagree. So who am I to say like, that's a correct question, that's a wrong question?

• 1:18:09Speaker 01

  Yeah.

• 1:18:09Speaker 00

  Right? So like, I don't know.

• 1:18:14Speaker 00

  Vent your frustration to someone that's done it.

• 1:18:16Speaker 01

  Exactly. And I think there's also you.

• 1:18:21Speaker 01

  It matters a lot about the promises. So if the promises, hey, if you've taken my course, you will not fuck up, you can't make that promise. Clearly. You could make a promise of like, here's the, some important things that everyone should at least know. And then you have to fill out the rest there. At least the promise changes. Of course, there's some subtlety in how do you communicate this stuff so people really get it. But I think it's important to dial in. And we have a section in our standard like, what is the promise and what is the promise not? Because it's impossible to guarantee that nothing will go wrong. If you need to guarantee that nothing will go wrong, you cannot work with frontier AI. But you can make some claims.

• 1:18:56Speaker 00

  Yeah, for sure. Cool. Wanted to end with open ended. Where is AIUC going? I think you talked about model stuff, robotic stuff. I just open ended like, you know, what is, what is in the future for you guys?

• 1:19:10Speaker 01

  Very near term, we've now started to work with some of the frontier companies in each of the categories that are taking off. And we'll, we'll continue that work to make sure that we cover all of the use cases that are really taking off. We see a lot of interest once the first one in the market moves. Lots of people want to follow them. And we think basically AUC one will get to a point where all of the Fortune 1000 will organize their risk processes around the standard.

• 1:19:37Speaker 00

  And you have 50%?

• 1:19:39Speaker 01

  No, we do not have 50% today. I think there is some world where probably by end of year, we might have representation in our consortium for 50% of the Fortune 1000. So that's on the agent layer. And then we think, yeah, the model layer, it's going to be, it just brings, are now surfacing the concerns that are most likely to slow down adoption of AI. And then, yeah, we think robotics comes after that.

• 1:20:03Speaker 00

  What are you hiring for? What's hard to hire for?

• 1:20:05Speaker 01

  We are hiring across the board, across go to market and members of technical staff. The people who do really well on our technical team are folks who are really excited about kind of being truly full stack. So let's say when we started working with Cursor, we'd never done coding tools before. So taking the standard and extending it, fleshing out what does frontier evals look like for long horizon coding agents? And taking that problem all the way from like working with Cursor and other folks in the space down to like fleshing out and shipping a new version of the standard. So that's like a truly a full stack entrepreneur, entrepreneur shaped, technical people do extremely well at AUC.

• 1:20:45Speaker 01

  The hard part is building one universal red teamer that works across from Harvey to Cursor and everywhere in between. It both has one consistent methodology, one consistent taxonomy of what are the risks and the attacks? And making, we think that's fundamentally the best way to make consistent promises. Jim Morgan is buying both. They want to have one framework, one consistent way that this comes out. And the mechanics of making that happen, you get to deal with a lot of the complexity of the real world. I think we have good answers in a bunch of that. But there's some pretty hard engineering problems in FinX and Q and A.

• 1:21:18Speaker 00

  Can I push a little bit? Like, must you have one? Why not just be like, okay, look, 40% of our use cases are coding agents. So we will specialize in coding agents. And that's the, that's the one of them. And then 30% is like rag. Just do rag.

• 1:21:32Speaker 01

  Yes. I think there's some wisdom in that question.

• 1:21:36Speaker 00

  Yeah.

• 1:21:38Speaker 01

  It depends on, what we found that there's a lot of value on is being able to, if the decision maker on the buying side, say you're the head of risk at a bank, and your biggest risk is not in coding, or in customer support, or whatever the top two biggest use cases, but it's somewhere else, you want to still make sure that that framework has something to say about it to the burning question you have. Otherwise, you'll not earn that trust. Now, it's true that a lot of the burning questions follow where there's a lot of adoption. And so, great, so do we. So we do today do not cover every single edge, but we have a framework that we can add all of these within. We have one global taxonomy of risks and attacks. That keeps adapting. As like every time a new incident

• 1:22:22Speaker 01

  occurs that has never been seen before, great, let's go and update the taxonomy. So we bake that in. So I think we have one coherent, universal approach. It doesn't mean that we spend equal amounts of time on code and insert niche use case. We do spend time where people, where people care. We think it's very valuable to have one language.

• 1:22:42Speaker 00

  Yeah, yeah, that makes sense. That's, that's, that's an important choice. We were going to end, actually, but I thought of one final ending closing question, which is, take this however you want, right? Let's say, one and a half years from now, OpenAI's secret panel of five experts declares that we have reached AGI. Do you expect your business to change?

• 1:23:03Speaker 01

  No, I think there's some important way. I think the, the last businesses to exist beyond the labs.

• 1:23:10Speaker 00

  Will be underwriting.

• 1:23:12Speaker 01

  Well, there's one, there's one job that the labs can never do for themselves, which is to be their own watchdog.

• 1:23:19Speaker 00

  There you go.

• 1:23:20Speaker 01

  So I, I think kind of to the extent that you believe this frame of like, you'll see hyper concentration, like the labs will kill all the startups, which we can go into the pros and cons.

• 1:23:32Speaker 00

  I feel like the labs actually care a lot about this, right? There was the whole superposition. What do we do? And we have models smarter than us, and a tier above, right? Models smarter than them, training them. So the labs actually think about this.

• 1:23:42Speaker 01

  They think a lot about that. I think there are, some of the smartest people on these topics work at the labs. So the problem is not whether they care. The problem is that they will all be stuck in a race where they might have incentives to cut corners, and they might have incentives to withhold information from the government, et cetera. And so, one kind of feels like eternal truth is that you need an independent third party to go and inspect that data and share information. In this case, say with the government, it's more of an incentive problem than an interest problem. I think they're fundamentally all trying to make this go well.

• 1:24:14Speaker 00

  What I'm not hearing is like, AGI, whatever that label means to you, to me, to them, doesn't fundamentally have like a qualitative shift in like, you still have to.

• 1:24:25Speaker 01

  And I think the one thing that would make this a qualitative shift is, for some definitions of AGI, it will just get nationalized. It'd be a threat to sovereignty.

• 1:24:34Speaker 00

  Yes.

• 1:24:34Speaker 01

  And then at that point, it kind of maybe every company is the government, and the government is every company. I struggle to think about that world. But at that point, you've kind of.

• 1:24:42Speaker 00

  We, I don't think we'll move fast enough.

• 1:24:44Speaker 01

  Right.

• 1:24:44Speaker 00

  You know, like, we're not, we're not set to do that.

• 1:24:47Speaker 01

  Yeah.

• 1:24:48Speaker 00

  But I have discussed this a lot on the podcast.

• 1:24:51Speaker 01

  Yeah, yeah, yeah, yeah.

• 1:24:52Speaker 00

  I mean, you know, as far as the watchdog concerned, I will also mention that because I have my finance background, I often think about the scene in The Big Short, where they talk to like, moody's, but also standard and pores. And then the lady at moody's is like, well, if I don't give you a triple A rating, you're just going to go down to standard and pores. So, so actually, the watchdog is a natural monopoly. Because if you have race dynamics in watchdogs, then the watchdogs will compete each other to the lowest possible standard.

• 1:25:18Speaker 01

  Correct. And so I think what, one of the things, one of the reasons why we're very excited about having insurers be around this table, is that insurers are the only ones that do not have this dynamic, because they pay the bill. If they keep lowering the prices.

• 1:25:32Speaker 00

  Yeah, you will find the market clearing.

• 1:25:35Speaker 01

  And this is not true for moody's where they don't directly pay the bill if they make recommendations that, that are off. So we think that balancing factor is, is pretty important. And I think it also highlights that there's like no system that's perfect. You need scrutiny of moody's, you need scrutiny of the watchdogs. For sure.

• 1:25:52Speaker 00

  Beautiful. Thank you so much for indulging. This is a beautiful conversation covering everything. Congrats on your success so far.

• 1:25:59Speaker 01

  Thanks for having me.

• 1:26:00Speaker 00

  Yeah.

• 1:26:00Speaker 01

  Appreciate it.