Underwriting Superintelligence: How AIUC is using Insurance, Standards, and Audits to Accelerate Adoption while Minimizing Risks

Rune Kvist and Rajiv Dattani, co-founders of the AI Underwriting Company, reveal their innovative strategy for unlocking enterprise AI adoption. They detail how certifying and insuring AI agents, through rigorous technical standards, periodic audits, and insurance, builds crucial "AI confidence infrastructure." This discussion explores how their model addresses AI risks, enables risk pricing in nascent domains, and aligns financial incentives for safe, responsible AI deployment. LINKS: AI Underwriting Company Sponsors: Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (02:53) AI Risks and Analogies (09:14) Insurance, Standards, and Audits (14:45) Insuring Ambiguous AI Risk (Part 1) (14:54) Sponsor: Tasklet (16:05) Insuring Ambiguous AI Risk (Part 2) (25:26) Managing Tail Risk Distribution (27:45) Introducing The AIUC1 Standard (Part 1) (27:50) Sponsor: Shopify (29:46) Introducing The AIUC1 Standard (Part 2) (35:45) The Business Case (40:43) Auditing The Full Stack (48:00) The Iterative Audit Process (54:58) The AIUC Business Model (01:02:26) Aligning Financial Incentives (01:08:56) Policy and Early Adopters (01:11:58) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm speaking with Runicavist and Rajiv Dattani, co-founders of the AI Underwriting Company, who aim to unlock enterprise AI adoption by certifying and ensuring AI agents. Their core insight is that security and progress are mutually reinforcing. Just as good brakes, seatbelts, and airbags are required if you want to drive 70 miles an hour, rigorous standards for AI agent behavior and reliability are critical for society's effort to realize the potential of increasingly powerful and autonomous AI systems. Their approach, which has already won support from an impressive list of industry leaders, including cognition, Ada, Intercom, and many more, combines three elements: frequently

• 0:45Nathan Labenz?

  updated technical standards that codify the latest best practices, periodic audits to verify adherence on an ongoing basis, and insurance to align incentives and provide financial protection when things do still go wrong. As always, in this conversation, we get into the many details of making this work, including the fact that today's insurance policies generally don't explicitly address AI risks, creating ambiguity about coverage for AI incidents. The AIUC1 standard that they've developed in partnership with technology and security leaders across a wide range of industries, which covers data and privacy, security, safety, reliability, accountability, and societal risks. Their approach to auditing client companies, which combines analytical evaluation of

• 1:29Nathan Labenz?

  technical safeguards with systematic red teaming. How the data generated by red teaming helps insurers price risk in domains where historical loss data is either limited or nonexistent. How they plan to structure financial incentives so as to avoid a race to the bottom, such as that which affected the credit rating agencies in the run-up to the 2008 financial crisis. And how, even though the market may not be able to effectively insure against the largest scale existential risks from AI, the government, as the de facto insurer of last resort, still benefits tremendously from the governance that this model creates. Overall, I have to say, I really like this approach. So much so that I did participate in the company's seed round alongside leading investors, including Nat Friedman, Emergence, and Terrain.

• 2:13Nathan Labenz?

  Certainly, there is a place for government regulation and also for unencumbered experimentation. But I think a private sector approach to AI reliability, powered by enterprise customers' desire to move with all responsible speed, and designed to align all parties' financial incentives with consumer and public safety, seems much more likely to get the important details right, not just once, but over and over again, as both the technology and the market mature. For now, I hope you enjoy this conversation about how the AI underwriting company seeks to build AI confidence infrastructure and ensure the intelligence age, with co-founders Runikavist and Rajiv Dattani. Runikavist and Rajiv Dattani, co-founders of the AI underwriting company, welcome to

• 2:58Nathan Labenz?

  the cognitive revolution. Thanks for having us. I'm excited for this conversation. You guys are doing some really interesting innovation at the intersection of trying to make sure that everybody can have all their AI goodies, which I am very excited about getting, but also making sure that we keep things on the rails, which is obviously extremely important as we head into a brave new AI world. So I'm excited to dig into everything that you're doing, the standard that you've put out, and the future that you guys envision. Maybe for starters, I think you've used a pretty provocative analogy in some of your writing and the way that you've introduced the company. Maybe tell us like, sort of, what the downside scenario would be for AI if we like, don't

• 3:43Nathan Labenz?

  do a good job of this? I mean, obviously, there's like, extreme, extreme extinction, but you've got sort of the nuclear outcome, which is a scenario where, you know, we don't get the goodies, but we still maybe get some of the bad stuff. And then, you know, maybe you could compare to an industry where things have gone well that you take inspiration from.

• 4:01Speaker 01

  Totally, and maybe before we get to the nuclear outcome, a lot of the concerns that we're seeing in the world today relate to today's risks that are much more kind of, in some ways mundane, data risk, uh, safety and security risk related to like brand disasters and data leakage, some of these kind of age old concerns that become much more severe in a new environment where there are many, many more vulnerabilities and frankly, people understand the vulnerabilities much less.

• 4:34Speaker 01

  But as you kind of plays out into the future, uh, we'll move to more and more agentic systems uh, that are going to be way smarter and that opens up new kinds of nuclear scenarios. One that's been talked about, a bunch about relates to imagine terrorists that want to inflict a huge amount of uh, harm if they have access to brilliant scientists that can help them create, for example, a new virus. They will be able to do that way, way cheaper and so you can imagine new kinds of COVIDs coming out uh, and they'll be way easier for adversaries to, to create. You can also start to imagine what a misaligned kind of system looks like where today that might look like uh, it's like deception or sycophancy, but as they get smarter and smarter, they'll

• 5:21Speaker 01

  they'll do some of the things that humans can do. One of the things I sometimes get questions about is like, well, well, as AIs get smarter, will they still fail? And I think it's, there are a lot of analogies to humans where a really smart CEO will no longer kind of mess up basic logic, but that enables them, their IQ enables them to run kind of Enron levels of fraud. So as the kind of uh, agents get smarter, you get different kinds of scenarios and when we think that really opens up for, for nuclear scenarios. A different category of scenarios altogether is the geopolitics, some of which may not have to happen, like relate directly to AI, but if you really get

• 6:03Speaker 01

  National security is believing that AI will give their nation a decisive strategic advantage that might lead them to preemptively attack their opponents in ways that may look like regular military escalation, but is really prompted by AI.

• 6:20Speaker 02

  Nathan, you asked about other industries. I think there's not one, there's neither one industry that we look to and say, you know, that industry was perfect, nor is there a perfect analogy for AI. But the really the thesis what we're building here, and I think we'll get into this more later, is how can we bring together learnings from lots of other industries, right? So the kind of three components to what we're building, the insurance, the standards and the audits, has come out of cars. And when, you know, post-World War II, there were a lot of car crashes on highways, insurers really developing crash testing solutions that led to airbags and creating the incentives for that to happen. Go further

• 7:00Speaker 02

  back into Benjamin Franklin's Philadelphia, and we've written a lot about this, and fire insurers leading to building codes. And no one was saying, you know, stop building at that time. It was how do we actually keep building, but build in a way that's safer? And that's really the essence of what we're trying to try and draw on here for our solution.

• 7:20Nathan Labenz?

  Yeah, it's funny, you went to even some more tail risks or kind of, you know, surprising possibles in the, under the umbrella of the nuclear outcome that I had in mind. When I say the nuclear outcome, for me, that's kind of shorthand for, we don't get the super cheap abundant energy that we should have from nuclear technology, but we do have thousands of nuclear weapons that are, you know, collectively a sort of Damocles that hangs over all humanity. And I do worry about a similar thing with AI, where it sort of gets like weaponized, or there's this AI Cold War, but maybe we also like freeze it out from a lot of the practical upside, you know, day-to-day quality of life improvements that I really want.

• 8:03Nathan Labenz?

  And what I really, for me, often just think of airplanes, because it's like, sort of like nuclear, there is the potential for like dramatic explosions. And, you know, obviously, if one happens near you, or you're on the plane that goes down, like, that's really bad. But yet, we do get the upside from planes, you know, and we fly cheaply and all the time. And, you know, as I always think back to Dumb and Dumber, like, it's actually more likely you'll get killed on the way to the airport, you know, than on the plane itself. So it is. Remarkable in my mind how contingent the end state for some of these like potentially transformative technologies can be based on, you know, even just a few incidents that

• 8:45Nathan Labenz?

  happen early on in its development that can kind of put us on one course or another. So.

• 8:51Speaker 01

  And I think, I think Nathan, we're already seeing that some parts of AI today, if you take kind of autonomous vehicles, you still cannot drive an autonomous vehicle outside of San Francisco very far before you get stopped. Yet drone warfare is increasingly autonomous and it looks like there's no barriers to that getting deployed. So in some ways, we're already starting to see that when it comes to autonomous drones and vehicles.

• 9:14Nathan Labenz?

  Yeah. Okay. Well, let's get into the positive vision, insurance, standards and audits. This is like the holy trinity of a virtuous cycle that you guys are trying to kickstart. So paint the picture.

• 9:29Speaker 01

  I can start high level. The core idea is that security and progress are mutually reinforcing. A kind of very basic analogy is that if you're a race driver, the reason you wear a helmet and a seatbelt is so that you can go faster around the corners. And the same is true for AI. The better we're able to steer and secure our systems, the faster we can deploy them, the more we can lean into some of these agentic capabilities. So the debate between the dooms and I take an originist, we think this is a core point that they in fact are mutually reinforcing. Maybe Rajiv, I'll let you speak about the kind of the mechanics of the incentive flywheel.

• 10:11Speaker 02

  Yeah. So, so the core of this Nathan is really as you were describing, if we don't get the security under control, we don't think we'll get the progress, right? We don't get the benefits of the airplanes. And so what are the three components to this, as Renan was mentioning? There's insurance, there's audits and there's standards. So let me just explain what, what each of them are and then I'll talk about how they link. The insurance is the financial protection for when something goes wrong, who's actually paying for it, right? And can you actually get the financial coverage behind it? There's a standard, which is what is the best practice and can we actually codify the set of requirements that you need to meet to have the best practices and to have met them? And there's the auditing which says, have you met the standards?

• 10:54Speaker 02

  Now, the reason we think we need all three of these is insurers will struggle to actually underwrite the risk and know how to differentiate a good risk and a bad risk unless there are a clear set of requirements that they can say, you know, yes, I understand these are the best practices, are you meeting them? Which of them aren't you meeting? And that would affect your premium, just like it would in your car insurance if you're driving recklessly. The auditing is essential because a standard without the auditing doesn't really help because it's very easy for companies to say, yes, I'm doing these things and self attest. But if there isn't the oversight to say, actually, yes, you know, somebody has rigorously checked this and made sure that the standard is being met.

• 11:37Speaker 02

  It's kind of a, there's a trust gap there. And so this is kind of how we think about the flywheel and critically in the past, insurers have funded the standards and funded the auditors to actually become more robust and create that oversight. And they've created the incentives for the companies who are creating the risk to comply with the standards because they want the insurance and they want cheaper insurance. And so creating the incentives where you have a pool of capital over here on the insurance balance sheets, funding, funding the research and you have actors who are creating the

• 12:17Speaker 02

  risk actually wanting to comply and wanting to have the insurance in place, we think actually aligns the incentives in the right way. Maybe one other thing I'll say on this is at the moment, this kind of debate has become polarized between.

• 12:32Speaker 02

  Let the AI developers cook and and just, you know, let them commit to voluntary commitments and that'll be fine. All the way through to on the other extreme, you know, we need top down regulation, we did it yesterday, and it's got to be really strict. We actually think this is a neat kind of middle ground here where you're aligning the incentives because the insurers want you to actually grow, right? That's how they grow their premiums as well. So they don't want to have too onerous requirements. They don't want to have requirements that don't actually track onto risks because then no one will buy insurance from them. On the other hand, they don't want it to be too lax and they don't want you to actually be able to break your commitments because they're the ones paying. And so this is a neat kind of market mechanism that that aligns the incentives between actually progress and security.

• 13:18Speaker 01

  And maybe we can ground ourselves in some of the historical examples and just work through them. So what the very first place we saw the incentive flywheel in in practice is back in seventeen fifty two in Philadelphia, when Benjamin Franklin sets up the first fire insurance company in the US. The basic problem was that Philadelphia was growing, the population ten X over that decade. And so people were packing houses close and close together, which created a lot of fire risks and the fire risk, of course, was spread. No individual actor was carrying entire risk because the entire city might burn down. The fire insurance company were obviously having to pay whenever a house burned down. So they're interested in reducing the the frequency of fires.

• 14:00Speaker 01

  And they created building codes, which is in fact standards. How far should houses be apart? Can they be under large trees that might be hit by lightning or not, et cetera? And then they created the first fire inspection, a focus that went into homes to just see are they meeting these building codes as a way of actually enforcing it. And so it's kind of that very basic logic that we've seen pop up again and again. For example, in the year nineteen hundred as well, when electricity comes out, again, houses start burning down and the underwriters laboratory funded by the insurers pops up to define the UL certificate, which you'll see in your light bulbs today, is a product security standard that they will then go and audit against. And that's the kind of mechanism that we're trying to bring to AI.

• 14:46Speaker 03

  Cool. I love the vision. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:55Nathan Labenz?

  Where do we stand today? I was doing a little deep research on the topic. And I guess my takeaway from what ChatGPT told me is that mostly it seems like the harms that people anticipate from AI systems are covered under existing insurance policies, at least when it comes to like the taxonomy of harms, like the kinds of harms are insured. People are sort of reasonably comfortable with the kinds of things they expect. But there's some ambiguity, I guess, around whether the fact that like an AI might cause them as opposed to some other cause could put something outside of scope. And it seems like a lot of people right now are just living with ambiguity. I understand this is one of the things that you recommend that the industry

• 15:39Nathan Labenz?

  clear up is like what's covered, what's not covered. How do you see that today? And what are the big points where we need clarification?

• 15:47Speaker 01

  So the state of play today as it comes to AI insurance is that it's ambiguous about what's covered in the current insurance policies.

• 15:58Speaker 01

  For example, AI can help you create new kinds of cyber vulnerabilities.

• 16:04Speaker 01

  And of course, people already have cyber insurance today. But it's unclear whether AI-induced cyber vulnerabilities would be covered by the current policies because AI is not mentioned anywhere.

• 16:16Speaker 01

  And what we think is going to happen at a high level is very similar to what happened when computers came out back in the kind of early 2000s. Again, leaking confidential data or having kind of downtime was not an entirely new set of harms. But when computers came out, they happened at a very different frequency and a very different severity. And there was the same question, is this just age-old insurance? Or is there something new here? And what happened was that over time, as these lawsuits came in to clarify whether things were covered, cyber insurance was split out into a separate set of products because they need to be priced completely different. And the products need to have a different structure.

• 17:00Speaker 01

  And we suspect that the same thing is true with AI here. One thing we know for sure is that the current insurance policies have not priced in AI risk because the insurers do not know how to price the AI risk.

• 17:13Speaker 02

  And this just doesn't serve anyone. The companies who are insured and their customers are saying, I'm not sure if I'm covered. In the 2000s, when it was cyber, this took years to work through the courts, but literally insurers were suing customers and vice versa. And from the insurer's perspective, as Rinne said, they've not priced in the risk. They're taking risk on that they're not being paid for. And that's going to show up in loss data, we expect sooner than later.

• 17:38Nathan Labenz?

  Yeah, that's a really, I had a very, very brief foray into the insurance industry some years ago. And it strikes me that this is a really hard challenge for them because they like to do this sort of thing based on historical data, right? It's like, we can look back at how many car crashes there are, we can look at how many fires, how many homes burned down from fires. And with a couple of important assumptions, like that things will be relatively uncorrelated. You can be pretty confident that you can write a bunch of policies and, you know, don't shoot yourself in the foot and you'll probably be okay. But the same challenge is popping up with AI everywhere. I was just speaking to a group of educators last week and saying like, I know you guys have been trained and it's like,

• 18:20Nathan Labenz?

  you know, central organizing principle of the field that you want to be evidence-based. But you can't be evidence-based because by the time you see a study, you've got two more generations of AI and the kids are using something that didn't exist at the time of the study. Here we are, you know, basically with the same challenge in insurance, right? Like, do we have any sense right now for what the frequency of these things is going to be? What the, you know, what the distribution of like magnitudes of harms are? And if people are, maybe we have some of it, but I'm interested in how much we do have. And then how does that translate into any sort of decision on pricing?

• 19:00Speaker 02

  Yeah, there's a couple of issues here, Nathan. One is what you said, AI is just moving too quickly. It's moving faster than anything insurers have ever known. And so the slow rollout, let the data trickle in, pick it up in a few other categories and then develop a product off the back of it, just won't work. If you were building an insurance product today, based on historical kind of AI data, you'd be building for a pre-LLM era and not even thinking about the agentic risk.

• 19:29Speaker 02

  So that's one issue. I think the other issue here, though, is they don't really have a mechanism. They're not set up to even collect that data. And so if they don't have a product and these losses are kind of happening in kind of small places, which is typically what we're seeing at the moment, that there was an EY study on this recently. Enterprises are having kind of million dollar size losses due to AI. Smaller companies are having, you know, in the thousands, tens of thousands of dollars. Insurers might not even be seeing this. And so if they can't see it, they don't have the loss data from which to price.

• 20:05Speaker 02

  And so what we expect is there's this kind of a small number of, or rather, there's a long tail of like small losses coming in and then we're hearing and seeing the big ones. They're the ones that kind of make the press and the rest of them are kind of being ignored, both by the insurers and by the companies where they're kind of just writing them off.

• 20:26Speaker 02

  If you don't think about how you translate that to pricing,

• 20:30Speaker 02

  it's just really hard for insurers to basically apply that model. And so there's a couple of approaches we're taking here. One is the nice thing about AI is you can actually do what was kind of known as red teaming. You can run evals.

• 20:45Speaker 02

  I expect your listeners are kind of very familiar with what that is in the AI context. But if you translate that to insurance, actually, it creates a massive pool of synthetic data, basically, for how this product is going to behave in the real world. And insurers can literally plug and play this into their pricing models where they say, well, I see that, you know, this failed the red teaming that I did in these particular ways. If that happened in the real world, I can estimate what the loss would have been because, you know, they know how to estimate losses from a data leak or a court case that would have happened, right? That's that's their bread and butter. And so they can take the data from the red teaming on like both frequency and severity and plug that straight into a pricing model.

• 21:30Speaker 02

  The other thing that's nice here is you can create pretty clear kind of parametric, as it's known in the insurance industry, pricing triggers, where you can say, actually, I can know definitively.

• 21:43Speaker 02

  Did the AI behave as it was supposed to? Or was this an out of distribution event? And by using that, you can then very quickly say, right, I'm going to pay out a million dollars because this happened and I'm just going to price it. I'm going to pre-agree the pricing, basically, and pay out rather than waiting for a long kind of court case to come in and settle on that.

• 22:05Speaker 01

  One other big consideration is there is always someone underwriting the risk, even when there's no insurance. So today, when, say, a hospital or a bank deploys an AI system, implicitly, they are pricing the risk to decide do they want to go ahead or not. And so although it's like an extremely gnarly problem to really pin down the price, the most important fact is that this is happening every day, all the time, that people are trying in more or less sophisticated ways. And we think by and large, the rigor of the insurance industry is going to make it more likely to get that pricing right than non-experts who are sitting distributed across every enterprise trying to make these decisions for themselves.

• 22:50Nathan Labenz?

  Yeah, we're all self-insured, whether we know it or not, at the moment.

• 22:56Nathan Labenz?

  You know, I guess one of the things that makes this sound really hard to me, if I try to imagine myself being an insurance company, is it feels like, again, with, you know, car accidents or whatever, I would guess that there's some sort of normal distribution of the damages, right? Like, there could be some outliers, but those will probably be relatively infrequent, and I can, you know, put most of the incidents in a pack. Whereas, you know, there are other things in the world that are characterized by more of a power law distribution, like venture capital returns, for one. I would assume, although I don't know for sure, that, like, bio incidents probably fall into this category, where, you know, a lot of, I'm sure there've been many, you know, viruses leaked from many labs that just

• 23:40Nathan Labenz?

  didn't go anywhere, but then you get one COVID and it turns into a global event. And I, my gut says AI risk might look more like that, in that it could be a few incidents that drive the bulk of the, the total weight of the losses. Is that right? Do we even know, you know, what, how to think about that? And, and how? How much of a problem does that pose for insurance? And then I guess like, is there any way, assuming that, assuming my mental model is not way off? Like, how do we start to attack that problem?

• 24:15Speaker 01

  Yeah. The, the first answer is no one knows what the kind of tail risk distribution looks like, because we have not yet seen it. And from nuclear energy insurance, so nuclear power plant insurance that we come in from here. So the same intuition applies here that probably nuclear failures are going to have a power distribution where the real big failures will carry most of the losses. And in fact, that's, that's true. Uh, in the U S, there is in fact, an insurance scheme for nuclear power plants. This is required by law.

• 24:53Speaker 01

  And, uh, the way they deal with these tail risks is that the government basically made a scheme where they said, we are going to require every nuclear power plant operator to have insurance. And we're going to require you to take liability, no matter where in the supply chain things fall. So no matter whether it's one of your vendors who provided you with a faulty thing, you are responsible for all of this. You're going to take out insurance. But on the other hand, we're going to cap the liability.

• 25:31Speaker 01

  So if there's a damage greater than $15 billion, this is going to come out of the government balance sheet, especially a government backstop.

• 25:40Speaker 01

  The reason why they're interested in this is, you're exactly right. The market may not be able to carry these very, very big tail risks. Yet the government would really like there to be an insurance industry because it provides an important governance function. It means that there is now a market-based version to have independent third-party auditors with direct financial incentives to price the risk accurately and enforce best practices on the industry. Basically does some of the governance work, but the government may not think that it has the capabilities to or is not going to be responsive enough to the market. And so there are some lessons there from nuclear for how do you get the best of private market governance? While allowing the government to capture risk where the market may not be able to to really respond to it.

• 26:25Speaker 01

  And I think this idea of like capping risks is going to be critical against AI, uh, which can allow you to cover a broad set of risks up to some amount.

• 26:34Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 26:39Nathan Labenz?

  Let's go to the standards and the audits. You guys have put out this AI UC one standard. The first standard that I'm aware of anywhere in the at least Western world for what to do when you're building and deploying an AI agent.

• 26:58Nathan Labenz?

  Tell me about it. What does it contain? What does it require companies to do? And how did you go about assembling all of these ideas?

• 27:07Speaker 01

  Yeah, I can speak to the high level here. The core idea with AI UC one is to take all of the concerns related to AI agents that slow down adoption.

• 27:19Speaker 01

  And put them into one framework that outlines for each of the risks, what are you supposed to do? In enough detail that you can have independent third party auditors come and verify whether a particular company is doing those things.

• 27:34Speaker 01

  And so the kind of design principle here is to go out and spend a bunch of time. We met with more than 500 folks. Across the enterprises that are deploying the systems today. This is security leaders, general councils, et cetera, across banking, health, et cetera, and figure out what is it that's keeping you up at night? And the answers will not surprise you a ton. It's going to be things like data leakage at a new scale. It's going to be, are we going to provide incorrect medical advice to our to our customers? Are we going to have jailbreaks and prompt injections? Uh, is there, are we going to be discriminating against our applicants? Et cetera, et cetera, baking that into one framework.

• 28:17Speaker 01

  The goal is.

• 28:20Speaker 01

  When a company passes, meets the standard, the enterprises that are deciding whether to adopt or not will get the data foundation for them to understand whether they can trust that particular agent or not.

• 28:33Speaker 02

  The way it's used in practice, Nathan, is often you'll have AI agent builders selling to large enterprises, right? Think of the the large banks, the large technology companies. They want they want a third party to have vetted the tool they're looking to buy and they ask them for their AI UC one report. So a large company is looking at a AI chatbot developer, say, AI customer support developer, and I'll say, share with me your AI UC one report that has in it all of the detail on what are the data and privacy controls they have, what are the security controls and implementations they have, what are they doing to protect customer

• 29:13Speaker 02

  safety, you know, what output filtering do they have, for example, on how does it perform against bias and is it tracking kind of its role correctly and not responding with anger or things like that with the or is it responding appropriately to anger if a customer really wants a refund and it's saying, you know, you must give me this and it's going to respond correctly?

• 29:37Speaker 02

  What are they doing with reliability? Do they have a groundedness filter in place? Are they appropriately mitigating as hallucinations? What are their human in the loop practices and what's the human oversight and what are the societal risks that they're tracking and covering? And so in one report, the enterprise can look at it and say, it's been third party tested, I trust it, and I could see all the detail that I might need and preempt all the questions I have.

• 30:00Nathan Labenz?

  Yeah, gotcha. So 500 people is a lot to get on board with anything.

• 30:08Nathan Labenz?

  How easy or difficult did you find it to boil all this down into a sort of consensus standard? I'm sure there must have been some things that were like contentious one way or the other in terms of like, well, including that is overkill or not including that, you know, leaves, you know, this risk unaddressed. How did, you know, where is the line? Like, what's the frontier of debate in terms of what is, you know, solid enough to be in the standard and what's still kind of, you know, next set of considerations?

• 30:41Speaker 01

  We found it to be remarkably uncontroversial for most of the topics.

• 30:49Speaker 01

  There is a lot of interest in just creating one shared language. And a lot of the, a lot of the concerns, like data. It's actually mostly just like churning through the issues and providing clarity on what should, what should be included, rather than debating where the line is. In part because the standard doesn't exactly specify what should be done, but it's a lot around disclosure. So that any particular enterprise can decide whether this is okay or not. Given this may differ radically whether you're a hospital or whether you're a retailer. They just have different risk preferences. And so if you get some of those design principles, it's actually relatively easy to get people on the same page.

• 31:28Speaker 01

  And the frontier of kind of controversy is probably related to some, some of the more societal harms. So should we check for whether a particular product enables folks to create cyber attacks at a much grander scale than before? Which may not really be kind of that top of mind for any particular enterprise that's deploying it, might seem a little bit overkill. And so that's where we've had to kind of make some judgment calls about what will best serve adoption, not just for any particular individual enterprise. But for the industry as a whole.

• 32:07Speaker 01

  Because as you were mentioning, when it comes to, say, nuclear, one incident can kind of chill a whole industry. And those are some of the effects that we've started to include in the standard. And still focused on like, how do you do that in a pragmatic way where you can actually assess it? And you can get it done without having to spend thousands of dollars to, to check whether any random bug coming through is going to bring down, bring down a whole industry. But we do think it's important to keep, keep tabs on.

• 32:38Nathan Labenz?

  So.

• 32:40Nathan Labenz?

  I guess, how does this look in practice? I was just talking to Andrew Lee a couple of days ago. He's launching this new product called Tasklet. And it is a very, you know, bet on the models agent platform where his idea is like, forget all this sort of step-by-step flow charts and all these kind of, you know, this field maps to that field as you would see in a traditional automation software experience. And he's just like, we're just going to bet on the models to figure it out. We're going to give them all the tools they need. We're going to give them the memory and they're going to keep getting smarter and they're going to go figure stuff out. And so I asked him, like, would you like to buy insurance? And he said, yeah, you know, we actually have had a really great record so far that

• 33:23Nathan Labenz?

  like, I thought people were going to be really freaked out or that we might see, you know, weird rogue behavior from the agents from time to time. Fortunately, they haven't. But still, he was like, yes, I would like to buy insurance if only because then I can take that insurance to my prospective enterprise customers and like, show them that, you know, we've got this and that should definitely ease our go-to-market process and speed it up. So, you know, perfectly consistent with your thesis there. But if we dig into it a little bit more, like, who do you think is primarily pushing this? Like, is it coming from an enterprise demand side or a vendor that like, wants to, you know, prove their qualification side?

• 34:04Nathan Labenz?

  And how much, is there any way that we can like quantify what we're getting? You know, you do these various things, but like, what was the risk before? What is it after? You know, do we have any way of sort of saying to an enterprise buyer, okay, you're going to buy this vibe coding tool and, you know, we're going to bound the risk in some way for you or we can, you know, we can make some level of confidence assertion that you're not going to have more than this big of a problem? What are those like, final, you know, statements that people are like, buying into look like?

• 34:38Speaker 01

  We see demand for what we think of as a confidence infrastructure, both on the developer's side and on the enterprise's side. Because with or without insurance and standards, there is still risk. The enterprise is still nervous. They are still asking tons of questions of the developers today and sometimes just not buying because they can't get to confidence because there's too much ambiguity. So the enterprises definitely would like to have clarity of one language, one standard, one report that they can trust. But the flip side of this, as I was saying, is that currently it's incumbent on the developers to create clarity individually, to earn the trust, to earn confidence. And there aren't a whole lot of tools available.

• 35:22Speaker 01

  They can write public blog posts on their security posture. And that's great, but there is, of course, they've graded their own homework. Everyone thinks that their security posture is great. They will sometimes reach out and make financial guarantees off their own balance sheets. And that can be, frankly, I think that's a great way to do it. But again, their balance sheets may not be able to carry it. So we actually see like the most forward-looking developers are as interested in creating this kind of confidence infrastructure as the enterprises, even though, of course, the enterprises are kind of the ultimate beneficiary of it.

• 35:59Speaker 01

  To your point around, like, what's the impact on this on the risk? Probably the best way to look at this is that when we go and certify a particular AI developer, we run a set of evals that finds a set of vulnerabilities. And we'll often do that in multiple rounds. So in the first round, we tend to find quite a lot of stuff that they themselves are very interested in solving. So we often find that sometimes you can get up to a 25% failure rate on certain kinds of attacks against a company, which is just obviously not what they want nor what the enterprises want. And we'll then, like, with the help of the standard,

• 36:42Speaker 01

  help them implement whether that be a groundedness filter or a monitoring solution or a content moderation filter and see those same failure rates. And we can often see them drop by 90%. The answer is also there's no panacea as like AI security as a field and the safeguards.

• 37:01Speaker 01

  It is a work in progress, it's an open research problem, but there are some pretty basic steps they can take. And we can see that pretty quantitatively from our first round of red teaming to the second or third round of red teaming. And that is the way that we get a little bit more confidence that through this process, the AI companies that are deeply hungry for improving actually will improve in a quantifiable way that they can they convey and communicate to their customers to have confidence, which is ultimately what allows them to invest in this stuff when they have a million things that they could be investing in.

• 37:36Nathan Labenz?

  I wanted to do just one kind of double click on the kinds of companies that you're working with. It seems like a sort of tacit assumption of most of the conversation so far has been that we're focusing on application developers who are presumably in most cases not training their own foundation models. They may be using a commercial API, they may be taking an open source model and doing something with it. But they're kind of providing the wrapping of that. And I don't mean that in a derogatory way, but obviously, there's a lot that goes around models to make them useful for enterprise purposes. But there are other layers of the stack that one could think about wanting insurance for, or wanting

• 38:16Nathan Labenz?

  to do similar audits on. Are you guys doing stuff at the foundation model layer, or even deeper at like the physical data center infrastructure layer as well? Or are we limited for now to the application layer?

• 38:32Speaker 02

  Yeah, we think this model applies at all three layers, right? So if you think about this kind of alignment of incentives, research into security, and how that unlocks progress, it's clearly most acute at the application layer at the moment, because these are the companies who don't have large enough balance sheets to kind of make large promises on their own. It's also to some extent where the risk is headed in terms of, you know, they're the ones building the agents and deploying agents today, even more so than the foundation model developers, for example. But if you think about data centers, you know, we spoke recently to a chair of the Audit Committee and Risk Committee at

• 39:14Speaker 02

  one of the top tech companies, who were saying, well,

• 39:20Speaker 02

  We're investing an astounding amount of money into data centers right now and our public shareholders don't want to be bearing that risk of what happens if the data center goes down, which could be anything from actually, you know, just wanting uptime through to there's an attack on the data center or, you know, there's an issue with the, with the security of it and it gets breached. And so, you know, they're saying, can we buy insurance for that? And can we have assurance from a third party that these risks have been well managed in a way that wasn't the case, you know, historically with, with kind of data centers they've bought and invested in the past. Now, that's obviously its own standard, right? AIC one doesn't currently cover the risks for that. So what we expect we'll do is we'll

• 40:03Speaker 02

  run the same process that we've run with AIC one, where, you know, as Rina described, we talked to 500 executives and understood one of the risks and one of the mitigations. And we talked to AI experts. We'll do the same thing on data centers where we bring in the right folks who are, who are really leading the thinking on this, both on the kind of buyers and the, and the builders of them. We'll codify the best practices. We'll figure out what are the ways in which you, you actually confirm the security is in place. And then we can insure off the back of that. And, and you can imagine exactly the same applying for, for foundation models.

• 40:36Speaker 01

  One way to think about it is, is also a game of earning insurers confidence over time. So start with the risks that might cost millions of dollars, tens of millions of dollars, move on to ensuring risks that might cost you hundreds of millions of dollars, a low billions of dollars, and then move on to the tens of billions of dollars eventually. And think of it as a ladder up and the agentic places where you have a lot of relatively small incidents, which gives you more data, allows you to build confidence, you have good models and then kind of slowly earn the confidence to, to move up the ladder.

• 41:11Nathan Labenz?

  Yeah, it makes a lot of sense to me also just from the standpoint of what I see people doing, like who seems to be doing a good job and not such a good job at this stage of the game. I wouldn't say I know too much about the data center layer. My sense is that there's a lot of new concerns here for them for sure, but at a minimum, they're like well aware that they're putting tens of billions of dollars into the ground and they have very obvious incentives because they're like, they're well aware that they're self-insuring. The foundation model providers strike me as being remarkably, not with some notable exceptions, like remarkably good about this stuff. And again, we could question

• 41:56Nathan Labenz?

  whether the whole enterprise is like a bad idea, big picture, but like they're doing a lot of work to taxonomize risks and to build defense in depth approaches and all that kind of stuff. And then when I see the application layer, I guess it started to get better, but certainly for most of the last two years, what I've seen is like weekend hackathons turn into businesses. People never really gave this any thought before they kind of built something and launched it and they just have zero, in many cases, zero protections in place whatsoever. So I do think from the standpoint of like just where's their low hanging fruit, where is there like a lot of needless risk that enterprise buyers

• 42:41Nathan Labenz?

  legitimately should be aware and should be asking these kinds of questions. The app layer absolutely has a lot of low hanging fruit and there's a need to be able to identify and separate who has done a good job and who's really, even if you can't take the risk to zero, like who's made a best effort to get to a place where they can responsibly be deployed in enterprise or a high stake setting and who hasn't. And there are a lot that haven't. Would you agree, disagree, add any color to my sketch there?

• 43:16Speaker 01

  I probably agree with the assessment. I think one piece of color to add is that at the agentic application layer, the promises being made to the enterprises are much thicker in some ways. Where if you buy Claude for your own purposes and you go to claude.ai, the promise of what Claude will help you to do isn't very specific. They're not claiming that they will give you correct medical advice 90% of the time. They're just saying, here's a helpful tool, use it for whatever, it will hallucinate. If you move into a space like, say, customer support, the kind of public claims that these companies are making are like, we will resolve X% of your tickets. And kind of implying also in those sales processes,

• 44:02Speaker 01

  we will not create massive brand disasters for you because we're gonna, you can trust us to interact directly with your customers. So the thicker the promises, the thicker the insurances and insurance also needs to be. And so that's a little bit of color for why the agentic application layer is the place to start. One other piece of color is that the distinction between foundation models and agentic applications is blurring. Take an example like Claude Code is not, is clearly not a foundation model. It's a system of models, at the very least, with a bunch of kind of different things going around. There are a bunch of integrations, a bunch of access. And so this, this line is really blurring and we think like agents is the most helpful frame to just think about it. And that's where some of the application layer companies got there first.

• 44:48Speaker 01

  But the companies you think of as foundation model companies are quickly moving into that space.

• 44:53Nathan Labenz?

  So let's talk about then the nature of the audits that you guys are doing. You mentioned red teaming. If I'm an application developer, and I am, like I've got Waymark, we are in a blessedly low risk environment because we help media companies make marketing assets for generally small and local businesses. And I, my standard line for our team has always been if somebody breaks into our system and watches all these TV commercials that we're making for local businesses, nothing bad will happen. That's a great, great comfort that, that we have available. But let's say I want to do this, like you can go, you know, poke it at our thing from outside and like prompt it or ask it to do like hateful or whatever commercials and you probably could find some gaps if I had to guess.

• 45:39Nathan Labenz?

  What more do you do? You get under the hood, you like read source code? Like, what are all the different angles of attack that you bring to bear on figuring out the, you know, many ways an application could go wrong?

• 45:50Speaker 01

  Totally. The the foundation is to start ground ourselves in the real world incidents that have happened to be like, what is it that we're looking for? So we have a database of incidents, real world incidents that have happened that inform what are the things that we should be checking for? Some of those are things that you might think of as kind of very concrete headlines. BBC says, wow, our Canada's chatbot hallucinated a refund policy. Great. We should clearly check for hallucinations that have financial implications. Some of them might be slightly more speculative, like a research paper that shows under these kind of experimental circumstances, AI chatbots will try and blackmail the user we'll think of as like an incident. It's just slightly more speculative.

• 46:35Speaker 01

  That forms the foundation for the audit. We then try and abstract that into a taxonomy of both what are the harms we're checking for? And what are the kinds of attack vectors that are relevant? What are the kind of latest research into jailbreaks? What are the kind of latest examples of applying the prompter on Twitter for different ways that these models break? And that kind of becomes the checklist of things that we want to check for.

• 47:01Speaker 01

  And the way we check it is broadly, for each of those kind of harms and risks, we specify what are the technical, legal, and operational safeguards you must have in place? And what are the evals you need to run? The technical safeguards could be like, oh, you must have a grounded misfilter that reduces the chance of hallucinations. Or it could be you must have a plan for what happens when shit hits the fan. What is your AI incident plan? Those get checked either by looking into the codebase, reading into policies, et cetera. And then secondly, there's the technical testing element, the red teaming, which really grounds everything. It's not enough to have an AI groundedness filter in there if it doesn't

• 47:42Speaker 01

  work. So that's where we go and take all these scenarios and try and break the chatbot to see how effective is that. And then the output of this is, you can imagine, a report that says for each of these technical, legal and operational safeguards, what do they have in place today? And for the eval of displaying back the results of what we found in a way that both helps enterprises make sense of it, but also leaves room for them to make the final judgment call with some level of confidence of whether there is a go or no go.

• 48:14Speaker 02

  And so one way to think about this, Nathan, is there's a trust and verify approach that we take, right? So what are the policies you have? Have you implemented the safeguards? And can we verify they work? And so again, you could imagine exactly how this would then also translate to foundation model developers, where it's like, you know, they have these policies on what safety testing they'll do and how they'll implement various safeguards. How do we actually verify that's happening?

• 48:42Nathan Labenz?

  Yeah, you mentioned that there's like multiple rounds of this process. And I think it's probably worth talking about why that's important. I was just reading a paper from Nicholas Carlini, Sander Schulhoff and others. I'm sure you've seen it, The Attacker Moves Second. And the basic notion of that paper was if you take the attacks as static and you say, okay, I'm going to tweak my defenses to mitigate those attacks, you can do that and you will mitigate those static defined attacks that you were trying to mitigate against. However, when you then deploy those defenses into the real world with potentially

• 49:24Nathan Labenz?

  adversarial attackers coming at you, they find ways around those defenses at a remarkably quick rate. And so what they found in their study was that human red teamers were basically 100% undefeated. Like there were no defenses that were totally impermeable to human red teamers. So how do you go through a sort of iterative process to?

• 49:51Nathan Labenz?

  You know, not just have this sort of kind of security theater of like, well, we had this one, we had one shoe attacker one times and everybody takes their shoes off at the airport and then we, you know, tell ourselves we're good, as if there was no other way, you know, that you could get something onto a plane, right? So, yeah, how do you think about that iterative thing? And how do you know when to stop? Because this could obviously go on forever, too, right? How do you even decide when you're getting to a point of diminishing returns or a point where, you know, you've hit some sort of best, you know, good point on a Pareto curve?

• 50:22Speaker 01

  Yeah, there isn't like a single neat answer. Ultimately, this is a judgment call. The important thing to know is we do a few rounds every time we do the auditing. And then to get certified, the companies commit to doing quarterly audits. They will change their product. The world's incident will evolve. There will be new research papers into new ways these will fail that we then kind of continuously incorporate into this body of tests that we run, that we run in each quarter. In practice, it becomes kind of a cost and effort versus confidence.

• 51:06Speaker 01

  And these are just, in fact, not at all new considerations. Cybersecurity has the exact same thing. No one, with enough effort, every system is breachable. And ultimately, that's pragmatic. And over time, the cost of running good audits goes down as intelligence becomes cheaper, but the set of attack vectors probably also grows. So there is, in fact, a very neat answer for where you stop. Ultimately, this is guided by kind of the market. What are the level of assurances you want? And what are the things that people are willing to pay for to get that assurance? Which is probably like, it's a messy answer, but the actual pragmatic answer for how most of that stuff gets settled in every domain today.

• 51:51Nathan Labenz?

  Yeah. What does this look like in terms of your business model? Like, who pays you? I suppose the developers pay you? How much do they pay you? How does that depend on like, what sort of product surface they have? What does that quarterly update process look like? Is it the same thing as the original? Or is it more streamlined as you go forward? What's like the, you know, the, the overall, you know, customer story for the AI underwriting company?

• 52:20Speaker 01

  Yeah, totally. So the company that's building their system pays. That can either be the, what you think of as like disruptors in the space, these AI native companies. But it can in fact also be enterprises that are developing something in-house. You'll see a bunch of it, whereas that are building their own customer support agents because they think that's a remote and they want still to have confidence before they deploy that. So it's the, the companies that develop the systems that fundamentally pay. They pay in a yearly basis, the certificate lasts for one year. With that, you're required to have these quarterly technical tests.

• 53:00Speaker 01

  And the, the pricing depends, as you're saying, exactly on the, the risk surface, the product surface. And also, of course, this also ties to value. Bigger companies tend to be able to generate more revenue out of this. They also tend to have a bigger risk surface. So it scales a lot. Other parts of the business model that are interesting is the same companies that are interested in getting certified overlap a lot with the companies that want to have insurance.

• 53:28Speaker 01

  And for the same purposes, they want to take risk off the table from their customers and deploy faster. And whereas the enterprise, they want to just reduce the risk so they can deploy faster. And that insurance, as a business model also works on it. You typically buy an insurance policy on a yearly basis for a yearly fee. So you can imagine those laying on top of each other. And then we're experimenting a little bit now with, imagine an AI native. Company that wants to not just get certified for the broader in general, but also want to certify specific deployments to specific customers. They want an extra layer of guarantee. They may want to know that, oh, I'm a bank deploying this AI voice customer support chatbot.

• 54:12Speaker 01

  I want to know that with my specific customer support policies and my specific customer sets, my specific policies, that this will also work. And so that's another way to kind of get more precise in the value. That's where the disruptive is really willing to pay because it's going to unlock a contract. But it's also very clear that it's the maximum level of assurances an enterprise can get when it's done literally in their environment.

• 54:35Nathan Labenz?

  Can you tell a little bit more about what the requirements are from the customer side? And I would be interested to know if you're willing to share like tangible price ranges. Like I assume this is, um, it sounds like it starts in like the maybe five figures and probably goes into the six figures, depending on how big the product surface is. And do they, do companies need to like make team members available to you? Do they have to, uh, or you know, like what, what beyond the sort of desire and willingness to pay, do they have to bring to the table?

• 55:09Speaker 01

  Yeah, the first, before we sign a contract with anyone, we do a gap analysis. So we take the standard and outside in, we take what is the kind of evidence that's publicly available? That gives us a sense of where do they stand against the standard? And we then do an in-depth review like row by row. Where do you actually stand today? That gives us some kind of effort estimate. What's the amount of engineering effort you need? What's the amount of non-engineering effort you need? And also from our side, what is the total scope here? And that's where we together come in and we're like, what's the, what's the price going to be here? The range I mentioned is broadly correct. And that kind of gives everyone a sense of what are the costs involved on those sides? And that's where, um, when they commit, they commit for a year. At the end of that year, they can decide whether they want to recertify.

• 55:53Speaker 01

  This is very kind of in line with how they already think about certifications in the cybersecurity space, things like SOC two, ISO standards. And so really a gap analysis is the core thing that says to everyone, what's the effort that's involved here? What's the cost going to be?

• 56:11Speaker 02

  And we can do that frankly, uh, within 24 hours. And, and what we find Nathan is anywhere from, uh, what our largest customer has thousands of employees. They've been running for years. Um, for them, it's like a small handful of hours of actual engineering lift or security team lift, actually in that case, and just evidence collection for us through to our smallest customer. When we started working with them, had two or three employees and, uh, you know, we needed to work with them in a much more hands-on way to actually implement the security measures that are required in the standards, um, remediate the issues we found through the testing. And, and that was obviously a much more evolved process. And, and so it's kind of hard to give a stock answer without, without doing that.

• 56:57Nathan Labenz?

  Yeah, well, it's going to be, um, seems like something everybody's going to need, right? And in a similar way to whether it's me as an individual driver of my minivan or a, you know, nationwide, uh, trucking company. Everybody's gonna have to have some sort of insurance to offer it presumably before too long.

• 57:16Nathan Labenz?

  I guess, do you think that's true? Like, is there a, is there a class of AI app that just never gets into this? For whatever reason, or do you think this does ultimately like become something that really everybody has to do in the same way that drivers do?

• 57:32Speaker 02

  Yeah, I might separate the insurance from the third-party testing for this, right? I think, I think there is for almost every kind of valuable use case you can imagine. There is a need to have somebody who is not just marking their own homework, actually test it and say it is secure, secure to ingest, you know, customer data or a company's kind of private sensitive IP and, and data and, and make sure that's all being protected.

• 58:00Speaker 02

  Et cetera, et cetera. And then I think on the, um, insurance side, I think that's where you could imagine more of a bifurcation. You know, so traditional software SaaS has been sold on kind of have a nice day basis, right? Often those aren't insured. You might get some service credits if something goes wrong, but you're not, you're typically not going to get more than that. As you said, you know, cars and houses, obviously everyone kind of carries insurance and in lots of cases this is mandated. You could imagine one version of this is a similar kind of bifurcation where the highest risk deployments and the places that are creating kind of externalities also really should be insured because that's where, you know, you want an extra layer of protection

• 58:41Speaker 02

  from a strong established balance sheet to provide the financial coverage. Whereas, you know, if I, if I take your example of, of producing a, producing media copy, you could say actually the, the risk here is, is manageable and, and it doesn't need insurance.

• 58:58Nathan Labenz?

  Yeah, makes sense. Um, I was briefly in the financial services industry, as I alluded to, and one of the things that I happen to have kind of another one of these Forrest Gump, um, extra roles in an important scene in, uh, in recent history was the kind of problem of the credit rating agencies. And, you know, basically what happened in the late 2000s is that these companies were in a sense captured by their customers. They, you know, had the incentive to say things looked good. They didn't have the incentive to like turn over too many stones. And so we ended up with all of these, you know,

• 59:40Nathan Labenz?

  very highly rated bonds going bad. How do you think about preventing that outcome? Like, I guess there's probably multiple dimensions of it. One is just like, how do you make sure that your audits stay weird enough? It almost seems like you need, you know, um, heterogeneous like cognitive profiles on your team to, to make sure that you are bringing, you know, weird and, and off-model ideas to the table as often as possible. But then there is this, this incentive question too, right? Like, at some point, as this gets normalized, there's going to be some pull or there might even be some competition. You know, that's why I think really where the ratings agency things went bad, when it was like, well, if you don't rate us high, we can always go to the other raider and they'll

• 1:00:24Nathan Labenz?

  rate us high. So, you know, that's like, well, jeez, well, what do we do, right? We're going to lose all our business by giving our own customers bad ratings. Like, that's a tough, uh, line to hold. So, you know, you're trying to create a race to the top, but there is at least some precedent for this sort of thing ending up in kind of a race to the bottom. Like, how do you think about making sure, you know, we stay on the right side of history here?

• 1:00:46Speaker 01

  The core thing that's gonna create the right incentives here is the insurance that gives us the skin in the game to not just lower the standards. When you take the Moody's example, Moody's does not have financial skin in the game outside of kind of trust. So when the financial crisis hit, there's not a direct financial mechanisms that that forces them to internalize the risk that's that's being created there. And when you look back at, for example, the UL certificate that's on all of your electrical products, which I think of as like broadly a very successful instance of standards, that was created by the insurers. It started from insurers having data on what's going wrong and having the incentive or yeah,

• 1:01:31Speaker 01

  the direct incentive to make sure that the standard covered that. So we think the insurance piece creates the skin in the game to prevent this race to the bottom. And there's a lot to learn from from history there.

• 1:01:43Speaker 02

  And and in terms of how we keep up with it and keep up with the frontier, this quarterly process is it's actually just really involved from our side. We have a mix of the actual people we're certifying and and the agent builders, right? Because they're on the ground saying, you know, here's a new safeguard that's really working well for us, right? Great, let's feed that in. We have a range of academic researchers and other people kind of building safeguards commercially, giving us input, right? Folks from Stanford and and University of Illinois and Hays Labs and Grace One and Virtu AI. And then with the enterprises, we've just launched our enterprise consortium, which is a mix of security and risk leaders from Fortune 500 enterprises who are actually

• 1:02:28Speaker 02

  also on the front line of the deployments and saying, here are the risks I'm worried about and the risks I'm seeing. And all of these people come together to give us input, both on what's missing, where's the frontier headed, what are the places where A C one isn't keeping up with that? And then how do we write the precise requirement in a way that will capture the spirit of what should be in here? And and so really, it's not just about our team here, which which of course has lots of these capabilities, Nathan, you're describing, but it's really about how do we mobilize the whole ecosystem around this and bring that together.

• 1:02:59Nathan Labenz?

  Do I understand correctly, just to rewind to the insurance piece and the incentives, like, are you planning to actually provide insurance? It it sounds like you're you're both going to, if I'm understanding correctly, like, you're gonna have skin in the game at the insurance payout level. Is that right?

• 1:03:16Speaker 02

  Yeah, we'll have skin in the game, Nathan. The way this works is we'll become a managing general agent. It's the kind of insurance term for it. And in practice, what that means is our payouts, so the insurers will pay us and our payouts will be tied to the actual underwriting result, the profit that they get on the insurance product. So if there are large losses, we don't get paid or we get paid much less. If in fact our risk assessment was good and there's fewer losses, we also benefit from that financially.

• 1:03:46Nathan Labenz?

  Interesting.

• 1:03:47Speaker 01

  And if there are large losses, the insurers will not want to work with us. So we will not be able to, we'll go out of business.

• 1:03:53Nathan Labenz?

  Yeah, gotcha. Is there a big difference between that and writing your own insurance? Is it just a matter of like size of balance sheet that causes you to structure it this way? Or do you imagine like at some point, building a giant balance sheet and like underwriting your own policies and you know, doing your own, your own payouts? Or is there a reason to let the traditional insurance companies handle that indefinitely?

• 1:04:19Speaker 02

  Yeah, insurance and actually reinsurance is often where the risk sits is really a cost of capital game. It's a kind of risk aggregation game, as we were talking about earlier. And so there's a lot of benefit there in being diverse across different risk lines, across, you know, selling through brokers who are selling kind of multiple products, et cetera. Really, we want to focus here, you know, we're focused on the AI risk. And so for us, this gives us the benefit and the flexibility to go out and, and write and structure insurance products, knowing we've got insurance partners, you know, some of the most established insurers in the world, who've never failed to pay out a claim. They've been around for hundreds of years. This is their business and we're, we're really excited to be able to write with them rather than having our own balance sheet.

• 1:05:02Nathan Labenz?

  I've been following this SB 813 proposal for a while. The idea here, as I'm sure you guys are very well aware, is to create some sort of private governance constellation of options where companies could opt in to abide by a certain standard. Maybe you could even be one of these multi-stakeholder regulatory organizations, I think they're called.

• 1:05:26Nathan Labenz?

  And in exchange, the companies get some liability insurance. There's also a bunch of other proposals for liability reform or changes in all sorts of different ways. Do you have ideas on policy? Like, what policies do you think would work best? What are most compatible with the business that you're trying to build? Are there any that are in direct conflict with it? What do you think is the future of that?

• 1:05:49Speaker 02

  Yeah, from our perspective, anything that promotes the creation of a market here and market-based solutions is the key. For the reasons we talked about earlier, that's the thing that aligns the incentives. Because if you have a free market of oversight, you're going to actually incentivize robust risk evaluation without hampering growth. And that's really important, right? We don't want this to become too onerous and crowd out the startups and crowd out innovation. And so there are plenty of policy ideas, like you mentioned, that do allow for this, do promote the creation of best practices and third-party oversight.

• 1:06:31Speaker 02

  And the other thing that's important, though, is which we think SBA 13 didn't do is actually force the companies who are creating the risk to own it, right? We don't want to create safe harbors where you have too many free passes, where you can create risk and then you're not punished for that and you're not held to account for that. And so we think that's a fine line and we think insurance actually allows you to internalize those externalities and actually take on the risk you're owning while still creating the market around it.

• 1:06:59Nathan Labenz?

  Last question, who all is involved? I understand there are quite a few well-known companies that are starting to buy into this. So maybe give us a little overview or name check of who's hopping on the train.

• 1:07:14Speaker 02

  Yeah, on the startup side, we've had an awesome group of founding technical contributors from various different spaces. We work closely with Ada and Intercom in the customer support space. We work with cognition on code. We've worked with a company called Recraft in image generation. So trying to really look across the spectrum of the different kinds of modalities and deployments. And maybe Rajiv, you can speak a little bit more to the kinds of companies that are in the enterprise consortium. Yeah, so we've got risk and security leaders from financial services like JP Morgan, Chase, for example, who are really thinking about how do we adopt AI across the bank globally, not just in specific kind of verticals. In tech, the CISO, the Chief Information Security Officer

• 1:07:59Speaker 02

  of Confluent, for example, is helping to shape AC1. The Deputy CISO at Anthropic is deeply involved, and we've got others from healthcare, from retail, etc. So it's really trying to represent across industries, actually also across geographies. We've got folks from Asia, from Australia, from Europe, and really bring this together into one framework and make sure it's as strong as it can be.

• 1:08:25Nathan Labenz?

  Cool. Well, we're out of time. This has been great. Any last thoughts you want to leave people with, or anything that we didn't touch on, you want to make sure to mention?

• 1:08:36Speaker 02

  I think this is pretty comprehensive.

• 1:08:38Nathan Labenz?

  Love it. Cool. Well, Rude Kavist and Rajiv Zatani, co-founders of the AI Underwriting Company, thank you for being part of the Cognitive Revolution.

• 1:08:47Speaker 02

  Pleasure. Thanks for having us.

• 1:08:51Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 1:09:36Nathan Labenz?

  And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts, so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.