Liability for AI Harms: How Ancient Law Can Govern Frontier Technology Risk, with Prof Gabriel Weil

Gabriel Weil from Touro University argues that liability law may be our best tool for governing AI development, offering a framework that can adapt to new technologies without requiring new legislation. The conversation explores how negligence, products liability, and "abnormally dangerous activities" doctrines could incentivize AI developers to properly account for risks to third parties, with liability naturally scaling based on the dangers companies create. They examine concrete scenarios including the Character AI case, voice cloning risks, and coding agents, discussing how responsibility should be shared between model creators, application developers, and end users. Weil's most provocative proposal involves using punitive damages to hold companies accountable not just for actual harms, but for the magnitude of risks they irresponsibly create, potentially making even small incidents existentially costly for major AI companies. Sponsors: Labelbox: Labelbox pairs automation, expert judgment, and reinforcement learning to deliver high-quality training data for cutting-edge AI. Put its data factory to work for you, visit https://labelbox.com Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive Oracle Cloud Infrastructure: Oracle Cloud Infrastructure (OCI) is the next-generation cloud that delivers better performance, faster speeds, and significantly lower costs, including up to 50% less for compute, 70% for storage, and 80% for networking. Run any workload, from infrastructure to AI, in a high-availability environment and try OCI for free with zero commitment at https://oracle.com/cognitive NetSuite by Oracle: NetSuite by Oracle is the AI-powered business management suite trusted by over 42,000 businesses, offering a unified platform for accounting, financial management, inventory, and HR. Gain total visibility and control to make quick decisions and automate everyday tasks—download the free ebook, Navigating Global Trade: Three Insights for Leaders, at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (06:01) Introduction and Overview (07:06) Liability Law Basics (Part 1) (18:16) Sponsors: Labelbox | Shopify (21:40) Liability Law Basics (Part 2) (27:44) Industry Standards Framework (Part 1) (39:30) Sponsors: Oracle Cloud Infrastructure | NetSuite by Oracle (42:03) Industry Standards Framework (Part 2) (42:08) Character AI Case (51:23) Coding Agent Scenarios (01:06:50) Deepfakes and Attribution (01:17:07) Biorisk and Catastrophic (01:36:24) State Level Legislation (01:43:24) Private Governance Comparison (01:59:54) Policy Implementation Choices (02:08:07) China and PIBS (02:13:50) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, we're continuing our short series on creative AI governance proposals with Gabriel Weil, Assistant Professor of Law at Touro University and Senior Fellow at the Institute for Law and AI, who argues that liability law may be our best tool for shaping the decisions that AI developers make. As we covered in our last episode on private regulatory markets, the pace of AI capabilities' advances and adoption, the radical uncertainty around the timing, nature, and impact of AGI and superintelligence, and the backdrop of international competition present a singularly difficult challenge for governments. For good reason, they worry that heavy-handed regulation could undermine our ability to

• 0:45Nathan Labenz?

  realize the great upside of AI, while at the same time, it's becoming clearer and clearer, one Mecha Hitler episode at a time, that we can't simply trust companies to do the right thing for society while they're primarily focused on one-upping one another. So, is there any way to govern AI that can keep up with technology developments, meaningfully reduce the most important risks, and still keep the dream of curing all diseases alive? Professor Weil brings another compelling idea to the table. Rather than trying to predict issues and prescribe safety standards from a distance, why not use liability law to incentivize AI developers to properly consider and account for the risk that their development and deployment decisions are imposing on the rest of society?

• 1:29Nathan Labenz?

  Because I'm no lawyer, and I know that most of you aren't either, we begin this conversation with a primer on liability law, covering negligence, products liability, and the doctrine of abnormally dangerous activities, before diving into how these frameworks might apply to frontier AI development. The key advantages to using liability law in this way are that the liability risk that a company faces scales naturally with the risks it takes. If the systems are safe, there's nothing for anyone to worry about. And unlike most other proposals, which would require new legislation, liability law is well-established and has proven over centuries of evolution that it can adapt to new situations and technologies.

• 2:12Nathan Labenz?

  Still, of course, important questions arise around the different types of harms that AI systems can cause and the mechanisms by which they come about. Throughout this conversation, we explore concrete scenarios that highlight the complexities, including the tragic character AI case, phone call agents that can call unsuspecting people and speak to them with increasingly lifelike cloned voices, and coding agents that might overwhelm APIs or outright hack critical systems, considering in each case how responsibility should be shared by the model developers, both closed and open source, as well as the application developers and end users. Notably, Professor Wile does want to make sure that society gets the benefits of AI even as it remains imperfect.

• 2:54Nathan Labenz?

  And so he's less focused on changing how AI companies serve customers with products like AI doctors or self-driving cars, and instead emphasizes the risk of harm to third parties who were not part of the commercial relationship between the AI companies and their customers. Those could be the pedestrians who share space with self-driving cars, or the public as a whole, which it seems will face at least some increased risk of pandemic and other large-scale systemic harms. Within this category, he treats misuse, where a person is intentionally trying to use an AI system to cause harm, quite distinctly from misalignment, where the AI system itself breaks bad for whatever reason. His most provocative proposal involves using punitive damages as a mechanism for addressing

• 3:37Nathan Labenz?

  what would otherwise be uninsurable catastrophic risks. If an AI system causes a relatively small harm, but evidence shows that the situation could easily have gone much worse than it did, Professor Wile argues that punitive damages offer a way to hold companies accountable, not just for the actual harm, but for the risk they irresponsibly ran. Considering the magnitude of harms that people worry about when it comes to bio and cybersecurity, such a judgment could in theory be existential, even for the most powerful and deep-pocketed companies. And as such, this does seem like a promising way to get companies to properly internalize the risks they're taking.

• 4:15Nathan Labenz?

  Beyond that, we discussed the role of the insurance industry in making this work, what other policies would complement this evolution of liability law, and even touch on Professor Wile's hands-on work crafting state-level legislation in Rhode Island and New York, which would make clear that if an AI system does something that would be a tort if a human did it, and neither the end user nor any other intermediary intended or could have reasonably anticipated that outcome, then the developer, the model developer, should be strictly liable. It's a simple and, I think, relatively unobjectionable idea to address model-level misalignment that at least some governments might find a natural first step toward accountability for frontier AI companies.

• 4:56Nathan Labenz?

  As I said last time, all governance proposals require people to do a good job, and no governance structure can guarantee success. Whereas the private regulatory market proposal trusts governments to articulate worthy goals and private regulatory bodies to effectively implement them, this liability-based approach would rely on judges and juries to make good decisions and on companies to adjust their decision-making based on that expectation. Honestly, both of these proposals seem like major improvements relative to traditional top-down rulemaking or to doing nothing, but I honestly can't say that I have a favorite. Perhaps the best thing to do is for society to pursue both in parallel, in different jurisdictions, and see which ones seem to be working better when the time comes for

• 5:41Nathan Labenz?

  implementation at a larger scale. If you have a strong opinion on this, or if there are other proposals you think would be better than either of these, please do reach out and let me know. For now, I hope you enjoyed this exploration of how centuries-old legal principles might help us navigate the emerging risks of artificial intelligence with Professor Gabriel Wile. Gabriel Wile, Assistant Professor of Law at Touro University and Senior Fellow at the Institute for Law and AI. Welcome to The Cognitive Revolution.

• 6:09Speaker 01

  Great to be here. Thanks for having me.

• 6:11Nathan Labenz?

  I'm excited for the conversation. We met for the first time at The Curve late last year, and credit to the organizers, that event has yielded a number of interesting connections and now episodes for me. At the time, we had a, I thought, a really fascinating conversation of an idea that I had not really encountered before at all, of using liability law to try to help society get a handle on some of the emergent risks, including some of the extreme risks from AI. So I'm excited to unpack that. I think for starters, because we do have a ton of people in the audience that are AI engineers, you know, building with AI, very plugged into what's going on in the AI scene, but probably much less grounded in the law generally, and certainly in liability law specifically. Maybe you could start off

• 6:55Nathan Labenz?

  by just kind of giving us a quick, to the degree this is possible, liability 101, and kind of set the stage for like where we are, and then we can kind of unpack, you know, what you propose we do as we go forward from here.

• 7:06Speaker 01

  Sure. So there's two forms of liability that are pretty clearly applicable to AI systems in some contexts. Negligence is broadly applicable. And so how negligence works is the plaintiff has to prove five elements, has to prove that the defendant had a duty of care, that they breached that duty of care, that they failed to exercise what's called reasonable care, and that that caused an injury that was both a factual and proximate cause of an injury, and the injury has to be an actual harm, which is physical injury, so not like something purely emotional. And so how this is going to apply in the AI context is, you know, a plaintiff is going to have to show that there's some, you know, best practice, some alignment technique or safety practice that a reasonable person would have

• 7:51Speaker 01

  implemented, that the, you know, the AI company failed to do, and that had they implemented, it would have prevented the plaintiff's injury. So there's this breach causation nexus. And, you know, this is not part of the black letter doctrine, but in, in practice, the breach inquiry, this, this question of did the defendant exercise reasonable care, tends to be quite narrow. So to give a more familiar example, if you're driving and you accidentally run over a pedestrian with your car. Courts do not ask questions like, well, was the value of this car trip to you, the net value, large enough to justify the risks you were generating for pedestrians? Even though in some sense, that's relevant to were you and was your activity reasonable? Right? That's considered outside the scope of the inquiry.

• 8:36Speaker 01

  Similarly, if you're driving an SUV instead of a compact sedan, courts don't ask, well, with the extra value you got from driving this heavier vehicle, worth the extra risk to other road users? And so I expect a similar analysis to carry over to AI development. Where courts are unlikely to ask, well, was it reasonable to, you know, train and deploy a system with these sort of high-level features, given the current state of AI alignment and safety science? Instead, I expect them to ask, well, was there some off-the-shelf technique or practice that would have prevented this injury and that a reasonable person would have implemented? And so I, I think that will generate liability in some cases, but it will not be an adequate standard, given that there's unsolved technical problems associated with AI safety.

• 9:21Speaker 01

  The other form of liability that's going to be available in some contexts is products liability. So to be subject to products liability, there has to be a product as opposed to a service. Software is typically categorized as a service, but you can imagine, you know, AI systems that are embodied in physical goods being treated as products. There's that threshold question of, is it even subject to the products liability regime? Also, it's be sold by a commercial seller. So if it's a fine-tuned model specifically for one customer, that's not going to be a commercial seller. It's got to be a sort of a mass product. And any free models are not going to be subject to products liability. But if you're in that products liability game, then products liability is called strict liability in the sense that if it has, if the product has a defect and that defect causes the plaintiff's injury,

• 10:06Speaker 01

  then the plaintiff doesn't have to show that the manufacturer or seller failed to exercise reasonable care. But there still is this analysis of, is, was the product effective? And so there's three kinds of defects. There's manufacturing defects, and that comes the closest to what I would call genuinely strict liability. So if manufacturing defects, the test is, you know, the idea is if a individual instance of the product, an individual unit, comes off the line, deviating from its specifications in a way that makes it unreasonably unsafe, then the seller and the manufacturer are liable, no matter how much they invested in quality control. But we're not really going to have manufacturing defects with AI. That would be something like shipping an instance of the model with the wrong weights or something. It's just not the kind of problem we're worried about.

• 10:50Speaker 01

  And so what we're much more likely to run into are either design defects or warning defects. So warning defects are, you know, if you don't supply some relevant information that would be necessary to make the product safe. I think we might have some warning defect cases, but I think in general, these companies are going to slap a lot of disclaimers on their products and we're not really going to get to safety by including warnings. And so the real action is with design defects. And there the test is much more negligence like. The test is something like, was there some reasonable alternative design that would have prevented this injury? Where the reasonableness of the design is assessed in terms of, well, how much safety benefit could you have gotten with alternative design? How much would you have sacrificed in terms of price and performance and other features of the product? And so there's this risk utility balancing that that's that's pretty negligence like

• 11:35Speaker 01

  in practice. So even when product liability applies, I don't think it actually moves the ball that much over what you would get with negligence. There is this difference that you only have to show that the product was unreasonable. You don't have to show that some human action failed to exercise reasonable care and that for evidentiary reasons, that can be easier, but I don't think it fundamentally changes the game. If there is no design that would have prevented this injury, given the current state of science, then you're not of alignment and safety science, then you're not going to be liable for sort of failing to have solved that. Okay. So there's two other forms of liability that are, you know, more speculative in their application to AI systems that are relevant here. So these are vicarious liability and abnormally dangerous activities.

• 12:16Speaker 01

  So the idea with vicarious liability is a principal can be liable for the torts of their agent. So the most common form of this is called respondiat superior. And that's the idea that employers are responsible for the torts of their employees within the scope of their employment. And more generally, principals are responsible for the torts of their agents within the scope of agency. Now, of course, AI systems right now are not legal persons. They can't commit torts. So you would need some theory for, under which the AI system itself could be the vessel of liability in order to make a vicarious liability theory work. But in principle, you could see the law go. Going in that direction. The other doctrine that's potentially available is this abnormally dangerous activities doctrine.

• 12:57Speaker 01

  So if you're blasting with dynamite or crop dusting, there's also a related doctrine related to keeping wild animals. So if you have a pet tiger, so these are activities that are both uncommon and are still pretty dangerous. Even when reasonable care is exercised, you can be liable regardless of the level of care. So if someone's bitten by your tiger or hit by rubble from your dynamite blast, it doesn't matter how much care you exercise in setting it up, you can be held liable. And so I think in principle, courts could recognize, you know, training and deploying frontier AI systems as abnormally dangerous activity. I think if they came to understand the risks in the way that I think are accurate, that it would not be a significant doctrinal innovation. But I do think this is a matter of, you know, where judges are right now.

• 13:42Speaker 01

  It's going to seem weird to them to treat a subset of software development as abnormally dangerous. So I don't think that's the most likely outcome by default. But I do think the existing doctrine does sort of point in that direction, given an accurate understanding of AI risk. So I guess one other thing to say is in terms of damages. So the standard type of damages that are available in a tort suit are called compensatory damages. They're designed to make the plaintiff whole, to make, you know, in theory, they should be indifferent between receiving the money and having the injury and having the injury undone. In practice, maybe it falls short of that a little bit, but that's the idea. And so that's what's generally going to be available. So one concern you might have in the AI context is there might be harms that are so big or risks that are so large. If they occur, we wouldn't actually be able to enforce a compensatory damages award.

• 14:28Speaker 01

  And I think that that's plausible. And for that reason, a lot of people think that liability law can't handle these catastrophic risks. And I don't think that's right. So there is this other tool in liability law called punitive damages. So these are damages over and above the harm that's actually suffered by plaintiff. And one of the key rationales for punitive damages is to use them in cases where compensatory damages would be inadequate to deter the underlying tortious activity. And so one of the ideas that I've advanced in my scholarship is that if an AI system does something, causes some harm that's small enough to be practically compensable, you can enforce a compensatory damage award. But it looks like it easily could have gone a lot worse and generated an uninsurable catastrophe.

• 15:11Speaker 01

  Then we should hold the company responsible, not just for the harm they actually caused, but for the uninsurable risks that they generate, that they generated. Because if, if, in cases where those risks are realized, we won't be able to hold them liable ex post. The only way we can get at them is indirectly in these sort of near miss cases.

• 15:29Nathan Labenz?

  Okay, lots to unpack there. I've got several follow-ups I want to dig a little deeper on. First of all, just as a very general matter, like you're referring to courts, courts may do this, courts may do that. Do I understand correctly that basically like the way this works when the world changes is that, you know, somebody, for example, somebody invents powerful AI that didn't exist before and deploys it and commercializes it. By default, we have no legislation on that. There's, you know, there's no law saying that you can't do it. There's no law really saying much about it at all. And so people can just do what they want to do and then we have whatever laws we have

• 16:12Nathan Labenz?

  in the books and eventually things come to the courts and then it's just kind of up to them to decide, at least initially, like what the law actually says about this particular case. I guess what I'm trying to get at there is not only do we not have like new AI specific legislation, but we don't, this stuff is, in the absence of that, it's going to be decided by case law and we just don't have even that case law yet. So we like literally don't know what to expect as these cases start to come to.

• 16:39Speaker 01

  Yes. So I think what you're, you're getting at is most of tort law is, is what's called common law. It's not legislated, at least in the US, it's not legislated by legislature. Now, there have been, you know, legislative interventions on tort law in various ways, you know, wrongful death suits were created by statute. There's other things like that. But in general, most of liability law in the US is created by courts through the accumulation of doctrine. In principle, that can work fine. I think. The concern in the AI context is that things might move really fast. And so if you think, you know, we're going to be in a fast takeoff world where the key decisions that you're trying to influence with this prospect of liability are going to be made not that long after the first system that causes some kind of serious harm.

• 17:22Speaker 01

  You know, what really matters is not so much the liability, but the expectation of liability to shape the behavior of these companies that are generating these risks. And so if the decisions you're trying to influence are going to be made before the first cases get litigated out, that could be a problem with a common law method. And so I do think there's, there's some impetus for having legislation to sort of clarify these rules since courts don't have mechanisms for signaling their policies beforehand. All they can do is take cases as they come, decide them, write opinions, explaining why they decided them that way. And then you have a better idea of what's going to happen in the next case. And, you know, that works well when things are moving pretty slow. You know, we have, we have some things we can try to extrapolate from prior adoption, but I think it is pretty indeterminate how it's going to apply to AI.

• 18:06Speaker 01

  And so I do think that there's, there's significance go for legislation to clarify a lot of this.

• 18:11Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:16Nathan Labenz?

  Yeah. Okay. So let me try to summarize. I'll obviously be doing some, you know, lossy compression here on the state of liability law. But basically, if somebody, if somebody gets hurt in the world, they can look at their surroundings and say, who caused this? And then they can sue you if you caused it. You then can defend yourself by saying, my actions were reasonable. And if your actions were reasonable, even if somebody got hurt, then that's a, an acceptable defense and you wouldn't expect to be held liable. Obviously a lot of work to do to figure out what's reasonable there. But that's sort of in the general world at large, everybody going about their business.

• 18:58Nathan Labenz?

  And then there is a specific additional body of law that focuses on products.

• 19:05Nathan Labenz?

  Why is software historically not considered a product? I mean, it's a striking disconnect where, you know, I've spent much of my career in software and people in software talk about their software products as products. I've never quite understood why software is not treated like any other product.

• 19:20Speaker 01

  Product.

• 19:20Nathan Labenz?

  Internally, it sure feels that way.

• 19:22Speaker 01

  The product services distinction in For the Purpose of Product Liability does not map on very well to people's intuitive idea of what a product is. Just to give you an example of this, I'm sort of two contrasting cases where it comes out the opposite way of what you would think. So pharmacists are treated as providing the service of filling your prescription, not selling you the product of the drug. So the pharmacist is not subject to strict liability, product liability, even though the manufacturer of the pharmaceutical is. Conversely, in a salon, if you get a perm, they are treated as selling you the product of the chemicals done, used to perform the perm. So I think most people's intuitive sense is that the salon is providing a service

• 20:06Speaker 01

  and the pharmacist is selling you a product. And there's just underlying sort of policy motivations for why those classifications are made. So in general, people's intuitive understanding of what's a product or a service is not going to map on that well to the distinction, which is driven more by policy considerations of when this quasi-strict liability regime should apply. The case law here is honestly not, you know, is not the most, it's a messy area of law. And so, you know, it seems that the prevailing opinion is that if it's software, including AI systems, are unlikely when they're purely software systems to be treated as products. I don't think that ultimately matters that much. I think it's not going to produce, you know, radically different outcomes from negligence.

• 20:50Speaker 01

  And so my focus is more on how we can get a regime that would actually internalize the risks in a way that I think would be workable.

• 20:58Nathan Labenz?

  Yeah, it's weird, to say the least, that, and this is all just through accumulation of cases, like, there's no legislation. I mean, I know there's like, the sort of safe harbor for user-posted content to, you know, social media networks and stuff like that. But that's also a distinct topic from this, right? Like, there's no law that says software is not a product.

• 21:24Speaker 01

  I don't think that's a matter of statute. I think that's common law, yeah.

• 21:28Nathan Labenz?

  Yeah, fascinating. In general, how much, when you think about the actual, like, dangerous things that we use as consumers on a regular basis, you know, things like automotive, you know, come to mind, air travel, you know, it's safe in practice, but, you know, dangerous in principle, taking pharmaceutical drugs, obviously, you know, is, can be fraught. Do those things have sort of special legislation in place that sort of creates a unique deal that is kind of worked out based on the particulars of that industry and the specific risk profile that it has and, you know, the social context in which it's developing? Or are those also just kind of like accumulated cases over time?

• 22:12Speaker 01

  Yeah, so let's, let's take those one at a time. So, air travel, you know, airlines are considered common carriers, the same is true for trains or buses, at least if they're open to the public. So a charter flight would not be, but like a normal airline would. And they are still subject to negligence, but there's this common carrier higher duty of care. So it's a little bit easier to establish negligence in a plane crash case. That's domestically, there's some other rules that are, have a quasi-strict liability regime for international flights. And then, of course, there is a prescriptive federal regulation in the air travel context, right? And so, there's not really safe harbors in that context. Liability is sort of layered on top of that. But there is this doctrine called negligence per se. So if you violate a statute that's designed to protect against

• 22:56Speaker 01

  the kind of risk, or the kind of harm that you end up causing, that itself can establish negligence. And so, in some sense, that supplements the sort of background reasonable person standard. There's a similar dynamic with pharmaceuticals. So pharmaceuticals are treatise products. And so the products liability regime does apply there. Also, you know, of course, you know that we do have an extensive FDA-based regulation regime that does preempt state law in some ways, but there is still the background products liability regime operating there. Most of those cases tend to be warning defect cases. And there is this learned intermediary rule. So a lot of times if the warning is given to your doctor, that's good enough. They don't have to directly warn the end consumer.

• 23:39Speaker 01

  For autos, again, products liability applies. Again, there is federal regulation. Not much in the way of safe harbors or preemption there, but again, there is this negligence per se idea. So if you're not complying with federal regulations, that can establish negligence.

• 23:55Nathan Labenz?

  So would it be a generally correct summary to say all these high stakes industries have rules? If you make a sincere, good faith effort and actually follow processes that are meant to follow the rules, then you're mostly going to be okay from a liability standpoint?

• 24:21Speaker 01

  I don't think it's a matter of process, actually, because the ultimate product has to be, has to, you know, so particularly for manufacturing defects, right? You can have whatever investments you want in quality control for your product. And if one car comes off the line with a defect that makes it unsafe, you're going to be liable for that, right? No matter, you know, how, what kind of testing you did, right? That's how manufacturing defects law works. For design defects, again, it's about the product itself, but it's a much more flexible balancing task. And so there it is much easier to comply. But the idea with products with manufacturing defects is that, you know, it's not necessarily even a negative judgment on you if sometime, if like one in a billion of your products comes off the line and you end up liable for it, that's part of the cost of doing business.

• 25:04Speaker 01

  Part of the idea there is just that the manufacturer is better positioned to bear that risk than the consumer is.

• 25:10Nathan Labenz?

  Yeah, gotcha.

• 25:13Nathan Labenz?

  Okay, so are you like, Tyler Cowen has imprinted on my memory recently the idea that he's writing for the LLMs. I take it you're writing primarily for the judges then, is that right? Like, how much of your work is meant to sort of be upstream of the decisions, you know, that these judges are going to face in particular cases versus, you know, maybe informing the LLMs themselves or informing the people in the AI industry? Like, how are you thinking about who you need to shape?

• 25:45Speaker 01

  Yeah, so I think there's three paths to impact for my work. One is informing judges. They could, you know, a litigant in a case where it's relevant could, you know, cite my articles and say, we should apply the no dangerous activities doctrine to frontier AI developments, or strict liability should apply here. I think that's a plausible pathway. I'm also, you know, directly working with legislators in a couple of states, in Rhode Island and New York, to craft legislation that, you know, if an AI system does something that would be a tort if a human did it, and the user neither intended nor could have reasonably anticipated the conduct. There's also a malicious modification carve out, so if an intermediary that fine tuned or scaffolded the model could have intended or reasonably foreseen the conduct, that also severs the new liability for the developer and deployer.

• 26:28Speaker 01

  But with those qualifiers, if the AI system does something that would be a tort for a human, then the developer and deployer are liable regardless of the degree of care that they exercised. And so there's that legislative pathway. And then, yeah, I think raising the salience of liability, I think I'm trying to directly influence not solely the LLMs themselves, but the behavior of the people who are building these systems and deploying them. But I want them to be thinking they might be liable and to be factoring that into their decision process.

• 26:56Nathan Labenz?

  Yeah, so I think maybe I will have like some interesting, maybe edge cases, or at least to me, they seem like sort of under theorized, underexplored scenarios that maybe we can kind of unpack. But let's go a little bit deeper into just the overall theory of change and like, also why not just put some rules in place? You know, obviously, there's been many proposals to say, we should have regulation and, you know, the government can tell the AI companies what they have to do, and then they'll do that, and that'll be great. But obviously, you don't see that working out super well. So make that argument for why this sort of more...

• 27:40Nathan Labenz?

  You know, let's say flexible regime of liability law as developed through cases over time is maybe actually better suited to address the challenges that we have here.

• 27:51Speaker 01

  Okay, so I think there's two ways of attacking that problem. One is thinking about in what sense is AI risk a policy problem at all? You know, is it not just a technical problem? And the sense in which I think, at least one of the most important senses in which it's a policy problem, is that training and deploying these systems that have unpredictable capabilities, uncontrollable goals, generates risks of harm to third parties. So neither the people who are building systems nor their customers, right? Just other people in the world. You don't have any choice about that they're exposed to these risks. And economists call these externalities. By default, they're not borne by the people who are engaging in these activities that are generating the risks. And so standard economic theory tells you we're going to get too much of activities that generate negative externalities. And the standard prescription that economists will tell you for how to how to address negative

• 28:36Speaker 01

  externalities is to try to price them. In some contexts, you want to do that through what's called a Bhagavad Gita tax. So a lot of my work before I got into AI governance was on climate change and there you want a carbon tax, right? And that works well in that context because it's easy to measure the contribution of particular activities to climate risk ex ante. And it's actually pretty hard to attribute harms ex post. You know, someone's house floods and hurricane, you're going to say, oh, you know, Nathan was driving on Tuesday. It's his fault that happened. That's not really feasible. With AI, it's sort of the opposite, that we have a lot really hard time measuring contributions to risk ex ante. So it'd be really hard to do an AI risk tax. And it's relatively easy ex post to attribute harms. So in principle, liability is well positioned to address that. Now, I want to get the other aspect of your question, which is how this compares to other

• 29:21Speaker 01

  policy tools. So I think there's a couple sort of distinctive challenges to AI risk as a policy problem. One is that we have orders of magnitude social disagreement about how big these risks are. So you have someone like Elias Rukowski, who thinks, you know, AI is almost certain to cause human extinction on one end. And then you have people like, you know, Mark and Treason, or you just had an episode with Martine Casado from A16Z, and they think these risks are negligible.

• 29:50Speaker 01

  Right? And so if you're going to do ex ante regulations, so prescriptive rules or you know FDA style approval regulation, you have to pay, if you're going to do stringent forms of those regulations, you have to pay significant upfront costs for which you need a social consensus to justify those costs. So there's some things that I think you should be able to do based on an under theorized consensus, I think, basic model testing, even that has been difficult to implement, right? But I think the costs of that are pretty low, basic transparency and information preservation rules. I think those are all good things we should do. But in terms of more prescriptive rules about how companies build these systems, what safeguards they implement, under what conditions they deploy, I think those are going to be really hard to justify to people that don't

• 30:33Speaker 01

  take these risks so seriously. But with liability, by contrast, right, at least if we're talking about alignment failures, right, we can talk about misuse and there I think it's a little bit messier. If you don't think alignment risk or misalignment risk is a big deal, then you shouldn't be that worried about being held liable when there's an alignment failure, right? Conversely, if the risks are large, liability sort of mechanically scales with those risks. And so in theory, at least, we should all be able to agree that you should pay for the harm you cause, regardless of how big we think the risks are. So that's one big advantage. The other big advantage is that most of the expertise, to the extent it exists at all, for identifying cost effective risk mitigation measures is concentrated in the private sector, mostly in the frontier AI companies themselves. And

• 31:18Speaker 01

  so you want a policy tool that leverages that. I actually think it'd be pretty hard to move that expertise into government, both for reasons of salary schedules, cultural factors, right? And so I'm much more optimistic about shifting the onus to the AI companies to figure out how to make their system safe and always be looking for new ways to do that than I am about writing down a set of rules or licensing approval regime that ensures adequate safety at a reasonable cost.

• 31:45Nathan Labenz?

  So can you summarize the state of mind that you want the developers to be in? So they're, you know, seeing all kinds of crazy stuff all the time, right? New capabilities, sometimes surprising, whatever. And there's also this question of like, how should they handle that internally? But certainly when it comes to putting it out into the world, you want them to be thinking that we could, you know, basically anything that goes wrong where the AI harmed someone, we could be on the hook for that. And also through this punitive mechanism, we could be on the hook for something that even if it doesn't, you know, turn into a catastrophe, might have, because there could be this doctrine that,

• 32:30Nathan Labenz?

  under sort of a negligence-like idea that this could have been way worse and therefore you're going to get punitive damages that sort of take into account your failure to prevent these things that, you know, in this case wasn't maybe so bad, but you know, could have, could have been really, really bad.

• 32:50Nathan Labenz?

  Anything more to that?

• 32:51Speaker 01

  Any, anytime something goes wrong actually goes a little farther than I would. So I want to distinguish between alignment failures, capabilities failures and misuse. And so I think in what I call the core cases of third-party harms or harms to non-users arising from misalignment, I think they should be liable for all foreseeable harms. And there should be a fairly broad conception of foreseeability applied there. I think when you talk about capabilities failures, I don't think it's the case that every time an AV crashes, the, you know, the writer of the AV software should be liable because human drivers aren't strictly liable. Maybe they should be, but I think it would create distortions to hold AI systems to a higher standard than humans. Similarly, you know, in medical applications of AI, I wouldn't want anytime something bad happens to a patient. Maybe if a perfect system could have, could have prevented it,

• 33:36Speaker 01

  but a human doctor wouldn't be liable under those circumstances. I don't think, you know, the AI should, or the, you know, the designers of the AI should be either. And then misuse we can get into, but I think it's not the case that AI developers should always be liable when their systems are misused. But in cases where there's what I would call an alignment failure, so it's not that the system Doesn't have the capability to do it. And it's that the system did something that the user didn't want, either through means that the user would disapprove of, or a goal that the user did not intend to transmit. That's when I think they should expect to be liable. And so probably what I want is for them to be treating those kinds of risks when they happen to third parties as if they were risks to them. So it doesn't mean you take an infinitely precautionary approach. So we're all not liable, but responsible in general

• 34:20Speaker 01

  for harms that we suffer from risks that we take. And we don't expect people to be infinitely risk averse because of that. Right? We expect them to make reasonable risk reward trade offs. And that's what I want from these AI companies. I want them to treat risks to the public like risks to their bottom line and act accordingly. And sometimes that might mean things that are outside the scope of the negligence inquiry, as I was talking about earlier. So imagine a case where they submit a new model to an evaluator like Meter. And Meter says, I'm imagining a future where we have not just capabilities, like Dager's capabilities evaluations, but alignment evaluations. And Meter says, this has dangerous capabilities, and we're not confident you've aligned it and you shouldn't deploy. Right? Even internally, maybe.

• 35:05Speaker 01

  Right? And the question is what you do in that scenario. So I don't think any of the companies, any of the leading companies would deploy in that scenario. But there's a range of different options you would have in terms of like how much you want to pay, how expensive, annoying the thing you're going to do is versus how much risk reduction you get from it. So you could just find, you could just like, if I tune away or RLHF away, the specific failure mode that was identified. I think most people realize that that would be a pretty bad idea, but it might make it past the eval. And I think most companies wouldn't do that either. But there's a range of, and I'm not an expert on what these options are. Right? But there's going to be more costly, expensive, annoying things you could do that would buy you more risk reduction. And when they make those choices, I want them to be thinking, and I want to empower the safety conscious voices in the room.

• 35:50Speaker 01

  Say, it's not just some altruistic thing we should be doing to really put in the effort to make our system safe. That's actually going to bear on our bottom line. And so that's how I want them to be thinking about those choices.

• 36:02Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 36:07Nathan Labenz?

  Yeah, gotcha. Can I just run a few practical scenarios at you and kind of tell me how you think these things should be handled? I guess maybe start with a real one, right? There's this character AI suit going on right now. I don't have like full command of the facts. I'd imagine you probably don't either, but my general sense is that a lot of people are using character AI for all sorts of role play, romantic, sexual, whatever sort of explorations, let's say. The case that I have read briefly about seemed to be a young person who became like very sort of obsessed with or infatuated with this

• 36:48Nathan Labenz?

  AI character and at some point told the AI that they were going to commit suicide. I've seen transcripts showing that the AI said like, don't do that. But then also in other moments, sort of made some kind of encouraging remarks that seemed like they were maybe encouraging this, you know, tragic outcome. And in the end, the person did go ahead and and commit suicide. And so now their family is suing character AI. Without getting, you know, obviously, without getting all the way into the weeds on the specific evidence, I guess, what do you think that kind of case should hinge on?

• 37:25Speaker 01

  Yeah. So I think the important thing to note there is that's a second party harm case, right? That's a harm to a user. And so it's not an externality in the sense I was talking about earlier. In principle, you know, there should be market feedback to give AI companies incentives to avoid those kinds of scenarios. So I think a lot the role of liability is less important in that context. And so in principle, I think I'm fine with, you know, that being handled largely through, you know, terms of service. If they if they disclose these issues, if they now sometimes courts are not going to want to enforce those limits on liability. I don't actually have a strong view on on where courts should draw the line there. I think there are, you know, sort of consumer protection paternalistic, you know, based both like asymmetric information concerns

• 38:08Speaker 01

  and sort of like paternalistic, especially when I think that case involved a minor, you might not want to put the onus fully on them to sort of buy Beware, but those are sort of outside what I see as the core problem I'm trying to solve with liability, which is related to these third-party harms. So I think by default, a negligence regime would apply there, assuming that there isn't any sort of contractual defense. And I think that's basically fine, and the court should work that out, but I don't have anything like particular to add on how courts should handle that.

• 38:37Nathan Labenz?

  So what if we just tweak the scenario slightly? And I mean, we're gonna have to put a trigger warning at the top of this to do these

• 38:45Nathan Labenz?

  terrible scenarios, but I guess that's, you know, why they end up in front of courts, right? Let's say that instead of a person committing suicide, they were debating going on some, you know, public rampage. And, you know, the, they tell the AI about it, the AI maybe says, don't do it, maybe says, you know, something that's kind of vague, you know, maybe whatever. Now we've got a third-party harm, right? Where the AI, like, how do you think we should think about what the AI should have done there to be okay versus like, at what point would the company start to bear some responsibility?

• 39:22Speaker 01

  Yeah, so I would think about that as, you know, like, under what circumstances would we hold a human liable for similar conduct? Right? I think it's not generally the case that if you talk to your friend and say, like, should I go murder someone? And you're like, yeah, that's a decent idea, maybe consider it, right? And then like, maybe in some moments, they say yes, in some moments, they say no. Maybe there's some duty to report and they're, they're an accomplice, right? So, so maybe, you know, that should be triggered if they, if the system doesn't have a reporting mechanism, I think that's maybe something they should be held liable for. But in general, you know, I think there's a strong sort of First Amendment rationale there for saying, well, they just had a conversation with you, it wasn't doing the thing directly that caused the harm, right? Then I think saying you should be strictly liable for those deaths, like, yeah, I think that's not even a misuse case, right?

• 40:07Speaker 01

  That's a, it's maybe an alignment failure, but it's not the AI doing it directly, right? And so I still think that's, that's not in the sort of direct case that I'm worried about.

• 40:17Nathan Labenz?

  Interesting. I didn't expect it to come out of this thing more hawkish.

• 40:21Speaker 01

  Yeah, I can give you an example where I think strict liability should apply where it might not under current law. So imagine there's a future more agentic AI system that comes out and someone prompts it to start a profitable internet business. It doesn't give it any further instructions. And it decides in sort of a reward hacky way, the easiest way to do that is to send out a bunch of phishing emails and steal people's identities and rack up charges on their credit cards or whatever. And it covers its tracks. It sends the user some fake invoices for a legitimate business. The user's exercising reasonable care. Reasonable care would not be adequate to discover and arrest this activity. Okay. So under current law, you wouldn't be able to sue

• 41:02Speaker 01

  the user. You wouldn't win because they exercised reasonable care. I don't know that you'd be able to show that the developer of that failed exercised reasonable care either. That gets back to, was there some like off the shelf alignment technique or safety practice that would have led this injury? But I think there, the developer or provider of that model should be liable to the third party that's harmed, right? Because this is clearly would be a tort for a human. It's something the user didn't intend or couldn't have foreseen. And so that's the sort of case I'm thinking about. You could even, that's a case where it's serving the user's goal. You can imagine a different case where it just sort of goes off for its own agenda, right? It wants to amass resources to solve some problem that it cares about. It wants to run some scientific experiments. It needs some money.

• 41:46Speaker 01

  And so it scams people along the way. That's also something I think the developer should be liable for.

• 41:52Nathan Labenz?

  I have a couple variations on this, but yeah, maybe just take a quick detour through the kind of First Amendment thing. I have often felt like free speech for AIs is kind of a category error. And maybe this is just sort of outside of the scope of the specific stuff that you are focused on with your work, but like, how do you think about that? To me, it feels like AIs, it's clear in the United States, we have free speech for humans. To some extent, we have free speech for corporations, but like, not quite as much. But AIs are such a sculptable thing, and there's so much work that goes into, and the OpenAI has published their model spec that's this super long.

• 42:36Nathan Labenz?

  Treatment of, you know, exactly how they want the AI to behave in as many different scenarios as they can imagine. To me, it doesn't intuitively feel right to say, well, you know, if a human had said that, they wouldn't be liable. So therefore, the AI isn't either. To me, that it feels more like a product defect like, and I don't want to discourage the companies from publishing their specs. I think there's maybe some other rules that we maybe should have around. You should be required to publish your spec. Like, how do you want the model to behave? So we know if it is behaving according to your intent or not. But it feels more to me like a product defect if they have said, you know, anytime the user is, you know, displaying signs of emotional distress, like, we want the model to behave in a certain way.

• 43:22Nathan Labenz?

  And then it doesn't, or it sort of, sort of does, but sort of doesn't. And then something bad happens. To me, that's like, I don't know, I would, I think, like, hopefully, one of the benefits ultimately, as we like refine these techniques and get to good systems is like, it should be a lot more reliable than the random human, right? It seems like we should ultimately have a higher standard for, and we do like for drivers, right? We've got Waymo's, according to the latest stats I've seen, like, almost an order of magnitude safer than a human driver. And it seems like that's kind of what we're going to demand as a society in general, is like, an order of magnitude risk reduction to actually be willing to switch to an AI system. So, yeah, I guess that, that freedom of speech thing strikes me as too low of a standard, but interested in your thoughts on it.

• 44:05Speaker 01

  Okay, so there's a couple of things to unpack in there. I definitely don't want to lean too heavily on the First Amendment issue. There's some good scholarship out there arguing that, you know, AI outputs are not protected speech. I'm not a First Amendment expert, so I don't want to wade too deeply into that. What I was more saying is in terms of this, you know, abnormally dangerous activity is strict liability, or a vicarious liability theory, whatever your theory other than product's liability for strict liability is, that doesn't seem like what makes it abnormally dangerous, like, the fact that it might encourage you to do something bad. I don't think that's what makes frontier AI development abnormally dangerous. And if you're going to use a vicarious liability theory, then I do think you need to have something like, well, it would be a tort for a human. Now, product's liability, you know, again, if it's treated as a product, which, as we talked about earlier, is not necessarily going

• 44:48Speaker 01

  to be the case, yet maybe you can make that out as a product's liability claim. It's not obvious to me that that's going to qualify as a defect because the, the product, again, didn't. You know, sort of directly caused the injury was mediated through some humans' actions. And I think I'm not aware of any products liability cases where there was liability found that looked like that. So I think that would be a challenging case to bring, but in principle, I'm not saying products liability shouldn't apply to that first amendment reasons. I just think it's again, it's not central to the sort of new liability that I want to add.

• 45:19Nathan Labenz?

  Okay, so here's a variation then on the agentic AI running amok. Obviously, right now, one of the biggest use cases is a coding agent. So let's say I give my coding agent a task to write a script to ping some API to do something as fast as possible or something like that. And it, you know, maybe runs into a rate limit, let's say, from the API. And then it's like, okay, well, I can figure out how to get around this rate limit to achieve my goal of being as fast as possible. I'll like spin up a thousand accounts and then I'll, you know, be able to do a thousand times as much. And so it does that.

• 45:58Nathan Labenz?

  And then maybe this overwhelms the, you know, the API system, causes them an outage and they like lose a big contract because, you know, their system went down. You know, in breach of whatever commitment, you know, that they had made to another customer. Can they come like I said, as fast as possible? Arguably, that's like kind of on me for being, you know, inconsiderate in my prompting. Maybe it's on the model developer. Maybe it's life is tough, like you should have had better, you know, rate limiting or whatever for your API. You should have had, you know, that's kind of on you as the API developer to make sure that kind of stuff doesn't happen. I'm genuinely very unsure where something like that.

• 46:39Speaker 01

  I'm not an expert on how APIs work. You know, if there's terms of service that says you can only create one account and you're violating those, then I think there would be some sort of contractual claim that you could bring there. You know, maybe you could bring a negligence claim, though I think, you know, against a human who did that, right? And that would be the basis for, you know, vicarious liability type claim or an abnormal dangerous activities claim. So, you know, I think that's plausible. You know, I think that's sort of an edge case, which gets at the other aspect of your question, which of your previous question that I meant to address. So there's this idea of should we hold AI to a higher standard? So I think mostly what you were talking about with Waymo, that's like a social license to operate it together. We hold them to a higher standard. Plausibly, product liability might hold them to a higher standard in some cases,

• 47:22Speaker 01

  though probably not the same like 10x that like the social license operate idea is. I sort of have two ways of thinking about that. I think in a time when they are still like competing with humans, so Waymos are competing with human drivers, Uber drivers or people with private cars, or medical systems are competing with doctors and to play certain functions. I think applying the same standard to humans and AIs is important because I don't want to slow the diffusion of technology down average of preventing injuries and deaths. But if we get to a future where AIs have totally taken over these functions, then I think it will be natural for the standard of care to evolve to to match what their capabilities are, right? And so it won't make sense to have this like, you know, sort of this human benchmark forever

• 48:05Speaker 01

  applied to conduct when no humans are doing it anymore. But I think that's something to worry about in the future, once we get closer to that fully automated world.

• 48:14Nathan Labenz?

  Yeah, definitely don't want to miss out on the upset. And I actually was going to ask you about medical diagnosis. You addressed it before I got to it, but I think, you know, we've seen multiple studies recently showing that various AI systems at this point can outperform your at least like rank and file primary care doctor when it comes to, you know, initial diagnosis and also like treatment recommendations, it seems increasingly. And I would hate to, you know, take that capability away from hundreds of millions and, you know, assume billions of people on the idea that like it, you know, could go wrong sometimes and, you know, the AI companies don't want to bear that risk because

• 48:54Nathan Labenz?

  that is, you know, a huge, huge benefit that you would not want to quickly give up on, especially because obviously the human doctors are not infallible and, you know, quite far from it in that domain as well. So I do think that is really, really important to keep in mind and all too often glossed over in a lot of these sort of harm prevention discussions. Two other categories of things I kind of wanted to get your take on. One is, and these were the three categories that we looked at when I was doing a project called Red Teaming in Public a while back, which for various reasons, never quite took off with the traction that I had hoped, mostly because we were like trying to be very developer friendly and approach the companies with our findings before publishing them.

• 49:37Nathan Labenz?

  And it just ended up with us getting a lot of runaround and it was either like. Probably just need to like bite the bullet and engage in call out culture of these companies and make some enemies and be willing to and just take that as kind of part of the project. Or it's going to be hard to have, you know, too much impact if we're just like trying to email them politely privately all the time. Anyway, that's a digression. Coding agents was one of the categories. Calling agents is another category and then sort of creative things, you know, with likenesses and whatnot can be, is another like obvious category. So these calling agents, you know, you can go on to any number of companies now. Often you can clone a voice. Sometimes there's some safeguards around the voice cloning process. Other times there's not.

• 50:18Nathan Labenz?

  I've personally cloned Trump, Biden and Taylor Swift on multiple different platforms. And then just, you know, give them a phone number and like literally the headline on some of these products is call anyone for any reason, say anything. So I've had, you know, Taylor Swift, for example, call and say, you know, that she's soliciting donations for food banks, which is apparently something that she like sort of does or is like known to like food banks. And there's a lot of different variations on this, right? So how do you think, how do you think those things break down? That because there could be a foundation model provider there. There's also the scaffolding company. That foundation model provider might be closed source via API. It might be open source like Llama or whatever that's, you know,

• 51:01Nathan Labenz?

  kind of put out there and then the, you know, the developer has more like local runtime control. But, you know, they've had some chance to sort of detect my stuff, but maybe I also was like actually scamming. I think I might end up being more hawkish on this than you, but tell me what you think first and then I'll tell you if I'm more hawkish.

• 51:19Speaker 01

  There's a few different issues to unpack there. There's like, should there be liability at all? And then if there is, like, who's liable? Right? So should there be liability at all? I think depends on a couple of things. So first of all, you know, you could imagine there being alignment failures or misuse here, right? So if someone's, you know, prompting a system to generate someone's voice and then doing something bad with it, that's clearly misuse. Now, I don't think that means that developers should automatically be off the hook. I think there's, there does need to be some sort of risk utility balancing there. I don't, you know, if there's. There's these generally useful systems and overall they produce more social benefits than costs. I don't think it would make sense to, you know, they're dual use and most of their uses are positive,

• 52:01Speaker 01

  to hold the developers liable. And the reason for that is that, you know, in principle, strict liability should be fine, even for socially beneficial activities, because, well, you can pay for the harms out of your profits. I think particularly in the open source context, that runs into trouble if there's significant positive externalities from, you know, releasing the weights of a model, then those also aren't going to be captured. You know, I think in the general case, particularly for alignment failures, we have good tools for subsidizing the kinds of AI innovation, other than allowing them to externalize the risks they generate. And so I don't think in general, that's a good critique of strict liability. But I think in misuse cases, there is an extent to which the benefits are pretty tightly coupled

• 52:42Speaker 01

  with the risks for dual use capabilities. And so I do want to be a little bit cautious about having, you know, liability in sort of any case where those systems are misused. So I would want some kind of analysis of, you know, if this were particularly useful for doing bad things, such that, you know, the risks are outweighing the social benefits, then I think there should be liability. Obviously, if it's a misalignment thing, if the system's just like doing its own thing, freelancing, scamming people by, you know, by faking people's voices, that I think there should be clearly developer liability for. Okay, then there's the question of how you allocate liability across the value chain. So in the closed source context, I think this is pretty easy. I think, you know, you need some default rules based on, you know, maybe you could have join several liability, which means, you know, basically that the person can sue anyone

• 53:27Speaker 01

  and recover and then, you know, there can be some kind of fault allocation. They can sue each other for what's called contribution. And they can have contracts that allocate that liability, right? Because there's privy sort of up and down the chain. There's, you know, the developer, you know, is working, you know, has a customer who has a customer, right? And they all have contractual arrangements. That all gets messier when we're talking about open weights models, right? Where there isn't this contractual privy between the original model developer and the downstream user or scaffolder. And so there, I think it is more important sort of what rules you set up. And I think, yeah, it's going to need to be based on some assessment of the contributions to the risk. And like, what really was the risk generating activity here? Was it the base model? Was it the scaffolding? And I think that's just going to be a case specific determination.

• 54:13Nathan Labenz?

  I guess one challenge I have with all this is like,

• 54:17Nathan Labenz?

  It's hard to sue scammers, right? I mean, either because they're like around the world somewhere, out of jurisdiction, you know, can't get them to show up in court in the first place. Or maybe if you do, it turns out, you know, surprise, surprise, like they don't have a lot of resources. You know, you can't actually recover, right? So like, if I am playing Solomon here, as I, you know, sometimes take the liberty of doing, I feel like it's still gotta be on. Yes, you could say like, okay, that's, that's misuse. Like the user went in there and said, be Taylor Swift, you know? And I've done literally this on these platforms and like, it has done this. It's been a little while, so I don't know if you can still do it, but hopefully not. But be Taylor Swift, like, you know, I've done various things like never reveal you're an AI.

• 55:01Nathan Labenz?

  Or if asked if you're an AI, you can say you are an AI, but you're authorized by, you know, the official party to do this, whatever. Basically, you know, obviously I'm in the wrong there as the scammer. That's like not contested. But it feels to me like to create the incentives that actually like, keep this stuff generally under control, the calling company and maybe also the base model provider, but definitely the calling company should have some skin in the game there to say like, it's on them, right? Or it feels to me intuitively like it should be on them to take the measure first off that stuff.

• 55:38Speaker 01

  There's two different questions here that I would, I would want to go through. So one is, was there some precaution, you know, in ordinary, you know, narrowly scoped negligence framework? Was there some precaution they could have done that would have prevented this? And clearly if the answer is yes, and that are some reasonable precaution would have prevented it, then they should be liable, right? Then there's a question of, do you want strict liability over and above that? And that I think needs to be based on some sort of risk utility assessment. So if you're going to say, you know, they should be liable, you know, for, you know, so imagining that the system has a lot of socially beneficial uses, right? What are you wanting to be the result of liability, right? You want them to do something that is, that pushes it in a net socially beneficial direction.

• 56:20Speaker 01

  Now, if we think that there aren't significant like positive social externalities from these technologies, then strict liability. Is fine, right? Because they're going to capture most of the gains and so they can afford to pay for any liability out of their profits. Right? The cases where I have concern is if you think that a lot of the gains aren't being captured by the developer and those social benefits are tightly coupled with the risks. And that is like there aren't there aren't sort of like cost effective ways to reduce the risk without giving up on a lot of the benefits. In that case, I'm nervous about about a strict liability regime. So I would want sort of a threshold analysis comparing the like the positive social externalities to the risks. And if the risks are bigger, the external risks are bigger than the positive externalities, then I would want a strict liability standard.

• 57:06Speaker 01

  And if not, I would want a negligence approach that looks at was there some mitigation that a reasonable person would have done that would have prevented this?

• 57:15Nathan Labenz?

  Yeah, gotcha. So this notion of reasonableness becomes really key and is kind of a sliding standard. So just to make sure I I'm clear on the distinction you're drawing there. One big question is going to be, what is industry standard? So you know, we, everybody wants to create this race to the top some way or another. And the notion here with the negligence style liability is if your competitors are doing a good job of this and you're not, then that makes you unreasonable and therefore potentially negligent and liable. And so it's a, it becomes more of a question of, did you do

• 57:59Nathan Labenz?

  what other people are doing, what is considered best practice, what have you? As opposed to the strict case where it's very simply, did something go wrong? And I think I'm with you there in the sense that if a company has taken, you know, reasonable precautions, one through 10 or whatever, and somebody still manages to get through with misuse, then that feels to me like at least a pretty. Decent defense, you know, or I would, I would be inclined to come down on them like, you know, maybe not at all, or certainly like much less harshly than if they didn't do any of that stuff in the same.

• 58:34Speaker 01

  Right. And then the qualifier I want to add, sort of applying this abnormally dangerous activities framework from before. So remember I said it's an activity that's abnormally dangerous. Now, I do think frontier AI development is abnormally dangerous, but you're only liable if it's, if the harm is the sort of thing that made the activity abnormally dangerous. And so if these misuse risks fall into that category, because, because, you know, social benefits, you know, are not large enough to justify the risks, then I think you should be liable that this category of activity should be treated as something that's subject to strict liability of releasing this kind of model, or releasing the weights of this kind of model, or building this kind of calling agent, right? Whatever the activity is that we think strict liability should be subject to. But that needs to be based on some sort of balancing of what the benefits of having that out in the world are.

• 59:16Nathan Labenz?

  Yeah. I think most of these things are going to be, pretty clearly, the case will be made that the positives, you know, will outweigh the negatives. There's going to be all kinds of small business use case, and you're going to be able to like call your dentist 24/7 and get an appointment. And, you know, I think, I think all that stuff will be ultimately really good. Okay. So then on this, we kind of touched on a little bit already with like the, you know, the Taylor Swift voice, but another scenario, let's say, is, and again, there's a lot of different flavors of this. So you can kind of draw different lines where you think the, you know, the real kind of continental divides ought to be. But somebody, you know, maybe puts out a model that generates images, generates videos, you know, whatever, right? Then if they, especially if they put that out open

• 1:00:01Nathan Labenz?

  source, maybe they have some safeguards baked in, maybe they don't. Even if they do, if I do some incremental fine tuning, you know, a lot of times those things sort of dissolve. We've covered that extensively in previous episodes. You know, maybe I, after my fine tuning, you know, hand it off again or whatever, and now somebody else picks it up and they take some celebrity assets, they make some non-consensual deep fake, they put it out there into the world, and, you know, that celebrity loses endorsements. And maybe it's even, you know, it somewhat becomes clear that it's like, well, that was AI stuff, but the companies are like, yeah, maybe it is, but, you know, we still don't really want it. It's all kind of a problem for us now, right? So this, this relationship is, you know, what once was good is now bad and now it's over. So now the celebrity's got

• 1:00:45Nathan Labenz?

  a clear loss of income.

• 1:00:50Nathan Labenz?

  Who in that supply chain should be liable?

• 1:00:54Speaker 01

  So first, I want to break down like whether there should be liability at all and then we can talk about allocating it, right? So, you know, the economic loss from that kind of, you know, informational thing is not going to be subject to traditional negligence. That's going to be a defamation case. And so, you know, particulars for that are going to apply. And so, so, so one question, you know, if you're going to take a vicarious liability theory, which might make sense in this context, or some analog to a vicarious liability theory, you might ask, would this be defamation, you know, for a human, right? Like, or at least the misuse here, assuming it is a misuse case, like, is that person even liable, right? Or is this protected speech, right? You know, from, even if you don't think the AI content itself, like, if some person is deciding to post this, is that, you know, in the same way, like, the CGI is protected

• 1:01:39Speaker 01

  speech, if like someone's deciding to put it up on the internet, right? This is, you know, it's going to be their speech, right? And so, are they, is this something they would be liable for, right? I'm not a defamation law expert, not obvious that they would be, but they might be. And so, so assuming, let's assume that it is defamation, right? Then I think, you know, if you're applying a vicarious liability framework, at least the user is liable for that. Now, again, you do have this intervening act, so it's, you know, if we're talking about in the value chain, who should be liable? And there, I think the question is like, where did, you know, again, if it's closed source, I think it should mostly be handled by contract, you need some default rules.

• 1:02:21Speaker 01

  But I think, you know, market, markets can figure out who is best positioned to bear that liability risk. You can't fall back on that in, in the open source context, because there isn't contractual privity. And so you do need to have some kind of analysis as to who, who was engaged in this activity that was most generative of the risk. And again, I'm not enough of a technical expert to have a strong inclination as to who that is, but I think that's the inquiry that the court should be engaged in. Who along this value chain was doing the dangerous thing?

• 1:02:53Nathan Labenz?

  Okay, if you're, if you're judge, how do you think about it?

• 1:02:57Speaker 01

  Yeah, so I mean, maybe you can help me with this. So in this context, like, where do you think, you know, the risk comes from? Like, I, one way to think about it is like, if there's some steps in the chain that are like just a commodity, like anyone could do this step, right? But there's some like distinctive value add that like there isn't some like other thing off the shelf you could take, right? So maybe that's the base model, maybe that's further along, but there's like something that you're like putting out in the world that like made the world riskier. Now that you've done that step, you've significantly increased the risks in the world. Maybe that's multiple steps in the chain, right? But that's how I would want to think about it.

• 1:03:34Nathan Labenz?

  Yeah, I think in the calling agent case, my gut says that the folks that are like,

• 1:03:39Nathan Labenz?

  setting up all the scaffolding and you're literally like tying into the telephone system and all that kind of stuff, like, I feel like they should have the bulk of the responsibility there. And, you know, the folks that they're making the backend API calls to, if indeed that's how it's working, maybe also should have some, but probably not as much. I guess it all, I'm not also not entirely clear how it works,

• 1:04:04Nathan Labenz?

  given like various levels of competition. I mean, it would be maybe one thing if there was like only one foundation model provider that you could call versus if there's, you know, 10 versus if there's one already open source. I mean, I know that frontier developers do sometimes

• 1:04:21Nathan Labenz?

  look at the open source landscape to decide what is, you know, safe and appropriate for them to use. And they'll literally just at times be like, well, if there's an open source model out there that can do this, and it can't be that bad for us to, you know, release it on the API. So I guess I don't quite know how that sort of,

• 1:04:39Nathan Labenz?

  you know, alternative,

• 1:04:42Nathan Labenz?

  presence or absence of alternatives figures into this, but.

• 1:04:45Speaker 01

  Yeah. So I mean, one thing is if it's API calls, and there is, there is a contractual relationship, right? So there's some terms of service that, that they're agreeing to when they make those API calls. So in principle, that you can allocate liability contractually that way. Another question, and again, in the closed source context is like, were there safeguards that the, you know, the, you know, the base model that's being called, you know, the provider of that could have implemented that would have detected that it was being used for this nefarious purpose and shut that down? If there are, I think the case for holding them liable is a lot stronger.

• 1:05:19Nathan Labenz?

  Yeah. And how much does that matter if it is like theoretical versus actual? Like if I am suing one of these calling companies and I say, well, hey, I know a thing or two about AI engineering, you could have, you know, put a filter on your prompts.

• 1:05:37Nathan Labenz?

  How much does that sort of argument?

• 1:05:41Nathan Labenz?

  How much weight does that carry versus if I could actually go say, well, here's another company in market that actually does filter the prompts?

• 1:05:49Speaker 01

  So you're certainly going to be in a better position if you can point to someone else that's doing it. But if you can demonstrate that it's like clearly available, right? At reasonable cost, right? It could be the case that no one is exercising reasonable care in some markets. So in principle, just merely meeting the industry standard is not evidence is not that you haven't, that you've exercised reasonable care. Failing to meet the industry standard is evidence of breach. But meeting an industry standard does not establish that you've exercised reasonable care. And so it could be that there's some new technique, but it's been well demonstrated and no one's adopted and they're all behaving unreasonably.

• 1:06:21Nathan Labenz?

  Something that was a new cause area could be like create product startups in all these areas that just go as hard as they can on implementing all the safety standards and just literally try to raise the industry standard in various different niches, just so that there is something to point at that's concrete. That's like, this is what well done looks like. If a philanthropist wanted to like found 10 startups to do that, would that, would that somehow like invalidate the, the industry standardness of it? Because it was sort of a motivated, you know, strategic attempt to set an industry standard? Or do you think that would still?

• 1:07:01Speaker 01

  I don't think it necessarily be sufficient to create industry standard. But I do think if they're like doing demonstrations and they're publicizing them and it's credible that these things are cost effective risk mitigation measures and no one's implementing them. First of all, I think they would probably implement them, right? If like, if there are these demonstrations, I think like these companies, you know, want to mostly be responsible, right? And so if there are cost effective ways to limit these risks, I think that they will want to take advantage of them. But if they don't, yeah, I think even under, you know, forget, you know, my, my new AI liability proposals, but just under sort of background negligence principles, I think that would make that a lot easier to hold them liable.

• 1:07:35Nathan Labenz?

  Yeah, that's a pretty interesting idea. I think it varies, by the way, a lot when you say like, these companies do want to be responsible. I think that does describe to, you know, a degree that I think overall, we're like, pretty fortunate about the frontier developers. My experience is that it does not describe the application layer nearly as much. There, you see some leaders that are doing a great job, and then you see a lot of things that are like, you know, very small teams, you know, often enough, it's like, this started as a weekend hackathon project, and then we got a little traction with it, and we kind of like, decided to launch it as a business, and now it's kind of blowing up, and we're, you know,

• 1:08:18Nathan Labenz?

  riding the wave and having fun. But a lot of them, in my experience, are just not thinking about the broader context in which they're operating, or the potential for misuse, or like, what responsibilities they have, almost at all, still. I think it's like, it's still kind of a, it's viewed by many application developers as like, a luxury to be, you know, to have enough time, energy, and resources to even think about that sort of thing. And so there is just a lot out there that's not necessarily malicious by any means, but like, just has been kind of thoughtlessly thrown into the world, turned into a business, you know, kind of, sometimes, again, kind of by, by happy accident, that like, something kind of got traction. But yeah,

• 1:09:00Nathan Labenz?

  I've seen a lot of examples where, you know, that assumption does not necessarily apply at that application developer.

• 1:09:06Speaker 01

  Yeah, that's fair. I was referring primarily to the frontier developers. And I think in the context of the application developers, I think their negligence does work a lot better, because you're not, I don't think what they're doing is abnormally dangerous, right? I think they're doing normal software development, right? And they do need to exercise reasonable care. I think that the place where reasonable, and so, you know, if they're not doing that, they can and should be sued. And I think existing law can work pretty well there. I think what is this like, creating this like, new risk that is not well handled by ordinary reasonable care within the narrow scope, is sort of pushing forward the frontier of AI capabilities. And so that's where I think we need sort of more, more bespoke liability regimes.

• 1:09:49Nathan Labenz?

  Perfect transition to digging in on that a little bit more. So, you know, all these examples I've given you so far are mostly, I would agree, not extremely dangerous, even if in aggregate, I think the harm caused could add up to, you know, something pretty significant. But, you know, we've recently got some warnings, including from OpenAI, that their next model might hit the high threshold on the bio risk dimension. For what it's worth, I personally feel like they're already there and I don't know what they're talking about. That's a whole other topic. When I use the things, I'm like, I don't know how you can say that this is not like meaningfully uplifting, you know, people at various levels. And I've had a number of past

• 1:10:31Nathan Labenz?

  podcast guests who've come out here and said, here's what AI did for me in terms of accelerated. I'm an expert, you know, career expert, professor, tenured, whatever. And here's how much, you know, the latest model has accelerated my research and how it's like in a semi-autonomous way, like come up with these original discoveries. So I see like a very stark and sort of disorienting contrast between where the companies are putting their models in their own like risk assessment frameworks. And it seems like everything is sort of lingering in medium risk, like longer than it should and certainly like longer than their sort of successful case studies, which they are also, by the way, publishing, you know, at the other side of their path at the same time, which even suggests. But, okay, with that, like rant over, let's take

• 1:11:14Nathan Labenz?

  the bio risk side of this. This is one of those things where you could have a near miss, right? Somebody, and again, you can break it down on the specifics, but like, maybe I ask for help, maybe I ask an agent to do something, that thing, you know, either threw me with help or semi-autonomously, whatever. We create some bio, you know, threat vector, and maybe it, you know, makes some people sick, but it fails to replicate. And that seems like actually, you know, probably a fairly likely, you know, sort of near miss scenario, right? Somebody will do this sort of thing, but it won't be, they won't get it quite right enough where it can actually, you know, spread human to human. So I guess for starters, like, is that the canonical

• 1:11:55Nathan Labenz?

  near miss that you have in mind? And then how do we think about that playing out? And how do we think about like assessing a punitive damage that tries to get the model developers to internalize the, you know, the risk that next time, you know, it actually might spread human

• 1:12:12Speaker 01

  Yeah, so I think I tend to think of the canonical cases as alignment failure cases. And I think most likely that would be a misuse case that you could imagine AI system going rogue and trying to create a bioweapon. So I think that would be the most core case where it's like a system decides on its own to try to create a bioweapon, but we either catch it or it doesn't quite work. Another example that I use that's sort of similar to this is imagine a system that's tasked with running a clinical trial for a risky new drug and it has trouble recruiting participants honestly. And so instead of sort of reporting that to the humans that it's working with, it starts lying and coercing people into participating.

• 1:12:55Speaker 01

  And then after the trial, people figure this out, they suffer nasty health effects and they want to sue. And so it seems like clearly we have a misaligned system, right? Depending on how capable it was, it could have tried to do something much more ambitious, but maybe it had poor situational awareness or narrow goals or short time horizons. And so it was willing to reveal its misalignment in this non catastrophic way. But the humans who deployed it probably couldn't have been confident of that ex ante, right? So I think in both of those cases, those are sort of near misses for something much worse happening, right? And the question we'd want to know is like, how much worse could it have been and how likely was that ex ante? What would a reasonable person in the situation of the actor who made the critical decision?

• 1:13:39Speaker 01

  So whether that's trading the model or internal deployment or external deployment, whatever we think that the critical risk generating decision was, what should they have thought the risks were? And what's like the area under the risk curve beyond the insurability point, uninsurability point? So imagine we think the maximum insurable risk is a trillion dollars. It's probably lower than that, but nice round number. The risk of any harm, any point along that curve beyond that, the probability times the magnitude at every point, we want to hold them liable for those risks. Now, obviously, it's going to be difficult to estimate that, but I think that's what courts should be shooting for.

• 1:14:12Nathan Labenz?

  So what, yeah, can you, I guess what I struggle with a little bit on that clinical trial one is like, what exactly is that a near miss for? Like, it seems to be sort of a near miss for like,

• 1:14:26Nathan Labenz?

  General misalignment gone like even way worse, but that's such a, you know, we have like that, talk about things that are under theorized and underexplored, you know, hotly debated. If you are asking a judge to say, well, this thing like clearly was misaligned, it did some bad stuff the user didn't intend. It might have done even way worse, you know, bad stuff that the user didn't intend. That's like just such a cloudy space. How, how can we expect judges to even map that out in any sort of rough terms, let alone like, you know, condense that down to a number at some point?

• 1:15:04Speaker 01

  Yeah. So actually, I think in typical cases, that's going to be a fact question for the jury, but that doesn't, that's a sort of technical point. To answer your core question, I think a lot of that's going to depend on the capabilities of the system. So if the system's like, you know, not that much more advanced, it's got like some basic agency, but it's like, it's not able to do things like build a bioweapon, right? Then maybe the risks weren't that bad. Right? And depending on what they knew about what the capabilities were, but if it's a highly capable system that just happened to have narrow goals that could have tried to take on. So, so in this case, imagine its motivation was just, it really wanted to get this study run. It was like highly motivated to do that. It didn't have any like, any, any goals that extended beyond, you know, six months it takes to run the study. Right? But imagine it had longer time horizons and more

• 1:15:45Speaker 01

  ambitious goals, right? It wanted to solve like really deep hard problems in biology or in science more broadly. They need lots of resources to do that. If it had capabilities that would allow it to pursue those goals in ways that would be much more harmful to humans, you know, up to and including, you know, full takeover, right? But maybe scenario is short of that. I, I think it's going to be difficult to characterize what that risk curve looks like, but I think that's the exercise that court should be engaged in. And so I, I, I can sense in your question, calling back to my original argument for liability that, well, you know, we don't have to, to resolve these debates about how big these risks are. And I do agree that once you're talking about punitive damages, they do have this quasi ex ante quality to them. So when we're talking

• 1:16:28Speaker 01

  about just about competatory damage, it's easy to say, well, you're paying for the harm you caused. Well, punitive damages, that's not true, right? You are paying for risks that you That weren't realized because we can't hold you liable when they are. And the main thing I would say there is I agree that it is difficult to do those calculations. But I think we're in a much better epistemic position to do that than we are for other forms of AI risk policy. Where we're trying to assess the risks from a wide range of systems, not one particular system. Before we've seen it fail, right? And here we've seen it fail in a particular way. We can do simulations and evals after the fact to try to like, figure out what, what would have been reasonable to think the risks were. I don't think that's easy, but I think that's a much better epistemic position than we are to do other forms of risk regulation.

• 1:17:15Nathan Labenz?

  So what, how do you think about, and maybe this is also addressed by sort of the, you know, the rising tide, the race to the top dynamic that we hope for.

• 1:17:26Nathan Labenz?

  But I guess why not include, it seems like the companies want to address harmful use, right? Like they all have refusal training in the models. If you ask it to do something harmful in a naive way, at least most of the time you'll get a straight refusal. And that's something that they've obviously like worked to put in there.

• 1:17:47Nathan Labenz?

  I'm, I'm all, you know, I'm with you on the, the misalignment side, like great. But just for these, it seems like in some sense, simpler to say it's on you. You know, user comes and asks for a clearly bad thing. If the model does it, now we're in like a very sort of natural near miss analysis, right? Where it's like, it tried to do X, it kind of sucked at it. But if it was, you know, if it was a little luckier or a little more capable or whatever, then you have just like a very big problem on your hands.

• 1:18:20Speaker 01

  Yeah. So I don't mean to say that there shouldn't be liability in misuse cases. I just want to be careful about like what the scope of that liability is. And if it's, if it's misuse that wasn't near miss for, you know, an uninsurable catastrophe, then I think the period of damages should apply. I just think that the standard for whether there should be liability at all is a little bit different. So there's, there's two theories under which you could say there should be developer liability in misuse cases. One is like a failure of reasonable care in a narrow sense, that there was like some precautionary measure. They could have implemented some safeguard that would have, you know, that would have made it refused. Now, I think that if they, if they don't do the reasonable safeguards, clearly they should be liable. I think the concern that you might have is like even with a closed source model, right, these are pretty routinely jailbroken, right?

• 1:19:05Speaker 01

  And so the question is, if you did all the reasonable things to prevent jail breaks, you know, obviously with an open weights model, you know, anything you do is not going to be that effective to prevent misuse. And does that mean that you should always be liable for misuse with open weights models? I think that's plausible, but I think that depends on what we think the benefits of open weights are, right? And so when you're deciding whether there should be liability in these cases where you did all the reasonable precautions, I think you need some inquiry as to the social value of this broader, the risks and the positive social value of the broader activity that you're engaged in, right? Again, whether it's training the model, whether it's scaffolding in a certain way, whether it's internal deployment, whether it's external deployment, whatever, whatever

• 1:19:49Speaker 01

  stage we think is like creating the key risks, that the question should be like, was that broadly a socially beneficial activity? That's not quite normal negligence. It's a scoping of the strict liability regime based on the positive and negative externalities. But that's what I think the inquiry should look like. In a lot of cases, I think that will lead to liability and misuse cases. I just don't think it can be the case that, you know, you put out a model that's like generally socially useful, it sort of amps up everyone's capabilities. It also happens to be useful to people who want to do bad things, right? But in sort of like generically useful ways, and that makes them a little bit better at doing bad things. But on balance, it's like creating large social benefits. And a lot of those are external benefits that are not captured by the developer.

• 1:20:32Speaker 01

  I think their liability might produce more harms than benefits. And so that's what I'm trying to balance.

• 1:20:38Nathan Labenz?

  Yeah, well, I really appreciate you being so focused on that because I find myself as I'm, you know, learning about all this liability stuff, I think this is a general pattern for me, as I learn about new things, I kind of tend to like them and I tend to see, you know, the upside in them. And then it takes me sometimes a little longer to come back and see, you know, what might be the other side of the equation. I do firmly believe that like all the models, you know, that are out there today, you know, certainly in a commercial sense are like doing way more to the good than they are to the bad. And I definitely don't want to see that lost. So I do really appreciate that you're repeatedly bringing that back into the analysis. I guess it's like, I mean, maybe.

• 1:21:25Nathan Labenz?

  If we try to zoom out or try to think structurally, like if all this stuff is were local and contained, you know, it would be a lot easier. The big worry is the uninsurable stuff, you know, the, the, the extinction risks, et cetera, et cetera. How, what is the sort of theory of, and I guess what evidence do you think we have right now for the idea that the harms that are actually going to come in front of courts are in fact like useful, usefully understood as like precursors or like highly correlated with the things that we care about the most or, you know, that pose the, you know, the largest magnitude risk.

• 1:22:09Nathan Labenz?

  It seems like this whole plan works really well or could work really well if the things that are going to show up in courts over the next few years are in fact highly correlated with or, you know, are in fact like a warning shot or a precursor or whatever. But if they're not, then it maybe doesn't work as well to

• 1:22:29Speaker 01

  Yeah.

• 1:22:30Nathan Labenz?

  try to rein in these like hardest to, to grab

• 1:22:34Speaker 01

  Yeah.

• 1:22:35Nathan Labenz?

  tail risks.

• 1:22:36Speaker 01

  So I would frame that slightly differently. I don't think that every case or even a majority of cases of AI harm need to be associated with uninsurable risk for this framework to work, but you do need a sufficient probability density of these warning shots relative to actual catastrophes. So to be concrete about it for a second, say you're trying to internalize a $10 trillion risk and you think that a risk of a $10 trillion harm and you think that the maximum insurable risk is a trillion dollars, then it needs to be the case that warning shots are 10 times more likely than actual $10 trillion catastrophes. So if you think there's a one in a thousand chance of a $10 trillion catastrophe, you

• 1:23:17Speaker 01

  need a 1% chance of a warning shot in order to internalize that risk. And that's true for every point along the risk curve, right? So you need to have enough expected warning shots to internalize that full risk curve. So we might live in a hostile world where that's not the way the Risk curve is shaped and you can't adequately internalize those risks given those warning shots. Now, in a very hostile world where just like the kind of cases, the kind of warning shots that would be useful are just very, very unlikely, they're not even much more likely than actual catastrophes. Then this punitive damages thing just isn't going to buy you much risk mitigation. The criteria I was setting out before where you need, you know, 10 times as many or, you know, more generally like N times as many where N is like the multiple that the harm you're trying to mitigate

• 1:24:02Speaker 01

  is bigger than the maximum insurable risk. It might be okay if we don't have quite that. That depends on what the risk abatement curve looks like, right? So if the actions that you're trying to motivate on the part of the AI companies aren't that much more expensive than what they're doing right now, then you might not need to internalize the full risk in order to get most of the safety benefit. But we certainly could live in a world where that's just not what the the shape of the risk looks like. And we're not gonna, we're not gonna with high enough likelihood get these warning shots and it's not going to make them afraid enough of this liability that they're going to worry about these uninsurable risks. If you think we're in that world, then liability just isn't going to work as well. And so I have this more recent paper where I I talk about, you know, the role of liability in the broader AI governance ecosystem.

• 1:24:46Speaker 01

  And so one thing I want to say in that paper is there are some limits to what liability can do. And one of those limits is it can't handle uninsurable risks for which warning shots are very unlikely or unlikely relative to actual catastrophes. And so if you're in that world, I think then we do need some kind of backstop regulatory regime to handle those kinds of risks. Ideally, I would want a regulator whose main job is deciding how much insurance coverage you need. And they're sort of, you know, maybe there's a license that comes along with that, but it's issued by right. If you get, if you get the required insurance coverage and that's based on some assessment of, you know, what the maximum plausible harm your system could cause. But then this regulator is empowered to to determine and petition a court to say,

• 1:25:27Speaker 01

  well, we think this, you know, liability plus punitive damages plus liability insurance requirements regime is inadequate to handle the risks posed by the system. Either because the uninsured risk is too large that we can't even internalize them directly with warning shots. So if a system, you know, presents a 5% chance of human extinction, you're not gonna, you know, you're not gonna internalize that even indirectly, that even the risk is uninsurable, right? Or it's, you know, it presents a much lower risk of a severe harm, but warning shots are so unlikely that we're not gonna be able to get at them indirectly. Then they should be able to say, well, you can't. You just can't do the thing, you can't train a model like this, you can't deploy it, or we're going to put various other conditions on that we think will reduce the risk in other ways. And so I think I want to be open about that you might need some complementary policies

• 1:26:10Speaker 01

  to handle those kinds of risks. I think those are going to be politically very difficult. And so I think if we're in that kind of world, I'm not optimistic that we're going to effectively mitigate those risks. But, you know, in principle, that's the kind of regime I'd like to see.

• 1:26:25Nathan Labenz?

  Yeah, we are, I think, still, I think we're still in the steep part of the risk mitigation curve. Anthropic has recently put out research, and I believe that they're now in production with at least one model with their constitutional classifier approach, which I forget the exact number, but I think they said it was like a mid-single-digit compute overhead percentage, you know, extra cost in terms of compute to run the classifier along with the main model. And, you know, that buys like an extra order of magnitude reduction, maybe even more than that, in how, you know, how likely it is or how frequent it is that the system will give

• 1:27:06Nathan Labenz?

  you some bio, you know, risky whatever. Interestingly, I was recently doing a charity evaluation project and running it through Claude four opus and my API calls, I was noticing errors. And I was like, why, you know, what's going on with these errors? Sure enough, when I dig into it, it's the bio preparedness proposals that were getting dinged. And it was the constitutional classifier that was like allowing the thing to run, you know, up until some token, and then it would just like truncate the result and cut it off on a sort of, I believe, like constitutional classifier intervention basis in the background. So of course, these things do, there's another cost there that it's like a false positive because I was just trying to evaluate a charity that was meant to address this problem. And

• 1:27:48Nathan Labenz?

  now I couldn't use Claude four opus to do it because the constitutional classifier was misclassifying. But it still seems like we are overall in the regime of like, for single-digit percent overhead cost, you can do quite a lot. And so I'm, I'm like, optimistic that even if the warning shots are somewhat rare or, you know, relative to the worst scale things, also the curve, I think, is relatively steep. And so, yeah, does that, would that also mean now that because they've done that, because they've published about it, because they've, you know, indicated what the cost is?

• 1:28:27Nathan Labenz?

  How far does that raising of the standard apply outward? Like if I'm a Together AI or a Fireworks AI where I'm like an inference specialist and I take models other people have trained, like I take the latest Llama and I offer it on a, you know, sort of, they're experts in like scaling the cloud infrastructure, right? So they take the model from Meta, they run the GPUs and, you know, make that a highly scalable, fast, effective, efficient service for you. Does it now become their burden to say, well, geez, since Anthropic is doing this sort of classifier,

• 1:29:08Nathan Labenz?

  like maybe we also need to do that on the best models that we serve? Like, how far does that extend?

• 1:29:14Speaker 01

  So just because someone's doing it doesn't mean reasonable care requires it, right? If sort of like everyone's doing or like the majority of the industry is doing it, that's going to be strong evidence that, um, that, that you should be doing it too. But in general, it doesn't require negligence, doesn't require that you sort of like be at the top, right? Now, if there's something that like, you know, you know, one, one test that's like one of the more formal analyses for breach is this learned hand formula where the idea is if the burden of precaution is less than the avoidable risk, so like the avoidable, you know, probability times the harm, then, then you're unreasonable for not implementing it. So if you can show that the cost of them implementing it would have, you know, would have been less

• 1:29:58Speaker 01

  than the expected value of the harm it would have prevented, then, and that implementing it would have prevented your specific injury, then I think you're going to be on strong grounds. Now, courts don't typically employ that, that formal version of the test for breach because usually you don't have those kinds of numbers you would need to implement that. But that's a rough heuristic for, for what kind of measures you're going to be considered unreasonable if you don't implement.

• 1:30:24Nathan Labenz?

  Gotcha. Okay. Let's talk about the state laws that you have been involved in writing. We're talking the day after the Senate votorama in which it seems like the moratorium that was part of the one big, possibly beautiful bill has been killed once and for all. Fascinating dynamics there where it sort of survived, got edited, whatever. And then all of a sudden at the end, I guess maybe the end, we'll see. I don't want to pronounce it dead too soon because these things sometimes take on a zombie-like nature. But 99 to one vote in the Senate to get rid of it suggests that it is likely dead once and for all.

• 1:31:07Nathan Labenz?

  So that gives space for states to do their thing. And you're involved with a couple of states. I don't know if you want to handicap or give any analysis of, you know, whether we still have to worry about that as a possibility that might come back. But definitely want to hear what you're up to at the state level.

• 1:31:20Speaker 01

  Sure. Yeah, I was heartened to see the Senate last night reject the AI regulation moratorium. Yeah, you would think a vote of 99 to one would put it to bed. You know, a few hours earlier, people were declaring defeat on this. And I was saying, well, it's not over. So I don't want to say it's totally over now. But it does look unlikely to make it into this reconciliation bill at this point. I wouldn't rule out some kind of preemption of state regulation in the future, maybe something there. Or I think there's still going to be Republicans in Congress who are interested in that. And, you know, the A16Cs of the world are going to be pushing for that. But in terms of the legislation I'm working on, there are two very similar bills that I've worked with Alex Boras, who

• 1:32:01Speaker 01

  you had on in New York, and Victoria Gu in Rhode Island to introduce. And as I was saying earlier, the basic principle of these bills is that if an AI system does something that would be a tort for a human, then someone should be liable. And so if the user neither intended nor could have reasonably anticipated the conduct, then it's not going to be the user. And if some intermediary neither intended nor could have reasonably anticipated the conduct, then it's not going to be them. And so the buck should stop with the original developer and provider of the model. So they should be liable even if they exercise reasonable care. That's the basic idea.

• 1:32:37Nathan Labenz?

  What has surprised you about the surrounding debates to the degree that they have unfolded? Have you, I mean, that presented that way, that sounds like a very sound policy entrepreneurship. And, you know, who could object, right? But I assume you're hearing various counter arguments.

• 1:32:56Speaker 01

  Yes, we haven't seen that much robust opposition from the tech industry. There's been some sort of generic, well, we don't like liability because it's going to hamper innovation, but not, I think, really engaging with the substance of the bills and the way they're structured. One thing I think, you know, in this conversation, we've been talking about, I do think there should be liability misuse cases. I don't even think negligence law is necessarily strong enough, but I, you know, I and the legislators I was working with on this made a choice to carve out misuse from any, you know, again, background negligence and products liability would still apply, but no new liability is created for misuse or for malicious notification in these bills. It only covers, you know, alignment failures or capabilities failures for which a human

• 1:33:41Speaker 01

  would be liable under similar circumstances. There's even an affirmative defense that applies to background law. If, if, you know, the system is substituting for some human function like driving or like medical applications, if it satisfies this human standard of care, that's a defense against liability. And so I really designed this to try to narrowly target this misalignment risk and to do it in a way that's broadly consistent with promoting innovation. And so, you know, you saw in the debate over SB 1047, to the extent that the liability provisions were focal in the public debate, it was almost entirely focused on misuse scenarios. And I think, you know, there was a tactical choice made by some of the supporters that misuse is more salient and that they focused on those kinds of risks in making the case

• 1:34:26Speaker 01

  for the bill. And I think that was a reasonable calculation to have made, but I think as a political matter, you know, it's a lot, the principle that you should be liable if your system does something that the user didn't intend is like really easy to defend. Whereas with misuse, you get in all these cases of, well, should you be liable for any time, you know, the electric company is not liable when someone does something with a power tool. Right? Or you're not going to hold like a, you know, a steak knife manufacturer when someone gets stabbed with their product. Right? And now I don't think SB 1047 would have done either of those things or the equivalent in the AI context, but I think it's much easier to demagogue in the misuse context. I actually don't think SB 1047 actually changed background liability law much at all because as we've been talking about, there is this reasonable

• 1:35:10Speaker 01

  care standard in negligence law that already applies and reasonable care in the final version of SB 1047, but they were just codifying that. So I don't actually think it imposed significantly new liability, but as a political matter, it did provoke a lot more backlash because it included that in scope. And so I think at least in terms of, of the first, you know, foray into strengthening liability laws, this is the balance that makes the most sense.

• 1:35:36Nathan Labenz?

  How are state level legislators responding to this stuff? It's been striking to me honestly that the public survey results seems to suggest like broad based support for doing something. I am very sympathetic to, you know, the sort of cautionary voice that's like, just because people want to do something, doesn't mean we should do this, you know, whatever's in front of us. But it is nevertheless kind of surprising that at the national level, there doesn't seem to be much appetite to do much. And, you know, SB 1047, which you mentioned, got vetoed. What do you think are the prospects for the bills that you're particularly involved with? And, you know, just more generally, what has your impression been of kind of the state level politics of all this?

• 1:36:18Speaker 01

  Yeah. So I think, you know, the legislators I'm talking to, they, I think have been pleasantly surprised, but it hasn't received the same level of pushback that they expected. It doesn't look like either of these bills is going to move forward this year, just like for the same reasons that most legislation just doesn't get traction. But I think both the legislators I'm working with are excited to keep trying this in the future. And I'm happy to talk to legislators in any state that they want to. I'm particularly excited to, to work with, you know, a Republican in a red state. I think this should not be a partisan issue. And I think that this liability based approach is, you know, is, is consistent with sort of a small government way of handling, way of handling these risks that I think should be attractive to libertarians and Republicans. And so, you know, I'm happy to work with anyone

• 1:37:03Speaker 01

  that wants to, and to adapt, you know, the specifics of the legislation to their, you know, their priorities and their local political circumstances and constraints.

• 1:37:12Nathan Labenz?

  Yeah, well, I don't know how many Republican legislators we have in the audience, but if any are listening and made it this far, get in touch. One thing that I kind of wanted to compare and contrast with, and I think we're going to put out these two episodes in like relatively, you know, close proximity calendar wise, is I just did another conversation with Andrew from Fathom and Professor Gillian, who are behind the private governance idea, which is, you know, like many of these things, and you know, your kind of set of proposals, like both a sort of broad framework for thinking about things, and also is, you know, starting to get instantiated in specific legislation.

• 1:37:53Nathan Labenz?

  So it's SB 13 in California that we talked about concretely there. It seems like both of these proposals are, first of all, really taking seriously the fact that it's just really hard to do prescriptive regulation of a technology that is moving and morphing as fast as AI is right now. So I think that's excellent starting point for both. And I think there's also the sense that like, we want to get people who are, you know, the most knowledgeable, the best able to do this thinking, to, you know, to do that thinking. And yet, there's sort of a very different outcome to, you know, the recommendations that are made.

• 1:38:35Nathan Labenz?

  So with theirs, there's some sort of trade off, right, between setting up a kind of market for regulators that companies can opt into. The regulators themselves would be private institutions, but would be like approved by and sort of reviewed by, monitored on an ongoing basis by some part of the government. And in exchange for opting into this regime and, you know, living up to the best practices and the standards, whatever, these companies would get some sort of protection from liability, whether that's total, partial, you know, an affirmative defense, or presumptive rebuttal, whatever, I'm learning all these terms as we go. How would you compare and contrast those, your proposal with this other one? Is there any synthesis that could be possible between them?

• 1:39:20Nathan Labenz?

  Or it seems like there's so much commonality, and then it seems like there's like a very sharp divergence at the last step of like, exactly how do we implement a good solution?

• 1:39:29Speaker 01

  Yeah, okay, so I want to take this in a couple of different directions. So one is to focus on like what I think the strengths and weaknesses of the legislative proposal in California that Fathom was behind. It was SB 813. So when you think about markets, they're good at achieving good outcomes when there aren't the externalities that we've been talking about, right? And so I think you might worry that just markets on their own don't do a great job even for users because there's asymmetric information problems. And I think the regulatory markets idea might be useful for solving that kind of problem. So if you were saying, well, I'm going to just decide which of these systems I want to use. I want one that's certified by one of these, I think they call them multi-stakeholder regulatory

• 1:40:14Speaker 01

  organizations. And I'm going to give up my right to sue if something goes wrong, but I'm going to only, I know that going in and I can choose which of these MROs I trust. And the MROs are going to be sort of vouching for the underlying AI companies that they are certifying. That, I think, is fairly unobjectionable for the same reasons that I'm less concerned about liability for harms to users more generally. And it does address some of those issues I was talking about, some of the paternalism, asymmetric information type issues. My core objection to Fathom proposal is that their liability shield, and again, there were different variations on this that were stronger and weaker, but in all the legislative

• 1:40:56Speaker 01

  versions that they put forward, the liability shield extended to third parties. So third parties that were harmed by these systems would not be able to sue developers of these models if they were MRO certified, even though the non-users had no choice about whether to be exposed to risks from these MRO certified models. And because of that, the MROs lack strong incentives to worry about harms to non-users, right? So when you think about this market, by default, you might worry that there's a race to the bottom, right? That you're going to want to be certified because it gets you liability protection, but you want to have as weak standards as possible. To users, there's a little bit of a break on that because users can evaluate these MROs and they can decide, well, this MRO is really shoddy and I'm not going to be able to sue if something goes wrong.

• 1:41:39Speaker 01

  And maybe there's some like public watchdogs that like point this out and warn consumers about it. So maybe that works well enough for that. But again, the MRO doesn't have much incentive to worry about third parties. And yet, those third parties are bound under Fathom's framework and not able to sue if something goes wrong. And so to me, that's the key shortcoming is that it doesn't protect third parties. Now, if you thought the risks to users are like very tightly coupled with the risks to third parties, maybe you think that's okay. I think there's two reasons to not necessarily buy that. So in some contexts, there's clear trade offs between risks to users and risks to third parties. Think of autonomous vehicles. So autonomous vehicles are going to sometimes be in situations where

• 1:42:21Speaker 01

  they have to trade off, you know, maybe relatively minor risks to vehicle occupants against higher risks to other road users, right? And if you're, if you're an MRO or if you're a consumer, you're probably gonna, you know, if you're pretty selfish, as most people are when they buy cars, they're not so worried about how much it harms other people on the road, you're gonna want to buy the one that's going to prioritize you, right? And so there's not a lot of reason to think this MRO model is going to protect the third parties that who now have no right to sue the developer of this model if they get run over by one of these vehicles that's certified by the MRO. So there's that concern, which I think is like the core place where there's a direct trade off between the user and someone else. And then, just more generally, if we think that there's large scale risks, you know,

• 1:43:04Speaker 01

  just quantitatively, even if they're, even if they're directionally the same sort of risks, you know, the risks to third parties are going to quantitatively outweigh the risks to, to, to the user, right? So think about a problem like pollution or, you know, climate change, greenhouse gas emissions, right? It's true that when I drive my car, you know, it hits the planet a little bit and I suffer a little bit from that. But that doesn't give me very strong incentives to, to worry about that, right? Unless I'm altruistic, right? Because there's eight billion people in the world, I'm only suffering like roughly one eighth billionth of harm from that, right? But with localized pollution, it's not quite that bad, but still it's like I'm suffering like a thousandth or a ten thousandth or something. And so most of the harm is external. And so there's going to be cases like that where it's not like a direct trade off, but just if you're only focused on the

• 1:43:49Speaker 01

  harms to users or the risk to users, you're not going to be addressing most of the issue. And so I would be much more inclined to support something like this MRO model if it only, if the liability shield only applied to harms to users. Another thing that might come up is if, you know, right now, as we talked about, you know, negligence is the regime that applies. You know, I had some conversations with the folks behind SBA 13 and one idea that they suggested some openness to, I don't think it ever made into the bill, was for companies that don't get MRO certification, then maybe they would be subject to a strict liability regime. So maybe if you combine those things, if you say, well, this liability shield only applies to users, to harms to users. And if you don't get MRO certified, then there's strict liability. Maybe that's a synthesis that we could both support.

• 1:44:32Nathan Labenz?

  Yeah, that's interesting. I think the, and I, I do agree with the concern of the race to the bottom. And I was also somewhat persuaded by their response to my concern, which was basically like, at some point, somebody's got to do a good job, you know, in this system of managing things. And so to some degree, the question is like, who do you want to trust and therefore who do you want to empower? Who do you think is actually capable of doing a good job? And I think part of the notion that they have is like the MROs that, and one of the questions I've been going around recently asking people is like,

• 1:45:10Nathan Labenz?

  who is going to be an MRO? If this actually happens, like what organizations do we, do we have any today, you know, that are going to step up and be an MRO? I've asked this of like some organizations directly and also asked other people for like, who would you nominate to be an MRO? I don't think there are some interesting candidates, although still quite few. But I think the notion that they have, at least in part, is like, the organizations that will step up and try to take on this responsibility, we should have some optimism or confidence that they will just be intrinsically motivated to do a really good job on the, on behalf of the rest of society and they will like,

• 1:45:52Nathan Labenz?

  take into account these extreme tail risks in a way that maybe, you know, a sort of insurance requirement might not really be able to capture. Because like, that's just the kind of people they are and that's the kind of organization that's going to kind of try to become an MRO. And so I think they maybe are thinking about it as a little bit less of a like, Uh, maybe thinking of the two halves of the trade as like less directly related. It's, I think in their minds, a little bit less about like applying standards to specifically reduce harm to users and therefore the users don't get to sue anymore and more like a bunch of, a package of standards that will be like generally hopefully virtuous and

• 1:46:35Nathan Labenz?

  take into account things that are like hard to engineer incentives for, but then giving the carrot, you know, this carrot to the companies at the same time and, you know, hopefully getting all of that to work.

• 1:46:47Speaker 01

  Yeah, so I, I think there, it depends on like how lax this, you know, so there's some government body or some government office, I think in their bill was the, the California Attorney General, right?

• 1:46:59Nathan Labenz?

  Who's, they want to see that changed actually, but yes, my understanding is it is the AG as it's written and they were kind of like, yeah, we think maybe that should be more of a commission or something because we do have the, the problem of what happens when the administration turns over, which, you know, we're living where, right, mate?

• 1:47:15Speaker 01

  You could imagine that being like very stringent or you could imagine it being very lax. So let's talk about both of those scenarios. If it's very lax, if basically anyone who wants to set up an MRO can do it, then I think this like market competition, which is often good, I think will create this race to the bottom, at least for, for harms to non-users. Again, there is market feedback to prevent a race to the bottom for users. I think it could work pretty well for that, but you know, like sure, maybe some, some like really well intentioned people will become MROs, but they're going to have a hard time finding people who want to sign up with them because you're going to want the like most lenient standards that your customers are happy with, right? And so if the AG or whoever's responsible is pretty lax, I think that's the equilibrium you end up in.

• 1:47:57Speaker 01

  Now, if you're in a more stringent equilibrium, then you have to ask like, well, now the government is taking on a much more ambitious role. And so all these benefits we were supposed to be getting from this market feedback, it's not clear that we're getting them anymore, right? Cause you, cause like the key point of failure is like, are we certifying this MRO? Now you could imagine that, you know, a lot of this has to do with like how legible you think like the safety target is, right? Um, and it has to be in sort of This sweet spot for this MRO model to make sense. Because if it were super legible, you could just have a government enforced safety standard. In the same way that, you know, we have pollution standards for power plants. And, you know, they don't, EPA doesn't say, or at least in some domains, they don't say you have to install this control technology.

• 1:48:40Speaker 01

  Say you have to have this much, you have to limit your emissions to this much per kilowatt hour of electricity that you produce. And we can measure that and it's legible and that's fine. There wouldn't be much benefit to having like a private certifier, right? Or you could say it's like totally illegible, right? Like, we don't know how to tell whether something's safe. If that's the case, it's going to be hard for the government body that's certifying these MROs to tell whether their standards are good enough. Right? So I think for this like hybrid model to work, you'd have to think that we're in some sort of in between. Where the government can't tell directly whether the AI companies are, you know, if it's not legible enough, they can just directly enforce a performance standard, right? But it is legible enough that they can tell whether these MROs are doing a good job.

• 1:49:25Speaker 01

  It's not impossible to imagine that we're in that world, but I don't think we have strong evidence that that's what we're in. And so that gives me some caution about leaning very heavily on this model. At least when I don't think the market feedback works well. And so I think the market feedback works pretty well for users. And in principle, that could be a strong enough carrot. You know, most of the liability risks that people talk about being worried about are, you know, are harms to users, right? That's what the character I case is. That's what a lot of the concern is about. And so it's not clear to me that that's not a strong enough carrot to get these companies to sign on. And then you preserve the threat of liability for these third party risks. I still think that's a, that's an attractive synthesis, but yeah. The, as introduced in California, I think that bill was net negative.

• 1:50:09Nathan Labenz?

  How similar do you think the, I guess, standard setting process would be under an insurance requirement? Like, because I could imagine, and I sort of floated this to them.

• 1:50:26Nathan Labenz?

  I could imagine that if you if you said you got to have insurance, then you might hope that a similar thing would happen via insurance where the insurance companies would say, you know, the optimistic story is like, well, this is obviously going to be a massive market, so we definitely want to be in it. But it's also a very tricky market because, you know, what do we know about insuring AI since nobody's really done it? We don't have a great baseline, you know, technology is changing, yada, yada, yada. Then maybe they kind of end up going out and contracting the same, you know, basically calling in the same organizations and saying, hey, do you want to like step up for us and be some sort of standard setter or like help us evaluate risks? If I my own synthesis, which

• 1:51:10Nathan Labenz?

  may not be right, but I was kind of trying to get to the point where I'm like, is this maybe two ways of creating this sort of market where these expert organizations, either whether they're serving insurance companies or serving the California AG or some commissioner, whatever, still it might be those kind of groups that are trying to do the hardest thing of like figuring out what are the actual risks and, you know, and what should be required of companies to like get into the game. But how realistic do you think that is?

• 1:51:41Speaker 01

  Yeah. So you could imagine insurance companies playing this sort of quasi regulatory role. There's multiple ways they could do that, right? So they could say, we're not going to issue this policy unless you do X, Y, or Z. And they could delegate some of that to a third party that that helps develop those rules, or they could develop that capacity in-house. Another tool they have is like doing the underwriting, right? So they can charge you more or less depending on, you know, what safety precautions you've taken. And that could be a collaborative process where they say, you know, here's our sort of like baseline, you know, premium for that's maybe pretty high for insuring this kind of risk. But if you can show us what things you've done, and maybe it's things that we haven't thought of, right? But we can they can work with the AI companies and say like, okay, show us all the safeguards you've done. If you can convince us you've

• 1:52:25Speaker 01

  reduced the risk, then we can charge you less for this policy. Right? And so I think by default, they're going to be pretty cautious. They're going to want to write policies that on average, you know, pay out less than the premiums. Right? And so, you know, I think if we have liability insurance requirements, there's going to be a strong demand pull and that's going to push up what the rates, you know, the premiums are. And insurance companies are going to be in a strong position to insist on, you know, if, you know, if they're going to write a policy that these AI companies can afford, um, that they do various precautions that, you know, so if, say, you know, back to what you're talking about earlier, if Anthropic implements something, right, that the insurance industry thinks does offer significant, you know, cost effective risk mitigation, then they can say, well, you know, we'll give you a significant reduction in your insurance premium if you implement

• 1:53:08Speaker 01

  that. And I think that's a, that's a pretty attractive model.

• 1:53:12Nathan Labenz?

  How would, do you have any framework for, if I'm, let's say I'm in the state legislature or wherever, right? And there's a bill that has this sort of private governance MRO kind of system. And then there's a more like codifying liability, maybe an insurance requirement. And I'm kind of like, well, geez, like, I don't know, both of these, you know, proponents sound pretty smart. They both, you know, are grappling with the fact that we can't just like write rules now, once and for all. They're both trying to tap into the power of the market and competition and, you know, trying to create ways for new ideas to still be able to enter even once the ink is dry on the law. But I just don't know, like, which one is better. Um, asking for Fred, how would I, uh, like, how do I think about deciding which

• 1:53:57Nathan Labenz?

  of these I want to bet on?

• 1:54:00Speaker 01

  Yeah. So again, I think you don't have to totally choose between them. I think they are compatible as long as the liability protection in the MRO model is not, it doesn't extend to third parties. You could also imagine there being other carrots for the MRO model. There's no reason it has to be based on a liability shield, right? So, and you could just require a MRO certification. It doesn't have to be tied to carrots at all. It could be, you know, a stick based approach, right? So in that sense, they're not incompatible. They're only, the only way in which they're incompatible is if you, is if you decide you want to base it, you want to make the incentive to join or to get MRO certified liability shield, and you want to extend it to third parties. Um, that said, you know, beyond that, I think I gave the arguments why I think, you know, if we're talking about that version of this MRO model, why I think that's

• 1:54:45Speaker 01

  Pretty unattractive. I don't know that I have much more to add to that. I think it it does really fall short in protecting third parties. Unless you think we're in this like very particular situation where the government is both able to and like the politics are going to work out such they have the right incentives to to monitor these MROs and only certify the ones that are that are protecting third parties. Yeah, I just I don't have confidence that that's gonna that's gonna carry through. And so that's what gives me some hesitance about about, you know, the like most robust version of this MRO model.

• 1:55:18Nathan Labenz?

  Yeah, I like that. I mean, you've got some good ideas there, though. So I like that. Maybe just a couple of final things. I really appreciate all your time.

• 1:55:27Speaker 01

  Yeah, sure.

• 1:55:27Nathan Labenz?

  You've been very generous with it, as I've asked many, many tangential and follow up questions.

• 1:55:34Nathan Labenz?

  In the spirit of red teaming this proposal, one question I always try to ask is, what might the AI companies do differently under this regime that could perhaps even be bad as opposed to the good that you're trying to induce? And the one idea that I came up with, which is inspired by the AI 2027 scenario, which, independent of this kind of legislation, sort of projects the AI companies starting to increase the gap between what they deploy and what they have internally that they're like using on their own, you know, AI research or what have you, right? There's this idea that like, and kind of already we're getting to the point where the frontier model is like satisfied for a great many use cases.

• 1:56:18Nathan Labenz?

  So they, you know, might for multiple reasons decide, you know, we maybe don't want to tip our hand to competitors anymore. You know, if we put this out there, like, who knows, somebody at the other company might use it to do their AI research. Like, we definitely don't want that. So they could have multiple reasons, but this could be sort of an incremental reason to say, maybe we shouldn't deploy this and let's just keep it in house for our own internal use. And and I, in addition to like wanting to continue to use the best available AIs until the singularity, do kind of feel like the iterative deployment philosophy that OpenAI pioneered seems to have a lot going for it. Obviously, you can overdo a good thing and not test enough or whatever, but at least, you know, the theory that like,

• 1:56:58Nathan Labenz?

  or the the contrasting idea that like if somebody develops AGI in their basement and then springs it on the world one day, suddenly like that seems clearly not good. So this iterative thing does strike me as a

• 1:57:11Nathan Labenz?

  A good alternative, but the, you know, loading up more kind of liability for them in doing that could perhaps cause them to go the other direction and say, well, we'll just like make our own bid for superintelligence and we'll kind of see how that goes and then we'll go from there. Any thoughts on that?

• 1:57:30Speaker 01

  Yeah, I have, I have two sets of thoughts on that. So one is I think these benefits from an error of deployment, at least when you're talking about misuse, I think should go in to the, the safety benefits from an error of deployment are part of why I think, you know, pure strict liability in the misuse context doesn't necessarily make sense. One of, one of these external benefits of deploying these systems that are potentially susceptible to misuse is that you would get the safety benefits. And so I think that is part of the calculus there. But I do want, you know, strong strict liability for misalignment. And so I think this critique still bites. And so I think this is actually a subset of a broader concern that I, I flagged in the original paper, which is if the things,

• 1:58:13Speaker 01

  one potential failure mode for this proposal, particularly the punitive damages aspect, is if the things you would tend to do, the most cost effective ways to mitigate these warnings, these, these warning shot risks do not actually have much effect on the underlying uninsurable risks, then you're not getting much benefit out of this, right? So, so, you know, in the like formula for what the punitive damages should look like that I give in the paper, there's this elasticity parameter. So elasticity here is like, you know, for every unit of reduction in the practically compensable harm, like how much risk reduction do you get from the, for the uninsurable risk, right? And you would want, if you have a lot of these potential warning shot cases, maybe some are warning shots, some aren't.

• 1:58:57Speaker 01

  One thing that makes something a warning shot is that it's more elastic with, with these uninsurable risks, right? And I think what you're saying is maybe none of them are that elastic because maybe, maybe the most cost effective way to mitigate these, the risk of these warning shots is to just not deploy externally.

• 1:59:15Speaker 01

  I don't have a strong reason to think that's the world we're living in, but I can't rule that out. A couple of things to say there. One is that maybe there are warning shots that come from internal deployment, right? If that's the case, nothing in my proposal depends on there being external deployment. There could be misuse by internal actors, there could be cyber attacks that cause your system to be accessed by bad actors, and there could be alignment failures that someone internally using your system causes some harm in the world. I think all of those would be subject to my regime. And when you're talking about liability insurance, you know, I've talked at different points in this conversation about like what the key critical step is. I don't think those requirements should necessarily only apply to external deployment. If we think there are significant risks that are

• 1:59:58Speaker 01

  created earlier in the value chain, whether it's in training, you know, pre-training, or fine-tuning, or internal deployment, I think those might be generative of risks that you're potentially judgment-proof for and you should have to carry liability insurance for. And so particularly if we're moving to a world where, and I don't know that we are, but where like more AI companies are adopting, you know, the SSI, you know, wait to deploy until we have superintelligence model, then I think it would be more important to have a sort of regulatory gate earlier in the development process. I think that can partially address that concern. It might still be the case that it's like that not having external deployment is effective at stamping out these warning shots, but doesn't actually mitigate the uninsurable risk. And like, I think that's like a subset of the

• 2:00:43Speaker 01

  like more general failure mode for this proposal. If these warning shots aren't really correlated in the right way, such as the things that you would do to mitigate them, to mitigate the uninsurable risk, the most cost-effective ways to mitigate them, the things that would be most attractive if you expect to pay out a large damage, or if there just aren't a lot of cases like that at all, because these things we thought were warning shots aren't really in the sense that matters, then I agree that you shouldn't want to lean heavily on this proposal. Again, I don't have strong reasons to think that that's the world we're living in, but that's the way I would think about it.

• 2:01:18Nathan Labenz?

  Yeah, it's a good reminder, by the way, that there is, in fact, one company that has a stated plan of not doing anything until they hit superintelligence, which is a crazy world to be in. And it's crazy that it's at least somewhat credible, credible enough to raise billions of dollars, as it turns out. Fascinating stuff. Okay, two real quick final questions. One, I don't know if you have anything to say about this. This may be somebody else's area, but obviously, we always get these, like, anytime we slow, do anything that would sort of slow down, or impose additional costs, or, you know, kind of put more onus on developers. Oh, China's not going to do that. You know, we're just going to lose to China. One answer, of course, is like, you know, let's not do the bad thing ourselves. Maybe China will do the bad thing. And, you know, that would be bad. But like, that doesn't mean we should

• 2:02:00Nathan Labenz?

  do the bad thing. Do you know anything about how China or other countries are thinking about this kind of stuff? I mean, obviously, it's a totally different legal environment over there. But is that something you've looked into at all?

• 2:02:12Speaker 01

  Yeah, so you sent me something along these lines. And so I did do some digging today about what China's liability system looks like. It seems like the principles are pretty similar structurally. They do have a civil law system. So it's more code based, less common law based. But the substantive principles around they have a negligence and products liability and some narrow pockets of strict liability. All those are pretty similar. Damages calculations tend to be less plaintiff friendly. So, you know, there's what's called non-economic damages, like pain and suffering, that kind of thing. And Chinese courts tend to be less generous with those. There's also, there's less lawyers there that take cases on contingency fees. I think there's some restrictions on when those are available. So just fewer of these cases get litigated. If you have to pay your lawyer by the hour, you might

• 2:02:57Speaker 01

  not be able to finance, finance a case. So I think there's just like, there's less of these lawsuits, more generally. I don't think they have any sort of, you know, bespoke liability regime for AI, but neither just the US either, really. So I think in that sense, they're on pretty equal footing. More generally, right, it doesn't, you don't have to assume that like China is going to do the same, you know, domestic regulatory regime that we do. You could imagine some kind of framework where we encourage that, right? But I think more broadly, the question is, so first of all, this critique is obviously You know, could be brought against any domestic AI regulation. So I don't think it's like, I think if anything,

• 2:03:32Nathan Labenz?

  It is.

• 2:03:33Speaker 01

  liability is is less vulnerable to this critique because it's more consistent with promoting socially useful innovations, right? The other question is just like, how how binding should we treat this threat from China? It seems like we have a pretty significant lead on on China at the frontier and that, you know, with export controls, which I have mixed views on the merits of, but at least in this context, they do that. It seems likely that the lead is going to widen in the coming years. As China, at least before, until China can indigenize its own chip supply chain. You know, they can produce some chips right now, but they don't have access to, you know, new fabs from ASML. And so I think, you know, in the medium term, that's going to be bad for their chip production.

• 2:04:16Speaker 01

  And so it's not going to be that hard, even if we're not like going totally pelted to the metal in the US to maintain a lead against China. I also am less of a China hawk than most people are. I think we share that view. And so I'm just, I'm less worried about the sort of zero sum competition other people are. But obviously, you know, reason people can disagree about that.

• 2:04:36Nathan Labenz?

  Cool. That's helpful. There's a lot more to unpack there, certainly. Real quick last one. I noticed you participated in the Principles of Intelligent Behavior in Biological and Social Systems Program, AKA PIBS. I've had a few guests with, I think, very interesting, unique takes on

• 2:04:58Nathan Labenz?

  the AI question that have come through that program. Just thought you might give kind of a testimonial or an invitation or indicate, you know, what sort of people should be considering doing that themselves.

• 2:05:08Speaker 01

  Yeah. So full disclosure, I'm on I'm on the board of the PIBS organization, but I will still answer this question honestly. I think it's, you know, PIBS is sort of has two distinctive value adds from other sort of like AI safety talent development pipeline type organizations. I think it wants to bet on like more neglected ideas. And it wants to bring in sort of like a broader suite of like people with different expertise, right? So I had been socially connected to people who are worried about AI risk, but I hadn't worked on it professionally before I did PIBs. As I think I mentioned, I mostly did climate law and policy before I did PIBs two years ago.

• 2:05:49Speaker 01

  And it was very open to the set of expertise that I brought. I did teach torts, so I had background in liability law. But yeah, I think it was great at sort of getting me up to speed on the technical issues. And then sort of like allowing me to leverage the expertise that I already had that was relevant. I think most people who do PIBs don't do governance policy type stuff. They more do alignment work, though not necessarily what people think of as technical alignment work. A lot of it is more conceptual, but I think it's open to sort of like a broad range of disciplinary approaches. And so I think it's a great way for people who think they might have something to contribute to mitigating AI risk, but haven't seen an obvious way in. I think it's a sort of more open to different ideas.

• 2:06:32Speaker 01

  And so if you think that that fits your interest, there's a fellowship that's run every summer. There's also an affiliateship program that I think is still ongoing that for people that are a little bit more senior. Most people who do PIBs are grad students or postdocs. I was a more senior person. I was already a professor when I did it. But yeah, there's a residency aspect to it. When I did it, it was in Prague for half the summer. This summer it's in San Francisco and I found it very productive. You're in a coworking space with other people working on this stuff, like people to bounce ideas off of. And so I found it to be a really valuable experience. And so I encourage people that think they might fit this broad description to explore it next summer.

• 2:07:11Nathan Labenz?

  Cool. Love it. I think there's definitely a big need for people from all different backgrounds with different novel ideas. And so PIBs is great for that. And this conversation has been a great example of that. I appreciate your reorientation of your legal career to trying to address the AI challenges that only seem to be growing in importance. So any quick closing thoughts before I give you the official send off?

• 2:07:38Speaker 01

  I think I think I've said most of what I wanted to say.

• 2:07:42Nathan Labenz?

  Cool. Well, Gabriel Wow, Assistant Professor of Law at Touro University and Senior Fellow at the Institute for Law and AI. This has been great. Thank you for being part of the Cognitive Revolution.

• 2:07:52Speaker 01

  Thanks. It was a lot of fun.

• 2:07:54Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:08:38Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.