Pod Insights The Cognitive Revolution Emergency Pod: Examining SB 1047, California's Proposed AI Legislation About this episode Dive into the nuanced debate over California's SB 1047 AI legislation with Nathan Calvin, Dean Ball, and Steve Newman. Our expert panel discussion explores the bill's intent, dispels misconceptions, and debates its potential impact on technology and regulation. Learn about AI's tail risks, the role of government, and hear suggestions for improving the legislation. Share your thoughts and join the conversation around shaping a positive AI future.
You can reach out and share your thoughts by using the contact form here : https://www.cognitiverevolution.ai/contact/
RECOMMENDED PODCAST:
The AI Daily Brief If you’re looking for an edge to stay up to date on everything AI, subscribe to Nathaniel Whittemore's AI Daily Brief which delivers tight episodes covering all things AI from legislation to new technologies to the philosophical debates around generalized intelligence.
Spotify: https://open.spotify.com/show/7gKwwMLFLc6RmjmRpbMtEO
Apple: https://podcasts.apple.com/us/podcast/the-ai-daily-brief-formerly-the-ai-breakdown/id1680633614
SPONSORS:
The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR
Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/
Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention "Turpentine" to skip the waitlist.
CHAPTERS:
(00:00:00) Introduction
(00:05:45) Opinions
(00:11:57) The Bill
(00:17:05) Elevated discussion
(00:18:51) Sponsors: Brave | Omneky
(00:20:18) AI Doctor
(00:22:55) Centralized Regulator
(00:26:29) SB 1047
(00:27:47) Critical infrastructure
(00:31:51) Should we regulate?
(00:39:00) Sponsors: Squad
(00:40:28) Safety standards for training
(00:46:09) What is 10^26 flops?
(00:52:10) Regulating general purpose technology
(00:53:42) AI regulation
(00:58:31) Societal adaptation
(01:01:18) AI safety
(01:04:20) AI regulations
(01:07:19) The current state of the industry
(01:10:08) What industry wants changed
(01:25:42) Safety is hard to legislate
(01:33:06) How to decide the trajectory of AI?
(01:45:52) Testing, Transparency, Whistleblower Provision
(01:55:17) Closing thoughts
Transcript 22,265 words · 4 speakers
Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, we have a timely and important discussion about SB 1047, the proposed AI legislation that's working its way through the California State Legislature, also known as the Safe and Secure Innovation for Frontier Artificial Intelligence Models. To understand this complex and all-too-easily
0:43 Speaker 00 misunderstood topic, I've invited Nathan Calvin, Senior Policy Counsel at the Center for AI Safety Action Fund, which is one of the bill's co-sponsors, and Dean Ball, Research Fellow at the Mercatus Center, to share their perspectives. We're also joined by Steve Newman, a self-described recovering entrepreneur who is now working to raise the level of discourse around AI issues. To set the stage, I can say with quite high confidence, based on my own close reading of the bill and related analysis, including that of friend of the show, Zvi Moschowitz, that contrary to some of the noise being made online, this bill is not some sweeping anti-technology power grab, nor is it the cynical ploy to protect industry incumbents from competition that some have alleged. As some evidence of that, consider that California State
1:28 Speaker 00 Senator Scott Wiener, who is advancing the bill, is probably best known for his pro-housing work. And he's also proposed legislation to hasten the transition to solar energy, to extend alcohol sales hours in bars and restaurants to 4:00 a.m., and even to decriminalize psychedelics. What's arguably most striking about AI discourse in general is how the strangeness of the technology creates space for people to think for themselves, and how that often results in the scrambling of familiar political coalitions. At the same time that we have ostensible political opposites like Tucker Carlson and Snoop Dogg emitting similar soundbites, our discussion today is among people who are generally very pro-technology and very skeptical of government regulation.
2:12 Speaker 00 As you'll hear, while Nathan and Dean do still ultimately come down on different sides of this legislation, they identify many, many points of general agreement along the way.
2:22 Speaker 00 Personally, while the bill may not be perfect, I appreciate that Nathan and his co-sponsors are making a genuine effort to focus on the critical issue of tail risks posed by advanced AI systems and not placing an undue burden on startups and smaller players in the AI ecosystem, even though that means leaving other important questions, including the risks associated with algorithmic bias, the future of professional licensing, and the general impact on the workforce at large for another day. At the same time, as someone who has long identified as libertarian and often been frustrated by government overreach, I'm definitely sympathetic to Dean's concerns that a new agency could exploit the bill's ambiguity to gradually expand its scope and power in very problematic ways over time.
3:05 Speaker 00 Ultimately, it seems to me that some form of government regulation of AI is inevitable. And so I feel that it's well worth genuinely engaging with and attempting to improve SB 1047 as it goes through the legislative process. With that in mind, toward the end of this conversation, in search of a productive synthesis, I put forward and we discuss several possible amendments that I hope might make the bill a bit better. Overall, I think you'll find this to be an unusually respectful, nuanced, and insightful discussion on what might be the defining policy issue of our time. If you agree, I'd ask that you take a moment to share it with your friends. And on this topic in particular, I really want to hear your best ideas for how government can help foster a positive AI future.
3:49 Speaker 00 I have an interview scheduled with Senator Weiner for next week, and I will definitely read everything that you send me and attempt to bring the best ideas to him directly in that conversation. You can reach me, as always, via our website, cognitiverevolution.ai, or by DMing me on the social media platform of your choice. For now, I hope you enjoy this deep dive with Nathan Calvin, Dean Ball, and Steve Newman into California's AI bill SB 1047. Nathan Calvin, Dean Ball, and Steve Newman, welcome to the Cognitive Revolutions.
4:20 Dean Ball? Thanks so much. Happy to be here.
4:22 Speaker 00 Likewise.
4:25 Speaker 00 So we're gathered together for a discussion about the proposed legislation that is working its way through the California State Legislature, which is known as SB 1047, or as the authors want it to be known, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. This legislation would be first of its kind. Obviously, we've seen the executive order, but there hasn't really been anything passed through a legislature that would try to meaningfully govern what's going on at the frontier of AI. And it's predictably been the topic of a lot of discussion, debate, seemingly some miscommunication, some misunderstandings.
5:08 Speaker 00 Well, so I wanted to bring this group together just to explore those and see what we can potentially hope to come to some agreement on. Also, it's worth noting, too, that we're not yet at the point of an up or down vote. So in terms of how to approach this today, I got a bit of outreach from Senator Weiner's office in California, and the guidance there was like, this is probably going to play out over the next several months. There are going to be a lot of opportunities for people to make amendments. So we're very much still in the discussion phase. And potentially, if we come up with anything that we all see good about, we could even recommend that to the authors. So that would be a big win in my book. For starters, Nathan, you are the Senior Policy Counsel at the Center for AI Safety Action Fund. You guys are
5:52 Speaker 00 one of three co-sponsors behind the bill. You want to just start off by giving us a little bit of an introduction on who you are, who the organization is, your kind of background worldview, and why it's something that you're motivated to work on? Then, Dean, I'll give you the a chance to give a similar story next.
6:10 Speaker 02 Yeah, absolutely. So to start with the organization, so the Center for AI Safety Action Fund is like the final one before affiliate of the Center for AI Safety, which listeners might know from the statement we put out about insurance for AI, as well as a lot of the technical research work that this center does, including by our Executive Director, Dan Hendricks. The 501(c)(4) was partially created because we were getting lots of inquiries and incoming from policymakers, including Senator Reiner and we wanted to have a vehicle that could do more direct quality work, which is something that I don't want to be aren't able to do. And yes, specifically for involvement in the bill, Senator Reiner put out an intense bill
6:53 Speaker 02 mid last year, talking about being interested in doing a bill on these issues. And he approached us and the other co sponsors to help him really flesh that out in something in this space. But I think Senator Reiner became concerned, as a lot of us have, that it seems like the pace of development in AI is really exciting and I think has a ton of benefits. But I think we are starting to see some of these glimpses of these really large models have pretty dangerous capabilities. I think we've got similar discussion at the AI executive order and that's important. But I think there are things that you can only do in statute and California is an important jurisdiction and it seemed like a really great opportunity. And I'm yet been really impressed with him as an author. It's a really smart guy and has been really into the details on this and it's been really
7:35 Speaker 02 just a pleasure to work with and with our other co sponsors of the bill, the Economic Security Project and in Pro Justice. Yeah, just speak a little bit about my own worldview about this. I think that I am just very uncertain. One of the main things, which I think is that I have a lot of excitement about AI. I think these tools are really amazing. I use GPT four and Pod because I'm a daily day business and I'm a big fan of these models and think that they have a ton of potential to improve our lives and work and to improve, bring a lot of innovation that we desperately need to solve really hard problems. I'm also someone who in general is like very, I think, critical of unnecessary regulation
8:17 Speaker 02 or bureaucracy, big fan of Senator Reiner's housing work and trying to build additional pure energy of trying to reform the FDA, a lot of the things that I think you and I on a separate podcast to probably chat about and not a lot to to agree on. I just happen to think that really the largest
8:34 Speaker 02 AI models that in addition to their potential to do a lot of good, I think they might be able to do just really powerful thing. And that we should be having some amount of precaution and kind of leading it totally up to the companies and voluntary measures about whether they want to be doing testing. I think we're hearing from both in the last that there's just tremendous commercial pressure to get involved as fast as possible. And I think there need be a role for governments to say, let's be thoughtful here, let's get the benefits, but let's not make sure that we're, you know, raking ahead and causing catastrophe. I think the one thing I want to say is I'm a very big fan of the general structure of the bill and will defend that very strongly. But I think a lot of the details about the big point, like the complicated issue of the bill has already been amended.
9:16 Speaker 02 It will continue to be amended and I really speaking in general, I mean, I'm going to be speaking for myself and kind of for a big reaction fund. But on this area, I'm confident the author also shares this, that he's really interested in amendments. We want to get this right. There are a lot of different technical terms and definitions. We have our bare intent, but we want to make sure that those definitions reflect what our intent is. And it's really to target specific, very dangerous concerns of models while really generally letting the AI ecosystem bloom and flourish as then continues. So that's my general standing. I think I'm worried about a variety of different risks. I think I'm worried about like a super powerful, it's a model that doesn't count humanity.
9:54 Speaker 02 But I think more and more correctly, I'm also just like worried about like the ability for bad actors to use this for for malicious biotax, really severe attacks on critical infrastructure. I think some of the risks that were identified in the Biden executive order from like dual use foundational models, I think similarly of novel DDR and developments, attacks on critical infrastructure and models that like deceive or self replicate or are otherwise really hard to get back control of. So that's what we're broadly thinking also care about and a lot of other issues also really important within AI and a lot of the things about deepfakes and the future of work and racial bias in government systems. And all those are super important that I think are also important discussions we had.
10:34 Speaker 02 But this bill is really focused on kind of these catastrophic national security risks from the largest models and similar in DNA to what was attempted in the Biden order. But that's broadly where I'm tied from. I think that. We do need to take some action here. I think that like we really are on an exponential curve and I think that there's some temptation to be like, this is really complicated. We could wait a year, we should wait. And so we can make everyone incredibly comfortable with it. I think legislation takes a long time. We already have in the legislation and as I think some folks have noted, some like delayed implementation or even after the bill is passed, it takes a while to get government set up around these things. And I really do think that we're looking at all possible to pass something this session so we can start that process going. And I really think that not how it being interpreted and on
11:17 Speaker 02 Twitter and things like that, but I think there are already a lot of concessions in this bill and trying to be a moderate, not maximal political. We can get more into all this. That's probably where I'm coming from.
11:27 Speaker 00 Quick follow up question. When you refer to the author of the bill, is that State Senator Weiner?
11:32 Speaker 02 Senator Weiner, yeah. Terminology is weird because at the federal level, you refer to the bill author as a sponsor, which is the opposite. So when I say the author, Senator Weiner. We have bonded, provided like technical advising and not dealing with that amendment to language, but ultimately him and his legislative director have been very involved in this and it made the final call on all the regions on the direction of the bill. And yeah, the time referring to the author, I'm referring to Senator Weiner and his team.
11:56 Speaker 00 Cool. Okay. Thank you. Dean, you are research fellow at the Mercatus Center. You've been a previous guest on the podcast when we talked about all things brain-computer interface. So obviously a big and pretty deeply technical engagement with technology, AI, all these sorts of things, though your day-to-day focus is on policy analysis. Give us a sense of where you're coming from on this.
12:19 Dean Ball? Sure. So first of all, I am at the Mercatus Center. That is a think tank affiliated with George Mason University. We are a 501(c)(3) and what that means is we're in the same tax status as a university. We try to educate the public and policymakers about matters of public policy using insights from academic research. So that's where we come from in a broad sense. Specifically on AI, I think that AI is a general purpose technology, quite possibly the most powerful general purpose technology that humans have ever invented. Certainly the most powerful one of sort of the recent wave, at least it could be, of
13:02 Dean Ball? general purpose technologies, computing, the internet, et cetera.
13:07 Dean Ball? As a general matter, we don't tend to regulate downstream use of general purpose technologies with a centralized government regulator. That's just not typically how that's done. There's not a Department of Electricity. There's not a Department of the Internet. There is FERC, the Federal Energy Regulatory Commission. They set standards for interstate transmission of electricity and all kinds of stuff like that. But they're not regulating my downstream use of electricity, which surely can be used to do all kinds of dangerous things, nor is there a Department of Computers. I think that if we want the AI revolution to be successful and to broadly benefit people,
13:48 Dean Ball? then even to speak of AI policy as a discrete area of public policy is probably going to seem a little bit anachronistic in 20 years. That's my hope. I wouldn't necessarily predict that, but that is my hope. Because I hope instead that AI is so integrated throughout our economy that it is regulated in all the myriad ways in which economic activity and public policy interact with one another. And so I think that creating a centralized regulator is just likely to lead to all kinds of political economy problems that will hinder the impact of AI,
14:33 Dean Ball? the good impact of AI, in ways that I can't even predict. And I'm not really sure that it gets us that much more safety either in the long term. In particular, because most of the concerns we have here, CBRN, cybercrime, these things, certainly they could come from domestic actors. Nathan, Calvin, you you reference national security concerns. I don't believe that the CCP cares about
14:58 Dean Ball? Qualifying for a limited duty exemption under the state of California, nor do they care about the NIST AI Risk Management Framework. So I'm not really sure that this gives us safety in any meaningful way from those types of actors. And instead, I think to the extent it hinders AI progress, it actually creates danger, which is what regulation often does of these kinds. It creates a false sense of security, but in fact, has a tendency to lead to long-term danger. I really think that nuclear energy is a good parallel to think about here. Except for at least nuclear plants, there were some nuclear incidents that raised a lot of concerns across the world.
15:42 Dean Ball? There is no AI Chernobyl so far. There is no AI Three Mile Island. And we created the Nuclear Regulatory Commission and legalized nuclear energy effectively after some major incidents. There have been no such incidents here. So if we're going to legalize AI progress, I'd at least rather do it after we have at least one concrete example of a major problem. I just think not only is it premature, but it's the wrong way to go in general. So that is, I just want to be very clear, though. I am not opposed to governments passing laws with the word AI in them. I'm not opposed to a vigorous government role in the regulation of AI.
16:25 Dean Ball? I just am skeptical of the idea that there is one big thing you can do, one major bill you can pass that will do what you want it to. Instead, I think the answer is every agency internalizing the risks and benefits of this technology in a serious and considered way, and taking steps within their own jurisdictions and their own areas of expertise across the city, state, and federal levels. I think there's not one big thing. There are five million small things. And that is the work that I hope we can do. And I think that frankly, centralized regulators hinder that work.
17:06 Speaker 00 Thank you both. Steve, to introduce you briefly as well, you are a exited entrepreneur. Most people will have interacted from time to time with one of your creations, which is now known as Google Docs. And having exited that line of work, you're now focused on raising the standard of discourse related to AI, which is obviously a noble pursuit. I think we're in a pretty good position to hopefully have some elevated discourse here because for one thing, I don't hear very extreme positions. Like you will sometimes go online and find people saying things like, oh, who cares if humanity goes extinct, the machines will be more worthy in some, you know, sense than us. Anyway, not hearing anything along those lines.
17:49 Speaker 00 I'm also not hearing argument from protection of Weaver employment, so to speak. I'm not hearing anybody saying that the current mode of production and the current employment relationships and dynamics are like the primary concern. It seems like everybody on this call is broadly pretty enthusiastic about the upside of AI and technology in general and broadly, and certainly this is true for myself as well, skeptical about government overreach into areas that are probably best left to develop on their own. Do you have any other thoughts on how we can make sure we have the most elevated discussion that we can? Well, I think the first step is just bringing the other people who have a positive attitude and are interested in engaging and with an under ten and that we're going to have a positive discussion.
18:33 Speaker 00 I think you've done a great job of queuing that up. And I think really looking forward to seeing Nathan and Dean dive in. And I think we've already seen just from the everyone's introductory remarks that we're here to kind of talk about reality and talk about things in a reasoned way. And that's 90% of the battle right there.
18:52 Speaker 03 Hey, we'll continue our interview in a moment after a word from our sponsors.
18:57 Dean Ball? I want to pick up Nathan Labenz on something you said about protecting Weaver employment.
19:05 Dean Ball? This transformation that is coming will be an economic transformation. I don't think anybody has intelligently modeled the labor market impacts of AI. I know some people who are trying. I don't know how possible it is right now, but it's going to be a struggle because there are a lot of areas of the labor market that are protected with barriers to entry and barriers to entry will be erected to keep AI from entering, whether it's driverless cars, medicine, law, there's a thousand things where, you know, current industry groups are going to try to continue extracting rents. And that's going to be a real struggle. I think that will be
19:47 Dean Ball? a huge policy struggle over the coming decades. And if you don't believe a centralized regulator will be a weapon that will be used against AI for diffusion, I think that you have not been paying appropriate attention to the way that government has generally conducted itself at the state and federal level for my entire lifetime at least. So I think that's one thing that's important is that I am very concerned about maximizing the benefits. I want to do that. And I think a centralized regulator presents all kinds of challenges to doing that. That's just one additional point I would make.
20:29 Speaker 00 Yeah, I think probably you'll have agreement in the general sense that we can find plenty of examples of government gradually overgrowing the original intent of the legislation under which it's operating. Maybe we can approach that from the standpoint as we get into the details of like, how can we prevent that from happening in this case? I often say, when people ask me like, why should we want any of this? Isn't this all just going to be terrible? I'm like, AI doctor, that's my first thing. When I first got my early puberty four access, I couldn't stop asking it all these medical questions. Oh, that I have that many. Fortunately, it was really good and so convenient. And that is, in my view, almost the canonical thing where if we end up
21:10 Speaker 00 in a world where we prevent people from getting quality medical advice from AI, whether out of irrational fear or bad cost benefit analysis or protecting doctor employment, it seems like we'll have made like a very bad mistake. So. If that's something we can agree on, then we can get into that tendency, which definitely is out there, could be mitigated or prevented. Bear?
21:34 Speaker 02 Bear, I guess I just want to say that I agree with you that there are a lot of really large benefits here, and I think it would be quite bad to not get them. And I think some of that will involve pages in the future at work. I think among the coalition that we're working with here, there is like a variety of different views with it among you. And I feel like to come to be like this bill is not necessarily taking a position either way on the labor market future and whether you could be trying to have some women. I have my own views about that, but I think either way, I just don't think what this bill is about. And if you are someone who is either concerned or optimistic about AI displacing human jobs, that I don't think the bill is necessarily going to have a lot to say in either direction, to be totally honest. Just want to clarify that. And then also, and I think we'll dig a lot more.
22:17 Speaker 02 I would push back on the idea that this bill is a centralized regulator. It's not a licensing regime and the only government's body that has created a new division of the California Department of Technology. You're just like receiving a certification. It doesn't have rulemaking power. It doesn't issue licenses. Most of this a liability regime for the Attorney General. So that still is probably more centralized than you would like. But I think there are other proposals that really are like centralized AI regulator. We're getting pre-approval to be able to do training runs where they really are doing the task themselves, doing a lot of that. I agree that there are really big challenges of that, of having government capacity to really understand all the things going on at the cutting edge and doing that effectively. And I just want to say I also, I think, have concerns about the efficacy
22:58 Speaker 02 of a centralized regulator, particularly given the existing technical expertise or lack thereof within government. And so I'm sure we'll continue to debate about how we're correcting the bill, but I also have hesitation about a centralized government regulator here. And I don't think that's actually what this bill does.
23:14 Dean Ball? Only thing I would say is the Frontier Model Division in a recent amendment to the bill does have the power to change the compute threshold. It has the power to change the specific standards for models that fall under its jurisdiction. So that already is starting to look a little bit centralized. And the other point that I think is under discussed here is that we talk about repeating the mistakes of nuclear energy. The Frontier Model Division, because of California's current $60 billion budget deficit, the Frontier Model Division is funded by fines and fees that are levied on AI companies.
23:54 Dean Ball? That's exactly how the Nuclear Regulatory Commission is funded. The regulators, the regulated entities pay for their regulation. This obviously creates regulatory capture potential, right? Like on its face, their salaries are paid for by the people they regulate. But in addition to that, it insulates the frontier model division from the normal appropriations process and legislatures where maybe five years from now, we're not as worried about AI risk. Maybe all the concerns of folks like Jeff Hinton don't pan out. And we want to lower the budget for that. But guess what? You can't because the FMD funds itself.
24:34 Dean Ball? Or it's inconvenient to, because the frontier model division funds itself from the AI industry. And so not only does that insulate it from the budget process, but in essence, it insulates it from the trade offs of the political process of democratic input. And I think it's probably worth California maybe thinking seriously about why they have a $60 billion budget deficit before they create a new centralized regulator for the most promising technology in the world. But also, I think that it would just create all kinds of problems. Just the funding alone.
25:06 Speaker 00 So come back to this question of how centralized the regulator is in a minute. I have done my own close reading of the bill and broken it down into basically six things that I would say it does that seem to matter. And we can then dig into the details of each one, either one by one, or you can add categories to the list if you think I missed any. First one, as I think about it, is it sets a standard for what are we talking about here? What is the actual risk that the bill is trying to address? There, it sets what I would say is a pretty high standard of either.
25:48 Speaker 00 Mass casualties or $500 million or more in damages to things like critical infrastructure. So that for me was like, okay, that's a pretty big thing, right? Like mass casualties, $500 million in damages. I think if you were to go ask most people on the street, at what level of harm would you think the government should start to get involved in AI? They would probably come up with something that is lower than mass casualties or 500 million. I don't know if Dean, you would say that you would be worried would be subject to future discretion or lowering of that threshold. But for me, it's a high number.
26:25 Dean Ball? So I don't think, I don't know this for a fact. I don't think that the bill allows the frontier model division to lower the hazardous capability threshold. I don't believe it can do that. However, I would say the following things. Critical infrastructure is an extremely deceptive term. Critical infrastructure is, it's a little bit Soviet in the sense that critical infrastructure is in fact a larger category than infrastructure. So critical infrastructure includes things like hospitals, all financial services, all telecommunications infrastructure. It is the majority of US economic output. So when you say critical infrastructure, don't think that you're talking about a small subset of things like nuclear power plants
27:08 Dean Ball? or something. You're actually talking about most of the US GDP. $500 million is a lot of money. At the same time, cyber crimes worldwide in 2023 were responsible for about $9 trillion of damage. So 500 million not indexed to inflation. That could be not so much money. It already is not so much money. You could probably, depending on how you calculate it, if you knock a wastewater treatment plant out, you don't blow anything up, you don't do anything like that, you just knock its computers out for a couple of days, you've probably done $500 million of damage to critical infrastructure. And obviously, we don't want people knocking out wastewater treatment plants. I'm not saying that's like a light thing. I'm just saying that you should think.
27:53 Dean Ball? Very carefully wastewater treatment plants go down, right? They go down a lot. Cities issue boil water notices on a regular basis in this country. We shouldn't think that's like some unprecedented thing. And just because it happens to be caused by A, it's just a little like, uh, I don't know. It's the new thing, right? But, oh, it was caused by computers. I don't know. Like, I think that it's a big deal and we should do everything we can to protect our critical infrastructure and be very serious about cybersecurity. But I actually think $500 million is not as big as it sounds.
28:26 Speaker 00 So to read the definition from the bill of critical infrastructure, it says, "Critical infrastructure," quote unquote, "means," and it is pretty broad, "assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state."
28:47 Dean Ball? Yeah.
28:48 Speaker 00 So that is, I think we are probably really relying on the number there more so than any sort of narrowness of definition because, as you said, like, a lot of things, you know, would have the potential for debilitating effect on physical, economic, security, public health, safety. Now, this is maybe more of a nitpick than anything I'm sure really matters, but for what's worth that nine trillion cyber crime number? I just came across that and I don't really buy it, to be totally frank, because that would be like pushing 10% of global GDP. If global GDP is $100 trillion and US is like $20 trillion, where that number has been flying around, I've seen that actually used on the, on both sides of the debate now because I've also seen people on the AI safety side being like, look how big of a problem this already is.
29:33 Speaker 00 And I'm like, there's no way that we're losing 10% of global output to cyber crime. That just seems crazy. I actually saw that a fundraising got struck too long ago for an AI assurance tech company. And I was like, I'm gonna have to downgrade your market size a little bit, but.
29:49 Dean Ball? Well, no, but because of, I don't know how that number is calculated, but because of the broken Windows phenomenon. That some of that $9 trillion might be accretive to GDP, right? It's not necessarily that it's a $9 trillion reduction in GDP. Costs of things are accretive to GDP, right? If I break a window and I fix the window, that's GDP. I would just point that out. And I think, but I think that that $9 trillion figure, it might be like totally fecockta. That's entirely possible. That should tell you something about $500 million in the context of cyber attack damage. How do you calculate it? I don't know. You could, there's probably ways you can fudge those numbers.
30:28 Speaker 00 Yeah.
30:30 Speaker 00 So let me guess, we take a step back for the moment. Dean, I'll address this question to you. Should we be regulating anything? Is anything bad at a large scale plausible to happen? Or in other words, are we debating exactly what's the best way to write this? Or are we actually debating, is there even a problem here?
30:49 Dean Ball? So I would definitely say we should be regulating things. I think I have a rather circumscribed perspective on that which government can effectively regulate through a single statute. I think there are many different things that government should be doing to incorporate AI into various criminal statutes and scale up their enforcement of things and increase defenses and all that kind of stuff. But is your question, what do I think the outcome would be if the bill passes?
31:19 Speaker 00 So the bill doesn't pass if no bill passes. In other words, we're talking about the wastewater plan being knocked offline. And from one perspective, like that could be a really big problem, could cause, I don't know, half a billion dollars in damage. From another perspective, it's a big economy and a big world and big problems happen. And maybe if AI was a little bit involved there, it doesn't really mean that that was fundamentally an AI issue. Taking a step back from the specifics of half a billion dollars, should we be worrying about AI? Is there a problem that we could be regulating, whether or not this regulation is the right way to go about it?
31:52 Dean Ball? I get the sense you're asking about catastrophic risk. Am I worried about existential type major, major risks from AI?
32:01 Speaker 00 I'm not trying to go all the way to existential, but just big stuff like cyber attack that brings the whole US grid down for a month or knocks all the wastewater plants offline. Or things that are happening in a big way that traditionally have not happened in a big way, or just get a lot easier to happen. Or rather, should it be half a billion or should it be two billion? Just is there a here and there as you see it?
32:26 Dean Ball? So I share Nathan Calvin's and I think probably Nathan Labenz's general uncertainty, like fairly strong uncertainty about the future trajectory of AI capabilities. So I would say, yeah, like we should take the concept of a cybersecurity attack that takes major electricity grids down for a long period of time. I think that is definitely something that we need to be worried about. I question the wisdom of passing a law that says it is illegal to take down the US electricity grid, because I suspect that it already is. And I think that basically if we pass a law that says it's illegal for the AI transformation to not go well, well, that doesn't do much for me.
33:08 Dean Ball? That doesn't advance the epistemic ball in any important way, because we all want it to go well. We all want this transformation to go well. We don't know what that means. Nobody knows what that means. And if the government just passes a law that says it has to go well, then we all have this epistemic uncertainty. And now you've introduced lawyers into the equation. Not that lawyers shouldn't be part of this whole dynamic, but it's, oh, great. Cool. We can go to court and talk about this. Sounds fun. I just, I question the wisdom and really the efficacy of laws of this kind. But I would definitely agree. The AI existential risk, inherent risk, I'm not all that worried about Claude four itself, developing its own motives and trying to kill all of humanity or whatever.
33:52 Dean Ball? I think that's a little bit of a Jejun debate. But, but I'm definitely worried about future models being used for all kinds of things. And cybercrime is at the top of the list for sure. So yeah, there's definitely a there there in my mind.
34:07 Speaker 02 One thing that I think could be helpful is that I agree with that in a lot of settings, the right way to handle regulation is to focus on kind of the downstream uses of the technology. I think it's fair to say, look, we're not gonna, you know, place a bunch of burdens on email or the internet. We're gonna focus on people who use those tools to do bad things and go after them. And that's appropriate and our laws are able to handle that. I think there are lots of settings where that work and I think that's appropriate. I think there are settings where that doesn't necessarily make a lot of sense. We have things like in in nuclear policy where we don't say, oh, we're gonna go after terrorists who use plutonium or something. We know the people who are enriching nuclear materials have some obligation to secure those materials and to be thoughtful about it.
34:50 Speaker 02 I think similarly, I'm very enthusiastic about biology. I think biology has tons and tons of benefits and I'm very optimistic about a lot of different factors. I don't think that therefore means like we should not think at all about gain-of-function research. And if folks are doing lots and lots of gain-of-function research, then therefore our solution should be, hey, let's just go after the individual terrorists or folks who are doing that. I think that we have to have some process also higher up on the screen. But I agree that I think that this is difficult this way, but I think in a lot of cases it makes sense to focus on the downstream applications. I just think that fully advanced, the largest AI systems that have a potential of having these pretty devastating impacts are one of a small handful of technologies where we do need to be thoughtful about whether regulations are necessary.
35:33 Dean Ball? I agree with that general framework you've laid out. The fiendishly difficult aspect of this, though, is that plutonium is not a general-purpose technology. It's not a consumer technology that I might use on my phone. So that complicates it considerably. I think I agree with you that there is room for regulation and a need for regulation somewhere between the downstream use and the creation of an AI model. I actually think AI might just be a little too low in the value chain or the production chain of threats, one might say. So a concrete example is the Biden administration just a few days ago
36:14 Dean Ball? released what in my mind is a very productive framework for nucleic acid synthesis machines.
36:21 Speaker 02 Also a big fan.
36:22 Dean Ball? Yeah, where they basically said, if you're gonna make, if you're gonna do federally-funded life sciences research, you have to use machines, DNA synthesis or nucleic acid synthesis machines from.
36:36 Dean Ball? Specific providers that will comply with these standards and those standards include that sequences that you want to produce, genetic sequences you want to produce, have to be screened against a database of known threats. Great, excellent. But I think that sort of going one level removed from that to like the statistical analysis of genetic data sets and regulating that, you know, I'm an American and that does raise constitutional problems in my mind, right? I think I do have a constitutional right to do statistical analysis on genetic data. I don't think I have a constitutional right to synthesize DNA. And I think that's like a pretty clear distinction that like at least most Americans can intuitively rock. Constitution, as V. Moshevitz has said to me, constitution's not a death pact.
37:21 Dean Ball? So I'm not going to rely just on a constitutional defense. I will also just say that it's too low to be a productive regulation. It's like regulating metallurgy rather than missiles.
37:32 Speaker 00 Hey, we'll continue our interview in a moment after a word from our sponsors.
37:38 Speaker 00 It's a good time for me to interject the second point of the six, and maybe I should even just run all six down real quick. We covered point one, which is the standard of the scale of harm that would be of concern. That is, again, mass casualties, $500 million damage to infrastructure. The second point is, how would you reasonably expect that you might be playing in that regime? Here, the bill uses the same threshold as the executive order, at least on the first definition, which is 10 to the 26 flops. So if you're going to train a model that is going to use 10 to the 26 flops going in, then that is presumed to be a covered model.
38:17 Speaker 00 That is to say, if you can't make a positive assertion for good technical reasons that this thing is going to be safe, meaning it's not going to be capable of causing these sort of downstream. Mass casualty or $500 million worth of harm, then you have to do certain things. And those things are you have to implement a lot of safety standards during the training process. So the the weights can't leak to the public or to the Chinese government or whatever. You have to, that includes having the ability to do a full shutdown. Not exactly clear on what would happen to a full shutdown, but the idea is basically you want to have
38:59 Speaker 00 an off switch. So if things get crazy, you can flip that switch and stop the process. You have to, at the end of that training process, again, this is assuming you didn't have good technical reasons upfront to say that it'll be safe. Then at the end of that process, you then have to do the testing and say, okay, now that we've made this thing, let's see what it is actually capable of. And at that point, you have another chance to say, okay, actually, we did all this testing. And now we can say, even though we didn't have a theoretical reason upfront, we now have a practical reason based on testing that we can confidently assert that it's safe. And then if you still can't do that, you have to deploy only with surrounding safeguards, which are meant to prevent these
39:42 Speaker 00 large scale harms from happening. Basically, you have three tiers. You could say, I have a theoretical reason, and I really would love to see that emerge, right? That would be the real silver bullet, which doesn't seem like anybody's super betting on at the moment, would be like a conceptual breakthrough that, hey, if we do these things, design the system this way, create the data set in a particular way, whatever, that we can just be confident that that's going to be safe from the get go. Now everything is awesome downstream. There's, you know, minimal imposition, right? From the bill, if you can make that kind of assertion. But if you can't do that, then you got to hammer it with a bunch of tests, find out what it is in fact capable of. Again, you have a fork there. Hey, we tested as much as we could. We determined that it can't do these things. So we're
40:27 Speaker 00 Or if still not, then we have to have guardrails, which would be like moderation, monitoring of usage and potentially also not releasing the weights into the wild, because this is probably one of the more contentious pieces of the bill, that if you do release something into the wild, then modifications of it downstream are still your responsibility. So you have to be able to assert not only that it will do this in the form that we release it, but also that it can't be like relatively easily modified to that state. And this is where I would refer listeners back to the FAR AI episode that we did with Adam Gleve, because they showed that it really doesn't take much in a lot of cases.
41:10 Speaker 00 If the capability is in the model, then the sort of RLHF finishing that they are generally treated with, the end of training is maybe enough as long as that stays in place. But very little fine tuning typically downstream can reverse those sorts of safety fine tunings, even in a non malicious way. That was one of their most notable findings is that even just taking a small chat data set and saying, hey, we don't, we want to have you chat in a particular way, not training it to do bad things, but just omitting the fact that it's supposed to refuse bad things, that that final fine tuning step can, in a way that the developers didn't even intend, reverse that.
41:51 Speaker 00 So it's on the, the model trayor to make sure that not only is this thing not exhibited, but that it can't be easily unlocked with downstream fine tuning. So that's like the big one. And then the one other thing that's worth noting there is, and I think this is also where a lot of the debate and the kind of heat is right now, is that 10 to the 26 flops is like clear enough. But then there's also this other definition that's if you're going to train a model that could reasonably be expected to be similarly powerful to what 10 to the 26 would get you, then you're also in that regime. And that's where I don't really know if I'm in that regime or not. So anything that you guys would clarify or refine on my description of that? Obviously, a lot of people have offered their summaries, but how would you grade mine?
42:37 Speaker 02 I think good summary. Definitely hear you on the point that yeah, I think one part that that is harder to know is saying, I know I'm treating a model below 10 to 20, but I'm not sure whether it will be as capable as a model at data about the art at 10 to 20 to 24. How can I make sure whether I then have to do good things and offer added vision, which says that basically if you are surprised by how good the model is, that's fine. You're not in trouble, just like institute those big cards after discovering that you were surprised and that's okay. And but in just some understanding that like, maybe in the future, we will have more accuracy and not exactly how high performing a model will be prior to training, but at least in the current regime, we recognize that you might not know that until after you started,
43:20 Speaker 02 until after you train the model and started doing various thing about its capabilities.
43:24 Speaker 00 Just to give people a sense of what is 10 to the 26 flops. First of all, it's not entirely known whether any already trained model has hit that threshold or not. If it has, it's almost for sure happened at either OpenAI or perhaps Google. The biggest meta trained still in training Lama three is expected to be halfway there. It's funny because I think it's been estimated at four times 10 to the 25, which is sounds really close, but it's still only 40% of the way there because orders of magnitude. It's the last one is always the one that matters the most. So what meta is spending on this Lama three, 400B still in training now
44:09 Speaker 00 is at retail price, something like $200 million in compute. They have published a bit of guidance on how many H100 hours they've put into the first two models that they put out. They put in over 6 million H100 hours into the 70B version. And so extrapolating Andre Karpathy had a nice analysis of this, which I'm taking as closest thing available to ground truth for these purposes. And then just putting a price on what you in the market today and H100 hour. If it's not dramatically subsidized, push is usually about $4 an hour for an H100. So roughly speaking, you do the back of the envelope math, you're getting to about a $200 million.
44:56 Speaker 00 Compute budget for the current Lama three. So you're in, you're in somewhere in the sort of hundreds of millions to half a billion dollars just to buy the 10 to the 26 slot. Uh, now I know Dean, you may have some thoughts on this because I know you've written a bit about how you think this is not a great way to frame this in part because efficiency is definitely a major driving force here. I don't know if there's any alternative that you would propose if we were going to keep the general structure of this. Is there anything other than a slot number that would make sense to anchor off of? But in any event, that that's a sense for the budget.
45:32 Dean Ball? I, I would say a couple of things about the compute threshold. First of all, yes, uh, it is the case that oftentimes regulation will be triggered by a firm's size, the revenue that they have, the number of customers that they have, whatever it might be. Firms as a general rule want to get bigger. When you impose thresholds like that, there's ample evidence in the economic literature that when you impose thresholds like that, you create all kinds of weird problems. Well, firms where firms will just skate right below the threshold. The difference is that firms as a general rule want to be more compute efficient. Uh, so with a threshold like this, you are swimming against the current and I think you're ultimately incentivizing
46:15 Dean Ball? the frontier model division to lower the threshold over time. Um, and they can do that under this bill. Um, and the other thing is I'm just, for me, it's not entirely clear what 10 to the 26th is based on. It does, I've never really heard a scientific basis for 10 to the 26 slots. So Nathan, if you have any answers there, I'd love to hear it, but that to me is the main thing about the threshold, but I think we shouldn't be regulating at the model layer. And I think that the fact that it is so hard to come up with good thresholds and make this stuff legible to the state is like evidence in favor of my argument. So I don't have a good alternative
46:56 Dean Ball? solution. My alternative solution would be not doing this and doing other things instead. It would be not doing nothing, but doing other things.
47:04 Speaker 02 Great, just to briefly justify where the 10 to the 25th number comes from. I think the idea here is just that this is the next generation of models, so not GPT-4 and Puffier, but really the ones that companies might be releasing potentially later this year or next year. And I think it comes from the understanding that like, we don't think that any of the current generation models are capable of causing these really severe harms. But we think that there are like some glimmers or warning signs that future generations that they should be like keeping their minds open to the potential that they could. And I think that's where that is coming from. The stuff like OpenAI's study on biorisks from GPT-4 saying, look, maybe it helps like a little bit, but it's really not very much. It is not the stuff that you couldn't find on Google.
47:48 Speaker 02 But saying that, look, as we create larger models, we can keep our minds open to the fact that this really might provide quite substantial assistance to a layperson in creating a really nasty biological weapon. The idea is we have some sense about what the abilities are of current models. I think on the lower threshold thing, I think you're talking about it's guidance about what sort of threshold might be done, like covered under like B. It's not a formal rule. It's guidance about it. There was consideration about giving them rulemaking power to change the threshold, and we didn't do that.
48:17 Dean Ball? Yeah, I guess I hear that. And I think if we were debating all of this and the current frontier was GPT-3 or GPT-3.5, we would 100% be saying this about GPT-4. And in fact, people did. People said it about GPT-4 and GPT-3 and GPT-2. And not to say that it's not eventually going to become a thing. I'm not dismissing that entirely. But it does seem to me like a vibes-based argument for imposing a threshold like this of it's the next generation and we think that could happen maybe. But otherwise, we have no empirical or scientific basis for these claims. I just think we should not create an unprecedented regulatory regime for software. And that is what this is.
49:01 Dean Ball? It is unprecedented. We have never done precautionary principle, generally applicable precautionary principle or liability regime for software. Never done it in the history of this country. This would be the first time. Software has given us a ton of economic and social benefits. So I think the bar for that is high. And I just don't think we think the next version Might be dangerous is a sufficiently high bar for something so unprecedented.
49:26 Speaker 02 I do think it's fair that if we were having this conversation around when like GPT three would release that people might say maybe there are some warning signs. And I think basically like OpenAI and Anthropic and the companies basically did the things we're describing in the bill, right? Like they tested them, it felt they determined like, hey, can't actually do these nasty things. And we can release in the form of that. Okay. I think it would have been fair then to say, hey, there's a potential. They should do these types of things, which I think they fact did and then released the model and everyone using it, including myself. And it's great. I think there's some level where you should just have uncertainty. I think at this level and the next generation, we're at that level. I think it's debatable. And I think from where they were sitting with GPT three, I think they should have kept an open mind to the fact GPT four might have, it turned out it didn't. I'm glad it didn't. And maybe someone could made a more bulletproof argument for why it definitely wins.
50:09 Speaker 02 Again, I agree that this is a weird situation we find ourselves in. Different from a lot of the things, I don't know, we don't have with like email. The creators of email and the leaders of all the email companies signing open letters saying it might kill everyone. Fundamentally different thing here. We have these rules of thumb and we think have have worked, but I think we really are like. Nvidia is worth more than Saudi Aramco or what video like went down a bit. This is like a crazy, unprecedented, wild thing that people are promising might be able to solve all these crazy problems and do all these insane things. And I think it's not unfair to say that are like normal intuition for how to regulate general purpose technologies or software. We, I think there's reason to think that these really incredibly large models that gave the belief that we don't understand. It's an area where our intuition should run.
50:54 Speaker 00 Yeah, two important things there from my perspective. One is that this is qualitatively an unusual situation. All the effort that is going into making the models agentic or capable of mid-range planning and overcoming barriers, finding solutions to the first problems that they encounter. This is definitely a horse of a different color compared to normal software. We're fully on the verge of really entering that moment. GPT four can't do much on its own. It falls down at the kind of friction point most of the time. And
51:33 Speaker 00 Yet, as Zuckerberg recently said in his Dorkish interview, like you observe these weaknesses, you have this process of building a lot of scaffolding to try to overcome those weaknesses, but you basically have a TikTok development cycle where you build the scaffolding and you also try to build the next generation of the models. You don't have to have all that scaffolding. And so it seems pretty clear that the next generation coming out of OpenAI, and this has been broadly reported on, is going to be a lot more agentic. And I basically define that as saying, if you give the model short instructions, it can unpack those into a plan and then go pursue that plan until it either succeeds or it's trying, or somebody comes and turns it off, right?
52:15 Speaker 00 But there is a qualitative difference to the sorts of things that we're starting to get into now. That's also measurable, just in terms of like how close these things are to expert performance in a lot of different domains. They're already better than the average human. They're closing in on human expert. And so there's not that much farther to go. Either progress would have to like totally stall out, which definitely doesn't seem like it's a bet anybody really wants to make, either commercially or safety wise, or something else really weird would have to happen for us not to be entering into very unprecedented territory. The other big thing is, key point is, the leading developers basically are doing these things, right? If we consider the live players today to be OpenAI, Google and Anthropic,
52:59 Speaker 00 those would be like definitely my top three global model developers. Anthropic has led with their responsible scaling policy. OpenAI had a pretty robust, I would say could have been better, but nevertheless, non-trivial effort in GPT-4 safety a year and a half ago. Clearly, they're bringing way more energy to that now with the next generation. And Google, if anything, gets criticized for being too slow and conservative. So it seems like all of the leading companies are basically already doing something like this. So then you would say, okay, who is this bill really for? And it seems like it's for Meta, maybe, slash like the next half dozen companies that might say, hey, we do want to participate in this multi-hundred-million-dollar training scale.
53:46 Speaker 00 Game and it's not that many companies that could plausibly do that. And and expect to get anywhere close to the state of the art, but there are a decent number. So it seems like it's really for them. And maybe also for the decisions that they might make to open source. If they can't, which again comes really right back to meta. The decisions that they might make to open source if they can't make certain assurances about what might happen with light modification downstream. Fair analysis?
54:18 Speaker 02 I think that's fair. One thing that has been the subject of a lot of conversation online and I just want to set the record straight is that there's been a lot of folks saying you're trying to make it so that no entrance can compete with OpenAI and Anthropic and Google. And that this bill is basically like secretly authored by them. They're the ones doing it. That is just not true. The bill also has like provisions around like whistle law protection, but different things they have to do that they're not have that want to give them props and recognition. So they said that like, I do think that these last half folks have taken these issues very seriously. And I put out these practices that like, we think are good and different in kind of experts with no financial association with them. That kind of Zen and Yashra Benjio and and others like are saying are good.
55:00 Speaker 02 And we also kind of, we have one in in view, which like one of the upstart meeting with them like to work the bill and do it all. I don't think like that crazy of a super radical thing. I think these are things that are doable. And what we're just saying is you're doing this now, but as the rates intensifies, don't stop doing it. So important and also new entrants come in and think that you want to not take these pretty basic common sense precautions. That's not okay. But that's some of what we're getting at. And I think it's fair to recognize that I think we really are trying to look at what different companies and folks are doing that seem smart and appropriate. And saying that, look, you've made voluntary commitments to the White House by doing this and other things like that. But we don't think that these voluntary commitments, we don't think these can be then you just decide you're out going to do. If your investors are really pushing you, like we made clear that we want you to do.
55:46 Dean Ball? I think that's interesting because this bill doesn't have any this associated with it other than sign a document that allows you to be sued. Like it doesn't, there is no this in this bill. There are no, because we have no standards for what companies should be doing. If we had those standards for either red teaming or alignment standards, safety standards, whatever you want to say, I would be so much more in favor of laws that say, for example, if you want to commercialize an AI model in XYZ industry, then you have to refer to these standards to do that. To get to that point, which would actually be like, standards like that would actually be
56:31 Dean Ball? moving the epistemic ball forward as opposed to saying, you guys figure it all out. And if we don't like what you do, we're going to sue you or potentially bring perjury charges against you. I know the perjury thing is complicated, but certainly there's a civil enforcement aspect of this bill. It's very European. That's very much how like EU regulation works where the EU says, do XYZ, you figure out how to do it. And then if we don't like what you do, we'll come after you and we'll like, we'll let you know we don't like your compliance by suing you or fining you, which again, the fine structure here, how big would the fines from the frontier model division be? We have no idea and they can set them to be whatever they want.
57:12 Dean Ball? So that's not great from a political economy perspective, but I think that we're getting the order of operations for good AI policy or AI regulation entirely backwards with this bill. I think that what we need to make good laws are good standards. And what we need to make good standards necessitates active scientific inquiry into AI. And active scientific inquiry is best supported by open source investing, by open source software, by open investigation in a decentralized way by many different people. And this bill makes all of that much harder.
57:53 Dean Ball? So I think that what we instead need to do is take a process of science to build standards to then create laws. And we're doing it in the opposite direction right now.
58:03 Speaker 00 Yeah, I guess my immediate reaction to that is that sounds almost like the case for a pause. If you feel like we don't know what the standards even should be.
58:14 Dean Ball? I don't make the case for things that won't happen. So I don't, I don't make the case for things that aren't possible. So no, it is not a case for a pause. It is a case for scientific investigation.
58:24 Speaker 02 I agree with you. I would love to live in a world where we could sit down for five years and figure out what these rules and standards should be. And then write those clearly into law. I just think that if we sit down and do that process for five years, we might have horrible bio weapon attacks and critical infrastructure attacks and like crazy agentic models running around the internet committing law crime.
58:45 Dean Ball? But in what world is that actually true? If agentic AI models are running around the internet committing crime. Society adapts, right? There's this model that I think a lot of people that are fixated on AI safety have where society is static. And like things happen to society and society is this inert object. But that's not true. Like you're living in the symphony. You're part of the orchestra and it's happening. It's unfolding. Our response is unfolding all the time. And so I, I think that like society will respond. The bio risk thing is the AI models are not the enabler to do to, to bio risk.
59:24 Dean Ball? There's bio risk potential on the table right now, major bio risk and CBRN risk that exists regardless of AI. There are other bottlenecks to that, that we need to be more proactive on policing, not just in America, but across the world. And we are, right? That's part of what the Biden administration executive order did that I very much applaud. We are doing those things. So it's not that we would do nothing and that society would be inert and that then we would scale. We would pursue the scientific investigation while doing many other things to ensure downstream security and resilience.
1:00:02 Speaker 02 I really love that you're taking seriously the idea around safeguarding society from some of the bio risk. I feel like there's some flawiness in some proponents of, oh, if you just have the open first models, you can defend the bio weapon with your own bio weapon. And like the way to defend it is more like actually having RUBC and gene centric just printing and like effective vaccines and acting platforms. And I agree, like, wish we lived in a world where maybe we just all stopped debating this and just all worked on trying to safeguard the world against horrible bio. I think that is very important and, and good. That still enough. I agree. Society in general is pretty good at adapting. I think the issue is that this rate of change is so incredibly fast. It's just like so much faster than the previous levels where we have had at a taken.
1:00:44 Speaker 02 And when I'm in Sacramento, when I'm like talking staffers and legislators from rural California, like they are just terrified. And I'm not necessarily that often think about the ability of society to effectively adapt to the just letting it out at an optimum speed. And I agree that something's gonna happen no matter what, and to some degree that you pay to that of the two. But I do think the US and California in particular is really leading the industry right now. We do have these export controls on China and just kind of their own problems with their tech industry. That means that I think if they don't have access to cutting out joke and first models, it's unfair how much that they can actually be competing at this frontier.
1:01:22 Speaker 02 I think we also need to be adapting and putting in societal adaptations and realizing that stuff can also slip through the cracks. But I also don't think we can just rely on adaptation on its own, given just how incredibly fast it's happening relative to other societal adaptations.
1:01:34 Dean Ball? Yeah, I agree. There's definitely policy steps. I just think models are too low in the stack for a variety of different, both practical and principled reasons. But it's a combination of policy and societal responses. I will also just say that there's, I hate when people use IQ to apply to language models, but there is like a 130 IQ language model out there who's going around talking about its metacognitive properties quite freely. And it's called Claude three opus. And I don't know, the that happened and the world, the republic has stood, has it not? Nobody seems to care outside of me and four other people.
1:02:15 Dean Ball? No one really seems to care about it. I wrote about it the day it came out and I was like, oh my God, look at this thing. But nobody cares. And it seems like it's been fine. I'm not saying that that's great and there should be much, much, much more of that. But also, one of the things that AI safety advocates love to, to say is that their regulations only apply to people that are very large companies spending lots of money. That's true today. Obviously, that's just not true in five years. If the thresholds stay the same, everyone knows that. And I've always felt like it's a disingenuous thing when anybody says this only applies to big companies because we all know that's not true. In the long term, that's not true. But It is also the case that currently it does only apply to big companies that are spending
1:02:57 Dean Ball? a lot of money and they're big companies that have shareholders and they're spending a lot of money. And so there are certain, they have strong incentives to investigate the safety aspects of this and to not release things that do awful stuff. And maybe, yes, do they maybe do it by accident? Possible? I doubt it. I doubt, not in an irreversible way. I think these companies are very capable. I really admire all of these companies for those things. I think it's great that we're leading, right? I think it's great that American firms, people that are embedded in an intellectual community that has worried about these issues for a long time, really worried about these issues before the technology was real, right? That's great.
1:03:42 Dean Ball? Sometimes I worry though, that because these concerns were initially born before the technology was real, they're maybe a little divorced from the technology in some sort of fundamentally irreversible way. There's, there's like a genetic, there's like a genetic problem with the concerns that people have that just is like, we had all these concerns before deep learning was a thing. We were not anticipating a statistical approximation of the internet as being the thing. We were anticipating like a Bayesian pure reasoner that could infer special, that could infer general relativity from two frames of an apple falling from a tree, right? Without having seen any other part of the world, right? That was like the original vision.
1:04:25 Dean Ball? This is quite different and I just think that's generally important to keep in mind. We're not on the path that L A Zer thought we were.
1:04:33 Speaker 02 Yes, sir. One thing I will say is there's been some movement in the opposite direction where you had folks like Jeff Hiddington who weren't initially particularly worried, who then when working with the latest generation of models were like, holy crap, you'll have more than I thought it was. And it seems like if I thin out the last five years of progress five years forward, I am pretty surprised about that we're the best. I do think we need to make sure that we're continuing to think about like in the realities of these models and not in the Higher level ideas about how intelligence might work. And I think that's why I think all of us, I think, for our credit, I think, for acknowledging that tremendous amount of uncertainty here. I think it's reasonable. I'm glad that we agree on that.
1:05:10 Speaker 00 Can I ask what you have heard from the leading developers? They've been strikingly quiet in the public about this. I haven't seen any official statements from OpenAI or Google or Anthropic on this bill. I wonder, like, why that is? What what do you think their attitudes are? Would you characterize industry as like working against this behind the scenes or saying, hey, this is okay with us? We're not like going to be vocal supporters of it for whatever reason. I do want to talk a little bit about the whistleblower thing because I do think one of the weirdnesses of the current moment is that this likely society changing and potentially society devastating
1:05:50 Speaker 00 technology is being developed in a way where it seems like not even everybody at the labs has visibility into it anymore, which is weird. But I guess for starters, how would you characterize what industry is saying or doing behind the scenes on this?
1:06:04 Speaker 02 Yeah, what I'll say is that the Senator's office, with the co sponsors included, have had conversations with OpenAI and Anthropic and Google and Meta and Microsoft. And I think they have a variety of different views. I think there are aspects of the bill that they want changed. I think there are aspects of the bill that they do to you and say, hey, these are things we're already doing. That seems okay. This doesn't necessarily seem that crazy. We have seen come out in public opposition, places like the Chamber of Progress, TechNet, GAI, which are tech associations, which have Microsoft and Meta and Google as members. And so we've seen some of the companies like speak through these tech associations.
1:06:50 Speaker 02 And that's sometimes how it goes, where in general, the companies are happy to let their like practice speak through them. Though I don't want to necessarily pay with a super broad breast because I think there is just like actually a lot of heterogeneity, even among companies that you might anticipate would have like similar interests here. The thing I can like very much say is like, They did not write this bill. They're not pushing like Scott to do that. That is just 100% false. There's been talk about that and it's just wrong. Scott Wiener got freaked out about AI risk and then came to sponsor saying, can I do an AI risk bill? It was not like anthropic approach. Didn't you do a secret thing? Anyway.
1:07:24 Speaker 00 Can you share anything more about what industry wants changed?
1:07:29 Speaker 02 Yeah, I think they are wanting to be clear about some of the specificity of the language and saying, like, we understand your intent of what you're trying to cover, but are you being sufficiently precise about that? This is something that, like, we're confident we can, we can follow and understand what it is. There's been like some like micro debate about do the KYSC provisions violate the Storage Communications Act. There's some like fairly big provisions, but their view in general has been that we're not going to or or pose publicly, but that we want to weigh in and and provide technical information. And some of the amendments for the time, is this the technical way that's talked about and is that accurate? I also want to be careful to the fact that, like, the center's team is leading negotiations, they're ongoing. And again, I want to push back
1:08:12 Speaker 02 very strongly on the idea that the motivations for this bill are to lock anything coming. That is just like 100% false. Then I can't emphasize enough the degree to which it's false. I think ultimately a company do have in support because they think these are doable. I think that's great. We would love that. Again, you did that. The other ones did a good. But like, I just, you know, want to work on that as strongly as possible, but it's false.
1:08:32 Dean Ball? Yeah, I don't believe that OpenAI or whoever from the big companies wrote this bill. I think that's true. In terms of the motivation, in terms of the effect, this bill undoubtedly benefits OpenAI, Anthropic and DeepMind at the expense of Meta, whose strategy relies upon openness. That is the explicit strategy of the company and also most of the startup community, which broadly speaking hates this, and a lot of the scientific community and academic community, which also, I know Jeff Hinton and Yashua Benjio exist, but there are other people, I don't want to betray confidences because I didn't get permission from anyone to to say names, but there are very prominent researchers who are just as respected as Hinton and Benjio,
1:09:15 Dean Ball? who don't like this. So, you know, the startup community has a lot of problems with it. Scientific community has a lot of problems with it. Those are important. And no, I definitely don't think that the regulatory capture is the explicit aim of the bill. It is, however,
1:09:30 Dean Ball? Very likely to be the effect. And that's how we should judge things, not by the motivations, but by the effect. California is repeat with legislation, which should be judged by its effect and not by its motivations.
1:09:46 Speaker 00 Seems like a lot of the discussion has focused around this is going to make open source illegal, which is obviously a bit of a hyperbole, but does maybe have a kernel of truth to it. The main thing that I see is this idea of you release a model and what people do downstream with it is your responsibility. That makes sense to me in as much as I think if you have trained this monstrous Shoggoth Lama four or whatever, and it's just got a thin veneer that's like causing it to refuse to make your bioweapons when asked through the meta search bar. But with like fine tuning, that capability can be exposed again.
1:10:32 Speaker 00 That's going to be a big problem if something like that does get released. I presume everybody would agree on that.
1:10:37 Speaker 02 I don't think everyone agrees on that.
1:10:39 Speaker 00 I don't know if you can object to that, Dean, but it's, let's say we have a, we train Lama four. We do not have a theoretical breakthrough. We can't make a good argument for why this is not going to be a problem. The pre-trained version, we can even say, hey, look, okay. Um, it'll do these various things, right? It'll find J zero exploits, it'll design bioweapons, it will maybe break itself off of its server. Now we RLHF it so that it refuses to do that. But we know from plenty of research that capability is still in there and can be re-exposed with downstream fine tuning. That to me seems like it would be a big problem if that's released. Tell me if you think that's either not a problem or.
1:11:20 Dean Ball? I would say the biological weapons thing, it's not an intelligible concept to say that Llama 4 can design biological weapons because the design and software aspect of the bio risk thing is an important part of the equation, but there are other parts of the value chain, as I've already, or production chain, as I've already talked about. So there's other things you can do about that. In a world where Llama 4 or whatever can identify Euro Day exploits and have that be fine tuned out of it or otherwise removed by downstream open source users, I would, I think that the equilibrium we have reached right now is actually a quite
1:12:02 Dean Ball? positive one, where the frontier of capabilities are generally reached by the closed source companies who take these things very seriously and they test it. And Microsoft owns GitHub, and if GPT-5 and GitHub haven't already met each other, I suspect they will in the future, probably before the public knows about it. I think that there is definitely a lot of responsibility that Meta would be taking. And if it's not a responsible decision for them to open source it, I guess it's, you're asking me what I would want to happen. It's not my decision. It's ultimately, part of this whole thing is that I don't think it should be any single person's decision. I think the world is too complicated to make decisions for people on things like that. But yeah, I would certainly hope that Zuckerberg and
1:12:44 Dean Ball? Yann Le Cun and others involved in that decision would think quite seriously about that and at least want to robustify society much more, our digital infrastructure in particular. And I think steps are already being taken along those lines, by the way. But yeah, would I want that to be released tomorrow?
1:13:03 Dean Ball? No, I wouldn't if it were my choice. And I think that most people don't, and that probably includes capital allocators and people who are responsible to shareholders and things like that. So I think they probably won't. And if such a thing is even possible, which we don't know, I think a problem gets introduced when you legislate things, because legislation is different from conduct.
1:13:33 Dean Ball? When you legislate things, you lock certain dynamics into place and create sort of a freezing in amber of industry and industry dynamics. And that's just problematic for all kinds of downstream reasons that are hard to predict. Yeah, I guess, no, I wouldn't want it to be released and no, I don't suspect it would be.
1:13:52 Speaker 00 Yeah, I have a reasonable amount of trust in Netto leadership these days. I think Zuckerberg has had some notable comments recently suggesting that he's not purely ideological about this and is not not going to YOLO an open source model if he thinks it's like genuinely not safe to put into the public. I wonder if there are things that we could do to allay some of these concerns. And on the open sourcing one, I do think most everybody thinks that open source is good unless there's these crazy capabilities that are getting inadvertently exposed. So one proposal that I was noodling on myself and then got a more concrete version of it from Z and a blog post that will
1:14:33 Speaker 00 be published by the time this is released. So you can go check it out on Z's blog is to create a little bit sharper definition on what counts as a derivative model versus what counts as a non derivative model. The idea right now is that if you open source something and somebody makes a derivative of it, it's still on you. If they fine tune and reexpose some latent capability that you masked, that doesn't get you off the hook as the foundation model developer. But there's not like a limit to that. And you could imagine a scenario where somebody does 10% as much additional training as the original model had, or 25% or 50%. And at some point it's like, okay, that's no longer just reexposing some latent capability, but you're actually using
1:15:17 Speaker 00 this base to build something that is fundamentally new and qualitatively different and maybe should be considered non derivative. So one idea I might advance would just be maybe we have some sort of reined in notion of how much downstream modification is
1:15:35 Speaker 00 Fill the responsibility of your original foundation model developer versus at what point it becomes like, okay, now it's really I'm the new developer. You put in your own tens of millions of dollars of training and now that's on you. Another idea would be maybe some sort of sunset to this bill would be a good idea. I think everybody agrees that we're in a regime of fast change. We would be very surprised if we don't know a lot more in even two years relative to where we are today. So again, I don't know why this has never seems to be done, but one could imagine a, hey, for the next 36 months, this is what you have to do. And at the end of that, this bill will sunset. Obviously, it could be extended,
1:16:16 Speaker 00 but ideally it would be like updated and improved upon and it could maybe bake some sort of forcing function into the legislation now to try to create something like that in the future. Any thoughts on my, my fixes?
1:16:31 Speaker 02 I think these are thoughtful suggestions that I think author really appreciates. And again, I think we agree on more than I think you may think. But I think Llama 3 being open source was good. I'm happy about it. I think it's great. I think it's gonna be able to do lots of awesome things. And I think there will be harms, but those are best addressed at the application developer and at other stages other than the developer. I do think that there is some level of capability where just going it out is not a great idea. And Mark Zuckerberg seems to be acknowledging that. I do worry that if we're in a regime where we're like relying on the will of Mark Zuckerberg, avoid catastrophe. And a lot of people are uncomfortable with that. And I think that possible there will be other people who maybe do think it's great machine takeover and do want to just blow it out
1:17:13 Speaker 02 at some point. And I think that is a rough place to be if that's what we're taking everything on. I definitely, I think it's fair to distinguish between a situation where you are fine tuning, remove some guardrails relatively versus like building your, your own model with 10 or 50% of the training costs. And again, I think it's definitely fair to consider amendment that clarified out. I think that seems reasonable. On the sunset, too, like there are things within the bill that try to make it future proof. All these references to like reasonable and myths and all of these best practices that will continue to evolve over time. But this is. Us trying to reckon with the urgent nature of the problem and wanting to put up what is a thoughtful and educated
1:17:57 Speaker 02 push at that, but I definitely don't preste this if that should does task of it being the end all be all of there being no later revisions or subsequent that you needed. I think that's super fair. Again, I think that open through great, and I am I genuinely think that there are like a ton of benefits and like in the vast majority of cases, the benefit that way, the cost similar to how I work pretty much like the biosecurity issues. And when you're working around like any restrictions on gain of function research, a group that was opposing any restrictions on gain of function research was called like scientists for science. And basically saying, if you don't, I can't function research, therefore you're anti-science, you hate all of science. And the fact that I don't want there to be models that can shut down wastewater plants or design bioweapons or do things like that.
1:18:37 Speaker 02 You I hate it, but I love open source. I just think there's a very small category of very common things where it can be often. And the opportunity for societal adaptation are not there, where I think that we should be more cautious. And honestly, not that different from where Mark Zuckerberg is at.
1:18:53 Speaker 02 I don't think it can just be them entirely making those calls on themselves and what mood Mark Zuckerberg wakes up on a particular day. If you just feel like maybe Elon put out a big model and I really want to beat him today, they'll look for it. I I don't know. That's my thought there.
1:19:08 Dean Ball? I'm not sure if we read different newspapers or something, but I'm not really sure I I see anybody YOLOing right now. This isn't like the least YOLO technology in the history of technology. It is such a profoundly unfun conversation. The unfunniness and seriousness and fake everything about this conversation uh is a form of regulation, I would argue in some ways. But I guess I just don't really see where the person is who says, I I don't give a crap about. Humanity, I guess I fail to understand who is the person that like isn't thinking about
1:19:49 Dean Ball? these things that doesn't care about the dangers of taking down critical infrastructure or wiping out humanity or whatever it might be, but who does care about the California frontier model division? I guess I'm like failing to understand what part of the Venn diagram do we think we're really moving the needle on here, other than just maybe creating new forms of liability and new regulatory risks for the legitimate players? That's what I guess I fail to see. And by the way, my comments should not have all been taken to suggest that we're relying on Mark Zuckerberg's whims. We're relying on a lot of people's whims, right? That's the nature of the world. I can't help with that. That's just the reality.
1:20:30 Dean Ball? Some people have more power than we do, but, but at the end of the day, what I'm relying upon is a system of incentives that tends to work. And by the way, which regulation often tends to break in weird, like extremely high dimensional ways that are like very hard to diagnose or to foresee. And I think we all know that about regulation. Everyone's seen that. So it's not like what I'm saying is empirically unobserved. Everyone's seen that.
1:21:01 Speaker 02 I feel like sometimes there's an idea that currently California has no rules at all about AI and then this is stepping into the void and creating those. There's lots of existing liability law. There's lots of actually existing uncertainty about like a situation where you did release a model and then someone modifies it, does something bad with it, whether you might have liability. I think to some degree, these are already questions and uncertainties to exist. I view this as actually clarifying what those are and some degree, given company more confidence in what they're doing, that aspires their burden. I think there's actually been some point back, there's no formal safe harbor in this legislation. Like we have a status clause that makes clear that like all other remedies and rights and actions that are laws still exist, but I think there is some more confidence that comes with companies about what do I do if I want to prevent these really bad things from happening?
1:21:46 Speaker 02 We're saying these are some things to do. These are some good ideas and this is giving some additional clarity to what it means to take reasonable actions as a frontier developer. Again, I agree so far, how people are acting is I think very in line with the bill and with what I would view as like responsible development. I guess I'm just worried is that I don't know if that necessarily extrapolates in the future. There are people who. Like the Beth Kajos's and Richard Sutton's of the world, who I do think have very different worldviews for myself or the vast majority of people. And I agree, there are going to be some group of folks that are not going to care about what the law says at all, but maybe they're going to have investors who just think that they're making a bunch of money and they they're like, oh, I want the machine to take over.
1:22:27 Speaker 02 And investors might not care about that ideological thing if they don't believe that, but maybe they do care if you're violating the law. And so I do think there are the people for whom this does have force. And I also do think, again, there's the question of what some of these developers think. Upon reflected in their better nature, it's in everyone's interest versus what is happening when there is incredibly intense competition and ego. And I just think we use shape their incentive and like having some pushback against that drive, I think is super important and necessary.
1:22:57 Dean Ball? But also not killing the entire industry and running their own company out of business shapes their incentive. If OpenAI releases a model that brings down the internet tomorrow, I don't envy Sam Altman's job, but it sounds incredibly hard. Uh, the pressures are there, right? And certainly there's internal stakeholders within OpenAI. Uh, that are very worried about these things and that should be the case. My point is that safety is a very hard thing to legislate. It's integrated into a lot of different economic activity at the frontier of technology. It's incentivized by a lot of different things. I think everything you're saying is fair. I just think that the dynamic we already have is a perfectly fine one.
1:23:37 Dean Ball? The first thing I looked at this bill and said was if someone uses an AI model to do a hazardous capability of the kinds described here. You're gonna get sued like for sure a thousand different ways. So like, what is the marginal utility of this bill? Other than creating again, a regulator whose objective is to ensure safety, but which does not actually do any safety work themselves. And which does not build AI systems. That is a regulator whose incentive is to cover their ass. That's the public choice reality of this. Uh, and so you're creating like what will ultimately, I suspect, be a quite pernicious.
1:24:20 Dean Ball? Addition to the AI industry dynamic with the frontier model division.
1:24:24 Speaker 02 So.
1:24:27 Speaker 00 One idea a simple person like myself might advance would be to say, why don't we just make this a lot simpler and just make certain things illegal? Like, what if we said it is illegal to certainly release or open source an AI model that can do X? And just have some concrete Xs and say, you just can't do that. Boom, simple. Now it's on you to make some, if you want to open source something, you got to make sure that you comply with that standard of what it can and can't do. And there you have it. Now, it's tough because these are general purpose technologies and there's a lot going on there. But if we had a pretty high threshold for the sorts of things that we are concerned with,
1:25:12 Speaker 00 then it seems like a lot of people would be like, okay, that reduces a lot of ambiguity. I don't have to worry about whether I fit into this or not. I just have to be very careful about 10 different things, maybe even fewer than that. And at least it's clear what I have to do, right? What would be the downside, maybe you would be more inclined to support that, but what would be the downside of a sort of simplification, clarification and just the following types of AIs are illegal, boom?
1:25:38 Speaker 02 Yeah, I think definitely room to make bill fair in part. And you and others have given some good suggestions that I think us in the office are going to think hard about. I think it's hard when you're trying to enumerate those and also recognize that you can't be 100% sure. Like we spent some time trying to think about like definitions of deceptive or self replicating systems or different. Very hard to define in statute. And I think describing more in terms of the severe consequences that we're trying to prevent, we thought was clearer. It's easier to describe the process of you should do tests for these types of things. And you should have some level of reasonable confidence. I hear you and I think open to specific language that tries to make that clear. I doubt that addresses the incurrence we should say, but that's my initial thought.
1:26:24 Dean Ball? The bill you've described, Nathan, I'm not sure if it would be like a single piece of legislation, but that is the legal outcome that I would most prefer. The problem with it is that we don't actually have the knowledge to legislate an outcome like that at this time because of the lack of good technical standards and evaluations and safety best practices and all that kind of stuff. And that stuff simply takes time to build, and I think we should invest so much into building those things as quickly as we can. I think that this bill hinders our efforts to do that, not helps. It does take time, and I mentioned that earlier, that it might take five years to do that. And Nathan Calvin, you said, "We don't have five years."
1:27:09 Dean Ball? And in my mind, nobody has made a compelling case. That is true. The only case that I hear on that is pointing at charts that don't tell me anything concretely, calls to authority, et cetera. We don't know how much time we have. And so I don't think, just, I, like, this is a general, it's not a critique I'm making of you in this conversation, I just want to be clear, you've been like, awesome, but it is a critique I have of the AI safety discourse in general. I'm not, this is a societal negotiation. The AI safety people don't just get what you want because you were here first. This is a societal negotiation, and I don't personally like to negotiate with a gun to my head. And that's oftentimes how it feels.
1:27:50 Dean Ball? When Mustafa Suleyman, last summer, goes on a podcast saying, "We're all gonna die if we don't create a brand-new regulatory regime from scratch tomorrow." That isn't true. And we're giving up a lot. We're potentially setting up radical precedent here. By the way, that is an important part of this. It's not necessarily specifically what this, I mean, it is specifically what this bill does that I don't like, but it's also the precedent that it sets about government pre-approval of software and killing that whole system that's been quite valuable. We're creating, and no one's talked about the enforcement infrastructure for this bill, also a major thing, especially as models of this level of power proliferate. Good luck enforcing it without creating a policing regime over the internet, California
1:28:34 Dean Ball? state government, which is a long-term outcome that, that might be inevitable either way, but. My only point is we're sacrificing a lot and we're setting a pretty radical precedent by doing this and we need really compelling evidence for that. And it just, it could happen because of exponentials is so unbelievably far from good enough in my mind.
1:28:56 Speaker 02 Yeah, I think we are honestly in similar places on the epistemic. Like when I say five years, you know, I put money in my retirement account. I do plan beyond five years. I'm not somebody who says definitely in five years it's going to be this. But when I look at these graphs, I have uncertainty. They could turn off at any time. We could have another air winter. We could run into a block. All that could totally happen. I've talked to a lot of smart people who have different opinions about this. There's, I think, healthy uncertainty there and there's a question of what to do in face of that tremendous uncertainty. I hear you saying that you think even if those risks do emerge that this will make us in a less good position to handle them. But I do think that they are at a high enough level to be worth taking seriously in one direction or the other.
1:29:36 Speaker 02 The other thing is I 100% agree with you though. Just because a bunch of people on Les Ron thought about this 15 years ago, doesn't mean they get to decide what happens. I 100% agree with that. I also don't think, Kate, that just because Mark Zuckerberg buys X thousand numbers of 8.100s, that he gets to decide what the rest of the world looks like and there's no opportunity for democratic input or things in that either. And so I think that's what is happening now. And we're going through this multi-month, tons of open hearing, lots of discussion on Twitter, talking to dozens and dozens of people. And I think it's fair to be pessimistic about government and lots of respect. And there are things with good intentions that go poorly. But I think this process of trying to figure out what should this should look like and taking seriously a variety of views from people inside the AI community and people who are terrified who are outside
1:30:19 Speaker 02 the AI community, I think that is appropriate. It's the way we have to change things. And I prefer that to like just saying that these leaders are going to make their calls and I hope they woke up on the right side of the bed this morning.
1:30:30 Dean Ball? To a certain extent, I actually would dispute the notion that we make decisions about the trajectory of technologies or industries based on democratic input solicited from like anyone, right? I don't think I have a vote in that. I have a vote as a consumer in the marketplace. But that's not something I exercise through my government. And I think that government thinking too much, that it's Is responsible for setting the trajectory of technology or the destiny of mankind. Really, anything even remotely adjacent to that gets you into all kinds of dangerous problems. It's a bad attitude for policymakers to have. I'm not sure that things in California are going so overwhelmingly great.
1:31:12 Dean Ball? That Scott Wiener has a lot of time to be working on the destiny of mankind. I just, I think that we make these decisions in more diffuse and complicated ways. And we don't do it through the democratic process and history doesn't unfold by show of hands. Gabriel O'Prinship did not ask everybody in Europe whether he should kill Archduke Ferdinand. He just did it. Sometimes that's the way it goes. Sometimes nonlinear things happen as the results of one person. History doesn't unfold by show of hands. Uh, and so, no, I don't think our fate is in Mark Zuckerberg's hands. I don't think our fate is in any one person's hands and I want to keep it that way. And actually, my opposition to something like the frontier model division existing
1:31:57 Dean Ball? is partially connected to that instinct I have of not wanting to put all my eggs in a small number of baskets.
1:32:05 Speaker 00 Let me ask another kind of hypothetical question. It seems like
1:32:11 Speaker 00 there was openness to this notion of if we could effectively enumerate a bunch of things that would be really bad for AI models to do. Then we could all potentially get on board with a law saying you can't make an AI that can do any of these bad thing. We have definitional challenges there. One thing that I see missing from the bill, or it's not entirely missing, but it's very limitedly addressed, is independent testing of the models. I would love to see a little bit more independent access to models during the training process. In particular, there's a lot of debate in the literature and community as to what should be considered an emergent property?
1:32:55 Speaker 00 And how fast do these emergent properties? Pop up in the training process, it seems like in general, there is time. That last order of magnitude is always the biggest order of magnitude, of course, and like, it seems to take an order of magnitude for a lot of these things to come online. To the best of my ability to synthesize the literature. So what about a scenario where we say it's illegal to distribute an AI that can do these things distribute could be via API or could be open source, but it's illegal to distribute an AI that can do any of these things. And we will determine whether the AI can do those things by giving access to a bunch of third party testers who don't necessarily get the weights, don't necessarily get to know all your trade secrets, don't have a
1:33:38 Speaker 00 sense for what the parameters are or whatever, but can just hammer at the thing for a while and satisfy themselves that either they are able to get a certain behavior out of it or finally give up. I know a lot of people who would be motivated to do that. I have a little or all volunteer project red teaming publicly deployed applications. And I could tell you, any of those people would be very excited to get the opportunity to go do some of this work on stuff that's not yet released. There's only one line in the bill that mentions third party audits as appropriate. At least that I was able to find. So quickly, I think there's more in there, but I'm reminded too, we had Paige Bailey from Google DeepMind on the podcast some months ago.
1:34:22 Speaker 00 And she was talking about the experience of doing these training processes at Google, and she described it as like coming in the morning and oh my God, we just hit new state of the art over here or this thing that we didn't expect to be this good for a while. Yeah. Suddenly gropped something and it's actually hitting these thresholds sooner than we expected. And I took away from that, that there is not enough internal during training testing happening to really have a sense for what is going on. Open AI is not that big, right? And a thousand people ish. And like half of those are, I think on the non-technical staff side at this point, so we're not really able to put enough actual human cycles into trying to figure out. And meanwhile, you've got Pliny online who's jailbreaking everything.
1:35:07 Speaker 00 I'm pretty confident with current techniques, like if it's in there, he'll pull it out. He's got the lead speak to, to figure out how to, to really talk to the models in the way that they want to be talked to. So.
1:35:18 Speaker 00 Trying to square the circle here by both giving, you know, the best standard that we could have would be to throw a motivated community at it, I think, and try to elicit problematic things. And then just say, hey, if we hit the crowd on it and we can't find them, then we guess they're probably not there. That doesn't give us a guarantee, but it would give us much better sense, I think. So what do you think? You can't do these things. You have to allow testing. If the testers can show that the models can do these things, you can't distribute them. Nathan's SB 1048.
1:35:52 Speaker 02 Yeah, I am also very enthusiastic about third-party testing. And one of the things I was a co-presenter and the author have thought about, we just want to make sure that the incentives are set up right about who those third-party testers are going to be and what that will look like. I love the idea of like finding on Twitter, going in and messing around with people's models and trying to break them. I think realistically, you're probably going to get like Deloitte, like going through a checklist. And I think there's some concern that like companies, or if they just hire Deloitte or who's Allen Hamilton and they give them a check or they're like, okay, I'm good. Let's do it now and not do that more like deeper bulk that searching and sense of responsibility. And so I think that there's something that not wanting to roll that out.
1:36:33 Speaker 02 Feel like companies can just push the responsibility onto those folks before we're confident that they will actually do a good job of it. You do have some language in the road, Bill, around the addition of being able to like establish a accreditation process for third-party testers to do them could be credited by the FNG, be able to do tests. But I think requiring them to then use those specific testers, I just think we want a little bit more information about like, who counts as a third-party tester? How are you going to enumerate the provisions by which they're doing it? You're going to rely on accreditation process? How is that accreditation process judged and not opportunity for capture weird things? Yeah, I'm enthusiastic about it. I'm very open to the idea of there being like a role for greater amounts of third-party testing in the bill.
1:37:14 Speaker 02 And I agree, very important part of it. And some of our whistleblower protections also do apply to folks that labs are bringing in to do testing. And I think we definitely view a important role for them in this. But just like, the attitude, I think, that legislators feel for this is responsible, but very humble about their abilities. I think that they should feel like they have a role and important part of those systems to protect California residents. But then I think they should be pretty thoughtful about what to what degree government is going to be able to go to toe to toe in an effective manner with the company. That is what we had tried to build into the bill, saying that like, let's look at what the companies themselves are doing, what they think they're currently publicly saying they're doing.
1:37:56 Speaker 02 Let's say, no, you actually have to do it. And I think that has been trying to take that approach of not having like a centralized regime that's trying to do pre-approval of model releases, which is not anywhere in the bill, and trying to go in and like how the government of student testing themselves, because like, they don't know how to do it. Like, we're a very far place of like, government being able to pay the seven figures salaries of folks at these firms, be able to stand so and so with them. I a hundred percent agree with you that there have been lots of problems and lots of legislation about the government, you know, like, we're gonna take the world into that government office and have that go better. That's part of how we've gotten stuff like TQL, which Cowenier has spent a decade fighting. And I'm very happy to end it fighting. I think the last thing we would ever want to do is create an AI TQL. I think that would be a disaster. And I just don't think that's what this is.
1:38:39 Speaker 02 So, anyway, I think it seems like part of your concern is not necessarily that like, this itself is doing those things, but that it will be grown over time to create something more like centralized AI belief in the future, rather than that like, the specific language and it actually does that today. Anyway, this is a side for Cardi thing, and I think that the author be quite open to suggestions there. And I agree, there's a super important role for them. It's just tricky to figure out exactly how you want to specify it.
1:39:06 Dean Ball? Yeah. To make some of my contentions clear. Contention number one, I do not believe that if this bill is passed, current open source models become illegal, right? I would never make that claim. There might even be like another generation-ish of open source models that are legal. But they become illegal eventually. They become essentially unlawful over time. And that's how the bill is designed.
1:39:32 Speaker 02 There is some level of capability where I do think that that lest you come up with methods like self-destructing malware, the resistance to tuning, yeah, they have those capabilities.
1:39:41 Dean Ball? It's a legislated gradual death of open source AI. I think that's just basically the case. And it's designed so that it doesn't happen like immediately. The opposite of whatever Sam Altman says, iterative deployment. It's like iterative destruction of open source. I mean, like existing open source models, Llama 3 will always exist, but it's a sort of gradual devolution. But to your point, Nathan, I think third-party evaluation is great. One of the things that I support at the state level and at the federal level too, is actually safe harbor for good faith, third-party, and like truly independent, not like I paid Deloitte, but like truly independent, no financial relationship, research into AI models.
1:40:24 Dean Ball? Right now, a lot of that work violates these companies acceptable use policy. And so I think that fixing that would be a great move. I would note that truly independent, again, not like we paid you, but actually independent research into models is harder without open source, right? Like, that's a, that's a trade-off that you're willing to make that I am less willing to make. I think that's like a fair characterization. I just want to say one other thing, though, to go back to your hypothetical SB 1048, Nathan, which is just that we have two problems with saying XYZ is illegal. First of all, it's what is XYZ? How exactly do you define it? But number two, I actually think the more serious problem is that We don't have a way of actually making that effective.
1:41:09 Dean Ball? Right? Like, you could say right now, if you make an image generation model that is specifically designed to produce child pornography, then that's illegal under current law. But if you say it's illegal to make a DNA foundation model that can't produce a dangerous bacteria, I don't know, can we train that out? What is a dangerous bacteria? Dangerous in what circumstances? How do you model that? How do we know? How would you ever actually legislate that in, in practice? And then on a technical level, in as much as you can legislate that, can you actually remove that specific set of things from the model's capability without damaging anything else? Right now, the answer to that is no. It might always be no.
1:41:52 Dean Ball? Maybe it'll be yes, probably it'll be some complex combination in between. Which is why I think that we need to just accelerate research into all of that kind of stuff. We need to be doing as much as possible. Why I support national compute um, infrastructure. If California wants to do it, that's great. I don't think California has the money. Uh, and I would prefer to see California spend the money on like addressing their overwhelming deficit, uh, their overwhelming budget deficit. Uh, I am a fiscal policy guy at heart, so I look at that and I'm like, I don't know, you'd probably be thinking about that. That's probably the primary threat that your citizens face right now. Probably not so much C burn from AI.
1:42:32 Dean Ball? But yeah, like I'm in favor of national compute infrastructure to give academic researchers all over the country the same, at least a modicum of the kind of resources that that they have in the big labs. I want Lee to have a 10,000 B 100s to do whatever she wants with. Uh, and Stanford's not going to buy that for her. I, I think that the public should in some sense uh, for all researchers. So I, I again, yeah, like I, I think that's what we need to be thinking about. And I think that like this work, this kind of a bill just slows down the scientific research.
1:43:05 Speaker 00 So if I had to sympathize this, it sounds like. General agreement on it would be good to have a discreet list of things that you can't make an AI that can do these things. But for the challenge of actually defining that with clarity. And there's also agreement on third party testing is good. And obviously we don't want it to be a credit bureau type of situation vis a vis the mortgage companies from 15 years ago was example of how that can go wrong.
1:43:38 Speaker 00 But
1:43:40 Speaker 00 that does seem very overcomable to me. I think there are plenty of people now who would volunteer or who would take a grant from a philanthropist to do that sort of work. And really the only barrier to them doing that sort of work right now is access. And so that seems like the kind of thing that legislation could like rather easily address and say, you have to provide some amount of opportunity for testing to happen during flash after the training process, but before deployment. That seems like quite easy to do. Then the other thing I would wonder about transparency on the results of those tests. This is a personal hobby horse for me because in the process of doing the GPT four red teaming,
1:44:25 Speaker 00 I was under an NDA where I was not allowed to visit, talk to anyone about anything related to this entire thing. And they wouldn't tell me anything about their other safety plan, other than the fact that I was in a black channel with a couple dozen people. And not many of them were doing much of anything. So I was like, I feel like I'm like one of a couple of people in the world that is doing this testing. And I got a little freaked out, not so much because of what I was seeing. My conclusion to that at the time was that I did think it was a safe thing to deploy. But just the yikes, if it's just me and like a couple of other people here that are active in this one black channel, like that doesn't seem to be enough. So I went and talked to a few people in the AI safety community to see what they thought about what I was seeing. And when it was discovered that I had done that, then they kicked me out of the program.
1:45:10 Speaker 00 Now, that doesn't seem great for a lot of reasons that I do understand that they viewed me as a bit of a wild card, potential loose cannon, whatever. But this is where, again, I think legislation could really help. One thing I've considered in the past is A special whistleblower provision for observed model behavior. If you saw an AI do something, you should be able to say what you saw the AI do, right? They sort of, what did Ilya see? Now, that could be distinct from what was the technique, right? I think there's still a place for trade secrets. At least I'm not trying to make a case that there should be no trade secret. But if you saw an AI do a certain thing and it freaked you out, I think you should probably be able to tell the public that.
1:45:55 Speaker 00 And I don't really see what the harm would be to the companies that, hey, it did X, right? This thing broke itself out of our test server and got onto our other test server. And I'm freaked out about that. That kind of thing doesn't seem like it should be swept under the rug. So if I was going to take 1048 to 1049, it would be like, you have some things that you definitely can't do. We could make those maybe like fairly narrow and as concrete as possible for now. We might even feel like that leaves something on the table. But in the interest of clarity, try to make it narrow, insist on access for testers to be able to do this sort of thing. And also make it clear that those testers can disclose the things that they see.
1:46:39 Speaker 00 And then if we could do all of that, I think we're headed for a scenario where I would feel a lot better about it. As opposed to, nobody really knows what they're baking. And they'll show it to us when they're good and ready. And who knows what they did in the meantime that didn't get any light of day? And again, I don't need to know how much visibility there is in the labs at this point. There've been some really interesting, somewhat cryptic treats recently from people that have departed OpenAI, where one who's now at Google said, what's really nice about Google's culture is that I have visibility into what's coming. Somebody responded, did you work at another place? Or that wasn't the case? Like, man, what is going on there? So I don't know.
1:47:21 Speaker 00 I'm building my own bill here, but I feel like I'm on the verge also of getting to a point where you guys would both support this. And it feels like it might be pretty good.
1:47:31 Speaker 02 I do think there are areas where you and I have some substantial agreement and I think some of them is around trying to figure out these appropriate transparency measures and I agree there's a problem where there are just far too few people actually investigating the safety of these models. I think one solution to that is just to release the models in an open weight form, which then enables anyone to then tinker and understand that, which I think is again, good and very net beneficial for current generation models. I think there becomes some point in the future where that worries me and where I don't think that is sustainable. But then when we do enter that regime where it is maybe not safe to release in an open weight form certain models, then I do think you need some way of like actually
1:48:13 Speaker 02 allowing other experts and people outside of it, not even just because the companies are like selfish and profit motivated, but because there's not enough bull and not enough like brain there and folks actually evaluating it. I think you need ways to take the wisdom of the crowd in important respects. I think that is an area where there just is a lot of tension among companies of getting access and of disclosing trade secrets and what exactly those things look like. In the bill is written, we have some provisions around the companies reporting I safety incidents like to the FMD and there's some mechanism by which they can then be like made available to the public, but I think fair to say that should be cleaner. We really don't want some of this just stuffed in a drawer somewhere when it really would be
1:48:53 Speaker 02 vital for the broader public and technical community to be aware of. We talked about this briefly, but I just do think it is super important. One area of agreement that I've seen among thoughtful people who are taking both sides of the open source for frontier models debate is thinking seriously that potential that advanced AI models are going to lower the ability for bad actors to make bio weapons. And I think some of the open personal strengths is going to exist anyway and exist today, which I agree that it does. And that would not be effective way of combating it. I feel differently, but I think you can go back and forth. Clearly, there are good faith and thoughtful people who disagree. But I just like really want to say there are people listening to this podcast on either side of the debate. Are you see Jean said this is creating just dealing with the potential
1:49:36 Speaker 02 of the future. We are going to have worse delivered bio incidents. Please, let's all agree on this and do something about this. I worked in Congress in 2022, a bit after COVID and there was just no energy to do anything on pandemic prevention because it was like all like everyone's COVID fatigue. Dude doesn't want to talk about this. These are just like incredibly wise and good investments. Do you think we are pretty likely to see some intentional bad news of biology in the next 10 years that could just be really horrific? And I just think that we could make just incredibly wide investments now for that. And it's good to see that scenario working. I agree.
1:50:07 Dean Ball? Just as a response to your idea, Nathan, I think the whistleblower provisions are great. Safe harbor for researchers is, I think, a really smart move to make. I don't know if it's actually possible. In a world where it's possible to legislate the kind of things you're talking about and actually technically possible to do, to say, oh, you can't make a model that does XYZ, and XYZ is a thing that everyone can agree is bad? Sure. If that exists as a thing, then it will be an example of a technological free lunch. And I'll be very happy for that. I'll be very happy if that's the outcome that we get. I think we also need to be prepared for a world in which it's really not epistemically
1:50:47 Dean Ball? or technically possible to talk coherently about that kind of thing, to say biological foundation model that can't make a bad DNA sequence. I think we have to be ready for that world. And we have to be ready, yeah, for a world that will be radically better in so many ways that we can't understand, and it will also have higher stakes, bigger dangers, more capable actors that are smaller, where intelligence and agility and speed and flexibility matter, matter greatly. Uh, that's gonna be an adaptation for all of us. We're transitioning into that. And I think it will change the nature of statecraft.
1:51:31 Dean Ball? I think it will change the nature of business. I think it will change many things. And it just, I, when I look at bills like this, what I see, frankly, is a rather desperate attempt to, to freeze the present in amber and to not go into that new future. And I don't know about you, but I'm not all that happy with the current status quo in the world. I'm open to changing the status quo in quite radical ways. And the trade off to that is that it might change in some ways that I don't like. So I think that contending with the reality that we have, rather than casting an anchor
1:52:14 Dean Ball? into a seabed that's not actually there, into a seabed that is of our imagination and exists purely in legislative fiction. Is I think that's ultimately going to put us in a lot more danger than simply contending with the reality and building capabilities to be able to thrive in that world. And that latter thing, that building is what I work on primarily every day.
1:52:38 Speaker 00 Sounds like a good closing statement for you. Do you want to give any final high level thoughts, Nathan?
1:52:44 Speaker 02 Sure. I guess what I want to say is that I think the proposition that this bill is standing for is like a lot more intuitive and common sense than I think I've seen it deframed. I think that we're saying that like companies have to test for extremely dangerous and hazardous capabilities. I think we welcome continued iteration on how exactly those are defined and making sure that they're not overbought in what respects. But like, I think we can talk about there being really dangerous thing, air models shutting down, breakwater plans and doing bioweapons and like the people and going around doing crazy thing, like the very bad things. Companies should test for them and they should make like appropriate and reasonable mitigations, not asking them things that are impossible,
1:53:27 Speaker 02 but asking them to take seriously what is possible that is happening at other companies, that is happening at NIST and other places. And to do that, I think that seems like pretty fair. And I think the thing from the open source community is that if you're saying at some point in the future, you think it might be the case that there's going to be a model of Llama 5 or whatever that can do really dangerous things. And just saying it's just completely impossible for us to prevent it from doing those incredibly dangerous things if we open source it. And just like everyone else, you guys got to deal with it now. Sorry, this is the beautiful like march of progress and you're just going to be crushed under its wheels. I don't know. I just think that's like pretty intolerable. And I really welcome, I think, other solutions.
1:54:08 Speaker 02 I also am like really enthusiastic about trying to invest in a method to like send things around, ways to open source that still allow additional open inquiry and transparency in those models. Again, I think we don't know about what those research directions are going to look like, but I think this will be some promise in ways to open source that are resistant to adversarial crime tuning. I think there's a long way to go, but I don't think like fundamentally against the laws of the universe for that to be possible. And I guess I just want to say that so the large number of people who are like Worried about this bill and worried that we're going in the wrong direction, but who do believe that dangerous capabilities of future AI models are real. Like, I genuinely say, reach out to the author, reach out to myself. We genuinely want to get this right. It's complicated. In a perfect
1:54:51 Speaker 02 world, I would love to just like freeze where we're at for a few years and have a really long deliberative period so we can get this exactly right. I think that's not the world we live in, and we have to be setting these processes up and having this conversation now. And I think they're just like sufficient enough fundamental disagreements that I don't think Dean will ever be able to get to the point where he supports this bill. But like, that doesn't mean that he still can have valuable insights about ways that the bill could be better or problem that like that we should try to fence. And but my message that the appropriate action in face of uncertainty and in face of trade-offs is not to just freeze and just accept that's what's coming is coming. And I, which I know is not what you're advocating for, and one other thing as well. But I think that like, even on the developer level, that I think that we are like, we need to be thoughtful and we need to make sure that it's not creating perverse incentives and not causing harms. But I don't
1:55:36 Speaker 02 think it's fundamentally impossible. And just the fact that like government has been bad at things before doesn't mean that like, we shouldn't weigh and hear where everything takes are quite enormous. Anyway, it was a really pleasure chatting. I really appreciate it. I feel like I've been just spending too much time on Twitter, watching things go by and wanting to actually have a conversation with that person and realize that I'm engaged in with a caricature of God, we're myself, we're opening our shelves to just want to lock in the incumbent for evil reasons. And instead just make clear that we are both thoughtful people, want things to go well, and are trying hard in a challenging world to do that. And so, really appreciate you convening Nathan and Dean. Again, oh, like if even we don't think that you can ever support this, like I know your feedback and with your any additional ideas that goes for anyone else
1:56:19 Speaker 02 listening who is also, you know, wants to make it better.
1:56:23 Dean Ball? I think what matters is that we had a respectful, civil, and nuanced conversation.
1:56:27 Speaker 02 I'm glad we did as well. Steve, any closing thoughts?
1:56:30 Speaker 00 I just want to express my appreciation to everyone. This has been a fascinating conversation. I wish we had 50 more hours, literally. Thank you, guys, for being a part of it. Certainly, the conversation will continue. But for now, I will say Nathan Calvin, from the Center for AI Safety Action Fund and Dean Ball, Research Fellow at the Mercatus Center. Thank you both for being part of the cognitive revolution. Thank you. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.
Previous Infinite AI Interns: How Young Professionals Can Win in an AI-Powered World Next TinyGPU, Massive Learning, with Adam Majmudar Page updated October 26, 2026 · transcript and summary are machine-generated
