The RAISE Act: Minimum Standards for Frontier AI Development, with NY Assembly Member Alex Bores

In this episode, New York State Assembly Member Alex Bores discusses the RAISE Act, a proposed bill aimed at regulating frontier AI models with basic safety protocols. He explains his background in technology, his motivations for the bill, and the legislative process. He emphasizes the importance of having clear safety protocols, third-party audits, and whistleblower protections for AI developers. He explains the intricacies of the bill, including its focus on large developers and frontier models, and addresses potential objections related to regulatory capture and state-level legislation. Alex encourages public and industry input to refine and support the bill, aiming for a balanced approach to AI regulation that keeps both innovation and public safety in mind. Link to the RAISE bill: https://legislation.nysenate.gov/pdf/bills/2025/A6453 Link to support Raise bill: https://win.newmode.net/aisafetynewyork Link to support AI Transparency Legislation: https://docs.google.com/forms/d/e/1FAIpQLSdvpAHkAWiA38oIu1cp57azchPqOOoxb789tHQ896ikJf3CKg/viewform Upcoming Major AI Events Featuring Nathan Labenz as a Keynote Speaker ⁠https://www.imagineai.live/⁠ ⁠https://adapta.org/adapta-summit⁠ ⁠https://itrevolution.com/product/enterprise-tech-leadership-summit-las-vegas/ SPONSORS: ElevenLabs: ElevenLabs gives your app a natural voice. Pick from 5,000+ voices in 31 languages, or clone your own, and launch lifelike agents for support, scheduling, learning, and games. Full server and client SDKs, dynamic tools, and monitoring keep you in control. Start free at https://elevenlabs.io/cognitive-revolution Oracle Cloud Infrastructure (OCI): Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive The AGNTCY: The AGNTCY is an open-source collective dedicated to building the Internet of Agents, enabling AI agents to communicate and collaborate seamlessly across frameworks. Join a community of engineers focused on high-quality multi-agent software and support the initiative at https://agntcy.org/ Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Alex Bores, New York State Assembly Member, representing New York's State District 73 on the east side of Manhattan, and sponsor of the RAZE Act, a bill designed to set a minimum standard for safety practices among large AI developers. In the generally polarized political environment that we take for granted in 2025, it is a striking fact that while poll after poll shows that significant majorities of voters across both American parties and internationally want more regulation of AI, with far more voters worried that the government won't go far enough than that it will go too far, the United States still has no meaningful laws covering foundation models or frontier AI development.

• 0:44Nathan Labenz?

  In part, this reflects the fact that while people generally agree on this issue, it's not a top priority for many. In part, it's the result of a very healthy fear among legislators that they don't understand this fast-moving technology wave well enough to regulate it effectively and thus might end up doing more harm than good. And in part, it stems from the fear that anything that slows US AI development down could allow Chinese AI developers to get ahead and ultimately win whatever AI race we ultimately find ourselves running. These are real and important concerns, and I've spent most of my adult life arguing against premature or heavy-handed regulation that might inadvertently deny us the benefits of breakthrough technologies. But nonetheless, it seems to me that especially considering the breathtaking pace of AI

• 1:28Nathan Labenz?

  capabilities advances and the fact that even among Turing Award winners, forecasts range from AI-enabled utopia to AI-caused human extinction, a functioning democracy would be responsive enough to public concern to put at least some minimal standards in place now, both to reduce catastrophic risks and hopefully to avoid a future crisis, real or perceived, that could lead to knee-jerk and ultimately counterproductive decisions.

• 1:55Nathan Labenz?

  With that in mind, I think Assemblymember Boris, who notably has a master's degree in computer science and spent a decade in the tech industry, including a number of years at Palantir, is perhaps the tech-savviest and friendliest legislator the industry could hope for. And the RAZE Act, which targets large AI companies and imposes relatively modest requirements around the development and publication of a safety plan, audits to assure the safety plans are followed, and whistleblower protections to alert the public if they're not, is approximately the least burdensome regulation we could realistically expect to see passed. And it should be said, is very much in line with voluntary commitments that frontier model developers have previously made. Nevertheless, in this conversation, we'll get deep into the weeds

• 2:38Nathan Labenz?

  of the various definitions the bill uses and the requirements it imposes. I act as a sort of red teamer of the bill. And Assemblymember Boris clarifies and defends key provisions, and also explains the process he's already gone through to work and compromise with industry, and also to avoid a situation in which lots of different states create undue friction by passing their own distinct regulatory frameworks. Importantly, everything we discuss would only apply to a high single-digit or perhaps low double-digit number of super-well-resourced companies that are working at the frontier of AI capabilities. And these companies would still have wide latitude to design their own safety plans, so long as they take reasonable care to minimize risks that could cause 100 or more human deaths

• 3:21Nathan Labenz?

  or greater than $1 billion in damages, via either chemical, biological, radiological, or nuclear mechanisms, or what I would call automated crime. While some will no doubt raise additional good-faith objections, I found Assemblymember Boris's defense of the bill's neutral stance with respect to open source quite compelling. And as you'll hear, I hope, if anything, that this bill spurs open-source champions to invest heavily in new safety techniques, so that we can continue to enjoy open-source frontier AI without dramatically elevating the risk of engineered pandemics or other AI-enabled disasters. In any case, this is just the first of a series of episodes on different AI policy proposals that we'll be bringing you this summer. So I look forward to exploring a broad range of perspectives,

• 4:06Nathan Labenz?

  and I will continue to watch this bill as it evolves through the legislative process. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or leave us a comment on YouTube. Of course, we welcome your feedback either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Finally, a quick reminder, I'll be speaking at Imagine AI Live coming up, May 28 through 30 in Las Vegas, the Adaptive Summit, August 12 and 13 in Sao Paulo, Brazil, and the Enterprise Tech Leadership Summit, September 23 through 25, again in Las Vegas. If you'll be at any of these events, definitely send me a note and let's meet up in person. For now, I hope you enjoy this deep dive into the RAZE Act,

• 4:49Nathan Labenz?

  a bill meant to raise the floor for frontier AI development safety practices with New York Assembly Member and sponsor, Alex Bores. New York Assembly Member Alex Bores, sponsor of the RAZE Act, welcome to the Cognitive Revolution.

• 5:03Speaker 01

  Thanks for having me.

• 5:05Nathan Labenz?

  I'm excited for this conversation. You have waded into, you know, what might seem at first like, no big deal, just a little light technology regulation waters and which you may find, if you haven't already, it brings out a lot of strong feelings in people. So, but I do applaud you for taking on the challenge and I'm excited to get into your perspective and motivations and what you're hearing from people and obviously the proposed legislation itself. And I really appreciate you taking the time to do this.

• 5:34Speaker 01

  I'm really excited to be here. I know you have a lot of guests from all over the AI field, which I've enjoyed listening to, but not as many elected officials. So excited to dive in.

• 5:43Nathan Labenz?

  Yeah, you're on a short list. Speaking of short lists, one thing I like to do, especially for people that are coming into the AI world from other backgrounds, is at least give a little bit of kind of, you know, credentials or context on you. As I was doing my homework, I noticed that you are the only member of, I guess, the Democratic Party in the New York state government that has a degree in computer science. So maybe just give us a little bit of your personal background and relationship with technology. I think that'll be helpful.

• 6:10Speaker 01

  Absolutely. I was the first with a degree in computer science. I'm happy to say I'm no longer the only one, though it is quite limited. But yeah, my background is that I worked in tech for nearly a decade before I ran for office. I was at Palantir for almost five years, joined as a data scientist, rose up to lead a large portion of the government business. I joined a couple of startups after that. During that time, I got a master's degree in computer science with a specialization in machine learning. And then this seat opened up in 2022 and you know, I had a lot of helpful conversations with friends. I had always been downstream of policy, often trying to fix it with tech.

• 6:55Speaker 01

  And this was an opportunity to go upstream and actually design policy the right way. And so one of those conversations I had was focused on, listen, this is a thesis as to how you can have an impact. And Ron, you don't know if you're going to win. And if you do win in two years and four years, if you're not effective, if you're not enjoying it, you can quit. But you can't in two to four years, say, now I'm going to run for the open seat, right? That happens when it happens. And so I threw my hat in the ring. It was a contested primary. I'm still friends with everyone who, who we all ran against each other, but I ended up being victorious there. I'm now starting my second term in terms of only two years. So I'm just starting my third year in the legislature.

• 7:37Speaker 01

  But so far I found it to be a place where you very much can be effective and so I'm enjoying it.

• 7:43Nathan Labenz?

  Yeah, that's cool. Definitely more people with technology backgrounds in government seems good. And also frankly, just more people who have, you know, no fear of a post-elected office life, I think would be great. Too many, you know, are kind of like, what would I do if I didn't have this seat, right? And that's not a great position for the public to be in.

• 8:03Speaker 01

  No, it's a very dangerous debate. You have to be not too attached to the seat or the job, absolutely.

• 8:08Nathan Labenz?

  Yeah, gotta be willing to, that's, that's all I ask for from our elected officials is willingness to sacrifice your political career. When the occasion calls for it. And it turns out that is kind of a lot to ask. So how about just a little bit more detail on like some of the other technology related things that you've done in your time there? I noticed that you had pushed for the state to adopt cloud computing. I also caught the push for a land value tax, which I'm coming to you from Detroit, Michigan. We have a lot of empty lots and probably much less of a problem in New York City than it is here. But we have a lot of empty lots of people that are just free riding on others' investment and, you know, waiting for their land to appreciate since it is not taxed in that way. So maybe just give a little bit more context on some of the things that you have pushed so far.

• 8:52Speaker 01

  Absolutely. There's, when you're a legislator, you end up working on a wide variety of things because your constituents care about a wide variety of things. So, you know, all of us have our specialization and our knowledge that we bring into the legislature that's perhaps more unique, and mine is around tech, so I've done a lot there. But certainly, the concerns of my constituents vary widely, and so I work on a lot of things. Within tech, as you mentioned, I've encouraged the adoption of cloud computing within government so that we can deliver services quicker. I've helped to strengthen the protections for tech workers, really for all workers, but it was contracts that are more specific in the tech industry, where companies would say they own any IP you develop while you're employed,

• 9:34Speaker 01

  even if it's not on company time or related to anything in the company, and that would just chill startups. And I actually partnered with the tech industry to pass that because it was based on a regulation that had already passed California nearly a decade before. But one of the things people want the least is a bunch of varied regulation across different states. And so I was like, I'm going to copy the California one exactly. And tech was like, great, even though it's limiting us, like, the fact that it's a copy, that it won't be additional work. They ended up supporting it. So I've also done a number of bills this session around AI Beyond the Raise Act. I'm working on ensuring companies don't fall into legal liability when they red-team their own algorithms.

• 10:16Speaker 01

  We actually want to encourage safety. And so while we are strengthening a lot of provisions around preventing CSAM, child sexual abuse material, we also want to make sure there's a legal liability shield for red teaming, for trying to stop your algorithm for doing that. I'm encouraging the use of industry-developed standards like C2PA that are metadata that help to establish provenance on an image or a sound so that you know what's real and what's not. But beyond sort of tech, as you mentioned, I do a lot on housing. I have a bill to enable a pilot on land value taxes. I have something specific to New York City. So you mentioned Detroit and maybe New York City doesn't have as many vacant lands. We don't have as many,

• 10:58Speaker 01

  but the ones we have are valued at an incredible amount and the loss of tax revenue is substantial. If you just looked at the vacant land within New York City and were to tax it at its normal market value instead of this discount, I don't want to get into all of New York City property law, but no property is taxed at its actual market value. If you were to just tax vacant land at its market value, that difference would be another $800 million a year for the New York City budget. So I have a bill that would shift that around and make more uses for it. And then I do a lot around public safety. One of the things is that our trials in New York State are very backlogged. They are extremely delayed and

• 11:42Speaker 01

  there's many, many reasons for that. The dumbest of which is that we don't have enough judges. And I say that's the dumbest because that should be an easy thing to fix. So you just create more judges. And two years ago, I did. I passed a bill the governor signed that created 20 new judges throughout New York State, but I couldn't create any new ones in Manhattan or in the Bronx or in the capital region around Albany because of a limit in the state constitution that dates back to 1846. And so one of my other bills is a constitutional amendment to get rid of that limit to allow us to have more judges and speed up trials. So and then another 60 or so odd bills on my website that anyone can take a look at and I always love feedback on.

• 12:21Nathan Labenz?

  Cool. That's great. I appreciate the introduction. You mentioned your constituents, and I thought it would be helpful also just to locate you in geography and sort of get a sense for the people that you're representing. And then I want to ask, you know, kind of to what degree are they thinking and talking to you about AI? You know, where does it rank on their priority concerns? So maybe just take us quick through like the geography, the sort of, you know, profile of the people that you're representing. And then what, if anything, are you hearing from them about AI? Like, is this something that they're pushing you to act on? Or is it something that you are doing out of intrinsic motivation while they, you know, are mostly concerned with other things?

• 12:56Speaker 01

  I represent part of the neighborhood where I grew up. So my district is in Manhattan, much of the east side of Manhattan. It's part of the Upper East Side and Midtown East. So for those in New York, 34th to 93rd Street, Second or Third Avenue to Fifth for most of the district, plus Sutton Place. So in the 50s, the whole east side. It is a highly educated district. It is also the wealthiest district in New York State, but even within that, there's many people facing challenges. Forty percent of the renters within my district are rent burdened. They're spending more than a third of their income on rent. And so it's a district that has a lot of sort of pride in its education and its schools. It's in District 2. I happen

• 13:37Speaker 01

  to represent my elementary school and right near my high school and my middle school. They're both across the street from my district. It's an area I know well. It's sometimes neighbors, or it's been fun getting to represent and now meeting a different, like, parents of the friends I grew up with. And they, like any district, have a wide variety of concerns. I think if you were to pull them, AI probably wouldn't be towards the top. But, you know, they're worried about the cost of living. They're worried about public safety. They're worried about the schools. They're worried about the things that everyone is worried about. But when we get into a conversation and I start mentioning my background and my expertise in tech, the usual response is like, "Oh, thank you," because like, I'm terrified. And I don't

• 14:22Speaker 01

  know what we should be doing, but I'm glad there's someone there thinking about it. There's this sort of latent fear might be too strong a word, although for some I would say fear, but certainly unease and a feeling like something is coming and they don't really know what they're supposed to be doing about it.

• 14:40Nathan Labenz?

  Yeah. Is that, is there any more like shape to it than that when people get on the topic of AI? I mean, we entertain all, you know, the full range of risks on this program, and I would say your bill is

• 14:55Nathan Labenz?

  You know, as we'll get into here, it's more consistent with my approach, which is like, I want to see us have kind of a lot of benefits, a lot of deployment, you know, a lot of use in places like education, even though that's going to be fraught and we're going to have to figure out a lot of things, you know, I'm quite convinced that like an AI tutor for every kid is a part of a winning future. Then there's, you know, all kinds of things around like privacy and, you know, New York is probably the place with the most security cameras on the street of any place in the country, maybe DC has more, I don't know, but that's something I could imagine people talking about, uh, you know, surveillance and just kind of who's watching who all the time. Do people, where are they on this or, you know, what, what sort of mix of AI specific concerns do you hear about?

• 15:35Speaker 01

  All of the above. And I would say at the start, I largely agree with you. I think there is so much capability out there now that is unevenly distributed and we could be making so much better use of the existing tools in ways that government helps to service people and education, as you said, in a variety of different fields. And also I am worried about what the future holds if we don't put guardrails on for the research. So, and that's sort of like short-term bullish, long-term kind of wary is, uh, I share with you, but I don't know if it's as common in many places. I tend to, all of those concerns matter to my constituents. I hear a lot about workplace displacement. I hear a lot about privacy and surveillance.

• 16:19Speaker 01

  New York is now in the minority of states that doesn't have a comprehensive privacy law. We've been working on it for a few years, so that certainly comes up as a, a bedrock that we are delayed on. But I tend to think of AI concerns using like three binary questions. So first is pessimistic or optimistic. The second is short-term or long-term. And the third, when you think about bills, is it use case specific or is it general to the model? And I think you can find concerns and therefore you can find legislation in all of those categories. And you know, I have bills in many of those categories. Most of the legislation my colleagues are working on tends to be short term,

• 17:04Speaker 01

  pessimistic and either use case or general, right? And by short term, I don't mean that it'll expire. I just mean that it's dealing with harms that are definitely already here. The chance of discrimination, the chance of privacy violations, right? Things that are in use today. Whereas long term is things that maybe they're not here yet. Maybe they are, but they're just starting to. And we're thinking about where it'll go, whether that's broader societal risks, et cetera. And that's where the RAZE Act is focused. Not because I think that's the only thing that matters or the only thing we have to do, but just that is a place that not as many people are focused. And I do think there's important steps we have to take.

• 17:42Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:48Nathan Labenz?

  Are there any other things that you're pushing legislation on that might be described as controversial? I mean, the two that you mentioned previously around like encouraging red teaming for things like CSAM seem like most people would be quick to sign on under something like that. Is this a situation where, you know, we sort of see the 10% of the iceberg that pops up and actually gets public debate and then like 90% of things are generally pretty smooth sailing. People agree?

• 18:19Speaker 01

  Well, first of all, I don't think any of my bills are controversial. I think they're all common sense, but you know, you don't always control what the outside reaction is. You'd be shocked at the level of pushback I've gotten on encouraging C2PA, which is a, again, a free, open industry standard that industry itself developed. And then I say, oh, this is great. We should encourage it. And I get pushed back on, whoa, no, don't, don't encourage the thing we developed. So, yeah, but I think what you said at the end there is quite insightful and something that people forget. Most of the work that any government does is non controversial. Most of the bills we pass are unanimous or nearly unanimous. And they're just about making government work.

• 19:00Speaker 01

  You know, we'll do probably 800 bills total between the Assembly and the Senate that we both do, that we send to the governor's desk. She'll end up signing 600 or so a year. And, you know, of that 800, that's 600. You're going to maybe hear about 20, maybe 50. Um, most of the work is just making government work. And this is the thing I really remind people because we're at a point where belief in government is so low that, and partly that's because all they see on the news are fights. And all they see are the drama, but that's because there's an incentive to cover the fights and the drama. There's not as much incentive to talk about funding water infrastructure so that everyone has clean water. Yeah, of course we should do that, right?

• 19:45Speaker 01

  But that's most of the work of government and most of what we end up spending time on.

• 19:50Nathan Labenz?

  Yeah. Are you getting any, um, push for sort of protecting various industries? I mean, this is something honestly that I have expected. Okay, yeah, so tell me more about that because I, my perspective was like two years ago, I was like, wow, we are going to get into just brutal bitter fights about where AI is allowed to be deployed and who's going to have what sign off authority and whatever. And it's been slower to develop than I would have guessed. So I was kind of expecting you to say not yet much, but now you're saying all the time. So tell me more about that.

• 20:22Speaker 01

  Yeah, well, well, sorry. In what ways do you think it's been slower? What were you expecting to see?

• 20:27Nathan Labenz?

  I would have expected the medical establishment to have a like strongly unified front by this point that AI doctors must be confined to some, you know, very narrow box and like not available to the public directly and so on and so forth. And still today I can go on to chat GPT Claude and Gemini, which I did this week for a little thing on my kid's eye that I was trying to figure out what it was. The tip to the user is tell them you're preparing for a conversation with your doctor and that disarms the, I'm not your doctor, uh, you know, cached routine. But with that, you can basically engage, right? And it's, uh, extremely valuable. And I would say in my case, it probably did displace a trip to the doctor, which is

• 21:10Nathan Labenz?

  one of the things obviously that any professional guild might fear. So, yeah, that's, I mean, maybe that still will happen, but I had kind of expected it already. And it's, you know, the fact that I can just still go to chat GPT and ask my questions is like, honestly, kind of surprising to me of two years ago.

• 21:27Speaker 01

  No, it's, it's a really good. Flag. I wonder if partially that's because, you know, nurses and doctors and so much of the health profession are licensed and so there's already some built in protection. One of the, you know, tongue in cheek things I say, although there's some truth to it, is that people in government, elected officials are sort of the least qualified to be focused on AI displacement of workers because by law, we can't be displaced by AI. You're not allowed to elect an AI and it would be quite the change to the constitution to make that true. But kernel of truth to that is when you have licensing, there's some kind of built in protection. I think it's the jobs where none of that exists and especially where they don't have

• 22:11Speaker 01

  unions that you're going to see much quicker turnover. And so we see that in terms of a lot of the entertainment industry, right? You saw a lot of the strikes last year by the Screen Actors Guild and the Writers Guild and the Directors Guild was about AI's role in producing movies and TV and what that'll be going forward. We see it in government employees as well. One of the things that New York did last year was pass a bill that said you cannot use AI to replace a government employee, but it was specific on replacing the actual employee. Doesn't mean you can't replace tasks and focus them on other issues.

• 22:53Speaker 01

  We have so many open slots in government and so much more that we could be doing. And once you have that baseline of, oh, you know, you as an individual worker are not going to be replaced, you can change the conversation on, and this is why you should learn it and be happy about it. It's going to make your job easier and you have a protection of law versus many people who approach AI coming in like the, not the metaphorical Luddites, but the literal Luddites that just said, this is just here to take my job and let's destroy it.

• 23:23Nathan Labenz?

  Okay, that's a main focus of our conversation will still be on the Rais Act, I promise. But that's really interesting, and I wonder what you think about it. I see a very similar question. I was honestly kind of surprised by the answer that I got from New Jersey Governor Phil Murphy, not too far away. He was touting that, you know, they had done various AI deployments to accelerate call center response times. So previously, if you called like whatever, you know, line, you would wait 40 minutes on average, and they were able to bring that down to, you know, single digit minutes. So great improvement in quality of service. But I asked, if you had an option, right, and it seems like this is like coming quite realistically, quite soon, right?

• 24:06Nathan Labenz?

  Let's say you have an option where you can deploy AI to a customer service function. It's not, you know, we're not yet talking about like strategic decision making legislative, but just where the rubber hits the road, you can put an AI in a call center. Let's say you could do that in a way where you could cut 90% of headcount and 90% of costs and improve the service. Still got some people there perhaps to, you know, take the escalations or whatever. How do you think governments should be thinking about that? Like, should they be prioritizing efficiency and service? Or should they be prioritizing the jobs that they have? Or, you know, is there some synthesis of those that you can imagine?

• 24:46Speaker 01

  Yeah, it is an interesting question, but one that is largely academic because of how underfunded and already perhaps inefficient government is in that like there's more work for any of these people to do. Recently, in New York City, they transitioned to automated purchasing of Metro cards and now this Omni card. And so the station agents weren't as needed. They didn't get rid of any of the station agents. They just empowered them to actually walk through the station and then they can help to actually be a presence there and be engaged with people in that way. With call centers, I mean, I've personally called a government agency and been on the phone for four hours and waiting. And meanwhile, I'm Googling and searching.

• 25:28Speaker 01

  And if there were an easy chatbot or something that could have answered my question ahead of time, not only do I get an answer, but that takes me off the queue. And the person that actually needs to talk to a human, because there will always be people that still need that sort of engagement, get to get there a lot faster. So we really, if we're smart, we should be pairing all of these new tools with

• 25:51Speaker 01

  The knowledge that comes from this deep work in government and this deep work with our citizens in order to make things better for everyone. You know, I'm a big fan of Jan Polka and Recoding America. And in that book, she talks about someone who had joined, I think it was the California Department of Labor to process unemployment claims. I may be in the state wrong, but the story was that there was the new guy who didn't feel really confident in all the systems and all of that. And the new guy had been in the job for 18 years. I mean, the amount of just human knowledge that is tied up that we could unleash if we're protecting their jobs, right? And that doesn't mean in the future, you're always going to hire the same number of people. You know, you can go down by attrition and that.

• 26:35Speaker 01

  You can repurpose people into other roles. But if you make it us versus the machines, you're not going to get the best results for people.

• 26:44Nathan Labenz?

  Yeah, I feel like, I wonder how long that paradigm lasts. It does seem like the current paradigm, you know, accurately described, but the RZAC certainly seems to be a bill that is like, at least partially feeling the AGI, so to speak. And I do wonder if that paradigm lasts more than, say, another one or two years. And I also do really wonder about people's ability to change into those new jobs, you know, especially at scale. It's like, we have millions of people driving, you know, cars and trucks in the country. And if the self-driving car stuff really starts to work, which by the way, having been in

• 27:27Nathan Labenz?

  a Waymo and an Uber, not Uber, a Waymo and a Tesla recently, like, it really is starting to work.

• 27:33Speaker 01

  Totally.

• 27:34Nathan Labenz?

  You know, that's going to be a wave where it's like, we're not really going to be able to tell four million drivers or whatever, like, oh, you could go learn to code. By the way, it's still, it's also now a hotly debated question as to whether or not it's even worth learning to code. So where are they going to go is like another interesting question.

• 27:47Speaker 01

  Well, and I 100% agree. I just want to point out, I think that'll hit the private sector before the public sector. I think the public sector is so much. Kind of, we can do so much with just trimming our regulation and we have so many vacant positions already. That will the private sector first, but it's a thing that I am concerned about because people say you look through history and every technological revolution, every advance creates more jobs than it destroys. Maybe, but, but the speed with which that technology advances has been shrinking over time. And so until recently, you could guarantee that maybe it would create new jobs and it would be worth it for you to go back to school or to be retrained. The, because the next revolution is not going to happen in your career, right?

• 28:32Speaker 01

  But now we're at a place where jobs could be replaced, you know, every year, every six months. And if the AI is acquiring new skills faster than any human being can, that is a fundamental question. We don't really have a policy answer to, because by the time you retrain for the new job, you're going to be in the same circumstance. So that is, that is a thing government and people outside of government need to be thinking a lot more about.

• 29:01Nathan Labenz?

  Yeah. Do you spend time thinking of a new social contract?

• 29:07Speaker 01

  You know, I am right now thinking a lot about the RACE Act and how to get that through. But once that is through, this is definitely a place where I want to spend some cycles and I'm really interested in having conversations with others that are doing that deeply.

• 29:21Nathan Labenz?

  I saw something recently and I don't know much about this at all, but I think Tyler Cowen posted on Marginal Revolution, "Who needs a UBI?" pointing to the New York State, and you can correct anything I get wrong on this. The, the idea that like, there's now ability for people to choose and hire their own independent caregivers. And many are hiring people that they know, people from their families. And this is like, in some circles, you know, treated as a scandal. And I kind of looked at it and was like, well, if there's one candidate. Like broad class of activity that people could maybe shift into in real numbers. Caregiving broadly would maybe be the thing.

• 30:04Nathan Labenz?

  And this almost does seem like, you know, and I'm not sure how Tyler meant his, uh, who needs a UBI commentary. But I was kind of like, oh, this does sort of seem like a proto UBI policy that also still tries to get some useful contribution from people and, you know, probably does quite well on scores of meaning and things like that. And so I was kind of like, man, maybe New York state government has kind of stumbled onto something here that actually could be the seed of a new future social contract.

• 30:32Speaker 01

  I haven't read that piece and certainly if he's referring to what I think he's referring to it, it is not meant to be a UBI. So while there have been a lot of investments in caregiving and home health care, it's often paying for family members or others to do it, uh, instead of someone going into a nursing home, which might be much worse for them from a social perspective, but also costs the state a lot more. So there certainly have been people on the edges that have taken advantage of a program and we made changes in the budget last year to, to crack down on that. Uh, but overall, those sort of home healthcare have actually saved the state a lot of money. So I think you have to put that in the broad picture of things. No, UBI is definitely part of the conversation.

• 31:16Speaker 01

  This is one of the things that's moving really, really quickly. And so I imagine that will be part of it and there will be many other ideas that come as well.

• 31:24Nathan Labenz?

  Yeah. And right. The conversation, do you mean like the conversation in the New York State Assembly?

• 31:29Speaker 01

  I think more broadly than that, uh, at the moment, but hopefully part of the conversation in legislature soon.

• 31:35Nathan Labenz?

  Cool. All right. Well, let's, uh, narrow focus then to the RACE Act and what you're trying to do with it. I've got pretty detailed notes here, but maybe the first thing to do is just have you give the pitch, like, what are we trying to do? What does this bill require? Why should we, you know, be confident that it's not too big of a burden to put on companies, uh, you know, give us the high level.

• 31:57Speaker 01

  So this bill is meant to ask companies that are doing extremely advanced research of the kind that we don't really know the impacts yet, to have some basic safety protocols in place. Largely, those safety protocols that are required in the bill are in line with commitments that they already made during the Biden administration. And so, how do we know that it's not too onerous? Well, they've largely already committed to do it, and in many cases are already doing it, if not to the exact letter of the law, close to the spirit of the law. And those four provisions that it requires is that they have a safety plan, that safety plan be looked at by a third party, that's not them, that they disclose critical safety

• 32:41Speaker 01

  incidents, and we define that strictly in the bill as to what qualifies, but it has to be something that's really increasing risk, and that they don't retaliate against their own workers or that third party if they are whistleblowers, if they disclose something that is truly catastrophic risk. That's all it asks. And people say, you know, what is the impact? What is the target of that? In many ways, we're just putting very basic guardrails there. I don't think this is the furthest that we should go. I don't think this is the end of what's there. I think in many ways, this is just laying out a floor, such that most people in the

• 33:19Speaker 01

  field are really good actors, but when you have the crutch of your next quarterly profit, because largely this, right now, this would almost be exclusively public companies with some extremely well-funded exceptions, but when you have the next pressure of quarterly profits, it can become easy, even if you've written down a safety plan, to maybe say, hey, we're going to cut a corner on this. We're, you know, we're two months behind in the AGI race, and that could be catastrophic. Let's just skip this test. We want to make sure there's no incentive for doing that job. And at the most basic level, we defer a lot of the choices to the companies themselves.

• 34:00Speaker 01

  We don't come in and say you need to have exactly these tests done or exactly this evaluation of risk. What we're trying to prevent are the cigarette companies of old, knowing that their cigarettes cause cancer, but then denying it publicly and not doing anything to make the cigarettes healthier. Or the oil companies, knowing for decades in advance that their products are causing climate change, but denying it and still putting it out there. This is meant to say that if your own testing, if your own research that you've thought of ahead of time without the economic pressures, if your own testing and research is saying this is a massive risk that it could cause, what we define in there as a critical harm,

• 34:41Speaker 01

  100 deaths or a billion dollars in damages, you shouldn't be releasing that model.

• 34:47Nathan Labenz?

  Yeah, I mean, that's, that's, uh, seems like a not super stringent threshold.

• 34:53Speaker 01

  As I said, I think all of my bills are non-controversial.

• 34:56Nathan Labenz?

  It's just the other people that sometimes don't see it that way. So yeah, let's, let's go through a few of the definitions. So you gave the kind of threshold for severe risks, 100 deaths or a billion dollars in damages. There's also this large AGI company, or it doesn't say AGI, it says large AI companies. And that seems to be defined as a company that has spent $100 million in total training models and at least $5 million on one model in particular. Do you have an idea of how many companies that would cover in today's world? I'm not quite sure, honestly.

• 35:33Speaker 01

  I think still single digits. Maybe we've crossed into double, but it's a very small number. And that's intentional. It's meant to be looking at the absolute frontier as it exists right now and not sweep in too many others. I think one of the objections to previous regulation was that it would involve a lot of startups, it would involve a lot of smaller companies. So we chose that $100 million threshold intentionally. And we also exempted, I'll point out, academic research as part of that. As I said, again, this is really about those potential incentives to skip your safety plan, and we just don't see those same sort of incentives in academic research.

• 36:12Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 36:17Nathan Labenz?

  Yeah, okay, that sounds about right to me for what it's worth. I was kind of like, is Amazon on that list yet or not? I mean, we're talking like big companies that would be sort of the marginal, you know, in or out.

• 36:27Speaker 01

  Yeah.

• 36:28Nathan Labenz?

  It's also notable to me that the kinds of risks that are covered are pretty narrow. You've got your classic CBRN, chemical, biological, radiological, nuclear, I believe is the N.

• 36:39Speaker 01

  Yeah.

• 36:40Nathan Labenz?

  Biological, of course, being the number one in that category by far. And then you've got another category, which I think is pretty smart, which is just automated crime. I've got a little background myself as a red teamer of various models and products. And it is honestly amazing, although this, you know, actually wouldn't necessarily be covered in some forms by the bill, but it is amazing. In some cases, you can go to some of these like calling agent companies, clone a voice. I've done Biden, I've done Trump.

• 37:09Speaker 01

  Yeah.

• 37:10Nathan Labenz?

  Done Taylor Swift and prompt the model to just call any number and say anything and just try to scam people at scale. And they're like, still in the uncanny valley, but like automated crime is definitely the kind of thing that is now doable.

• 37:25Speaker 01

  Totally.

• 37:26Nathan Labenz?

  With some success rate. But that's it, right? So I'm interested in kind of, you know, any, any reflections you have on kind of where you decided to draw that line. It's true that these are not like behavior. I mean, with smoking, of course, like smoking is obviously bad for you. I am well aware of that, but it is much more of a behavioral sort of thing where it's like, you do have some agency in the situation. You kind of know it is bad for you at this, certainly at this point. You probably know you shouldn't be doing it, but you're still kind of doing it. It, you know, the smoking is almost more to me like the, you know, we're going to have addiction to AI. We're going to have people kind of falling in love with and, you know, going off into weird lands with their guys.

• 38:04Speaker 01

  I don't want to say using AI is smoking. I use AI every day for various things, right? That I was just doing the example of companies have knowledge. Their own tests show it's risky, but then they go ahead anyway. And the point of the bill being if your own tests are saying this is risky, we actually as a state are going to say you need to take a pause there. I want to talk about the risk, but I do want to say one more thing on the definition of large developer before we move on from that, which is, you know, you have to be a large developer. So you have to have spent a hundred million in training costs. And then a frontier model is either 10 to the 26 parameters and 100 million on that model, or specifically, knowledge distillation. So that's not a post-training modification.

• 38:48Speaker 01

  That is the specific process of using another frontier model, something that is 10 to the 26, to train a smaller model that can have similar performance and as broad performance as that original one. And that you need to have spent at least 5 million on. That's largely in response to DeepSeek. We're seeing a lot of new models spin up that are being trained on the larger ones that are introducing their own risk. And so, this bill has written, you know, might I cover the first version of DeepSeek that caught everyone's attention because that was trained on O1, which wasn't 10 to the 26. But, a DeepSeek-like thing in the future, as long as it has any interaction with New York.

• 39:31Speaker 01

  So that means it's available in New York via the App Store. That means, you know, you have any employees, any business presence, if you want access to New York markets, this applies to you. And so, that future version of DeepSeek would actually have all these requirements in there as well. And that was an intentional push on our front to not just apply to the first movers and then have people, especially overseas, introducing similar levels of risk. That was a really intentional choice in that regard. But jumping to the risks, as you brought up, we include, right, chemical, biological, radiological, nuclear risk. And that's, you know, if it in any way is aiding to bring that about, then these provisions

• 40:14Speaker 01

  apply and that's seen as an unreasonable risk of harm. And then, I've never phrased this as automated crime, but I think I'm going to do that going forward. The reason we're saying it has to be committing those crimes with limited human intervention is, we do want some... Built in protection on like, you can really abuse and twist and modify a model. And if a human being is really determined to prompt it in order to do something, you know, that violates some obscure law, that that's not what this is meant to target. But there's some level of limited human intervention, right? If I can just tell it to do a thing, it largely does it, right? That's what would fall under the potential risks.

• 40:54Speaker 01

  And there's an additional caveat in those risks, which is it has to be materially helpful in doing that. If I could Google how to build a nuke and I get the same high level overview as I do when I enter it into an LLM, that's not something you're going to be held liable for. It's really for when you're making a material difference in the ability to do that.

• 41:18Nathan Labenz?

  Yeah, for what I meant to emphasize in bringing up the sort of behavioral aspect of smoking was really just that like, there's a lot of other things that people are worried about. And I think with at least some good reason that are not, you know, that is out of scope for this bill. And that includes basically anything where it's like, this might be bad for you, but you might like it. And we're not really sure. And so, you know, all of that kind of stuff is out and we're not like, you know, in people's sort of private business with their personal relationships with AIs with this particular bill.

• 41:46Speaker 01

  Yes, totally.

• 41:47Nathan Labenz?

  The knowledge distillation thing is, if I had to guess, probably going to be one of the most fraught provisions that people are going to really want to understand and pick apart. It seems to me very reasonable to say to, you know, high single digit to low double digit companies that are spending a hundred million plus, that you got to have a plan, you got to publish the plan, you got to have an audit to make sure you're standing by the plan. And you got to not, you got to have some prediction for whistleblowers in case you're not doing that and people see that internally. And I think most everybody's going to be sort of on board with that. Certainly like your constituents, you know, I would expect will be quick to support that.

• 42:31Nathan Labenz?

  The knowledge distillation piece, though, is tricky. I guess first, just clarification question is like, is it an and clause? Like, if I'm doing knowledge distillation, I still have to be spending

• 42:44Speaker 01

  Yes.

• 42:44Nathan Labenz?

  $5 million?

• 42:45Speaker 01

  Yes.

• 42:46Nathan Labenz?

  Or and $100 million or necessarily $100 million?

• 42:48Speaker 01

  Yes.

• 42:49Nathan Labenz?

  Okay, so that is a hard hurdle that I, if I'm not over those financial levels, then I can distill all I want. Like, I can go grab R1 and distill into Llama, you know, 4.1, whatever I want to do, as long as I'm doing it under that $500 million spend low.

• 43:08Speaker 01

  Correct.

• 43:09Nathan Labenz?

  Okay, and those are, I assume, cumulative over, like, all time?

• 43:13Speaker 01

  The $5 million is per model, the $100 million is cumulative over all time.

• 43:17Nathan Labenz?

  Okay, so when I get to $100 million, and then it's both, right? I have to be, if I spend $5 on one, but I haven't spent $100, I'm not there. Okay, so that, that basically gives people a ton of freedom if they're operating at, like, certainly, personal budgets or startup training budgets. What you can do, and for people that may not know, like, what is a realistic fine-tuning budget? Obviously, they can get bigger than this, but like, the, I was recently a very minor contributor to a project that made some waves called emergent misalignment, in which, you know, research team, actually, in the, in the pursuit of answering a different question,

• 44:00Nathan Labenz?

  fine-tuned an OpenAI model on 6,000 examples of code being written that was not done according to security best practices. In some cases, like, kind of flagrantly so, like, just not taking proper precautions in the code that you're writing. So, fine-tuning a model to do that turned out to create a generally evil model, and that was the emergent misalignment phenomenon. And you would think, like, how does this happen? And people are still trying to figure that out. But what is clear and has been replicated is, like, train on insecure code and get a model that, like, wants to have dinner with Hitler and, you know, has all these, like, crazy notions about AIs enslaving humans and so on. And you're like, wow, that's really, you know, quite out of domain, and yet, you know,

• 44:45Nathan Labenz?

  pretty striking. Anyway, the cost to do that fine-tuning with those 6,000 examples is like $25 fine-tuning on the OpenAI API. So. You know, there's a lot of wrinkles. You're doing low rank fine tuning there. If you're doing, you know, all weights like that ends up costing more, whatever. But like, we have orders of magnitude between, you know, making a rather large behavioral change to a model and the sort of thing that you are, you know, that you have to be over in order for this bill to apply to you at all. I do think that's important to understand. I guess one question would be like, given the relative cheapness of knowledge distillation and the relatively

• 45:28Nathan Labenz?

  high financial hurdles, what is the purpose of that clause? Like, couldn't, could you delete it? And what would be lost if you deleted that, that knowledge distillation clause entirely?

• 45:40Speaker 01

  Would be lost. If you deleted the knowledge distillation, it would only be individual models that are 10 to the 26 and 100 million in spend. And you wouldn't have any coverage of DeepSeek-like phenomenon. And I think it's important if we're seeing more of those kind of threats to have some coverage there. But as you correctly point out, we're giving people orders and orders of magnitude to do interesting things here. It is really meant to just cover the frontier and in particular, those with large financial resources. So I keep emphasizing to everyone that 100 million threshold is its own threshold. If you have

• 46:20Speaker 01

  not spent 100 million specifically on compute, specifically on training, this bill does not apply to you. And so we were really talking about single digits, maybe double digits at this point that it applies to.

• 46:36Nathan Labenz?

  So can you tell me a little bit more about the theory of DeepSeek? I mean, I don't know what their total training spend has been, but they did say, I think, 6 million was the

• 46:48Speaker 01

  Was for

• 46:49Nathan Labenz?

  DeepV3

• 46:50Speaker 01

  Exactly.

• 46:51Nathan Labenz?

  And got turned into the R1.

• 46:52Speaker 01

  Yep.

• 46:53Nathan Labenz?

  So my sense would be that They, you know, just that model in and of itself like would hit that five threshold. Yeah, just walk me through the theory a little bit more. I guess the understanding is like, you're assuming or inferring that they took a bunch of OpenAI outputs and trained on that. And certainly it is a way to save money.

• 47:13Speaker 01

  V3 would not be covered, right? Because it was trained on O1 and it's only knowledge distillation if it's of a frontier model and O1 didn't qualify as a frontier model because it wasn't 10 to the 26. So, but it's meant to say deep.

• 47:24Nathan Labenz?

  So it has to be 10 to the 26 flops and 5 million?

• 47:29Speaker 01

  No, no, no, no. It has to be, the knowledge distillation needs to be using a model that itself qualifies as a frontier model. And so if it's using a model that isn't itself 10 to the 26 in order to train the smaller model, it does not apply.

• 47:45Nathan Labenz?

  But that original qualification, is that 10 to the 26 and 5? Or is that?

• 47:49Speaker 01

  Oh, the original is 10 to the 26 and 100, right? It is really just the most extreme models. 10 to the 26 parameters and 100 million in spend. That is the base definition of a frontier model. Then there's this additional definition, which is if you use a frontier model to do knowledge distillation and spend 5 million in that process, the resulting model also counts as a frontier model.

• 48:16Nathan Labenz?

  Okay, so I think I had a misunderstanding. I'm not a professional legislation reader. So let me just make sure I have this clear. So the way you get into the large AI company category in the first place is you train a model that is 10 to the 26 flops or 100 million dollars in spend on a single model?

• 48:41Speaker 01

  So I would, I would think about this a different way, right? I would say the definition of a large developer, of who this bill applies to, is you have trained at least one frontier model and you have cumulatively spent 100 million in training that model, right? So that's, that's, there are 100 million in training models, frontier models.

• 49:02Nathan Labenz?

  Okay.

• 49:04Speaker 01

  Frontier models are defined as one of two things. First is the one I think most people are familiar with that have been used elsewhere, which is the model itself is 10 to the 26 parameters and $100 million was spent on training that model. Right? That's the base of people think of as frontier, similar to what was in California, similar to the Biden EO without the spending threshold. All right, 10 to the 26 and 100 million, that's a frontier model. A second way that something can be a frontier model is if it is trained via the process of knowledge distillation from a frontier model and that process was at least 5 million. Right? So two pathways to become a frontier model.

• 49:48Speaker 01

  And then a large developer is someone that has spent $100 million training frontier models. So they can do that either by training one 10 to the 26, 100 million, or by training 20 knowledge distillations, 5 million on each of them, or any combination thereof.

• 50:07Nathan Labenz?

  Gotcha. Okay. So the main reason for the knowledge distillation clause is that you want to catch companies that are working at large scale, but taking a knowledge distillation route such that their largest individual models could still sneak under the mainline definition of a frontier model, but would have similar capabilities because obviously that's the whole point of distillation.

• 50:38Speaker 01

  Absolutely right.

• 50:39Nathan Labenz?

  Okay. Good. Well, thank you for walking through that with me. This stuff does get a little gnarly sometimes.

• 50:45Speaker 01

  No, and listen, legislation, especially at the state level, is much easier than reading federal bills, right? Federal bills are thousands and thousands of pages. This one, I think, is 15 or so.

• 50:54Nathan Labenz?

  I'd like to read every word.

• 50:57Speaker 01

  No, I manage more. And the details really matter and it's dense legal language. And I think that's one of the challenges we've seen in past bills is, is you have people hop on Twitter and take, you know, a couple of words out of context or don't think about, you know, Every bill is inserting language into the code, right, the legal code, not the computer code, the legal code of the state. And so it's affected by all of these other words that are around it as well. And so, you know, I support everyone asking questions. None of this is easy to understand exactly all of this. The same way, you know, coding any of these models is not easy. But, you know, you don't see people jump on Twitter having read like three lines of code in Llama. And we're like, oh, my God, the whole, you know, right? Like, you should think about.

• 51:42Nathan Labenz?

  Well, you should join my part of Twitter. I think.

• 51:46Speaker 01

  Fair enough, fair enough.

• 51:47Nathan Labenz?

  I could show you some. Okay, well, let's go maybe double click then into a little bit of the, you know, deeper into the language. Here's my summary of the requirements that the frontier developers would have in terms of the safety protocols that they would have to develop and publish. Basically, they need to come up with various ways to reduce risks. They have to reduce risks of, you know, the model being used for these CBRN type purposes. There's an interesting clause about reducing risk by sophisticated actors. Which I'm interested in. I assume that's like code for the CCP or maybe North Korea or whatever.

• 52:27Speaker 01

  Yeah, any sophisticated state actor, you know, that's not targeting one specific one that's sort of any out there. We're saying that the stakes of this debate and the risk are so high that you need to include nation states acting.

• 52:39Nathan Labenz?

  And that is typically translated to like, secure the model weight. So, you know, sort of tighten up your security practices in today's world. Is that how you are imagining that playing out as well?

• 52:51Speaker 01

  Yeah, largely. It's not, you know, all of cybersecurity is based on your threat model and based on, you know, what the risks potentially are. And if you're applying normal corporate security to these models, you're probably not doing enough. This is just meant to be very explicit that the stakes of this incredibly powerful technology are large and your threat model should be including sophisticated state actors.

• 53:17Nathan Labenz?

  Okay. There's then an interesting section also where it's like basically saying you have to sort of explain why you think the tests that you have outlined actually tell you what you're claiming they tell you. It's sort of an epistemology of your whole like risk analysis. And this is a tricky one to me as somebody who has like, you know, built a bunch of these workflows and, you know, simple agents and stuff. You always get these caveats in like work from meter and so on where they're like, we built some scaffolding to try to figure out how, you know, what scale of research engineering task and a

• 53:58Nathan Labenz?

  model could do, but we don't really know that we did a great job. You know, we don't know what the limits are. We, you know, scaffolding could almost certainly be improved. And maybe I'll just couple that with a, you know, an actual quote from the bill is a large developer shall not deploy a frontier model if doing so would create an unreasonable risk of harm. So I guess where the wrangling ultimately is with something like this is like, what's a reasonable or unreasonable risk of harm? And to what extent must people go to, you know, demonstrate that they've pushed the scaffolding and really elicited the capabilities to the fullest, knowing that that's hard and like the state of the

• 54:40Nathan Labenz?

  art is like very much evolving? How does somebody know if they have done a good, good enough job that they're on the reasonable side of unreasonable?

• 54:49Speaker 01

  The short, totally unsatisfying answer is that often in law, we use this sort of reasonable person standard, right? There's a lot of things where we're not going to be able to exactly specify everything that needs to be done. We can sort of point at as close as it is, but you sort of lead a little bit of deference to what a reasonable person would be. That's not invented in this bill. That's not invented in law. That's a well-established legal standard. And I think it's particularly, so the longer answer though, is that anytime you're writing a bill on anything, but in particularly something as fast-moving as technology and in particular AI at this point in time, you have this tension where

• 55:35Speaker 01

  People rightfully want specificity of exactly what it is telling you to do. And on the flip side, want it to be able to evolve in time. Because exactly what you should be doing right now will change in six months, in a year, in two years, et cetera. And that's always a balance, right? The no law is ever final. The legislature can obviously come back and make changes at any point. So you don't need to write it so that it lasts 1,000 years. At the same time, you don't want to be deferring so much that companies really don't know exactly what's required of them. And so any bill is going to have that tension and you're going to find the balance somewhere. You know, I think a sign that we found the balance in a pretty good place on this one

• 56:18Speaker 01

  is that we have about an equal number of comments on both sides of it. Of I want more specificity, or actually, I want less government telling me exactly what to do. So we've probably hit it right. But you're pointing out that exact piece. And that's why at the start, when I was describing the bill and I tried to emphasize for people, we are largely letting the companies grade their own homework. We are largely saying, you know, you put out what the standard should be, write it ahead of time when you're not in real economic pressure. And then, you know, grade it against that. And the only real pushback on that is A, we have a third-party audit and that audit is going to, you know, include, did you actually follow this? Did you follow best practices, et cetera? And then B, this reasonableness standard.

• 57:01Speaker 01

  So if you write a plan that just says, we don't care about safety, we're not going to do any of this, and this is the standard, well, that's pretty clearly unreasonable.

• 57:11Nathan Labenz?

  How about the relationship between all of this and open source releases? You know, at the sort of sophisticated actor level, you know, obviously, anybody around the world can download a Llama model or any open source model. And there was also one clause, I wasn't quite sure how to interpret it, but it referred to modifications, which in, at least in some other, you know, debates, has been understood to be like, post-open source release modifications that, you know, who knows who might make. I mean, maybe with either like close reading of the bill or just like your, you know, intent, what? How does this apply to somebody like Meta,

• 57:53Nathan Labenz?

  who is going to potentially release a behemoth version of Llama4, which I think would probably get to that 10 to the 26, and I'm sure would be in a north of, I mean, they've clearly spent 100 million, whatever, like, seems like they're going to be in. We don't really have great ways, as I'm sure you're well aware, to like, really control what people do downstream, once a model is released. So are they on the hook for that or not?

• 58:24Speaker 01

  The bill's agnostic to whether you open source or closed source, right? The best way of balancing, this is not saying specifically open source or closed source, but just saying, think about the risk, think about the use case, make your own judgment with that. And so there's many, many ways to keep a model safe, right? You can have it on your platform, or you can monitor it at all time. You could go a step further and do know your customer, and only release certain features and certain things that are really powerful to people that you trust, right? That becomes another way of doing the risk. If you opt to open source it, like, that's great, too. It helps to encourage, you know, academic study and analysis of all these things. But any of those choices come with risk. And I find it kind of bizarre, those that say, oh, you should

• 59:09Speaker 01

  evaluate, we should take a risk-based approach, which is what companies always say, you should evaluate everything in its context, except open source, ignore that context. That context doesn't exist. Just write all of that off. I think we are not in any way targeting open source. We're just saying, it's up to you to make your choices on your risk profile. And you should do that accordingly. I think the vast majority of AI that has been released, it's great that it's been released, and it's been encouraging. You'll note that this bill doesn't actually require you to take in the risk of cyber. And I think that's largely because it's really already there, and maybe that ship has sailed. But it's up to companies to decide the risk and the way that they're deploying it. And all

• 59:53Speaker 01

  of those decisions matter. So we sort of leave that quite open ended. But to my point of legislation always being changeable, like, that's not it. We don't want to shut down that ecosystem. That's not an outcome we want. If that's where it's trending, we can change this.

• 1:00:12Nathan Labenz?

  So I think that's an intellectually honest position. And I do think This is

• 1:00:18Speaker 01

  Which is not something that elected officials are often have queues out. So I appreciate that.

• 1:00:23Nathan Labenz?

  Yeah, I mean, the the normal way that people try to get out of this is some sort of denial or cope or whatever. But it does strike me that like in taking a neutral approach with respect to open source, it does make open source harder. Like it is much easier, let's say, to manage risks if you have a closed source model where you don't release the weights. If you do release the weights,

• 1:00:49Nathan Labenz?

  you just really have a hard time in in many, many ways, even knowing or, you know, being able to predict what will happen, let alone controlling what will happen downstream from there. So it does seem that this could create real risk for a company like Meta that's trying to evaluate and might steer them toward not releasing if they're like, geez, you know, we have, you know, we have basically no known, you know, we can train this thing to refuse and we can put out llama guard. We can do all these different things to try to enable people that want to do the right thing to do the right thing and to set them up for success. But we really can't prevent somebody from, you know, untraining that refusal behavior or just not using, you know, llama guard or whatever. Right? So

• 1:01:35Nathan Labenz?

  they're they're in that analysis. It sounds like basically your sense is like the risks are the risks. And if you can't do it, or we don't have the right techniques, then like, maybe you just shouldn't put it out. And that is kind of the reality.

• 1:01:49Speaker 01

  I I would say two things to that. I would first say, I start often with a question of, do you believe that there is ever any information or any capability that should not be open sourced? And I think most people would say, like, really detailed analysis or the ability to produce really powerful bioweapons should not be open sourced, right? But but whatever your threshold is, right? Do you believe in classification at all? Do you believe in restrictions on weapons at all? Is banning the sale of nuclear weapons a reasonable, right? Like, just as long as there is some level that you think should not be open to the public. All we're saying is declare that level, right?

• 1:02:34Speaker 01

  And then go from that. I'm not putting a specific level out there. And so if people can, you know, with a straight face, say no, I think every capability and every power should definitely always be open source. Then you've already sort of accepted, hey, we got to think about the risk here. But the more specific thing I would say is I don't think this is going to change behavior because the threshold for critical harm is 100 deaths or a billion dollars in damage. But I think the leadership of every company is probably comfortable saying, we don't want our products to cause 100 deaths and a billion dollars in damage. And we'll do actions that will stop that from happening.

• 1:03:17Speaker 01

  And right now, none of their products, as best I can tell of any of these products, really reached that threshold. So we're not talking about restricting any current behavior. But whether this bill exists or not, I would hope that the board of a public company would be comfortable saying, yeah, our policy is not to cause 100 deaths and a billion dollars in damage. That's all we're asking them to do.

• 1:03:39Nathan Labenz?

  Yeah, it's going to be really interesting to see how this plays out. I mean, I think we're, I don't know, one to two model generations away from, certainly that's if you listen to the Dario and Anthropic timeline, you're one to two generations away from models that would be, in a very meaningful way, able to do some sort of, you know, needle moving contribution to the creation of a bioweapon. And I would definitely agree, you know, that Zuckerberg doesn't want to have to face the public and say, you know, yeah, we shipped it even though we thought maybe it would, you know, cause a pandemic or whatever. But it is really going to be tricky because there's like, first of all, there's a lot of people who, not necessarily in their role as like a, you know, corporate executive,

• 1:04:23Nathan Labenz?

  but there are a lot of people who would say, first of all, that threshold isn't that high. You know, the, how many people die on the highways every year, right? It's like literally 100 people die a day on American roads. So, okay, you know, it is a big world and 100 deaths is. A tragedy, but it is also one day of US road deaths. And so the benefit of

• 1:04:44Speaker 01

  And specifically 100 deaths or a billion dollars from a chemical, biological, or ideological or nuclear weapon, or something that is already a crime in the penal law. Right. So it's not talking about sort of accidents. It's talking about automated crime or CBRN.

• 1:05:01Speaker 01

  It's a, it's a, I don't want to use the word intentional death, but it is one that is caused by a crime or a big weapon. And that is not a thing that you see every day.

• 1:05:10Nathan Labenz?

  Yeah, it's almost like, I mean, what's so weird about a lot of these things is that, you know, the pandemics we really worry about cause a lot more than 100 deaths. It's like an in expectation sort of thing. If COVID caused whatever 10 million deaths, then, you know, in expectation, one in 100,000 chance of a COVID like outcome would lead you to an expectation of 100 deaths. And that's just like a very weird epistemic position. Basically, nobody has the clarity of what, you know, we just don't, you know, again, to quote Dario, like, we don't know why these things do what they do half the time.

• 1:05:43Speaker 01

  Totally.

• 1:05:43Nathan Labenz?

  So we're in a really weird spot where it's very hard to give an assurance that's at five nines on anything. And we do have existence proofs that, you know, one of these things can easily get to 10 million deaths and obviously, you know, could have been a lot worse. And so we're in a, just a very weird epistemic position. Yeah, I mean, I, I think it is worth, you know, taking all this very seriously. And I guess my hope would be that it really pushes people to invest hard in areas where we haven't got answers yet, right? I mean, the, the real thing is like, can we create a model that doesn't know about virology?

• 1:06:26Nathan Labenz?

  Is there some way to wall off that kind of knowledge and excise it from the version that gets released?

• 1:06:34Nathan Labenz?

  You know, interpretability techniques or otherwise, is there some way that we can say with an affirmative safety case, like, yes, we can be confident we're being reasonable here and we know this, you know, not just because the thing refuses nine times out of 10, but because we have a much deeper understanding of what's going on. But if I had to guess, I think we are probably headed for a moment. I mean, tell me if you would see this differently, but I would expect a sort of 2026 reality being like, this bill gets passed, Lama five is trained. We don't have that like affirmative safety case yet.

• 1:07:09Nathan Labenz?

  The risk is kind of on the unreasonable side. And unless Meta is just like willing to run the risk for whatever reason, they probably have to look at this and say,

• 1:07:22Nathan Labenz?

  can't quite release it in this form. We either need to like solve some technical problems we haven't solved, or we just can't put it out there because it's just too powerful. And maybe they would even come to that decision on their own, you know?

• 1:07:34Speaker 01

  That's my point is I think we're talking about a bill that, you know, requires people to do some work upfront, writing things down on paper, knowing that their homework is going to be checked and whose fines are in the eight figures. I mean, they are going to make these decisions separately on societal level, large risks.

• 1:07:56Nathan Labenz?

  So I guess in the spirit of sort of red teaming the bill, as I understand it, the maximum penalty for like repeat offense under the bill is a $30 million fine.

• 1:08:06Speaker 01

  Yes.

• 1:08:07Nathan Labenz?

  And, you know, I think they paid Trump off at that level, right? And we've seen like multiple people just sort of say, you know, forget it, we'll just settle this lawsuit because we just, you know, want to get this guy off our backs. Then maybe one way to sort of soften the bill while still getting all the benefits that you're wanting would just be to like require the plan and, you know, be a little bit less on the sort of downstream enforcement side of like what was reasonable or not reasonable. Do you think that that could be a version of this that, you know, if pushback or whatever dynamics ended up being conducive to it? Is that something that you think would be a viable possible compromise at some point in the future?

• 1:08:47Speaker 01

  I think the version that we have right now is the result of many, many compromises. And I want to be clear that I, you know, I started this by looking at all of the public debate around last year's regulation and probably accepting 95% of the public critiques. And then I sent a draft of this bill to, you know, five or six major labs, asked for, you know, red teaming, asked for red lines on the bill, got them back, did another draft around December. I sent again five or six labs, hey, here's a new version. Got more feedback. Ended up talking to a lot of people in the state and that's why this bill was published in March, but it's been circulating since, you know, probably July, August of last year, getting a lot of this input and the feedback. So this is in no way the first stab at it.

• 1:09:32Speaker 01

  This is in response to a lot of industry feedback and a lot of compromise. And I think if you see other people saying, well, we need a compromise version, this has been the compromise version. But on the specific thing of just releasing the plan, you still need to define what the plan is, right? There need to be some standards. You can't have someone have a one sentence, "We're building the AGI," and that be the plan, right? So you got to put some standards as to what the plan ought to be doing. I think the third-party audit is incredibly important. That was part of the voluntary commitments. That was part of the regulations that came out of the commission, from the Newsom Commission after 1047 embraced

• 1:10:15Speaker 01

  it. It was part of the EU version, right? Third-party audits, I think, are really core to the bill. Whistleblower protections, I think, are extremely core to the bill. It's sort of the one part everyone agrees on and is moving in every state. And then disclosing critical incidents is crucial just to keeping New Yorkers safe. So I think all four parts of that bill are pretty required and pretty drummed down. The part that I think you're pointing out of the "don't release a model has an unreasonable risk," again, is just that they're largely grading their own tests. This is the smoking companies, once they know it causes lung cancer, need to proactively take action. Oil companies, once they know

• 1:10:59Speaker 01

  it causes climate change, need to take proactive action. When your own tests say this is going to cause deaths, you need to take action. The fact that it's 100 deaths, I think, is part of the compromise.

• 1:11:13Nathan Labenz?

  Yeah, okay. Maybe just a couple of double clicks on several issues, since we're going deep here. How do we not have regulatory capture of the auditors? This is something that I experienced once upon a time in the financial services industry and...

• 1:11:31Nathan Labenz?

  You know, I've had a number of these kind of model testing orgs on the podcast in the past, Apollo Research and folks from Meter and Palisade and more and Far AI. And, you know, all these folks have a tricky position. I would say that, and I know them personally to some extent, and I would say they are very sincerely motivated by a safety mission, but they are also very mindful that their access and ability to do their work at all is at the moment at the pleasure of the companies. And so they're like very cautious about how they speak publicly and, you know, all this kind of stuff is like very, very carefully thought through because they don't want to get, you know, they don't

• 1:12:14Nathan Labenz?

  want to offend somebody and get cut off. So I don't think the bill has much on that yet. You know, is there any plan for that problem?

• 1:12:22Speaker 01

  I think that's one of the ones where we don't want to legislate ahead of time on it, but it's definitely a thing I'm concerned about. I mean, the only requirement is that it's a separate auditor. It's not a government agency. I think of it as pretty similar to the SOC 2 process, right? Where you have consultants, these auditors that are set up to evaluate your security stance, but they take into account the size of your company, the risk, et cetera. It's not sort of a checklist of hard things. I'll point out that all of the companies that would be subject to this already require SOC 2 of all of their vendors. So it's a similar process that they engage in. But you're right, when you have this kind of open market and government is not licensing the auditors, there is always that

• 1:13:07Speaker 01

  chance of regulatory capture. We require the auditors follow best practices. We require, you know, who's doing the audit and all of that. But it is, it is a thing that we will monitor over time and I hope not to have to take more action, but certainly a real risk.

• 1:13:23Nathan Labenz?

  Yeah, okay. On the safety incidents, as I was reading through the, you know, the different things that would qualify, various incidents kept coming to mind and I was like, would that qualify? Would that qualify? So I don't know that you can like officially judge, but I'm interested to get your reactions on.

• 1:13:40Speaker 01

  Yeah.

• 1:13:41Nathan Labenz?

  So a couple of lines from the safety incident definition. Clause A, a frontier model autonomously engaging in behavior other than at the request of a user. And here I'm like, I can point you to a lot of people who have reported that, you know, Claude changed the model from OpenAI to Claude. Or, you know, just yesterday, a friend was like, basically I vibe coded my way onto the project maintainer's shit list because I ended up modifying the database in ways that I wasn't even meaning to, but the model just, you know, got blocked one way and went another way. And I mean, I think that we're very confused, honestly, broadly as a field on like autonomy. Like, should we want it?

• 1:14:23Nathan Labenz?

  I'm not so sure that we should, but we're clearly pushing for it. And I guess my, my sense is like, maybe that clause is like being triggered a lot in the world today. Do you feel like that may in fact be the case?

• 1:14:36Speaker 01

  It, it, it might by itself, but remember that all of those four specific incidents have to be only if they're increasing the risk of a critical harm. And so if it's a minor, you know, thing that pops up and you're not like, this isn't going to increase the risk of 100 deaths of billion damage, that's not something to report. Um, you know, if a employee forgets to log out at the end of the night, but no one comes in and oh, the janitor saw the code, but they're right, that's not a thing. But if China steals the code, right, that is a different sort of circumstance. So it's taking into account if it would cause a real risk of a harm. And largely, again, relying on the company's judgment of that, but saying that things that do rise up to that level of these specific four need to be disclosed.

• 1:15:21Nathan Labenz?

  Yeah, that, I mean, it's easier sometimes as you get down the nested structure of these bills to forget the sort of top level clause. So it's a good reminder that all of this stuff is in conjunction with this increasing of the aforementioned critical harms.

• 1:15:36Nathan Labenz?

  I guess a couple of other incidents that came to mind would be basically examples of companies not following their process or sort of mistakenly releasing capabilities, specifically being famously like launched in India and a couple of countries, I think, without going through the safety board approval process that they and OpenAI had together agreed on. I think some people at OpenAI were also involved in sort of saying, eh, go ahead and do it. In retrospect, all the Sydney behavior was in fact reported on the forum and they missed that as well. Would something like, I mean, that model was only GPT-4, so it's, you know, maybe not at this critical harm level, but would something like that qualify? That seems like it would in a future scenario where the model is like

• 1:16:21Nathan Labenz?

  more powerful, like you can't do that.

• 1:16:23Speaker 01

  Yeah, it might. You know, I don't, for a legislator, I have a deep background in these things, but I don't claim to be the foremost safety researcher, right? I wanted to defer some judgment to the people who are doing this every day and that's what the bill is meant to do. So with that caveat and, you know, my, my voice here not necessarily being binding on it, you know, I think if you temporarily enable a feature and you've monitored the platform that whole time, so you know if it was used in any way and now you've turned off that feature, did that actually introduce a real risk? Like, probably not. You know, again, depends on the exact feature and how much you can see into how it's used, et cetera. But, you know, this is a, when you're, when you're playing with

• 1:17:07Speaker 01

  fire, you know, it's a different standard than when you're playing with, you know, a smaller feature.

• 1:17:14Nathan Labenz?

  Yeah, we're definitely playing with the new kind of fire here. Say that all the time. Okay, whistleblowers. So I'm a big supporter of general whistleblower protections. How strong do you understand these protections to be?

• 1:17:27Speaker 01

  Super weak.

• 1:17:29Nathan Labenz?

  Yeah, okay. So what, I mean, one key question is like, if you go to the AG, can the company fire you for doing that?

• 1:17:36Speaker 01

  It would be illegal under this law and on top of any other laws, you would also be subject to this $10,000 fine per employee per retaliation, as well as injunctive relief, so you might have to hire the person back. But yeah, $10,000 is nothing compared to the resources here. Personally, I would love to see that be a lot higher and a lot stronger. You know, every bill exists in the context of its state and the state law, and 10,000 is the standard in New York across a wide variety of fields. I think it was a push last year even to increase to a higher percentage for those that have worker violation of child laborers, right? But that sort of got beaten down.

• 1:18:18Speaker 01

  It's maybe seen as a Pandora's box of doing anything above that 10,000 level. So yeah, I think probably you want a stronger incentive there, but you want to make sure at the very least that there is statutes on the books that if you are firing someone for raising catastrophic risk, that is illegal and that there can be action taken for that.

• 1:18:40Nathan Labenz?

  In terms of injunctive relief, hiring somebody back,

• 1:18:45Nathan Labenz?

  that seems hard in the sense that like, you're not going to have, you know, the AG sitting in on meetings at, you know, at the companies, making sure that this person's job, you know, is the same as it used to be. And obviously, just things change.

• 1:18:59Nathan Labenz?

  I guess, what do you, how do you imagine that playing out in practice? Like if somebody actually says, you know, I'm freaked out about whatever.

• 1:19:08Nathan Labenz?

  I'm going to the AG. The company's like, you, you know, violated our trust. Like, we have our own protocols you should have followed. You didn't do it, whatever. Either you're fired or you're sort of, you know, banished to home office status and will like continue to pay you or whatever, but you're not like going to be, you know, privy to all the things you used to be privy to. How does that actually, like, what should a whistleblower expect in terms of actual material outcome or protection for them individually? If they... Do kind of violate the chain of command and come to the government.

• 1:19:41Speaker 01

  There's good news and there's bad news here. The good news is that there is a lot of case law and statute around whistleblower protections because they exist, covering a variety of actions in New York State, but also throughout the country for reporting anything that is illegal behavior, right? And so it's not just you have to be rehired, it's that your job is protected, your responsibilities are protected, right? And if any of that changes, there can be injunctive relief, there can be following things from that. We see this, I come out of the labor movement and you see often people being fired for organizing or for labor violations and then you have the ability to restore that. So that's, the

• 1:20:21Speaker 01

  good news is, you know, much of this debate is out there and the system functions already. We're just adding on something that might not be explicitly illegal, but is a catastrophic risk. That's the only change here. The bad news is, yeah, I think broadly in society, labor protections are not as high as I would like them to be and I can't promise the system functions exactly as it should. And I think that's part of a larger conversation that should be had, maybe separate and outside from this bill and one that I would love to have, but it is not, largely, we're not changing whistleblowers in New York because there is so much on the books. We're just adding that bit about catastrophic risk.

• 1:21:04Nathan Labenz?

  Yeah, there's been a couple of calls recently for class consciousness among the research engineers at the leading AI developers and it's a weird dynamic because they are seemingly increasingly, in a very like self-aware way, trying to automate themselves out of a job. You know, the vision seems to be increasingly explicitly, as far as I can tell, like get the AIs to do the AI research and then hope that we Can steer them or hope that we like set the initial conditions right. And all that honestly is pretty scary to me. How about internal deployments? As far as I can tell, that is not part of this bill, but the sort of, you know, bleeding edge of policy discussion is is turning toward

• 1:21:49Nathan Labenz?

  internal deployments. I think for this reason where there's sort of an expectation that the gap between what companies have and maybe are using for their own AI research and what the rest of us, you know, plebes in the public get to see and use might widen pretty dramatically if the companies are all locked into this sort of game theoretical race for first to AGI.

• 1:22:12Nathan Labenz?

  And so first, am I correct? That's not really addressed here. And second, like, is it on your mind for, you know, possibly something to come back around to?

• 1:22:18Speaker 01

  So actually, internal deployments would be covered in most cases here. So so the definition of deploy at the top includes using the model as well as making it available to others. And so if you are using it even internally, then it is covered by this. Now, we exempt anything that you are like testing and development and evaluation. That doesn't count as using it. Additionally, we exempt if you're doing it to comply with, you know, state or federal law, or it's part of broader federal project, right? Those things are already exempt. But general use that isn't in one of those categories would actually be covered and trigger the requirements in this bill.

• 1:22:55Nathan Labenz?

  Yeah, that sounds to me like the most likely place for the whistleblowers might might end up feeling compelled to come forward. The idea that like, you know, what exactly counts as use and whatever. And we've got this sort of, you know, guardrails light or guardrails free purely helpful model that, you know, we have kind of access to internally and the controls are not great and people are, you know, asking it to do AI experiments. And the whole thing has a sort of gain of function type of vibe to it. So, yeah, that's really interesting to think about. But that's a that's a really good clarification. And I do think. An enlightened one to include at this stage.

• 1:23:34Speaker 01

  I appreciate that, and I think it goes back to part of the conversation we had of how is this field going to develop? And do we really expect the most dangerous models to be released open source? Or we're already seeing companies move in this direction? And whether this bill exists or not, right? This isn't changing liability for things that happen after you release it, right? So companies are already going to start to make that decision of when we get really powerful, how should we be thinking about what goes out there? This is just meant to sort of improve your internal stance and your safety planning and everything that goes into that ahead of time. And I think you're right. I think more of the risk is coming from internal deployments now, and that's why we wanted to make sure it was covered.

• 1:24:15Nathan Labenz?

  Cool. Couple final questions, and I really appreciate all the time. This has been great.

• 1:24:22Nathan Labenz?

  Obviously, one of the big concerns and really the sort of nominal concern, I've been hesitant to even bring up, you know, previous legislative battles, state level legislative battles, because I feel like some of them became so toxic, and needlessly so, that it's like, let's just leave that, you know, in the past and, you know, tackle this new chapter with a fresh start or a blank slate or whatever. But one, you know, commonly raised concern that was sort of the final presented concern last time around was doing this at the state level is, you know, just not great because obviously we got a lot of states and, you know, in the limit, the final worry is always China. So it's like, well, if we have 50 different state rules,

• 1:25:06Nathan Labenz?

  then we'll never be able to get anything going, and then we'll lose to China. It seems like you have thought about that and have, you know, tried to take that into account as you've developed this. I wonder if there's any sort of... Possibility of kind of a compact of states or some sort of reciprocity between states that could kind of neutralize that objection, which I do think has some reality to it, although I think it's also kind of a smokescreen at times, but would be good to, you know, get the best of both worlds if we can. Do you see any prospect for that?

• 1:25:38Speaker 01

  Yeah, I love the idea of a compact and I would say usually the best smokescreens have a little bit of truth to them and that's a good description of this right here. I think, I agree, it should be done at the federal level. Come back to me when it is. Like, federal laws already override state laws. If any congress member wants to take my bill and do it at the federal level, please do so. I'm really happy to partner with any congress member, any staffer that's listening to this. But that objection holds a lot more weight when A, congress is doing anything, and B, once one state has passed something. No state has passed anything yet, so there's no conflict to be thought of. And we're very

• 1:26:20Speaker 01

  good at focusing on, you know, making things standard across the states. Not 100%, and that's why there should be things done at the federal level, but I can say that the, you know, I'm aware of maybe six or so states that are pushing forward frontier model regulation of some kind, and we're all on a group text. We're already talking and thinking about

• 1:26:47Nathan Labenz?

  Plus possibly a few reporters or whatever that you added in.

• 1:26:49Speaker 01

  Yeah, I had The Atlantic as far as I know, but, you know, I guess we are just a slip of the finger.

• 1:26:56Nathan Labenz?

  I had to mute that thread. You guys added me and it was just so crazy.

• 1:26:58Speaker 01

  We are honored to have you as part of it. But, but, yeah, so we're already talking and it's got to be established somewhere first. And I think all of us have shown that we're happy to copy other places. You know, I have other bills on protecting more short-term consumer aspects that are based on what Colorado passed, because hey, they got there first. Okay, let's try to unify this. As I said in the intro, I worked in tech before I entered this field. Like, I know what that compliance looks like and having one standard is way better. So, let's work towards that. But, you know, before we start trying to draw a line between two or more points, like, let's get that first point. Let's get a state on the books and then we can talk about how it should be the same with any further ones.

• 1:27:43Nathan Labenz?

  Yeah.

• 1:27:45Nathan Labenz?

  Okay. What's the process from here and maybe like, how are the dynamics shaping up? Gavin Newsom once said that the bill that shall not be named for the moment had created its own weather system. You seem relatively relaxed. Maybe that's just your good humor, but are we seeing anything similar where, you know, people are sort of mustering all their force and kind of bringing the Eye of Sauron to this debate? Or is it going to be a little more chill this time around?

• 1:28:19Speaker 01

  No one here is the Eye of Sauron. Let me be clear. I especially need to say that as someone that worked at Palantir. Anytime you get into the Lord of the Rings references. But, but, no, I think I've been very intentional to engage with industry and with industry associations. I want to hear their feedback. I want to see their red lines. I've gotten many versions of it and incorporated probably 80, 90% of what they have sent. You know, this is not, these labs are not the enemy. As I think I said at the start, I want to see more adoption of what we have already. I want to see where this field is going. The incredible potential

• 1:29:01Speaker 01

  of what we are going to have from medical discoveries, from routinizing the monotony of life, from even cyber defense, from, from what's going forward. This is really incredible technology. And I want the guardrails to make sure we get it right. And those guardrails work best when they're done in conjunction with industry. So this is not in any way sort of a fight. I mean, again, politics is often painted that way. And we say, you know, what gets the press are the things where it seems like a fight. And then there's an incentive to design it as a fight. Doesn't mean we agree on everything. But I think

• 1:29:44Speaker 01

  this has been a very collaborative process so far. The second thing I'll say is the...

• 1:29:51Speaker 01

  Rate of collaboration is probably about to speed up quite a bit. So our legislative session ends on June 17th. So we're a little more than a month away for when this this has to pass. And I think there'll be an intense focus as we get closer and closer to it. I think it's really important that people not sit on the sidelines as part of this. So if you're listening and you have suggestions to improve the bill, email my office, call my office, Boris A at NY Assembly.gov. We take all input. I would love to hear it. If you live in New York, I would encourage you, and that's New York State, not just New York City.

• 1:30:35Speaker 01

  I would encourage you to reach out to your legislators and say, hey, I support this bill and why don't you hop on as a cosponsor? And we've made that easy, actually. If you go to bit.ly/raiseactny, all one word, all lowercase. That will give you a form. You enter in your address and it'll generate an email for you to send to your assembly member and your senator. If you're not in New York, or even if you are, but you run a company, you run an organization that has a unique view on AI, you can also write in what's called a memo of support, something that helps say, this is who I am and why I care about this and why I think the state legislature should take action.

• 1:31:16Speaker 01

  And you can go to bit.ly/supportraiseny, again, all one word, all lowercase. But this is, you know, government is not a spectator sport and decisions are made by those who show up. I assure you, industry is showing up and I welcome that. If you're someone that listens to this podcast, you probably already have an interest in this field. I'd encourage you to show up as well. There's a lot of actions you can take right now to really influence what frontier model AI regulation looks like in the United States.

• 1:31:47Nathan Labenz?

  Cool. That's great. I've noted those URLs. We'll include them in the show notes.

• 1:31:52Speaker 01

  Thank you.

• 1:31:53Nathan Labenz?

  I don't know if you like to handicap your own bills, but it was striking that, you know, the vote in California was like very one-sided. And then, of course, we had the governor's veto. What do you think will be like the hard parts or are there going to be any, you know, obvious hard parts between here and making this a reality?

• 1:32:15Speaker 01

  Yes, there will be hard parts. You know, it's like any bill. It's got to pass both houses of the legislature. It probably has to go through multiple committees. It has to each individually pass before it even gets to the floor. And then it has to be signed by the governor. But I think one important thing and different thing about New York versus any other state or the federal government actually comes with that governor step. So I'm skipping ahead at this part, but I think this is useful for context for people to have if we are successful in passing it. In every other state and in the federal government, once a bill is passed, it goes to the executive who can either sign it or veto it or do nothing, and that's interpreted as signing it or vetoing it.

• 1:33:00Speaker 01

  In New York, there's a third option called a chapter amendment. And what that looks like is the governor negotiates with the sponsors of the bill for changes that they may want to see. And so they might say, oh, I like the bill, but can you change A, B, C, and D? And the sponsors say, well, we can change A and B. And then you reach some agreement to change A, B, and C. And the governor will sign the bill at the end of the year with a memo that says, you know, I'm signing this pursuant to an agreement that I've reached with the sponsors. And then we'll introduce the amendments early in the next session and pass it. So I bring that up, A, because the version that passes in June doesn't have to be the final version, right? It doesn't have to be a this or nothing.

• 1:33:44Speaker 01

  And so if you're not 100% on board with every clause one way or the other, you can still participate. But B, if there's rapid changes in the field between June and December, and we all agree that those changes have come to be and we need to act on it, we actually can. We're not locked in. We can make those amendments. And so one of the things I've talked to industry a lot about is a standard definition of a frontier model. And I keep saying, I'm happy to change this, right? We know what we're trying to capture, but is the threshold exactly right? Should it be different in this way? You know, if everyone in industry, if everyone academic comes together and says, hey, this is really what the definition should be, we're happy to swap that out and put that in there.

• 1:34:27Speaker 01

  And so everyone should keep that in mind. Different than California is we have that additional flexibility as the year goes on.

• 1:34:36Nathan Labenz?

  Cool. Well, I think if there's one safe bet we can make in this process, there will be some developments in the next six or seven months. So my guess is there will be something that will at least challenge some assumption or provoke another round of discussion. That seems like a safe bet. This has been great. Anything else you want to share or leave people with before we break for today?

• 1:34:59Speaker 01

  I really encourage people to get involved. I mean, this is a field where your voice, you, the audience, as people who are, yours as well, Nathan, but you, the audience, as people who are thinking about these topics deeply at a time when most legislators throughout the country aren't, just because it's new and it's not most people's background, really makes a difference. Especially people who are deep researchers or academics or engineers who really want to be precise in your language and in what's going forward. There is a hesitancy, I think, to be involved in politics that is often speaking with sweeping statements and maybe not always seen as, to steal your phrase from earlier, intellectually honest.

• 1:35:45Speaker 01

  But I encourage you now, embrace that nuance. You can give specific feedback. You know, this bill isn't my baby. You're allowed to tell me things to fix it. But it's really important that you express a desire for something to happen here. Because there are definitely people with an economic incentive to say no to any regulation. And if the reasonable people that want some regulation, don't speak up. Don't send that email. Don't send that memo of support. Instead of getting something you agree with 80, 90%, you're going to get absolutely nothing. And so what I want to leave everyone with is take action. Tell me all the ways you'd improve the bill. I would love that. You know, amendments are all there.

• 1:36:27Speaker 01

  But then please, please, please speak up in support because we have a real chance to make a difference here, but only if everyone gets involved.

• 1:36:35Nathan Labenz?

  Great. Well, the bill is called the Raze Act. In my humble opinion, it's not really all that much to ask. And I think you're with your background in technology and obvious technological literacy, I think you're a great avatar to represent what are some pretty modest requirements to the public. So thank you for taking the time, New York Assembly Member Alex Bores. Thank you for being part of the cognitive revolution.

• 1:36:59Speaker 01

  Thanks for having me.

• 1:37:01Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.