Governing Frontier AI, with CA Senator Scott Wiener, Author of SB 1047

Dive into the world of frontier AI development with California State Senator Scott Wiener's intricate detailing of SB 1047 bill. This podcast explores the delicate balance between accelerating AI innovation and ensuring stringent safety protocols, including potential implications for advanced AI models. Learn about the debates on government involvement, the need for independent third-party testing, and Senator Wiener's commitment to a measured approach to AI regulation. Join us for a thought-provoking discussion on the complexities of legislating emerging technologies. SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention "Turpentine" to skip the waitlist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ CHAPTERS: (00:00:00) Introduction (00:05:47) Senator Scott Wiener (00:09:38) AI policy (00:13:56) AI worldview (00:16:20) Sponsors: Oracle | Brave (00:18:28) Burden on Developers (00:26:10) Developers reaction (00:28:43) Big tech companies opposing the bill (00:31:09) Open source models (00:34:57) Sponsors: Squad | Omneky (00:36:45) Illegal activities (00:41:15) Penalties (00:44:33) Third party testing (00:48:33) Transparency requirements for model development (00:53:07) Closing thoughts

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm speaking with California State Senator Scott Wiener about SB 1047, his proposed legislation to establish safety testing and risk mitigation requirements for advanced AI models, which the bill defines, as does the Biden executive order, as models trained with 10 to the 26th flops of compute

• 0:43Nathan Labenz?

  in 2024 and other models with similar capabilities, even if they require less training compute to get there in the future. It's an admittedly complicated definition, the ambiguity of which reflects the general uncertainty about how this technology will evolve going forward. In our conversation today, Senator Wiener begins by sharing his enthusiasm for the potential of AI, as well as the story and motivation behind SB 1047. He explains how he's attempting to take a light-touch approach, not to eliminate, but at least to minimize risk, without harming prospects for continued AI innovation. He addresses some of the misconceptions he's seen in the online discussions surrounding the bill, and he responds to various good-faith objections that have been raised. For a more thorough analysis, I definitely encourage you to listen back

• 1:28Nathan Labenz?

  to our previous episode, where Nathan Calvin from the Center for AI Safety Action Fund and Dean W. Ball from the Mercatus Center joined Steve Newman and me to discuss the potential benefits, as well as the drawbacks of this bill, from a generally pro-technology point of view. I think that episode holds up very well in light of today's conversation. So where do I come out on SB 1047? Personally, I often describe myself as an adoption accelerationist, hyperscaling pauser, which means that, like Senator Wiener, I really want to see us get the incredible value of things like primary care AI doctors. And by the way, I'll soon have an episode on Google's new MedGemini model, which shows just how close we are to that exciting reality. And yet, at the same time, I do believe that we are playing with a new kind of fire

• 2:13Nathan Labenz?

  in AI, and that it's critical that we exercise caution on our way to superhuman AI strategists and scientists. SB 1047 is very much in that spirit, and for that reason, I am inclined to support it. While it's true that the small number of companies pushing the frontiers today seem to be proceeding responsibly without being forced to, and while I, as a longtime libertarian, absolutely recognize that a new government agency could become a bureaucratic nightmare, it seems to me that the stakes associated with frontier AI development are high enough and the competition is fierce enough that this bill constitutes a prudent step. Of course, there's more that government could and perhaps should do with respect to AI, and I did ask a number of questions about other possible rules meant to mitigate,

• 2:58Nathan Labenz?

  for example, arms race dynamics between leading developers. But in the end, I came away from this conversation quite sympathetic to Senator Wiener's point that you can't do everything in one bill, and that the focus of this bill is on ensuring that serious testing is done and safety mitigations are applied before models are released. As you'll hear, one change that I would still love to see, and which would turn me from a somewhat cautious to a very enthusiastic supporter of the bill, is stronger support for independent third-party testing. I believe it's critical that independent experts, who are motivated solely to find the truth, have the opportunity to thoroughly test frontier models during the training process, and to be able to report their results directly to the government

• 3:41Nathan Labenz?

  without fear of loss of access to the models or other retaliation from the developers. This is a tricky balance to strike, with open questions including who should be trusted to test, what sort of access should they have, and what remedy should the labs have if testers begin to behave in bad faith. But as challenging as those questions are, evidence is mounting that a government requirement for third-party testing is needed if we are to achieve Senator Wiener's stated goal for this bill. For one thing, my experience on the GPT-4 red team, and if you haven't already heard it, I definitely recommend my "Did I Get Sam Altman Fired?" episode for the full story of that experience, shows just how precarious an independent tester's position can be vis-à-vis the frontier developers.

• 4:24Nathan Labenz?

  And that's assuming one can get access in the first place. My background conversations with the leaders of several different AI safety review organizations suggest that access today ranges from nonexistent to extremely limited.

• 4:37Nathan Labenz?

  And recent media reports also indicate that even the UK AI Safety Institute has not been able to get advanced access to frontier models. That's despite voluntary commitments from the labs at last year's AI Safety Summit. That, I believe, needs to change, and an amended SB 1047 would seem a natural way to make that happen. In the end, whether you support or oppose this particular bill, I would hope that everyone can recognize that the governance of advanced generalist AI systems with their jagged capabilities, frontiers, and their many emergent and often quite surprising behaviors is an important and complex topic with no easy answers. With that in mind, I definitely appreciate the care with which Senator Wiener has crafted this bill and the open-minded approach he brings to the legislative

• 5:21Nathan Labenz?

  process. As always, if you're finding value in the show, please do take a moment to share it with friends, post about it on social media, or leave us a review on Apple Podcasts or Spotify. And if you have any feedback, you can contact us via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, here's my conversation on regulating frontier AI development with California State Senator and sponsor of SB 1047, Scott Wiener. California State Senator Scott Wiener, welcome to the Cognitive Revolution.

• 5:51Speaker 01

  Thank you for having me.

• 5:52Nathan Labenz?

  I'm excited for this conversation. So you are the sponsor, or I guess as it's known in California parlance, the author of the proposed legislation SB 1047, which attempts to get the government's arms around this question of what is going on the frontier of AI development, and has certainly caused a lot of consternation and attracted a lot of interest lately. So I'm really interested to get your story on how you got interested in this, what you aim to do with this legislation, and then we can get into the weeds on some of the concerns that people have and some possible ways to even make it better. How's that sound?

• 6:28Speaker 01

  Sure, it sounds great. Thanks for having me, and thanks for talking with me about about the bill. Yes, I have the honor representing San Francisco in the State Senate and I'm very proud that San Francisco is the beating heart of AI innovation. So much amazing and inspiring work is happening in San Francisco. Because I am a San Franciscan, I represent San Francisco, I am surrounded by some incredibly brilliant AI minds. And not just folks at senior levels, but frontline technologists who are doing the work every day.

• 7:10Speaker 01

  I'm thinking about, I've had a great opportunities to talk about policy issues surrounding AI. And probably about a year and a half ago, in a number of the confetti, folks in the AI world started talking to me about safety and the need for policymakers to try to get ahead of safety issues rather than playing catch up, which is what we often do when the horse is already out of the barn. And so we started to just have a lot of conversations about what that might look like and what might make sense. And how do we absolutely promote innovation? The last thing we would ever want to do, the last thing I would want to do is to keep innovation.

• 7:54Speaker 01

  How we do that and also make it safe and really address that issue. So we had an enormous number of conversations, lots of outreach. Last September, before we recess for the year, I put what we call, I put into print, we introduced a formal piece of legislation, really an outline of what we were looking at in terms of an AI safety bill. I put that out there so that it would be just floating up there for months and months. I wanted to be completely transparent about what we were doing, welcome lots of broad-based feedback. We introduced that outline, sent it around to a bunch of people. And then in February, introduced the formal bill.

• 8:39Speaker 01

  And my intentions are really threefold. First, set some basic safety and mitigation requirements that are

• 8:48Speaker 01

  Reasonable light touch and not micromanaging. Don't mention not be micromanaging this light touch. Reasonable safety and mitigation requirements that are super doable and super possible for developers of LLCs or models to accomplish. We want to focus on only the most capable models, upcoming really large, powerful models. They're beyond what's possible today with tools like Google and other tools that we have. We're talking about future really large models. And then how do we protect and foster innovation, including open source model development?

• 9:31Speaker 01

  And how do we do it in a way that takes safety to account? So those are our goals and that's how it came about.

• 9:38Nathan Labenz?

  You want to give us just a tiny bit of background on your legislative career?

• 9:43Speaker 01

  Yeah, so I'm just like a DGE, which by you moved to San Francisco from New York, so seven for same reasons, a lot of weird people came out here to find my community. Practiced law for many years, very involved in the LST two community. And ultimately ran for was elected to the San Francisco Board of Supervisors, which is like our city council back in 2010, representing the district of Carte de Vol. Used to represent did a lot of work on the board around housing, all the transportation and me and other issues and so forth. Within 2016, I was elected to the California State Senate, representing all of San Francisco and part of San Mateo County, right in the south of San Francisco.

• 10:27Speaker 01

  State Senate District, State California are huge. I represent about a million people. Then I served for a number of years as the Chair of the Senate Housing Committee and I currently serve as Chair of the Senate Gladiators Committee. And then I got a queer Gladiators in time for a massive Gladiators, which is always a lot of fun. And my focus, I'm quite best known for my work for a housing policy for trying to really area to building more homes. We've been way, way too hard to build housing. So we try to make it easier and faster to build new homes. Done a ton of work to make sure we're adequate for planning. I'm teaching a lot of work around healthcare access, including single care, healthcare, a lot of work around climate.

• 11:12Speaker 01

  Energy, issues, criminal justice reform, so forth. A note here, back in 2018, after the Trump administration got rid of net neutrality protections. You know, people know, but let's say, but though I doesn't know that neutrality is a notion that you are actually get inside where we go on the internet and DCT, Verizon and Comcast and other internet service providers are not be telling us or manipulating us into where to go or not go on the internet. And so it's something that I offer and we're able to tell for news and then it's all relapse, which is a song in some country and some people even democratizing the internet and not having concentration of power

• 11:56Speaker 01

  to making sure that the internet is a democratic place.

• 12:00Nathan Labenz?

  Thank you. So for this bill, one thing that is striking is that it really does focus on the frontier. And it leaves, I would say, a number of questions which you probably agree are important for another debate at another time. Those would be things like the future of professional licensing, the overall impact on the workforce, the potential for algorithmic bias or discrimination. None of those are really addressed in the bill at all. So question would be, why? Is that something you think you'll come back to later? And how have you assembled a coalition to support this, given that from what I understand of the coalition, I would imagine some of those folks would be pretty concerned with those issues as well.

• 12:44Speaker 01

  Yeah. And one thing it's important to know about this about AI policy, about any policy is you can't do everything in one bill. And so we have a number of members of let's just say, who have been or working on AI related issues and various bills that are moving forward now. So our bill, S B seven four seven focuses on a safety evaluation to mitigate for the. Very largest models that are being developed. There is a bill that's coming in the state assembly by my colleague, some member Rebecca Darkey, relating to algorithmic discrimination.

• 13:23Speaker 01

  There are there are a few different bills around watermarking of AI generated images. There are a couple of bills on AI generated revenge porn. There are bills around government contracting around AI related services and government use of AI. And there are others as well, but there are a few AI bills moving forward. And SB ten forty seven is simply one of them relating to safety and privacy.

• 13:57Nathan Labenz?

  Okay, do you have a established position on these other bills? Is there any broader context on your AI worldview that you'd want to share?

• 14:06Speaker 01

  Yeah, generally I'm a supporter of artificial intelligence. It has so much potential to make the world a better place, to make people's day to day lives better, to address some of the biggest issues of our time around climate or various huge healthcare issues, that the potential is limitless in a lot of ways. And I am so excited about what AI can do to make huge lives better. But we also know that there are impacts that we need to be mindful of. And so I am supportive of addressing algorithmic discrimination.

• 14:45Speaker 01

  The last thing we need is to have AI make the discrimination that exists in the world even more pronounced. And so that needs to be addressed. I think it's important for people to be able to know if an image is fake or not. There could be so much disinformation around audio and visual fakes. And deepfakes are a real issue, we need to address it. And in terms of impacts on the workforce, it's a really hard issue. Technology always impacts or often impacts job classifications and workforce. But we're looking at impacts on the workforce at a very large scale.

• 15:29Speaker 01

  And I know that there in a theoretical perfect world, if AI made work or made it possible for for less work to exist, in an ideal world, the benefits would be spread around to everyone and everyone could just work less and have more free time. But we know that in this society on this work in this world, we are very bad about spreading around benefits. And what I don't want to see happen is that AI generates huge economic benefits that are enjoyed by a few, whereas most people are made worse off because they don't ever have jobs. We need to be intentional about what the future looks like in terms of AI's impact on the workforce. That is a huge mega issue.

• 16:14Speaker 01

  It's not going to be solved by a bill or one bill, but that's something we need to focus on.

• 16:21Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. You said the goal is to have a focus on the frontier, the goal is to have a light touch. How would you describe the burden that the law would place on frontier developers? And how would you say they have reacted to it so far? It's been striking to me that I haven't really heard like an official position from the companies that would seem most likely to be directly impacted in the near term.

• 16:51Speaker 01

  Sure. So I'll describe what the bill does, and I'll also describe after that all the elements that I rejected, that some people wanted to put in the bill. My goal here is to be light touch, is to not to manage, not to, you know, undo burdens upon people. That's something I don't like passing laws that make people's lives harder just for the sake of making them harder. That's not what that's not who I am, not what I ever wanted. I do, I want my the laws that have my name on them to actually make the world a better place. So the bill that applies to models that are

• 17:36Speaker 01

  size threshold, we use the same size threshold that the Biden executive order on AI to use, which is 10 and 26 plus. And because we don't want to tie everything to where we are in 2024, we want this to be flexible over time. If you're developing that model of that scale, then before you train, The model and before you finalize it and make it available to other people, you need to perform a safety evaluation on the bill. There are various kinds of these evaluations, red teaming is one of them, but there are others as well. So you have to perform that evaluation.

• 18:19Speaker 01

  If the evaluation, the testing shows that there is a significant risk to model leaving a catastrophic harm, I'll evaluate how we define catastrophic harm. Then you need to take reasonable mitigations to reduce that risk, not to eliminate risk. Eliminating risk in life, that is usually not possible. And when we try to eliminate risk, that will take a lot of the brilliant innovation out. We'll all be sitting in a coffin in our basement, doing nothing. Life is about risk. And so if there, we want people to at least try to mitigate or reduce those risks. When we talk about catastrophic harms, how we define it in the bill is that it will cause,

• 19:00Speaker 01

  lead to the deployment or development, deployment of nuclear, chemical, biological weapons, that it will cause harm, damage to critical infrastructure, $500 million worth, where that it will create, trigger some sort of cybercrime or some sort of crime. That would be a crime that a human did that causing $500 million or more in damage. So we're not talking about small things. We're talking about large scale destruction and damage. Some people say that thresholds should be higher. Some people say, wait, why 500 million? Is a 50 million enough? And that's top of the conversation we could have. And then if there are other harms that are sort of equivalent in scale and destruction to those.

• 19:45Speaker 01

  And when someone engages in the safety evaluation, you then have to certify to the Department of Technology that you've done so. And if then one of those harms occurs from your model, then, and you have not. Done the safety testing. Or you did the safety testing and it showed that there was like really bad, it was likely to happen. You didn't do anything about it, that the Attorney General can sue you. I also just want to note that there are a number of things that I rejected, including the bill. Cause I wanted this to be like touch. But first is there are some who wanted us to enable the court private right of action so that anyone could file a lawsuit against the developer if something were to happen.

• 20:30Speaker 01

  Um, I did not accept that idea. We limited enforcement to the Attorney General of California. To with this limited resources, so they will focus their enforcement on the really bad actors. Uh, not just on someone. Uh, there are people who uh, wanted us to uh, have a licensing requirement so that you can't train or release any of these, a lot of these models until the less the state can commission them and gives you a license. I did not want to go in that direction, but I don't want the state uh, to be in the middle um, of all that. Um, some people wanted us to ban certain kinds of models, so we didn't go there. And some people wanted, let's call it strict liability.

• 21:15Speaker 01

  Strict liability means if you cause a harm, you're automatically liable, whether or not you acted reasonably. I rejected that as well. Uh, that's basically what this element of the bill does and some of the items that we did not. But I do just want to address one thing, because there was some conversations on Twitter saying such very inaccurate things about the bill. First, they said the bill is being fast-tracked, which is false, the bill. We first analyzed this nine months ago and it's on the regular slow goat. And so there's time of opportunity for additional dialogues, for amendments.

• 21:51Speaker 01

  And it's always not a gas trap. It's a super transparent process. And the other piece is always some people say, oh, this developers are in a prison for this. You don't make honest mistakes, you go to prison, which is also false. The only criminal aspect of the bill is about perjury. If you literally intentionally maliciously lie to the government, that's perjury. Just like it would be if you lie on a driver's license application or any other documents with the government. You actually intentionally lie, misrepresent, that can be charged as perjury. It's not about if you could fake mistake or an honest mistake or you submit something that was even something sloppy.

• 22:36Speaker 01

  That would not rise to that level. So anyway, that's, I'll stop there.

• 22:41Nathan Labenz?

  Yeah, so that's an important distinction. So just to make sure I get it in layman's terms. Basically, obviously, these frontier models are very unusual technology in that we don't really know what they can do until they've been created. And then we're still figuring out new things and better ways to prompt GPT-4 a full year after its public release, eighteen months after it was finished training. So there's this whole process. What you're saying there is that a developer is responsible for doing some amount of reasonable testing to try to get their own arms around what can this thing do? And then it's their job to report, obviously, honestly to the government that we have tested this.

• 23:25Nathan Labenz?

  Again, I'm a little fuzzy on like exactly how much they're supposed to test, how exhaustive that's supposed to be. Maybe you can clarify that for me, but if they do an honest testing and reporting, and they say, hey, we don't think we have a problem here. We can go ahead and distribute this model. And then something happens, they could be sued would be the base case. And only if it was later found that they actually knew better and were covering up information that they had willfully not reported, would they be subject to criminal liability. Do I have that right?

• 23:55Speaker 01

  Yes, but the lawsuit could only be by the attorneys out of California, not by other random people. So it's very limited, very focused, supported by one law enforcement agency.

• 24:08Nathan Labenz?

  Gotcha. Okay. So how have the developers reacted to this? I feel like there's a lot of crosstalk in that. Dimension as well. Some people are of course saying, oh, this is a ploy by the developers to create a regulatory capture environment. Others are saying that it's going to prevent them from doing what they want to do. What has your conversation with the developers been like?

• 24:31Speaker 01

  It's been a wide array. We're talking to all sorts of people. And there are a lot of people who, some publicly, but many quietly say, this is the right approach. Thanks for doing this. Largely, they don't want to be public about it. I'm learning a lot about the politics of Silicon Valley. And there are a lot of these people want to be funded. They want, they, and so there are a lot of people who don't want to be public about their support. Some do. And we've had some very prominent folks in the AI world who are supporting the bill. We've had a lot of meetings, both with the large labs, with the mega tech companies, but also with smaller developers.

• 25:10Speaker 01

  The large tech companies, there's been this narrative that this is about regulatory capture and trying to box out small folks in the startup. That is completely untrue. And it's not what the bill will in any way do. None of the big tech companies are supporting the bill. We're in conversation with them to try to get feedback, but none of them are supporting the bill. And in fact, TechNet, which is the trade association for all the big tech companies, the Metas and Googles, Amazon, Cerebro, is opposing the bill. The ideas in this bill did not come from big tech companies. It actually came from a lot of technologists and folks who focus on AI safety. We have a number of startups that are supporting the bill.

• 25:52Speaker 01

  I think that also speaks volumes. A lot of folks were sort of watching and staying away. We have a particular conversation with the open source community. Folks who are, think, who believe that open source should be treated differently from other models. And we're in active conversation with folks in that community. I think open source has enormous potential value. In terms of a stirring innovation and democratizing access to these models. But there are of course also risks. If you release a very powerful open source model, people could do amazing things with it. It could also potentially do harmful things. So that's a continuing conversation.

• 26:36Speaker 01

  We definitely are looking to undermine open source.

• 26:41Nathan Labenz?

  Objections that I've heard, you mentioned that the big tech companies in their network are opposing. One of the other objections I've heard has been that, well, all the leading developers are basically doing this already anyway. So what's the point? They've all put out model cards and walked us through, here's what we're doing for testing. Anthropic's got their responsible scaling policy. OpenAI has answered that. Google's getting there too. Even Meta with their open source approach has done pretty thorough testing. So interested to hear, what are they saying that they find a problem with? And how would you answer the opposite end of that says, if all the big guys are doing this, why do we need to make it a law?

• 27:17Speaker 01

  Yeah, just to be clear, the big tech companies, they have not come out individually. So we're having conversations with, I think, all of them. And the big folks with the small folks, with the academics, with everyone, it's super open door. And yeah, there is testing. So for the folks who are doing reasonable good testing, this still will have a very limited impact because they're already doing it. And so it's not an issue for them. There are other labs that we're aware of who are maybe not doing the best testing. There, we've heard some, this has been, this is not clear, so I'm not seeing this.

• 27:59Speaker 01

  Some folks who have concerns about Meta's safety testing, whether it's really where it is to be. And so I'm saying, again, just repeating what I've seen online. So there are diversities about whether all of the large labs are doing testing to the degree they need to. And I've also found in life, like it comes to any industry, any ecosystem, it's good to have minimum standards and not just trust that every lab is going to do the right thing. There are plenty of labs that will do the right thing. They're taking safety very seriously. We'll go above and beyond on testing and on guardrails and mitigations.

• 28:43Speaker 01

  But that doesn't mean that they all will. And I think it's good to have a level field where there are the minimum standard. And again, this is light touch. Safety rules. We're not letting the micro vanish, but we want to make sure everyone, if you're creating a model of this scale, and these are huge, powerful models, just at least do that baseline safety testing.

• 29:08Nathan Labenz?

  On the open source question in particular, this is obviously one of the hottest topics. The cause around models that have similar power to what 10 to the 26 would get you in 2024 has a lot of people projecting out a couple of years and saying, look, we're getting a lot better at this stuff. It's going to become pretty affordable to create something of that power over the next couple of years. And so the threshold effectively in terms of like dollar budget falls from the rare air today where we're talking hundreds of millions of dollars and relatively few companies can do it to maybe a couple of years from now, just a couple of million dollars and maybe like a ton of companies could do it, or even groups or clubs or whatever could rally that many resources.

• 29:52Nathan Labenz?

  It seems to me like that is right unless there's some sort of conceptual breakthrough that would allow people to really definitively say that their stuff is safe. Is that sort of a long-term bullet you're willing to bite as of right now, where you basically would just say, hey, look, if there's no way to say that this stuff is safe, then you really just can't open source it into the public. Is that kind of your default position and hope that there is a breakthrough?

• 30:17Speaker 01

  We know that no one can predict the future, but we know that right now and in the future, I think we can see that that it's going to, it's going to be incredibly expensive to develop models of this scale, this magnitude. And so we're talking about significant undertaking to develop models of this scale. And so we think it's reasonable, so it's just do a safety test to, at least again, not eliminating risk to at least try to mitigate the risk. With respect to open source in particular, we want to protect open source in very, very clear. And we're not looking to eliminate open source. That's not what I want to do. It's not what we're going to do.

• 30:59Speaker 01

  Uh, we're working right now, as we speak, in good faith with a lot of people who are a lot of really smart people in the open source space to try to figure out how we can. Address some their concerns in a way that still promotes safety in the open source context. And we know that when people, if you take a very powerful open source model, when you're building on that, there's a good chance that what you're building on top of it may be very small in terms of turning that model into some sort of real world application. And so we do think that it's the model developer is best positioned to do that safety analysis.

• 31:44Speaker 01

  However, we are really actively receiving feedback from open source developers and experts to make sure that we are, again, I don't like, I'm not looking to create any kind of structure that people can't meet. So we're going to continue to work on the open source issue. And I don't want to continue to have an open door all on it.

• 32:05Nathan Labenz?

  One of the best ideas I've heard there would be the idea of maybe getting a little more precise on how much more you can do on an open source model before the responsibility becomes yours. Right now, if I read the legislation correctly, it's like, open source developer puts out a model, anybody does downstream additional training or whatever on, that's still a derivative model. And so the original open source developer retains the responsibility. Some of the ideas I've heard have been like, if you do 10% additional compute, then maybe you should now own the ball. The original open source developer can be off the hook. Is that the kind of thing that you're entertaining now?

• 32:44Speaker 01

  No, I think that's a fair conversation and we're hoping to exploring that and other ideas about with the open source folks.

• 32:54Nathan Labenz?

  Okay.

• 32:55Speaker 01

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 33:00Nathan Labenz?

  One other idea that's just like super simple is what if we just made it illegal to distribute AIs that do certain things? What's the fault in that idea?

• 33:10Speaker 01

  Yeah, there's certainly a case to be made that there are certain types of activities that that question not be allowed. I don't think we want AIs to be creating biological weapons. For example, it's probably the most commonly discussed idea, but we don't really know, first of all, what the capabilities of these models are going to be in two years, five years, 10 years, even if we decided there are certain things we want to ban, we may not even know. What it is, uh, we would want to be up. And so we decided in this bill, rather than saying what you can't develop in terms of capabilities, let's, let's require safety evaluation. And again, I'm not going to pretend that this bill is going to solve all the central problems

• 33:55Speaker 01

  that could ever result from, from AI models. But I think it at least takes that step of requiring some introspection and evaluation. And then to discover, oh, wait, this is going to generate biological weapons, trying to mitigate that. And the conversation about whether to restrict or ban particular AI capabilities, that will be an ongoing conversation, but it's certainly a fair conversation to have.

• 34:21Nathan Labenz?

  I was surprised also that there's no AI sort of having to identify itself when it goes out into the world. This evolved, no Harari, the first rule should be AI must identify itself as AI. Is that just something that you think will be handled by other legislation?

• 34:36Speaker 01

  And I sort of, I do have a list of AI goals in front of me. I think that might be a bill pending this year. I can't remember. But that is certainly an active topic of, of conversation. And I think it's in a, that's, there's a dollar in that this year, but I could be wrong about that. But that's certainly an important issue.

• 34:57Nathan Labenz?

  I think one of the big ideas that people really worry about in AI broadly is the idea of an arms race between countries, certainly, but even between firms within the United States. The bill obviously insists on a certain amount of testing. But in a world where it's really easy to switch from one language model to another, there is, I think, risk of a sort of winner take all dynamic between the companies where, you know, giving the testing kind of the shortest possible window that they could give it starts to maybe become attractive. Because I think we might even see this like in the next couple of weeks here. Google's going to have their event, they're going to release something and then it's expected that like OpenAI is going to try to come back over the top and steal their thunder.

• 35:43Nathan Labenz?

  And so you can imagine inside the labs, the conversation might be like, look, we're launching at this date when we can most effectively compete our rival. And you get your testing done by then. And whatever we can do by then, like we'll call it reasonable. Do you worry about that at all? And do you have any ideas or has there been any conversation about things that legislation could do to try to mitigate this sort of winner take all AI arms race dynamic that seems like it is starting to take shape?

• 36:13Speaker 01

  You mean in terms of people cutting corners?

• 36:16Nathan Labenz?

  Yeah, exactly.

• 36:18Speaker 01

  Ultimately, under this bill, if someone cuts corners on safety testing, those really shoddy testing receives problems and so sort of tangentially tends not to see them, they're going to have to file a certification and they end up making it be that if something terrible happens, that could lead to liability. The Attorney General could file a lawsuit. Yeah, people, especially, again, this is light touch. We're not micromanaging. We're not just setting broad parameters and people can certainly violate the law, act in shoddy, irresponsible ways if they want to, but then there's going to be a risk to them. Maybe

• 37:02Speaker 01

  they'll get away with it. The best solution, life often works that way. They're trying from people to problematic things and get away with it, but sometimes they don't. And so they're, and we're talking about catastrophic harms. So they, developers will need to be mindful. And I, I think ultimately, a lot of developers want to do the right thing. They want, they don't want to release a lot of that, that's going to cause catastrophic harm. And so I, I think that we will have good compliance.

• 37:32Nathan Labenz?

  The penalties are, how would you describe the penalties? Cause I, I see in the bill, like a 10 to 30% sort of fine, 10 to 30% of what it costs to train the model, I believe. And then I think there's like additional sort of possibility for kind of punitive damages or whatever. But I could imagine a situation where you're at. Google, for example, and you're like, man, OpenAI is eating our lunch. How much did this thing cost to train again? Oh, it costs $500 million to train. Okay, that means we could be on the hook for 150. It's worth it, right? Do you think that the the financial penalties sort of probabilistic and deferred into the future as they are enough to be really persuasive to these companies?

• 38:13Speaker 01

  I think so, and especially punitive damages. It is punitive damages require a very high, it's a high threshold of malice. And if punitive damages are triggered, those are typically based on your net worth. And so punitive damages fought against me as a public servant, who doesn't have a lot of money, are going to look very different than punitive damages against a trillion dollar corporation or whatever the size is. And so I think the incentives are good in the bill.

• 38:46Nathan Labenz?

  I started my career very briefly in the mortgage space, and it just happened to be at the time when the mortgage industry blew up the American economy. And then it became clear that while there were a lot of people nominally watching after the behavior of the lenders and all the different steps in the value chain, like that had eroded to be more of a rubber stamp. And it is a big part of what the safety community most worries about. It's, okay, these people are going to be under pressure. It's such an easy change to switch from one API call to the other. I can switch from OpenAI to Google to Anthropic, literally in, you know, one line of code. So it does create these sort of, you have the best model, like you win the day dynamics. And then there's this sense of, okay, we're racing.

• 39:31Nathan Labenz?

  Maybe we want to do this, but do we really want to do it? Do we really want to find everything we could find? Do we really want to take all the time that we could take? It seems like there is likely to be a space where there would, it would be plausible for the developers to have a reasonable defense. Look at all the things we did. This is like very reasonable. But then somebody who's like really in the know would say, think about all the things you didn't do. And here's all the things that I really think you should have done and you didn't, but it's still reasonable. This leads me to, I think, one of the most common suggestions that I heard in preparing for this, which is a more forceful requirement for truly independent third-party testing.

• 40:12Nathan Labenz?

  What has there been any conversation around that sort of thing where who provides that testing, under what circumstances, how long they have, exactly what sort of access they have, there's a lot of particulars there. But I do worry along with a lot of these safety people that the motivation or the sort of standard of reasonableness could be met. And yet we might want a higher standard if we're really talking about catastrophic, or as you well know, the safety community even worries about extinction level risks. Is reasonable enough? Or should we sort of say, hey, you have to allow in these sort of red hat testers to go in and dig deeper than you might dig on your own?

• 40:50Speaker 01

  Now, the bill, and again, we are walking a fine year where we want to have good safety protocols without being too intrusive, right? And we're already there are people who are criticizing us for being too intrusive. I don't think we are at all. Other people are saying it should be more intrusive. There, as you know, are better than I do. There are some philosophical divides within that AI world. Right now, on this issue, the bill calls to developers to use third-party testing, quote, when appropriate. And I understand that is a little vague and certainly an issue we'll continue to take a look at. We're open

• 41:31Speaker 01

  to the possibility of a greater role for third-party testing. But we also want to know more about who those third-party testers would be. And I think as you alluded to earlier, and I didn't allude, you said, it's just some big Italian firm that comes in and checks the box. Or is it real? Our goal here is just to have good testing. And we're open to different ways of approaching that. Again, with that, I know you're going to get sick of me using this phrase, but it's real. As long as it is late touch, that is our goal for it to micromanage. And there are people who would like custom micromattes, right? They want people who want you to have to get permission from the government and get a license before you train or or anyway release

• 42:15Speaker 01

  at a model of this scale. And I I respect that point of view, but that's not where we focused on this bill.

• 42:22Nathan Labenz?

  Yeah, I do think it's a tough one to figure out exactly how you would specify who the third parties should be. And again, exactly what kind of access they should have. I had a personal experience as a member of the GPT four red team. And I was really taken aback by what a leap it was, and the public hadn't really seen anything like it. My feeling at the time was that the testing that they were doing appeared to be inadequate, and I became sort of a lost in the wilderness, like under NDA on the one hand, like not trying to blow up their spot on the other hand, but like feeling like I was one of just a couple of people in the world that had the time to devote. And I literally dropped everything else I was doing and worked on it full time for a while. And I felt like, where do I go?

• 43:07Nathan Labenz?

  One of the problems that I had was that I was totally at the pleasure of OpenAI. I do think one thing that would be really useful would be some protections for the third party testers, some perhaps like accreditation or some status that they need to achieve, but also some ability for them to report their findings without being at the risk of immediately being cut off because the access dance that they have with the developers is a really tricky one.

• 43:35Speaker 01

  Yeah, I think so. And it's totally different. I authored a law that was passed in sign last year. It was a law to require large corporations to disclose their carbon emissions, including from their supply chain. It was a huge fight with a lot of the large business associations. We got it passed and the government signed the law. It puts California in the forefront. And in that law, we require these disclosures and exposures have to be audited. At the California Resource Board, it certifies the auditors that are qualified and trusted to do that auditing, different contexts, not just subject matter, but also because carbon emissions are not necessarily like trade secrets and but not.

• 44:20Speaker 01

  I guess it could be, but are typically not particularly sensitive in terms of some sort of confidential part of a business model or a technology. Here, obviously, there's a much higher sensitivity in terms of this is like the core products of the business and there's a lot of confidentiality issues as well. So it's a little touchy, but I think that this is an absolute word to talk to the conversation about how to make sure the testing is as solid as possible.

• 44:50Nathan Labenz?

  What about sort of a similar transparency requirement for model development as well? What if you were to say you have to just disclose how big your training runs are going to be? If you're going to do two times 10 to 26, okay, but you got to put it on record. That would maybe help address these sort of race dynamics where the different developers don't know just how fast the other companies are going, how big. And you might also imagine extending that to things that people have seen models do. I do think there is definitely a place for trade secrets. I don't think total transparency makes sense, if only because then we're also sharing all our secrets with all the governments of the world, which we probably don't want to do. But I've often thought, does it really harm the company if somebody says, hey, I saw

• 45:35Nathan Labenz?

  an AI do X and it freaked me out? Because I was under an NDA where I couldn't even disclose that sort of thing. And people have been fired, of course, for saying that they think their AIs are conscious and whatnot. I don't necessarily think they are, but I also don't rule it out. It certainly seems like something we should be discussing. I wonder what you think about these sort of, are there ways to put strategic transparency, either requirements on the company or sort of opportunities for very careful, like narrow, but hopefully illuminating disclosure for employees or testers that could help everybody at least get a sense for kind of what's going on across the environment?

• 46:17Speaker 01

  Is that possible? Yes. And is that worthy of a conversation? Yes. In this bill, we're really focused on just creating, which does not exist now, baseline requirements to do safety testing, which a number of the labs are already doing. And some are probably doing better and more consistently and fairly than others. And we want to make sure that labs that are creating these huge models are doing that baseline testing. Are there other things that may be reasonable to put on top of that potentially? But we're really focused on let's do this, take this important step. That is already going to be controversial and we're already seeing that controversy.

• 47:02Speaker 01

  So that's what we're focused.

• 47:04Nathan Labenz?

  Yeah, makes sense. So on the core testing concept, I hear you. The bottom line seems to be, we want to make sure that this testing is happening. We want to make sure that it's good quality testing. Want to create the right incentives for that. Open this, you've expressed several times to discussion about potentially a bigger role for third parties or a more kind of forceful insistence on that. Where should people come to have that conversation with you, particularly those that are motivated to do that testing that feel like they have something to contribute, but are currently boxed out of contributing in the way that they would like to?

• 47:40Speaker 01

  Yeah, so people can reach out to my office. We're super easy to find. They can also message me on Twitter. Scott underscore waiter, I have before Ian waiter. I can message me there. You can reach out to office. We really welcome feedback and ideas. And people have been very gracious and helpful in pointing out maybe things in the bill that could be better or things for columns that we may not have anticipated. And we very much welcome that feedback. You can never guarantee that everyone's going to be in complete agreement with everything in any bill that I left there. But I do really value feedback to make the bill as good as it can be. And there are times, I've had bills where there are

• 48:23Speaker 01

  There was one philosophy of legislating, saying, if you're opposing my bill, then I'm not going to listen to anything you have to say. I don't subscribe to that philosophy. My view is that even if you are fighting me on a bill, if you then come forward and say, hey, there's somebody you missed, and this is going to play out in a problematic way. There are times when I'll say, oh, my God, yeah, you're totally right. Thank you for bringing that to my attention. We changed the bill. Even if the person's going to continue to oppose it, because my take, because I just want my bills to be as good as they can be.

• 48:53Nathan Labenz?

  I appreciate what a complicated situation this is, how much uncertainty there is on so many of the key questions. And I know you're balancing a lot in terms of different constituencies and trying to create something that has a positive effect and still has a light touch. I definitely appreciate the fact that you're focused on these frontier tales, sometimes dismissed as speculative, but I don't think wisely dismissed as speculative risks. I appreciate the work that you're putting into this and the openness that you've expressed to further input as well. Any other closing thoughts from you?

• 49:27Speaker 01

  I appreciate that. First of all, I'm excited about what AI has to offer. I appreciate the engagement that we're seeing. I would also just, long ago I was asked, my only request is, this is true in length, not by AI policy, but just about our bills. But if you see something on Twitter or elsewhere online about the bill, just please don't assume that's what the bill does. We've seen some real inaccurate information about the bill on Twitter in particular over the last week. I'm not saying it's all malicious. There are some people who just see things or hear things and then just post about it. And feel free to reach out to us and ask, because we're happy to ask some questions.

• 50:12Nathan Labenz?

  California State Senator Scott Wiener, sponsor of SB 1047. Thank you for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.