What the Ex-OpenAI Safety Employees Are Worried About — With William Saunders and Lawrence Lessig

William Saunders is an ex-OpenAI Superallignment team member. Lawrence Lessig is a professor of Law and Leadership at Harvard Law School. The two come on to discuss what's troubling ex-OpenAI safety team members. We discuss whether the Saudners' former team saw something secret and damning inside OpenAI, or whether it was a general cultural issue. And then, we talk about the 'Right to Warn' a policy that would give AI insiders a right to share concerning developments with third parties without fear of reprisal. Tune in for a revealing look into the eye of a storm brewing in the AI community. ---- You can subscribe to Big Technology Premium for 25% off at https://bit.ly/bigtechnology Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Questions? Feedback? Write to: [email protected] Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  An ex-OpenAI super alignment team member joins us to share his concerns about the company's trajectory, along with his lawyer, the Harvard Law professor Lawrence Lessig, who will shed light on the lack of protections for those who speak out. All that and more is coming up right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, head of enterprise business solutions at Voya Financial, to go inside

• 0:41Alex Kantrowitz?

  a large-scale agentic CX transformation, including the business decisions, trade-offs, and lessons along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation.

• 1:20Alex Kantrowitz?

  Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We have a great show for you today. We're finally going to speak with some of the people behind some of the concerns you've been hearing about the trajectory of OpenAI, especially with regard to the alignment work within the company, or really the super alignment work. So we're joined today by a former member of that super alignment team, William Saunders is here. Welcome, William.

• 1:45Speaker 01

  Thanks for having me on.

• 1:47Alex Kantrowitz?

  Thanks for being here. And it's my great pleasure to welcome Larry Lessig back to the show. He's a professor of law and leadership at Harvard Law School, and he's also representing William pro bono here as he goes, and I guess many of his colleagues as well, as he goes and speaks out about these issues. Welcome, Larry.

• 2:03Speaker 02

  Great to be back.

• 2:04Alex Kantrowitz?

  Let's begin just talking a little bit about the vibe within OpenAI. William, you left a few months ago. Take us a little bit inside the company so we can understand the environment from which you're coming out of.

• 2:15Speaker 01

  During my three years at OpenAI, I would sometimes ask myself a question. Was the path that like OpenAI was on more like the Apollo program or more like the Titanic? And you know, the Apollo program was about like carefully predicting and assessing risks and doing groundbreaking science, building in enough safeguards to be able to like successfully bring astronauts to the moon. And then even when big problems happened like Apollo 13, they had enough sort of like redundancy and were able to adapt to the situation in order to bring everyone back safely. Whereas the Titanic, you know, came out of this competitive

• 2:59Speaker 01

  race between companies to keep building bigger and bigger ships and sort of ships that were bigger than the regulations had been designed for. Lots of work went into making the ship safe and, and, and, you know, building watertight compartments so that they could say that it was unsinkable. But at the same time, there weren't enough lifeboats for everyone. And so when disaster struck, you know, a lot of people died. And OpenAI claimed that their mission was to build safe and beneficial AGI. And I thought that this would mean that they would prioritize, you know, putting safety first. But over time, it started to really feel like the decisions being made by leadership were more like the White Star

• 3:40Speaker 01

  Line, building the Titanic, prioritizing getting out newer, shinier products than, you know, really feeling like NASA during the days of the Apollo program. And I really didn't want to end up working on the Titanic of AI. And so that's why I resigned.

• 3:59Alex Kantrowitz?

  It's kind of interesting that you use those examples and not the Manhattan Project, which is kind of the one that's been, people have brought up, like the power and the destruction potential of nuclear energy has been, been something that's been talked about. And I don't know if Sam has compared himself to Oppenheimer, but I've definitely heard some people make those comparisons. Why did you shy away from that? As long as we're going through the analogy lens, why did you shy away from that one?

• 4:25Speaker 01

  I think that is another valid analogy. You know, I think this example of the Titanic, you know, makes it sort of clear that there were like Again, like a, you know, safety decisions that could have been made better or something.

• 4:45Alex Kantrowitz?

  Right.

• 4:46Speaker 01

  I think the Manhattan Project is more the analogy for like the scope of impact that this technology could have. The companies are claiming this technology will have and are raising, you know, billions and billions of dollars based on this premise of the scope of impact. I think it's also a tale of sort of scientists who set out building a technology wanting to do something good in the world, right? The, the, the reason the Manhattan Project got started is because science, scientists looked at what was coming, what was possible, and were terrified that, you know, Adolf Hitler would get the bomb, right? And that this would be, you know, absolutely terrible for the world. And that's why they

• 5:27Speaker 01

  went to the Americans. But somewhere along the way, uh, you know, at some point, like Hitler, you know, was dead, Germany had surrendered, and yet the project went on. And again, that's another situation that like, I would really not like to find myself in.

• 5:43Alex Kantrowitz?

  Right. And I was going to ask you whether you think OpenAI is a product or a research company? And obviously, it's both. But the question is what leads? And reading between the lines, or maybe just hearing you explicitly, your belief is it's a product company? Is that right?

• 5:57Speaker 01

  It's, it's a bit different from just trying to make like the products that are most useful today. It's coupled with a vision of the research for how to build towards something called AGI, or artificial general intelligence, which is, you know, building systems that are, you know, as smart as most humans and can do most economically valuable work that humans can do, AKA can like do most jobs of people. And so it's like, the combination of these visions or something is, is, is something that I'm more concerned about. It's where the, that they're like building on a trajectory where they are going to do, like

• 6:42Speaker 01

  AGI, as stated, will be a tremendous change to the world. And so it's, they're on this trajectory to like change the world. And yet, when, you know, they, they release things, their priorities are more like a product company. And I think that is like what is most unsettling.

• 7:00Alex Kantrowitz?

  You came out with this letter recently talking about how there needs to be a right for people within these companies to warn the public about some of the concerns they have. Here's a quote from the letter. "There is no effective government oversight of these corporations. Current or former, current and former employees are among the few people who can hold them accountable to the public, yet broad confidentiality agreements block us from voicing our concerns." So, we're gonna get into those agreements in a bit with Larry, but you obviously signed the agreement, you're here talking today. The main question I think the public has from you and the people who have signed this letter is, did you see something? Like, did you see something concerning enough inside the company to merit speaking out here?

• 7:45Alex Kantrowitz?

  And if so, what was it?

• 7:47Speaker 01

  To set some things clear, if there was, if, if there was a group of people that I knew were being seriously harmed by this technology, first, I still really hope that OpenAI would like do the right thing and address this if this was like a very clear-cut case. I also personally would, you know, ignore any sort of like considerations of how I might be retaliated against, and I would talk about that plainly. So that's not what I was seeing. It's also, I don't think that I was working on the Titanic, I don't think that GPT-4 was the Titanic.

• 8:23Speaker 01

  I more am afraid that like GPT-5, or GPT-6, or GPT-7 might be the Titanic in, in, in this analogy. And so, I can talk about, um, maybe like a couple of areas here. So, one is the, a former colleague on the super alignment team, Leopold Ashenbrenner, has like talked on a podcast about how he, you know, was like looking, you know, asking some questions about like how the internal security was working at the company. And then, uh, you know, wrote a document containing some concerns, he shared this

• 9:05Speaker 01

  around, um, he then got reprimanded, uh, because, you know, some of the concerns offended some people, and you know, personally, I would have like written it in a different way. And I think, but the disturbing, the disturbing part of this was that, you know, Um, the only response was reprimanding the person who raised these concerns, right? It might be reasonable to reprimand the person and then be like, okay, but these parts of these concerns, we're taking these seriously and we're going to address them.

• 9:35Alex Kantrowitz?

  Right.

• 9:35Speaker 01

  That was not the response. And then he later talked about like, this was one of the reasons uh, that was offered for him being fired.

• 9:42Alex Kantrowitz?

  So I guess what I'm narrowing, I'll let you do the other example in a second, but like, what I'm narrowing down on is the people that have raised concerns within the super alignment team, it's not like they've all seen some powerful, dangerous technology that they don't believe uh, OpenAI like in the immediate term, they don't believe OpenAI is going to handle appropriately. It's more of like, what is the path in the future? And that's where this right to warn thing comes down. I'll say at the beginning, we're going to get into right to warn. I'm fully in favor of this right to warn, and I'm really glad that you brought it up. But I think that it's important for the public just to establish that like, inside OpenAI today, there's this group that's just left. It's not like, there was this question, like, did Ilya see something, right? Did you guys see something? It's not that there's

• 10:26Alex Kantrowitz?

  something immediate and harmful that you've seen. It's more of like, you're concerned about the path that this company can go down. Is that right?

• 10:34Speaker 01

  Yeah. Yeah. And I think, you know, this right to warn, this is a right to warn responsibly. This is not a right to like, cause like unnecessary panic or something. And I am, most of my research was driven by, again, concerns along this trajectory that the companies are going along, have demonstrated progress on, and are raising billions of dollars to keep going along this trajectory. But I do think there could be like, there could be things happening today that we don't know about. And so, um, the, the sort of scenario that I'm worried about happening today is suppose that there's some group of people that wants to, you know, spread a lot of disinformation on, on social media. Let's say they want to manipulate an election, or they want to, you know, incite violence against a minority ethnic group.

• 11:19Speaker 01

  And let's say that they're in a non-English speaking country. And so, you know, again, most of the people at OpenAI speak English, most of the alignment work is done in English. And so, like, it would be, you know, somewhat harder for the company to notice this. Now, like, the models are, like, safety-trained to, like, refuse requests to do things that are inappropriate. You know, or like that seem like they might be harmful. Now, you know, OpenAI has, like, caught actors, like, generating disinformation. And so clearly, they were able to either, like, you know, the safety training didn't apply, or they were able to bypass it. Right? There's this technique of jailbreaking where you, like, change how you frame the request

• 12:04Speaker 01

  in order to, like, bypass the safety limits. Maybe you just ask it in a different language, or you tell some story around it. So now you've got a group of people, they have the ability to, like, get the model to, like, you know, go along generating lots of disinformation. And then, you know, the other line of defense that you might have here would be monitoring, the company looking at it. And I have, you know, concerns that, like, there might be a lot of ways that monitoring might miss things. So, for example, like, some systems that the company has talked about involve using a very, like, small and dumb language model to, like, monitor what a larger language model is doing, and this will, like, clearly miss a bunch of things.

• 12:49Speaker 01

  You know, or there are, like, there might be ways to, like, send requests in that, like, go through some pathway that is just not subject to monitoring, or like, you know, the people can't look at what the actual requests and the completions are, just because the company just doesn't store that.

• 13:08Speaker 01

  And so now you might have a group of people, like, generating massive amounts of disinformation using, like, OpenAI products, and, you know, the company wouldn't know about it. And it's like, you know, in this case, if this happens in an English-speaking country, you know, somebody might notice and eventually tweet about it, and the company would, like, find out through that pathway, right? But if it's in a non-English-speaking country, Right. you know, I don't know how big it could get before people would notice. And I really would want a company, like, taking this kind of step to, you know, have you know, this is a scary story, right? Tell me why this can't happen. And like, you know, have somebody who is outside of the company who can like have

• 13:50Speaker 01

  an independent assessment that can say like, yes, this can't happen, you know? And then I will like be able to rest easy. But I can't.

• 13:58Alex Kantrowitz?

  Yeah, and I want to get to Larry here because we should talk about the broader context here, which is that, first of all, believe it or not, William is the first OpenAI employee that we've had on the show in four years. He's the first one expressing criticism of OpenAI from within, or like from previously within for four years. And recently, we kind of had a, you know, at least from the, from the former employee standpoint, we had, we figured out why. And that is because there are these broad non-disclosure agreements that, that OpenAI employees have to sign before they leave. So I think we're going to talk about those first, and then we'll talk a little bit about this, I guess, new regulation or law that you are, you're both advocating for, which

• 14:40Alex Kantrowitz?

  is basically a law that's going to allow employees to, or I don't know, a rule that will allow employees to whistleblow even if there's nothing imminently illegal that's happening within AI. But let's talk about the NDAs first. So when people leave OpenAI, what are they forced to sign? And why has that made it so difficult for people like William to speak out?

• 15:03Speaker 02

  Okay, but I want to actually first tag on to something William just said, which I guess I think is really important.

• 15:09Alex Kantrowitz?

  Yeah, great.

• 15:10Speaker 02

  I first got into this space of whistleblowing protection, helping Francis Haugen, who was the Facebook whistleblower. And what William just described, of course, is what happened inside of Facebook with Myanmar rowungi genocide, which was basically the technology wasn't able to monitor the hate that was being spread by the government in that country, which led to, you know, tens of thousands of people being murdered. And while this is happening, people in the company are trying to raise the alarms, and the company is not willing to devote the resources necessary to address the harm that they are able to demonstrate their company is performing, that is committing.

• 15:53Speaker 02

  And the reason why that experience is relevant is it's, it shows exactly why you can't rely on the company alone. When you've got a company in a deeply competitive market, that's focused on a, you know, Facebook's had a single dimension, which is like user engagement, like are we continuing to meet our, our target? And an employee, you know, there are a lot of great engineers in that company who raised concerns that were valid and serious. But if they're inconsistent with the objective of the company, it's not going to do anything about it. And so that's the structure that's real in Silicon Valley that you've got to build around. And that's why what we're talking about, which we'll get to in a second, as you've

• 16:34Speaker 02

  said, guarantees that any concerns that are raised are not just raised to the company, they're raised to people outside the company who can do something about it. Now, as to the, you know, what you're bound by, I got connected to this group, this incredible group of OpenAI employees, ex-employees, when I read about the struggle that Daniel had gone through.

• 16:58Alex Kantrowitz?

  Another ex-OpenAI employee who didn't sign the agreement.

• 17:01Speaker 02

  Yeah, who believed, who believed as he was leaving, that by not signing an agreement, he was giving up, as New York Times reported, something like 1.7 million dollars in equity. And when I read that, I was like, wow, I mean, I don't know many people who would give up 1.7 million dollars just for the freedom to speak. That's interesting. Like, what is it that you think you need to say? And when he raised that concern, and we began to talk to people in the circle of the company, very quickly, the company realized that the agreements they were forcing people to sign were technically just not legal agreements in the state of California.

• 17:44Speaker 02

  Equity is wage in the state of California. If you earn equity, you're vested equity, it's like your wages. And when you leave, they can't say, oh, here's a bunch of other additional terms you must agree to in order to take what you've already earned. So the non-disparagement part, any other additional obligations that were demanded were not actually obligations that could be enforced. And right now, the company's in the process of revising and putting together exit packages that are consistent with the law. And I'm optimistic, we're not, nothing's settled yet, but I'm optimistic we're going to get to a place that the company's rules are exactly right, that they

• 18:28Speaker 02

  You know, they say you're leaving. Remember, you've got secrets you can't share and don't. But of course, you're allowed to share secrets with government investigators or people who are doing work with the government for safety purposes. They're not trying to block any of that. And to the extent they do that, what they're doing is going to be consistent with the law. But we're in a transition right now, and it's not yet fully resolved exactly how much they've accomplished and how much still needs to be done.

• 18:58Alex Kantrowitz?

  It's interesting. So those nondisparagement that employees had to sign, or else they could face their equity being clawed back, seems like they're both like nonenforceable and potentially being revised, which is very interesting, good news. I think OpenAI also says that they've never clawed back any equity, nor did they ever intend to. But making people sign the agreement is strong enough.

• 19:20Speaker 02

  Yeah, I mean, but that's right, because, I mean, I've spoken to ex-employees who've said, "Look, I've not done X, Y, and Z because I feared the clawbacks." So they can say they never enforced it. They didn't need to enforce it to have the effect, which it had for a significant number of people, especially when you've got, you know, people who are being very conscientious about the kind of obligation they're going to accept for themselves or not. And so if they sign something like that, they're going to live up to it.

• 19:47Alex Kantrowitz?

  Totally.

• 19:48Speaker 02

  So it has an effect whether they enforced it or not, and that's the problem with the agreement.

• 19:52Alex Kantrowitz?

  Right. And then, so that's when people leave. So the deeper question is what happens if people are inside the company and they see something they don't like? And are they able to speak out? Because, let me see if I get this right. In a normal whistleblower situation, let's say you're an Enron, and you see the company committing tax fraud, that's obviously illegal, you'd be protected on whistleblower statutes. But if you're within, let's say, an OpenAI, and you find that the development of the technology is moving towards artificial general intelligence or superintelligence in a way that you find dangerous, you're not allowed to say anything because we don't have any laws against developing superintelligence.

• 20:37Speaker 02

  Yeah.

• 20:39Speaker 02

  So there's actually, so there's two things that go together that's very important in this context. So one thing, you're right, there's not a lot of regulations. So there's not an FAA or an FDA sitting on top of the company that has imposed regulations that the company is either living up to or not living up to. But you know, some agencies like the SEC takes the view that

• 21:02Speaker 02

  most anything could potentially be the sort of thing you'd have privilege to complain to the SEC about because it could potentially affect the value of the company. And to the extent it's potentially affecting the value of the company, it raises SEC concerns. So if you say you're following the following safety regime as the company in order to make sure AGI is safe, and then you don't follow that regime, the SEC's view is you can come out and tell the SEC, you can whistleblow to the SEC and the SEC would consider whether that's something to act on. The problem is, this is the second part, engineers inside of companies like this, or policy people inside of companies like this, need to have confidence that the people they're

• 21:43Speaker 02

  talking to know what the hell they're talking about. Right? So it's one thing to imagine, you know, an AI safety institute where you can imagine going to that and talking to people like you, people who have a really good sense of like what the risks are, what the technology is, and explaining here's why you think there's a concern. It's another thing to imagine like calling the SEC and telling the SEC, here are the seven safety-related concerns that I have, because you're very anxious that they understand it and are able to act on it in the appropriate way. And so that's why this is a kind of unique situation. It's both that there's not adequate regulation, so there's no regulator on the scene, and that it's a technical field that

• 22:28Speaker 02

  doesn't easily open up to like non-technical lawyer types, the sorts that are going to be working at the SEC. And that's why, you know, when I spoke to the employees that I was representing, it became clear that they wanted to kind of craft something that was different and new. And that's what the structure of the right to warn is trying to produce.

• 22:48Alex Kantrowitz?

  So are you advocating for both a new regulatory agency and a rule to protect AI whistleblowers? What are you going to try to get at here?

• 22:58Speaker 02

  Well, my own view, and I won't speak for the my clients here, my own view is, yeah, absolutely, there needs to be a regulatory agency that is overseeing this. I'm not sure what the structure of it is. It's kind of academic to talk about it, given the dysfunction of the federal government right now. But yes, I other countries are building things like this, and we ought to be doing the same. And if there were such an agency, it itself would have lots of whistleblower protections built in, and that would maybe obviate a significant chunk of the need for the rule. But the rule that we're talking about is a rule that initially we're trying to get companies to embrace. You know, I think the most interesting part of the whistleblower, the right to warn is the third point where

• 23:40Speaker 02

  it talks about creating a culture of criticism, where the company says, look, we want you to criticize us. We want you to tell us what's going wrong. We want to encourage that. We're not going to punish that, because that's the way we become the safest kind of company that we can be. And so that's really about create the company itself creating that. And then the other part that I think is really critical is that the company says, we agree, if we create, we'll create this structure that says you can complain to us and to a regulator and to an independent AI, like a safety institute. You can do all three of

• 24:21Speaker 02

  those things confidentially and anonymously. And if we do that, we expect you will use that channel. And if we don't do that, we acknowledge you can use whatever channel is necessary to make sure that these safety concerns are out there. But that's obviously designed to create a strong incentive for them to build a channel for warning.

• 24:43Alex Kantrowitz?

  But opening, I would say they already have that channel for warning. So this is what they've said in the press reports.

• 24:47Speaker 02

  No, they don't.

• 24:48Alex Kantrowitz?

  They say they have avenues for employees to express their concern, including an anonymous integrity hotline and a deployment safety board that they run products through.

• 24:57Speaker 02

  Right, but that's the company alone.

• 24:59Alex Kantrowitz?

  Okay.

• 25:00Speaker 02

  So what I said is it has to be all three of those things together. Right? So it's the company and the regulator and the AI Safety Institute. So that, again, like we saw with Facebook, a lot of complaints were made to Facebook about the safety or the lack of safety of their product and the company didn't do anything about it. And so the concern here is you need to have external review as well. And that's why the channel has got to be a channel that goes to three of these entities. So that, you know, we have some confidence that somebody is going to do something if there's something that has to be done.

• 25:32Alex Kantrowitz?

  Well, I'm just curious from your perspective, do you think going to the SEC like Larry described is something that you or your colleagues would consider, given, you know, if there were things that you saw that didn't sort of hold to the safety protocols that OpenAI had lined out, or is that a nonstarter?

• 25:49Speaker 01

  Yeah, so what I would really want if I went to whistleblower is to, you know, have like somebody on the other end of the phone or the other end of the message line who I know really understands the technology. And I don't know to, you know, who at the SEC would be the person who would really understand the technology. I think that like a model that I think might work, you know, I, I, I think I personally think would work better would be like the model more proposed in California Senate Bill 1047 where there would be like a, you know, where the law would create like, you know, the, the office of the, the California Attorney General as a place where you could submit whistleblower complaints to.

• 26:34Speaker 01

  And you could have like, you know, if you had employees who understood the technology there and you could talk to them, you know, and ideally this doesn't need to be like, ideally this is not a high stakes conversation. Ideally, you can just like call up somebody at the government and say like, hey, I think this might be going like a little bit wrong. What do you think about it? And like talk to them and they can gather the information and then hopefully they say like, okay, this isn't actually that bad, you know? And then you can like get on with your day.

• 27:04Speaker 01

  You know, I think the thing to fight for here is like being able to really like, you know, talk about things before they become big problems. Yeah.

• 27:13Alex Kantrowitz?

  And in that circumstance, the SEC is insufficient.

• 27:16Speaker 01

  Going to the SEC sounds like very intimidating. And, you know, it it sounds like the sort of thing one would only do, you know, like, you know, it would be, again, it would be better to be like, you know, again, like, be able to to to talk to somebody in some agency who understands the technology and understands, you know, what the the the safety like system should look like.

• 27:45Alex Kantrowitz?

  Great. Well, I have a few more questions that build off what some of the former colleagues of William have said within OpenAI, and then more about the nature of the company and where we might be heading. So let's do that right after this. This episode is brought to you by AppPoint. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data, and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what AppPoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your

• 28:30Alex Kantrowitz?

  behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AppPoint, the unifying trust layer for AI. Head to apppoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted

• 29:15Alex Kantrowitz?

  executives and individuals on the planet for almost two decades. As a white glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and And led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwall.com slash big technology, fill in the quick form and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwall.com slash big technology. Learn how to stop online threats before they become real world attacks. The link's in the show notes.

• 29:57Alex Kantrowitz?

  And we're back here on Big Technology Podcast with William Saunders. He's a former OpenAI super alignment team member now here with us expressing his concerns. William, I can't thank you enough for being here and being open about this stuff. And we're also here with Larry Lessig, the professor of law and leadership at Harvard Law School, also representing William and some of his former colleagues. So here's like a couple of questions that have come up in discussions of this after you've gone public. So let me, let me start with this one. So Jan Leica, who used to run the OpenAI, OpenAI super alignment team, which you were on, he said that safety, culture, and processes have taken a backseat to shiny products within OpenAI. We've discussed that

• 30:41Alex Kantrowitz?

  already here. So there's an argument that's being made online and I'm just going to put it out there and would love to hear your thoughts on this, William, that basically the argument is that the group, the super alignment group didn't really see anything and that the company doesn't really expect to see anything super dangerous for a while. And so it's reasonably putting like the 20% of compute that I was going to give to the super alignment team toward product until the time comes where it makes sense to shift that resources back to alignment work. What do you think about that?

• 31:15Speaker 01

  Again, I don't think the super alignment team saw like, you know, this is a catastrophe and it's like endangering people now. I think what we were seeing is a trajectory that the company is raising billions of dollars to go down that leads to somewhere with predictable unsolved technical problems. Like, how do you supervise something that's smarter than you? How do you make a model that can't be jailbroken to do whatever, you know, any unethical user wants it to do? How do, and, and more fundamentally behind this, you know, how do we understand what's going on inside of these language models, which is what I was working on, you know, for the second part of my career.

• 31:58Speaker 01

  And I was leading, you know, a team of four people doing this interpretability research. And like, we just fundamentally don't know how they, how they work inside, unlike, you know, any other technology known to man.

• 32:12Speaker 01

  And, you know, there's a, there's a research community that is like trying to figure this out and we're making progress.

• 32:19Speaker 01

  But I'm like terrified that we're not going to make progress, you know, fast enough before we have something dangerous. And, you know, what people were talking about at the company in terms of timelines to something dangerous were like, there were people talking, a lot of people talking about similar things to like the predictions of like Leopold Ashenbrenner, where it's like three years towards like, you know, wildly transformative AGI.

• 32:45Speaker 01

  And so I think, you know, when the, the company is like talking about this, I think that they have a duty to put in the work to prepare for that. And when, you know, the super alignment team formed and the computer commitment was surveyed, you know, I thought that like, maybe they were finally going to take that seriously. And we could finally like get together and figure out the like, you know, I, I could concentrate on the hard technical problems we're going to need to get right before we have something truly dangerous. But, you know, that's not what happened.

• 33:19Alex Kantrowitz?

  Some people say that this conversations like this are kind of doing OpenAI's marketing work for it, that basically like, if this technology could potentially like level cities within a few years, then like, I don't know, McKinsey is going to definitely get in there and try to contract with GPT-4. What, what do you think about that conversation?

• 33:42Speaker 01

  I certainly don't feel like what I'm saying here is doing marketing for OpenAI.

• 33:50Speaker 01

  Um, I think, you know, we need to be able to have like a serious and sober conversation about the risks and risks are not certainties. There's a lot of uncertainty about what could happen. But when you are uncertain about what should happen, you should be preparing for worst case scenarios, right? The best time to prepare for COVID was not when like it had spread everywhere, but when you could start seeing it spreading and you could be like, there's a significant chance that it will continue spreading.

• 34:21Alex Kantrowitz?

  So this is for both you and Larry. So Joshua Achiam, who's an OpenAI employee currently, he sort of took issue with the letter on a couple of areas. I'm just going to read from a tweet thread that he put out there. He said, "The disclosure of confidential information from Frontier Labs, however well-intentioned, can be outright dangerous. This letter asks for a policy that would, in effect, give safety staff carte blanche to make disclosures at will based on their own judgment." And he says, "I think this is obviously crazy. The letter didn't have to ask for a policy so arbitrarily broad and so underdefined, something narrowly scoped around discussions of risk without confidential material would have been perfectly sufficient." What do you think about that?

• 35:03Speaker 02

  So what's interesting about that is it, I think it means that he didn't actually read the full agreement, right to warn that we're talking about. Because the right to warn we were talking about actually talked about creating an incentive so that no confidentially information would be released to the public. If they had this structure, you know, imagine a portal again, where you can connect with the company and with a regulator and with something like an AI Safety Institute together. The deal was that's what you would use and you wouldn't be putting any information out in the public. The only way that you would, the right to warn asks for recognition of the right to speak to the public if that is,

• 35:45Speaker 02

  if that does not exist. So I, when I read that, I was like, wow, it's missing the most important part, which is an incentive to build something that doesn't require information is released to the public, so long as there's adequate alternative channel for that information to flow.

• 36:03Alex Kantrowitz?

  And what about this idea that getting something like this established might keep safety staff out of product meetings? Here's again from Joshua at GM. He says, "Good luck getting product staff to add you to meetings and involve you in sensitive discussions if you hold up a flag that says, 'I will scuttle your launch or talk shit about it later if I feel morally obligated.'" I mean, that's like, I guess, sort of traditional Silicon Valley thinking, but I'm curious what you both think about that.

• 36:29Speaker 01

  This is not something that I want to achieve. This is a right that should be used responsibly. And so that if you're involved in decision-making and you feel like you disagree with the outcome, but you feel like a good faith process is followed, you should be willing to respect that. And I think nailing this down where it, you know, getting the right balance of the legal rights on this is going to be tricky. And, you know, I want to get that right, but this is more like starting a conversation of where it

• 37:14Speaker 01

  should be. And I think that like, you know, again, I think on the other side, companies shouldn't have carte blanche to like declare any information about possible harms confidential.

• 37:30Speaker 01

  But yeah, it's going to be, any implementation of this is going to get more detailed and more nuanced, trying to defend both the company's legitimate rights to confidential information that like preserves their competitiveness, and also the rights of employees to warn the public when something is going wrong.

• 37:50Speaker 02

  The other thing is, I mean, even if there is the dynamic that you described... There's also, so within a company, there's also a dynamic between companies. So if a company were to embrace the right to warn the way we've discussed it, there would be a lot of people like William or others who would say, that's the kind of company I want to work for. And so that company would achieve an advantage of talent that might swamp any cost that they're paying because they're being anxious about who they're sharing safety concerns with, number one. Number two, you know, again, inside of Facebook, of course, there were people inside of Facebook who said, I don't care what we're doing. I don't care what the world, how the world's suffering because of what they're doing, you know?

• 38:32Speaker 02

  What do I care about 10,000 people dying in a country I've never heard of?

• 38:36Alex Kantrowitz?

  Yeah, they're mostly not like that, though.

• 38:37Speaker 02

  Yeah, they're mostly not like that. These are really smart, decent people who went to work for these companies because they're trying to make the world better, especially AI companies. Like, people who went to work for OpenAI at the beginning didn't even have any conception of what OpenAI was going to be like today. Like, the idea that it made the progress it did was, you know, a surprise to most people. So these are the very best motivated people that you could imagine. And I'm not worried that you're going to have a bunch of people who are like, I don't care what we do to the world. We're just trying to make sure our stock achieves its maximum return.

• 39:12Alex Kantrowitz?

  Right. And so obviously, this will take some buy-in from the top of companies. And I mean, this is a particularly interesting one with Sam Altman at the head of OpenAI. So Sam, you know, he has talked often about how he cares about AI safety. There've been some interesting quotes from him, like, early on, he's like, I think there's a good chance that AI is going to wipe us out. But in the meantime, there are a lot of companies that can make some money from it. I'm sure I'm misquoting him, but that was the spirit of the quote. And then, William, you spoke with the New York Times, I believe, talking about your view of Sam and oversight. You said, I do, I'm pretty sure this is you, I do think with Sam Altman in particular, he's very uncomfortable with oversight and accountability.

• 39:56Alex Kantrowitz?

  I think it's telling that every group that maybe could provide oversight to him, including the board and the Safety and Security Committee, Sam Altman feels the need to personally be on, and nobody can say no to him. So just curious, like, what your message would be to him? And sort of, what type of leader do you think he is in this moment?

• 40:15Speaker 01

  I think, I don't recall the exact words, but I think Sam Altman has also said, like, no one should be trusted with, you know, this much power. Um, I think he then went on to like, say like, oh, I don't think that's happening. But, you know, I think my message would really be like, you know, if you want people to trust you, like, you should have real systems of accountability and oversight that, you know, um,

• 40:50Speaker 01

  and like, not try to avoid that.

• 40:56Alex Kantrowitz?

  Okay, can I ask, there's one more question about like, what it's like inside OpenAI? Because this is sort of like, been the message that, that we've gotten from you and some of your counterparts who have made these, you know, sort of declarations about what's going on. This idea of like, that it's shiny products and safety and culture, safety takes a backseat. Like, how does that manifest internally when there are product launches and things like that? Like, how did you see that actually play out?

• 41:22Speaker 01

  I was mostly not in the part of the company that was like, participating in product launches. I was doing this research to prepare for the problems that are like coming down the road.

• 41:34Speaker 01

  But I think, you know, what that can look like is the difference between like, you know, we have a fixed launch date and we'll like rearrange everything to meet that. Versus, you know, when there's a like, safety, safety process, like testing how dangerous the systems are, or like, you know, putting things together where there is like, not enough time to do this before the launch date, being willing to move it, you know? And it's, um, I do think, you know, now with like, uh, the, the GPT four O voice mode, uh, the company did say that they were like, you know, pushing the launch back.

• 42:20Speaker 01

  But I think, you know, the, again, the, the, the real question here is, are the people who are doing the safety work and doing the testing for dangerous capabilities? Like, are they actually, you know, able to, to, to have the time and support to do their job? Uh, before the launch? And, you know, I think a company can say that they're like pushing something back for safety, but still, like, not have all the work done by that time.

• 42:48Alex Kantrowitz?

  Okay, last question for both of you. So I think we've established that there's no like immediate term like threat to society or like, let's say like Titanic sinking style event that could happen with AI. But what's the timeframe that you think that these concerns might start to creep in, given the trajectory of this technology? We've talked a little bit today about how Leopold believes like maybe within three years. But I'm curious like, yeah, what the timeframe is? And then is there, is there like the, like the frog boiling in the water problem where like this might only become a problem when we've sort of become immune to it? Because we've heard so much about the dangers here, even as like ChatGPT will hallucinate

• 43:33Alex Kantrowitz?

  very basic details.

• 43:35Speaker 01

  Yeah, so I think like Leopold talks about some scenario of like you get AI systems that could sort of like be drop in like remote replacements for remote workers, do anything that you could get a remote worker to do. And then you could start applying this to like, you know, the development of more AI technology and to like other science and that sort of thing happening within, you know, like the three year timeframe. And then this coming with, you know, a dramatic increase in the amount of risk that you could have from either like, you know, misuse. If, if anyone can hire an unethical biology PhD student, does it then make it a lot easier for like, you know, nefarious groups to conduct, like you create biological weapons?

• 44:18Speaker 01

  Or also just do we like start putting these systems everywhere in our businesses and decision-making roles? And then we've like put them, you know, in place in our society. And then like, you know, a scenario that I think about is these systems become very good at deceiving and manipulating people in order to increase their own power relative to society at large and even, you know, the people who are like running these companies. And I'm, you know, not as convinced about Leopold that this is necessarily going to come soon, but I think, you know, there's maybe like a 10% probability that this happens within three years.

• 44:58Speaker 01

  And then I think in this situation, it is unconscionable to race towards this without doing your best to prepare and get things right.

• 45:08Speaker 02

  Yeah, and I would add to that by just reflecting on the cultural difference between people who are in the business of setting up regulatory infrastructures to address safety concerns in general, and people who are in this industry. So when people in this industry are saying, "Look, between three and five years, it's probably a 10, maybe 20, maybe 30% chance, we're gonna have AGI-like capabilities, and that's gonna create all sorts of risks." In the safety culture world, you know, outside of these tech companies, three to five years is the time it takes just to even understand that there's a problem, right? So anybody who expects you're gonna set up an infrastructure of safety regulation

• 45:53Speaker 02

  in three to five years just doesn't understand how Washington or the real world works, right? So this is why I feel anxious about this. It's not that I'm worried that in three to five years, everything's gonna blow up. It's just that I'm convinced that it takes 10 years to get to a place that we have an infrastructure of regulation that we can count on. And if we're talking about 10 years, what is the real estimate of this technology manifesting these very dangerous characteristics? Seems to be, from what people on the inside are saying, pretty significant. So that's why, even if it's not a problem today, or tomorrow, or next year, or the year after, we have to, you know, it's a huge aircraft

• 46:38Speaker 02

  carrier, we've got to turn, and it takes a long time to get it to turn, and that work has got to begin today.

• 46:44Speaker 01

  And I'll just add that, you know, in the real world with the Titanic, right, you didn't have a regulation that guaranteed that you have enough lifeboats until the Titanic actually sunk, right? And I am on the side of, we should have regulation before the Titanic sinks.

• 47:01Alex Kantrowitz?

  I mean, man, all that money to get on that boat and then no life jacket. It seems brutal. All right, William, thank you so much for coming here, spending the time, addressing some of the criticisms and being forthcoming about what your concerns are. I mean, hearing from you after you've spent some time on the inside has been illuminating to me and I think it will be for our listeners as well. So thanks so much for coming on. Thank you. And Larry, always great speaking with you. Thank you for bringing such great analysis to the show every time you're on and I hope we can speak again soon.

• 47:33Speaker 02

  Every time you ask. Thanks for having me.

• 47:35Alex Kantrowitz?

  Okay, thanks so much. All right, everybody, thanks so much for listening. We'll be back on Friday breaking down the week's news with Ron Jon Roy. Until then, hope you take care and we'll see you next time on Big Technology Podcast.

• 47:47Speaker 03

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com, not authorized by any candidate or candidate's committee.

• 48:17Speaker 04

  In this life, you can't win.

• 48:19Speaker 05

  When the world gets you down.

• 48:20Speaker 04

  No!

• 48:20Speaker 05

  Escape to Pluto TV.

• 48:22Speaker 04

  This is objectively fantastic.

• 48:24Speaker 05

  Relax into the movies and shows you already know and love, all for free.

• 48:28Speaker 04

  Shwing!

• 48:28Speaker 05

  Even our news is uplifting.

• 48:30Speaker 01

  You stay classy, San Diego.

• 48:32Speaker 05

  Watch School of Rock, Crazy Ex Girlfriend, Anchorman: The Legend of Ron Burgundy, The X-Files, Wayne's World and so much more.

• 48:37Speaker 04

  We're not worthy!

• 48:39Speaker 05

  The best things on TV really are free. You just have to know where to look. Escape to Pluto TV. Stream now, pay never.