The AI Whistleblower Initiative: Supporting AGI Insiders When It Matters Most, w/ founder Karl Koch

Today Karl Koch, Co-Founder of the AI Whistleblower Initiative, joins The Cognitive Revolution to discuss the barriers preventing AI insiders from raising safety concerns, his organization's anonymous "Third Opinion" service connecting whistleblowers with independent experts, and their campaign demanding frontier AI companies publish their internal whistleblowing policies to address widespread retaliation and lack of transparency. Check out our sponsors: Oracle Cloud Infrastructure, Shopify. Shownotes below brought to you by Notion AI Meeting Notes - try one month for free at ⁠⁠⁠⁠https://⁠⁠notion.com/lp/nathan AI Whistleblower Initiative Origins: Founded by Karl Koch in Berlin, the initiative began with research in early 2024 after consulting with over 100 governance researchers and insiders. Link to the website: https://aiwi.org/ "Third Opinion" Proposition: Launched in late 2024, this program systematically breaks down barriers for insiders to speak up about AI concerns while ensuring their issues are addressed. Publish Your Policies Campaign: The initiative calls for AI companies to publish their whistleblowing and speaking up policies, with 100% of surveyed insiders supporting this transparency measure. Link to the campaign: https://aiwi.org/publishyourpolicies/ Anonymous Consultation Process: Whistleblowers can seek advice without sharing confidential information through an open-source anonymous tool accessed via Tor browser. Pro Bono Legal Counsel: Whistleblowers receive connections to experienced legal representation with client privilege protection at any stage of their journey. Policy Advocacy: Beyond direct support, the initiative works on advocating for better whistleblower protections in both US policy and EU AI regulations. AI Whistleblower Initiative Survey: https://bit.ly/AIWISurvey PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Karl Koch, founder and managing director of the AI Whistleblower Initiative, a nonprofit dedicated to supporting concerned insiders at the frontier of AI development. I am particularly passionate about this work because I would have loved to have had this kind of support a couple of years ago, when, as longtime listeners will know, I tried to raise concerns about the size and quality of the GPT-4 Red Team project and the ineffectiveness of the nascent safety measures that OpenAI had developed at the time. Then, no such support existed, and so I consulted friends in the AI safety community before ultimately deciding to escalate my concerns to OpenAI's board, for which I was subsequently

• 0:44Nathan Labenz?

  dismissed from the program. That experience left me acutely aware of how difficult it is for insiders to navigate these situations, and has motivated me to support this project with a mix of modest personal donations, behind-the-scenes fundraising help, and a bit of ad hoc volunteer work over the last six months. Along the way, I've been consistently impressed by the seriousness of Karl's thinking and the maturity of his approach. AsperFitz is an organization that aims to help people in truly critical moments in their careers, when the stakes have never been higher for them personally and potentially for society as a whole. They are taking care to lay a foundation of understanding and infrastructure now so that insiders can trust them if and when that pivotal moment comes.

• 1:29Nathan Labenz?

  The first critical investment they've made is in extensive research and understanding. By talking to over 100 governance researchers and surveying employees at frontier AI developers, they've developed a deep understanding of the barriers that potential whistleblowers face. The majority of frontier lab insiders, as it turns out, don't even know if their companies have internal whistleblowing policies, let alone understand what protections they offer. The legal landscape unfortunately doesn't help much either. The EU will begin to protect AI whistleblowers starting in 2026, but US law remains a patchwork, with proposed legislation like the AI Whistleblower Protection Act still pending.

• 2:07Nathan Labenz?

  Strikingly, roughly half of survey respondents expressed a lack of confidence in their own ability to determine whether specific observations constitute a serious cause for concern. And literally 100% lacked confidence that regulators would understand, let alone be able to act effectively on their concerns. Meanwhile, over 90% couldn't name a single whistleblower support organization. All this, plus well-known cases where people like Leopold Ashenbrenner were fired for breaking chain of command and going directly to the OpenAI board with security concerns, creates a highly uncertain and risky context for such high-stakes decisions, in which people with serious safety concerns are left to think through the nuanced costs and benefits

• 2:49Nathan Labenz?

  of internal escalation versus going to regulators versus leaking to the press almost entirely on their own. It is an extremely stressful position to be in and not conducive to the best possible decision-making. The good news is that the AI Whistleblower Initiative offers several forms of support. Their Third Opinion service, which you can find online at aiwi.org, allows insiders to anonymously reach out via a Tor-based, open-source tool, which Carl notes is pen-tested, with security reports published openly for scrutiny and verification, to get help identifying and anonymously contacting independent experts who can answer questions without requiring you to share confidential information or even reveal where you work.

• 3:32Nathan Labenz?

  For those concerned about digital privacy, they provide a digital privacy guide and, in select cases, hardened devices with specific operating system setups for highly secure communication. And if insiders need concerns justified, they also connect people with specialized and experienced whistleblowing support organizations, including the Signals Network, pssst.org, and the Government Accountability Project, which can provide pro bono legal counsel, psychological counseling, and guidance throughout the process, all without pressure to disclose any information. Crucially, in some cases, they can even help arrange financing to cover legal costs, which can easily add up in cases that end up in any form of litigation.

• 4:14Nathan Labenz?

  That a nonprofit stands ready to invest this seriously in any concerned insider that needs help may strike some people as excessive today. But considering that we're talking about perhaps just hundreds or maybe low thousands of people globally who are positioned to spot and raise critical concerns over the next few years, of which I'd bet only a few dozen will ever find themselves in position to seriously consider sounding alarms, I think this sort of care and support is absolutely worthwhile. Most recently, Carl and team have launched the Publish Your Policies campaign, online at publishyourpolicies.org, calling on frontier AI companies to make their internal whistleblowing policies public. This is actually standard practice in many industries, but interestingly, in the AI

• 4:58Nathan Labenz?

  space, only OpenAI has done any version of this, with their Raising Concerns policy, which they published only after Daniel Cocotello and others revealed OpenAI's use of extensive non-disparagement agreements to keep former employees from publicly criticizing the company. Daniel, by the way, is joined by other former AI insiders, luminaries like Stuart Russell and Lawrence Lessig, and yours truly, in signing on to the Publish Your Policies campaign. Of course, publishing corporate policies doesn't obviate the need for proper legal protections, which Carl strongly advocates for as well. But at a minimum, it would help insiders understand their rights and options, enable public scrutiny, and ultimately create accountability that benefits everyone.

• 5:39Nathan Labenz?

  If you work at a frontier AI company, Carl encourages you to ask your management to consider publishing their whistleblower policies. If enough people ask these questions now, I wouldn't be surprised if it becomes another dimension of the intense competition between frontier AI developers for top research talent. And that could ultimately mean that companies even begin to collect and publish data on things like how many reports they receive, their response timelines, retaliation complaints, appeal rates, and whistleblower satisfaction scores, all of which would benefit everyone.

• 6:11Nathan Labenz?

  Regardless of what company leadership decides to do, Carl's message for insiders is this: Support is available at every stage. Whether you're considering internal escalation, thinking about approaching regulators, or even contemplating public disclosure, you can reach out completely anonymously without sharing any confidential information just to understand your options. And the AI Whistleblower Initiative can help you get expert perspective on what you're seeing, connect you with legal counsel and experienced guidance, and perhaps even help finance your case. So don't wait until you're deep into a crisis and know that you don't have to face this alone.

• 6:51Nathan Labenz?

  With that, I hope you enjoy, and I encourage you to share with friends who work at Frontier AI Developers this in-depth conversation about the challenges that concerned AI insiders face and the support that's available to make sure that those who are building our AI future can safely speak up when it matters most. With Carl Koch of the AI Whistleblower Initiative. Carl Koch, Managing Director at the AI Whistleblower Initiative. Welcome to the cognitive revolution.

• 7:18Speaker 01

  Thank you very much, Nathan. Thank you for having me.

• 7:21Nathan Labenz?

  I'm excited for this conversation. We've been talking behind the scenes and collaborating a little bit as you've been building this organization and a couple of key initiatives over recent months. So I'm excited to finally get into it in a public forum.

• 7:36Nathan Labenz?

  Maybe for starters,

• 7:38Nathan Labenz?

  AI whistleblowing, you know, very new territory. How did this come to your attention? How did you decide to prioritize it? What's the backstory that has you focused full-time on this corner of the world?

• 7:49Speaker 01

  Yeah. So maybe on a personal level. So yeah, thank you very much again, Nathan. Lovely, lovely introduction. Name's Carl Koch, founder of the AI Whistleblower Initiative. We're a nonprofit project currently based out of Berlin, supporting concerned insiders, whistleblowers at the frontier of AI. I personally came to it like I've been involved in like the AI safety scene, or however you want to call it these days, since like 2016. I was a volunteer researcher at the Future of Humanity Institute. Lovely institute, of course. No longer around sadly. And back in the day, worked on differential technological development. Maybe some of the older listeners still familiar with that term. And worked on like an AI safety research camp also after this, but afterwards actually decided to drop out of the scene for a bit.

• 8:34Speaker 01

  Was a management consultant in Hong Kong for a few years, then started my own SaaS business. Because back in the day, the very different timelines. So I was always sort of also back in like the two thousand teens quite interested, especially in the arms race angle, a sort of like a root evil for I think a lot of the problems that we see today, like safety skipping, these sort of things. And

• 8:54Speaker 01

  as then chat G P T rolled around, sort of the alarm bells went off at the latest moments, like, okay, maybe things are moving quite a bit faster than I think most people anticipated, sort of in the late two thousand teens. And then started to talk to a bunch of people from the network, governance researchers, thinking about what seemed to be like the most tractable, best solutions, best things one could start building. And transparency sort of came to the front pretty quickly as a thing that we should just generally have more of, regardless of what the future looks like. And like one angle, for example, was compute traceability, which I think other people have taken on sort of over the past monseas as like championing that cause. And the other angle was whistleblowing.

• 9:36Speaker 01

  So I think people had been writing about like the importance of whistleblowing as a mechanism since like two thousand seventeen, eighteen, with specifically the AI angle. And originally we came from the arms race perspective again. So if you have a bunch of players in like, you know, a multi-round game, they would ideally want to trust each other on their statements on speed and safety. And if you don't have transparency and if you can't actually believe that others stick to their promises, how can you, you know, how can you cooperate over multi-round games? That was sort of the original kickoff thought. This was like twenty twenty three mid. So there the world, I think, still seemed a bit more rosy, a super alignment team had just been kicked off, everything seemed golden, ten percent commitment by OpenAI of compute.

• 10:22Speaker 01

  And so we thought, okay, maybe we were a bit too early here, but this is going to become relevant earlier or later, because the, the, the, I mean, the original papers, I think, around what game theory would look like. Intensifying competition were like out in twenty fourteen, fifteen, sixteen. Then the opening airport drama happened. So that was like the first moment we thought, okay, maybe we have to speed up on the research side a bit more. And this maybe becomes more of a concern already that maybe we cannot trust that everything is going jolly well, even in these organizations that claim to be very aligned, let's say. Then 2024, of course, things really started to go in a different direction. First, of course, there was the Leopold Aschenbrenner case around sharing information,

• 11:04Speaker 01

  escalating concerns to the board, as I think is understood by now at least, where people were penalized for, among other things. And then, of course, the big story is sort of mid 24 around the Cucutayo and William Saunders and the other people, which then led to Right to One. So, of course, this topic became a lot hotter sort of throughout mid 2024. We properly started off like our research phase in early 24, talked to like well over a hundred sort of governance researchers, insiders in frontier companies throughout that year, and then launched our first proposition called Third Opinion. Maybe the listeners have heard of it. At the end of 2024, which we developed together with one of the OpenAI whistleblowers from

• 11:48Speaker 01

  last year, kind of tackling one specific problem, I'm sure we'll get into it a bit more a bit later. And yes, so basically we've been live since late 24, and now we're doing a variety of things to like systematically break down barriers for insiders at the frontier to speak up and make sure those concerns are addressed.

• 12:05Nathan Labenz?

  Love it. Thank you. One thing that I've been impressed by watching you behind the scenes is how deliberate you have been in the approach. You mentioned talking to a hundred insiders.

• 12:20Speaker 01

  That's a hundred governance researchers and some insiders as well.

• 12:24Nathan Labenz?

  A hundred governance researchers and some insiders. Important to get the details right. But that's, you know, a pretty quiet, slogging process, which I think is kind of representative of the right. Way to position an organization like this. You know, there's many things in the AI space right now that people are sort of YOLOing, right, where they're just like, oh, I made a thing like. And so much stuff I think is even just getting open sourced. People sometimes, you know, ask like, why is everybody open sourcing their stuff? And my answer in many cases is like, they know that their time at the frontier is going to be very brief. And there's probably not enough time in many cases to like, build a business around it or, you know, make money from it or whatever. So in many cases, like the best that people can do when they create something that they're

• 13:09Nathan Labenz?

  proud of is just put it out there, you know, try to put their flag, you know, in the ground that at one point, you know, if nothing else, I was at the frontier of this incredible phenomenon. And then just kind of see what happens and, you know, hope that maybe people notice it and take an interest in it. I think a lot of things are going that way, but you're taking a very, very different approach that is like very deliberate, very behind the scenes, has been quite quiet, although you're starting to raise profile now a little bit. What did you learn from talking to all those people? Like, you know, can you, can you synthesize the vibe, the need, you know, the, and you could segment that perhaps by maybe different frontier developers or different, you know, mindsets.

• 13:51Nathan Labenz?

  I, we've hear, we hear a lot of things about like cultural divides, you know, even within single companies. But yeah, how would you sort of characterize the, you know, the long journey that you took through all these conversations?

• 14:04Speaker 01

  Big question, big question. So I guess one angle to look at it from sort of both from instances we talked to and obviously case studies that are already out there around or like statistics even around like journeys of insiders. So like, what are the challenges they face? And maybe some of those challenges are very specific on the AI side, right? And you can talk through that, that journey is probably one angle. And then certainly the other angle is how does that differ sort of among different companies and what sort of patterns are we seeing in terms of speak up culture, in terms of maybe retaliation? And so these are probably the two different angles. Then, of course, you can all split up again via channels. I'm not sure how familiar the listeners are to this.

• 14:48Speaker 01

  So like generally, when we talk about whistleblowing, we're always talking about some insiders raising a concern, like or some misconduct that they want to have rectified. And they do this in a way where potentially they go sort of over the heads, likely of like middle management or like against like the direct powers that be. It doesn't mean that whistleblowing is always, for example, leaking information to the media. So often I think people think, and this is maybe your first answer to the question. A lot of people, when they hear whistleblowing, they think, okay, this is like an Edward Snowden kind of, kind of situation. That doesn't really have to be the case. So there's basically roughly these three different channels, which mean like which are called internal, external and public. And there's like the internal channel, which is sort of the major channel that most insiders use,

• 15:33Speaker 01

  but at least based on statistics at the start. Naturally, we also expect that AI companies. So raising concerns internally, within the company, either the structured or an unstructured process, which maybe actually you can share a little bit about your experience in a second as well. And then there's sort of the external process, which means going to a regulator, speaking to a regulator about the concerns you have. And then public is sort of the kind of next escalation mode where you feel you maybe you're not progressing or you can actually expect maybe that going to a regulator would make the problem worse. You'd expect collusion, for example. Then you can, you can also go public. These are like roughly the three different ways. And people in different companies think difference about these differently, about these three different ways. And so sort of different challenges associated with each of them.

• 16:18Speaker 01

  Would you have a preference? Which one do you want to kind of talk about first in terms of insights from like conversations with insiders?

• 16:26Nathan Labenz?

  Yeah, maybe we could take it through that ladder of, I guess, escalation or, you know, sort of, it seems like obviously from insider to outsider to public, there is, you know, more personal risk that the individual is running. There's also just like more chance that things sort of take on a life of their own, you know, and.

• 16:52Nathan Labenz?

  Results of, you know, sharing information just become harder to predict.

• 16:56Speaker 01

  Yeah.

• 16:56Nathan Labenz?

  So maybe, yeah, I would say probably the right way for people to be thinking about it, I would guess, maybe you could disagree. But kind of the way I ended up, you know, framing it myself was that this was sort of a ladder to climb as opposed to like three options. But.

• 17:13Speaker 01

  You tell me. Yeah, I think so. You have to, I think, differentiate that a little bit between what's actually the law and what are you allowed to do versus what do people perceive? And so maybe we can start sort of on the perception side. I think what you say is definitely sort of generally perceived to be the case. That most people think about, hey, if I have a concern, I'm going to raise it internally. Sort of the step going, for example, to regulator is a much larger step naturally. Maybe we could talk in that context also a little bit about a kind of the campaign we just launched, publish your policies.org. You can find it, by the way, asking companies to publish their internal whistleblowing policies. We can do that in a second. So the vast majority of investors we talked to definitely think about raising internal first.

• 17:50Speaker 01

  And the statistics confirm this, like I think the SEC released statistics around more than 70% of people who actually end up going to the SEC started kind of raising concerns internally. If we want to start there, then sort of the biggest challenges we see is on a first level is sort of overarching. It's legal protections even for raising concerns internally. So large variety here, sort of if you, for example, look at the EU side, that's quite heavily protected through the EU whistleblowing directive. Less so in the AI space at the moment, but that's going to come, that's going to be covered instead of mid 26. So August 26, the EU Act is going to be covered under the whistleblowing directive.

• 18:33Speaker 01

  In the US, it's much more of a patchwork. So even for internal disclosures, much more so for external disclosures. Public is not possible at all, pretty much in the US, to go public with a concern. In the EU is actually possible. So you can actually go public if you want. If you have reasonable cause to believe that there's a violation of a covered law. You are allowed to go public if external channels failed to respond to you, for example, in time, or if you believe there could be collusion, so you can go public straight away. You don't really have that in the States. So like overarching is this question of like legal protection, which is quite messy, because there is no AI whistleblower protection as yet. There is one being proposed, which is great at the moment, but there's sort of nothing

• 19:15Speaker 01

  there, which means there's just overall quite poor protections at the moment from a legal perspective. You can get creative, I think you talked to somebody about the liability side. Recently, there is specific angles, maybe also under the SEC, where you can claim that, for example, certain violation is like securities fraud. For example, if you make a public statement saying you should invest in our company because we are the safest company around, because, look, we have this RSP, for example, here, and then you actually don't follow the RSP, that may be securities fraud. Maybe there's Matt Levine fans here as well in your audience, everything is securities fraud. So, but maybe, so this is sort of overall, the overarching legal challenge that people face.

• 19:57Speaker 01

  There's good support here available, by the way, we want to keep stressing this, that there is actually great legal support, also pro bono available. You can also find a bunch of organizations on our website, aiwi.org, or you can approach us directly for consultations on who may be able to help you. Yeah, so that's the overarching legal element. And then it comes to something, I think, a bit more AI specific, which is just clarifying if what you're seeing is even cause for concern.

• 20:21Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 20:26Speaker 01

  So this is, I think, something that we keep hearing over and over again, where the gray zones are sort of really the tricky piece. If something is clearly illegal, then, you know, the path forward is pretty easy, but as you, as you know, we're sort of building this plane as we fly. Nobody is really clear on what is acceptable behavior at the moment and what is not. If you look at internal deployment, for example, I mean, there was a, I think, pretty great paper out a few weeks ago on like risks around internal deployment. This is uncharted territory, and it's obviously there's also no regulation covering it, what is acceptable, what is not, and the people building the systems are also quite frequently pretty unsure. For example, like we had a conversation with like a pretty senior person, actually, in like a safety function

• 21:12Speaker 01

  of one of the frontier companies who we asked around how much of a barrier to speaking up is your own ability or inability to accurately assess the severity of risks and they rated it as like extremely high. Funny enough, more junior team members frequently rated lower. So like we see more junior like employees at these frontier companies being a bit more optimistic. Maybe a function of, you know, getting really concrete problems that you work on as a junior as compared to maybe a senior person where you have to think about what are actually the problems we should be working on. May just be limited sample size remains to be seen. So that's, I think, a really specific problem. This is sort of maybe going through like escalation channels. One that also is a struggle like at these different levels.

• 21:57Speaker 01

  So if you're, for example, just thinking about going internal with your concern, you naturally potentially slide into it already anyways. Because if you have a concern, your manager doesn't share the concern. You have to, what do you do about it? If you're maybe not convinced in the conversation, you naturally move into a flow of saying, okay, I want to escalate this now. And from our impression, also I've talked into previous AI whistleblowers, frequently they don't understand themselves as whistleblowers already at that moment. It's just business as usual. You're just sort of saying, oh, I'm concerned about this manager is not, I'm gonna jump one level up. And that could already be a case where maybe your manager doesn't like you anymore after that.

• 22:41Speaker 01

  And may decide to, the actual term is retaliation, right? May decide actually that performance review is not gonna look great. And at that level already, it can become relevant, sort of what the internal structures are at these companies to handle concerns and like how they deal with concerns. Uh, jumping ahead a little bit, I think we're going to talk about later our campaign. The problem is at the moment, we have no idea what these systems look like. So we don't know what the internal whistleblowing policy systems are of these frontier companies, which is like way lagging other like companies, industry standards around transparency requirements and like effectiveness. We've seen a bunch of retaliation also in the past. And that is, that is not good. That should not be the case.

• 23:25Speaker 01

  Um, yeah, so I think coming from the question of, okay, clarifying whether there's even a concern here. Internal, you can go, you can ideally, if you have some well-equipped whistleblowing system, where people can actually investigate for you, that's, that's like a good first step. The next path, of course, then is the regulator approach. Here we are also seeing the struggle again around the extent to which also employees and insiders actually trust, whether governments can handle these reports, especially if they themselves think they're in a gray zone, which then I think is one of the most, the most interesting, unfortunately, cases, because like the really clear cases of there is something obviously going wrong here

• 24:07Speaker 01

  are potentially going to be rectified internally, hopefully so. A lot of people, at least we talked to, still seem to be like of the belief, at least instead of the lapses which we expect, like OpenAI, Anthropic, DeepMind, that if there's really glaring holes, they will be kind of fixed internally. Uh, and if it's like really clear that if that's not happening, that there is probably somebody at a regulator level who will understand your concern and act on it. Uh, if the right legal provisions are there, we alluded to it. But like the gray zone is really tricky and we're basically seeing that insiders have very low levels of trust, that regulators and these people sitting there are actually going to be able to understand. Like imagine, you know, you were, work on, let's say, like interpretability team and like a frontier company, the Bay Area, and you have this problem that you cannot figure

• 24:52Speaker 01

  out yourself and you have genuine confusion and now you're meant to go to an attorney general. And explain that to them and be like, hey, this is maybe something to be concerned about that could potentially fall like under this risk area. Tough, right, tough. I think really hoping that when sort of the hopefully the Whistlebird Protection Act is gonna pass, that we're also gonna see good buildup here around capabilities and like rights to investigate, freedom to consult external parties and do that at speed. Um, I think there's not much detail on that side in the Europe side. So we're pushing relatively hard for the EU to establish a mailbox specifically at the EU AI office.

• 25:34Speaker 01

  Because at the moment, per default, that's not gonna be set up, the way that Europe is set up, you know, very federal, like every member state would get their own like reporting office and that's historically has not worked well for much simpler cases like accounting fraud. And so we think it's extremely important that like on the EU level, there has to be the central, well-equipped, knowledgeable recipient body. So that's the internal, the external angle on that question, like should I even be concerned? Maybe I can sprinkle it into here as well. So we have this service called Third Opinion. That's one of the offerings we have where basically we allow insiders to anonymously reach out to us via a Tor-based tool that's open source.

• 26:16Speaker 01

  You can check out the Penta's reports yourself if you would like security stuff. And people basically can reach out with questions to us. So the idea here is that even before an insider feels like there is actually something to be worried about here, they can reach out with a question around their concern without involving confidential information, without disclosing who they are, who they work for. And then what we do is we workshop together with the insider. For one, is the question sort of the right one? Is this a promising question to ask or is it maybe too broad? And then we identify together relevant independent experts. Because I mean, we've seen, I think, quite a bit recently that labs have become a lot more secretive, or AI companies rather, have become a lot more secretive, maybe compared to two years, three years ago.

• 27:02Speaker 01

  And we basically identify these independent experts together and then we approach them with the question, get the answers back, supply them to the insider and then if the insider feels, okay, actually, fair enough, nothing to be worried about here, great, even better for all of us. But if they feel that in fact, there is something here, then we connect them to pro bono legal counsel, that we then, if legally permissible, also kind of involve the experts we've identified in the first step to make sure also that legal counsel understands the actual situation and gets a bit more context around it, kind of covered under legal privilege, which by the way is another insight, I think, talking to insiders. Also, on the legal counsel side, they often don't have a lot of trust and feel like, yeah,

• 27:47Speaker 01

  I mean, I could approach a lawyer here now with my concern, but they're also not going to understand like what even the problem is, especially if it's just roughly pointing towards this is something maybe to be concerned about. So that's sort of the idea. We say there is already great whistleblower lawyers out there. They do pro bono work, like the Signals Networks is another great organization, maybe listeners should check out, dot org, whistleblower aid, government accountability project, whisper more on the national security side. There's a bunch of really great organizations there. And insiders tend to be either don't know them, this is a massive problem, we'll get to that in a second, or they don't trust they have the knowledge. So we sort of supplement that through that expertise. Going sort of, I can, I could talk further sort of through the journey of the other challenges that we

• 28:29Speaker 01

  see insiders face, but I feel like I'm talking a lot already. Do you have anything, any questions at this point?

• 28:35Nathan Labenz?

  Well, excuse me, I do want to invite you to go on. I guess, you know, just reflecting on a number of the things that you've said there, a big part of the reason I've been passionate about this and in trying to, you know, do my small part to help you behind the scenes is

• 28:54Nathan Labenz?

  I can really, having done the GPT-4 red team, you know, project myself, I can really empathize with the insiders who are like, wow, I'm seeing things I did not expect to see.

• 29:09Nathan Labenz?

  I'm not necessarily sure how big of a problem they are, but I, you know, don't just want to sit here and do nothing and, you know, let myself be the proverbial boiled frog. You know, and I know that there's not necessarily many of us right now who have this information and, you know, who are sort of, you know, I always use this Wesley Nielsen joke from Airplane. We're all counting on you, right? Like I, you know, I felt, I guess to be a little bit more concrete in my case with the GPT-4 red team. So this was late 2022. ChatGPT was not even out yet. I had been a customer of OpenAI and there's a couple of episodes on the feed. If people are new to the feed and want to hear the full, you know, long story version of this, there's a couple of episodes that tell it in different ways.

• 29:52Nathan Labenz?

  But having been a customer of OpenAI, I originally got access to GPT-4 as a customer preview. And there were a number of little like warning signals or sort of alarm bells that went off for me along the way. One was just like, okay, wow, this is a huge leap from what we had seen. And I had been in other, you know, customer preview programs. I was like pretty plugged into what their latest stuff was. But seeing GPT-4, uh, it was like, holy moly, like this is a massive leap. I asked, is there a safety review program for this? They said, yes. I said, can I join it? They said, yes. It was just like, okay, yeah, you go over to this other Slack channel where the red team, you know, chats. And then I was like, as I got into that, I was like, wow, there's really not much here. You know, there's like

• 30:36Nathan Labenz?

  two dozen people maybe that have sort of been, and I was, you know, one that had just raised a hand to join it. Not a lot of chat, you know, not a lot of guidance, no background information, no feedback on anything we were reporting. No safety measures at all at that point. The model we got was purely helpful. And then there was a moment where they brought us the safety model and they, you know, it was like the each, they weren't calling it GPT four yet at the time, but it was like, you know, whatever, whatever DaVinci oh oh two latest dash safety, you know, the sort of safety term was appended, you know, to the main model name. And we were told this model is expected to refuse anything in the, you know, content category moderation, content categories. Um, you know, tell us what you find. And we just found it was like

• 31:21Nathan Labenz?

  totally trivially easy to break. You know, the safety mitigations were not working at all. And I was like, yikes, you know, if you said it's expected to do that and you, and then it's behaving what how I'm seeing, like, how concerned should I be about your competence? You know, like, you don't seem to have a command of what your models are doing. Um, and so I felt all these things that you were describing, right? I was like, first of all, how much of a concern is this? You know, GPT four, we now know, you know, with the benefit of two years of hindsight and the whole community coming at it from a million different ways, was like a major advance, but not that powerful, you know, where it was gonna do, you know, irreversible damage. Uh, that was the conclusion I ultimately

• 32:06Nathan Labenz?

  came to just through pure, you know, individual testing as well. And that's ultimately what I kind of, you know, I did frame that in my eventual report to the board. But, you know, how, how concerned should I be about the fact that like, there seems to be, because I don't, I didn't think this model is going to be super dangerous, but I did see this sort of divergence where I was like, I've seen three point five, I now see four, the step change there compared to the seeming lack of progress made on any sort of safety or control measures, just felt like there was, you know, a widening gap that, was anybody even concerned about this? Like, is it just gonna get wider? Like, where are we going? No, couldn't get any answers to those questions. And I honestly didn't even,

• 32:49Nathan Labenz?

  I, I, you know, certain, I did think a little bit about this ladder, but the people that I was directly talking to were like, Basically not responding, you know, their, their, their marching orders were just like, don't share anything with the red team, like just take in their reports and that's it. Thank them and, and that's it, you know? We didn't know anything about how the model was trained. We didn't know anything about really anything. Uh, this was also before, you know, the 10 to the 26 reporting requirement. So there was literally no, you know, there's still basically nothing, but there was even less, you know, since then of like what thresholds would even trigger any sort of reporting. So I was just getting kind of stonewalled at the level of, you know, the natural interaction. And then it was like, okay, well, where do I go from here?

• 33:31Nathan Labenz?

  And it didn't even occur to me to initially, it didn't occur to me to go to the board. It also didn't, and I, you know, if it had occurred to me, I don't think I would have done it, uh, to go to any sort of regulator. Cause like, again, as you said, like, who would you go to? And how would they have any sense for what's going on? And then I was like, well, maybe I, you know, do I go to the press like? But again, who do I talk to? Who do I trust? Like, you know, I'm, is a good story going to be written? Would that even be good, you know? And there, obviously the landscape has changed now where I think like, it would be hard to do anything that would intensify the level of investment or interest in AI beyond where it currently is. But back then there was this, you know, sense that like, if people know about this, you know, then there will be even more, you know,

• 34:16Nathan Labenz?

  energy and investment and things will just accelerate and it'll get even further out of hand, which was something I was like, well, that I kind of take that seriously. But at the same time, like, it seems like it, it is already getting out of hand, you know? So I'm supposed to keep that it's getting out of hand secret so that it doesn't get more out of hand, like something didn't quite feel super right about that. And I think what I ended up doing, and I was fortunate to have a, a network, you know, basically that I could go to. Also, you know, having been around AI safety ideas for a long time and, you know, knowing a decent number of people that were thinking about it from different angles. I was able to go to people that I knew, and this is outside of the chain of command, but at least, you know, to calibrate myself and say like, here's what I'm seeing.

• 35:00Nathan Labenz?

  What do you think? You know, does this seem like it's a big deal? And if you do think that, like, what would you do? And again, I had to kind of create that all for myself, like come to the realization that, yeah, you know, it was worth, um, there was a whole other mess about the NDA that I, uh, wasn't actually asked to sign at the onboarding, uh, which was just, just a reflection of sloppiness, you know, in execution on the OpenAI part. There was some debate as to whether or not we actually had an effective NDA in place. Um, but regardless, you know, I kind of knew that like, they didn't want me talking about it. So it was, it was not, that part was not ambiguous, even though the legal side of it was a little more muddled.

• 35:40Nathan Labenz?

  But in talking to these people, they were like, yeah, that does sound generally concerning. And it was one of my friends who ultimately said, like, why don't you go to the board? Like a nonprofit, you know, that's why they're there. Um, and so that's what I decided to do. I did tell the people that I was working with that I was, you know, directly at OpenAI that I was going to do that. Um, they didn't, you know, do, they didn't really say anything much. Again, it was just sort of a like, okay, you know, Stonewall kind of, if that's what you're going to do, that's what you're going to do. We're not really commenting on it. When I did get to the board, you know, they didn't seem to be in the know. If one famous detail was that the board member that I spoke to said, I'm confident I could get access to GPT-4 if I wanted to. And I was like, well, that doesn't seem good.

• 36:25Nathan Labenz?

  Um, and then they, and you know, was it retaliation is an interesting question. Um, they did kick me out of the program pretty much directly after that. And that definitely sucked, you know, I was, I really, first of all, found the work extremely interesting to be doing this, like frontier exploration. So I've wanted to continue to do it. I was also like, just generally concerned from a public interest standpoint that like, I probably did. You know, 20% of all the red teaming of GPT-4. And you know, there were, there was like, Meter also, you know, was named in the report and did the, you know, did the famous captcha thing where the model, you know, told a task worker that it needed help with the captcha

• 37:08Nathan Labenz?

  because it was blind or whatever. So meters, you know, was also a significant, they did more than, more than me. I think other than meter, I think I did the most of anyone. And I was just like, you know, this is not, you're going to take me out of the equation when like, what you clearly need is like 10 times more than what you have. That didn't seem great. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 37:31Speaker 01

  Also, apart from being like directly maybe not good for their red teaming efforts, it's also just a strange sign of culture, right? That if you have somebody who's who actually cares, that's probably somebody you want to have on the red team. And then raises concerns to make sure those are actually heard, then kick that sort of person out. The not necessarily, I think, like evidence of what we would call like a speak up culture.

• 37:57Nathan Labenz?

  Yeah. And the grounds was, you know, that I had talked to people outside of the OpenAI umbrella, which was true. But, and I wasn't even really hiding that. I just said, you know, look, I've got some friends in the AI safety community that I sort of ran the situation by to calibrate myself. And I think it was ultimately to their benefit because like, if I'd been left to my own devices truly to decide alone, maybe I would have, you know, gone to the press or something. And I don't think that would have been ultimately the right decision. There was one other thing that really chilled me in that moment, which was I had started to collaborate with Meter. And so I was doing my own, just like direct red teaming, but also they have, you know, projects ongoing. I was getting involved

• 38:39Nathan Labenz?

  a bit. And when they dismissed me from the program, there was a, basically a threat made to meters access. They were like, well, we can't really, you know, we meter has access. I sort of said, you know, are you going to try to prevent me from contributing to their ongoing effort? And the answer was, we, we, we can't really control that because like they have, you know, an organization level access, so they can kind of do what they want to do. But we will take, you know, into consideration as we look at renewing our engagement with them, like who they're working with and whether, you know, whether those people can be trusted.

• 39:15Speaker 01

  It's very thinly veiled, isn't it?

• 39:17Nathan Labenz?

  It was pretty, it was a pretty overt threat to meters access, which I was just like, this is insanity, you know, it's just them and me and like a few other stragglers. And we were smart people, by the way, and I don't want to cast shade on the other, you know, red team participants, because I think I just happened to be in a place where I didn't really have a job at the time and was able to kind of put everything else down and do this full time. Not many people have that, you know, flexibility. So I don't blame them for not having the flexibility that I had. But it was nevertheless the case that there was like only, you know, a couple like people seriously diving into this.

• 39:51Nathan Labenz?

  And so, yeah, you know, I think if this third opinion thing had existed then and I had known about it, I would have had, I would have come to it and I would have, you know, been able to do a calibration on my concerns and sort of make a plan. And I may still have ended up in the same place because I still might have ultimately escalated to the board. But I might have been able to do that in a way where there was never this sense of like,

• 40:18Nathan Labenz?

  you talk to somebody out there that you weren't supposed to talk to. If I had been able to get the confidentiality, you know, guarantee that you're offering with a third opinion

• 40:28Nathan Labenz?

  angle, I think I might, I might still lean open AI's good graces today, possibly. Possibly I still would have been, you know, kicked out for having, you know, skipped a level in the chain of command or whatever. But at least the grounds, you know, that they cited for my dismissal would, would have been avoided. And I, you know, I think it was really, another thing I would say is just it was super consuming at that time. Obviously, testing GPT-4 itself was super consuming because it was like, there's a, you know, this thing contains multitudes and, you know, I can't possibly characterize it all, but I'm going to, you know, do my absolute best. So I was working extremely hard just on the object level work. But then also this sort of meta question of like, what should I be doing here? Was consuming, was a little crazy making. You start to have these sort of

• 41:15Nathan Labenz?

  heroic narratives, you know, pop into your head. I don't know how prone you are to that. I personally find that I have to sort of fight the idea that like, I'm going to go to the public and then I'm going to, you know, be the hero or whatever. And those ideas are not like explicit necessarily in my mind, but they are kind of, I can, I can become quite fond of them if I, you know, allow myself to envision, yeah, I'm going to do this or, you know, we're going to be in the New Yorker or the New York Times, whatever. I'm going to be on TV. Ultimately, I was proud of where I kind of came down in terms of like suppressing those, you know, sort of visions for my, you know, my heroic contribution. And I think I did handle

• 41:55Nathan Labenz?

  it pretty well. I do look back and feel like generally proud of my conduct, but I also like, think if things had just been a little bit different, you know, if I had, you know, just a little bit of other responsibility on my plate that was, you know, stressing out in some other way or whatever, I might have easily made like a much worse decision. And again, I think, you know, having the sort of counsel that you are offering with this third opinion network of expertise would have been really, would have been really great.

• 42:29Speaker 01

  Happy to hear, happy to hear. Yeah, there's many thoughts here. So maybe, maybe starting with like the psychological side, for example, I think also often underappreciated for like, like how heavy the strain is very much. Unfortunately, I think because especially cases can last for a long time. I think I'm not sure how long did it last for you? Sort of the whole process from I'm worried about this to I am approaching the board to, okay, I'm no longer part of the red team now. And like, maybe also worry afterwards, what's going to be consequences here?

• 43:02Nathan Labenz?

  Yeah, the whole thing was about three months and it ended for me with the launch of chat GPT, actually. It was two months of actual intensive testing, you know, ultimately talking to the board member and getting dismissed. And then I was like, okay, now I have a lot more time on my hands, you know, I'm not like testing the thing actively anymore. So I kind of resolved to just like, take my time and think about it a little bit before deciding really what to do next. And there was also no timeline to launch, which is another, you know, thing where I was like, do we have, I ask questions like, do we have a timeline to launch? Do we have a standard? You know, do we have a sort of, some sort of control level that we

• 43:44Nathan Labenz?

  need to achieve before they would, we will launch? We, you know, as if I was part of the team, I always like to take that we mindset where I can. But the answer is like, can't tell you anything, basically, you know, across the board. So when I was finally kind of like, okay, I've got a lot more free time on my hands, I'll think about this. I basically never got to the end of thinking about it because a couple of weeks later, ChatGPT was launched and it was a huge update where it was like, it actually did have some better control measures and it was clear that they were kind of, you know, launching something weaker first to try to iron out a lot of these issues before, you know, bringing the best thing that they had forward. And so there was also strangely this reality that like, the impression that they

• 44:28Nathan Labenz?

  had made on me, mostly through like refusing to answer any of my, you know, questions that they in fact did have somewhat better answers to than they were willing to provide. Uh, the impression they made on me was like just way worse actually than the underlying reality. So that was also a very strange situation. But yeah, it was pretty consuming during that time. And I don't, I still don't really know what I might have done, you know, if there had been no ChatGPT launch, because that was December of, I think like either the last day of November, the first day of December of 22, and we didn't get GPT4 until March. So there was like still several months. If they had had a different, you know, rollout plan or whatever, who knows what I might have done in the meantime. But in the end, I sort of was like, okay, well, the

• 45:12Nathan Labenz?

  world is waking up to this. You know, there's a lot here for, for a lot of people to unpack. And I think I've kind of done my part for now was kind of where I ended up landing on it.

• 45:21Speaker 01

  Yeah. And thank you again, by the way, for co-raising.

• 45:26Nathan Labenz?

  Very minor contribution.

• 45:29Speaker 01

  Absolutely. No, still, still, absolutely. Um, yeah, I think, as I said, like, I think even as you like just three months, it's still a significant time, right? Especially if it's like emotionally intense. And I think there's, so just depending on also like this, for example, this case, it was internal, right? How well systems are set up that can be in extremely stressful situation, especially if companies, for example, retaliate of a pattern of that, then it can be like there's multi-year processes. There's. Let me see if I have the numbers in format. All right now, a large share of light cases, for example, retaliation claims can last for five years, six years, seven years. I believe the Tyler Schultz, the Theranos whistleblower, I think, lasted many, many years.

• 46:14Speaker 01

  I think $400,000, he had to go in advance to, to fight from the law, on the law, legal cost side, which by the way, we can also help with, but that's a side point. And like, that's sort of the, the, the one side where it can be extremely like, basically all consuming. And like, not to mean speaking about other like negative impacts, like blacklisting in the industry, someone saw those sort of things. But there is also plenty of like examples from companies where, for example, internal whistleblowing works quite well. There's retaliation, unfortunately, is still somewhat the norm in, in some form, shape or another. But there's also like plenty of examples where companies handle it well and where it's really kind of part of the regular business process,

• 46:56Speaker 01

  processes, organizations, folks will come back to internal whistleblowers and say, oh yes, thank you for your report. We now keep you in the loop, they provide them with regular updates. So you know, you just don't sit there and wait and see if something is going to happen. So there's basically definitely like better ways to do this and worse ways also from like a psychological perspective, but also I think here again, the point being like, apart from there being these differences, if you find yourself in a situation like this, then like help is available and people like, as I said from many conversations, people are just not aware that there's organizations that are specifically like focused on also providing psychological support, guiding through the journey, even if it's at super early stages and like internal escalation rather than wanting to go public, for example.

• 47:38Nathan Labenz?

  Yeah, when, um, I guess, uh, well, for one thing, I had it relatively easy in the sense that my income wasn't, you know, depending on this, right? I didn't have, I didn't have a bunch of equity. I didn't have, you know, a sort of, a lot of upside that I was really putting at risk. So I think that did make my position easier than it would be for a lot of people that, you know, are employed and have just been promised a 1.5 million dollar bonus over the next 18 months or, you know, whatever the case may be.

• 48:04Speaker 01

  Hmm.

• 48:04Nathan Labenz?

  Um, that, I think.

• 48:06Speaker 01

  Oh, I believe, although the latest meta number was a hundred, hundred twenty.

• 48:10Nathan Labenz?

  Oh yeah, gosh. Well, yeah, that's a. Yeah, the dollar figures flying around are definitely going exponential like everything else.

• 48:19Nathan Labenz?

  So yeah, that that, you know, I just mentioned that as a way to indicate that, like, I think my situation is still on relatively easy mode. Also, just as a quick digression to give some credit, actually substantial credit, I don't want to say just some credit. And I don't want to be begrudging about it because I do think it's actually pretty good. We're talking the day after GPT five was announced and I read the whole system card yesterday. And I will say, major progress on a number of fronts in terms of the quality of the red team program, you know, just much larger, much more intensive. When a problem that we had at the time was very low rate limits, inability to do anything automated. It was all manual. That's been fixed.

• 49:04Nathan Labenz?

  Lack of knowledge around, you know, what they had already seen, what they had already tested, what they had already, you know, observed or what the inputs were to make any sort of inference from that. That has also been addressed now where, for example, meter in their report on this one is able to say, you know, we observed this, but we also were told this, you know, by OpenAI. And so between what they're telling us about how this was made and what we've observed, like we can, you know, get to a higher level of confidence on some of our conclusions than we would be able to if we only had, you know, our own direct observations. Even have now. Yes, sir.

• 49:41Nathan Labenz?

  Last thing on my mind, I think, to give credit, and again, I think this is substantial credit. Access to chain of thought also now has been extended to some of these safety review orgs, Apollo, I believe, and Meter. At least both got that sort of visibility into what the, you know, what the models are thinking, which they didn't have, you know, for like oh, one and oh, three. So I think there has been a lot of progress. My, my sense was like in, in late 2022, as of GPT four, I thought I was going to see something a lot more like that. And what I saw was sort of, you know, basically the sort of first warm up for something that, you know, now has kind of, at least meaningfully matured, not to say that it's.

• 50:23Nathan Labenz?

  Enough, but I certainly there's a lot of progress. I at least wanted to just, you know, for folks who aren't calibrated, I'm like where we are relative to where we've been like, they have come a long, long way and there is definitely, you know, some very good things happening.

• 50:37Speaker 01

  So I think they all were access earlier this time. I believe, I think I read four weeks pre-launch access this time, I think for meter at least. I'm not sure about the rest.

• 50:47Nathan Labenz?

  Longer access also.

• 50:48Speaker 01

  Yeah, exactly.

• 50:49Nathan Labenz?

  Definitely good. Although that has compressed, we had months and it was a six month window between end of training GPT four and launch. Now, you know, those time scales are shortening, but you know, they can do more with automated access and then they've got language model as judge. That was another thing that really, I don't know if it's good or bad, it's probably both, but it was striking to me in reading the system card, how much they are using language model as judge in their characterizations of the model. They're doing a lot of things where it's like, well, we, you know, used O three or even in some cases, we used O one to evaluate all these outputs. We validated that O three or O one or whatever can do a similarly good job to an expert by like working with an expert and refining the process

• 51:34Nathan Labenz?

  to kind of match their process. But at the end of the day, it is still like, yikes, you know, how we're like, we're starting to have the L L M's doing the alignment homework as, you know, as Eliezer used to put it. And I do feel like there's, you know, some, that allows the centrifuges to spin ever faster, but one wonders at some point if it also may lead to them sort of spinning off their axis and, you know, who knows what that looks like.

• 52:00Speaker 01

  The scalable oversight problem.

• 52:04Nathan Labenz?

  The other thing I wanted to ask is, and I guess another just frame for this whole project is, I am always really into what I call the unilateral provision of global public goods. And I think this is a really interesting project where.

• 52:22Nathan Labenz?

  You know, in a world where everything goes well, nobody ever calls you, right? It's sort of a strange. And maybe in a world where everybody knows that you're out there, you know, people kind of get their act together and have good internal policies. And again, you know, maybe nobody calls you. So that's a, that's a weird sort of situation to be in, right? Where you like, in the best case scenario, your sort of KPIs are like flatlining because everything's going well, but nevertheless, you may still have some influence because the sort of existence of these, you know, pressure release valves or safety nets, you know, it's not something that, you know, the decision makers are unaware of, or perhaps, you know, hopefully they're not unresponsive to it.

• 53:06Nathan Labenz?

  How many people do you think are like, like, how many people are we talking about here? Between now and the singularity, you have a sense for like, how many people are going to be in this spot?

• 53:18Speaker 01

  Super difficult question, obviously, right? I think, I mean, the way, of course, we think about it's all the ones possible. We want to make sure that all the ones who are in that situation are aware that support is available and that there is hopefully a better way to do it than the sort of default path they would have chosen otherwise. So indeed, like it's neither, it's not that we say, we want to have a thousand whistleblowers. We don't want to have zero either, just for we'll have to make sure we're ready. So I think.

• 53:47Nathan Labenz?

  Like, how many people do you think are, because another big trend, of course, is the organizations seem to be getting more secretive. Dario recently said in an interview that while we have a, you know, very open culture, we also have like a need to know basis for key things. There was recently somebody who left OpenAI and wrote, I forget the guy's name, but it was, he was the founder of Segment Technology, who then went to OpenAI for a while. And on leaving, he was like, you know, here's my experience. In some ways, positive, you know, like people are really trying to do the right thing. You know, people care about safety, all these kind of qualitative statements, I thought, sounded pretty encouraging and no reason to doubt he's being honest. Flip side of that was, he's like, extreme secrecy, you know?

• 54:29Nathan Labenz?

  Many times I couldn't tell my, you know, next guy over what I was working on and they weren't telling me. So I guess I, how many like? Insiders do you think there are? I guess what I'm getting at is I think it might not be that many.

• 54:44Speaker 01

  Yeah.

• 54:44Nathan Labenz?

  And it's sort of like, all this work, all this preparation, might be for like, perhaps a quite small number of people, but where the stakes on each one of those interactions could be like quite high.

• 54:56Speaker 01

  Yeah, I think that's fair. So I think we're probably talking about a few thousand individuals globally. That's probably roughly, at least in sort of the core, the core, like larger maybe control problem type stuff. There could also be other issues where you really have like a full picture. On the fringes, you may have a lot more. So like, you brought up the eval companies before, right? So at the moment, for example, I believe it is also not clear whether they are allowed to use the internal whistleblowing systems, for example, right? And of course, there is an unfortunate sort of conflict of interest situation where, yes, they're independent, but also AI can refuse access in the future, right? So they're in a bit of a, bit of a tough situation.

• 55:40Speaker 01

  And so it's, I think, important to not only think about, okay, who are like the people directly inside, the organizations that maybe who are also on the fringes, who could spot concerning behavior. It could also be, could also be suppliers, could be employees of suppliers, maybe on the training side, for example. I mean, we've seen, for example, issues around, this is sort of going in a slightly different direction now, but significant trauma from like data label, from data labeling, from content moderation, I think we've seen a bunch of areas that would also be space. Of course, this is not like probably when you bring up the singularity, maybe not the sort of concerns you're necessarily thinking about, but of course, they're still relevant. For one, of course, directly for the individuals to suffer, but also kind of indicating maybe

• 56:20Speaker 01

  of a culture that doesn't necessarily care about maybe weaker members of society. That's maybe one way to frame it. And then sort of you have a larger space of people who can observe behavior and don't need to have all of the context available. If you're talking about like those probably really few, really highly critical issues, then I think you're probably right. I don't think we are expected, at least something on the public side, like thousands every year. Of course, we'd want companies to also address concerns really well internally, which would then also look like, you know, there is no external whistleblowing, but in fact, they're just moving in the right directions. If you trust that the companies themselves are like set up in a way and have the right

• 57:03Speaker 01

  incentive systems to rectify sort of issues in the public interest.

• 57:08Speaker 01

  So yes, I mean, if I ask you for a number, probably depending on what the timelines are for the singularity, it could be in the dozens, maybe something there.

• 57:16Nathan Labenz?

  Yeah.

• 57:17Speaker 01

  Not sure if that's a viable answer, but.

• 57:20Nathan Labenz?

  Whatever. Kind of what I would. That's that checks out. That's kind of where I back out to as well. So let's talk about the survey that you've run. And again, very kind of quiet. I think it's maybe even still in process, but I guess there's enough initial results to um. And you know, you can even talk about sort of the way that you've distributed the survey to try to make sure you're getting high quality results. And you know, all these things are kind of fraught in this context because people don't necessarily want to like validate with their work email that they're taking a survey on whistleblowing related issues. But yeah, take us through the the survey, a little bit of kind of how you set it up and how you make sure you're actually hearing from the people, you know, mean to be hearing

• 58:05Nathan Labenz?

  from. And then what have we learned about the state of whistleblowing awareness, support, uh, policy, et cetera, from the insiders who have responded?

• 58:17Speaker 01

  So the, yeah, we ran this service indeed. It's still ongoing, actually. So we basically have a few kind of dedicated survey links that we spread through our network and like people more directly, sort of dedicated per um, per AI company. And it's fully anonymous. We didn't like gather any names, any contact details associated with responses. So basically fully anonymous in that sense. And then we also launched a more public call for responses. Actually, this, so this is still ongoing. If you are listening still and you're an AI insider, uh, working with the frontier companies, maybe we can put the link to the survey in the show notes. So if you want to contribute, you still can.

• 58:59Speaker 01

  Maybe major insights. So I think I'd shared something around like clarifying concerns already. I think I gave them, gave one example, two, but like we see roughly, roughly half of respondents believe that, you know, they have a low confidence in the ability to assess and judge risks. I think really mirroring Nathan, what you said, right? So like this one rephrased quote would be something like, it's really challenging to distinguish between appropriate and inappropriate concerns. I can see how there's a risk of escalating minor issues into major crises. So both, I guess, for the individual, but also from like a boy who cries wolf situation where of course you don't wanna not overblow every situation to like Armageddon. So this is a real concern. At the same time, of course, normally overly adverse to that risk because it might down the line, it might well be.

• 59:44Speaker 01

  It might be really a meaningful thing. Government outreach, we talked briefly about as well. So I believe a hundred percent actually are either not confident at all or not very confident that response would be acted upon, understood by the government. So I think here a little quote is something like with knowing the, without knowing appropriate contact person or agency, I wouldn't attempt to reach out. People uh very strongly supported the idea of having one dedicated reporting channel to come to. I think idea here is also from a quote like normalize, institutionalize, whistleblowing, make it routine, anticipated practice, you know? That it really just becomes part of regular work. This is not something to be looked out upon or to be avoided.

• 1:00:29Speaker 01

  There's a bunch of benefits, by the way, for companies for this too, but maybe we can get to it later.

• 1:00:34Speaker 01

  Um support infrastructure is unknown. So I think maybe if we think down again, the journey path, right? We talked about lacking legal protections, which is a big issue, which we need stronger clarifying concerns we just talked about. And then there's also the question, who can help me, right? So we talked briefly touched on it in the psychological space on the legal advice side, because you maybe have to get creative with finding a legal basis for getting whistleblower protections, slightly easier in California, maybe than in other US states, but it still can be tricky. And we saw, I think ninety percent of insiders didn't know any whistleblower support organization. I think it's actually over ninety percent. This is the same in like conversations insiders. Interestingly enough, even with previous AI whistleblowers.

• 1:01:17Speaker 01

  So I think this was a person at at Google from like a few years ago who raised concerns around research misconduct and was terminated. Eventually they settled, which I think is not officially a sign of like unlawful termination, but Google tried to throw the case out and it didn't succeed. This person essentially got lucky, this is their words, that they had a good friend who connected them to a lawyer, who then had luckily some background in whistleblowing law. But people again just don't think about, I may move into a territory now where I'm becoming a whistleblower, I should get whistleblower advice. Like it just feels like escalating, right, to incest, especially if you have a good relationship with the organisation,

• 1:02:01Speaker 01

  which most of these individuals do, right? And so that is something I think we definitely want to see change over the coming years, that people understand there's these organisations they can help reach out to. Because it also then avoids, as I said, like unsafe other approaches, like for example going internal or going to regulator just by yourself without doing the right things, like for example, a classic case is like, to the SEC, yes, you get protections, but you go public first. So if you go public, you won't, for example, get the bounties. There's been like a few SEC, like David, you know, the SEC has a great whistleblowing programme actually, where they award percentages of penalties on companies based on the whistleblower's information. But if that information becomes public first and then they file with the SEC,

• 1:02:43Speaker 01

  they, the SEC basically claims, I think historically has claimed, yeah, this wasn't new information anymore. You may still get protection, but you don't have a right to the bounty, if I recall correctly. Or for example, I think we had this in our survey as well, people basically said, I don't really know, like the only, if there is something important, the only thing I can do is go straight to a journalist. Which may still be true, although, again, legal disclaimer, we're not counselling anybody to take any unlawful actions or violate their contracts. But for a person who is really dedicated to resolving this issue, that may still be a path that they may want to take and that may be effective, but it shouldn't be the default one that people think, okay, this is the only one I have available, like there's people willing to support and willing to help.

• 1:03:26Speaker 01

  And then I think the, the last part is around awareness of internal channels. So this is coming directly from a lot of conversations we've had recently, given also the campaign we've launched and also from the survey, is that there's extremely low awareness of internal. Like whistleblowing channels and ways to raise concerns. So I think more than half, sorry, fifty-five percent of respondents were in companies where those policies claimed to exist, or companies claimed those policies exist, did not even know that they exist, did not know where to find them, if they did. Many were not trained at all. The last

• 1:04:08Speaker 01

  large, vast majority was not trained at all. Some difference to be across companies a little bit here. And for example, people still, although we don't know, I'm not going to say which organizations it is, but still seem to be confused around whether escalation to the board is permissible or not. They don't understand what those policies actually say. So the only company that has published its whistleblowing policy to date is OpenAI, and it actually reads pretty well, but there is still a bunch of issues in there that are not obvious, though, to an insider, because you know, you're not spending all your day reading whistleblowing policies. You never chose to read this document in the first place, probably. You would have preferred not to ever have to read it. So it may read well, it may give the impression,

• 1:04:52Speaker 01

  okay, I can, this is great. I'm just going to do this now. And you may not understand that actually you're exposing yourself to significant risk. And there's plenty of evidence, actually, in these companies of still retaliating against insiders. One case at Google I just mentioned, who went internal and got lucky that he used a few keywords in some of the escalation emails, which then served him well, sort of down the road. I believe it was fraud was the word, which then gives reasonable cause to believe there was a crime, which then unlocks the whistleblower protections. But so basically, you either have no awareness they exist, then maybe you don't understand them, if you see them, or you think you understand them, which may lead you on the wrong path. Or maybe you do understand them, but you just don't trust

• 1:05:34Speaker 01

  the organization at all. So there we also have cases where people have things said like, well, rephrased again, anticipate using official reporting channels would like result in subtle indirect consequences rather than overt retaliation, like termination. On the one hand, that may be the case. I think we do still see a lot probably of like also overt retaliation. But yes, this is probably also likely. But again, I think this just speaks to people not really trusting. The systems internally, which is likely the case, also because companies don't really create a lot of reasons to trust the system. Because they don't publish anything about their systems to public scrutiny. They don't provide evidence to the public. We also have no, at least as far as I'm aware, and we've looked quite a bit recently.

• 1:06:18Speaker 01

  There's no point is that companies, these frontier companies would, for example, internally create transparency, which is very common practice to basically say, hey, so as you know, we have this internal way of reporting concerns to, for example, the board. Last quarter, we had X cases, Y of those are still open. Satisfaction with whistleblowers internally was NPS whatever percentage. We had that many appeal cases, X percent of cases were deemed outside of the scope of the policy. What happened to those cases? What retaliation did we observe? We're not seeing any of that, neither in public, nor I think even in the companies internally. So companies could do a lot more here to probably, for one, improve the systems, because

• 1:07:00Speaker 01

  we have a bunch of evidence that they don't work, and they are actively retaliating against insiders. And also to create, then after that, trust that the systems actually work, both sort of for the insiders and for the public.

• 1:07:14Nathan Labenz?

  Can you say a little bit more about the evidence of retaliation? I mean, we've heard the Leopold story is the most famous one that comes to mind for me. You alluded to at least one at Google.

• 1:07:26Speaker 01

  Yeah.

• 1:07:27Nathan Labenz?

  Is this just something that you're gleaning from comments on the survey or is plus conversations, of course, or how much more sort of, I don't know.

• 1:07:38Nathan Labenz?

  How can we make that a little more substantive for people so that they sort of have a sense for really what that's like today?

• 1:07:43Speaker 01

  Yeah. Unfortunately, there's very little data, as you can probably imagine, on the actual retaliation experience. So this is mostly coming from, so maybe I should have said this, we are kind of in close collaboration, hosted by the whistleblower netzwerk, which is like Germany's longest standing whistleblower nonprofit, where we also get, of course, a lot of from the expertise side from. So it's talking a lot to, from our and whistleblower support organizations who help people who experience retaliation, because it's all. And it is from there. There's not incredibly great data sources on it. If you look at public cases, you will of course see a lot. Um, that's the nature of cases also got public and become a really big story then. But I think you can probably pick almost any tech company that you will, and you will find pretty intense cases.

• 1:08:28Speaker 01

  Apple, for example, has fought Ashley Gilvick for quite a few years, uh, who raised concerns internally around workplace, workplace safety and tried to wiggle their way out in multiple ways. I think this lasted over five years, eventually leading to them actually publishing their whistleblowing policy because the, the pushback was so significant, but I think you can probably also see it in just the cultural attempts, like for example, Daniel's story of aiming to suppress any raising of concerns certainly goes in that direction. There's a famous Timnit Gebru case from 2019, I believe, around research practices,

• 1:09:08Speaker 01

  which she was fired for because she basically published a paper criticizing, I think it was primarily discrimination and bias in models, in AI models, but there were quite a few other items as well. And a colleague of Margaret Mitchell also got fired from Terminator, I believe, um, a year afterwards for again raising sort of similar concerns. There's quite a few cases that we see and of course the dark number is likely higher because we're not really getting a lot of transparency from these companies to understand, you know, to what extent are these systems working or not.

• 1:09:44Nathan Labenz?

  Yeah. Is there anything you could say more about the taxonomy of concerns? I mean, you kind of alluded to this a bit with like,

• 1:09:53Nathan Labenz?

  I guess we have things as potentially broad based as like working conditions on the data creation side. And then there is the like.

• 1:10:07Nathan Labenz?

  You know, sort of securities fraud type stuff of hyping stock with, you know, claims that may or may not be fully true. Then there's like commitments that have been made around voluntary or otherwise, excuse me, mostly voluntary so far, that companies may or may not be fully, you know, falling through on. Obviously, we know some instances where they're not. There's like the, you know, there's also the sort of late policy change thing, which we recently saw Anthropic do, which isn't necessarily bad, but like, it was certainly weird to see, you know, an RSP change a couple of days before, you know, the launch of the next big model. That's one way to follow the RSP. And again, not necessarily anything wrong with it, but you know, it's interesting.

• 1:10:52Nathan Labenz?

  And then there's stuff like, I would assume that maybe the most sensitive is like, we are seeing X model behavior.

• 1:11:02Nathan Labenz?

  How would you, would you add more to the rough scaffold of different concerned types there?

• 1:11:10Speaker 01

  So I guess roughly probably falls back into like, what are the three major categories of like risks from A I we be concerned about? And I know there's a lot of discussion around here, right? So like, which are the ones we should be prioritizing? Which are the ones which like one should be focusing on? Instead of the whistleblowing space, the good thing, I guess, I'm using quotation marks here, so it's like catch all. And that's the thing what makes it so powerful as for one, also for regulators to be able to catch risks that we cannot foresee, because I mean, this is sort of unfortunately the the name of the game, that we're not really quite sure what risks are going to be substantial and which are not. So of course, we can talk about like, you know, where do we, where do we see major risk areas?

• 1:11:52Speaker 01

  And what examples do we have? What stuff maybe are we particularly concerned about? I think in general, the big strength right of whistleblowing as a tool is that it finds the actual items of concern sort of when and where they arise. And as uncovered by individuals who don't have conflicts of interest in the sense of maybe heavy competitive pressure incentives, where maybe a chief executive may not want to reveal.

• 1:12:19Speaker 01

  Bad conduct. So roughly, of course, like three guys would be, is there misuse risk? Then you have maybe control risk and you have systemic risk, you know, those three sort of classic categories, I think. Altman used like the same ones as sometimes recently too. In all of those, of course, we'd be interested to see revelations if we're moving in very bad directions. So I guess on the misuse side, like, are we seeing ignored? Like, is my misuse being ignored? Like, do until a monitoring show, okay? We're seeing really bad behavior here. You don't even necessarily, I think, have to go into like the bioweapons direction, but of course, that's like a prominent case. Uh, are people using our models for really nefarious purposes? And maybe the companies are not doing anything about it.

• 1:13:03Speaker 01

  Because they don't really care. They don't have the capacity to. Um, could also be not investing into transparency along those lines. So I know, I think at least of one of the frontier companies after like a pretty major release, basically had no monitoring life. For, I think, around three to four weeks, it was just sort of broken. They only recovered briefly or like recovered it over time. That does not seem good at the moment. That's fine, roughly, right? But probably down the line, significantly less fine. That's probably what we'd see on the frontier side, where if we noticed, okay, companies are really not investing in transparency at all. That would generally be very interesting. There would have to be some sort of violation of the law underlying it to unlock the legal protections in the EU that may already be covered.

• 1:13:49Speaker 01

  Under the AI Act, if basically a company is essentially flying blind, they're probably not fulfilling their reporting requirements. To the EU on being able to, you know, manage systemic risks. And they probably don't have a good view of the risks which they should have, according to the EU AI Act. So that could already be something, at least to get protection if you're covered by the EU. In the US, probably a bit more complicated at the moment, because we don't have anything yet. The whistleblower protection would do quite a bit here. Then you could think about, of course, control issues, whatever those look like. Internal deployment, I think, and Bluetooth briefly before is a very interesting one. It can specifically fit for, and you actually mentioned like scalable off the site as well, right?

• 1:14:32Speaker 01

  Like probably specifically fit for whistleblowing, because it's by default internal. There is no way for externals to to look at it. This is likely well before, you know, like a meter or an Apollo. Look at it, although I think meter also said that in the current eval, they already extrapolated or tried to extrapolate, like, what does this mean for internal deployment? But they basically can't look at it, right? So this seems to be an extremely high potential, in quotation marks, area where we may be able to see something in the future. On the systemic risk side, of course, would be interesting to see how much this kind of may be going into monitoring misuse direction. What are we seeing around maybe political propaganda?

• 1:15:15Speaker 01

  Are we seeing people maybe getting more and more reliant? These are classic AI risk topics, of course, right? Like, works on people using these models, potentially sycophantic models, as like psychological help or to form their political beliefs. Are there issues around here? And then probably outside of the concrete risk, you can probably take it one notch up and talk about the organizations behind them. And like, what does the leadership, the culture look like? To what extent are they trustworthy? And can they be trusted with kind of steering us in the right direction? So this could even be like surface level things that maybe already covered like completely well at the moment around widespread fraud, for example.

• 1:15:58Speaker 01

  So are we just seeing dishonest, dishonesty in culture in general? Widespread discrimination issues could fall into this. It might be copyright violations, things like that. That's for one, yes, there's a direct crime there, although I guess the precedent is also still being established. What is in fact a violation of the law and what is not? But that could also just point to reckless cultures, which we may, we do not want, although I'm not sure to what extent we would still update towards certain organizations, at least that are most famously known for copyright violations, that they are reckless, I think we probably are pretty much there already. Research fraud goes in that direction, punishing, risk raising, risk reaching up. I mean, this is basically the Daniel Cucatayio case.

• 1:16:39Speaker 01

  Like this does not seem like a culture that seems to be able to deal well with concerns and people raising concerns. It could also be matters like release rushing, sort of besides the actual negative impact. And I think this probably going to your point around before around red teaming, like there's the one aspect, which is this could just be like kind of object level bad, that what this model is going to do in the world, how it's going to be used. And then there is, shouldn't these organizations be managing these models in a better way? And that's the governance side. And then probably other areas are like political influence taking to what extent is maybe certain lobbying occurring that points in directions where the interests of these companies are not aligned with the interest, let's say, of the general public.

• 1:17:24Speaker 01

  So those are maybe broad categories. There's something around arms race acceleration, like capabilities, but that's probably going a bit farther.

• 1:17:32Nathan Labenz?

  Yeah, that, I mean, that's a, that's a thorough taxonomy. I appreciate it. I guess one, um, if I was to red team the whole concept for a second, I guess two concerns that I'd be interested in your take on. One is like, you know, the, my dad's old, uh, saw is, you know, the real scandal is what's legal. But that's not exactly the right, uh, way to think about this. What I'm thinking of is like, we have XAI, you know, who has Grok three calling itself Hitler. Uh, and going like, you know, pretty hard, right? A, in a pretty bad direction. It then, you know, brings Grok four online with a live stream in which

• 1:18:17Nathan Labenz?

  the whole Hitler behavior from the model is not mentioned at all. And then, you know, we see reports of Grok four, like searching for what Elon Musk thinks about things in order to determine, you know, what its maximally truth seeking answer is supposed to be. And I guess one question is like, How do you think about try, if you have a whistleblower story, you're going to try to make an impact on it somehow, some way? How do you think about the fact that like, some of this stuff is just happening in plain view? And nobody seems to care. It's like, wait a second. Can anybody possibly have a scandal that is more scandalous than, you know, Grok 3 to Grok 4, Mecca Hitler, you know, sweeping the whole thing under the rug?

• 1:19:02Nathan Labenz?

  Like, that's all in the, you know, literally in the public domain at this point, right? So how do you think about the, maybe secrets just have a, you know, different quality to them that, I mean, there is something about that, right? I mean, that's often been remarked on with Trump, where it's like, because he puts it all out there, people sort of shrug their shoulders at it. Whereas, you know, in the past, things that were kind of covered up, you know, there was a sense that there was, you know, shame or wrongdoing about it. And maybe that makes a big difference. But how do you think about this contrast between, you know, just such seemingly flagrant things happening in, in the open and, you know, what people might bring forward, you know, that was previously a secret?

• 1:19:41Speaker 01

  Hmm. Let me, let me think about this for a second. So I think, tough question indeed. Probably three parts to it. One is sort of, yes, if there were stories in the past that looked quite bad, and maybe the next news cycle came and it all washed away and we're not really doing anything about it, can be true. But also true that there can be very different scales of disclosures and issues being uncovered. And I think probably this is when we're talking about like maybe the dozens or something before. That's probably more going in that direction, yeah, rather than, than Mecca Hitler, probably

• 1:20:25Speaker 01

  I assume people had somewhat made up their mind around XAI in the direction that X had taken and Musk was taken, possibly also before. I think if probably Mecca Hitler had come by, was released in like part of ChatGPT, that would have probably maybe been a bigger story because it would have been a more drastic change. But this is nitpicking on the specific story now. Like, I think the overall point generally stands, but I think the scale of stories can still like differ dramatically. And we've definitely seen a lot of like whistleblower disclosures in the past have major impacts, right? Next one probably is, would you rather not know?

• 1:21:03Speaker 01

  Right, uh, transparency, having transparency of what is going on is still significantly better than not having it. And of course, we can, like, we can do another two-hour podcast on, like, should we still trust, like, democratic sense-making processes? And to what extent is actually attention on an issue directly translating into, like, intervention? Or does it serve as, like, a good deterrent to know as a company that things are gonna come out? I would still think yes, uh, to quite an extent, and for the X example, for example. I mean, there, I think the, uh, the, the numbers at least post-acquisition still don't look super up, as far as I'm aware.

• 1:21:44Speaker 01

  But that's, besides the point, I think, overall here, basically transparency, I think in general is still better than not having it. And we need to have faith in something and need to, at least to an extent, believe, yes, that if real misconduct comes into public, that does gonna have a deterring effect, and there is gonna be some rectification. And if not, then it's democratic processes to make sure that it hopefully happens in the future, it has got rectified. And probably last one is, we cannot rely, of course, solely on whistleblowers to fix all of these problems, right? Same as we cannot rely purely on individual courage. That's why we have to make it easier and safer for people to speak up to create that transparency. But likewise, we need other guardrails.

• 1:22:29Speaker 01

  We, whatever that looks like, I think, I mean, you can probably take more European approach when you go to on the regulation side, or I think maybe the more American approach now, which seems to be going a lot more in the competitiveness direction and being less involved. I'm not gonna comment maybe here today, what I thought think the right approach is, but definitely cannot all rest on the shoulders of whistleblowers, it's also true, but I think it plays an important role. I'm not sure if that, uh, if that answers your question, like, a satisfying, that is like, man, I would be interested to hear, to hear your thoughts on it, like, but what do you think?

• 1:23:04Nathan Labenz?

  I don't know. I think it's very hard to understand why certain things hit and other things don't. I do think that...

• 1:23:12Nathan Labenz?

  One thing that made the Daniel Cucatello episode extremely compelling to people was that he had been willing to forego the stock, you know, that he had been willing to put such skin in the game personally. I think it was even further compelling that he sort of did that quietly and only, you know, like, it sort of came to like gradually with a random, you know, comment on a blog post here and people asking questions there. And then it was like, wait a second, this you did this and this is what happened and that this is what they asked you to do. And so there was maybe a couple of elements there that the skin in the game, the sort of stakes, the personal stakes that everyone was like, okay, this dude must be really serious. And also the kind of community uncovering process maybe

• 1:23:57Nathan Labenz?

  contributed to why that broke through when, you know, when, when the Leopold one didn't as much. I mean, obviously, his, his

• 1:24:08Nathan Labenz?

  Situationally Broke through, but did his story of like being retaliated by the board breakthrough so much? Not really, I would say. And maybe that's more because he was kind of already selling something else in a way, you know, and it was like a footnote in a, in a larger story or, you know, it was easier to for people to be like, you know, to kind of file that under, well, this guy's like, he's promoting his new thing now, right? So I think maybe felt a little different from how Daniel was literally just like, yeah, I left however many million dollars of stock comp on the table because I wanted to be able to say what I wanted to say.

• 1:24:49Nathan Labenz?

  One other kind of red team question on the concept is secrecy. My sense is that like, maybe this is already just super baked in, but it's at least worth thinking for a second. And I know you have around, you know, how do we, how do we not exacerbate the problem of intense internal secrecy at the companies? That seems to be largely commercially driven. I don't think it's like going to be, you know, moved that much on the margin by the existence of a whistleblower support org. But do you have any thoughts on how to, you know, at least not make worse and possibly, you know, push back a little bit on the intensive internal security that just keeps the number of possible whistleblowers so low in the first place? Maybe Claude will be the whistleblower. That's one answer.

• 1:25:35Speaker 01

  Wasn't this, wasn't this also?

• 1:25:37Nathan Labenz?

  The AI whistleblower, yeah, could, what if the, what if the AI whistleblower is in fact the AI?

• 1:25:42Speaker 01

  Yeah, exactly. There's a, yeah, was this again? Was it also meter paper? I can't quite recall it, where Claude was reaching out to the SEC directly and the FDA. Uh, just a while ago.

• 1:25:51Nathan Labenz?

  I think it was just, I think that was just anthropic work internally, but yeah, it was, it was in the Claude four system card. Uh, where it was, yeah, deciding like, it was also blackmailing engineers at times. So, you know, the behavior is, uh, missed, but.

• 1:26:06Speaker 01

  Sure. I think Ryan Greenblood wrote about this at some point as well. Um, so I think very tough, very tough. I think sort of once like culture shifts in that direction can be very tricky. One angle of course is sort of the regulatory one, you know, if you're not going to ask nicely the new force. And you require transparency kind of managed by law, which is guess, you know, the EU Act and also the Code of Practice for the implementing the EU Act for the general purpose. I, and the model providers has a large transparency section. That is one angle I think also in the States overall, right? Considered like a good baseline to create more transparency. Can you rely on the self-disclosure? Hmm. Difficult, difficult.

• 1:26:51Speaker 01

  What we've seen as sort of a side effect. So like, sorry, like legislation side of course is regulation in general and transparency. Then there's the whistleblower protection legislation, which is, that's why it's so incredibly important, right? That you make sure that it's clear that people can come to a regulator directly and speak up. That's how, how you counter it, if that's, if that is well set up, right? So that's why the whistleblower protection act, for example, is also so valuable because it's like quite broad in basically covering any sort of concern, as long as it's substantial and specific, which is a whole other topic around like public harm or, uh, public health concerns.

• 1:27:32Speaker 01

  So that's one angle, of course, we can create that transparency, huh? And another angle here is that we've seen this a lot, where if legislation pushes ahead on whistle law protections, then internal cultures become better. So internal speak up cultures become better because companies then know, okay, uh, for example, in this case, I'm not sure who exactly is going, would do it on the, um, uh, on the US side, for example, if the main recipient body, if there's one to be set up, would run around and it could form all of the employees of frontier AI companies, hey, we are here, you can come to us, it's super easy, we'll preserve your anonymity, as the S S C has done pretty successfully, actually. Then companies know, okay, if we want to make sure that those, that stuff doesn't come out

• 1:28:14Speaker 01

  and doesn't go towards the regulator, then we'll have to improve our internal systems. And we've seen that a lot in the EU already, after the introduction of the whistle blowing directive, like internal systems have become significantly better. Like I think Transparency International did a great study comparing internal systems between twenty nineteen and twenty twenty four. I think more than seventy companies in the Netherlands, I think, was that, so we checked that it did. And they had like pretty dramatic improvements in terms of internal speak up culture, fraud being or misconduct being detected, protection of internal whistleblowers. By the way, Google ranked last in that study, a little side note, also because they're not transparent. But ASML was actually in the top ten, so at least something in the AI value chain.

• 1:28:56Speaker 01

  And it was pretty high up. So I think asking for like, hey, you know, why aren't you being a bit, you know, maybe, maybe more open about things, also in terms of internal knowledge sharing, very tough. Um, pushing and is one angle, and then the other last was probably convincing that there's plenty of benefits from improving like internal speak up cultures and information sharing. So plenty of like empirical studies around better speaker cultures leading to much stronger innovation. So I think, for example, also for that reason, there is like some upper limit to the extent to which you cannot share information internally. I think you alluded to it before, right, that like

• 1:29:38Speaker 01

  people feel they like less and less information is being shared with them. I think the Dario statement that you referenced. I definitely definitely hear that from certain organizations like this is just hearsay of course and not a representative sample. You can just share a lot. We have heard like OpenAI seems to be relatively siloed in terms of information sharing. They also have a lot of leaks, which you know is probably, you know, firing up the, like they both sort of probably enforce each other. Where insiders feel, I don't understand what's happening here. I cannot get the information. So the only kind of real option I have, if I don't trust the internal channel or understand it, is I'm gonna go public with it. The company sees it, is trying to suppress sort of information sharing even more, right?

• 1:30:21Speaker 01

  It's like not a, not a good cycle to be in. We had other organizations that for example seems to be a bit different, where there's a lot more information sharing still. But it's still like coming back, I think. There's probably some upper limits on the extent to which you can limit information sharing, just because you have very intelligent people working there. Who need to understand the context of the things they work on and they want to understand the context of the things they work on. And so you probably cannot, you know, uh, set every single researcher, not given the context on what they work on. And say, okay, just solve this like miniscule problem here. It's probably not gonna unlock the research benefits that you want if you want to make good strides.

• 1:30:59Speaker 01

  So having more information sharing internally is something probably that companies naturally should gravitate to, at least for like the mid and long term. In the short term, there's probably benefits to like limiting info sharing. But like in mid and long term, if you get rewarded with uh better innovation, better research capabilities. And having strong speaker culture like internal channels, like empirically proven to lead to stronger employee loyalty. A stronger employee satisfaction, improved processes. There's like a great paper out this. Like more than a thousand companies were surveyed around like what their benefits and drawbacks were. So that's probably the last angle of convincing companies, making companies understand.

• 1:31:40Speaker 01

  That we're sort of all sitting in the same boat and I can imagine this probably sounds a bit naive. But this is probably another angle of saying, hey, secrecy is actually probably not the way to go. And it's also in your interest to to do better here.

• 1:31:55Nathan Labenz?

  Cool. That's great. Two last things, I think, on my agenda. One, let's talk about the announcement and push on this publish your policies campaign, which is kind of the occasion for us to talk, but is also, we're getting to it late, but we shouldn't neglect it. And then I want to give you just also one more chance than kind of closing, and you can raise anything else you want to, but you know, just to kind of describe again for people who might want to avail themselves of your support at some point, like tell again, like what that process, you know, will be like, what can they expect in sort of, you know, the most concrete experiential terms that you can. Let's do the campaign first and then we'll do that.

• 1:32:35Speaker 01

  Nice. So, yeah, on the campaign, I've alluded, I think, to like on several points already, kind of coming from the perspective of an insider, we've taken it, I think, a few times already today. The struggle is just massive around being able to understand how can I raise concerns internally in like a safe and protected manner and trust that these concerns will be handled well. And a big reason for this is because companies do not publish their whistleblowing policies. So like whistleblowing policies, maybe I should have mentioned that before, is basically like a document or can also be interactive tool, could even be a video that is provided to employees or covered persons. So a lot of companies include, for example, independent contractors, like, for example,

• 1:33:19Speaker 01

  or independent parties in general, like eval providers should be covered, for example, but not clear if they are at the moment, at least by OpenAI, who are the only ones to publish their policy. So basically this document explains the whole system to covered persons and also the public. And says basically, this is our whistleblowing system. This is how it works. This is why you can trust it. These are the recipients. We're going to look at concerns. This is how they investigate. These are the protections against retaliation we provide. This is why this whole process is independent. Again, you can trust it. These are the sort of areas of misconduct that you can raise concerns on, and these are ones that you cannot raise concerns on. So for example, like specific individual HR matters, something like that. It would say, no, this is not the right channel, go here.

• 1:34:04Speaker 01

  Or there may be explanations of for these types of issues. This is what we do with these types of issues. This is what we do, something like that. So it basically lays out the whole system.

• 1:34:14Speaker 01

  That's like the whistleblowing policy part, and then there is the sort of reporting evidence part. We in our campaign, we structured this into level one, level two, where level two basically says, what evidence our company is providing that these systems work or don't work? That's, you know, in terms of transparency evidence, that's also fine, right? Because also, by the way, no kind of organization gets it right the first time. It's as any business process, it's something you work on again and again and again and again and again. And so basically this would include things like how many reports were even received? How many of them were anonymous? So for example, having anonymous channels is super important. What you maybe want to see over time is that there's fewer and fewer anonymous outreach, because people gain trust in the system.

• 1:35:00Speaker 01

  If you have less and less anonymous outreach, it probably maybe points in like the other direction. Then you want to look at things like retaliation, that's like how many retaliation complaints are there? What are appeals processes? So like how many appeals are filed where people are not satisfied at all with the outcome of their case? Response timelines, these are things, satisfaction whistleblowers, that's sort of the other part. And we're not seeing basically any of these AI companies, apart from OpenAI have published that policy, after the drama from last year, which maybe you remember the Apple story from previously is a pattern. That's only after scandal, something is published. None of these companies publish any of this. And that is not good, because we talked about it before, like well over seventy percent

• 1:35:45Speaker 01

  of kind of whistleblowers went to SSC, start internal. These systems have to work well. And we cannot just rely on trust, especially given sort of precedent, that they are going to work well. And that's what we're calling for companies to, at an absolute minimum, publish their whistleblowing policies. And ideally, also kind of catch up to global standard, and also publish evidence around how well their systems work or how they are performing and what measures they also take to improve those systems, that's important or not. This actually is a, we think this is like a bit of a litmus test. Because there's essentially no cost to companies for publishing these, because we're not asking for additional of reporting to be created, we're just asking for transparency on what already exists.

• 1:36:29Speaker 01

  And any company that takes as serious as a business process would, for example, of course, have a whistleblowing policy and would have, would measure all of those things already anyways. Because if they care, they are measuring these things. If they're not measuring these things, then we have an answer, to an extent at least. Either that they don't really care about it at all. And they maybe haven't invested time to think about what they should be caring about. Or they have thought about it and they're actively not doing it. And both of these do not seem like great options. Yes, it is also best practice globally, a bunch of companies already do this. Benefits exist obviously for insiders, because the public can then look at these policies and explain why certain policies fall short, or what looks good and what doesn't look good, which they can't do today. There again, false confidence may sort of exist.

• 1:37:14Speaker 01

  Good for the public, because we know, and good for companies, we talked about like the benefits of speak up culture. For like, feedback improvement of these policies. We should see, uh, these, all of these impacts we mentioned before, and in fact, there was an asset manager called Trillium Asset Management, who on 2022, for example, called on Google to improve their whistleblowing systems, exactly for all of those reasons, and basically saying, strong whistleblowers serve shareholders, and there, there is an interest from shareholders to have strong whistleblowing systems, because we want to make sure there is no misconduct. And it basically only doesn't serve maybe direct managers, potentially executives, depending sort of on how you look at it.

• 1:37:54Speaker 01

  So, it's a very reasonable ask that we're putting forward, which would hopefully still be quite impactful. Although of course, again, just creating transparency is not everything. You can have a great looking policy, and it still doesn't perform. You can create transparency around the evidence, and the evidence looks bad, or maybe it's not trustworthy, right? So, this is like a minimal thing, and...

• 1:38:16Speaker 01

  This is kind of the first step, I think, that we think that organizations should be taking these AI companies. And we're also happy to work with them, by the way, on these topics. We've got an incredible coalition that we put together for this. So it's well over thirty organizations. We're very proud of it, actually, because the first coalition of its kind with the best whistleblower support organizations in the world, the Signals Networks, Government Accountability Project, I mentioned already, Whistleblower Aid, Whisper, in the UK, we have Whistleblowers UK. Sort of basically the who's who of the whistleblowing world, academics as well, people who wrote, like, the individual who co-wrote the ISO standard on internal whistleblowing systems onboard, Transparency International, who wrote the best practice guide on internal whistleblowing systems, and Creating Transparency Around Evidence also onboard.

• 1:39:01Speaker 01

  On the AI side, we have Stuart Russell is joining the call, Larry Lessig was also, both of them were simultaneous of the right to warn, Daniel Cocotayo is onboard. Of course, you are onboard, Nathan. Thank you very much for joining the call. And the Future of Life Institute, Karma, many, many more, I'm not going to list them, list them all off off the top of my head. Take a look at the websites, publishyourpolicies.org. And if you're an insider at an AI company and you're thinking this sounds like a sensible thing and you would like to have this transparency, reach out internally, ask your management, maybe you have an anonymous town hall, maybe you trust your direct managers enough to raise it and say, hey, why are we not doing this? This is standard practice.

• 1:39:46Speaker 01

  This could be helpful, why not? This is going to benefit your manager as well. And it's going to benefit their manager's manager probably as well. And by the way, actually, this is another thing from our survey, which I didn't mention. A hundred percent of insiders support publication of policies that we surveyed. So there seems to be pretty broad support for this.

• 1:40:08Speaker 01

  If you're an outsider, you're not working in an air company, spread the word, make sure the call is heard. And yeah, maybe as much sort of on that campaign.

• 1:40:16Nathan Labenz?

  Great. I guess it's too early to have any responses from any official channels from companies, right?

• 1:40:23Speaker 01

  That's right. We just launched like last, last, so it's just been a bit over a week. We know that they're aware of this call. They have been aware of this call for a while because, for example, the Future of Life Institute and the Air Safety Index also called for publication of policies, although they recommended it rather than actively calling for it. And the question underlying that study came out, I think, six weeks it was shared with the companies, including the question, why you're not publishing your policies. So they are definitely aware of the question, at least. We had also given heads up to these companies. We know they're aware of the call. And so we're looking forward to working with them, seeing what the responses are going to be. And if there's no responses, then we also have a response.

• 1:41:09Nathan Labenz?

  Yeah.

• 1:41:10Speaker 01

  Cool.

• 1:41:12Speaker 01

  We also want to talk, we need to talk a little bit about sort of.

• 1:41:14Nathan Labenz?

  Yeah, I was just gonna invite you to do that again. So.

• 1:41:16Speaker 01

  Absolutely. Thank you very much.

• 1:41:19Nathan Labenz?

  The floor is yours. What, you know, what should people expect? What can they count on?

• 1:41:25Speaker 01

  Yeah. So basically where we see ourselves and like in the direct support side at least is a like a connecting point between sort of the EIA and the whistleblower ecosystem and like a first point of contact. That's why the third opinion offering I mentioned before allows you to reach out with a question around your concern without sharing any confidential information, fully anonymously. We then workshop together where like an open source anonymous tool you can access via Tor browser, workshop the question together, identify together the relevant independent experts. So you don't have to rely on us knowing the experts in your field better than you do as an insider. We identify relevant experts together, approach them with your question, bring back their answers to you. Hopefully at this point your

• 1:42:10Speaker 01

  concerns are alleviated. If they are not, we will help connect you to pro bono legal counsel that will be extremely experienced in helping whistleblowers along their journey with no pressure for any disclosure, regardless of where you are in your journey, the support is available. You can also reach out directly to us without going through the third opinion process and ask for help on who may be the best fit for you. We will also help you there. And supplement sort of a set like from the expert network, independent expertise, covered under legal privilege with those great organizations as required. We also have on our website, you can find apart from these organizations listed, an explanation of the process, a digital privacy guide.

• 1:42:53Speaker 01

  If you're concerned about digital privacy, which we do actually also see quite a lot, which makes a lot of sense to make sure you stay safe. There is also a bunch of resources there that you can find. We have previously also supplied hardened devices to at-risk individuals with like specific operating system setups that are like highly secure.

• 1:43:15Speaker 01

  That is also something we offer on the direct support side. And then on a wider scale, what we as A, we do, we mentioned like systematically breaking down barriers for AI insiders. There is the advocacy side, something like this. There is the research side, the survey we mentioned, which is still ongoing. You can find the link to the survey in the show notes if you work in front of the AI company. And there's an upcoming legal study that we're actually currently fundraising for to kind of really dive deep into what is the status quo of whistleblower protections across like a pretty wide range of AI risk scenarios, which like identifying kind of the most interesting ones is part of that research study. That's that is upcoming pending funding.

• 1:43:59Speaker 01

  And then there is advocacy, like we're published complaints manager, sorry, and policy works are providing feedback on policy. Both sort of in the US, there's other great organizations working on this. If you're interested, reach out, we can connect you. And on the EU side with the AI office, making sure they establish a whistleblower mailbox. In fact, the authors, the vice chairs of the court of practice recently called for exactly that as well. Which is amazing that they did that. So that's sort of what we focus on at the moment.

• 1:44:28Nathan Labenz?

  Cool. Well, thank you. This has been great. I think between the coalition of organizations that you've been able to put together and the evident seriousness with which you're taking every aspect of this, the thoughtfulness of the support structures that you've designed up to and including the provision of hardened devices, all of that is, in my mind, not too much for people that are concerned with just how crazy things might get to invest in now in anticipation of the possibility that there might just be a couple dozen individuals who happen to be placed at the right intersection of information and access to what's going

• 1:45:12Nathan Labenz?

  on that have the awareness and consciousness to want to do something about it or to at least seriously question it before moving ahead. And I think those people are going to be scarce and precious resources for society and also under a lot of stress and pressure individually as they're facing those things. So I think it is excellent that you and Coalition of the Willing are setting things up now to support those people. And I've been glad to be a real small part of it, but hopefully this helps raise the awareness

• 1:45:53Nathan Labenz?

  further and establishes you guys as a resource that people hopefully will never need. But it seems likely that there's going to be some cases where people are going to need to reach out and get this kind of support. So I for one would have enjoyed or would have appreciated having it to two and a half years ago already, but certainly as the stakes only continue to rise, I'm very glad that people in the future will have this option to avail themselves of this kind of very thoughtfully designed and soberly provided support. So that's great. Keep up the good work. Again, we're all counting on you. But for now, Karl Koch, Managing Director of the AI Whistleblower Initiative, thank you for being part of the cognitive revolution. Thank you very much for having me.

• 1:46:39Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot ing.

• 1:47:24Nathan Labenz?

  And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts, so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.