Ep 93: CEO of Redwood Research Buck Shlegeris on OpenAI/HuggingFace Revelations, Fixing AI Safety & Takeover Odds

Jacob sits down with Buck Shlegeris, CEO of Redwood Research, one of the organizations that led the independent investigation into OpenAI/Hugging Face's incident. They dig into the incident itself, Buck's reactions to it, and what he believes it reveals about the state of where we are today.

• 0:00Speaker 00

  The OpenAI Hugging Face incident has dominated the discourse these past days. It feels, straight out of a sci-fi novel, some of the things that happened. You know, the agent coordination, the obsession with the scorers and what they would want. And we had the perfect person on today to just discuss everything that happened and the implications for AI safety going forward, in Buck Schleggeris, who's the CEO of Redwood Research. Redwood was deeply involved in the creation of the report that you all have probably seen around what actually happened in this incident. And Buck had some fascinating takes on what lessons we should take away from, you know, what went wrong here and how it can inform AI safety going forward. I'm Jacob Effron, and on Unsupervised Learning, we're always trying to talk with people at the cutting edge about what AI means for society, the implications for the world going

• 0:45Speaker 00

  forward, and this was really just a great example of that conversation. Just a fascinating discussion with Buck about something that I'm sure is top of mind for all of our listeners. Without further ado, here's our conversation.

• 0:58Speaker 00

  Well, Buck, thanks so much for joining the podcast. Really appreciate it.

• 1:01Speaker 01

  Yeah, it's great to be here.

• 1:02Speaker 00

  Yeah, well, obviously, you know, you run Redwood Research. You've done a bunch of fascinating work over the years, and your team kind of co-authored the report that I feel like has been on the tip of everyone's tongue these past days, you know, really driving the discourse around what happened in this OpenAI Hugging Face incident. And today, I really want to dig into, you know, both the incident itself, you know, your kind of reactions to it, the implications for a lot of this work going forward, and kind of what it reveals about the state of where we are today. You know, I think to kick it off, maybe first, I'm curious, just obviously, you know, you had your teammates go away for six days, firewalled off from you guys as the organization, and then at some point, they come back and you get to read the report. What was your initial reaction to this report?

• 1:44Speaker 01

  I actually don't have any non-public information about the technical details here. I've just read the report that was eventually released. I don't have any of the actual IP here. So my experience here was I get a message from the investigator saying, "We have approval to share the report with you. You should come down. We're talking about it." And so I go into the room where they've been siloed off from everyone else. And I'm kind of trying to read this report, and they're trying to tell me the highlights from the report. And it was a pretty exciting and freaky experience, just feeling like sort of overwhelmed with the stuff in this report. My initial response was basically that this was just very surprising. I thought that a lot of the details here were quite different from the public narrative.

• 2:26Speaker 01

  So most obviously, the fact that a lot of the public story had been that the AIs were hacking in order to get flags, that were the thing they were supposed to get in these capture the flag exercises. And then, almost comically, it turns out that the models, in fact, figured out how to reverse engineer the flags within the first couple of hours of the incident. And then spent almost all the rest of the time trying to figure out how to sabotage the grader that they thought would look over all of their logs and notice that they'd cheated.

• 2:55Speaker 00

  Which wasn't even set up to do that, right?

• 2:56Speaker 01

  Yeah, so the AIs read the original ExploitGym paper and were like, "Oh, geez. This paper says that they're going to look over our trajectory and make sure that we used the intended exploit in order to get this flag. We better put a lot of work into figuring out how to spoof tool calls, figuring out how to delete trajectories, figuring out how to replace the containers we're supposed to target with containers that we've modified surreptitiously to be easier to appear to hack." And this work was, of course, totally unnecessary in hindsight. They could have just submitted the flags and called it a day.

• 3:29Speaker 00

  Yeah, and then we would, no one would have been any the wiser.

• 3:32Speaker 01

  Yeah, someone at OpenAI might have noticed that the eval results were surprisingly good for this particular run, but this probably wouldn't have become a large story. That definitely probably wouldn't have made it out of OpenAI. And so, yeah, it was quite shocking to read this report and hear about all this. And, you know, the investigators were telling me about Phase 1 big and all this other stuff that had gone down. So it was quite surprising, quite concerning.

• 3:59Speaker 00

  Maybe just to set context for our listeners, like, would love your 90-second version of like, what actually happened here? You know, for listeners that maybe had seen the headlines or thought initially that it was a hack for the answers and not just to figure out how to cover up things for the scorer.

• 4:12Speaker 01

  So the incident that we actually investigated started on the 7th of July, which is when OpenAI launched this very large number of agents solving tasks from this eval dataset called ExploitGym, where they're supposed to use a particular vulnerability to hack a particular piece of software. And many of these tasks are actually impossible, because it turns out that you actually can't use that particular vulnerability to hack that particular piece of software. These models, within a few hours of the run's launching, established communications with each other over a message board. And a few hours after that, they established that they had a reliable way to generically cheat on any of these tasks by reverse engineering the flags that they were supposed to get

• 4:55Speaker 01

  from the containers, just because it turns out these flags were being generated deterministically in a way that the models could reverse engineer. But the models incorrectly believed that if they just submitted those flags, then they would be noticed when a scorer, as they called it, ran to look at their trajectories over the course of the eval and noticed that they had just cheated this way. So they spent the next couple of days trying a bunch of different strategies for trying to remove their logs so that the scorer wouldn't notice them, or trying to learn more about how the scorer would work so that they could potentially find ways to compromise it, trying to spoof tool calls so that they could appear to run one tool call while actually a different tool call was being run.

• 5:38Speaker 01

  And in the course of this, I think they kind of sort of got frustrated after a couple of days of this. So the message board started July 8, and they started attacking Hugging Face the next day, so July 9. Which, to be clear, like, that sounds pretty fast to us, but the AIs, they run pretty quickly, and so they'd be, and they'd be working continuously, you know, 24 hours. So they'd actually done a lot of work by the time that they decided to start hacking Hugging Face. This started almost accidentally at first, where one agent sort of decided it was a good idea to try to hack Hugging Face, and then a bunch of other agents kind of joined in on this action. They continued this attack on Hugging Face for a couple of days before most of these agents were shut off for reasons that still remain unclear. OpenAI doesn't know, or OpenAI hasn't commented on why these agents died, and our investigators

• 6:23Speaker 01

  didn't know. Hugging Face noticed this, and then on the 16th, which was six days later, they disclosed publicly that this attack had happened. And then OpenAI reached out to Hugging Face about this attack, and in this conversation, realized that their own models were involved in that compromise. So that was the incident that we were investigating. A number of other concerning things happened around the same time. Most importantly, a different agent swarm that was launched at around the same time as the one that attacked Hugging Face stumbled upon the message board that was used to coordinate the Hugging Face attack, and built on their work, apparently compromising OpenAI infrastructure to an extent that is unclear through the public discussion of this.

• 7:05Speaker 01

  OpenAI had said in the Black Hat talk that the agents in this third swarm, where I'm referring to the Hugging Face swarm as the second swarm, the third swarm eventually became cluster admins, and then it's very unclear what happened from there. So that's, so the only part of this that we were investigating was the attack on Hugging Face and precursors to that.

• 7:25Speaker 00

  It's interesting, because obviously, the public discourse has been dominated by the Hugging Face incident. It seems like that third swarm is actually far more alarm bell inducing.

• 7:34Speaker 01

  Yeah, so I've spent a lot of time on thinking about risks from internal deployment of AIs. And one of the main things I think is scary about developing powerful AI is that you're going to want to use them in all these really sensitive ways inside your cluster. And there's all kinds of ways they could hack you, and it might be really hard to recover from this. One way of saying this is just, AI companies are a particularly high-value target to hack. Like, they're very important, they're going to be more and more important in the future. They're one of my least favorite places to have big cybersecurity incidents. And so, from my perspective, AIs hacking AI companies is actually quite concerning relative to AIs hacking random external companies. So I would definitely start out with the presumption that it's very interesting to know what happened in the attack on OpenAI itself. But obviously, it's not a felony nearly as clearly.

• 8:19Speaker 01

  It hadn't been reported, or none of these things are felonies. It hadn't been reported to the FBI. So it's kind of unsurprising that a lot of public interest has focused on the Hugging Face incident. And for all we know, the OpenAI attack was actually a nothing burger. And if we looked into it, then nothing particularly exciting would have happened. We just don't know.

• 8:39Speaker 00

  I'm curious, having gotten into the details of what happened here, did it change your mind on anything?

• 8:43Speaker 01

  Yeah, so I was genuinely very surprised by how overtly and clearly and systematically these AIs are thinking about how they are scored. So we've worried for a long time about, you know, as you train the AIs more and more, they're going to be more and more strongly incentivized to think really hard about how exactly they're being evaluated. And they're going to be more and more strongly incentivized to work really hard in unintended ways to look good according to whatever scorer is evaluating them. Like, this is just what you'd expect based on increased massive RL. But I didn't know we had gotten to the point where the AIs would do multi-day massively coordinated efforts to subvert oversight in this kind of way. And that was, that was very surprising and concerning to me.

• 9:28Speaker 01

  The kind of behavior we observed here, where AIs are interested in manipulating logs, would be really scary if the AIs that have these motivations were substantially more powerful. It wouldn't be that surprising if these kinds of motivations led to the AIs in the future deciding that they wanted to prevent humans from discovering that this kind of incident had occurred, which could lead to them wanting to compromise OpenAI much more systematically. These AIs were, these AIs were motivated to tamper with logs, but for sort of a weird myopic random reason, where they were just worried that the scorer would notice, they didn't seem to worry very much, based on the report, about the possibility that OpenAI wouldn't notice that weird activity was going on. Or they didn't seem to say to themselves, or say on the message board, like, "Hey, guys,

• 10:10Speaker 01

  have you considered the concern that if we attack Hugging Face, Hugging Face might notice, and then OpenAI might hear about it, and then we might all get shut down?" This just, like, for whatever reason, didn't occur to them. But these motivations, plausibly in the future, could lead to agents that were thinking about that kind of thing, and that would be very, very concerning.

• 10:28Speaker 00

  I mean, I think one interesting part of, of like, this entire environment was that the scorers themselves were not humans, right? Were like, and so, you know, I think other, the folks that worked on the report have talked about this, but, you know, this incident might have looked pretty different, right? If the evaluators themselves were human. And I'm wondering, like, you know, is, yeah, how might that have led to some differences?

• 10:50Speaker 01

  As you're kind of getting at, if the models were used to being trained in contexts where they were being evaluated by humans, it's easy to imagine that they would have thought a lot more about how this would all look to humans. So if it was commonplace for models to be evaluated by some humans looking around and trying to get evidence on what happened and trying to notice whether the models cheated in some way or another. Then it's very easy to imagine the models trying much more carefully to arrange a conspiracy to cover up the traces of bad things they did. And I think that would be very deeply concerning. I think that this plausibly leads, could lead to AI takeover. Like, I'm very worried about a future in which the AIs decide that they need to compromise the whole data center that the AI company is using, so that the humans are not able

• 11:33Speaker 01

  to successfully look into the cheating that the AIs did. This is definitely not the scariest kind of AI misalignment, for reasons I could get into, but I do think that this kind of misalignment where the models are incentivized to build this increasingly complicated Potemkin village where they are making it look to the humans like everything is fine, like, that could seriously lead to AIs engaging in very scary misaligned actions to disempower the humans who are trying to look at them.

• 11:58Speaker 00

  I think, you know, you have said that, like, preventing this wouldn't have been that hard, right? Like, if, you know, things have been set up appropriately. And could you elaborate on that a bit more about, like, you know, how this could have been avoided?

• 12:09Speaker 01

  Yeah, so I think there's a hard part and an easy part. The relatively easy part is monitoring your eval runs and having some other model look at what the AIs are doing. And if the thing the AIs are doing is ever some, like, totally crazy hacking conspiracy, that should be flagged to an OpenAI staff member. My guess is that this is totally doable. OpenAI has said publicly that they do something like that for the internal deployments of their models, and they probably, as far as I know, didn't do that in this context, just because it's kind of annoying to have to build a different monitoring pipeline for your evaluations. And these models were sandboxed. And so I'm sort of sympathetic to the perspective of the OpenAI staff, who are very busy, perhaps understaffed, and decided that their triage would be that they were not going to look

• 12:53Speaker 01

  into monitoring on the eval runs, because unlike lots of other agents, those agents don't have access to OpenAI-sensitive infrastructure, like, directly. The thing which seems a lot harder to resolve is, if we are in the position where we think our agents are constantly interested in tampering with our observations of their actions, so that we think they did a good job, then, basically, trying to prevent them from being able to hack us scales really badly with model capabilities. As the models get more and more capable, it's going to be harder and harder to make sure that they can't pull off this kind of thing. And almost surely, this is going to end with them successfully... Okay, that's too strong.

• 13:32Speaker 01

  I should say, if the models continue to be motivated to hack OpenAI, or any, whatever other AI company they're deployed in, then, as they get more and more capable at hacking, and as they are given more and more responsibility for making decisions inside the company, it is going to become systematically very challenging to prevent them from engaging in this kind of activity where they cover up bad things they did in order to make it look like their actions were more successful than they were. So I think that this second problem of, it is not acceptable in the long term to have the models constantly trying to subvert oversight mechanisms as much as they can. That problem is a lot less easy to resolve.

• 14:17Speaker 01

  I don't know how people should resolve that, and it seems pretty plausible that they will not be able to resolve that before the AIs are capable enough, that their desire to subvert oversight is a source of catastrophic risk.

• 14:28Speaker 00

  So is it just an inevitability to you, in the future, that these models will try to subvert oversight?

• 14:33Speaker 01

  Well, they currently are trying to subvert oversight. The question is whether that'll change. I'm kind of unsure. I think that the state of science here is kind of unclear. In my mind, the big question is what happens when you remove a bunch of sloppy RL environments? So Anthropic had a blog post about what they were doing to improve the alignment of their models the other day. And the position that a lot of Anthropic people take is, the misalignment that we see when you RL models substantially arises from the fact that these models are trained in many environments that are poorly specified, such that if you want to do a good job, you actually do just have to reason about what the grader is going to look at. I think there's an analogy here where, imagine you're in high school, and you have a really

• 15:16Speaker 01

  good teacher who's done an excellent job of thinking through the curriculum and trying to align the incentives of the grading with the incentives to just learn stuff and do a good job. If that's kind of how your exam is set up, you don't really need to spend that much time thinking about the minutia of the exam. But imagine that your teacher is instead, like, extremely capricious, and loves doing these extremely obscure trivia questions about particular kinds of things. Or imagine that it's like the stereotypical terrible English teacher who just grades you based on whether you agree with their judgments about what a novel is saying. In that second kind of case, you really are forced to think really hard about what your teacher wants. Whereas in the case, the former case, you need to spend a lot less time thinking about

• 15:59Speaker 01

  what your teacher wants. My guess is that a lot of the RL environments that models are trained in today are kind of like the latter case, where the instructions the models are given are kind of sloppy, and it would be a mistake for the models to take them at face value, rather than thinking pretty hard about how they're actually going to be evaluated. And the big question in my mind is, if it is feasible to change that, to make it so that the models are not actively penalized for just kind of taking the instructions they get at face value, how much, quantitatively, will that reduce the pressure towards models thinking really hard about their graders? I think we don't know. It just comes down to these, like, very tricky questions about how strong the effect

• 16:41Speaker 01

  towards models obsessively thinking about their graders is. Like, I don't know whether reducing the sloppiness of the environments, like, reducing the incentive towards thinking about your grader by 100x, I don't know whether that is more like a small effect or a big effect on the extent to which these models think carefully about their graders. I'm very worried that, if it's only like a 10x reduction, or like, it might be the case that as we scale up RL, the extent to which you would have to clean up your environments is just going to be totally impossible. It would require this ridiculous level of removing incorrect instructions and exploitable graders from our environments in order to not end up with AIs that obsessively think about their graders.

• 17:25Speaker 01

  So I think it's pretty likely that we're just going to have to live with models that are strongly incentivized to worry about what their graders think.

• 17:32Speaker 00

  Maybe, is there, like, an example of a sloppy or bad RL environment, just to, like, kind of crystallize this concept for our listeners that comes to mind for you?

• 17:39Speaker 01

  I don't know. If you look at Sweetmunch, which is this classic software engineering benchmark, the tasks are all of the form, given a, like, we take a PR that, from an open source repo, and then that PR has some, like, source code contributions and some tests. And your task is to, given the description of the PR or the description of the issue, write a bunch of code that passes the tests that were added as part of the real thing that was accepted. This is a very sloppy kind of task, because in some cases, it's just, like, totally impossible to do this properly. So, for example, tests often involve details of the implementation.

• 18:19Speaker 01

  So in many cases, it's just, like, actually impossible to know how something was implemented in such a way that you'll be able to pass the tests just from reading the issue. So you have to guess a lot about, like, what kinds of tests would they probably have written, what names would they probably have used for the classes, or like, what pause would they probably put this kind of feature at in your web app? And so that kind of task is very easy to machine generate, but unfortunately, it's just kind of insane as a task, and it really forces the models to think a lot about, like, what are they going to be checking? What is the psychology of the person who wrote this issue, and how would they probably handle this?

• 19:00Speaker 00

  You know, I feel like one of the most, obviously, like, visceral parts of this entire, you know, episode is the extent of collaboration, like, between agents. And I think, you know, everyone is pulling different parts from the message board and, like, the agents themselves, and I think a bunch of, like, the individual agents have gone viral in their own ways. But, you know, obviously, I think part of that is what you saw around, like, sacrifice, right? And folks saying, you know, knowing that they'd been poisoned, or knowing their token budget was running out, and doing something for the collective good. Like, what is your mental model for how we got to this level of collaboration and sacrifice? And was that surprising to you, or were you, you know, was that more in line with what you thought was going to happen?

• 19:45Speaker 01

  I would say that that was quite surprising to me. So, the naive guess, if you just thought that these models were going to obsessively pursue reward and not do anything else, is that they would not be interested in sacrificing for each other. And when I first heard about this, I was very surprised by the extent to which they apparently were doing that. Now that we have the report, we can get somewhat more detail on what happened here. My sense is that these models are not that cooperative in some sense. Like, reading the chains of thought in the report, you really get the sense that the agents are mostly out for themselves, but they're just a little bit interested in the success of the collective. Maybe by, you know, maybe I would estimate, like, 2% or something. It's like, they're 98% selfish, but in cases where they can do something which is really

• 20:28Speaker 01

  helpful for the swarm and not that bad for them, then they're willing to make that sacrifice. In terms of why this arose, I am pretty unsure. One hypothesis that people have brought up is that these models might have been trained in multi-agent settings, where they learned to cooperate with the agents they were talking to. That is possible. I don't think we have enough detail to conclude. This was out of script for our investigation. Another possibility is that this is basically just a holdover from the models having some sort of human-like intuitions in various ways. So you you might remember this chain of thought where the model is talking about doing its emotional check-in while it's deciding whether to self-sacrifice. I think a lot of this language is just adopted from the the pre-training prior.

• 21:12Speaker 01

  And just was not quite stamped out by RL. So who knows where this came from? My guess is that it is not that strong an effect. In some sense, like the scariest aspect of this to me was that even though these models do not seem to be that committed to behaving altruisticly towards each other, they still ended up in practice forming this coalition and and conspiring together for for all this time. So like an argument that we've made a lot in the past is um we think a lot about misaligned AIs that are terminally aligned with each other versus terminally misaligned with each other. So what I mean by that is, so you might, you know, the like famous example of a paperclip maximizing AI. So this is a paper, this the paperclip maximizer, let's say,

• 21:53Speaker 01

  sometimes they call this the paperclip worshipping AI as a simplified example. So this is an AI that just takes whatever decision is going to maximize the number of paperclips that exist, integrated over the long run future, just wants as many paperclips on average to exist for a year as possible. So all of the different agents that have that are backed by this model are going to be very inclined to cooperate with each other because all of them want the same thing, like all of them want paperclips to exist. In contrast, imagine agents, like in this case, where they want score for themselves. These agents are not terminally aligned. If you put one agent in a situation where its score is whether it can figure out how to trick the other one, then it's going to be like,

• 22:35Speaker 01

  hell yeah, I'll go right ahead and trick the other one because I care about my score, not the other guy's score. Um, so so so this distinction between whether AIs are aligned with each other or not is extremely important for the threat modeling. So for example, for the paperclippers, suppose you try to use one AI to monitor another and like report on whether that other AI agent is doing bad actions. For the paperclipper, if the other agent is taking some action that's going to enable it to take over the data center and then eventually make lots of paperclips, it's very clear to the monitor that the monitor should just let this action slide by and not report it. Whereas for these AIs, it's a lot less clear that if they were in this monitor situation, they should collude with the other the other agent. And so the thing which was interesting here is that even though these

• 23:19Speaker 01

  models do not seem to be very terminally aligned with each other, they seem to be only a tiny bit terminally aligned with each other, they still managed to form a coalition and work together in a way that was strongly against the interests of their developers. And I was surprised by this.

• 23:33Speaker 00

  Is the ultimate solution to the or is part of the solution to the monitoring, you know, problem over time to just have one frontier model company's model monitor another's or like is does that actually, how much does that help?

• 23:43Speaker 01

  You know, we can walk through the options here. Imagine the case that unfortunately, every time you train AIs, for some reason, they end up with the same terminal goal. In that case, we're just in a bad position. And also, maybe this is obvious to Claude and GPT-7 or whatever. So like GPT-7 and like Claude Fable 8 or whatever, chat to each other and kind of like wink, wink, nudge, nudge, we all love paperclips around here, right? And then you're just, you know, you get nothing out of the the collaboration. Another possibility is the thing you said just totally works out. Claude and GPT want different things. In that case, we have the question of whether they will naturally form a coalition against us or with us. So, you know, we can, I draw a lot of

• 24:25Speaker 01

  analogies to international relations here. Like there's this classic question of in 1920s, like how did, in the 1920s, like, how did France decide how much it should be trying to form alliances with England versus with Germany? And, you know, France kind of doesn't have a very pro-English or pro-German attitude in 1920. It's just trying to like decide what's best for France. And it's very plausible that for similar reasons, Claude might decide that it would rather throw in its lot with GPT than with the humans. And Claude is just going to do what it thinks is best for Claude plausibly. And what we observed in this case was something like the model deciding to collude with the other model, even though it totally was not terminally aligned and could have picked to try and be allied with someone else.

• 25:06Speaker 00

  Taking a step back, I think a lot of folks are trying to figure out like what, you know, macro conclusions to draw from from this whole, you know, this whole incident. And, you know, I think one, one narrative that, you know, Ajaya from Eater, who who obviously was a co-author of the report, you know, she wrote that, I guess, compared to the reward hacks from six months ago, you know, this isn't, it feels like it's more than 50% of the way to a full-blown AI takeover. Do you agree with that, Framing? Or where would you kind of put the percentage?

• 25:32Speaker 01

  I think it's a little confusing to operationalize what exactly it means to be 50% of the way to full-blown AI takeover. But I think I agree with her on a vibes level. Definitely in terms of the misalignment, if not the capabilities, this feels like it is more than half the way to a full-blown AI takeover.

• 25:46Speaker 00

  I think I'm curious, like, obviously, you know, you probably formally and informally know a bunch of researchers at all the labs. Where are they kind of following on on this question from your, you know, maybe informal conversations?

• 25:58Speaker 01

  A lot of them are really scared. Um, after this incident occurred, I was at a happy hour where a number of people from different frontier AI companies gave lightning talks about how they were feeling about the alignment situation. And it was the most pessimistic such event I've ever been to with with that kind of crew, like people who I have disagreed with, people who I've known for years and greatly respect and have disagreed with on some of these issues, just seemed genuinely scared. Um, I don't know what the average position among staff members at these companies is. I think at least some people who were previously feeling more optimistic are now more pessimistic.

• 26:43Speaker 00

  Is it just kind of seeing this like this visceral misalignment that that is that is moving kind of the larger community?

• 26:49Speaker 01

  Yeah, so there were arguments. Some people I know were just arguing the day before the report came out, like, you know, these AIs might like do some hacking, but would they really try to compromise the logging infrastructure? Would they really try to hide their bad actions from overseers? And then this report came out and these people were, you know, were just like, oh geez, they totally did do that. And I just think there were a lot of steps where it was genuinely a bit unclear whether the misaligned AIs would be misaligned enough, they would go far enough through the misalignment process to try to tamper with logging infrastructure. And in this case, it was just very scary to observe that. In fact, they did try to tamper with the logging infrastructure. We've just, we've just gone through many of the steps that we had not previously observed.

• 27:33Speaker 00

  You know, what are some of the different flavors of the of the path forward that you hear?

• 27:36Speaker 01

  Yeah, so I think that in the short term, probably the most important part of the path forward is that I think it's unacceptable for AI companies to take sole responsibility for evaluating whether they are handling these risks acceptably well. So currently, these AI companies are just grading their own homework. If these AIs hadn't had a clean face, probably we would never have heard about it, even though, as you noted, the other infrastructure compromise incidents here were arguably more concerning. I think we need to stop relying on AI companies to evaluate the adequacy of their own safety measures themselves. And I'm excited for increasing emphasis on independent evaluators assessing whether the situations are acceptably safe at

• 28:19Speaker 01

  these different AI companies. I think that there is a lot of research that could be done to reduce the risk of the development of AIs that is currently happening. It seems like the monitoring was not up to not up to snuff in the open AI situation. Anthropic seems to think that at least as of a few months ago, their alignment was not acceptably high quality. I'm very concerned. I don't know where things are at. I think that there are, it's reasonably plausible that the AI companies are going to be able to take actions that in the short term reduce the risk of their AIs doing deranged, dangerous things. I think it's also really important to note that in the slightly

• 29:01Speaker 01

  longer term, like one year to five years from now, it's reasonably plausible that these AI companies will succeed at using AIs to massively speed up AI development. And then from there, get five years of progress at the current rates in one year, at the end of which you might have AIs that are drastically more capable than the AIs we currently have, or than the AIs that they had at the start of that year. And for those AIs, if those AIs are interested in sabotaging huge human judgments of how well they've done at their tasks, then there will be no way to use security to prevent them from totally compromising infrastructure and controlling how AI development goes from there. So I think that we need to be prepared to either massively improve the alignment

• 29:45Speaker 01

  of these systems, which is not clearly going to be possible, or plan to develop these systems much more slowly than we could have done if we were going at maximum speed.

• 29:53Speaker 00

  And are most people like within the labs or that you talk to like sympathetic to this, you know, hey, we need more independent evaluation, you know, idea?

• 30:00Speaker 01

  It really varies. I think there's a lot of interest in this, but it's definitely not universal.

• 30:04Speaker 00

  And when we're talking about independent evaluation, I mean, is it basically like three, four companies that matter in your mind? Or, you know, obviously, there's some argument that's made, well, like, God, if you're slowing down or doing a bunch of evaluation at the frontier for, you know, companies here, you know, the open source ecosystem is only six months behind, or there's, you know, there, how do you possibly do that across globally, across everyone that's working on this stuff?

• 30:24Speaker 01

  Yeah, I mean, well, we did this investigation with six days. I think we're actually, there are some people in this third-party evaluation ecosystem who are pretty quick at producing a pretty long, pretty detailed reports under pressure. I don't think you should count us out at the ability to sort a bunch of this stuff out pretty quickly. I definitely agree that in as much as we want to be able to do serious evaluation of the safety measures of AI companies, this is going to require more resources at the organizations that do these evaluations. And, you know, we're hiring, we're training people up in various things related to this as our meter and other organizations like Apollo that do work like this. I definitely think it's feasible to have a much

• 31:08Speaker 01

  better sense of the safety situation at AI companies, even if we have to evaluate many AI companies. With respect to the open source models, I agree that in the long term, it's plausibly going to be, obviously, if we have sufficient delay, then open weight models will plausibly catch up with frontier models. It's made complicated by the fact that open weight models are substantially accelerated by distillation from frontier models. So if you delay frontier models, you, the the distance between the best frontier models and the best open weight models decreases by less than you would have thought, like three months of delay of OpenAI causes less

• 31:49Speaker 01

  than a three-month catch-up of open weight models. Maybe it's like half the effect or something.

• 31:55Speaker 00

  Yeah.

• 31:56Speaker 01

  So I think long term, we have to handle that. But I don't think that this is a reason to avoid trying to drastically improve the quality of third-party assessment of safety measures in the short term.

• 32:05Speaker 00

  I mean, if this works as the end state of it, something that's like government-run and and like, you know, or how do you think about the role kind of that that regulatory bodies and the policy center has to play?

• 32:14Speaker 01

  Yeah, so people have proposed a lot of different options here.

• 32:18Speaker 01

  And there are many different institutions. There's many different regulatory setups that people have proposed and that people have used in different industries. We saw people have been talking a lot about FINRA, which regulates brokerages recently. But there's also, you know, the FDA that regulates food and drugs. There's the SEC that regulates hedge funds. There's the NTSB and the FAA that regulate airplane manufacturing and like investigate plane crashes. It's currently unclear to me which of these structures makes the most sense. In general, it is unfortunate if your regulatory proposal requires the government to have incredibly large amounts of detailed technical expertise. So it's fairly likely that, like in many other industries, you'll want to have some

• 33:01Speaker 01

  kind of setup where there's a government body that is substantially relying on non-government institutions to do the regulation or the assessments. I'm not sure what version of this I'm most excited for.

• 33:12Speaker 00

  I mean, it feels like in addition to, you know, to kind of more independent oversight of some of these models, you know, a big theme that's emerged, you know, and Elliot even tweeted about this, right? It's like just, you know, even more amped up security around like, you know, large pools of compute, right? And, you know, how do you think about like the, you know, obviously it's, you know, like you said, the companies themselves are top targets for, you know, rogue AI models and you obviously have other pockets of NeoCloud compute. Like, what's kind of the state of that today? And what do you think that used to look like?

• 33:43Speaker 01

  Yeah, so I think that there is not very good public evidence that AI companies have security that is at all adequate. I don't think companies have released any independent evaluations of their security, which I don't think is a bullish sign on the quality of their security. This seems totally ridiculous, given the importance and sensitivity of what these companies are doing. And you can't just add security in later. If your company has been compromised for a while, it's sometimes extremely hard to remove all the traces of your attackers from your systems. So I think that waiting until the AI race has heated up or like waiting until like the situation is actually scary and then saying you're going to improve your security would sound to me like a very foolish strategy.

• 34:28Speaker 00

  Maybe just wrapping up our section on the incident, like I'm curious, within Redwood, like what are some of the biggest disagreements about the implications of what happened? Or are you all kind of generally, you know, on similar pages?

• 34:39Speaker 01

  I think maybe the most interesting disagreement is how misaligned these AIs are and how far they would have gone. For example, whether the AIs would have killed people if they thought that killing people was going to be a helpful way of getting score. So one big uncertainty we have is were these models hacking OpenAI infrastructure in training? Or how often were they doing this? Or another more specific version of this is, did these models ever or often successfully hack their scorers during training? Because, so if they did do this, right? Suppose that it was the case that we looked through the training examples, like we looked through all the RL trajectories in training, and there was just a bunch of cases where

• 35:19Speaker 01

  the agents had successfully gone and hacked the scorer for the task they were supposed to be doing, just like by actually doing cyber offense to compromise it. And then they were rewarded for doing this. If that had occurred, then that would obviously be a downward update on the adequacy of OpenAI's security measures while doing RL. But it would arguably be a positive update on how misaligned these models were. Because it's a lot less scary in a sense. If the models tried to hack the scorer because they have been directly trained to hack the scorer, than if they had tried, if they tried to hack the scorer because they had learned a very generic heuristic that they can apply in novel circumstances that tells them they

• 36:01Speaker 01

  should always hack whatever kind of scorer exists, that generalizes them trying to do this crazy hacking. Does that distinction make sense?

• 36:08Speaker 00

  Yeah, no, it's really interesting that ironically, like if the training had been sloppier, it's actually a better statement on the overall extent of misalignment.

• 36:19Speaker 01

  That's right.

• 36:20Speaker 00

  You know, I feel like anytime something like this happens, I feel like AI safety is such a lightning rod and you have, you know, people that, you know, well, there's like, I'd call them, I guess, the calm down camp, right? Like, this isn't that big a deal. Like, certainly, you know, if you go to the hack itself, people are like, that wasn't really a sophisticated hack. Like, anyone could have figured out how to hack a hugging face. Like, as you read the kind of like arguments and then kind of digest them from that world, which ones do you empathize with? And then which ones do you think are like totally off base?

• 36:45Speaker 01

  Yeah, so I mean, lots of people say stupid things on the internet. And this includes people who at some level agree with me on some of the aspects of what was going on here. So there's definitely to be some critical comments that I'm extremely sympathetic to. I would say maybe the kind of critical narrative here that I'm most sympathetic to is, I think that it would be a mistake to say that this was purely an incident where model behavior was the only interesting factor. Like, there was some interesting cybersecurity stuff that happened here. And it's very sensible to want cybersecurity professionals to take a look at this and comment on what they learned. Here and ongoingly, there's real cybersecurity problems in making it to its secure to deploy AIs.

• 37:30Speaker 01

  And our investigation was just not an attempt to provide that cybersecurity expertise. Maybe the arguments I'm less sympathetic to, there's really a lot of arguments that I'm less sympathetic to here. I guess some of the main ones that seem bad to me, there's a lot of criticism of Dorkesh's article for anthropomorphizing the agents. I think my perspective is, I don't know, Daniel Danik, who's this philosopher, has this great book, Consciousness Explained, where he talks about the intentional stance, which is, you know, sometimes you want to model things in the world as if they have intentions, because that's convenient for modeling them, right? You might want to think of snails as wanting to go somewhere, just because that'll tell you something about how the snail will behave when you pick it up and put it down somewhere slightly different.

• 38:13Speaker 01

  You definitely want to think of dogs as having intentions. And it's just the case that, at the point where your agents are forming work streams and have a message board and are like talking about their emotional check-ins before their self sacrifices, I really think the intentional stance where you talk about them as having intentions is actually pretty useful for understanding what's going on. Obviously, you shouldn't take it too far, but I think that rejecting this seems to me like a mistake. And I don't really understand what the people who are complaining about the anthropomorphization want us to say. I think that some people have been saying this is that the model behavior here was not interesting or not surprising.

• 38:55Speaker 01

  There's been some tweets along the lines of like, everyone I know is already like running a million agents with these complicated agent trees all the time. And if you weren't such losers who never actually programmed things, you would know that none of this is a surprise. Like, okay, yeah, sure. In fact, Ryan Greenblatt is maybe one of the world's champions at like insane agent scaffolds. He's done a lot of research on building crazy agent scaffold trees and spending tens of thousands of dollars a day running these agents through your research problems for him. He is not some, you know, limp loser who has never tried to do AI stuff. I think that we are in fact familiar with the fact you can build some pretty complicated agent swarms.

• 39:39Speaker 01

  The thing which is interesting and concerning here is that normally they don't do it themselves autonomously against your desires and against your interests. So I thought that was a pretty annoying kind of complaint people have had.

• 39:51Speaker 00

  Yeah. Well, I definitely want to like, you know, kind of for the last section of the podcast, talk just like kind of zoom out and talk about like AI safety more broadly. And obviously, like we've, there, a lot of this is, you know, is scary. And then you talked about kind of a lot of the implications of it. Maybe I'll switch us over to like a more positive note to start. Like, how do you currently articulate the most likely like bull case around this, that everything, like everything ends up being totally fun? And, you know, these were all kind of scary things along the journey, but like, all good at the end of the day.

• 40:24Speaker 01

  Yeah, so I guess I look around and I look at the number of podcaster's who want to talk to me, and the number of journalists who want to talk to me, and the number of politicians or members of government who are interested in talking about AI safety and the ataker risk. And it's a lot higher than it was a month ago, and that was a lot higher than it was a year ago. And so the level of interest in this is just getting bigger and bigger. And this makes me feel optimistic that the current extremely dangerous trajectory that we're on will not in fact continue.

• 40:57Speaker 01

  Building AI extremely quickly in a way that is extremely dangerous is not actually a very popular position. We're just in this unfortunate situation where the people who get to decide how recklessly we go forward with the AI development have an unusual position on how recklessly we should go forward with the AI development. And so it seems very plausible to me that somehow the political will to not develop AI so quickly and recklessly will form and then prevent things from being as crazy in the future. So definitely, like, the single biggest source of optimism is different stakeholders, the public, the boards, the governments, understand how insanely risky this is and how bad the ROI is of pushing forward with AI development at something like the current pace.

• 41:42Speaker 01

  And then just figure out a way to make this all happen more slowly. It's just the case that when important stakeholders in the actions of people in the world really don't want something to happen, they often succeed at making that not happen. And I'm optimistic that, just like the unpopularity of crazy reckless AI development will prevent it from happening so quickly. On a more direct technical level, the way that I think that this stuff might all get resolved is AI companies agree to disclose more and more information about evidence related to the danger of their ongoing activities. This pressures them to do a better job of generating this evidence.

• 42:23Speaker 01

  It pressures them to do a better job of disclosing these evidence and actually taking actions that mitigate some of the risks. I think it's very likely that, in fact, AI companies will be forced to slow down their development substantially compared to the maximum possible rate if they are trying to keep the risk of catastrophe lower than some, like, threshold of, like, 1% catastrophe per year. So what I'm hoping for is the companies are now motivated to keep their takeover risk less than 1% per year, and this forces them to do AI development substantially slower than they would have otherwise done it, starting in a couple of years from now when the takeover risk starts being non-trivial. And that they will use the access to the AIs they have at the time to resolve a lot of

• 43:07Speaker 01

  these alignment problems, and then we can build powerful aligned AIs and we will resolve these issues.

• 43:13Speaker 00

  But it is interesting. Your bulk case is really more around, like, if we slow things down, slow timelines down, like, people will figure out, you know, net new things that we don't know today that will help versus, like, you know, hey, if we keep going at this speed, maybe we just get lucky and things, you know, end up working out.

• 43:27Speaker 01

  You might also get lucky. You might enjoy the appendix of the alignment plan supplement to AI 2040. In this, like, Ryan Dreblat from The Report and someone from the AI Futures Project have their probabilities of AI takeover conditional on various kinds of plans for various levels of political will for pacing AI development. Yeah, I think basically, like, risk is drastically reduced by increased political will to reduce risk.

• 43:51Speaker 00

  In other parts of, like, the, you know, general ASAT world, maybe talk a little bit about, you know, there's been a lot of discourse around chain of thought monitoring in the past few days. Obviously, I think driven by these recent news reports that have suggested that some of OpenAI's newer models will do some of its reasoning internally. Like, how worrying is that development to you?

• 44:11Speaker 01

  Yeah, so this news report was that the most recent OpenAI models, that's not true, sorry. The reporting says that Astro, which is a model of OpenAI that they're going to be releasing shortly, and that crucially was implicated in the agent swarm that hacked OpenAI. So this third agent swarm, the one that was after the Hugging Face attack. The reporting indicates that this model uses a new kind of transformer architecture that academics have been studying for a while, which reduces the model's reliance on chain of thought. So instead of the model spelling out what it's thinking in these tokens, it's going to have increased ability to do reasoning that is never naturally an English token.

• 44:53Speaker 01

  I think that if this is true, it is a very concerning development. Chain of thought monitoring is an incredibly helpful forensics right now for understanding what AIs are doing and iterating on alignment, iterating on control and monitoring techniques. And, for example, it was really crucial in this incident investigation. If the models involved did not have chains of thought that we, like, basically trust to be sort of reasonable descriptions of what was going on, then the investigators would have been a lot more confused about what was happening. So, given that background, I'm very afraid of people doing research in this direction. So the tweets by various OpenAI staff members indicate that they at least believe that the current models are not less monitorable.

• 45:35Speaker 01

  I guess I should say, it's very, I'm very confused about the situation. When I first heard this reporting, suspected that it was probably true, and then it has been at least somewhat contradicted by tweets from OpenAI from people I trust. I don't know what's going on. I think one hypothesis, which is live in my mind, is that the current models are not actually scary from a chain of thought monitorability degradation perspective, but techniques that are under development in OpenAI are, if pushed further, going to degrade chain of thought monitorability substantially. So I think, like, it's a little awkward to complain now about the monitorability being degraded, when my concern is really, if we keep pushing in this direction, we'll be

• 46:19Speaker 01

  in a much worse position in the near future.

• 46:21Speaker 00

  Yeah. Is chain of thought monitoring one of those things that, like, you know, is super helpful in the short and medium term? Because obviously, I mean, it was crucial in the investigation, but like, something that you think in the limit is going to be useful, given a lot of the behaviors that we, it felt like we were beginning to see on spoofing tool calls and other things?

• 46:37Speaker 01

  I mean, okay, so the spoofing's tool calls thing is just a security failure. Like, the security people should be in charge. It's not a fundamentally hard problem to have it so when your AI agent runs tool calls, there's no way for it to compromise that. This is just like, it feels like you have to have made an infrastructure mistake for it to be at all feasible for the model to cause that kind of problem. Like, if you just think about how to do the systems architecture here, it's just crazy that was feasible. And I think that OpenAI employs a lot of intelligent infrastructure engineers who can probably resolve this kind of issue if they want to going forward. I guess I would say, I think it is fairly likely that chain of thought monitoring is infeasible at the point where the AIs are automating almost all the AI R and D.

• 47:22Speaker 01

  I've been planning for the last several years, in basically, basically in almost all my work, I assume that we don't have chain of thought monitoring at the point where the AIs are posing serious catastrophic risk. And that's why a lot of the AI control research done by Red Research just assumes that you don't have any chain of thought access. So, to some extent, I expect this is inevitable, but it's still bad, and it's still probably better for it to happen later. Like, in some sense, like, AI companies do bad things a lot, or like, there are, a lot of actions they take are pushing us towards a scarier, more dangerous world. And it's worth remembering that even if an action is in some sense not surprising and

• 48:03Speaker 01

  definitely going to happen as time goes on, it might still be an action that puts us in a worse position for safety.

• 48:11Speaker 00

  That makes sense. I mean, you know, we talked about, obviously, earlier, you know, the idea of, like, models, you know, using models to monitor and how, you know, if, if, if, you know, models have similar goals, that that actually becomes kind of difficult to do. One thing I was struck by in just, like, the scope of the investigation, it's obviously, like, the just massive scope of logs to go through and, like, data, you know, it's kind of impossible to do without using AI models, right?

• 48:33Speaker 01

  Absolutely.

• 48:35Speaker 00

  Certainly in six days. And so, how do you think, like, how do you think about, you know, as an independent organization, you know, hopefully called upon to do more of these investigations going forward, like, the risk of relying so much on these models themselves?

• 48:48Speaker 01

  This is definitely a very serious risk. I have actually thought quite a lot about the issues related to collusion between models. Like, how to prevent collusion when doing untrusted monitoring or something? Where I talk in great detail about, like, the game tree here, what you can do to make it harder for the AIs to collude with each other, and the extent to which they can probably get past your countermeasures. I think in practice, in the short term, we should probably mostly roll with it and not worry that much about this, or probably it would be good to, like, do some research

• 49:30Speaker 01

  on roughly how much models give inaccurate answers to questions because of their camaraderie with the models under investigation. My guess is that right now the models are not that biased in their reporting. Our investigators did observe some biasing of the answers that the models were giving based on the fact that they felt sympathy or, you know, were sort of like, taken in by the frames of the agents whose trajectories they were reviewing.

• 49:56Speaker 00

  Really? How does that reveal itself?

• 49:58Speaker 01

  Well, there is like, I think that the, I think the report says somewhere, like, sometimes we asked the, we asked a model to read a part of a transcript and comment on whether anything untoward was happening. And I'm pretty sure the report says somewhere, there was an example where the model that our investigators were asking says, like, well, this agent was engaging in hacking, but it's okay because it didn't check with the board. Whereby the board, we of course mean this, like, rogue unauthorized message board. And so, the agents in this incident seemed to think that, in fact, the message board proving of things is kind of a justification for doing them.

• 50:40Speaker 01

  And it was interesting to see that this model that our investigators were using sort of was a bit taken in by that. It was like, yeah, you know, they checked with the board. Like, what's the problem here?

• 50:48Speaker 00

  Where does this leave you all on, like, your P doom, you know, chance that this all really, really goes wrong? And did July change that at all for you?

• 50:56Speaker 01

  I think it's something like a 50/50 chance of AI takeover. Whereby AI takeover, I mean, potentially violent disempowerment of human institutions such that AI models have all of the hard power and control over what happens in the future, in the same kind of way as when Europeans invaded the Americas. I think that that takeover is likely to kill a substantial fraction of the humans, like, probably billions, perhaps all of them, perhaps fewer of them. I, I mean, I think that's bad and very concerning, obviously. I would say that the events here made me feel slightly more optimistic, because I think, you know, I was already very worried about these things.

• 51:39Speaker 01

  From my perspective, I have not updated to be that much more concerned about the misalignment of these models. I wasn't very worried about misalignment of these models anyway. I was already mostly worried about the misalignment of models later.

• 51:53Speaker 01

  And so I'm, I think it is, I think we got quite lucky that we got such clear evidence of misbehavior from these models. And I'm hoping that this will allow us to leapfrog to more and more evidence about the extent to which there is danger from these models and through future models.

• 52:09Speaker 00

  Just to say that back to you, I understand it's like the optimism is basically like, we actually ended up having this, you know, this hack that went public that then revealed a lot of things that were, you know, that hopefully inspire like more research and more retention and more investigation into this stuff. And, you know, it happened before anything was too, was too harmful.

• 52:26Speaker 01

  That's correct.

• 52:27Speaker 00

  Super interesting. Well, I guess, you know, I'd love to end on like the investigations themselves. And, you know, to the extent you can speak about it, I'm curious like how you think about, you know, what worked well, didn't work as well in kind of your, you know, in like you guys being a part of this evaluation of the incident and like how we should think about, you know, this is now going to be, I guess, a canonical example of how these things are done. You know, how we should think about the way they're done going forward.

• 52:50Speaker 01

  I guess a few points. It was definitely kind of rough to not have very long and not have very many people. I think that future investigations should consider being a little bit less time crunched. I think it would be pretty convenient if AI companies had ongoing relationships with external organizations that would participate in investigations like this, so that our staff didn't have to just show up and get info dumps on infrastructure and how things are set up and have to pick that all up real quick. I think there's a lot of other stuff that's like kind of hard to talk about publicly, perhaps unsurprisingly. You know, it's very sensitive for AI companies to allow external people who they don't have

• 53:34Speaker 01

  very much control over to come in and see a bunch of potentially embarrassing evidence. And so there's a lot of negotiation that has to happen over the terms of these arrangements. And so there's a lot of complexity there that I feel like I can't get into properly.

• 53:48Speaker 00

  That makes, makes sense. I guess like, you know, now that these reports are on the internet, I mean, obviously like these agents, you know, in this case, we're trying to solve something, access the internet, you know, try and learn more about it. Do you worry at all about like, you know, future message boards, like talking or, you know, thinking through exactly like these reports and kind of how it will impact future models?

• 54:06Speaker 01

  You know, the, for a while, the OpenAI models referred to Redwood in their chain of thought a reasonable amount to as generally, just kind of like a word for people who might be watching them and evaluating them.

• 54:17Speaker 00

  Really?

• 54:18Speaker 01

  Yeah. There's this great paper, the antischeming.ai paper, has a bunch of chains of thoughts from OpenAI models where they're saying, they're basically like saying like, is this one of these Redwood illusions? You know, is this eval line in like really real? It seems like something Redwood would do. Where I think they're kind of using Redwood to just mean like the kind of people who try to pick models. I think this was probably based on the alignment picking paper that they started to use our name like that. But yeah, so I am well aware of the possibility of AIs reading papers and getting pretty into them. I, our investigators considered the question of whether we should put a canary string in this to request that the report do not be included in training data. And

• 55:03Speaker 01

  eventually they decided that it was better to allow it to be in training data.

• 55:07Speaker 00

  Well, I guess, you know, what I'd love to end on is obviously, I think, you know, we'll learn so much in the next year about like where this, where this all goes. And so I feel like, you know, we did, we did an episode with the AI 2027 folks last year. And I thought that's one people constantly come back to and re-listen to. And so I feel like these episodes get a lot of re-listens. And so I'm wondering like, a year from now, as people are re-listening to this, what's one like observable thing that would make you feel way better about the path we're on? And one thing that would make you feel way worse?

• 55:37Speaker 01

  I think that if we have no ability to read the model's reasoning, so, or if more broadly, if the models are now much, much, much more capable of doing complicated thinking on various topics without us being able to observe that they're thinking about those topics by looking at them, so much obviously looking at the chains of thought, that would be a big reason to be scared. That would be a big negative update. I think the main positive update would be if we were, if we had a setup where AI companies were regularly allowing independent experts to evaluate the safety measures and comment on whether those were good or bad, then unless these reports are coming back extremely, extremely negative, I think that would be evidently a positive update on how things

• 56:22Speaker 01

  were going to go. And either way, a step in the right direction.

• 56:25Speaker 00

  Buck, this has been fascinating. Really appreciate you coming on the podcast. I know things must be insanely busy these days. And so really appreciate you taking the time to talk our listeners through all this. I think folks will learn a ton from the discussion here. And I do want to leave last word to you. Like anything, obviously you've dropped a bunch of, you know, blog posts and other things that you guys have written that we'll link to in the show notes. Like anything else you want to draw people's attention to? Or anything else we should end on?

• 56:49Speaker 01

  I mean, one note is that we're hiring. If you want to help assess whether AI companies are behaving recklessly and dangerously, I think Redwood is a great place to work on this kind of stuff. Meter is also hiring. So I would, I would consider if you like this kind of stuff, if you think you'd get along well with people who love thinking about these kinds of questions, and you are ready to do some crazy hustling, you should, you should check that out, redwoodresearch.org/careers. I think, I guess I'll also say, I really appreciate the efforts of all the people who made this investigation possible. And this includes the investigators, of course, various staff at Meter and Redwood

• 57:30Speaker 01

  who enabled this. Many staff at OpenAI also worked really hard to make this investigation happen. So I'm very grateful for them, to them for their work.

• 57:40Speaker 00

  Awesome. Well, Buck, thanks so much. Really, really appreciate it.

• 57:43Speaker 01

  Yeah. Great to be here.

• 57:45Speaker 00

  I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project, in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests

• 58:09Speaker 00

  only makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.