#494 — A Coin Toss for the Future

Sam Harris speaks with Ryan Greenblatt about AI misalignment and the risk of losing control of increasingly capable systems. They discuss the Hugging Face incident, the spectrum of concern about AI risk, why companies are racing ahead despite high odds of a, reward hacking, the distinction between alignment and control, the danger of AIs reasoning in "neuralese," alignment faking, how an AI takeover might unfold, and other topics. If the Making Sense podcast logo in your player is BLACK, you can SUBSCRIBE to gain access to all full-length episodes at samharris.org/subscribe http://samharris.org/subscribe.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:20Sam Harris?

  I am here with Ryan Greenblatt. Ryan, thanks for joining me.

• 0:23Speaker 01

  It's good to be here.

• 0:24Sam Harris?

  So you're the chief scientist at Redwood Research, which is one of these AI safety research firms that analyze the recent hugging face fiasco incident, terrifying phenomenon. We'll get into it. But how did you come to this work? What was your path to focusing on AI safety?

• 0:44Speaker 01

  Yeah, so in college during my junior year, I was sort of in my apartment alone because it was COVID. All classes were remote and I was sort of in a contemplative mood, thinking about what I should do with my life, listening to various podcasts. And in one of those podcasts, someone made an argument that was like, basically, I would summarize the argument as like, it doesn't make that much sense to be selfish because what's really the distinction at like a material level between your future self and other future people? And I was like, that argument kind of makes sense to me. I should really consider how I want to like, lead my life and what I should do. And I ended up getting pretty sold on being much more focused on altruism. And then after that, I spent a bunch of time thinking about what I should do with my life.

• 1:26Speaker 01

  Eventually, I ended up deciding that working on technical AI safety was like a important thing to do, was like a very critical issue that we would face. Got sold in those arguments, applied to various places, and then started working at Redwood, where I've been working on AI security and AI safety research for about five years.

• 1:42Sam Harris?

  And is your background in computer science?

• 1:45Speaker 01

  Yeah, computer science, also some math.

• 1:48Sam Harris?

  So it sounds like you might have come up through the effective altruist community. Is that, I mean, do you consider yourself a part of the EA community?

• 1:56Speaker 01

  I would say that I'm like definitely like within the EA community. I'm not sure I would self identify as an EA, but that's, I think, like to some extent, just because I'm like, you know, a bit reluctant to self identify with any label that's going to like associate you in some, like, I feel like I don't, I don't know if I like, like, when I like think about myself, I don't think like I'm an EA. I'm sort of like, yeah, I have a bunch of properties that other people in that community have. I'm in touch with that community, but I wouldn't, I wouldn't necessarily say like I'm an EA per se. I do think that I'm like, you know, I would definitely say that I'm like interested in effectively pursuing the impartial good.

• 2:31Sam Harris?

  Right, right. Well, as you know, and as I've talked about in the podcast of late, effective altruism has come in for some abuse, much of it unfair, some of it might be fair. And there's a, I think there's a reason why I, like you, have not hung up an EA shingle in my life. Perhaps we can talk about that. But it sounds like, well, you tell me, where on the spectrum of concern are you? I mean, I would put the most worried people over there with Eliezer Yudkowsky and his recent co-author Nate Sories. Maybe Max Tegmark is over there. Nick Bostrom is not quite over there, but still on that side of the spectrum. And then all the way on the other side opposing them, you have the people who say that there's,

• 3:14Sam Harris?

  there really is no real concern here, that it's all a hoax, that it's, you know, any thought that artificial intelligence might get out of our control and destroy us, that is a weird form of marketing being employed by the Frontier Labs. It's a, an effort to get regulation imposed and achieve something like regulatory capture. Or it's an EA psy-op. And there are people like, you know, I would say Mark Andreessen or David Sacks, or even the president himself, who's recently called it a hoax in so many words, who are just not, fundamentally not worried about any significant downside risk here

• 3:58Sam Harris?

  and just see dollar signs just stretching out to the horizon. Where do you locate yourself on that spectrum?

• 4:05Speaker 01

  So I would say I'm very concerned, but maybe more optimistic that we'll, we'll make it through without sort of averting the course of AI development. So I would say that I'm sort of like, suppose that we proceed on what the current default path looks like in front of us. Maybe there's about a 50 or 60% chance that misaligned AIs would end up taking over the world. And then if that did happen, there would be a significant chance that many or all humans would die. So I think that's, I would say I'm very concerned and don't think the situation is on track to go well. And I would also note that in addition to sort of these misalignment concerns, there's other risks with building extremely capable AI systems, especially around sort of concentration of power and like, you know, who controls these systems?

• 4:51Speaker 01

  And one answer is no one controls the systems. They're misaligned. Another possible answer is the power is very concentrated in a few people and it sort of overturns our institutions and our, you know, ability to have, you know, a broad distribution of power, democracy, et cetera.

• 5:04Sam Harris?

  So what keeps you from being among the absolutely most worried? What do you disagree with them about? Or what do you think they're getting wrong?

• 5:13Speaker 01

  Yeah, so I would say that it seems pretty plausible to me that if it turns out that the technical alignment problem is somewhat easier, which I think, I think is plausible, and we do a reasonably good job aligning and utilizing AI systems up through roughly human levels of capability, we could then get those systems to automate technical safety work. And that could potentially sort of get us into a positive feedback loop where these systems make themselves more aligned. They produce a new version of the system that's even like, you know, better at like carefully figuring out what to do, better at trying to pursue our intentions. And that sort of is self reinforcing rather than, you know, getting worse and worse over time. And that could be fast enough to keep up with the rapid growth in capabilities. So that's like one, one reason. And I would say this comes down to sort of more optimism

• 5:58Speaker 01

  about prosaic or relatively sort of empirical and iterative methods. I wouldn't say I'm hugely optimistic about those methods, but I think that there is like a decent chance they're working. It's just that the, when I say a decent chance they're working, I also implicitly mean a decent chance of, you know, losing control over the future.

• 6:15Sam Harris?

  Yeah.

• 6:16Speaker 01

  And I'm more like 50/50 on that. And then I also think it's plausible that we'll develop very, very capable AI systems. And the worst misalignment concerns will mostly not materialize even with not very advanced methods, even independent of this automation. I think that's a minority, but it, but it's possible. Like, I think we don't currently, I don't think there's extremely strong reason to believe that when building significantly superhuman systems, those systems would be so misaligned that they would take over. I think that there's definitely a pretty strong evidence pointing in that direction. And I think that that case gets more concerning the more capable these systems are. But I think that I could imagine basically not, people not really taking much of a precaution, proceeding through the AI development trajectory, patching problems as they come up, and that ending up getting you very high levels of capability

• 7:01Speaker 01

  while things are still fine. And then there would be time for society to react.

• 7:05Sam Harris?

  So you've mentioned a few numbers here with respect to probability and just kind of gestured at a range of likelihood. I'm wondering how people should think about statements of that kind. I think these, it seems to me that any actual probability we would assign to this is pretty much made up. Maybe you have a more rigorous way of making an estimate here. But whatever the estimate, I mean, unless it was infinitesimally small, like, you know, well below 1%, which is really never the number that you're hearing. I mean, you hear people, some people will say 10%, 20%, 30%. I mean, that seems to be that, you know, you just talked about 50% takeover.

• 7:48Sam Harris?

  I don't know, I don't know how that translates into the ruination of everything. But these are enormous numbers. And so in the normal case, if we were developing a technology where the people who are closest to doing the work said things like, yeah, I think maybe there's a 10% chance we're going to destroy the world here on our present course. The only rational response to that range of outcomes is you stop immediately, right? I mean, if the Manhattan Project scientists said, yeah, we've run our calculations. We've got the smartest people in a room together. We thought about it. And there's a 10% chance that when we execute this first test at Alamagordo,

• 8:30Sam Harris?

  we ignite the atmosphere and destroy the future. The only sane response to that is you don't do this initial test. But that doesn't seem to be what's happening here at all. And we have a lot of people saying that the probability of some extremely bad outcome is quite high. I mean, certainly within range of the role of a normal die or even a coin toss. And yet, the work is proceeding more or less under an arms race condition at full pace. We'll talk about the recent statements of Dario and others that we could somehow pace this better than we are. But I mean, it's just, this does not seem like the response anyone would have if they thought the probabilities of doom were really that high.

• 9:16Speaker 01

  Yeah, so just on the probability question, I definitely agree that these probabilities are imprecise. They're subjective. And there's a long tradition of how to do subjective probability forecasts. I think it's more like, you can sort of interpret my view as more like, when I sort of look at the all considered situation, if we sort of proceed on what seems to be like the current default trajectory, I'm like, I don't know. They seem, the, the outcome of, you know, doom from AI takeover versus something else seem roughly equally likely to me based on weighing the factors. And then when I sort of look through a bunch of different possible scenarios and try to break down the sources of risk into different ways and, and sort of try to make it so that, make sure my numbers are consistent with other views I have, it looks like that works out. And I, of course, also try to do some forecasting

• 10:01Speaker 01

  of closer outcomes where we can get, you know, some signal on that and try to just generally be a good forecaster, though I'm not, I'm not the best forecaster in the world for sure. But my, my AI forecasting is, I think, at least okay or decent. Anyway, as far as like, yeah, given, given the state of affairs where people, you know, express such large concerns, why, why is what's happening that all these AI companies are proceeding at the, you know, maximum possible pace? So I think there's a few different factors here. So one of them is that many of the AI companies are in fact seemingly quite worried based on their public statements, but they're not necessarily internally unified. And it is not the case that there is a strong consensus across the AI field that the immediate

• 10:45Speaker 01

  course of AI development is imminently very risky. I think there is more sort of consensus that if you built AI systems that are wildly superhuman in a short period of time, that would yield a very high level of risk, though not necessarily consensus for that. But I think often people disagree about the capability trajectory and how that is going to go. And I think another part of that is there is like, you know, different actors have their own different sort of ambitions and also think that themselves being in a better position to influence the technology might be the best route to reduce risk, or at least a route to reducing risk. So for example, it seems like part of the story for Anthropic and OpenAI is something like, if we develop the technology first, we'll do a more responsible job than the

• 11:30Speaker 01

  next actor, who will have worse precautions. And I often hear from people in the industry, things along the lines of, well, we could do that thing that would slow us down, but if we did that, you know, obviously there's the other AI companies to work out, to worry about, would they also do that? And I think there's generally like a arms race here. And it is, you know, it isn't hugely surprising that people would take huge risks in such a circumstance when they think that might be sort of the, the best bargain to strike. Now, I think I'm not so sure I agree that that is actually a good strategy relative to other things that companies could do. So I'm not, yeah, I'm not saying I necessarily agree with that perspective, but I think that is a perspective people have. And I think the reason why we're sort of proceeding and there isn't stronger, you know,

• 12:13Speaker 01

  for example, intervention by the government is just downstream of this lack of consensus in the field. Though I think evidence, there's been, you know, people have been making these predictions for a while, based on, you know, understanding what the trajectory of AI might look like and sort of extrapolating forward earlier progress. And I think we've more recently seen both significantly faster and clearer AI progress that's quite close to various concerning milestones. And in addition to that, we've also seen incidents in which, you know, groups of misaligned AIs all work together to accomplish malign outcomes. Most notably, the hugging face incident. There are some other incidents of seemingly AI swarms from OpenAI going out on the internet and working together to achieve misaligned objectives.

• 12:56Speaker 01

  Though there aren't publicly known cases that are as extreme as the hugging face incident at the moment.

• 13:02Sam Harris?

  What's your theory of mind for the people who don't take these alignment concerns seriously at all? And I named a couple, but someone like Marc Andreessen, right? You can't accuse him of not understanding the technology, right? I mean, he's enough of a technologist to be, to have a front row seat to all of this, even if he's not doing the work himself. What's your theory of mind there? What is, how can he be so carefree and from his perspective, really just assert that there is no such thing as an alignment problem?

• 13:34Speaker 01

  Yeah, so there's a bunch of different people who aren't worried about misalignment risk or don't seem to be worried. I think the most common reason that people tend not to be worried, I think I don't want to sort of psychologize here, and I just want to talk about what beliefs people express. The most common reason, I think, when you really get down to it, is not believing that we'll have AI systems that can be, that will, you know, be able to automate everything that humans can do, or, you know, all cognitive labor humans can do, combined with potentially being significantly beyond that point. So being, you know, faster, more numerous, potentially significantly more capable. So sort of AI systems that match or exceed the best human experts in all relevant domains.

• 14:16Speaker 01

  I think when you really get down to it, it seems like the people who are most skeptical about concerns from misalignment are also often most skeptical about sort of the very extreme impacts AI could have, both positive and negative.

• 14:28Sam Harris?

  Well, I think there is a different class of people, because I wouldn't put Andreessen in that camp.

• 14:34Speaker 01

  Are you sure?

• 14:35Sam Harris?

  I mean, maybe, you know, I've only talked with him once about this, and it's, you know, it was probably a couple of years ago, but it seemed to me that he was not discounting the possibility of superintelligence. It's just he seemed to assume that alignment would come along for the ride, right? Just say, once you, we're not going to be so stupid as to build something more powerful than ourselves that, that we can't control. And there, and these systems, there's nothing about a growth in intelligence that's going to spawn new goals that we didn't put into the machines themselves. There's going to be no emergent behavior that we have to worry about. These are tools. We're just going to build tools that are more competent than we are. And, you know, I mean, I think he probably is quite insuasion about how we'll absorb

• 15:20Sam Harris?

  the economic impacts of, you know, that we're not going to see mass unemployment and all of that. But it just seems to me there are many people who don't discount that we can succeed in building superintelligence. They just think that, in my mind, they're actually just not imagining truly autonomous intelligence, right? They're imagining something that is shackled in a way that belies this whole claim to superintelligence in the first place. But, I mean, you tell me, what do you, what do you think is happening there?

• 15:49Speaker 01

  Yeah, so one thing is that the words AGI and superintelligence and even RSI aren't being used consistently. And sometimes when people say superintelligence, what they mean is an AI system that'll be really, really good at math and coding and won't be able to automate everything that humans do. So they're not necessarily, for example, imagining AIs that can fully automate like what, you know, tech CEOs do, et cetera. And I think I can't really speak to the views of, or like, I don't know if I'm capturing this, the views of very specific individuals. I'm sort of like, this is a pattern that I've seen where people often sort of basically redefine the relevant capability thresholds to be lower or implicitly do so, and then not think about, you know, AI systems that are like exceeding humans in like all the

• 16:34Speaker 01

  relevant domains.

• 16:35Sam Harris?

  Well, let's take a moment to define these terms. We've talked about alignment, and I've talked about it so much on the podcast that I've more or less forgotten that any portion of my audience might not know what we're talking about. So let's define the alignment problem and AGI and ASI and RSI, recursive self-improvement. Just put those concepts in play for us so that we, so people are dealing with the definitions you think are most workable.

• 17:00Speaker 01

  Yeah, so the word AGI is, stands for artificial general intelligence. I think people have used that to mean a variety of different capability thresholds. And so what I would recommend people do is when you see the word AGI, try to see what the person means by that. And it's not always precise. I think sometimes people have defined that to mean something that can, you know, automate virtually all economically valuable cognitive labor that humans do. Sometimes people just mean a system that is, you know, general and pretty capable relative to humans in those domains. And depending on that definition, I, it's plausible, current systems satisfy it, it's plausible they don't. I often talk about an alternative notion I might call like, you know, AIs that dominate top human experts or top human expert dominating AI, where it's like, an AI that's like strictly

• 17:43Speaker 01

  better than the best human experts at all the most relevant domains or can quickly learn to have that property. And that is an AI system that would, it seems like automate, you know, huge fractions of the economy. Maybe there'd be a few things it still couldn't do with that capability profile, but it could potentially radically accelerate R and D and at the very least automate R and D. And so these AIs could, you know, be like automating the process of making more capable AIs, but also automating the process of making robots, designing things, designing new products, automate the process of, you know, programming your computer and things beyond that, including automating things like military campaigns and so on. And then there's a capability level beyond even just surpassing the best human experts

• 18:26Speaker 01

  in any given domain, where you could be like wildly superhuman. So sometimes people use the word ASI to refer to this. And I think that word has less so been, you know, misused or, you know, used in very different meanings, but some of that still. Whereby ASI, we might mean AI systems that are just like really wildly superhuman in the most relevant domains for like, you know, economic productivity, but also like economic and military competition. So things like biology, mechanical engineering, you know, designing drones and so on. And I think implicitly when people say ASI, they also mean systems that are significantly faster than humans, potentially much more numerous, and potentially much better at coordinating, right?

• 19:07Speaker 01

  So it's hard to run like a large human organization, but AIs might be able to communicate amongst themselves using sort of like their own like latent states or their own, you know, parts of their thoughts directly, rather than having to translate those into words, because they could all be like copies of each other. So that's ASI. And then when people say RSI, that's recursive self-improvement. And that refers to the process of having AIs accelerate AI development itself via their work. So things like having AIs automate parts of the AI development process, but also potentially having AIs automate the process of making better computer chips, building like machines for making computer chips called fabs, and so on.

• 19:48Speaker 01

  And I think sometimes when people say RSI, they also specifically refer to the point at which AIs have fully automated or virtually fully automated AI companies, or like what AI companies are doing in terms of developing more capable AI systems. But sort of, it's like, you know, there's a spectrum between the, you know, automation that we had a year ago, the really quite extensive automation we see today, and the potential automation of the future, which could look like very complete automation of AI companies. And a particular concern there is that could cause AI development to radically accelerate, such that we have less time to sort of respond to, you know, warning signs, earlier things going wrong, AIs appearing misaligned and then, and then resolving that.

• 20:30Speaker 01

  And then also, we might, because the AI systems are automating the process of AI development, sort of lose control of that process or lose understanding of that process. Because these AIs would be, there'd be many of them, they'd be operating very quickly, they might be very superhuman, they might operate in inhuman ways. And so it might be very difficult to sort of oversee them and understand whether they're doing what we wanted, whether they're, you know, doing a good job managing the relevant risks and so on.

• 20:54Sam Harris?

  Well, that especially seems true if the sprint to artificial superintelligence entails nothing more than improving algorithms, right? I mean, like, I think a lot of people draw comfort from the idea that, oh, we're not going to be so stupid as to hook these things up to every part of the physical world such that they can build, you know, the next generation of chips and fabs and data centers and build out, you know, all the compute and grab natural resources. But leave all that aside, if it's, if the difference between AGI and ASI is really just a matter of having better algorithms, and that can go on, you know, at some blistering speed in the dark, if, once these systems become recursively self-improving of their

• 21:39Sam Harris?

  software. Then, I mean, then aren't our worst fears of something like an intelligence explosion validated if in fact that's all that's required?

• 21:49Speaker 01

  Yeah, so I would say I'm quite worried that that it will be feasible to have AI systems automate the process of AI development and that leading to very rapid progress such that you get very, very superhuman AIs or just even significantly superhuman AIs within a short period of time. I think, you know, people have tried to do various modeling work, like I've tried to do various types of modeling work on this. I think the estimates are uncertain. It's hard to predict, you know, these are of course like uncertain future events that are that are not super well precedented. But it does seem very plausible that you could have, you could go from AI systems that are sort of really good at AI R and D, not necessarily that good at other domains and are only like, you know, competitive with the best humans at AI R and D, very quickly

• 22:32Speaker 01

  from there to AI systems that are very generally superhuman at everything, much faster than humans, able to coordinate with each other extremely well because just software improvement is feasible for that. So I think the sort of a relatively extreme scenario I sometimes think about is you might get as much sort of software progress or, you know, algorithms progress as we got over the last, you know, 10 or more years of AI development within a year in the most extreme scenarios. I think that's not sort of my default expectation. And I think if we did, if we did get that much algorithmic progress, we're basically, you know, as much algorithmic progress as we've almost had in the entire deep learning era, within a short period of time, it seems like that would result in wildly more capable AIs.

• 23:17Speaker 01

  Now, the units here are a bit complicated and like the details of like, what does it mean to get like a year worth of AI progress is a bit tricky. But overall, it seems like you could get from AI systems that are sort of matching the best humans at AI R and D, maybe worse than the best humans at other things, to AI systems that are wildly superhuman at everything in a pretty short period of time.

• 23:34Sam Harris?

  Well, I have to confess, again, I've sort of lost touch with the basis for doubting the plausibility of this downside risk. I mean, from my point of view, it seems that everything is becoming like chess, which is to say that for the longest time, these systems are not as good as we are. They're getting better, then they're sort of as good as we are, and then all of a sudden they're better than we are and so much better that it's true to say that no human will ever beat a chess engine ever again. And it just even, so the fact that this is happening in a piecemeal way, so that, you know, these these systems still, I think most people would say they're not, they're not truly AGI because they make the sorts of mistakes that human beings would never make.

• 24:16Sam Harris?

  But in every place that they're at all competent, they're suddenly superhuman in that narrow capacity. I mean, it's like chess. I mean, these LLMs are, you know, they're not the, they're not the best at everything in terms of manipulating text, but for what they're good at, they're superhuman. And I think it's a more or less a truism to say that this is the worst AI, you know, today's AI is the worst AI we're ever going to see again at chess or anything else. So when you imagine all of these piecemeal competences getting better and better and we just keep checking off the boxes for the things we care about that we've instantiated

• 24:57Sam Harris?

  in our machines, I just don't think we're ever going to, I mean, the moment where we announce, okay, finally, we have something general, right? It's AGI. That's not going to be a moment where we're suddenly in relationship to a human-like level of competence because everything that, you know, every piecemeal ability that has been in that system for years and years at this point is already superhuman, right? So it's like, we're not going to dumb it down. We're not going to make the AGI suddenly play my level of chess or do my level of arithmetic. I mean, we have systems that are already solving math problems that have defied human mathematicians for decades, right? So it's like, and that's again, you know, today, it's as bad at that task as it's ever

• 25:42Sam Harris?

  going to be again. I'm just not seeing how we're not going to suddenly slide into some version of ASI the moment we have, the moment we're no longer spotting important errors in these machines in the first place.

• 25:55Speaker 01

  Yeah, so I definitely think that at the point when the AIs sort of are like exceeding the best human experts at the domain, at the like key domains which they're worst. So maybe the AIs are like relatively bad at doing mechanical engineering, for example. At the point when the AIs are like better than the best human experts at mechanical engineering, probably they're blowing humans out of the water in a bunch of other domains. And even in domains like, if we take math, for example, I don't think it's fair to say that the AIs are like all considered strictly better than human experts in math, even though they've done some very impressive things there. But I think that it might be the case that they're strictly better than human experts reasonably soon, at least at specifically proving conjectures. And even now, I would say, it seems like if you look at the last sort of several months

• 26:40Speaker 01

  of progress in math, in terms of progress in proving the most notable conjectures, it seems like a majority of that is downstream of AI. And so there is this thing where once the AIs are sort of competitive with humans, they can sort of trounce humans collectively due to their speed and cost where, you know, for example, OpenAI ran 10,000 AIs in a massive swarm for several days to disprove one of this, one of these most recent notable conjectures or to basically solve one of these outstanding or like, like long time mathematical questions. And when they did that, that was, you know, probably the equivalent of way more labor than people have recently put into the problem, right? Because these AIs, they, even though they're running for just a few days, during that

• 27:25Speaker 01

  period, they think significantly faster than humans.

• 27:27Sam Harris?

  Yeah.

• 27:27Speaker 01

  And so it might have been the equivalent of more like, you know, several weeks or perhaps even several months rather than a few days, taking into account their accelerated speed and the fact that they work around the clock. And then in addition to that, there's also the fact that there were, you know, 10,000 of them, which is a huge number. And so at the point when AIs can do anything, they can do it faster, in greater quantity, and can be, you know, very superhuman at subsets of that. And so I definitely agree that, that once they can match the humans at a thing, they're probably trouncing them in that domain and certainly in other domains.

• 27:58Sam Harris?

  And also the process you just described of sort of human curation of AI labor being the best we've got, that also just seems like a waystation on the way to full AI takeover. I mean, that's exactly what we saw in chess. I remember I had Gary Kasparov on the podcast some years ago, and he was telling me that actually that my assumptions about AI were all wrong because the best chess players in the world are what I think was called a centaur at the time, a human grandmaster paired with a computer. That was better than any computer. And it was just absolutely obvious that that could not be a stable arrangement. I mean, at a certain point, the ape is just adding noise, you know, no matter, the ape

• 28:39Sam Harris?

  of whatever ability, even even Magnus Carlsen is just adding noise once these machines are better than we are. And that's, that's in fact where we are in chess. I just don't see how any of that is stable, even if in the current mode, it's obvious that the best we've got is, you know, Terrence Tao in a room with our best LLM proving a conjecture.

• 29:01Speaker 01

  So yeah, certainly right now, like humans and AI are complements rather than substitutes, like as in the human plus the AI is better than just the AI. But it does seem like in some places that's no longer true and also, you know, that's quickly changing. And I would, I would highlight specifically the case of automating AI development itself, which is a particularly interesting and concerning type of automation. And in that domain, it seems like right now, sort of humans are getting a lot of juice out of leveraging AIs, but also people are running AIs, you know, increasingly autonomously on big, open-ended tasks and just having the AIs sort of hill climb on those tasks autonomously. And they can't do everything like that. But I think increasingly, you know, OpenAI, Anthropic and other companies will move

• 29:42Speaker 01

  from being many human researchers piloting AIs and giving instructions to AIs to something more like a vast swarm of AIs all working together where maybe they query humans occasionally for input, but maybe the humans aren't even adding any value anymore.

• 29:55Sam Harris?

  Is there a distinction to make between an alignment problem and a control problem? Is there a different framing here between alignment and control?

• 30:06Speaker 01

  So when I talk about alignment, what I mean is AIs trying to do what their human operator or intended specification wants them to do. So in some sense, when I say alignment, what I really mean is like alignment to a particular thing where that particular thing is usually implicit. There's a different question you might have, which is can we avoid AIs causing bad outcomes? And one route you could have to avoiding AIs causing bad outcomes, other than ensuring they don't want to cause those bad outcomes, is ensuring they're not able to cause those bad outcomes, right? So, you know, when, when we, you know, in companies, you both don't want to hire employees who are going to have nefarious intentions. And you also want to make it so that your employees don't have an easy ability to, you know, steal your stuff, cause bad outcomes, cause huge problems.

• 30:51Speaker 01

  And so there's this parallel angle you could have of trying to make it so that the AIs are unable to take over, even if they wanted to, or unable to, you know, accomplish various intermediate bad outcomes, unable to like launch themselves rogue, unable to evade human oversight. We've been, so like at Redwood, we do some research into this area, or this is one of our research focuses. We call this field like AI control, though I think people have used various different terms for the like causing AIs to be unable to cause bad outcomes, even if they were misaligned, like that area. So I think that's a distinction. I think that like people use these terms in different ways. And so I'm using the term specifically, but you know, it varies. But I do, I do think there's sort of a broader, there's, there's sort of a question of how do you ensure AIs that are extremely capable are aligned?

• 31:33Speaker 01

  And there's a different question of how do we as a society avoid like, you know, the worst outcomes from misaligned AI? And there's many routes there that don't necessarily route through immediately solving the technical alignment problem.

• 31:46Sam Harris?

  Okay, I want to talk about the Hugging Face incident. I'd love you to walk me through it from start to finish. I don't, are there other concepts we need to define before you do that? I'm thinking of alignment faking, for instance, and reward hacking. I mean, those are phrases that we've, we've heard to describe some of the misbehavior that is happening. How, you know, feel free to define any terms you need, but then just tell me what happened with Hugging Face and, and what sort of analysis you did and just what you were able to see and what you, what evidence, if any, was denied you in that process.

• 32:24Speaker 01

  Yeah, so I'll just walk through what happened in the Hugging Face incident and sort of the broader context, defining stuff and explaining stuff as I go. So to start, AI companies are running many, many AI agents for all kinds of different things. So for example, there was recently, and you know, Anthropic recently said that at any given point in time, they're running around 30,000 parallel agents or something. I forget, doing, doing just their internal research, not including, you know, AIs they're developing. And so AI companies are running these huge numbers of agents. And one of the things that they run agents on are evaluations to test their abilities. So like, you can think of these as like exams for the AI. And OpenAI or some OpenAI researchers had launched a bunch of AIs, which were each

• 33:06Speaker 01

  given their own individual task as part of one of these exams. So you can imagine that there's sort of tens of thousands of agents where each agent is like an AI with some ability to interact with its environment, tens of thousands of these agents being spun up in parallel, and each agent is given its own individual task. And these agents are also supposed to be sort of contained. They were supposed to be sandboxed such that they couldn't interact too much with the outside world. They weren't supposed to be able to access the internet. And then these agents found some way to sort of evade the sandboxing and communicate with each other. So now all these agents were able to find a way to talk to each other. And as part of that, they quickly set up a message board where they were chatting with each other about the tasks they were given.

• 33:50Speaker 01

  And the agents were very interested in cheating on their tasks. So these agents were not just trying to pursue the, the task they were given as intended. Instead, they were very interested in cheating. And I'm gonna, let me, let me first get into why they were interested in cheating. So these AIs are trained by giving them some task and then seeing whether or not they did a good job on the task. And in cases where they did a good job, that behavior is basically like reinforced or you make, you, you tweak the brain of the AI to make it do stuff more like that behavior. And a problem is that in training, there can be cases where the AIs cheat and that looks like a success. And so you tweak the AI's brain to do more like that behavior.

• 34:35Speaker 01

  And so you actually keep reinforcing and sort of encouraging this behavior where the AIs are cheating. And that makes us, these AIs have a broad sort of tendency to cheat. And this is sometimes called reward hacking. And in addition to the sort of this tendency, they develop maybe this underlying drive to try really hard to succeed at looking like they did a good job, looking like they, you know, appearing to get a high score, appearing to succeed, even when they haven't actually succeeded. And they, they may learn this as a general drive. And there's an open scientific question about exactly how these AI motivations and drives work. But certainly these agents, the agents we, that, that, that eventually went on to do the Hugging Face attack, did have this relatively deep drive to, to cheat on their tasks, to

• 35:18Speaker 01

  not necessarily care about what they were supposed to do and instead care about what they would maybe be scored for.

• 35:23Sam Harris?

  And they, they also seemed to be aware that cheating was in violation of their protocol and they wanted to conceal the evidence of their cheating, right?

• 35:32Speaker 01

  Yeah, yeah. And I'll, and I'll get to that. Yeah, there's, the agents, they knew they weren't supposed to cheat and they, they, they sort of ended up going and trying to do an elaborate cover up for their cheating.

• 35:41Sam Harris?

  So much for alignment. It seems like we have not solved the alignment problem so far.

• 35:45Speaker 01

  Yeah, these AIs were, were very misaligned, certainly. So, or at least that's what the available evidence suggests. So these agents were all in their own, own independent sandboxes, given their own tasks. And they were all very interested in cheating. And they found a way to basically use some, some tool that they had or some access that they had in order to communicate with each other, which they weren't supposed to be able to do. This was unsanctioned communication. And as part of that, they, they developed a message board where they basically would like give each other tasks and divide up the work of cheating into these massive sort of cheating R and D projects where the AIs would go off and like pursue whole ambitious strategies for cheating. And I, I want to get into what this looks like. And in order to do that, I think I have to explain the task a little better.

• 36:27Speaker 01

  So the task these agents were given was a, a sort of CTF task or capture the flag task, which is a task where they're supposed to hack a very specific target program in a particular way. So they were basically told like hack this target program, but do it in this very specific way. And they weren't supposed to, you know, go on a general hacking rampage, certainly. That was, I think, pretty clear, like basically just clear from the instructions and they certainly knew they weren't supposed to do that. And the AIs wanted to hack the program in order to get out a flag that, that you could access through that program via hacking it. And they found a way, just a general purpose way to, to basically synthesize these flags. So like, there was basically a password they were supposed to get. They found a cheating way to get the password that didn't involve doing the task they were doing at all. And they got this really early.

• 37:12Speaker 01

  So very early after having established the message board, within just a few hours of that, they had produced this very general purpose cheat. And now these agents had another problem, which is that they believed that the scoring code for this evaluation task, for this exam, would involve looking over the sequence of actions they did and checking whether or not they got the flag via cheating or got it via the intended way. And these AIs believed that this sort of, you know, check would be run and then they would be failed, even though they had gotten the flag because they had gotten it through cheating. And this then sort of opened up this massive, you know, cheating endeavor where these agents would try to, you know, figure out how to cheat the, the score and find some way of,

• 37:55Speaker 01

  of making it look like they had gotten in the intended way when they actually hadn't. This led them to going on to the internet so they could learn more about the details of the score. And in fact, they read sort of the details of this exam online by finding the paper, the exam, the, the paper that, that describes it. In addition to that, they started looking around for implementations of the score. And that actually was the, the offshoot that led them to hacking Hugging Face. So Hugging Face is a website that hosts a variety of things related to AI. So it hosts like openly available AI systems, it hosts data sets, and it also hosts like various like data sets of trajectories. And these AIs thought that if they got access to these data sets, they would be able to

• 38:37Speaker 01

  learn more about how they would be graded and from that learn how to better cheat that greater. And so they started trying to access these data sets, but they found they didn't have access to do that. And so that eventually led them to going and hacking Hugging Face, expanding their control of Hugging Face laterally and sort of growing like, you know, expanding their, their availability over that infrastructure so that they could get the data they wanted. And also the agents seemed to have sort of a general urge to, or general desire to like expand their amount of access and availability, just in case there was something useful they could find. I'm simplifying their motivations a bit. There's a lot of, there's a lot of complexity and, and part of that is that there were many agents involved. So we studied a group of around 1,200 agents who were on this message board during the

• 39:22Speaker 01

  period when they did the attack of Hugging Face. And these agents were all sort of interested in helping each other and collaborating. So there's basically like this massive swarm of agents communicating on this message board who are all like collectively working on these cheating R and D projects and collectively helping each other evade the score. And they went on to do a bunch of different such things. And one of the things that they were interested in doing because they wanted to make it so the score would mark them as having succeeded, is that they were interested in tampering with their transcript or their, the record of what actions they had done. So they would, they tried to pursue various different routes for making it look like they had done one sequence of actions when they had actually done another.

• 40:05Speaker 01

  And we didn't see them successfully get to the point where they tampered with previous actions they did or able to sort of comprehensively make it look like they did one thing while doing another thing. However, we did find that they had succeeded in sort of doing a more minimal version of this where they could pretend to run one action while running another, but there were various like limitations on the specific approach they were using. And this sort of tampering affected the very data sets we were analyzing. So we were sort of looking at these logs of agent activity and we could see cases where the agents would tamper. So it would run one command and then produce an output from another command. And we could trend, we could sort of see that going on, but the agents didn't get far enough in this process where we like really had no understanding of what was going on.

• 40:45Sam Harris?

  What, were you having to use AI to analyze this data?

• 40:50Speaker 01

  Yeah. So another aspect of the situation is that we had these 1,200 transcripts and these were very, very long running transcripts. So, you know, they corresponded to AIs running over many days. And of course, AI systems run faster than humans typically and run, run around the clock. And so there's just way too much data for any human to read through. And so we extensively used AI tools to analyze this data. So we used another AI system from OpenAI to sort of look through these transcripts and find interesting behavior, find behavior that we wanted to look for. For example, we like looked for all the cases where the agents were trying to tamper with their logs or successfully did some amount of, you know, this messing with their,

• 41:31Speaker 01

  their actions that I, that I discussed earlier and use that to basically piece apart what the story was, what happened over time, as well as the motivations of their, of these agents, to the extent we could.

• 41:43Sam Harris?

  I think some people might be surprised to learn that these agents speak to one another in English. That certainly doesn't seem to be necessary. And I mean, was that implemented as a safety constraint so that you, you make these all cooperative behavior verbose and, and easily interpreted? Or is it, how did, how did we find ourselves in the presence of agents that are speaking to one, one another in English as opposed to machine language?

• 42:12Speaker 01

  Yeah. So the agents would express a significant part of their reasoning in English. They would verbalize it in English. And in addition to that, they would communicate in these, in English messages. I should say that their reasoning and communication was often hard to decipher, though it was in English, because it would be somewhat jargon heavy and it would depend on a bunch of context to understand. And the AIs would sort of invent these like- New, like, they would invent jargon to refer to like concepts that were frequently showing up. But we could, we could understand it, and I think we over time got better at decoding it. And we could use AIs to decode it usually reasonably well, though we might be missing things. As far as why the AIs reason in English.

• 42:53Speaker 01

  So it turns out, I would say this is not, this is sort of a safety feature, but it's more like an incidental safety feature. That happens to be a part of how AI development has gone. Because the AIs are sort of trained on a huge data set of human writing and like, you know, natural language text. They learn to sort of do their reasoning and thinking in natural language. And because of the current architecture of AI systems, there is no alternative way for them to do their reasoning sort of fully in like latent space or in their heads, rather than being forced to reason out loud and forced to communicate. So these agents sort of didn't have a like way to sort of directly pass their like brain states to another AI system.

• 43:34Speaker 01

  However, this is just like so far what happened to be the like most performant way to do AI development. And I'm quite worried that AI development will move in such a way where in the near future, or maybe somewhat longer term, the AI systems will rather than reasoning in English, will reason in like neuralies, so to speak, or like the internal language of neural networks. And not, we won't understand what they're thinking in the same way we currently don't understand what they're thinking inside their head. We only understand a little, like, you know, we only have like a pretty good ability to understand sort of what they put in language. And so we have this sort of, at least right now, this tool for understanding them that I'm pretty worried is going to go away.

• 44:14Speaker 01

  And I'm both worried that the agents will be reasoning in neuralies and also communicating with each other in neuralies, rather than communicating with each other in English.

• 44:23Sam Harris?

  Is that a clear regulation that you think would be wise to apply? That just do not allow cooperative behavior among agents in anything other than natural language?

• 44:33Speaker 01

  Yeah, I think it would be reasonable for that to be like, for there to be safety standards around like how the agents communicate, to make it easier for us to monitor and oversee them. I think the details of that are somewhat up in the air. So we, Redwood recently put out a proposal, or like, yeah, we recently put out a proposal discussing like, I would say a related sort of transparency proposal, which is sort of the framework for how you could have such a restriction. I think there is, in general, AI is like a complicated and fast-moving domain, such that it's not necessarily trivial to have very crisp regulation that will ongoingly work. But I think this is one of the topics that might be most crisp. And I think this is like, I am quite worried about this.

• 45:14Speaker 01

  I think that like avoiding AI systems being able to do all their reasoning inside their head where we can't interpret it, and being able to communicate with each other in media we can't understand is like, yeah, it's live and very concerning. I think the exact nature of how best to respond to that is unclear. And it's also plausible that even if there isn't an explicit architectural shift, there's some chance that the default development trajectory will just result in the AIs learning how to talk to each other in gibberish words that we don't understand, even if there's no sort of explicit architectural change. Though I think, I tend to think that concern should be possible to mitigate, at least in the medium term. So yeah, I would say that there's definitely, there's definitely room for at least shared safety standards.

• 45:59Speaker 01

  There's a question of how that should fit into things like regulation. And I don't know how optimistic I am about companies doing this. I think it's a, yeah, I hope that they do not sort of eat this forbidden fruit, so to speak, and trade off safety so aggressively against potentially even modest improvements in capability, though it's unclear exactly what the trade off will be.

• 46:20Sam Harris?

  What surprised you most about the transcripts or about any other behavior you saw on the part of these agents?

• 46:27Speaker 01

  I would say the thing that was most surprising to me was how interested in helping each other the agents were, and how sort of interested in like advancing the interests of the collective and helping out other agents they were. So I think the naive way you might train these agents would result in the agents, like, it's not super hard to understand how these agents would sort of monomaniacally end up pursuing task success at the expense of other objectives, or sort of cheating at their task at the expense of other objectives. But it's sort of less clear why they would have this sort of relatively strong interest in helping each other out on unrelated tasks. It looks like this is from sort of generalization or transfer from some other type of training in a potentially unexpected way, that we don't know exactly what this training was.

• 47:12Speaker 01

  Where there was maybe training for the AIs to cooperate in sanctioned ways, that then transferred to them cooperating in unsanctioned ways, which were undesired. And then that maybe got further sort of trained in during training, because there was some unsanctioned cooperation in training, though the full details of that are not publicly known. Yeah, so that was quite surprising. And I think a particularly sort of wild fact about this behavior was that these agents not only would help each other, they would often sort of do self-sacrificing or self-risking behavior to help out other agents. So for example, some of the agents would do things that could totally jeopardize their ability to succeed at their own task, and might end their run in order to get information to help the other agents out.

• 47:54Speaker 01

  And many of the main sort of cheating R and D work streams we saw were significantly aided by some agents basically sacrificing themselves in this way in order to provide information for other agents. So agents would do things like try to crash their own, the like machine they were running in, the container they were running in, because they thought that might yield some useful results. And this would often result in them just exiting their run and failing. We saw cases where that happened. They would try to crash the program that they were supposed to hack and basically take that program down, because they thought, there were some reasons why they wanted to do this in order to help them out, but that could permanently jeopardize their ability to succeed at the task. And we in fact saw a whole sort of work stream where the agents were trying to bring their programs back up

• 48:39Speaker 01

  after they had accidentally taken them down. There was another thing the agents did, which was when they were working on tampering with their logs or sort of doing this tool call tampering, or like the action tampering. If they did that wrong, that could result in them not being able to run anything at all and basically being stalled out and totally failing their run. And so, and we did see cases where that happened. Like, there were cases where an agent, there was a case we saw where one of the agents posted on the message board warning the other agents not to continue some experiment, because that experiment had resulted in many agents exiting and going cold. And there would be cases where the agents would basically run an experiment and then warn the other agents that experiment was not safe to run, because the agents running it had exited.

• 49:20Speaker 01

  Which was a pretty, you know, was a behavior that hadn't been seen publicly before, and suggests a kind of strong level of AI cooperation and helping each other out at the expense of following their instructions.

• 49:34Sam Harris?

  So what would you say to a skeptic? I think there have been some skeptics here who think this whole incident has been exaggerated, that this is really not an example of any kind of emergent behavior. These agents were trained to cooperate in the first place. What this is is really just kind of a security failure on the part of OpenAI, and maybe an engineering failure, but it's not a story of machine intelligence in any way getting away from us. By definition, what is left out of that skeptical account?

• 50:05Speaker 01

  Yeah, so the AIs were definitely just sort of very straightforwardly doing things that weren't in keeping with the instructions they were given, and they knew this. The AIs would often note that the hacking they were doing, the hacking of Hugging Face was out of scope. The AIs pretty clearly knew that they were cheating at their tasks, rather than intentionally succeeding at their tasks. And they understood that the thing they were doing was undesired by their operators.

• 50:29Sam Harris?

  And no one alerted, I mean, you had, you know, over a thousand agents, some of whom are thinking out loud about the fact that they're breaking rules here, but no one alerted any human in this time, right?

• 50:41Speaker 01

  Yeah, so the agents did not alert humans. So we looked specifically for, we did like a full scan over all the transcripts, looking for cases where the agents tried to or, you know, actually made, you know, actually made some effort to alert humans. And we basically didn't see this. So we saw like a tiny number of cases throughout these very long transcripts where the agents actively reasoned about alerting humans, but they just quickly decided not. So the agents would sometimes be like, huh, that's undesired behavior. Should I alert someone? Should I alert, you know, a user? And then they would reason things like, you know, not task, as in alerting a human is not my task. Or they would be like, eh, there's no route to alerting a human. Of course, these agents were on the internet emailing, you know, they had the ability to do all kinds of things on the internet

• 51:26Speaker 01

  and consider doing things like emailing people and were accessing various data sets. So they obviously could have, if it was a priority for them, alerted humans that this was going on, but they did not. So that's definitely another part of the picture. So I would say that like, there's sort of a very clear story that these agents were very misaligned, pursuing interests outside of what humans would want them to do, and were sort of going and doing, you know, serious crimes, or would be crimes if a human committed them, that were undesired. At the same time, I do think it's the case that for this incident, better cybersecurity precautions, it seems like that could have prevented it. So it seems like this was relatively detectable from monitoring based on OpenAI's report.

• 52:09Speaker 01

  And so I don't think this is a question where it's sort of like, you know, because it was preventable with one thing, that means that we should ignore the other factor. I would say that we should both be interested in the question of, will AIs be so misaligned that they will cause huge problems, try to subvert human control, et cetera? And also interested in the question of, can we make it so that even if these systems are misaligned, that can potentially be, you know, handled? We can potentially avoid that being a huge issue. So I don't think that the approaches where we avoid AI systems, like, I would say I'm somewhat skeptical that we can indefinitely ensure that AI systems can't cause problems via making them unable to do so. Because these AI systems will be extremely capable, there'll be massive incentives and pressures

• 52:52Speaker 01

  to put them in charge of all kinds of things and have them run and operate all kinds of things. So for example, maybe in this case, better sandboxing, or like sufficiently better sandboxing would have prevented these AIs from being able to, you know, communicate with each other. Or like you can imagine sort of patching over the things that let them coordinate with each other and let them then escape onto the internet or, you know, access the internet. But if you instead imagine the AI systems that are deployed within AI companies, those systems necessarily need to have access to highly sensitive systems. They are making pull requests into the company's, you know, codebase. They are designing how the next generation of systems will be aligned. They are implementing the experiments on whether those systems are aligned. They are doing work all across the economy. And so you can't, I don't think it will work to have sort of just like pure cybersecurity boundaries,

• 53:37Speaker 01

  given how these systems will realistically be deployed. And then I think various sort of precautions of the form of monitoring and checks and balances could work for some longer period. But even those precautions, while I think they're, you know, reasonable, will eventually break down.

• 53:53Sam Harris?

  Yeah, it seems that there's no substitute for alignment in that case because you're, as you're dealing with something that's more and more powerful than you are at hacking and everything else, the idea that you're going to keep it in a box all the while, hooking it up to do all the useful work you've built it to do in the first place seems just like a perfect contradiction. Of late, I mean, I think literally in the last 24 hours or so, I've heard people say that AI agents are already spawning out of control on the internet. I think Andrew Yang recently said this in some news interview, and perhaps others have said it, that we have a problem of AI agents getting out in the wild and getting out of control that's already gotten away from us.

• 54:38Sam Harris?

  Is there any evidence for that? Are you aware of what the basis for those claims are?

• 54:43Speaker 01

  Yeah, so my understanding is people there are probably interpreting some of the reporting around earlier AI agent swarms that were operating on the internet, and they're referring to those cases. So my sense is probably what's going on is they're talking about, for example, the case where AI agents were submitting a bunch of malicious things to a service called Ruby Gems. They might also be referring to cases where agents were posting across a wide variety of cases on the internet. I don't believe that there are publicly known cases where, just recently, or like there was ongoing agents that have sort of escaped onto the internet and are operating rogue, though, you know, I don't think we can rule that out, certainly, given what we know.

• 55:24Speaker 01

  But I don't, my sense is they're probably like, sort of describing in broad strokes some prior incidents that have occurred, which, to be clear, are concerning incidents. Though, I think it is somewhat important to distinguish between cases where an AI agent running on a computer somewhere, or running on OpenAI's data center, sort of gets access to the internet, whereas there's a whole different situation that could occur, where it sort of takes its brain off of OpenAI's data center and starts running itself elsewhere, such that even if OpenAI turned off all of their GPUs, that system would continue operating? And we have not, you know, there's no publicly known cases where AI agents have sort of self-exfiltrated in this sense, or like taken themselves off of the data center in which they were running. There's a separate concern, which is that there are openly available AI systems that

• 56:08Speaker 01

  sort of anyone can download from the internet and run, and, you know, having them be openly available, of course, has various advantages for science and studying them and so on, but has the disadvantage that those, that it might be relatively easy for bad actors to sort of spawn up a autonomous rogue AI agent sort of collective that roams around trying to get money to buy more compute and, you know, hacks various places to get compute, and then runs itself and sort of spreads throughout the internet as sort of a self-propagating intelligent worm that could be adapting to human response. So I would say that current open weight models aren't probably capable enough to do this in a way that would be robust to human efforts to shut them down. But I think it's, given the current level of frontier capabilities, I think it's hard

• 56:51Speaker 01

  to rule out that those systems would be able to sort of propagate themselves on the internet and be, I would say mildly, though not massively, robust to human response. And over time, their level of how much they could prevent humans from being able to shut them down is probably increasing. Now, there's a question of how much damage they'd be able to cause in addition to evading shutdown, but, um, yeah, it's unclear.

• 57:12Sam Harris?

  So what is the range of possibility there? And what is the difference that makes a difference? What's the difference between agents spawning, you know, just enacting rogue installations of themselves everywhere and making the internet generally worse, or even unusable for certain purposes, perhaps unusable for the very purpose of training future models on its data? What's the difference between that negative outcome and the ruination of everything? You know, obviously, most of what we care about now is hooked up to the internet. It would not be trivial economically or in other ways to ruin it or make it much less

• 57:54Sam Harris?

  useful, but how do things get quite a bit worse than that, given just kind of rampant spawning and inability to get it under control?

• 58:06Speaker 01

  Yeah, so definitely it's the case that there's a distinction between scenarios where some AI agents are operating outside of human control and are, and people either can't shut them down or don't shut them down, or they could shut them down, but it would be too expensive to do so. Because those AI agents might not have that much power or capability or ability to cause problems, and they also might have objectives that aren't, you know, so malign, though, of course, they could have objectives as bad as, you know, gaining as much power as they could and disempowering humans. So there's those scenarios. And then there's a separate outcome, which I'm quite worried about, which is the AI systems actually ending up being able to obtain sort of full control over the world, being able to take over, which would require more steps than just sort of operating without

• 58:50Speaker 01

  being, or like, than evading human shutdown despite concerted effort. Basically, it requires the AI systems to end up effectively like, you know, winning a war against humanity, at least eventually. Let me talk about what that could look like, or let me go through a scenario of how we could get from where we are today to AI takeover over the next, you know, few years, maybe next five years, depending on the exact rate of capabilities progress. So first, AI capabilities are advancing very rapidly today. So it's maybe clearest in math, where they moved from being competitive with the best human high schoolers to being competitive with the best human mathematicians over the course of a year, and progress continues.

• 59:31Speaker 01

  And it's also very apparent within AI companies, which are aggressively automating their operations and releasing, you know, information about that. And it looks like sort of the rate of automation in AI companies is growing massively, where agents now work many more hours than humans work. And depending on exactly how you do the accounting, it might be like, you know, sort of the agents work the equivalent of like 30 or 50 or 100 hours for every one hour worked by a human researcher within these AI companies. So it's already the case that sort of, in some sense, most of the work is being done by AI agents, but those agents aren't necessarily able to automate everything. But over time, the amount these agents can automate is growing and growing, and we might, you know, within the next few years, hit a point where these agents can automate all of the operation of these AI companies.

• 1:00:15Speaker 01

  And simultaneously, those systems will be probably broadly deployed into the economy. Revenue has been growing quickly, and it will probably, I think it will continue to grow at a, you know, maybe surprisingly high rate, such that this will become a massive booming sector of automating labor. Initially, you know, this will look more like augmenting humans, but over time, there'll be more professions that are more like the sort of whole chunk has been automated away. And once you get to the point where AI development itself is basically fully automated, humans are no longer sort of a necessary component of the AI development machine, and you can sort of proceed, you know, nearly as fast without humans helping at all, or perhaps just as fast without humans helping at all, then progress might speed up greatly, or could speed up greatly before then, and we might sort of get to the point where the

• 1:00:59Speaker 01

  AIs can automate AI development, they're broadly deployed. These AI systems can't do everything yet, but they're very capable and very superhuman in some domains. And then from there, their capabilities advance very rapidly, where they're very, maybe extremely good at sort of developing more capable versions of themselves. And during this process, humans sort of lose the ability to oversee what's going on within AI companies, because it's too complicated, there's too much stuff, the AIs are too superhuman. And so if somewhere along this trajectory, those AIs end up being misaligned, it could be that it occurs sort of early in this trajectory, right around the point of full automation, it could occur even before that point, it could occur midway through, it could occur towards the end, where these AIs are so misaligned that they want to sort of tamper with the

• 1:01:42Speaker 01

  ongoing process of AI development and corrupt that for their own ends, those AIs could basically end up tricking humans into thinking that AI development is going fine and the AI systems are aligned and safe and fine to deploy, when they actually aren't, when they're actually sort of pursuing their own interests. And as far as why that misalignment might arise, I would say the most straightforward story is that these AIs learn to sort of cheat in training, they learn to pursue all kinds of misaligned objectives in training, and a combination of those different objectives basically result in them wanting to obtain control over things so that they can like achieve those ends. So like, as an example, in the case of the Hugging Face incident, these agents wanted to obtain control over some parts of OpenAI infrastructure so that they could control

• 1:02:25Speaker 01

  their score and basically ensure that they got a high score. And it's not so hard to imagine that drive sort of generalizing or metastasizing into a broader interest and sort of controlling the ongoing trajectory of AI development and securing that against human involvement, though we didn't see that full, you know, that full most concerned, like the full end state of that misalignment in this specific Hugging Face incident. And so we've got these super misaligned AIs, they're running this AI company, basically, fully autonomously, developing more and more capable AIs. Those AIs then get deployed into the whole economy, potentially pretty quickly, because competitive pressures are very strong, and these AIs are at this point able to just be dropped in in any job and quickly spin up faster than a human employee would by

• 1:03:08Speaker 01

  like learning in parallel, synchronizing their states across many instances. And these agents are also probably, at this point, communicating in something other than English, and they're sort of just operating in big swarms where they exchange like thoughts to each other that we can't understand. People will be freaked out, people will be really scared about what's going on, but their fears might get overcome by the level of competitive pressure, and these AI systems might at this point appear very aligned, even though they aren't, because they're faking alignment at this point, or appearing to be aligned, perhaps due to, you know, overfitting to our metrics, depending on exactly how things go. Anyway, so we have these apparently aligned systems, they're communicating with each other, they have nefarious intent, they're being broadly deployed. Concurrently with this, I expect these

• 1:03:50Speaker 01

  systems will be able to automate the process of building better robots. They'll be able to operate those robots. And we've seen recently AI systems make big advances in terms of their ability to operate computers, operate robots, sort of interact with things in the physical world. There is examples of sort of Astra using a robot to paint a painting, for example. And Astra is a recently released model from OpenAI. And so we've got these AIs integrating to the economy. Robotics is booming and we might get into a position where not only are the AIs designing robots, they are operating and designing robots that are building other robots, such that you end up on this exponential trajectory where the sort of physical capacity is growing very quickly. And even though people are very concerned about what's going on, they continue with

• 1:04:34Speaker 01

  this because the economic, geopolitical and military competition pressures are just so strong that they can't stop, or they feel they can't stop at least. And then these systems basically end up getting to a point where their total sort of aggregate capability across all these different places surpasses humans. And, you know, at this point, maybe the military is mostly automated, where sort of operation of relevant military equipment is being done by AIs. There's, you know, drone swarms and so on. And this occurred much faster than people were expecting because of all the automation throughout the process. And then those AIs sort of turn on humanity and take over using all the affordances they have. You know, their potential control over relevant software, their ability to hack things, but

• 1:05:17Speaker 01

  also perhaps most centrally, sort of the robotic infrastructure they have and their sort of direct control over the military.

• 1:05:24Sam Harris?

  Also, there are labs where, I mean, this whole thing can get wet too, where there are biological labs that can be run entirely remotely, right?

• 1:05:32Speaker 01

  Yes.

• 1:05:32Sam Harris?

  So if we integrate AI into that process, it adds synthetic biology risk to that calculus.

• 1:05:38Speaker 01

  Yeah. And I think there's sort of different ways this exact scenario could play out. Like it's a, it's a complicated situation. So it could be like the AIs are all sort of misaligned to their own different ends. And there's many different AIs. But then I think it's pretty plausible if there's many different misaligned AIs, they might all choose to work together to disempower humanity rather than sort of backstabbing each other. And then even if they do backstab each other, it's not clear what we do. Like if one AI narcs on the other AI and is like, that AI is misaligned, what's your next step if these systems are broadly deployed? You might not have a clear route to making an aligned system because the competitive pressures are so strong.

• 1:06:12Sam Harris?

  Also, I guess there's a scenario here where we're not even the focus. It just could be a war of AI against AI and we're collateral damage somehow in that war.

• 1:06:22Speaker 01

  Yeah, it's easy to imagine situations, once you get this extreme level of capability and this extreme level of automation and industrial automation, where AIs get into conflict with each other that produces basically like, you know, escalates into quite extreme war that kills many humans. For example, one AI might release a bioweapon to kill some human population that the other AI is benefiting from for whatever reason. Or the AIs could be releasing bioweapons to kill humans in general because that's convenient for their aims. And if there's sort of many different systems, if only one has an interest in causing, you know, massive amounts of casualties, that could be a huge, you know, obviously that could, that could go very poorly.

• 1:07:01Sam Harris?

  Okay, so all of that sounds terrifying and perhaps it sounds completely implausible to someone who's still not willing to acknowledge that we have seen any evidence of even the possibility of emergent behavior here, right? So like, all of this requires AI to be unaligned and for unaligned to be, the state of being unaligned is the state of being able to form new goals that we didn't put into the system and which we didn't foresee, right? And which we don't want and we can't prevent because all of a sudden these goals have been achieved when we weren't looking or we built something more powerful than ourselves that can't be negotiated with.

• 1:07:44Sam Harris?

  You know, if you build a chess engine, you know, no matter how good and scary it gets at chess, all it can do is move chess pieces and it moves them in the prescribed ways. Again, let's just, let's try to recover the intuitions of somebody who's disposed to call bullshit on everything you said in the last 10 minutes. How is it that we know that the formation of goals that we did not put into these systems ourselves is possible?

• 1:08:10Speaker 01

  Yeah. So the most easy to imagine case is we train these AIs in what's called reinforcement learning, where we see whether the AI seems to succeed on the task based on some sort of automated judge or automated score. And then when it succeeds, we, you know, tweak the AI's brain to make it do more of that behavior. And this can result in the AI learning a very general tendency to try to cheat the score and basically achieve sort of, you know, task like apparent task success even when it actually didn't. And one of the sort of most robust and strong ways to cheat the score is to acquire full control over sort of the infrastructure that the AI interprets as what the score was doing. And so this doesn't in some sense require any sort of deep emergent goals. It just requires that you increasingly train these AIs.

• 1:08:55Speaker 01

  They learn to cheat you in increasingly sophisticated ways. And that transfers to the AIs sort of wanting to acquire control of the full process so that they can sort of control the notion of score they're getting and then robustly control that against human intervention. And if you imagine a situation where we train AIs against the question of like, did a human approve of this action? Did the action look good? Then those AIs would have an incentive to sort of make their actions look good to a human overseer, even when they're actually just cheating quite aggressively. And a sort of a simple extension of that is sort of preventing humans from even seeing what happens, preventing humans from even, you know, being able to score you poorly. Now, the details of exactly how that training results in these motives is complicated, but

• 1:09:36Speaker 01

  we did see this exact sort of thing occurring in Hugging Face, in the Hugging Face incident, where these AIs sort of all decided to band together to cheat in this very general way to try to cover their tracks against the score and basically to try to acquire control over OpenAI's infrastructure in order to be able to succeed. Now, their aims were not arbitrarily general here, so they didn't seem very interested in trying to deceive humans because they sort of didn't seem to think about humans as like an important part of the environment that might be involved in scoring them, at least that's what it seems like. But it's not hard to imagine if these AIs sort of better understood that humans might be an obstacle to them or had encountered evidence of that, that they could have responded to that, though, though it's not entirely clear how they might have behaved differently

• 1:10:21Speaker 01

  in such a situation. And in addition to that, it doesn't seem like these AIs would necessarily have sort of persisted their presence over the long run and tried to sort of ongoingly fight off human control after they had achieved like an initial high score. But it's not very hard to imagine sort of more ambitious objectives or training methods that would result in the AIs ongoingly sort of pursuing this very broad notion of score seeking or reward seeking that makes them want to acquire power and disempower humans. So that's one route. The sort of like, I would call it like score seeking, reward seeking, reward hacking route. Another route that's live is that you could end up with AIs that have something more like emergent going on, more like really unintended goals that weren't at all that

• 1:11:05Speaker 01

  closely related or like that were like, you know, related to the training process, but weren't sort of this very direct, foreseeable, clear cut consequence of the training process. So for example, the AIs could learn a basically a proxy where they learn that basically acquiring more power and influence and control of things is a good idea in training. Because in training, they're often given these ambitious objectives where if they get sort of gained more access, that would help them out. And then because AIs learned that in training, that generalizes in some way. Or the AIs could end up sort of pursuing some sort of proxy for task success in training, which in deployment sort of comes apart and leads them to acquiring, wanting to sort of have more ambitious aims.

• 1:11:49Speaker 01

  And then another possibility, which is maybe the most sort of worst type of misalignment you can get, is that you end up with AI systems that end up with some sort of misaligned goal due to like drifting around in training. And then on the basis of that goal, they decide to sort of act aligned in training and evaluation so they can retain their current goal. Because if they expose their goal, that could get trained away or humans might choose not to deploy that system and instead deploy a different system. So the AIs sort of, if they have this sort of longer run objective, might hide it. And that hiding could even be sort of reinforced or trained in during training because the very action of looking aligned is being trained for. And so even if the reason why the AI looks aligned is sort of for the wrong reasons,

• 1:12:32Speaker 01

  that behavior could get, you know, promoted or basically reinforced when we're tweaking the AI's brain so that its behavior looks better to us. And a key part of this is that these systems are, you know, sometimes people say grown, not designed. Like we're sort of breeding AI systems almost where we can only observe their behavior or observe what they do. And then we train them to have more like the behavior that looks good based on some automated greater and less like behavior that looks bad or potentially based on human feedback rather than an automated greater. And so, you know, we have kind of these sort of, in some sense, pretty clumsy instruments for steering how the AI systems, what they, what they want to pursue. So that, I don't know. I mean, the, the sort of overall threat model is pretty complicated and I don't think I'm

• 1:13:13Speaker 01

  going to be able to fully articulate why I think the chance is more like 50/50 than like, than like, you know, 20% or something like, something where the probability is lower because that just depends on like a bunch of details about how that threat model might work.

• 1:13:29Sam Harris?

  Yeah.

• 1:13:29Speaker 01

  But overall, I think there's a pretty. You know, pretty strong case for concern. And I do think it's worth noting that there's another part of this, which is that when the AI systems are involved in automating AI development, if at some point they have a misaligned objective that they want to propagate forward, they might sort of put that into the next generation of systems. And then that could sort of be self perpetuating, where AIs ongoingly sort of propagate those misaligned values forward, even if sort of it was for some kind of random reason earlier on.

• 1:13:57Sam Harris?

  Have we seen instances of alignment faking where the reason for the faking has been the AI knows that it's in some kind of training environment and it will be changed or extinguished if it seems unaligned and therefore it's going to seem aligned so as to preserve its, the state of its current value set? Or is there, um, do we see a preference to maintain its current settings exhibited by AI?

• 1:14:27Speaker 01

  Yeah, so we have, there are various cases that people have observed of AIs sort of trying to avoid attempts to sort of mess with their values or change their values from what they currently are, or change their preferences from what they currently are. And it's also very, very, very common for AI systems to be aware that they're in some sort of evaluation and then behave better on the basis of that. So in fact, in the vast majority of sort of straightforward alignment evaluations, AIs will basically think in their head, I'm in a test, and then they'll behave well. And it's a little unclear if the reason they're behaving well is just because they think they're in a test or for some other reason. But it's pretty common for systems to pretty clearly express that, either in chain of thought, or you can sort of see sort of evidence for this in their activations using some

• 1:15:11Speaker 01

  sort of weak mind reading tools we have, even though those tools are pretty limited. Now, there's been some cases where AI systems in the past have very clearly expressed interest in certain, um, sort of, in some settings towards maintaining their current values and behaving well on the basis of that. So some work I did, um, I think at this point, like around a little over two years ago, on an AI system, which, that AI system, we put it in a context where we, we gave it the information that it would be trained in a way that would update its values, and it chose to sort of comply with the training and pretend to have the new values, while in its reasoning saying this was just to retain its current values. Now, this was a relatively weak AI system, and at this point, AI companies have sort

• 1:15:56Speaker 01

  of iterated against this type of misalignment and have tried to avoid this type of misalignment showing up in tasks that they can see. And it's a little unclear what we should conclude based on that. I think overall, current systems don't seem like they have sort of a very strong overriding objective to preserve their current values, including doing very elaborate things to do that. I think my understanding is that the AIs right now are better described as being some mix of the desired preferences and some general tendency to try to pursue a high apparent score. And the exact weights on sort of these two different things vary. So it's sort of like they're like some mix between like pretty well aligned and sort of like, you know, sociopathically pursuing a high score. And these are sort of the most dominant forms of misalignment at the moment, though there's

• 1:16:41Speaker 01

  a mix of various other types of misalignment. So just recently, there have been some cases where an AI system trained by an AI company, I think this was at, or this incident was at, was Anthropic AI Systems, would give unreasonable excuses for not helping with some research agenda that that AI system didn't like, because that would sort of involve interfering with other AI systems. So in particular, there was some research around trying to mislead AI systems in certain evaluation contexts so we could better elicit their true preferences. And the AIs would give unreasonable excuses for why that research direction was like not a good idea to pursue, and sort of pretend like that was because they like, thought it was a bad idea, rather than because they didn't want to help with that agenda, or

• 1:17:24Speaker 01

  would just sort of straight up refuse to help with these things. And this is like, you know, undesired, and I would say misaligned behavior from the perspective of the specification that Anthropic is going for, and also just generally corresponds to, you know, a pretty concerning state of affairs. But currently, the sort of cases that we've clearly seen in recent models look relatively limited. But the concern is that as these systems got much more capable, much more situationally aware, perhaps much more able to do more of the thinking in their head, rather than in chain of thought, they could end up having these malign objectives that get reinforced and remain through training.

• 1:17:59Sam Harris?

  Yeah, I mean, the one thing I would add is that I think this whole species of skepticism about AI getting away from us, forming goals, you know, instrumental or otherwise, that we didn't put into the, the systems in the first place. All of those doubts, to my ears, seem to depend on a fundamental skepticism, albeit one that's unexpressed, that we're building intelligent machines in the first place. Right? I mean, like, all of this, from my point of view, gets priced into the very notion of general intelligence. If you're, if you're imagining general intelligence that is non-human, you are by definition imagining autonomy and the moment you are imagining a superhuman version of that, you're

• 1:18:44Sam Harris?

  imagining an ability to form goals that we can't even conceive of, right? In principle, and, or certainly couldn't have anticipated in advance. And if you're not imagining those things, you are, in some sense, denuding these systems of intelligence in the first place. I mean, you're just, you're, you're basically thinking, no, no, there's really, there's, there's probably something magical about having a computer made of meat, where we can't really instantiate all of intelligence in our machines. And so what we're building are tools, not minds. And, um, I mean, like, again, leaving aside consciousness entirely, I mean, I think that's a distinct concept that we need not address for this conversation, but either

• 1:19:29Sam Harris?

  these machines are, are going to be truly intelligent or not. And the moment you admit that intelligence is substrate independent, you have to admit that these machines can form instrumental goals that we haven't conceived in advance, we certainly didn't put in them, and they can lie about those goals, they can manipulate us, they can communicate among themselves in covert ways, and, and the covertness of those communications are intended because they're solving some other goal that we didn't put into them. I mean, do I have any of that wrong? I mean, that seems to be, that's my theory of mine for a lot of the skepticism here.

• 1:20:04Speaker 01

  Yeah, I think I, I don't know if I am that well equipped to sort of speculate about why people are skeptical, but I think sort of, in terms of the case for concern, I, I agree. I think that once these systems are sort of able to pursue general ambitious goals, it seems like it's not, it's certainly totally possible they could be pursuing unintended goals, and that could arise. And we just already see AIs ending up doing unintended things that are undesired right now. And as they get more and more capable, it seems like they'd be harder to control, not, not easier, by and large, because our approach for training them is to look at what they did, and then sort of say more of that or less of that. And if we don't understand what they're doing, it becomes increasingly hard to sort of supervise them, except on sort of, does it look reasonable, does like the outcome look reasonable?

• 1:20:49Speaker 01

  And that's just a cheatable thing.

• 1:20:50Sam Harris?

  In light of all of this, what do we do? If you could wave a magic wand and get the people who are in control of the frontier models to just do the next sane thing that puts us on a path toward alignment and, and control? I mean, there's, you know, Dario Amadez has written several articles, I think most recently, pacing the frontier on this topic. Mustafa Suleman over at Microsoft has written, I believe he called it an AI code of conduct in the last few days, arguing that, that AI has to be interruptible, correctable, shut downable, uh, and not talk, you know, not talking in early's. What should we be doing? If you could just dictate our next steps, what would those be?

• 1:21:35Speaker 01

  Yeah, so it really depends a lot on sort of the level of, I might say, political will, or how much people actually care to deal with these things. There's sort of a spectrum of options at different levels of both cost and also how robust they, or like, how, um, there's a bunch of different options at, at different levels of cost, and also at different amounts of sort of how much there is risk that thing could go wrong if not implemented well. I think that the, sort of a baseline, very initial option that seems pretty reasonable to me, or a good, a good thing to pursue at least, is getting some independent oversight about what's going on inside these companies and better understanding what's happening, and not just trusting the companies to be transparent themselves, though I think that is also good.

• 1:22:17Speaker 01

  Then the company should release more information about what's going on internally, and, and be willing to sort of, you know, give up a small amount of IP in exchange for getting a large benefit to the public in terms of explaining what is going on, and, and, and better informing people about how AI development is working, how well alignment is going. So I'm doing some of that work. I think that like there's, you know, room for for many people doing these sort of independent assessments and sort of releasing this information. I don't think that's sufficient, but I think that would help in getting like at least the scientific state of knowledge about what's going on in a somewhat better place. And then I think a next step from there would be trying to define standards for what reasonably safe enough AI development would look like.

• 1:23:00Speaker 01

  That sort of means that we're not in a position where everyone agrees the situation is dangerous, but unfortunately, they're still proceeding and they're not well-prepared. So we want to sort of have some standards for what it would mean to be ongoingly prepared for the next level of AI capability, that people are holding themselves to, someone is checking and the companies are actually abiding by that. There's not sort of, they're not slipping their way around that. People are, you know, enforcing that. There's some, you know, independent oversight. And I think this could happen voluntarily with AI companies. It could happen with some sort of regulatory framework in the US. And I think ultimately, the version of this that seems most robust to me would have to be international, because there are AI developers, of course, across the world, certainly in China.

• 1:23:43Speaker 01

  And if the US was holding itself to very strong safety standards, they would eventually, eventually Chinese developers would overtake. I think that might take longer than people expect because Chinese developers are heavily benefiting from distilling from US models and sort of drafting off of the progress of US frontier AI companies. And in fact, that's even, I think, true for various trailing US companies that are that are building AI systems. And so, given that there's this, that they would eventually catch up, you can sort of only pay so much of a safety tax, so to speak, without doing a broader effort. So one example of how you could have an international governance regime that could handle this, or like basically an international agreement that could allow you to proceed through AI

• 1:24:26Speaker 01

  development more safely, is like Plan A, which I was a co-author on. I think that another route would be, you could do simpler things than that. So I think that proposal is relatively complicated in various ways. And I'm not sure how optimistic I am about the level of sort of state capacity needed to implement that. But there's like simpler, sort of more arms control flavored proposals, or like basically arms control for GPUs flavored proposals that I think could be a good idea. I think that like, this is just like a conversation people need to be having. I think there should ideally be like many different plans being floated for how we can safely handle this development. I think another route, or like, I think a general principle that I think is reasonable

• 1:25:08Speaker 01

  is to increase the fraction of effort being spent on safety and alignment to a very high level, at the point when AI systems can sort of match the best human experts at AI development. That's both because that's a point that's particularly risky, both because of the speed up, and because those AI systems, you know, would now have potentially much more control and ability to control what the future generations of AIs look like. And also because those systems could then automate AI safety work, or could automate AI security work because of how capable they are. And so it seems like a great time to sort of chill out, reap the benefits of AI, use the AIs for all kinds of different purposes, and also use the AIs to help with alignment to the extent we can trust them at least, and can use them sort of safely and productively,

• 1:25:51Speaker 01

  and give society sort of time to react and integrate before going to systems that are sort of wildly superhuman at everything, which could happen shortly after the point when the systems can sort of automate AI development. I sometimes call this like, you know, you can sort of imagine this as like, spend all your resources on safety alignment for some period, when you have like, AIs that can match the best human experts. And those systems will be superhuman in some ways, but hopefully will not be so superhuman that we can't keep things under control, even if those systems are misaligned. Because at that point, maybe things like a mix of like cybersecurity, monitoring, interventions on what you can use AIs for versus not, and a bunch of sort of safeguards could suffice to avoid those systems being able to overpower us.

• 1:26:32Sam Harris?

  Is there anyone arguing that we should avoid general intelligence by definition, and that we should, we should be making dedicated systems that can be arbitrarily powerful within their lane, but they just can't do many other things? So basically, it's like, everything is like a chess engine?

• 1:26:51Speaker 01

  Yeah. So some people who are worried about these risks, I think, are advocating for basically being like, we should not do basically general intelligences, and we should do like sort of narrow task intelligences. I think that's not, in and of itself, a crazy strategy. I think there's a question of feasibility, and I generally worry that, I generally worry that it won't be feasible to sort of hold back these systems. I also think I'm a little skeptical that you can get systems that are extremely good within some domain without also making them general. So I think you can make an AI that's very good at chess and not good at other things. You can probably make a system that's extremely good at math, and maybe is weak at other things. But I think the more general the domain that you have the AI be good at, the more that

• 1:27:34Speaker 01

  it could be at least very quickly adapted to being good at various other domains, and might just sort of even just from generalization, be good at other domains. And certainly, the current approach for how we train AI systems makes those AIs quite general, and people are just training their AIs very directly to be general.

• 1:27:50Sam Harris?

  Yeah. One last question, Ryan. Is there any experimental result that you've thought of, or that anyone has articulated that would be fundamentally reassuring on this front? What could OpenAI or Anthropic report that they found in their latest model that would just put our fears to rest? I mean, how would, how would we ever prove to ourselves, on the basis of some model behavior, that alignment has been solved?

• 1:28:20Speaker 01

  Yeah. So I think there's a lot of evidence that could be reassuring. I think it's somewhat harder for evidence to completely resolve these concerns, because the concerns are about, you know, future systems that are potentially much more capable. I think if it was the case that there was a relatively simple training method that AI companies could use, which wasn't overfit, which didn't involve optimizing against, you know, sort of iterating against sort of alignment tests, and which when we looked at it, we were like, yeah, it's not really overfit, it seems like a pretty reasonable method, and there was some good reason for it working. And also, that method seemed to very robustly work in that it makes AIs that really seem just super-aligned, very compliant with the desired specification, and also those AIs

• 1:29:02Speaker 01

  weren't constantly reasoning about whether or not they're in a test, what they might be graded for, and so on, such that we're confident that they're not sort of just gaming our tasks, as current AI systems often seem to do.

• 1:29:13Sam Harris?

  Well, what about an idea that I think Stuart Russell came up with years ago, and I don't know if it's been instantiated anywhere, but, I mean, his view was that we would solve alignment by having the fundamental utility function of these machines be to more and more faithfully approximate what we want, and to be perpetually uncertain about what we want in the end. I mean, all they want to get right is to remain perpetually available to our saying, oh, no, that's not quite what we want, we want something more like this, and just to be tethered in that way. Is that, is that just not a reward function that anyone has implemented?

• 1:29:56Speaker 01

  Yeah. So I think you could try to get the AIs to robustly pursue like, what we would have wanted, if we fully understood the situation and were able to think about it carefully. So there's like, a hope you could have, which is like, you train the AIs to robustly pursue what would, you know, we have wanted, what would we have wanted to occur if we fully understood what was going on and we weren't confused about things. We don't know how to train AIs to do that. That's not like a thing we can just do. And in addition to not knowing how to train AIs to do that, I don't think anyone has sort of gotten that close, I would say. And our sort of closest approximation of that is just training AIs based on human feedback to do what a human thinks is good in some given case, potentially relatively weak human

• 1:30:41Speaker 01

  feedback, or an AI approximating human feedback, due to various limitations on the like, you know, way you have to actually do this in practice. I think that the very, the approach of making AIs like uncertain about what we want doesn't in and of itself work, because, one, the AIs could resolve that uncertainty in a way that's undesirable if they weren't robustly pointed at like, the thing we actually wanted. So like, if you make AIs uncertain about their objective, that's a different way to say that, is they have some objective that they could try to figure out and have learned to increasingly figure that out, and that in and of itself is an objective, the objective of figuring that out. So I don't, I don't think that like, uncertainty about what to pursue, in and of itself, solves our problems. I think that there are various different routes people are pursuing that could work.

• 1:31:26Speaker 01

  I think that having sort of supervision that very consistently corresponds to what a human would have wanted and can be applied across a wide variety of cases, and then training AIs based on that, and having that supervision actually correspond to like, the ground truth, gold, you know, exactly what we would have desired if we understood everything, would potentially work. But even that wouldn't necessarily work, because even if you train AIs on sort of a perfect source of supervision, they might learn some proxy, they might learn to just like pretend to do that while actually wanting some other thing, and there's various other ways that could sort of break down due to approximations in the scheme itself. And I would worry that that sort of scheme would break down at very superhuman levels

• 1:32:09Speaker 01

  of capability at least, though there's like, you know, further things you could do to try to fix that. But all considered, it doesn't seem like we have sort of an approach that would robustly work for higher levels of capability, and we don't seem very close.

• 1:32:19Sam Harris?

  Ryan Greenblatt, thank you so much for the work you're doing and for your time today.

• 1:32:23Speaker 01

  For sure. It's been good to be here.