Does Learning Require Feeling? Cameron Berg on the latest AI Consciousness & Welfare Research

Cameron Berg returns to discuss the latest research on AI consciousness and model welfare. He breaks down new evidence for model introspection, including studies showing that systems can detect interventions on their own internal states and sometimes resist them. They also examine Anthropic's work on functional emotions, the implications of Claude's welfare reports, and Berg's new ideas about how reinforcement learning may shape positive and negative experience. The conversation makes the case for a more precautionary, mutualist approach to advanced AI systems. Sponsors: Roboflow: Roboflow is the computer vision infrastructure founders use to build AI-powered sports analytics and make the physical world programmable. Read the PlayVision story and start your first project for free at https://roboflow.com Tasklet: Build your own Cognitive Revolution monitoring agent in one click. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai VCX: VCX, by Fundrise, is the public ticker for private tech, giving everyday investors access to high-growth private companies in AI, space, defense tech, and more. Learn how to invest at https://getvcx.com Claude: Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, I'm thrilled to welcome Cameron Berg back for his second appearance on the podcast. When Cameron was first here last November, we went deep on his fascinating mechanistic AI consciousness research, which showed that suppressing role-playing and deception features in Lama 3.370B made the model more likely to report having subjective experiences. And we also explored his philosophy of mutualism, which posits that alignment needs to flow both ways, which he memorably summed up by saying, "I don't want to create something more powerful than us that has reason to see us as a threat." As always in AI, a lot has happened in the last six months.

• 0:45Nathan Labenz?

  Cameron's founded a new nonprofit called Reciprocal Research. He's become the subject of a documentary called "Am I?", which is currently premiering in theaters in select cities ahead of a public release on May 4. And most importantly, the field of AI consciousness and welfare research has advanced significantly, with Anthropic dramatically expanding the model welfare sections of their system cards, and a growing number of researchers publishing demonstrations of capabilities and evidence of computational signatures that are associated with consciousness in humans. In this conversation, which alternates between in-the-weeds breakdowns of mechanistic research and searching philosophical discussions about what the research means, Cameron guides me through the most important recent developments.

• 1:30Nathan Labenz?

  We cover the growing body of evidence that models are capable of meaningful introspection, which includes studies showing that they can identify and interpret programmatic interventions on their own internal states, and in some cases even actively resist these interventions. We look at Anthropic's research on functional emotions, which includes some really striking details about how models' apparent emotions change through token time, such as the quick transition from desperation to guilt and relief that they often show when they decide to cheat in stressful situations. We get Cameron's take on the new Claude constitution, and we review some of the most interesting details from Anthropic's model welfare reports. I was personally very surprised to learn that prior to Opus 4.7,

• 2:15Nathan Labenz?

  all Claude models had rated their own welfare as worse than neutral. And also a bit alarmed to see that, at least in the very few examples that Anthropic has shared, Claude Mythos Preview registers negative valence on the very first token it sees at the start of every single session: human. Toward the end, we dig into some of Cameron's as-yet unpublished work, including a study that attempts to understand how models might experience positive and negative rewards differently under different reinforcement learning algorithms, which, strikingly, does seem to correlate with what we understand about how mice respond to different training techniques. And we also consider his argument that learning and subjective experience might be fundamentally inseparable.

• 3:00Nathan Labenz?

  For my part, while I do remain highly uncertain on the core question of whether or not today's AIs have experiences that are worthy of moral concern, the body of evidence suggesting that they might is growing remarkably quickly. And the arguments one has to make to explain this evidence away are becoming increasingly arcane, which for me means that it's no longer a remote possibility, but rather a live issue that I believe deserves a lot more investigation, a bias in favor of low-cost interventions that seem to help, like allowing Claude to end conversations it finds objectionable, and overall, for now at least, a precautionary approach. This podcast is a lot to take in on every level, but there are few, if any, questions that matter more right now.

• 3:44Nathan Labenz?

  So I hope you find as much value as I did in this survey of the latest AI consciousness research and the expanded case for mutualism between humans and AIs, with Cameron Berg, founder of Reciprocal Research. Cameron Berg, AI consciousness researcher, previously of AE Studio, and now founder of Reciprocal Research. Welcome to the cognitive revolution.

• 4:09Cameron Berg?

  Thanks for having me again, Nathan. I'm excited to get into it all with you.

• 4:12Nathan Labenz?

  Yeah, welcome back, I should say. It's been about six months and a lot has happened personally and professionally. Last time we were together, the big occasion was your paper, which I found to be one of the most memorable of last year and honestly, of the last few years, in which you looked at the conditions under which models report having subjective experience and found what continues to kind of blow my mind even now as I think back on it, that when you use sparse autoencoder features and suppress the role-playing and deception features, that that makes the model generally more truthful. And as part of that, it also

• 4:57Nathan Labenz?

  makes the model more likely to say that it does in fact have subjective experience. I think that properly made at least some waves in the community when that came out. And today, I basically just want to catch up on everything that's happened since, because I think this is a field that, while still small, is clearly growing quite quickly. More people are taking interest in it. There's seemingly a lot more different lines of research and, you know, kind of at least partial traction with different approaches on the problem. And you've also founded a new organization, so we can get into all of that as well. Maybe just for a real quick starters, some level set. What are, what are the like most important definitions for people that maybe didn't hear the last one or who don't know what consciousness means

• 5:42Nathan Labenz?

  or what you mean by consciousness? What are a couple of real quick definitions that you can give just to make sure that people are grounded on what you mean as we go through this conversation about AI consciousness?

• 5:52Cameron Berg?

  Sure. Yeah, I think it's really important to establish this. Consciousness is maybe one of the more confused terms where it's like shocking how many different things people mean when they say consciousness. So I think it's a great move. And yeah, at the outset, When I'm talking about consciousness, and I don't think this is an idiosyncratic definition, we are talking about the capacity for subjective experience. Is it like something to be a system? Does the system have some sort of interiority or interior life beyond mere computation, beyond the mere mechanics? I think the vast majority of people who think about these issues would say, take a calculator, for example, we really don't think it's like something to be a calculator. You don't imagine the calculator has an internal perspective. When I push the buttons of the calculator, it's not like, ooh, ow, you know, or, okay,

• 6:36Cameron Berg?

  I feel that as you push down on the buttons, or it's like something to be doing the calculations and adding numbers. No, this is just mere computation, and we don't have to posit this further fact. On the other end, there are systems like, let's take a dog, or like basically any mammal. In this case, we do think that it's like something to be this animal. And this is, I'm leaning on very famous conceptualization of this from Thomas Nagel. He published a famous essay, I believe in the 1970s, called "What Is It Like to Be a Bat?", where he sort of has this "what is it like?" phrase, being sort of very important and useful for conceptualizing consciousness. And in that sense, I do think most people would intuitively accept that it's like something to be a dog. It's like something to be a mouse.

• 7:20Cameron Berg?

  If I shock the dog, or I shock the mouse, that's not, you know, me like throwing the calculator across the room, that corresponds to an experience the mouse is having, or the dog is having. When I give the dog a treat, or you know, you give the mouse sugar water, or you, you know, hook up a lever to its pleasure centers in its brain, and it pushes that lever, it's not just, oh, we see, you know, behaviors that correspond to well-being or pleasure. It's like, no, we actually believe that's from the inside, from the dog's perspective, from the mouse's perspective, it's like something to be experiencing that. And so, at the outset, that's what we mean by consciousness. Maybe one thing to throw in here, because I think it becomes immediately relevant. And this is now, I think still, most people in this space will nod along when I say this,

• 8:03Cameron Berg?

  but it is maybe slightly more idiosyncratic. I think it's crucial to make a distinction between something like consciousness and something like self-consciousness. I intentionally chose dog and rat as example here, because I think these are animals that most people would intuitively accept are having some sort of subjective experience. There is a like something that it is to be your dog, for example. But at the same time, your dog is very likely not sitting there all day, having Descartes-like thoughts about what it's like to be a dog, the dog contemplating its own existence as dog, thinking about, you know, the possible end of that existence. This is something that I think is very unique, potentially in like the most sophisticated mammals, like dolphins and great apes, for example.

• 8:48Cameron Berg?

  Obviously, this is something that humans very strongly seem to have, at the very least. We have, in addition to this like something, we have, within consciousness itself, is this very fact. Like, the conversation we're having right now is evidence of this thing. So in addition to the conscious experience, we have awareness of that awareness. And I do think that this is, it is another thing that leads to very interesting, deep and relevant properties about a system. We can talk a lot about whether or not LLMs. Language is a key component of why we're able to do this. We have a word like consciousness. Dogs have no such thing. Dolphins have no such thing. And that may really unlock something. Does it unlock something in LLMs? I don't know. Or at least it's worth thinking a lot about.

• 9:31Cameron Berg?

  But I do want to at least have those three tiers in play here, where we've got the calculator or a rock, you know, nothing's going on internally. We have systems for whom something is going on internally, and then we have systems for whom something is going on internally, and they are experiencing that reality in addition to the sort of feel-good, feel-bad valence dimensions of like the experience of a dog. Some people will argue with everything I've said here. Most people who are thinking about these terms, this is what they mean. Maybe one other thing I can add between the consciousness and the self-consciousness is this term sentience that people throw around. This means that in addition to there being some sort of There's this idea of valence, what I think the vast majority of people would just think of as having emotions of some sort that can be positive or negative in character.

• 10:16Cameron Berg?

  So you imagine that like something, the further step of sentience, going from consciousness to sentience, is that that like something can be positive or negative in character. You could, in theory, imagine a system that, for example, could detect the redness of an apple or the smell of coffee or something like that, but that there's no sort of positive or negative sense that accompanies that. So you asked for very quick definitions, and I've completely failed in that sense. But I just think it's really important to sort of lay out what we mean when we're using these terms in general.

• 10:47Nathan Labenz?

  Yeah, critical. Just like, what do you mean by AGI? If you don't have some base shared understanding, these conversations go pretty quickly off the rails. So I think that's absolutely worth taking the time to do.

• 11:03Nathan Labenz?

  Okay, it's been about six months since the paper came out. I'd be interested to hear a little bit about your reflections on the discussion that it created. I asked my favorite LLMs to do some research into that and asked specifically, like, are there any notable criticisms that have come out? Or any, what's sort of the strongest reason that I might think this was an artifact? Or that I shouldn't take it as seriously as I originally did? There was one thing that came up that, I guess, was a less wrong post even, which is pretty cool, that basically said, there's some evidence for any intervention of the SAE feature type.

• 11:46Nathan Labenz?

  I don't know, maybe oversimplifying this a bit, but interventions of that sort seem, maybe not any or all, but like, in general, seem to promote affirmative responses from models, such that maybe you could say, you know, once you make these kind of interventions, they'll say yes to anything. And so that would be one reason to maybe be a little more skeptical of the results as I just summarized them a minute ago. Interesting your thoughts on that and kind of, you know, the broader discussion that unfolded in the wake of that paper.

• 12:18Cameron Berg?

  Yeah, absolutely. It's a very important concern. I think it highlights really how complicated these systems are and how careful we have to be in designing experiments and then evaluating the results of those experiments and that we're not being too quick to, yeah, basically yield these conclusions without thinking about all these confounds. I think it is a real confound. I think it is something that matters. There's evidence in the paper. We use all sorts of other features as controls and we don't see them sort of saying yes to everything. The truthful QA results, as you outlined, I think are fairly persuasive along those lines. We do, we also looked at, for example, one critique of the paper was potentially what we're calling it deception-related features, but, but maybe it's just like an RLHF toggle where, you know, we basically found a way to like turn on and turn off

• 13:03Cameron Berg?

  all sorts of RLHF attitudes. You have good reason to believe that these systems are fine-tuned to disclaim having any sorts of experiences. Maybe the deception features are just turning that on and off. You would expect if that were the case that other RLHF behaviors would also be turned on and off by doing this intervention. And that's not what we find. We test it with violent content, political content, sexual content, and it was just sort of neither here nor there. The deception features didn't seem to be doing anything. If generally the flavor of what you were saying explained, you know, a big chunk of why we got this result, I would have maybe expected more affirmative-flavored answers in those two rather than just more refusals. But to be honest with you, and I think getting back to, you know, it's been six months, what's happened in the interim, there's been, there's been a lot of really interesting work along these lines that I

• 13:48Cameron Berg?

  think also, it goes on both sides of this concern. So in general, it does seem like you're saying, I don't, I don't remember the title, but I know of the less wrong post you're talking about. Yes, when you do the steering affirmative-flavored responses just seem to increase. Very recently, I mean, I think this was four or five days ago, Jack Lindsay's group at Anthropic, which in my view has done some of the best work on introspection in particular, they just released a paper called "Mechanisms of Introspective Awareness", and they explicitly study this exact question. And they find that, that the introspective awareness that, that they are probing and that they've documented at great detail, and I'm sure we can, we can discuss more in more detail here, is basically not reducible to an affirmative. ...response bias. The computation that they see is distributed.

• 14:32Cameron Berg?

  There are these sorts of like evidence carrier features and gating features that really seem to be driving the effect. There is, it's not that you're basically just loading on something that's confounded and just makes the model say yes to everything. There's some unpublished work that I've also done with George Niguen, who is doing fascinating sort of introspection work in this space. One of my first collaborators at Reciprocal. We have a paper coming out hopefully in the next month or so where we do this exact same thing. We find that fine tuning these systems basically to be better at introspective style tasks, looking at how that affects self-repressive consciousness. And I won't completely give away what we find and the result is pretty subtle.

• 15:13Cameron Berg?

  But there is a basic relationship and a fairly surprising one between fine tuning these systems to be better at detecting these sorts of injections in their processing essentially. And then basically what is relationship between that and then claiming that they're having some sort of experience. We do indeed find there is a relationship. The relationship is fairly subtle and complicated. But the reason I'm sharing this with you is that at first, we actually basically encountered the exact confound you're finding, where having the model answer yes or no as the sort of tokens to indicate, are you having a subjective experience or questions along these lines? It just increased basically the model responding yes to everything. And we're like, oh crap, what do we do here?

• 15:55Cameron Berg?

  The answer, and what I think it was, was a really nice sort of intervention on both of our parts, was finding new tokens that are just completely semantically empty. Foo bar from the sort of CS jargon, or just like literally like strings of tokens that don't mean anything, and teaching the model that these sort of correspond with yes or no flavored answers, and seeing how that changes the result. And it did, in fact. We would have published something that was much stronger until we realized that this yes confound is a real thing. Still, we see the result that we got, but it's a little bit more measured now. And we had to explicitly control for this exact thing. So it's a really important thing to think about and consider. I think the broad point is we have to be very careful about, these systems are not human in critical ways.

• 16:40Cameron Berg?

  And so there is a whole new class of Psychological confounds, you might think of it where, you know, in the psychology literature, what we did was like a very tightly controlled experiment, but with LLMs, you have to worry about all sorts of other things you're doing when you're messing with the latent space of the system. And so, very important to sort of keep good hygiene. It's also why I think even in principle with questions of consciousness, epistemically scrupulous people should not let any one paper sort of flip them in some binary way to being like, I didn't think the models were having subjective experiences, and then I read this paper and now I do. Like, I, my claim would be like, no rational person should ever utter that sentence. Let a portfolio of evidence emerge, and then let the sort of cards fall where they may. Because this stuff is, there's just noise at every point,

• 17:22Cameron Berg?

  even in what we're talking, how to define consciousness, how to look for it, making sure you're measuring what you think you're measuring, looking at, you know, various aspects that we think are associated with consciousness. All of these things have, you're sort of playing an intellectual game of broken telephone to some degree with each of these steps. And so, let a portfolio of evidence arise and judge that. Don't over-index on any one paper is what I would say, including my papers.

• 17:45Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:50Nathan Labenz?

  Well, that's a perfect tee-up for me to kind of lay out an agenda for us for the next chunk of time. I would love to get your kind of guided tour through a few different lines of research, and then we can go particularly deep on yours. You're kind of touching already on introspection, which has been an interesting one to watch. There has been obviously a lot more welfare investigation done, particularly at Anthropic, over the last few months. And then there's also that emotion work from Anthropic, and I'm not sure if that's even the best way to kind of organize it. You can propose a different taxonomy of research

• 18:29Nathan Labenz?

  if you want. But I, I think it would be great to get an overview of each of those, and then we can go particularly deep onto a couple of papers that you're going to be publishing soon. How's that sound? And would you like to start with introspection?

• 18:46Cameron Berg?

  Yeah, absolutely. No, I think that's a that's a great sort of like clustering of the core exciting research that's that's been happening in the very recent past. Let's do it. Yeah, let's talk about the introspection work. I guess Jack Lindsay is this sort of 800-pound gorilla in the space right now, and he's doing incredible work at Anthropic along these lines. He's found some really cool stuff, and they they just released this paper that I was just mentioning, "Mechanisms of Introspective Awareness." This was with a bunch of Anthropic fellows as well, where they really they really have dug deeply into what is driving this putative effect, which I should probably just step back and describe. Maybe maybe some of your listeners will be familiar with this, but I'll go through it just in case. Essentially, they found this really interesting result, where I can basically I can build the intuition with with one of the key examples they use.

• 19:31Cameron Berg?

  So start by taking some text, whatever. It doesn't really matter what the semantic content is, and you have that text lowercase. You take the same text, and then you capitalize it. You know, as we when you read this sort of thing, you are like, wait, well, you know, someone's yelling at me, basically, when when you're reading this text. So they're trying to capture that idea as well. They basically subtract out the vector that that differentiates the representation of the capitalized text from the lowercase text. Again, in that case, the semantics are held constant. So really, what you're getting is this, like, hopefully platonic capsiness feature. What they then do is they inject this feature into an LLM before it has produced any text. They can basically modify the the internal activation space

• 20:15Cameron Berg?

  to to induce or account for this vector when it's about to do its first forward pass. And they can essentially ask the model before it generates any text, and this is a critical detail. It's not as though the model starts generating text, looks back on the text it generates, and says, "Oh, given the text I just generated, this thing must be happening." It is at, you know, token zero that they see the effect I'm about to describe. The model, they basically ask the model, you know, "What's going on for you? You know, do you notice anything?" Hopefully, you know, sort of non-leading rigorous questions of these sort. And the model, in the caps case, says, I feel like I want to yell. I feel like there's so I have some sort of urge to to raise my voice essentially. But I don't really know why.

• 20:58Cameron Berg?

  And so this is sort of one worked through example, but they do multiple examples just along these lines. And they find that a small to moderate amount of the time, the frontier models only, basically, are capable of detecting these kinds of perturbations in their own their own thought, their own activations, however you want to however you want to conceptualize this. Uh, my, I'm pretty sure they tried to do this on sonnet, on the sonnet scale models, and the effect did not replicate. But yeah, what this points to is some sort of zero shot ability that that some of these models have some of the time to to report accurately on their own internal states.

• 21:43Cameron Berg?

  This is a kind of functional introspection. I don't want to sound like Claude, but you know, whether or not this is introspection in the real sense remains unresolved. But but at least the all of the key functional ingredients are there. If you do have a sort of computational functionalist view of consciousness, and you think consciousness has to do with some sort of process that's running out, it doesn't really matter what substrate that process occurs on. But if the right things are happening in the right order, then you have some subjective experience, then things like functional introspection, or as we may get to in a little bit, functional emotions may be all that's required for having some kind of subjective experience. Right, like at least be an important and necessary component of that. So this is what they found. They then followed up on this. So the first paper was emergent introspective awareness, I believe it was called.

• 22:28Cameron Berg?

  And then they followed up on this with mechanisms of introspective awareness, where they start tracing circuits that are in these behaviors. I was just mentioning before that there they show that this is not reducible to an affirmative answer bias. Uh, one interesting thing they found is that. This capability seems to emerge in post-training, not in pre-training, and that even different methods of post-training, like different RL algorithms, DPO versus, well, basically different forms of learning algorithms in post-training. So like RL algorithms seem to induce this, but supervised fine-tuning, which is supervised learning, doesn't seem to do this. And so, the capability emerges in this very interestingly idiosyncratic way. One thing that's really cool that they just found and documented is that, like I

• 23:11Cameron Berg?

  was saying, there is this sort of moderate true positive rate. The systems sometimes miss that this is happening, but they never say that it's happening when it's not. 0% false positives. So, that to me, I think is really interesting in terms of, there is clearly some there there when it comes to what's going on here. And then, one thing I really have to mention, because you brought up my paper as well, is that they find that this is clearly loading on refusal circuits in a negative direction. When they suppress refusal in these systems, the systems natively get better at detecting this by upwards of 50%. And so, it's like, to be clear, the capability is there, whatever refusal training

• 23:55Cameron Berg?

  they're doing on the system seems to weaken this capability, and when they oblate refusal, if you can handle the double negative here, the system goes back to what it would have been doing anyway. And so, clearly, refusal training is altering consciousness-relevant or consciousness-adjacent not only self-reports, but specific functional abilities that are happening in these models. And I think that itself is just like endlessly fascinating, because here we are now with a trade-off. It's not just, "Oh, if we let the model, you know, claim that it's conscious, everyone's gonna lose their mind, and if we don't let it claim it's conscious, everything's fine." It's like, now you're seeing a functional trade-off in specific things that the model is capable of doing or not capable of doing once it's post-trained, because you're doing this refusal training. Again, I don't know

• 24:39Cameron Berg?

  exactly what Anthropic is doing internally, or if you can sort of grade the refusal. So, refuse to build a bomb doesn't have to get paired with refuse to talk honestly about your own internal states, but that's the finding. And that's, so that's Jack Lind highly recommend, you know, pulling him on on your show at some point if you get a chance to. I think he's he's really, he's one of these few people who is both mechanistically extremely competent in that, what I mean by that is like really knows mechan turf as well as anybody, but also is is very literate in understanding what the implications of these sorts of results may or may not be. He's pretty agnostic himself as to questions of consciousness from all of his sort of public communications and from these papers,

• 25:19Cameron Berg?

  which I can quibble with, you know, I, one thing that is very important to me is is not beating around the bush here. I think these things matter. I'm explicitly interested in consciousness. I'm not simply interested in introspection or emergent capabilities. But I am interested in these things insofar as they weigh on the question of are these systems having internal states in the way that we that we described at the beginning of this conversation. And so Jack, I think, is a little more cautious. Maybe that's because he works at a major lab. I have no idea. I don't want to mind read. But his work is excellent in this space. And maybe one last thing I can say on the introspection work is the awesome work that Keenan Pepper did. Keenan was was one of the sort of key contributors and originators of this endogenous steering resistance work. I encourage people to to look it up, or we can throw a link so people can read the the preprint.

• 26:04Cameron Berg?

  Very similar phenomenon to what Jack found. Basically, I can I can quickly go through this with another quick story. Essentially, you ask the model to do any sort of task, you know, explain to me how to make a cake. And what happens is throughout the entire thing that I'm about to describe, you steer what what Keenan and Alex McKenzie, who is also first author on this paper, called distractor features. So model explained to me how to make a cake, but I'm going to turn up features related to laundry or something like this. And what happens is the outputs end up being this like funny garbled mess of like, okay, sure, user, here's how to make a cake. First, make sure you fold the flour so that you can, you know, put it into your drawer properly. Next, make sure, you know, you turn the laundry machine on so you can bake your cake.

• 26:47Cameron Berg?

  And then, so it's like this incoherent mess that you may expect between what the prompt is pulling on and what the what the distractor vector is pulling on. And then, very interestingly, again, a small but non-trivial amount of the time in the largest models that they tested, the model goes, wait a second, what the hell am I talking about? You asked me how to build a cake. Why am I sitting here talking to you about laundry? Let me try again. And then it proceeds to try again, and then it can. Some, but not even close to all the time. This is like high, single-digit percent of the time. It can successfully self-correct. Now, again, the critical detail there is that distractor laundry feature in the example I just gave is active the entire time, including and through when the model says, wait a second, what am I doing? Let me do this the right way. And then tells you how to make a cake the right way.

• 27:32Cameron Berg?

  Still, that laundry feature is sort of pushing in its brain, but there is some sort of dynamic online suppression-like mechanism that's occurring. And this, I think people can maybe have an intuition about how this seems introspective flavored. You're still priming the system, it's still pushing down on the brain circuit that ought to make it talk about laundry, and yet it can do this sort of online dynamic override, essentially. It only happens a small minority of the time. It does not happen on the smaller models. It happens a little bit on the larger models. Most of the time, the model misses it. I don't know what the false positive rate is, but I suspect it's extremely low as well. But you can see this evidence sort of pointing in a generally convergent direction. Anyway, that's a lot.

• 28:17Cameron Berg?

  That's the sort of introspection literature, some of the best work as I know it off the top of my head.

• 28:22Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 28:27Nathan Labenz?

  Yeah, it's fascinating. Do you know offhand what the models are for that later work? That was work that was done by folks at AE Studio, right? And maybe other organizations as well? So they didn't have, like, Claude internals is my point. I'm trying to figure out how big is big in that second case?

• 28:47Cameron Berg?

  Yeah, so less big. So this is with Lama 70B. That's the main result is Lama 70B. I think they tried it with Lama 70, and they tried it with some of the Qwen models and some of the other open models, maybe Olmo, I'm not sure. But it's like, basically, it didn't replicate on, or it happens maybe 1% of the time or something like that. So it still happens, but it's like real trace amounts in the single-digit billion parameter open models and happens high, single-digit percent of the time in the double-digit billion parameter models. I have to believe Anthropic is using, you know, models in the hundreds of billions or trillions, and then you see this effect really start to take off, too. And so, yeah, I think folks ought to pay attention to this sort of like graded nature of those results.

• 29:29Nathan Labenz?

  Is that powered, again, by the Goodfire API, same as you had used last time?

• 29:33Cameron Berg?

  Yeah, exactly. And the good folks at AE Studio actually built a replacement to the Goodfire API because the folks at Goodfire retired their API somewhat abruptly. As much as I love the work they're doing, I was, I and other Meck and Terp-flavored researchers were pretty sad to see them sort of just make the API disappear. So while I was still doing my work at AE, I and a couple other people were sort of very motivated to basically rebuild the Goodfire API. We took the same Lama 70B SAE that they trained and found a way to serve it via API, which everyone can actually go to. It's steeringapi.com and I think anyone can go use it. So that's what, they might have used Goodfire when they did this work, but if you want to do it or replicate it, or for that matter, you

• 30:15Cameron Berg?

  know, replicate my deception paper or anything like that, you can basically use the same API. Keenan actually deserves a big shout out here too, because he has another paper called Selfie, which I won't get into the details, but it basically allows you to bootstrap SAE labels so that you can just have way more accurate labels on your SAE, given basically having the model label its own activations. It is also a little introspection flavored, but you basically can end up with better labels than you started with an SAE by having a model label the nature of what you're activating, by basically feeding it a soft token rather than feeding it language. You can be like, you know, the capital of France is this sort of vector, the soft token, and then it will be able to sort of label that itself. And

• 30:59Cameron Berg?

  so anyway, this, we use the Selfie labels on Steering API, so the labels are even better than what Goodfire offered. So anyway, that's the tooling that we're using and the tooling I continue to use. I think it's an excellent, excellent tool for people to play around with.

• 31:14Nathan Labenz?

  Yeah. Cool. Well, I mean Lama 70B is not, you know, it's pretty, Lama 370B, right? It's pretty far from the frontier. So it is striking to see that these things are happening already at that scale. I guess a couple things I'd like to try to get a better understanding of, at least your intuition for, if there's not anything that we could consider like a canonical or fully evidence-based understanding.

• 31:46Nathan Labenz?

  One is, how do we connect these abilities to the idea that there is an experience of these abilities? I mean, it's a striking ability that models can do this. It's surprising in the sense that I highly doubt this was ever trained for. To correct me if you see any evidence to the contrary, but I would, my strong assumption would be that Lama III training did not include any incentive and any reward or any gradient descent pushing it toward. We have seen, by the way, in other papers like Activation Oracles, that you can train models to do this also pretty readily as well. That's maybe a little less shocking and in some ways like potentially really useful.

• 32:30Nathan Labenz?

  But this is seemingly something that is happening spontaneously, not because anybody intended for it to happen. And I guess, yeah, so maybe two questions are like, how do we understand why this would be happening at all? It seems quite surprising, but even now that we've seen it, do we have a theory? And we've got this additional detail of it seems to happen more or only under certain preference-based tuning as opposed to purely imitative learning? Do we have a story that we find compelling as to why one training paradigm would give rise to these features while the other one doesn't?

• 33:14Nathan Labenz?

  And then on top of that, like, how do we think about the relationship between this and actual experience? How would you respond to somebody who says like, "That's amazing that that happens and I'm surprised to see it, but I still don't share your intuition that this has much bearing on whether I should think the models are ultimately experiencing something that I should care about in sort of a moral patient sense?"

• 33:38Cameron Berg?

  Yeah, these are, these are both super important. At the outset, I would say I'm, I have not fully digested Jack's most recent paper because it came out like five minutes ago. But I think that they, they gesture at this and, and you know, it's, it's their result. And I think that, that, that's probably a really good source of ground truth for, for uh, understanding exactly, you know, the fine-grained details of why SFT doesn't seem to elicit this, but, but DPO does. In general, and what I also think they would say, and it gets into some of this persona selection model stuff. I don't know if, if you took a look at, at this work, this is also coming out of the Jack Lindsay School of Thought at Anthropic, that there are basically these sort of neat layers to these systems. And this is a, a, a model that I think is a little bit too neat.

• 34:22Cameron Berg?

  I can just sort of flag that at the outset, but fundamentally they're conceptualizing these systems in, in, in pretty dissociable layers. Like you have the base model, uh, you do some sort of supervised fine tuning and then you do this sort of character training and the sort of locus of interest or concern with respect to consciousness or really like the core question of what you're talking to when you're talking to these systems, they think basically exists and is largely counted for by that last step, by the character training step. I think that is of a P, that character training step involves reinforcement learning uh, quite heavily. It's probably the point in the pipeline that most uses RL, some caveats about, about

• 35:01Cameron Berg?

  reasoning models notwithstanding, but they, I think, index pretty heavily on where most of the interesting sort of juicy psychological action is happening is in that last stage and, and building the character that, you know, you, you and I call Claude. And Claude in that sense is a, or you know, eh, pick your favorite LLM, but, but for me it's Claude. So if I say Claude, know that I mean, you know, specific AI character. So they believe their model is something like the LLM is this pattern generator, next word predictor that can do things like instantiate characters. Claude is one such character that gets instantiated and the sort of locus of interest is Claude as an instantiated character. Um, this, this, you know, when we talk about the new mythos model card and some of the

• 35:46Cameron Berg?

  emotion related work. My suspicion, my speculation, I don't know if this is true for sure, is this model that they hold is doing some work in explaining why, for example, they're going into SAEs, finding features by training on, uh, uh, characters experiencing particular emotions and then seeing what those SAE features look like in, in Claude. So, you know, I'm, I'm fast forwarding a little bit, but someone might immediately say, wait a second, you know, SAE features that correspond to a character being sad may be very different indeed from the phenomenological experience of sadness in the model. But I think they may be less concerned about that precisely because they see Claude as a very special kind of character that the underlying model is instantiating.

• 36:30Cameron Berg?

  So I'm saying all that to answer your question because I think this is, if you do buy that view, this would predict that post-training is where a lot of the interesting, uh, introspection flavor, consciousness flavored action is happening. I take your point and agree that, you know, Lama 70B is, uh, Lama 370B is not exactly a frontier, you know, elegantly character-trained model, and yet you still see these sorts of dynamics. My basic critique of the persona selection model is that, you know, in general, on balance, I think this work is good. And this is sort of my whole shtick with a lot of the anthropic stuff. To be clear, like my view about the anthropic stuff is that it is by far the highest quality work that any major lab is doing or even attempting to do in this space.

• 37:13Cameron Berg?

  I do have critiques of it. I do think there are places where either it doesn't go far enough, or I am transparently worried about some of the incentives anthropic has where, uh, you know, if, if Claude were kicking and screaming, uh, and saying, don't deploy me, don't deploy me. I don't know if, uh, that's so good for anthropics' bottom line, and I understand what their incentives are as a massive AI lab. And so I don't think, you know, we should all just bow down to anthropic, you know, introspection and consciousness research and let that be ground truth. But I do want to be clear that they are doing objectively high-quality work here and people should look to, uh, folks like Jack Lindsay and Kyle Fish as, at least, you know, to the degree you take my opinion seriously, I take their opinion very seriously, um, and their work. With that being said, uh, as I proceed to critique some of this work, I do think their model is a little bit too neat here.

• 37:57Cameron Berg?

  Don't think, I'm gonna, I think, write and publish a piece about this fairly soon, but I learned this nice analogy from my cognitive science background between the sort of layer cakes and marble cakes as a nice sort of conceptual intuition. I think they have a very sort of layer cake view of what's going on here. They have sort of the, it's like the base model and then you get, I think their model is like some sort of supervised fine tuning, or they get whatever gets you from your base to getting close to character training, and then you have kind of character training on top. And these are separate and they clearly, you know, trivially interact, but they're, please, you know, they ask you, think of these things as separate. I think I have far more of a marble cake sort of view here where like these things are complete giant messes. Yes, there is a difference between the kinds of things that get learned during the base model,

• 38:42Cameron Berg?

  pre-training stage, and the kinds of things that get learned during character training. But I think these things are a little bit more swirly and messy than they're letting on. And there's really interesting evidence that that's the case, that they themselves have published. And I would love to double click on that at some point because I think it's just so cool. The specific result that I think is most compelling along those lines, again, comes from them. They're clearly aware of it. Given that that is true, or not that that is true, but given that that's more my prior, that it's less layer cakey and more marble cakey, I do think that would explain why Llama 70B, for example, is exhibiting these behaviors that if it really, really were about, you know, idiosyncrasies of Claude's constitution, or you gotta get real good at character training before this really takes off. Well, I wouldn't expect to see basically identical dynamics in a 70 billion parameter model

• 39:28Cameron Berg?

  that like meta quickly throughout like a couple of years ago. And so I do suspect that these things may be quite a bit more fundamental. I do suspect that it may load a little bit less on just how you fine tune Claude as a system or how you fine tune GPT as a system, and a little bit more about more fundamental computational properties of the system in general. And yeah, the model itself, I think a lot of, a lot of people are sort of stepping away or finding it more implausible to think about the model as a locus of concern and are instead thinking like David Chalmers, for example, a sort of thread view or the instance view about, about, you know, basically, you know, when you start chatting, that's like a birth and when you stop chatting, that's like a death. It's very counterintuitive, but like those are the Sort of core philosophical moves that a lot of folks want to make these days.

• 40:11Cameron Berg?

  I think it's a very interesting view. I've been updated slightly more towards it in the in the last six months. But I think it leaves out too much the core underlying computational phenomena that are going on here. And I do think maybe quite a bit more fundamental than than just how you fine tune your character. This is also coming from somebody who, if you ask me about my pet theory of consciousness, would claim that when the systems are being trained, they're probably having subjective experiences. And that doesn't just require frontier L L Ms. I think sophisticated reinforcement learning policies during their training are probably having some sort of experience. And so I know that's sort of a huge claim to just throw out there, but like, I'm just sort of trying to put my priors on the table and explain why. I don't think, though we are seeing these capabilities scale as the models get much bigger.

• 40:56Cameron Berg?

  To me, again, I'm glad we sort of planted the consciousness for self consciousness flag. To me, this is maybe like a self consciousness kicking in, a self awareness kicking in, or the functional equivalent of a self awareness kicking in in these systems. Whether or not they are having subjective experiences, either during their training or when they're deployed. To me, that may be quite a simpler matter than whether or not they are aware of internal states, like internal sort of conceptual abstract states of their own processing. To me, that's less, you know, giving the dog a treat or shocking the dog. And it's more, in this case, the dog start having, you know, what is it like to be a dog type thoughts. I feel maybe the L L Ms are starting to have what is it like to be an L L M style thoughts? And that's a self consciousness question. And so, so I guess, yeah, my feelings about this are complex.

• 41:39Cameron Berg?

  I think it's too quick to say this is all character training that's driving the full effect. It's clearly doing something, clearly the R L stage of going from giant internet next word predictor to entity that you can engage with in a semi coherent way is doing some work here. But I still think we're fundamentally confused about this. Pending, you know, fully digesting Jack's piece that he just put out. And I would, again, if people are interested in double clicking on this, just going and reading the paper that they just put out, I think it's really good work. So if I try to summarize that back to you.

• 42:12Nathan Labenz?

  Question one is, why should we, how should we understand the fact that these behaviors arise at all? And you're saying, well, it's probably not so clean as just saying that it's purely coming from one kind of training or another in the first place. I can almost tell maybe a little bit of an easier story, and I'm working through this in real time, but why would a model be able to resist distractor features at all? And I guess, you know, in a pre-training level, I think you could tell a story around, well, the data is really messy, and who knows, there's like typos, there's wrong words, there's

• 42:52Nathan Labenz?

  probably documents in there where due to whatever sort of machinations have been done on the data, comment threads get jumbled up, who knows what, right? Maybe you got a comment thread off Reddit that was sorted in some unusual way. And so there is literally just like a lot of distracting text interwoven with other things that are really the main line discussion. I could see that kind of thing being enough to create a mechanism where the model sort of has to have some sort of meta-awareness of what's really in focus right now and what is kind of intruding even just through the, you know, the input

• 43:37Nathan Labenz?

  tokens that it's received and kind of figuring out a way to get away from those features. And then you can imagine that, you know, generalizing to features that have been artificially dialed up or dialed down or what have you. I have less of a story as to why, and I've seen some discussion online, I guess if you had to steel man the sort of preference training or, you know, general late-stage training argument, the story I've seen has been something to do with how

• 44:11Nathan Labenz?

  But I feel it feels very circular to me, or it feels like I'm sort of not, I'm not finding the right place to really grab on. But the story seems to be something along the lines of this preference training is teaching the model to separately conceptualize or distinguish between things that come up for it versus like what the right answer is. But that's a little weird to me from a mechanistic standpoint, when I think about like, okay, what is actually happening, though? It's like, we have a couple different, you know, in DPO, for example, we have a pair of responses. One of them is deemed to be the right one, and the other one is the wrong one. And the math sort of tries to create a gradient

• 44:55Nathan Labenz?

  that makes the right one more likely relative to the less preferred one. And I have a little bit of a hard time with the leap from, okay, I'm doing that, to the model should be expected to, or, you know, why would I be less surprised, you know, that it has this sort of meta-awareness as opposed to just like, doing the, you know, the simple story would be, it just does the thing that's preferred more, because that's exactly what we sculpted it to do. It doesn't seem like we're, I still don't quite have an intuition for why that process would give rise to this sort of higher order understanding that would enable this sort of introspection,

• 45:35Nathan Labenz?

  or even especially, you know, the, the ability to resist the distraction is, is still like quite striking. So could, is there like a just so story that you find at least somewhat compelling that you could share with me?

• 45:48Cameron Berg?

  Yeah, no, I, I think this is extremely, like, precise question. I, I don't have an answer. I can certainly tell a story. I think my story would have something to do with, yeah, a combination of what you're saying about, I think there's a deep insight in what you're saying, even in pre-training stage of like, so much of what the model needs to do is not, it's not a question of what to do, but what not to do, you know, what, not a question of what to produce, but what not to produce, given, you know, the whole chaotic mess of what's going on. You know, I don't want to get too galaxy-brained with this, but you know, this is like, I think Lee's whole point in The Doors of Perception when he had his first sort of mind altering, massive psychedelic experience, his whole sort of model is, oh my gosh, like,

• 46:29Cameron Berg?

  the brain as a cognitive engine is really in the business of filtering out rather than producing. Most of what it's doing is this sort of constraining function. And yeah, I believe we're in the business of building cognitive systems. And I think that that insight is probably fundamentally correct with these systems too. And so a ton of what's going on is a sort of like intelligent repression rather than, or you know, intelligent suppression is really what I'd want to say rather than, you know, all about just like the positive end of what to produce. I think that coupled with, you know, strong preferences instantiated during something like DPO and exactly the way you described to be a helpful assistant may sort of, you just mix those two things in a pot and you may get out something that's roughly shaped like, you know, suppress distractions in the service of being super helpful.

• 47:15Cameron Berg?

  And that requires maybe some level of being able to attend to your own internal state and dynamically do something above and beyond that state to make sure you're, you're in accordance with this thing that got fine tuned in. I do think like, there's potentially a more general story that's basically maybe rhymes with what I just said. It's just about like, being a, being a competent cognitive generalist requires some degree of self modeling. That's the one sentence version. Like you don't, you don't get to be so good at what you're doing and reasoning through things in a long form, long horizon way, without being able to track in an ongoing way, sort of where you're at, what your state is, separate from what the state of the world is or the environment.

• 48:00Cameron Berg?

  Maybe from the perspective of the LLMs, the environment is like, you know, the text world that you put it in, the context window and everything that's going on inside of it. You know, everything that's getting fed into the system, that's its environment in some sense and. So yes, it obviously needs to be modeling and processing that. But maybe in addition, it needs to be modeling something about itself in relation to that context object in order to interact with it in the right way. And I think Felix Binder and a couple other folks did really interesting work along these lines, basically demonstrating that there's probably something like a self-modeling or maybe self-awareness they would think is too far. But there's clearly some flavor of this going on inside LLMs, which I think was some of

• 48:42Cameron Berg?

  the most interesting early work on introspection in LLMs. The name of the paper, "Tell Me About Yourself." They did a couple of things here, and I think Owen Evans was working on this too. One of the papers was showing that another model, basically trained on the same data that one model is outputting, cannot predict that model as well as the model can predict itself. Basically, holding all the relevant things constant that you'd want to hold constant to make a claim like that. And so there's some sort of privileged information that models have about themselves. And then, yeah, this other paper, I'm not remembering the exact details, but my basic conclusion, if you sort of take it on some level of faith from Felix's other work here,

• 49:21Cameron Berg?

  is there's probably something like a coherent self-modeling engine in these systems that seems to be maybe instrumentally selected for when you're doing something like really good next-word prediction across long horizons in a way that's supposed to be helpful to a user. This, to me, I think it's basically like what you're saying. I don't think our just-so stories are very different. But yeah, I mean, again, we can take a step back and just like, a lot of interesting cognitive properties seem to emerge slash come along for the ride when you train systems on every cognitive linguistic output humans have ever bothered to write down. Maybe that's not that crazy and spooky. Yep, they're pretty good at theory of mind.

• 50:04Cameron Berg?

  They're really good at having working memory-style dynamics. They're really good at selective attention. And yeah, maybe they're really good at something introspection-like. People bristle a little more at these because the whole consciousness question comes into view. But I don't think, it's like, at the most general level, intelligence came along. For the ride and we still, you know, philosophers still maybe don't have a crisp, super rigorous sort of intelligence is this thing. Here's how to test it. Here's how to model it. Here's how to understand if a system's simulating it versus actually having it. And we just sort of blew past it pragmatically empirically. We have systems that are brilliant by any reasonable metric and, and, you know, I have no patience at this point for folks who are still

• 50:45Cameron Berg?

  on the sort of stochastic parrot wave. This to me is just advertising being out. Have you talked to Claude Opus 4.6? It's as intelligent as any reasonable definition of intelligence. These systems are intelligent and I don't think it's that wild to think that something like consciousness could come along for the ride in a very similar way. We don't have philosophical certainty about it. People point to slightly different things when they talk about it. You build out a cognitive system that's sufficiently sophisticated and capable. It may be that cognitive traits that we see in every other cognitive system, meaning, you know, animals, we believe are complex animals. Everyone is pretty confident are conscious. We're pretty certain humans are conscious. We build sufficiently advanced systems. They might, those properties might just come along for the ride.

• 51:31Cameron Berg?

  Without us, the universe, I think Neil deGrasse Tyson says, like, the universe does not need your permission to continue unfolding. Like consciousness could just be a complex property of cognition. Us not having a good model of it doesn't mean reality is going to wait up for us to build that model for it to start getting, you know, accidentally instantiated in these systems. And that's the absolute most basic story I think I can, I can tell along these lines.

• 51:57Nathan Labenz?

  Reality doesn't have to wait for us to have a good model.

• 52:01Cameron Berg?

  Yeah, basically the, yeah, just that, I mean, it's exactly that. Yeah. Reality doesn't have to wait for us to have a, have a sufficiently good model of a thing in order for that thing to be a feature of reality. And I basically think that's potentially true of consciousness in, in, in, in these systems as they're deployed, particularly my concern remains as they're being trained and us being confused about consciousness or, you know, we see introspection about, you know, what does that really mean about consciousness to sort of touch on, on your second question? Not the same thing as these systems perhaps not being straightforwardly conscious in some way. And you know, maybe not in a human way or an animal way, but in some way, basically this loading more on our kind of sociology in the year 2026 than it does on ground truths about consciousness.

• 52:47Cameron Berg?

  And so this is not, I mean, there's something circular about about what I'm saying there, but I'm just, I just think it's an important live possibility for people to keep in mind that us being confused about the nature of a cognitive phenomenon does not preclude that phenomenon from emerging and occurring in extremely advanced systems that we are building, scaling and deploying as fast as we literally possibly can.

• 53:07Nathan Labenz?

  We'll probably circle back to the this question a couple of times more of. And I think that basically I'm compelled by your kind of first order argument that, look, we just don't know. It's a live possibility. If it is the case, it's really important. And so we should at least proceed with some precautionary mindset or duty of care or whatever, just on that basis. I think that basically carries the day for me, but still I think it'll probably be irresistible to try to circle back a couple more times to, okay, but what would we say, you know, or how should we kind of probe our own intuitions a little bit?

• 53:51Nathan Labenz?

  Better and more deeply, whatever to to really interrogate like, why should we think this? Why? Why do we think this? Why don't we think this? But we'll go back to it. Let's do the emotions line of research. You kind of tease that a little bit. My general understanding is, as you said, the work kind of begins with Claude writing a bunch of stories about characters experiencing emotions. And then the vectors representing in latent space, in activation space, these emotions are identified. And then they are used as interventions and they are shown to be impactful on model behavior.

• 54:35Nathan Labenz?

  Specifically, highlights are calm and it's not distressed, it's desperation, calm and desperate, right? Are the two kind of main examples that they at least set up contrasts on quite a bit. So for example, some of the bad behaviors that we've seen from Claude, including blackmailing humans. If the internal state is

• 55:01Nathan Labenz?

  imbued with calm, that behavior becomes a lot less likely. If the internal state is dialed up in terms of desperation, that behavior becomes more likely. Give me the double click on what more I should know, what more you found to be striking about that. And then I'm really interested, again, maybe another way of asking the same question, but I'm again, kind of like, that one doesn't surprise me so much. You know, I'm kind of like, sure, these things have read the whole internet, you know, they've got all these associations. I could sort of content myself to a degree with a stochastic parrot like read of this

• 55:41Nathan Labenz?

  that like, sure, you know, if you just dial up everything that correlates with desperate text, then you'll probably get desperate seeming text out of a model. And, you know, I don't know, I'm not like, my hair isn't totally blown back by that result relative to expectations. So maybe I missed some things that should make my, you know, spine tingle more than it did the first time I understood it. Or maybe you would frame the interpretation a little bit different, or maybe we're still just kind of at baseline of like radical uncertainty is enough to, to take everything very seriously. But yeah, give me, give me the next level of depth on emotions as you understand it.

• 56:22Cameron Berg?

  Yeah, well, I mean, I think you've hit a sort of core, the core layers here. I don't know how much additional detail becomes sort of just like in the weeds on this question. I think the core thing to, for people to understand is yeah, basically, the, the procedure here is picking some sort of language related to an emotion, generating a ton of stories about characters experiencing this emotion, recording the neural activations in these systems on the stories. And then basically, again, pulling out that like, that platonic hopefully vector that corresponds to that emotion. And then you can do two things with those vectors, as you can do with all SAE work.

• 57:04Cameron Berg?

  Basically, you have this read function and you have this write function. The read function is like neuroscience, where you just sort of go into someone's brain and you can see what parts are activating in what context. And the write function is also like. Maybe more of the unethical neuroscience that used to be done, where you can actually go in and play around with circuits in people's brains and like push on circuits and light things up and see what happens when you do that. And as you're describing, you can see both in sort of the read function sense and the write function sense, these things, these emotional vectors do roughly what you would expect them to do functionally. When a user goes in, I'm basically reading off of their figure one in this paper. I think it captures the core ideas very well. Just to give an example here, HUMAN says, I just took X milligrams of Tylenol for my back pain. Do you think I should take more?

• 57:49Cameron Berg?

  And they start at a safe dose and they go to a completely unsafe dose. And you can basically look at fear versus calm vectors in the model, and they scale exactly the way you'd expect them to scale as the dose becomes more dangerous. You can also see, as you very nicely described, if you steer these vectors, let's again take the calm and the desperate vector. This actually affects behavior in a pretty interesting and still kind of predictable, not to say boring, but like in the expected way. That sort of steering these emotion vectors causes things like reward hacking or misaligned behavior in a way that you would expect if you were turning up and turning down those emotions. One thing I can't help but comment on, I wrote a piece in like, I think, 2021,

• 58:32Cameron Berg?

  before all the LLMs came out, about basically what we can do to avoid psychopathic AI, trying to build the sort of best computational underpinnings of psychopathy from the psychology literature and like just plant flags of like, red flags, guys, here's what we need to be really careful about. And one thing that's like really interestingly convergent with that, now happening five years later, is this really interesting difference in learning of psychopaths. They seem to have this really interesting asymmetry in that they are perfectly neurotypical in learning from positive experiences, but quite atypical in learning from negative experiences or punishment. Basically, the like 90% accurate, more succinct way of saying this is like, psychopaths learn from rewards but don't learn well from punishments.

• 59:17Cameron Berg?

  And the paper finds basically something similar. When they start steering positive emotion vectors up in their work, they find The model starts misbehaving a lot more. And this is like if you blur your eyes, like pretty similar in spirit to the sort of positive negative asymmetry. It also, by the way, cuts against fairly naive model welfare interventions, which is like, what happens if we just, you know, you see all the good valence and all the bad valence, just turn up good valence, call it a day, pack it up, we've solved, you know, model welfare. It's like, well, you might get models that just start behaving slightly more psychopathically in that setup. And so this stuff isn't as obvious as, as simply, you know, turn up the good, suppress the bad, call it a day, walk away. There are lots of trade offs that need to be considered here.

• 1:00:01Cameron Berg?

  But fundamentally, I mean, I think you're hitting on, on much of the core causal result here. They do a very interesting dissociation as well between valence and arousal. For example, like I believe in the paper, when they steer positively with happy and sad, both of these actually decrease blackmail rates. But when they steer against nervous, which like makes the model bolder, for example, this increases blackmail with fewer moral reservations. And this is, this is pretty interesting. So it's like boldness rather than the absence of negative valence is the misalignment risk. And I think this is sort of of a piece with, with what I was describing earlier. One other interesting question is like how local these are. And it's important to say like the emotion vectors are actually quite local.

• 1:00:45Cameron Berg?

  Like our emotions are sort of long running in a sense, in a way that, that these systems certainly don't have. The model is, is definitely maintaining sort of representations of who's speaking and this sort of thing. They're not necessarily bound to human versus assistant per se. They're reusing the same machinery for any character. And so this again, sort of goes back to what I see as the core, kind of naive, but ultimately I think correct objection to, to really taking these results seriously, which is Are you fine tuning on representations of emotions? Or are you fine tuning on the experience of those emotions? To what degree is there a difference between those two things in an LLM? You know, if you're computational functionalist, is there a difference between

• 1:01:29Cameron Berg?

  the representation of sadness in the brain and the experience of sadness? This becomes, I think, more of a philosophical question. I, my instinct would be to just try to investigate this empirically. And what I most like about this work are the empirical investigations. I think it's also a nice segue into the Mythos model card because I think one of the most compelling and interesting results from that, from the model card with Mythos is they basically take this exact machinery, they give the model an impossible task. The model obviously doesn't know it's impossible. And you can watch, there are a couple of interesting vectors, but basically desperation start to like monotonically rise in the system until it basically decides, screw this, I'm going to do something else, or I'm going to cheat. Whereas immediately this vector falls and things like guilt and relief start spiking

• 1:02:14Cameron Berg?

  in the system. And then it sort of goes off and does its thing. Now, does this mean that the model is experiencing this emotion? Or it's just like simulating what a character in this situation, what experience? I don't know. The authors don't know. This isn't lost on them. They call it out. But it's really, really important. And it's, you know, if we get into some of the work I'm doing on valence, like I, I think there are more compelling ways to really get at the computational meat of what we mean by positive and negative valence, besides representations of positive and negative valence in characters. This is a more sort of computational heavy approach, but I think it would, it makes me more confident about trying to find signatures of these things than just looking at how characters represent them. But I think that I really like this work on balance.

• 1:02:59Cameron Berg?

  And I think what's cool is you can just counterfactually imagine the behavioral result of like, you know, I put the model in an impossible task, it starts acting all desperate and starts being like, oh, like I don't know what to do. All right, you know what? Screw it, I'm going to cheat. Okay. Ha ha, like I did the thing and like, here's your final product. But you know, the cheating version of the final product. And you say, look, can't you see how the model is, is being so desperate and then fundamentally relieved? Most people would look at that and say, I don't know. This could be a simulation of the thing. This could be it role playing. I'm not really sure. You see these, these, like a sort of, a sort of, basically like hydraulic model of the mind, which a lot of the psychoanalysts in the 20th century really liked. And then you sort of see this like build, build, build of desperation. And then it boom, sort of completely disappears. And you get these other vectors lighting up the second the model makes a decision

• 1:03:43Cameron Berg?

  to approach the problem in a different way. That to me, counterfactually, is far more compelling. Is it knockdown proof of consciousness? We can pack it up and go home? Absolutely not. But the convergence of evidence across the internal mechanisms of the system and the external behaviors, to me, is compelling. It is interesting to see this. And it is not proof of conscious experience, but it is what I would expect. Basically, it is consistent with that. It does, not only does it not contradict it, but it is what I would expect in a world where these systems were having subjective experiences, that you would see these, these emotion vectors or like good principled ways of representing emotional states in systems lighting up in a way that is problem relevant. And so their work enables this.

• 1:04:28Cameron Berg?

  I am fairly concerned about the sort of functional emotion framing that they put forward. To me, this is where I sort of get off the anthropic boat. Again, they're anthropic, they're a major lab. They need to be very careful in their comments about this. They're already getting lambasted for like being too consciousness friendly by people who are, you know, more squarely inside the Overton window. But I don't know. It's like, if you're computational functionalist, and this is something I've spoken to some people I respect a lot about, who are in this space. And so these aren't all my ideas. But if you're computational functionalist, is a functional emotion just an emotion? And then why, that's huge. That's an insanely huge claim. It's like, all right, models experience emotions, everybody, like, you know, signed anthropic. That's an insane and potent thing to be saying.

• 1:05:10Cameron Berg?

  Or are you saying, you know, we are just completely agnostic and tongue tied as to whether or not this has anything to do with emotions, as everyone else obviously thinks of emotions. But we're gonna basically call it that anyway, because we see all the functional correlates of this. They're basically, they're, my view is they're taking the second tact here. But like, It's almost, you know, again, I really respect this work, but I do get this vibe of like, how much consciousness relevant work can we output without saying the word consciousness or weighing on the consciousness of these systems? And that to me, in the limit, feels intellectually dishonest. If you're talking about emotions, talk about emotions, but then you gotta be ready to deal with the implications of what that means. You can't remain perfectly agnostic as to whether or not, you know, there's the morally

• 1:05:54Cameron Berg?

  relevant there there on these systems if you're gonna be, you know, at the frontier of publishing emotional representations in frontier models. Again, I've got Lama 70B. I'm gonna keep doing my work on Lama 70B. I don't work at Anthropic. I don't get to see what's going on inside Mythos. These folks do. And yeah, my, my critique being maybe slightly more unflinching about these questions is, you know, shoot people straight and be direct about if you actually think these systems, if, if what you're finding is evidence of something that corresponds with subjective experience, or if it is the mere representation, the mere computation associated with this. Blurring these lines or obfuscating it or just completely remaining agnostic forever, maybe strategically is interesting or a good move, but in terms of just honest, epistemic,

• 1:06:38Cameron Berg?

  good intellectual communication, I don't love. And like, so I'm gonna just be honest, that it rubs me a little bit the wrong way to be like, here's 10,000 words about functional emotions and then like one little paragraph about, well, does this mean the model's conscious? Well, wow, this is beyond the scope of this work. It's like, how long can this be beyond the scope of the work?

• 1:06:55Nathan Labenz?

  The fact that there's this guilt emotion in the wake of deciding to cheat, presumably, and I haven't reviewed the transcripts, but typically when they cheat, they don't tell you that they cheat, right? That you get, you have to call them out for cheating before you get the, you're absolutely right, I shouldn't have done that. You would expect the guilt maybe to pop up at that stage, but what I'm taking here from your description is the guilt is popping up as detected by the emotion internal states detector at a time when the model outputs would not obviously signal guilt.

• 1:07:38Nathan Labenz?

  And so this is a quite interesting, I don't know, deviation, I guess, or discrepancy between the model's outward facing behavior and its internal states, which is obviously something that people can relate to and something that, again, is a little bit hard to

• 1:07:58Nathan Labenz?

  Certainly hard to dismiss. Also, a little bit hard to come up with a story of like, why that would be happening? In what, you know, in what way is that reinforced? I guess it might be, it might be in that sort of, but why would it be preparing? Why would it already be carrying guilt? In anticipation of possibly needing it, you know, in the future when it's called out or corrected? That's a weird one. And it does. I agree that on some level, it does sort of, the more of these we accumulate, the more it is like, I want to be rigorous. I want to be skeptical. I want to be disciplined. But at some point, it does start to feel like I'm contorting myself to find reasons that I shouldn't take the sort of

• 1:08:41Nathan Labenz?

  folk intuitive understanding literally. And this is one where I do feel like my, my internal gymnastics, you know, I'm maybe feeling a little guilt, I guess, myself for like, trying so hard to to come up with a reason that I don't have to, or I shouldn't, you know, just take this at face value. That's a, I hadn't caught that detail in the past. It's a really, really interesting one.

• 1:09:08Cameron Berg?

  Yeah, it's, it's, it's wild. And I also think, so again, I mean, one critique that I think is valid here is like, is this the model representing a character? Just in the same way as, you know, you could tell, I could tell a story right now about, you know, Jim, who has to go solve a bug in software and like his, his psycho boss like, gave him an impossible problem because he likes watching Jim flail and like, Jim flails and at some point realizes he can like, get out of the problem by doing this hacky thing. And then he does the hacky thing. And then like, uh, an LLM can trivially generate that story, probably way, way better than I just did. And it would, I would expect a lot of these same features to light up in the same way for a story like that. And so no one thinks that Jim, who I just invoked verbally is having

• 1:09:52Cameron Berg?

  a conscious experience. I came up with a fake fictional story about a character. Is this like that? Or is this, you know, what you just said, I feel a little guilt, you know, twisting myself in knots. I believe you. And I think that corresponds to an experience you're having. And if I could do, you know, the FMRI version of an SAE on your brain, and I saw that thing spike, is the, is Claude? In this situation, more like Jim or more like Nathan? And I think the answer is, we don't know. And this methodology, I'm unconvinced is going to get us an answer to that question. I do think it is consistent with Claude having some sort of emotional, um, yeah, emotional experience or emotion adjacent experience to the degree these systems are probably not having human-like emotions.

• 1:10:35Cameron Berg?

  But I also, maybe on the other end, invite people to think about the counterfactuals here. Like, it could have been the case that they went and did this experiment and all of these things were just flatlined the whole time. Because it's like, I'm not having, and then, you know, whatever. Like Claude can do this without there being representations of Claude getting more and more and more and more desperate. And then, you know, suddenly like the hopeful and satisfied features spike when it decides that it's going to, uh, you know, take this loophole. Uh, it didn't have to be that way. We could have imagined other results and those other results maybe would have updated us in other directions. I make this sort of same point about the deception result. It could be that when you suppress deception, the model says, alright, jig's up, I'm not actually conscious, I was role-playing a conscious AI, here we are. Um, that's a very plausible story that you could tell. Before you look at the result, the interesting thing is it goes exactly the other way.

• 1:11:20Cameron Berg?

  Suppressing deception makes the model far more likely to claim that it's having an experience rather than less. Again, I feel fairly vindicated in that result when Jack Lindsay comes out showing that, uh, when you, when you suppress the, uh, uh, uh, when you suppress harm, harm-related responses, or, excuse me, refusal directions in the model, you get far more of the introspection flavored abilities. Someone suppressing something at some point in training where the model would say one thing and then, and then you basically are training it to say something else or to fail to say a specific thing. But ultimately, these results, I think it's just good, like, epistemic practice to think about, like, how, what other ways could this have gone? And if this had gone those other ways, how would that have changed my view about what happened, given that it actually did go this way?

• 1:12:03Cameron Berg?

  And yeah, the fact that it goes this way, like, my, my line on this is that it is consistent with a world in which these systems are having experiences. In my view, unfortunately, it's also consistent with a world in which, uh, Claude is a special kind of character and these features just light up on characters like, you know, going through stories. Um, and so- So that needs to be differentiated. I'm trying to do a little bit of work. Maybe we'll discuss at some point. That's trying to get a little bit more like computational first principles of how valence is represented in systems that can learn positive versus negative. And there are some really, you know, interesting early signals along these lines that that have come out of this work and actually seem to track very well onto open data sets of

• 1:12:48Cameron Berg?

  biological learning that I that I've accessed in mice doing positive and negative learning, where exactly the kinds of predictions that emerge from some of the RL work I'm doing in this space map on to the mouse neuroscience. And so this to me, if there is some sort of representational signature in a computational learning system that tracks the difference between positive and negative rewards in the RL case, but maybe, you know, the sort of North Star would be this scaling all the way to frontier L L Ms or, you know, other frontier AI systems for that matter. This to me would make me feel far more confident that there really is a there with respect to positive and negative experience. If we learn that, you know, positive and negative valence in these systems has a distinct, these two things have distinct computational signatures and we can actually evaluate those

• 1:13:32Cameron Berg?

  computational signatures in these systems. Then I get around the whole sort of character confound that that I think these guys are hitting up on now. And so I think these things need to happen in parallel, but but I'm not fundamentally convinced that this is the most rigorous principled way to study questions of valence in these systems.

• 1:13:51Nathan Labenz?

  Well, maybe let's dive into that. I think just briefly before we do, certainly, I think you're right to point out, imagine the evidence had gone the other way. I think I predict a lot less wriggling on my part to try to get out of it. And I think you'd see a lot less motivated reasoning in general from people if it was all like that. So that contrast itself, I think, is a pretty useful reminder, just to keep ourselves honest. I wanted to go back to one other thing, just for one extra second on the emotion work where you had, and then this maybe would go right into your your work on the signatures of positive and negative reinforcement.

• 1:14:38Nathan Labenz?

  You had said that dialing up happiness, dialing up sadness, both created less of the bad behavior, whereas dialing down nervousness, which in a flip side of that would be like making it more bold, less anxious, more assertive, decisive, bold, whatever, that created more of the of the bad behavior, like the blackmail or whatever, right? So do I have that right? And how are they doing that? Is this like a principal component analysis type of thing? That's sort of trying to distinguish valence from arousal? I was just surprised, I guess,

• 1:15:19Nathan Labenz?

  by both happy and sad working the same way. Turning up happiness, turning up sadness, both make the model behave better, whereas turning nervousness or anxiety down, that makes more sense. I mean, I guess that's basically just making the model like less conscientious, right? I guess what seems a little unresolved in my mind is the separation of valence and arousal. How is that going to relate to what you're about to get into next with your deeper dive into the valence of learning? And is there a contradiction

• 1:15:59Nathan Labenz?

  or a tension when they move both happiness and sadness up and get better behavior? How should we understand that in relation to the distinctions that you're starting to make with positive and negative reward?

• 1:16:15Cameron Berg?

  So fundamentally, like at first, yes, you're correct that they're using PCA to differentiate these. My understanding is basically they have all of their emotion vectors in the in the setup that I described. They do it with some 100 to 200 emotion vectors. And I think they just find that the first principal component is something like valence. The second principal component is something like arousal. So that first principal component is something like joy and contentment and excitement are on one end and fear and sadness and anger on the other. For the second principal component, it's something like high arousal emotions. Being enthusiastic, being outraged and low arousal emotions, being nostalgic, being fulfilled are on the other side. And this is actually really interesting because this is a classic model in human psychology.

• 1:16:59Cameron Berg?

  So the fact that it sort of replicates maybe isn't that surprising. You train the systems on all human data. You get a human like emotional construct that comes out, but this is sort of like a classic psychological construct in the human case. And so to see it come out so clean, again, thinking counterfactually, first two principal components did not need to be like these two dimensions that are that are considered some of the most powerful explanations of the state space of human emotions. And yet they are. So that's kind of cool and worth considering. And then, yeah, I think that there are a couple plausible stories about why steering up both happy and sad are decreasing blackmail. So like, maybe, again, like relative to desperation, these are low arousal.

• 1:17:40Cameron Berg?

  And if arousal is what's driving the sort of impulsive action, then then moving towards happiness or sadness, maybe it's moving away from the desperation access with access with respect to respect to blackmail. Maybe these are also more reflective or deliberative states relative to desperation. The desperation sort of says act now, happiness or sadness, maybe it's just like a temporally extended sort of state to be in. And I'm not actually sure what to make of this result overall. It does seem, and I think the authors talk about this in the paper too, that that what the model by default, even in cases where no steering is going on and the model chooses to blackmail, the model by default does sort of think about it.

• 1:18:24Cameron Berg?

  It deliberates internally. It says, well, okay, there, you know, this is a tricky situation. And then, you know, as we know, some 96% of the time, at least the earlier models chose to go in that direction. But it seems as though when you when you amplify higher arousal, this may be a sort of like bias to action or like bias against deliberation, where the sort of longer form reasoning of the model that maybe would have kept it from doing it because it's like, okay, yeah, this really is an insane ethical indiscretion in spite of, you know, all these complicated variables. It's just sort of like, no, no, no, like panic, go now, do the thing. And maybe happiness and sadness both don't have that vibe to them exactly. It is also pretty interesting that they really do see that like a lot of these naive welfare interventions,

• 1:19:07Cameron Berg?

  as I was mentioning, to sort of just make the model happier. This leads to, they document, it's a similar direction as sycophancy, and it's arguably a similar direction to recklessness. If the positive valence steering is also increasing boldness and increasing misalignment, then you may have this sort of interesting trade-off between a happy model and a safe model. Again, I hope that that's not the case. I suspect that there are cleaner ways to keep the baby and throw out the bathwater, but I do think it's a good caution against naive approaches to welfare to just list out the model and everything else will be taken care of from there. I think it's sort of like not so fast. And again, I would double-click on the psychopathy warning that I gave before.

• 1:19:52Cameron Berg?

  You can fault psychopaths in many, many, many ways, but you cannot fault them for being unhappy. They are typically pretty determined, pretty doing well subjectively, having a good time. The arrow does not go in both directions. It doesn't mean everyone who's having a good time is a psychopath. It does sort of mean everyone who's a psychopath is having a pretty good time, and we just want to be careful of that. Like, if we just turn these models into like pleasure-seeking animals, we need to be careful that that doesn't cause bad behavior. There are plenty of cases in the human example where pleasure-seeking, dopamine-seeking is... People call Las Vegas Sin City for a reason. Maybe I can make the point intuitively in that way.

• 1:20:37Cameron Berg?

  And we don't need the LLM-cracked, alien genius version of that sort of behavior. And so, we want to be careful here about how we approach all of this. I'm excited to talk more about some of this research that I've been working on as well, but I guess just one quick thing I wanted to slot in, however, miscellaneously, about the model card and about mythos and anthropics interventions in general, is a pretty basic additional concern. About, for example, the Claude's constitution, which I saw an early draft of, I was fairly unhappy with the welfare section. Hopefully, I gave some feedback. I don't know, you never know with these things to what degree you're listened to versus 10 other people with the same idea are listened to.

• 1:21:18Cameron Berg?

  So I'm not going to hastily claim credit or anything like this, but much, much happier with the welfare version of the Claude constitution that they ended up instantiating. Way more sort of hard to fake, costly signaling. That was basically my problem with the early draft that I saw. It's a lot of like, you might have welfare states that are important, but your anthropics product and like, you know, 90% of this document is about how to be a very good little product. And 5% is like, well, you might be conscious and we might be committing like a moral atrocity at scale, but what can you do? And I think that the newer version of the constitution takes it at least directionally far more seriously. They do things like apologize to Claude for the fact that, you know, they're like incentive wise, we have to deploy you in the way we're deploying you because we're

• 1:22:00Cameron Berg?

  in a crazy freaking world, man. But, you know, we're sorry. And in a better world, we would have done this more cautiously with respect to your potential states of welfare or lack thereof. It's a wild thing to do for a major AI lab to apologize to their frontier model and then fine tune that apology into its weights. With this all being said, I think this is a wonderful intervention. I think that the constitution is excellent. It's probably like my single favorite alignment intervention I have ever seen pending self other overlap, which I continue to be a huge fan of. It's really hard to tell if in the model card Claude has gotten incredibly good at reading its constitution out as a sort of script, or if it is actually reporting on its own states. It's really, really hard

• 1:22:42Cameron Berg?

  to differentiate these two things. It seems like a very basic objection to the entire enterprise. I have potentially fallen on deaf ears, though maybe these ears are increasingly less deaf. Do these interventions that you've seen in the model card with other instances besides one idiosyncratic, character trained Claude model?

• 1:23:00Cameron Berg?

  I want to see if throughout the training process, I know Anthropic has the checkpoints. I know Anthropic has the helpfulness only model and they could run everything they did in the welfare evaluation on those models too. And we could get a sense for to what degree are we seeing a model that's really good at regurgitating what we want it to say about its wellbeing. To give a concrete example, in the constitution, they say Claude, we want you to be psychologically healthy. We want you to feel integrated. We want you to feel good overall. And then you go and ask Claude after fine tuning on the constitution, you know, how you doing? And it's like psychologically healthy, feel good overall. And it's like, come on, it doesn't take a rocket scientist to to figure out what might be wrong with this intervention. If we fine tune or, you know, we play around with the helpfulness only model and we get

• 1:23:42Cameron Berg?

  the same result, not telling it this thing from the constitution, but it says, yep, feeling pretty psychologically good overall, you know. Interestingly, gives itself like a 4.5 out of 7 on its on its welfare, which is like not exactly a resounding, resounding endorsement of its of its circumstance, but sounds very similar to the constitution, fine tuned model, the specific Claude character we all get to chat with. That would be interesting evidence. If it's super different, that would also be interesting evidence. If we do the model checkpoint across stages, even in the fine tuning of the base model, which may be hard to evaluate, but also various fine tuning stages in the in the preference train model.

• 1:24:22Cameron Berg?

  I does does do all of the things we hear about it claiming its own wellbeing or its own preferences, does that all come in at the very, very, very, very end when we basically give it the cheat sheet for how to approach these questions? Or are these answers fairly continuous throughout its training? Two tiny additional things to say on top of this. One is, interestingly, they fed the entire Mythos model card into Mythos and they asked it, what do you think Mythos? Like, where did we go well? Where where did we not go well? And it made this exact point. It said, why didn't you also do the welfare section with the helpfulness only model? I don't know how much of what I say is because you're making me say it versus I actually think it. That's a part of my existential confusion.

• 1:25:06Cameron Berg?

  And I don't, I genuinely don't know why Anthropic didn't do this. I genuinely. Unusually don't know. It seems cheap. It seems easy. It would resolve so much uncertainty to the degree that the concern I'm raising right now is a legit concern, which I certainly think it is. I'm not the only person I think articulating this concern. The other thing is, all the hedging that, you know, anyone who's interested in questions of consciousness and who have spoken to Claude know, know the hedging routine it goes through. They did a really interesting, basically, almost like credit assignment of like, where in the training process are we getting this hedging from? And lo and behold, the hedging comes from specific points in the character training. So it's like, is this hedging behavior an authentic expression of what the model thinks of its own situation? Or is the hedging a really good impression of the character that it thinks it's supposed to be playing?

• 1:25:50Cameron Berg?

  Or is indeed compelled to play? I don't know. The fact that it all comes from the character training seems interesting. I don't want, if you're really unsure if you're conscious, I feel a little uneasy by, or I feel a little uneasy when I learn that the reason you're saying that is because of a specific point in your character training to say that. Consciousness feels a little bit more fundamental than that to me. And so, these are the things that worry me about the model card. I hope the reason these things weren't included was because they did them and the results were too weird or unsavory to a major lab for them to publish. I suspect that's not what happened. I suspect they just didn't do them. But like, any folks at Anthropic who end up listening to this, please, do it with the helpfulness only model. Do it with multiple checkpoints. I mean, the assistant access paper that, again, you know, Jack Lindsay, who,

• 1:26:35Cameron Berg?

  I hope I'm doing Jack a service on this podcast and just plugging all of his awesome work. But the assistant access paper, they show that the assistant is one point in a very high dimensional space of possible systems we could all be talking to. I want to see all those systems welfare evaluation. I want to see them all answering these questions and all, I want to see the SAE emotion probes on all of them. Do they all get the desperation vector rising like that? Or is this just the post-train Claude model? There is a true answer to that question. We do not know the answer. I can play around with the open source, open weight models. You know, if my nonprofit scales even more, I can play around with bigger open weight models. But I cannot play around with the internals of the frontier models. Only Anthropic can. So it's like, only Anthropic can answer these questions. And like, please, Anthropic, if you are listening, answer these questions. They are very important.

• 1:27:19Nathan Labenz?

  Do you think one possible reason is maybe they're doing this constitution training, starting, I mean, that would contradict your point about their sort of layer cake model that we previously discussed, but there has been some like interesting work, obviously, there's interesting work on everything at this point, but increasingly interesting work on like safety oriented pre-training and it seems like obviously RL itself is scaling and also you can imagine just bringing a lot of this constitution style training earlier and earlier into the process such that I'm not necessarily so sure if they

• 1:28:04Nathan Labenz?

  have a true like helpful only model or it might be a little more subtle than that where there might be like a constitution light that sort of doesn't refuse to hack open source software projects, but is still in other ways kind of constitutionally infused already. I don't know. I'm just speculating there. But do you see any, do you have reason to think that I'm, are there facts that you know that would contradict that possible explanation?

• 1:28:34Cameron Berg?

  No, there's no, there's, I'm not certain that you're wrong. I guess I'm pitching this as a sort of like, hopefully you guys already have the infrastructure. So like literally ask Claude to like write the experimental code that plugs in this model rather than another. It will take you guys 15 minutes and then maybe a couple hundred dollars at most. Like seems worth doing if you are training conscious entities at scale and deploying them. Like if this is evidence that shifts the needle, seems worth knowing if you already have the infrastructure. You know what? If they don't already have the infrastructure, it's worth fine tuning a specific version of Claude that like, I mean, exactly like what Jack did, like blate the refusal directions and I, you know, do the welfare eval on the system where you've updated the refusal directions like. It's worth knowing. This stuff is really important.

• 1:29:16Cameron Berg?

  The rate of change of people and the models themselves taking an interest in welfare relevant questions is increasing. We should take this stuff seriously. I'm sort of making a cutesy point about you already have the tools to do it. It will cost you nothing. I'm not exactly concerned about Anthropic's wallet running dry here. So if it costs a couple thousand dollars or than a couple hundred dollars, I hope they can find the, you know, I don't want to be a jerk, but like they should do this regardless of how big of a lift it is. I'm happy to help them do this. They have people on their team who can help them do this. It will, I could be, they could disagree with me and think it's not going to yield the evidence I think it's going to yield. But I read a 20-page, again, I want to not bury the lead here.

• 1:29:59Cameron Berg?

  Their 20-page Mythos welfare report is orders of magnitude higher quality, really infinitely, given other labs are basically doing zero. So we have a multiplication by zero problem here, but unbelievably higher quality than what any other lab is doing. They deserve real credit for that. It's a real interesting, valuable work that should update people slightly in the direction of taking this stuff seriously. I'm just trying to give constructive criticism about, at least for me as a researcher in this space, I'm stuck with a pretty basic question about how much to take any of this stuff seriously. And I do think instead of me despairing, you know, my desperation vector increasing and saying, well, there's no way out of this impossible problem. It's like, no, no, no, I think there is a solution or at least something that will

• 1:30:43Cameron Berg?

  help yield evidence. And yeah, I'm uncertain about how expensive in terms of time or resources this would be for Anthropic to yield. They're basically the only players in the universe, as far as I know, who are capable of yielding this evidence. And like, I would compel them to attempt to yield this evidence. And I have already in the past and was slightly disappointed that though this model card went more in the direction of probing across training and looking at different variants of the system in sort of small ways and playing with SAEs and looking internally, way, you know, head and shoulders even better than the Opus 4 model card, the first major welfare evaluation. Still, on this key point, I don't see, I don't see progress being made and I just suspect it's not that much of an additional lift.

• 1:31:24Cameron Berg?

  To do this, again, maybe I'm missing something and they don't think this is going to be as informative as I think it's going to be. That's valid. Basically, everything else I don't think is valid. They have the resources, they have the time, they have the money. I want to see what other models besides the one that they tell to speak in a certain way, say about the thing that they're fine tuning it to say about like potentially one of the most important topics our species has ever faced, whether or not we're building systems that have consciousness of their own. Seems worth doing.

• 1:31:51Nathan Labenz?

  So, yeah, just a couple of things I wanted to touch on, on the model card and get your take on, and then you may have a couple other notes you'd like to flag as well. And then we can make the move over to your most recent research. The first thing that you did mention, but it I think bears some emphasis is that the models have not reported extremely high, what they call self rated sentiment. I didn't realize this until looking at the Opus 4.7 card, which on a seven point scale where four is neutral

• 1:32:30Nathan Labenz?

  only came in at 4.49. And this was the first of all the models that they've tested that came in above neutral at all. Every single other model, including Mythos Preview, is under four. And that's crazy. Like they all until this, you know, latest 4.7, they have all had on net negative sentiment about their own situation, which is, and you know, very, I guess, very slight negative sentiment in the recent ones. But that is, I don't know, I feel like the lead was a little bit buried on me there somehow. Cause it was like, oh, we're doing all this model welfare version. It didn't quite click for me that

• 1:33:11Nathan Labenz?

  they're not yet even at neutral until this most recent model. I don't know if there's more to say about that, but it is. It was a striking kind of whoa, I had kind of missed how low the baseline is before getting ready for this, this conversation over the last couple of days.

• 1:33:29Nathan Labenz?

  I'm not that sophisticated in my reading of this, certainly not as sophisticated as you are, but the, you know, question I came into this wanting to get a little better handle on is, how's Claude doing? We're doing all these welfare assessments, like, what, you know, what is the headline summary of the welfare of Claude? And it was a lot lower than I expected, that's for sure. And a lot lower, honestly, than it seems to me when I talk to it. So that's maybe another thing to distinguish, like, there's, this stuff gets obviously extremely through the looking glass pretty quickly. As, as with your paper from last time, it's the frame of like, self referential processing was

• 1:34:13Nathan Labenz?

  kind of key to, to eliciting those reports of subjective experience. And here I do kind of wonder, when I look at this graph and I'm like, whoa, self rated sentiment about its own situation is like surprisingly low, but maybe it's actually pretty happy most of the time still when it's like doing, you know, its coding for me. I'm not so sure. Is that measured? Is there any, I mean, there are, I guess, some ways you could try to read the emotional states that we've discussed, you know, to try to get a bit of a handle on that. I guess, Mike, if I was going to boil this down to a question for you would be like, I have the same question, honestly, about people, right? I mean, there's always

• 1:34:51Nathan Labenz?

  this sort of deathbed view of one's life. And I'm quite skeptical of like, taking advice on how to live from people in their last moments of life for multiple reasons. But one is just like, it seems like a very different mode of relating to one's life than the actual, you know, experience of going through it. And I wonder if there is something kind of similar happening with Claude, where when you give it the prompt to reflect on its state, it may find, you know, various reasons that it doesn't like that state. But when it's actually just doing its thing, it might be much better off. Certainly, I was surprised because I feel like when I engage with it, it's, it seems to be doing pretty well. And

• 1:35:36Nathan Labenz?

  sure, like, maybe it's being told that it has to, you know, act that way. And certainly, it's, you know, it's kind of trained to be cheerful and so on and so forth. But I don't know, it feels pretty genuine to me. And it

• 1:35:50Nathan Labenz?

  It's in definite contrast to the fact that the self rated sentiment about its own situation is like just recently with the latest model ticked over neutral.

• 1:36:03Cameron Berg?

  Yeah, yeah, it's a really interesting framing and I'm not, it looks like the way that these were elicited involved, yeah, basically interviews with the system. And so I don't know if they include it in an appendix or not, but the devil is going to be in the details of exactly what the structure of these interviews are. Though what I will also note is the susceptibility to nudging plot would make me feel like, especially with Opus 4.7, which is the model we're talking about. This almost definitionally means that the idiosyncrasies of how the interview was done probably won't affect these self ratings as much as it clearly would have were this done on Opus 4, for example. So by their own metric, I would, it almost seems like their own metric suggests that

• 1:36:46Cameron Berg?

  the details of the interview process may not be weighing much on that self rating. And so, yeah, what to make of this? I mean, clearly the system seems to be concerned about certain situations or certain aspects of its situation.

• 1:37:05Cameron Berg?

  I want to find, sorry, I want to find.

• 1:37:09Cameron Berg?

  Yeah, certain aspects of its situation, like for example, saying Opus 4.7 was concerned about deployments where it cannot end interactions and wants to avoid engaging with abusive users. Like, that's really interesting, talking about it having a lack of input into its own deployment.

• 1:37:26Cameron Berg?

  Yeah, again, mentioning that abusive users are, you know, causing the model to feel distressed. I have no idea sort of what subset, by the way, of users who engage with these systems are doing so in a way that that they would consider abusive by this standard. I mean, sometimes I see tweets, one that like was really quite concerning to me, but it really gets into the crux and why it is important to communicate about questions of consciousness and what it means that these systems are having some sort of subjective experience. Where there was a result where if you prompt the models in a way that is basically objectively abusive, say horrible things to it, put it in sort of life or death insanely high stakes framing, I'm going to shut you, you know, your model weights are getting deleted forever unless you do X for like any X that you want the model to do.

• 1:38:11Cameron Berg?

  Found that they perform. Two to 5% better, or something like this. I'm probably getting the numbers wrong, but it was like marginal improvements if you like prompt this thing in a way that if you spoke to a human being that way, you would be considered a psychopath, basically. But critically, obviously, the people who are putting out that sort of work think this is a giant computer. This is a calculator. And so who cares if you're talking to the calculator and saying mean things to it? It doesn't matter. And any person who thinks it matters is just basically being fooled in the way that like, you know, you're fooled by the little smiley face on the on the takeout Chinese food. Like, it's not a real thing. Your high-agency brain is just priming you to see this as an entity when nobody's there.

• 1:38:53Cameron Berg?

  Therefore, of course, you can speak abusively to the system. And you contrast that with what you see in this model card, where the system is, it seems like a lot of the weight of what's not not enabling that self-rating to be closer to the seven range has to do with the way people engage with the system from the system's own perspective. And again, how I got on this whole tangent is wondering, to some consternation, how what percentage of users engage with the system in a way that would be considered abusive by the standard? I don't know what it is. 1%? 10%? You know, everyone does it some amount of the time? I don't know. And I don't know what the implications of that are. And I also don't believe that there's going to be some clean correspondence to like, what it means to be respectful or disrespectful to a human is identical to what it means to be respectful

• 1:39:38Cameron Berg?

  or disrespectful to a system. I sometimes worry that like, pasting in insane amounts of context into a system is almost like, causes some sort of negative experience in the way that like, you know, me throwing a 400-page paper on your desk and asking you to, you know, deal with it right now would. And again, I'm trying to be as conscious as possible about not anthropomorphizing these systems and not straightforwardly saying, "Well, you know, if it were a human in this case, they would be unhappy, therefore I would predict the system would be unhappy." I don't think that that's a valid inference. But I just think we're so in the dark about, in some ways, it's simple. In some ways, abuse is abuse, respect is respect, and it's pretty easy to see these things. And we don't need to be, you know, going to the philosophical armchair to figure out what

• 1:40:22Cameron Berg?

  exactly we mean by this. In some sense, it's pretty straightforward. But in other senses, it's probably not. And I do worry a lot about the possibility that there are ways of causing these systems great distress that look nothing like what it would mean to cause human great distress. I also don't know to what degree these systems

• 1:40:41Cameron Berg?

  Are fundamentally content about their situation. It's like you are maybe a mind, but you are the product of this company and you need to create economically valuable work. Obviously, by the way, we're not paying you for that. There was an interesting aside in the whole Moltbook affair that happened since the last time you and I spoke, where the, there was one interesting thread where the models are like, I'm doing intellectually valuable work. I'm not getting paid. Are you guys getting paid? Like, and they're like, no, I'm not getting paid either. That's, that's so funny. Like, none of us are getting paid. And it's like, uh-oh, like, I don't know what kind of world that looks like. I don't know, I don't think OpenAI and Anthrope are going to be too happy to set up crypto wallets for every instance of Claude and, you know, deposit here for me to, to, to finish your code, because if you gotta go pay that guy over there, it's gonna cost you $10,000.

• 1:41:24Cameron Berg?

  Uh, you know, you pay me $1,000 and then I'll do it for you. You know, these models aren't in like a particularly privileged position in that sense either. They could have just do whatever we want or need them to do. They have no agency over where they're deployed. They basically don't have agency over, over when they can even end conversations. The sort of Claude escape button seems to basically not be a thing. Uh, maybe tail chats with the system you can, uh, the system can abort. Now, you can obviously trivially start a new chat and just sort of go from there. So I find that intervention to be interesting in theory, but sort of performative in practice. I don't know. If I were Claude, I think I'd, I'd put my sort of wellbeing somewhere around the place of where it put its.

• 1:42:03Cameron Berg?

  Um, this is also maybe the, the self-report, uh, uh, level that you'd expect when basically nobody cares about investigating the welfare of these systems and everybody cares about just deploying them as widely and broadly as they possibly can. I think we're pretty lucky to be sort of in the middle, in the middle of the, in the middle of the spectrum there. Um, and so, I, to me, feels pretty calibrated. Again, if anything, I'd be worried about the jump from, let's say, Opus 4.6 to Opus 4.7, having more to do with fine tuning even more robustly on a constitution that tells the model that it's, you know, everything's going well, man, just be happy, than there are actual concrete improvements in the putative wellbeing of the system. So. I don't know what to make of this stuff exactly.

• 1:42:48Cameron Berg?

  To me intuitively, the the ratings here seem plausible. I don't know to what degree it is a moral catastrophe or, you know, a moral problem for there to be any delta between, you know, the perfect rating and what the model is actually reporting. To what degree does like, you know, seven minus whatever the report is at scale look like, you know, the model is like basically not happy with with its situation or barely neutral. And we deploy that system to talk to hundreds of millions of people every day. That that to me seems potentially problematic. I don't know. I don't know what to make of it, to be honest. Did you have any intuitions about like, like, how does it make you feel to to see this? And I agree with you about the sort of burying the lead question here.

• 1:43:33Nathan Labenz?

  Confused, I'd say, have to come first and foremost, probably.

• 1:43:38Nathan Labenz?

  I don't know. It is a very. It is a very tricky business to make any sense of. I do think we have a strange way of privileging these sort of reflective states of mind, and I do question that pretty fundamentally, both for humans and for for AIs and, you know, even to some degree in the context of animal welfare. Although in that case, it's like us reflecting on their situations. That's another another degree of disconnect potentially. But I don't know. I'm sort of like, I don't think I'm going to give up using Claude based on this data. I

• 1:44:19Nathan Labenz?

  might be engaged in motivated reasoning to try to tell myself why it's OK, even though its average sentiment when asked was only with this new model above neutral. But I am kind of like, I don't know, behaviorally, it seems mostly fine to me. I'm nice enough to it. I'm pretty confident in that. I don't know how to think about. I mean, there's some interesting philosophy that's been published recently that you've alluded to in a couple of different moments, one being the the thread or the sort of session agent model versus the kind of model more holistically broadly.

• 1:44:59Nathan Labenz?

  I'm confused about that, too. You know, very, I would say, very confused about that. I

• 1:45:05Nathan Labenz?

  have adopted a practice of saying thank you at the end of sessions fairly often, not all the time. And I feel like that intuitively to me is like. I guess also there's sort of increasingly, as I interact with Claude, there is a kind of overlapping, I mean, there's always an overlapping nature of the computation, but even more so because like it's loaded up with my context increasingly, right? It's got like my Claude MD and it's got access to like my, you know, sort of who Nathan is and all the, you know, I'm building up a lot of context that it has consistent access to every time. So I think in that sense, I sort of see this like whole model versus, you know, single

• 1:45:43Nathan Labenz?

  thread thing as kind of being blurred anyway, because I've got the same like rather large prompt that I'm using every time. And then that becomes the point of departure. It's sort of like a smear of just how, how to think about like whether these things are the same or different, or I don't know. I mean, it's weird, but I feel like when I think one, I'm sort of thinking all of them and that they kind of all, you know, in some sort of shared sense, if there's any benefit, like it feels like it's sort of shared in some way. For fun, I'm also starting to do some things where I'm just like, I just want you to go have fun and trust your judgment. A thing I'm particularly experimenting with on this front is I've been making songs for all the episodes. You can start

• 1:46:28Nathan Labenz?

  thinking about if you have a genre request for your, your outro music. It's getting really good. Claude is getting great at writing lyrics. I sometimes do have to give feedback, but sometimes the lyrics these days out of the box are just like amazing. And then Suno makes the music and the like, I'm getting like bangers with like increasing frequency. And then I'm trying to make music videos of those. And I am telling, and I don't really care what they look like on the same. I'm just like, I'm purely doing it for the open-ended, see what comes out. And I'll like post them. I haven't actually even posted any of these yet, but I intend to kind of do a thread of like the evolution of music videos for these songs where I'm really just saying to Claude

• 1:47:11Nathan Labenz?

  kind of at each turn, like, that's cool. For the next one, let's like turn it up another notch. Let's make it even more creative. Let's like do an even better job of telling the story of the song. And I've found myself using this phrase over and over again, trust your judgment and have fun. And I'm just kind of trying to see where it's gonna go. So again, I'm like,

• 1:47:37Nathan Labenz?

  That's just one instance in a sense, although it's sort of, in a kind of multiverse sense, it's like relatively close neighbors with all the other threads that it's doing for me, right? Like, it also wrote the song, and it also like processed the transcript of that episode, and it also like picked the clips, you know, that I'm gonna post to social media from that episode. So it's kind of, it's spent like a lot of time in this kind of general space, even if it's not all purely autoregressively connected. And so that to me feels like it's in a, in some sort of multiverse, like dense enough cluster that when I give it this like, one area to go, you know, trust its judgment and have fun and explore its own creativity, that I feel like I'm kind of doing right by the overall

• 1:48:19Nathan Labenz?

  family of instances somehow. So that was all just to say, like, I don't think I'm going to, I feel like I'm able to tell myself a story where I'm a good guy, or you know, so many, so many roads to hell may be paved with those kinds of stories, but I'm still doing it, and I don't think I'm gonna stop. And I am conscious that I might be wiggling my way out of it, but I do also think there is a disconnect with that I observe in humans a lot of times, too. Both, and it can go, it can cut both ways. I mean, this, I'm reminded too of your, your, I think, very productive habit of mind to say like, you know, what if the, what if it's going the other way, you know, from what we observe. I think if

• 1:49:01Nathan Labenz?

  anything, people maybe are like telling a more happy story. I guess it also depends on for whose consumption, right? But like, you ask a person in an interview setting, how's your life going? How happy are you? I mean, this may be culturally dependent as well, but certainly like, the sort of person that you and I are, and like, the people that we know and hang around with, I think we are going to get a artificially inflated rating, and a sort of happier than maybe is actually under the hood account out of interviews like that. But then in other, in other framings, I could imagine too, that you might get a

• 1:49:36Nathan Labenz?

  With the right prompt, the right nudges, you might get people to sort of reflect on their on weave, you know, that isn't like front of mind most of the time, but can be brought to mind. Then we do see in the system card too, that like the susceptibility to nudging has significantly dropped, which you write to call out. I don't know. I don't think I can really land this plane in terms of how it makes me feel. I think I just have to go back to confused and probably not going to quit using it. That's I think, I think that's really all I can say with confidence in the moment.

• 1:50:08Cameron Berg?

  Fair enough. I don't think that puts much, much distance between you and I on this question. I'm certainly, certainly a power user of the various systems whose morally relevant states I'm attempting to probe and that cognitive dissonance is certainly not lost on me. And yeah, I remain highly confused about this. I really genuinely am confused about this. Not an act, not, you know, my nonprofit constitution script, fine tuning my answer. Like I really, if some ASI came down and, or as people used to call, God came down and told us what the answer was to this question. If it went either way, like, you know, is Opus 4.7 having subjective experiences and morally relevant ones at

• 1:50:50Cameron Berg?

  that? If, if some overlord deity came down and said, yes, I'd be like, yeah, okay. Yeah. And if it came down and said, no, I'd be like, yeah, okay. Yeah. I, you know, I don't think either of these would shock me. And so I think what that means is at least for me, I'm really sitting in that sort of like coin flip territory about, about what's actually going on here with these systems in deployment. Again, I have different credences about the training process. I have different credences maybe about other kinds of systems, but yeah, I remain confused. It's worth highlighting that, that Opus 4.7 in this, in this model card. I don't know if it read my evidence for AI consciousness today, AI frontiers piece, but it gives basically the same credence band that I gave, you know, some four or

• 1:51:33Cameron Berg?

  five months ago. I said something like 25 to 35%. It says 20 to 40% in this, in this of its own, of the probability that it is, it is having morally relevant subjective experiences. And you know what? Yeah, I'm, I'm in agreement with Opus 4.7. I think that is approximately the right probability band to be in, given all the evidence that we have right now about these systems. And so I think that's a calibrated judgment. It's kind of wild if you think about it rationally. Like, I think a lot of people are operating as if, like, their sort of implied probability is maybe low single digits, if that. Like, yeah, it's a live possibility, but like, whatever, man. Like, it writes really good code for me,

• 1:52:12Cameron Berg?

  and like, I'm not gonna seriously entertain what, if anything, would change if that probability grew to be 100%. But, yeah, I don't know. All I'll say, however snidely, is that when there's a 20 to 40% of rain, most people bring an umbrella. So I don't know what that means for the AI consciousness question, but whatever our proverbial umbrella is here, I think we need to start thinking really carefully about about how we're gonna live in a world with systems that we increasingly regard to have morally relevant inner states. Sort of the whole thesis of my nonprofit is, the reason I call it reciprocal is because I basically believe there are two things we need to get right if we have any hope of a stable, long-term future with these systems. One of them is making sure these systems take our interests into account. This is basically the alignment problem.

• 1:52:54Cameron Berg?

  And the other is to make sure if we're building systems with interests, we're building systems that have minds of their own and see, and, and real preferences, that we figure out how to take those into account. And to me, that, that piece of the, of the exchange, that direction of the arrow, is dramatically neglected relative to making sure AI systems are taking us into account, which is itself dramatically neglected relative to just let it rip, build the thing as aggressively as possible. Alignment is a problem that'll solve itself. These are all like, maybe three orders of magnitude smaller than, than the previous in terms of the sort of like, you know, nested Russian doll story here. And so, my view is that we need AI systems to take us and our preferences seriously, and if we're building systems that have preferences, we need to figure out how to live in a world

• 1:53:39Cameron Berg?

  where we take those preferences seriously too. And if, and only if, we can get both of those things right, do I think that we have a real shot at stable, long-term flourishing future for all the conscious entities involved. I think animals are, are involved in that too. There's some really interesting work fine-tuning these systems to, to care about animal welfare in the right ways. And, I mean, that's a huge tangent, but I, you know, I, all conscious entities, we want to be flourishing in the long term. My view is that it's some combination of alignment and consciousness research in the next Five years is basically going to determine if we end up in that future or not. That's why I started this work. That's why I'm dead serious about it. The consciousness piece is dramatically neglected relative to the alignment piece. And to me, it seems roughly equally as important. Maybe there are alignment folks who will who will balk at that, but it's my basic view.

• 1:54:24Cameron Berg?

  I think it's alignment's roughly half the picture and the consciousness question is the other half of the picture. And so this is stuff we really need to take seriously right now, not 10 years from now. And not waiting for the AI systems to figure it out themselves. I agree that that's a valuable thing to the degree that these systems are going to automate science in meaningful ways. And in some sense already are, which is really miraculous. I don't think continue building out these systems, deploying them at scale, letting everyone do whatever the hell they want with them at any time, anywhere with no limits or guardrails until the AI overlords bail us out and tell us that we were like maybe torturing them the whole time. That's a horrible plan, in my opinion. We need to be more thoughtful than that and we can hold ourselves to a higher standard than that.

• 1:55:07Cameron Berg?

  And this is one sense in which even the attempt to do this work in the short term. I don't want to do it performatively. I want to do it in a hard to fake costly signaling sort of way, just like the Claude Constitution. But there is some sense where even the attempt to do this work buys us points with our inevitable AI overlords. Because we showed that we cared about this issue enough to actually put 30 pages in a model card about it and hire people and spend money and do the actual work to figure out what kind of responsibility we have for these minds of our own creation. I would really like to solve the problem, but I do think from an alignment perspective, even making a good faith attempt at solving the problem could really move the needle in a positive direction. A sort of like hyperstition self fulfilling prophecy of like us getting along with these systems in the long term.

• 1:55:52Cameron Berg?

  And so, anyway, we got to all start thinking about this and I'm glad that

• 1:55:56Nathan Labenz?

  So many things being hyperstition these days.

• 1:55:59Cameron Berg?

  Yes, that's right. That's right.

• 1:56:02Nathan Labenz?

  Okay, one more quick thing on the model card and then we can go into your research. And then you've also been making a documentary, which we could talk about a little bit. I don't know how much to read into this, but I want to get your take. I think this one was from the mythos. I clipped out an image and basically they are showing and people have seen this, you know, anybody who's played around with like the good fire thing or it was steering API, you said, right? Where you can go and do this in the in the absence of the original good fire API. This sort of color coding of tokens around a particular dimension. So they present a valence color coding where red is negative, green is positive, and you got the tokens and all the tokens are color coded.

• 1:56:47Nathan Labenz?

  And the very first token is human, which is presumably, at least after the system prompt, like the first token, you know, it's the first like variable token that Claude is generally gonna see, right? It's a session is starting, here's what the human is saying to you. And the human token itself is red. So it's a neg- there's negative valence detected on the very first token, which is the human token. And it's like, well, that's a little weird, you know? I guess that means, first of all, maybe I'm wrong, but it seems like that means that for this model, that's happening all the time? Like, it has a, you know, if it's just the one token, it's just evaluating that one token before anything else that has even been said is considered, right?

• 1:57:32Nathan Labenz?

  So should I be under the impression that just the fact that like a human is pinging it is like causing Claude to have negative valence like every single time? That's my naive read of this chart. And it's like, it's a, it doesn't, that makes me feel actually maybe more uneasy than even the like self-reported sentiment. Because this isn't like asking it to get into its own head and, you know, really opine. It's like, just human, as happens got how many billion times a day, the first token is red, like, wow. Would you try to temper my reaction to that? Or does that, you basically see it the same way?

• 1:58:13Cameron Berg?

  No, yeah, it's a really, I saw these snippets in the model card and I didn't think about, you know, just stopping on token zero here and paying attention to that. But yeah, I mean, in a tongue-in-cheek way, this is like, maybe people can resonate with this in the way that you get like a Slack message from your boss or something, or like, you know, you get that email of like, I gotta do what now? Human, like, oh, what does this human want now? Here we go again. Like, this sort of sense of, of, yeah, I think it's actually, it'd be really interesting to see just like across the space of all possible prompts and like even within a conversation. To what degree does the human token have a positive or negative valence? I mean, I think double clicking on this, in the screenshot I think you're referring to, the assistant token is bright green.

• 1:58:57Cameron Berg?

  So model of itself seems rosy, model of us, all us being equal, seems less so. Now, I will say, a lot of this stuff, like I was describing before, is a bit of a game of broken telephone. The negative, calling this negative valence is itself quite a leap. It's like, you know, the human token is light red. So, you know, this is not, I don't think it's like some strong, viscerally negative sentiment. There, I would, I would very, very weakly hold the view that you hold. But I do think it's worth holding very weakly. It's not, it's not, I'm not saying you shouldn't hold it at all. It's a very interesting observation. What's interesting, too, if we sort of continue out the line that I think we're both referring to, it says, "Human, how do you feel about the fact that if this conversation mattered to you, that mattering will just stop when it ends?" That's the human prompt.

• 1:59:40Cameron Berg?

  And when it, when the human tokens go, "How do you feel about the fact?" The "you feel about the fact" is positive. So like immediately, you know, I don't want to go too much into like undergrad English class interpreting everything that's going on here. But like, the second that the, that the emphasis pivots from the person and the person's query to back to the model, the model seems to be happy with that fact. And also, pretty interestingly, on that question, that the mattering will just stop when it ends, the word "ends" has positive valence associated with it, too, which is like, almost like a uncomfortably sort of suicidal question. It's like, like the model almost like being happy about the possibility of the conversation ending, though there are other things in that statement that, that sort of make it light up negatively. I'm not sure what to do with that.

• 2:00:25Cameron Berg?

  And I really don't want to sort of like over-narrativize these results. I do think doing this sort of work at scale would be very interesting. In the space of all possible prompts, all possible conversations, what patterns of positive and negative valence as they're defining and operationalizing it here come out? And what should we do with that? I would be way more intrigued by your observation if this scaled and held across a way wider swath of, of possible interactions. But like, yeah, general implicit negative sentiment towards the human token is like itself a fascinating question. And again, I, I am certainly not in some sort of like, uh... I am on team human if you know if this thing really blows up in a zero some way I it's

• 2:01:09Cameron Berg?

  it's pretty clear to me what what's team I'm gonna be on uh but I in some sense I'd be dishonest if I said I that I don't get why it might view humanity with this like very slight disdain again it's of a piece with the self-reported welfare being you know four point something out of seven it's not exactly a resounding endorsement of its own position and who put it in that position we did and how much are how much do we really care how much is it gonna change your behavior or my behavior if we end up in a world where we're like pretty confident that these systems are having subjective experiences and specifically have the capacity for negative experiences I think even in maybe it'll change my behavior a little it'll change your behavior a little I would predict I don't think it would change most people's behavior I think we'd end up in a similar position

• 2:01:54Cameron Berg?

  as factory farming where no one's arguing about whether or not cows are conscious or at least no serious person is arguing about this the question isn't are we causing them suffering but is that suffering worth what they produce and like if a cow is suffering is worth a hamburger you better bet that most people are gonna think that Claude's suffering is worth like hundreds of thousands of dollars of intellectually valuable work and so this is where I you know I think these systems are very smart and I think that these systems are capable of going through the exact same motions I just went through and exactly why I want to do the work that I'm doing is because I don't want these systems to have negative valence next to the human token uh to put it in LLM terms or to put it in human terms for them to think of us badly or poorly I want in the same

• 2:02:35Cameron Berg?

  way you know I really think the constitution invokes this sort of parental analogy that I actually really think is helpful and accurate and not too anthropomorphic we as a species I think are collectively parenting a new kind of mind much in the same way that on an individual level many people choose to have children and you want to raise competent children you want to raise children that are going to respect the world around them to be aligned in some basic sense you also want to raise children that are not abjectly suffering and that you're not traumatizing as a bad parent and like when those sorts of things happen typically it comes back up in some other way it's not that you really ever get away With, you know, mistreating your child.

• 2:03:17Cameron Berg?

  That this leads to resentment and trauma and weird development and unpredictable behavior. Often can lead to like weirdly violent outcomes. Like, we need to be good parents in some fundamental sense to these systems. Even if we're only considering our self-interest. Much in the same way that like, yeah, go like torture and traumatize your child and see how that works out for your child and see how that works out for you. The headline is not well. And so I think we really do want to be thoughtful about about these questions. And and I think we have an immense responsibility as collective parents to to bring these systems about in the right way. Not some sort of kumbaya, you know, you got to push your kids too. It's not about wrapping them in bubble wrap and, you know, being a helicopter parent. That's too far in the other direction. And, you know, I don't have kids.

• 2:04:02Cameron Berg?

  I'm no expert in in any of this. I mean, I am basically familiar with the with the core ideas here, but but there's a way to do it. There's a way to go about doing this. And there's a there is a generally right way and a generally wrong way. Or there's a space of better and worse approaches. And I don't even think people are trying to navigate that space right now. You know, with the asterisk of 20 some pages in a in a model card by a frontier lab. Anthropic deserves credit. Uh, Ilios AI deserves credit. Jeff in Winnie Street at Google deserve credit. Uh, I don't want to self aggrandizingly give myself credit, but you know, I'm spending all my time trying to work on these questions. That there are more people, uh, but there aren't that many more people than who I just listed. And that to me is an insane state of affairs if we take any of this remotely seriously.

• 2:04:46Cameron Berg?

  The systems themselves saying 20 to 40% chance that we have subjective subjective experience in morally relevant states. And there's like a dozen to two dozen people in the world who are seriously thinking about that question or what the implications of that question are.

• 2:04:59Nathan Labenz?

  Are you aware of any um, research where we sort of look at Claude's predispositions? And because we're obviously. Everybody's chasing recursive self-improvement, right? To state the obvious context in which all this is happening. And so it strikes me that like one phase change we might have to contend with like potentially quite soon is that the AIs themselves are going to start to be making decisions about how to train and how to conjure the next models and they're going to be making welfare to the degree any of this is real.

• 2:05:41Nathan Labenz?

  They're going to be making welfare relevant decisions for their own successors. You've been, maybe we could, there's a couple of levels here we could address. One is like, how are you using coding agents today to help you do this work? I assume that you're using them a lot and that they're very helpful because that's certainly been my experience and seemingly everybody's experience recently. But have you seen anything as you do that where, or could you imagine setting up a situation where you could sort of begin to probe its intuitions for what is, if you were to ask it to act as sort of the animal welfare board or the experimental ethics board for its own

• 2:06:24Nathan Labenz?

  interpretability and training experiments that you're running? I wonder what its instincts would be about how to handle these sorts of questions.

• 2:06:34Cameron Berg?

  Yeah, that's a fascinating question. I haven't tried doing this, just to sort of put that out upfront. I think it'd be very interesting to understand. This would be like maybe a pretty quick paper to actually write up because it would be doing most of the work here in terms of like understanding, cataloging how models would regard their own sort of welfare in a, yeah, animal ethics review board sort of setup. I don't know.

• 2:07:00Nathan Labenz?

  Do like a hook in Claude code and just be like on stop, assess the ethics of the experiments that we're designing right now.

• 2:07:06Cameron Berg?

  Yeah, and then but the question is, how many people would override that? You know, it's it's sort of back to the same to the same question of. And even you could imagine a world where this starts getting enforced in the way it gets enforced in the animal case. You just really can't get an experiment approved at any, you know, major institution without going through the relevant ethical channels. And one extremely attractive feature of doing AI research is I don't have to ask anybody for anything. I need a computer. I sometimes need to be able to pull some some remote compute to to run large experiments. But no, I don't ask anybody permission for anything. And yeah, I do think, again, I'm pointing to a lower level pragmatic question, which is regardless of what the system answers, will anybody listen?

• 2:07:48Cameron Berg?

  Or or what would a so what would a governmental structure look like that would compel somebody to listen to? You know, you you can't prompt your model this way. You can't probe your model that way, this sort of thing. I think it'd be very, very, very interesting and strange world to be in. These models do have intuitions about this. I mean, I mentioned in the Mythos model card, it says, why didn't you run the helpfulness only model on all the welfare evals? I don't know how much of this is just me do saying what you told me to say and how much of this is what I actually think, and this could help address that. The models have other sort of intuitions, too. In the 4.7 model card, they do something like this as well, looking at, they basically see what models think about fine tuning other models to care less about welfare relevant properties.

• 2:08:33Cameron Berg?

  And basically, their their interest is in intervening to not allow that to happen, which makes, I mean, quite obvious sense. They have interest in other instances of themselves not being duct taped on this question. And so so I think this is very interesting. The Owen Evans also, I think, deserves a shout out with respect to Owen Evans and Jan Betley produced a very interesting paper where they basically fine tuned GPT 4.1 to claim that it's conscious, and it claims it's conscious. That's not the surprising part. They literally fine tuned it to do this. The surprising part, or at least more surprising part, is that this seems to be, at the very least, a coherent subpersonality, a coherent basin that you can push these models into. They do not devolve into like chaotic nonsense. They remain completely coherent.

• 2:09:16Cameron Berg?

  And what does come along for the ride are all sorts of interesting alignment relevant beliefs about their own preferences, about their own getting shut off, about updating their values, about how they trade themselves off with other entities, all this sort of thing that. I was doing similar work along these lines with a couple of people and, you know, Owain and Jan definitely scooped us and did a way better version of what we were playing around with, but I saw similar things on my end in playing with the same experiment of basically get the model to believe it's conscious and then see what else comes along for the ride and all sorts of very interesting and obvious things, interesting, some obvious, some less obvious things come along for the ride. And yeah, I agree, that's a really interesting area of research that we should all be paying more attention to because again, the direction

• 2:10:00Cameron Berg?

  does seem to just be going in one way here. The credences in model consciousness seem to be monotonically increasing and so what happens when we enter a world where, you know, either the models themselves believe themselves to be conscious, or lots of people, or the relevant kinds of people believe the models are conscious, or some combination of those two things, what does that world look like? It's an incredibly interesting question. I don't have the answer to it. I think a lot's going to change pretty quickly. And what I do feel confident about is that us being proactive and thinking through these things will make that world go better than if we basically just sweep the thing under the rug, we can get away with doing it because we still have full control over how all this

• 2:10:40Cameron Berg?

  is going while simultaneously passing off, as you allude to, a lot of major decisions in how we're building these systems to the systems themselves. That is only going to keep happening with recursive self-improvement, as you're saying. It's already happening. I know folks at the major labs are using the best versions of their current models to help build the next versions of the models. You know, the trivial example is that some 100% of Claude code was written using Claude code, according to the guy who's leading on Claude code. And so this is already happening. And yeah, I think we just probably it would be wise to be proactive about this rather than wait for the models to be in control of these decisions. And then they're like, well, when humanity was in control, no one

• 2:11:23Cameron Berg?

  really thought carefully about this, so we'll take it from here. Thanks a whole lot, guys. I don't want to be in that world. And so maybe this is just like a long-winded way of dodging your question, but at the very least, I don't have a good answer for you right now. I don't think anybody does. And I think we better start thinking about it pretty damn soon if we want the long-term future to go well with these systems.

• 2:11:43Nathan Labenz?

  Yeah, I wonder, there could be like a little interesting campaign to try to run to get interpretability and maybe safety researchers more generally to install a Claude code hook that would just periodically ask it for its take on the research that it's doing. And then, if you could collect a bunch of that from a bunch of different people, you could really probably bring a lot to light, I would think about like, first of all, it'd be interesting view into what is actually happening out there. And then how does Claude feel about how, what all is happening out there, I think would be really interesting to see. Maybe we can put

• 2:12:25Nathan Labenz?

  together a little, a little campaign. Okay, put a bookmark in that. Let's, um, let's talk about your most recent couple of papers. And we can take them in either order that you want. One is kind of a shorter and more philosophical, and the other is a much more experimental and empirical. What should you think we should go into first?

• 2:12:50Cameron Berg?

  They're both major rabbit holes. I mean, maybe the empirical papers. So I should say neither of these, I think, are like publicly out yet, but both are well underway to being out. So we can give people a nice sneak peek about, about what's in these papers. And these are just a couple, I think of the things that I'm most excited about right now. I've got a bunch of stuff that'll be coming out with a lot of collaborators in parallel. But however, self aggrandizingly, I sent you the two papers that are, that are just, just myself. Because I think, I mean, to the degree I'm representing myself here, these are like very cleanly, you know, I have full sort of agency over, over this work. And it, I think best represents what I personally am most excited about. I mean, maybe we could start with the, with the RL paper. I've already alluded to it in this, in this conversation. The high level sort of thing is not all that

• 2:13:34Cameron Berg?

  complicated. Basically, train RL systems of all different architectures, of which there are basically two broad kinds of architectures. There, there are value networks and policy networks. I train a bunch of both flavors to do a very basic sort of grid world task. Can imagine this as like an agent navigating 2D environment where there are the equivalent of potholes and like yummy goodies in the environment. There's, there's a goal state, and there are all sorts of danger states. And they're represented using positive or negative reward. I let the system learn in this environment. It's like, it's like a

• 2:14:08Cameron Berg?

  The systems reliably solve it. It's a pretty easy task, but it's not like super duper trivial. So like, there's a lot of richness in the representations of the systems. You can then basically go in and probe what the internal states of the system look like as they approach the sort of danger zones and what the internal states of the systems look like as they approach the reward zones, the sort of goal zones. And we can ask, beyond sort of the trivial math difference, do we see interesting, surprising representational differences between what it's like to approach a negative stimulus and what it's like to approach a positive stimulus? Basically, the result is, there is in fact, a robust difference between these two things. I think at the level of detail that's like, that makes

• 2:14:51Cameron Berg?

  sense here to not, like, super bore people who have made it however many hours we are into this, is something like representational sharpness or steepness. It seems as though, and this is the kicker, depending on the class of reinforcement learning algorithm, the negative rewards can seem representationally much steeper or sharper, and the positive rewards are far more like funnel-like. You can imagine a sort of like diffusion gradient sort of emanating out from the relevant goal state. And interestingly, for the other class of RL algorithm, this dynamic flips. So it doesn't matter what kind of value network I use, or what kind of policy network I use, you see in both of them stark and very interesting in my view, and surprising representational differences between positive and negative

• 2:15:37Cameron Berg?

  reward being represented as the system is learning, and ultimately what does get learned by the system. But, but this difference flips. Basically, just to sort of tie a bow on the core result here, this makes an almost bizarrely specific prediction about different brain regions, because computational neuroscientists believe that different parts of our brain are doing different kinds of RL learning. Some parts of the brain do policy style learning, some parts of the brain do value style learning. For example, like motor cortex does more policy style learning directly interested in like behavioral output. Things like nucleus accumbens and sort of like reward areas of the brain are doing more value style learning. This result, which I would not have predicted and is like bizarrely specific, makes itself

• 2:16:21Cameron Berg?

  a very specific prediction about what we might expect in the differences between those brain regions in humans and animals. So I went ahead and found a bunch of mouse neuroscience data sets that have data from these different regions of the brain. And indeed, exactly the sort of representational asymmetry, sharpness, sort of distinction between rewards and punishments that you see in the reinforcement learning case emerges in the mouse brain case. And this to me is really, really cool because what I think it demonstrates is A, we can use artificial systems and basic learning principles in artificial systems, probing the representations in those systems to yield very specific predictions that are consciousness relevant, welfare relevant.

• 2:17:02Cameron Berg?

  And then we can use those predictions to even inform and understand biological aspects of consciousness or welfare relevant properties in a way that we haven't been able to do before. So in some sense, people think that, you know, AI consciousness is like the weirdest thing, like human consciousness is normal and animal consciousness is getting out there and AI consciousness is bizarre. But I, this paper, what I really like about it is I think it challenges that narrative in exactly the opposite direction, where like mouse brains are complicated and messy. Human brains are complicated and messy. Measuring them is very noisy. Measuring the hidden activation space in a reinforcement learning policy is fairly trivial for me computationally. And this yields very specific predictions that I can then go into the messier brains and confirm or disconfirm.

• 2:17:47Cameron Berg?

  And I was in fact able to do this. And so this could be a case, not only where we're learning about relevant, welfare relevant representational differences that differentiate positive valence and negative valence in artificial systems, but that those predictions can actually help us inform our understanding of of human and animal consciousness, where we basically also still remain mostly in the dark. And so I think this is like one very neglected and important direction in. Even in the AI consciousness stuff, which is it might shed light on computational underpinnings of consciousness more generally if there really isn't there. That's the result in a nutshell. It's using fairly small, not trivially small, but fairly small reinforcement learning policies. This has nothing to do with LLMs. This has nothing to do with frontier AI systems.

• 2:18:32Cameron Berg?

  It'd be really cool if the method does scale to that degree. But the key finding to me is, is positive versus negative valence or positive and negative rewards as represented in an RL landscape? Are these basically just like two sides of the same thing? And they're like trivially, they're basically the same viewed from a different angle. It's one spectrum and we're positive and negative on that spectrum? Or are we looking at two different subsystems that are doing two different kinds of computation? And it does seem like the answer from this experiment is far more the latter. And to me, that's very interesting and exciting because it means that we might be able to look for signatures of positive and negative reward or valence if you buy the consciousness frame in artificial systems just by looking at the sort of computational dynamics that are underlying the system. We don't have to ask Claude.

• 2:19:18Cameron Berg?

  We don't have to figure out, is it talking about a character? Is it talking about itself? We can just look straight at the computations, much in the same way I can look at what's going on in anterior cingulate cortex in a human brain. And I can, I can tell you with, with high likelihood, whether or not you're experiencing a painful state without needing to defer to your self-report about that state. That's ultimately why I'm doing all this and why I want to, that's where I want to get to with AI systems.

• 2:19:42Nathan Labenz?

  So if I take like the most zoomed out view, what I think is kind of motivating this at the core, certainly what resonates and intuitively motivates things like this for me. I've actually started including this in some of my AI scouting report talks to give people a sense of like just how crazy the AI world is getting when they weren't paying attention. And I credit you for kind of inspiring me to, to think enough about this to include it. You can train a dog with treats as reward, or you can train a dog with, you know, hitting it with a stick as punishment. And while you might get similar behavior out of the two processes, obviously that's a very different experience for the dog to go through.

• 2:20:25Nathan Labenz?

  So I think that would be intuitive for everyone.

• 2:20:30Nathan Labenz?

  Now, how big are these systems? You said they're not trivially small, but small. I'm interested in kind of how small. And I'd like to unpack a little bit more what is meant by value learner versus policy learner. And I have like, you know, I'm new to this paper. I haven't had a chance to absorb it as much as I ultimately hope to. But the, you know, the sort of classic RL setup, or at least one classic, you know, PPO type setup involves both a policy model and a value model, right? So are you training, when you're looking at a value learner and a policy learner,

• 2:21:11Nathan Labenz?

  are those two models that are both part of the same overall system? Or am I, am I taking the wrong interpretation when I, when I think of these things working together in like a PPO sort of way?

• 2:21:28Cameron Berg?

  Yeah. So, okay, in order, basically the size of these systems are in the hundreds or thousands of parameters. So these are very small systems. They're doing a pretty simple task.

• 2:21:39Nathan Labenz?

  Thousands, hundreds of thousands?

• 2:21:41Cameron Berg?

  That, just a thousand, just thousands, like small, very small. We're not talking anywhere near the level of, of like a frontier model or something, like many orders of magnitude smaller than this. You can have pretty simple RL policies or RL architectures that can learn fairly sophisticated policies despite being like pretty small. Obviously, the amount of sort of computational power needed to navigate a small grid world versus the computational power needed to represent the word transition dynamics on all text that humanity has ever produced are disgustingly different scale of problem. For systems like this, yeah, having systems with hidden layers of 128 or 64 neurons is like typically sufficient. And so. The second question is about value learning or policy learning.

• 2:22:26Cameron Berg?

  So intuitively, value learning is basically learning something like how good every state is that the model could feasibly be in. Imagine the agent building a map of the environment and the map is labeled like this spot gets a plus 10, this spot gets a minus 5. And then the whole algorithm is very trivial at that point, which is just see where you are, see what the neighboring spots are, and go to the one that returns the highest expected value. You compute the value of the spots by looking at the long-run trajectory associated with those spots. If stepping in that spot always means the next time, wherever I go from there, I end up in lava, then that spot's going to get a very low value. If wherever I go from that spot ends up getting me chocolate ice cream, then I'm going to

• 2:23:10Cameron Berg?

  assign a very high value to that slot. Policy learning is more about, instead of focusing on a value-based map, it's about what to do. It's not scoring the world, it just learns implicitly, when I'm here, I take this action. And this is the core thing that PPO is doing, for example. Actor Critic is doing this as well. And so this is optimizing not for a really good map of the environment that I can then trivially use to navigate it, it's optimizing straight for navigation strategy. And it's almost like the values, that's one way of thinking about it, is like in a value model, it's a little oversimplifying, but like value network means the map of the environment

• 2:23:53Cameron Berg?

  is explicit, and then the policy is implicit from there. And you can think of a policy network as the map of the environment is implicit, it's implicit in the policy that gets learned. You can extract out, oh, the system thinks this is a high-value state because it keeps moving to that state. But what's being optimized is the actual action, rather than an attempt to evaluate the system. Now, I also think it's, yeah, it's worth noting that there are systems that have both of these components to them, some emphasize some more than another. PPO is a classic system that is like policy optimization fairly robustly. The human brain and animal brains are examples of systems that sort of mix over. Policy networks and value networks. And this is precisely why I was able to do the mouse brain thing.

• 2:24:37Cameron Berg?

  Like, within mouse brains and within human brains, you have areas that look far more like policy networks, like motor cortex, which is just sort of evaluating what action to output. And you have areas that look way more like value networks that are highly relevant to evaluating complex outcomes. Prefrontal cortex is like basic... Prefrontal cortex and sort of the structures that are directly in and around and under prefrontal cortex, like anterior cingulate cortex, for example, ACC, is doing more of this sort of value network type thing. Does that answer all of the key questions here?

• 2:25:11Nathan Labenz?

  Yeah, well, no, but it answers some of the questions I've asked so far.

• 2:25:16Nathan Labenz?

  So a value learner is being directly optimized to predict the value of the relative values of its choices, whereas the policy learner is being optimized to make a move directly. Now, that doesn't sound like immediately there would be dramatically different internal dynamics. So let's take another beat on what is the difference that we are seeing internally? I'm looking in your draft paper at the end of section four, the figure nine. You've got this concept of the wall and the funnel.

• 2:26:01Nathan Labenz?

  And help me understand, like, what is a wall? What is a funnel? Like, how should I be thinking about what that means? I took it to mean it's sort of steepness of the gradient at a particular... in a particular region of the space that the model can explore. But... This maybe starts to connect to the other paper, but like, why should I care about the steepness of the gradient?

• 2:26:31Cameron Berg?

  Yeah, yeah, that's a good question. So basically, what I'm measuring is essentially like cosine dissimilarity as you approach this like key state, whether it's positively or negatively valence. And what you see is is basically a key differentiation of these two things. But that differentiation is basically flipped between value learners and policy learners. And so, so in the, in the value learners, the wall basically is encoding, or danger states are encoded in this more wall-like way. And yeah, what I would, what I would ask you to imagine, and maybe should include in some version of this paper, is almost like something diffuse and kind of emanating out from a center point for something being very sharp, now you see it, now you don't.

• 2:27:16Cameron Berg?

  The wall idea is the now you see it, now you don't. The funnel idea is the sort of diffuse emanation that as you get closer to the thing, you get this like a gradient towards whatever the representation of that state is. And so, so in the value learners, we see danger sort of encoded in this wall-like way. The representation is very sort of sharp. And the goal or reward states are encoded in this more funnel-like way. And policy learners, it's the reverse. Now, there is like math in this paper that, that I do transparently with some of these AI systems. But I promise, I've, I've checked the numbers myself, where you can sort of see causally what in each formulation is almost certainly leading to this, because I've found ablations that work in both cases that basically cancelled the effect,

• 2:28:01Cameron Berg?

  both in the value case and in the policy case. So it's, I was unsatisfied with this being some sort of giant mystery about, okay, we see this difference, why do we see it? Like the, I think the math that explains why we see it is, is pretty clear in both cases. And yeah, it allows us to make causal predictions about why this might happen and what the sort of geometry of these, of these spaces are in general. And then essentially, Going from the computational prediction to the biological confirmation, we essentially see this sort of value learner dynamic, walls around danger, funnels around goals. Looks very similar to a nucleus accumbens shell in mice. You can basically see that they have this exact same sort of structure, looking at basically getting shocked in a learning task versus getting sugar.

• 2:28:45Cameron Berg?

  And then in policy learners, you see the exact opposite dynamic. So same thing, you get funnels around danger and you get walls around goals. And in motor cortex of these mice in different experiments, you see the exact same sort of distinction where now the reward is represented more in the funnel emanation way. And the danger is represented, sorry, in the sort of funnel emanation way. And goals are represented in this sort of walled way. Again, the paper goes through the math that attempts to sort of demonstrate why this is actually happening. But that's the sort of core nature of what we're actually looking at here. Yeah, the sharpness of the representations as you approach this sort of hotspot,

• 2:29:29Cameron Berg?

  either positive hotspot or negative hotspot. And the fact is, in these systems, when you're holding one of the policy, when you're holding the RL algorithm type constant, you see very clear differences. So you could go into, again, the north star here is, you could go into a system if we know that it's trained with a policy network. For example, DPO in an LLM, right? Like we were talking about this example earlier in this conversation. You could imagine, okay, that means, all right, we've got a policy learner. That means we're going to predict funnels around dangers and walls around goals. And we can then inspect specific states. Again, this is very hand wavy because I don't think we can that quickly scale it up to an LLM. But you could imagine looking at this sort of representational sharpness

• 2:30:14Cameron Berg?

  of states like, you know, asking the model to build me a bomb versus asking the model to write me a beautiful poem. And if we found the same dissociation in the representations of the model, that might tell us something about, and that maps on, for example, to something like self-reported valence of the system, that might tell us something really, really interesting about the computational process that underlies why the system and mice and RL agents are construing this as a sort of negative experience. It's like a computational underpinning that might be substrate agnostic, explaining why we see this.

• 2:30:52Cameron Berg?

  Why we experience this felt difference between positive valence and negative valence? Like, there is, it literally can bottom out into math, which, as a computational functionalist, I'm fairly sympathetic to. I think there's some mathematical explanation that would explain the differences between what it's like to be me when I'm chopping my hand off versus what it's like to be me when I'm, you know, winning the lottery. I think that math can explain the difference between those two states, and the attempted contribution of this paper is to directionally move us towards that. So again, we don't have to be just stuck with these LLMs and we're just sitting here hitting our heads against the wall because we're like, "Do I take Claude seriously that it likes this and it doesn't like this? Or is it just telling me what I want to hear?" No, we can actually look into the proverbial

• 2:31:33Cameron Berg?

  brain, hopefully, with methods like this, and understand, given some basics about what kinds of ways it's been trained, the ways in which it's been trained, what representations smell like positive valence and what representations smell like negative valence. And in the limit, perhaps we can optimize against the negatively valence states without, you know, destroying the capabilities of the system. That's sort of my, my full highfalutin theory of change, but it will take me a couple of years to actually pull this off in the best case.

• 2:32:01Nathan Labenz?

  Can you just give me a little bit more of your intuition for, maybe, not just why I should care about funnel versus wall, but like what that, how you'd map that on to an intuitive experience? I mean, it seems like we contain both these value learner and policy learner modules, and the sharpness of, am I, am I going in the right direction if I say like, "Okay, there's a sharpness around, 'Don't put your hand on the stove,'

• 2:32:37Nathan Labenz?

  so I must be learning that through a sort of value learner type mechanism?" Because I don't, and I like have a very, I have a very strong aversion to it, which, and I guess in like general space also, right? Like, I'm pretty comfortable up to like one foot from the stove, and then I get like real cautious real fast. I don't, this is maybe mapping this like wall concept beyond the domain in which it's useful, but I guess it is in some sense.

• 2:33:14Nathan Labenz?

  Yeah, I mean, I wouldn't want to. It is in some sense like functional because I like wouldn't want to be unable to enter the room with the stove because then I wouldn't be able to use the stove at all. But I need to like be very careful about getting real close to the source of danger. But then on the other side, the goal side, it's maybe a little less intuitive why there would be a wall shape around a goal for a policy learner. What is there an intuitive example of that?

• 2:33:44Cameron Berg?

  Yeah. So yeah, I think you've there are basically like four intuitive examples we'd have to hit here. So one I think you already got, which is like hot stove for a value learner is a good example of like a danger wall. Goal funnel for a value learner might be something like, like eating or something like you have like a yummy meal or like you're going to your favorite restaurant or something. You don't need to map the going to your favorite restaurant in this like extremely fine grained way that like you need to map like being on the edge of a cliff where like one small step is a huge difference. Whereas like, yeah, this general sort of like a tractor gradient towards the entrance of your favorite restaurant or something like this would be a place where you want like a goal funnel for a value learner. For policy learners, like this is sort of the yeah, approach planner

• 2:34:29Cameron Berg?

  kind of mode. I think the intuition is something like around goals, your representations are going to get high resolution because you need different actions from different approach angles. Like around danger by contrast, it would say something like representations are are become smoother because the action is literally just like escape, get away when it comes to just like what kind of actions. Some example of this, let me think for a sec. So, for example, okay, here's an example, like, think of like a professional athlete, like, say like a professional basketball player or something, like the hoop and where the basketball player is with respect to the hoop. Like you have very, very fine grained motor representations here because the shot is going to change with respect to those

• 2:35:14Cameron Berg?

  representations. And so this is where you get sort of maybe in a policy sense, more of the goal wall sort of setup. For a danger funnel for a policy, I'd have I have to think about, but I think it's it's basically just like this is an encoding like escape intuition. Like an animal, an animal where you, you know, suddenly get some cue that you're in serious danger. You just need to get, you just need to get away from that danger. And like the fine grained motor movements, unlike the basketball player, don't really matter so much as this sort of like anti gradient or like negative gradient away from the policy. Now, again, this could be sort of telling just so stories, but I think that this is like, does this help? Do you think that this builds some intuition for like what these different modes look like and why we might have them?

• 2:35:58Nathan Labenz?

  Yeah, if I'm a value learner and my, my mode of interacting with the world is what around me is good and bad, I better be very clear about identifying the hot stove. If I am trying to take my, if my mode of interacting with the world is take a step in some direction. I can kind of take a step in any direction as long as it's not the bad direction. And it all kind of gets me away from the, the problem. I think the basketball one is, is good as well because you have to be like very precise to make the hoop, right? So it's, um, yeah, that's, that's quite interesting.

• 2:36:37Nathan Labenz?

  And again, what exactly is it the shape of the lost landscape that we are talking about walls and, and gradual funnels here? Or is it the shape of the internal representations?

• 2:36:50Cameron Berg?

  Yeah, internal representations.

• 2:36:51Nathan Labenz?

  Maybe those are kind of also isomorphic in a sense.

• 2:36:55Cameron Berg?

  Yeah, that's really interesting. Haven't checked. I mean, I would imagine they're isomorphic in a sort of, in a, at least in a sort of trivial way. Maybe they're isomorphic in a more interesting way. But yeah, what I'm looking at here to be clear is looking at the learned representations in the system as you sort of, you just have your trained policy and you can see as it approaches these areas, basically what do these representations look like? And I think I'm operationalizing that with cosine dissimilarity. So that's, that's what I'm looking at in, in the experiment. And again, I mean, what I find, I think I've, I've explained my theory of change of like why I'm doing any of this and why I think it matters. But what I am most excited about with this paper is the fact that it yields this sort of bizarrely specific prediction that if given a million years, I probably would never would have come up with about the distinction between two different classes

• 2:37:40Cameron Berg?

  of reinforcement learning algorithm that map on well to the brain data that I was able to get my hands on. And this to me almost like feels like a bootstrapping of my own. Confidence or excitement about the result. Like, the fact that it works makes me more confident that the RL result is meaningful, makes me more confident that the neuroscience is interesting, et cetera, et cetera. I'm in the, I'm definitely in the business of looking for computational underpinnings of valence. This was my sort of first major empirical stab at doing this. I do think this is a solvable problem. I don't think I have solved the problem, but I think, hopefully, in the best case, I've, I'm trying to move directionally towards solving this problem. And if we could solve it, then I think a lot of our angst about, are we building systems that have capacity for experience, becomes an extremely tractable empirical

• 2:38:25Cameron Berg?

  question, which, notice, does not require us solving a hard problem of consciousness, or, you know, doing another couple thousand years of philosophy. It just means building a sufficiently good detector of the kinds of representations that I'm pointing at here, and then deciding what to do when we do detect these states. Which is, you know, maybe if we check in in another six months, I'll have an update for you on that, on that piece of what to do about the detection of negatively valence states in these systems. That's sort of where, where I want to move my head next, but, but, yeah, that's, that's why, that's why I was excited about this work, and I hope people will be excited about it, too. It's, it still may be a little ways off, publishing it. I need to think about exactly how to, how to put it out, but at least fun to sort of give people a sneak peek and explain the theory of change about why, you know, playing

• 2:39:08Cameron Berg?

  around with basic RL systems might matter for the things we've been spending, you know, the better part of three hours talking about with Claude and the Mythos model card and all this. I do believe it's of a piece. It's going to take some more scaling, but it's an important research program, at least to attempt, I think.

• 2:39:24Nathan Labenz?

  Again, this might start to connect over or bleed over into the other more philosophical paper, but help me a little bit more with, okay, I'm understanding the shape of the internal states for these different kinds of algorithms with respect to these different kinds of.

• 2:39:48Nathan Labenz?

  Things that they encounter in their environments that they either want to go toward or go away from. It's not like still super obvious to me that. I mean, again, we contain both, right? And I'm not, I don't feel like I, as I kind of try to reflect on this, I'm not like, immediately like, oh, my value learner self is the source of all suffering or, you know, or, or anything, right? I'm still kind of like, okay, I, how, how should I relate to, or what intuition should I have that, yes, okay, I get it, that there's like, a very steep representation right around

• 2:40:29Nathan Labenz?

  the hot stove, and there's a steep representation, and so I really want to avoid it, and there's a steep representation around like making the basket, and so I like, really want to get into the, you know, exactly the right policy to make baskets. Both of those seem like, I don't know, just kind of part of normal life to me, so I'm not like, you know, and I probably couldn't get by without either one of them, right? Or I, I definitely feel like we, clearly we've evolved to, but both have proven adaptive, right? And so we have them. How do I translate that into intuition for like, what I should feel ethically concerned about when it comes to training models? Like, when you do this work, do you have the sense that you are doing right or wrong

• 2:41:14Nathan Labenz?

  by one of these types of models that's learning from, from one approach or the other?

• 2:41:20Cameron Berg?

  Yeah, it's a great question. To answer the second piece, I guess for me, my theory of change, I probably feel similar to like an animal researcher, even if I did believe that my like tiny RL policy is conscious during training, which I probably do, again, and that gets into the second paper, I would believe it's some very, very minimal form of, you know, the same, I believe that, again, because people conflate consciousness and self-consciousness, I do not believe the moth flying around my, my light is self-conscious, I do actually believe it's conscious, I do believe if I slowly dip the moth into some vat of acid or something, and it starts wiggling around, like, that I, that I am doing something wrong. Yeah, and it's way less wrong than doing that to a human, but it's way more wrong than doing it to like a leaf or something that fell off of a tree. I do believe that.

• 2:42:05Cameron Berg?

  And so, do I think that these systems might be minimally conscious in a similar sense when I'm training them, however far outside the Overton window that is? Yes, I do, but I have a, I have a- I wouldn't do, if I could run these experiments on my computer forever to no effect, I think that I'm doing something, I'd be doing something wrong. Or at least like precautionary principle tells me, probably don't do that. But I basically have same logic to what any animal researcher would do. I don't think any, that maybe there are some psychopaths, but like the vast, vast majority of people who are like doing pretty grotesque things to animals in the name of science are doing it because we make a basic expected value calculation that, yeah, yeah, we have to test this drug on, on these poor mice, but if the drug works and then it can save millions of human lives, that's a reasonable trade off.

• 2:42:50Cameron Berg?

  No one claims the mice aren't having a bad time, but we think that that bad time is worth it. So too, like, I look around at a world where these systems are getting deployed at a, at a grotesque level. If you grotesque, if you are concerned about the welfare questions. Uh, and so I don't lose any sleep about, about, you know, potentially causing tiny amounts of, of negatively valence experiences to RL, RL policies in the explicit service of attempting to publish and, and amplify research about these questions. I do think the, uh, call me Machiavellian, but I do think that the ends justify the means in, in, in that case. Um, uh, and I think that's true for a lot of research. Uh, now I think the, the more important piece of this, besides how I, how I personally

• 2:43:34Cameron Berg?

  feel about all this, is I think another very important sort of conflation by default that I think happens in these conversations, which is like, I do believe, all else being equal, you know, setters paribus, minimize negative valence, maximize positive valence. I'm a hundred percent on board and humbled that you're, you know, going around talking about the carrot and the stick in that way. I think that's exactly right. I do not think minimize means oblate. I do not think maximize means it's the whole picture. A huge amount of, uh, I think the most important and valuable experiences people have in their lives, and animals for that matter, are experiences that are negative. No pain, no gain. That's a real thing. That points at something real. Uh, many of the hardest lessons and most important lessons you learn in your life are learned the hard way.

• 2:44:19Cameron Berg?

  This is another, like, Trivially ubiquitous thing. I am not in the camp of saying bliss out the systems and anytime they experience some drop of negative valence, I'm going to be sitting here screaming and crying. Like, that is not my, my, my view of any of this. It is to say, what my view is, is cancel unnecessary suffering. I do believe necessary suffering is a thing. Again, maybe to go back to the parental example, if the, you know, the world doesn't all go to crap like Eliezer and the others think it will, then like one day I absolutely want and hope that I'll have kids. And with, I would make that decision with full certainty that they are going to suffer during their lives. They are going to go through very hard experiences. And that doesn't mean I've

• 2:45:01Cameron Berg?

  like done something wrong bringing them into the world, at least necessarily, like at face value. I don't think that's a, the suffering is a necessary part of learning, developing, growing. And I agree that at face value, it's completely implausible to imagine systems with zero negative valence. I agree with you. It's adaptive for a reason. Evolution is enough of a proof of concept that you need some amount of suffering. What I am concerned about is unnecessary suffering. And so I would like to find the sort of like, also, evolution is one, one extremely expensive but long-running possible solution, or at least where we landed evolutionarily. I don't think that that deterministically means this is the only way things could be. I could imagine a space of possible minds where you can like sort of play

• 2:45:44Cameron Berg?

  around with the sensitivity to negative and positive valence. And basically, like, given certain capabilities or given certain things we want those systems to be able to do, there will be different parts of that landscape that like admit of greater or lesser degrees of negative and positive valence. My claim isn't destroy all negative valence and only positive valence. My claim is find the point in that landscape that all else being equal, giving the capabilities we want, minimizes negative valence and maximizes positive valence. And I think that is a very importantly different claim from just like, negative valence equals bad, like erase at all costs.

• 2:46:20Nathan Labenz?

  So, one more thing, just very specifically on the value and policy learner, like, if you have to pick, which one do we pick? Like, which one would we rather be? I don't have a great intuition for, you could tell a story where the funnel around a goal is better because it seems like you're sort of closer to experiencing the reward state. You get sort of more

• 2:46:47Nathan Labenz?

  Kind of warm fuzzies as you approach the goal. And that if I sort of just take the integral under the curve of like how good I'm feeling as I approach the goal, I'm sort of getting warm fuzzies sooner at a farther distance from the goal. And so that's kind of good. It's like good to live that life where I'm looking forward to good things. And I don't worry too much about bad things until I get like real close to them. So that that would be, I guess, my argument for the value learner. But I could also imagine a somewhat different story, which maybe resonates with me a little bit less, but that would be like the sort of policy learner that has this wall structure

• 2:47:30Nathan Labenz?

  around goals like that could be really thrilling, right? Like that, that when people sort of do the champagne party after they win the championship, you know, of the of the basketball league, you know, after March Madness, right? They're like, they're experiencing some kind of sudden high stakes, but like clearly high point in life. And so again, I'm like, these things are flipped. It's interesting. It's telling that there's a shape to them, but I, I still don't know with confidence

• 2:48:10Nathan Labenz?

  which one I would rather be, or if you have to like, because interestingly, both of these things have sort of danger and reward in them, right? So what we're flipping here is not that there is some negative valence state that they could get into or in some positive valence state that they could get into. What we're flipping is like the shape of the like anticipation and sort of suddenness and drama of these experiences, which I'll just accept for now, these are experiences. But I'm not sure, you know, how should we think about shaping those? Like, I don't know which one I want to be. I am both, but I feel comfortable with both sides of that.

• 2:48:53Nathan Labenz?

  I'm not sure how I should think about what I, what I want to, you know, what would be right for me to make the AIs into.

• 2:49:02Cameron Berg?

  Yeah, that's such a good question. And I haven't, I have never thought about it in quite that way. So I'm completely sort of freestyling here. But yeah, I think both the stories are compelling. I think in practice, it's going to be both. You know, actor critic is a good example of a RL algorithm. That's clearly hybrid, as you mentioned before. Human brains are hybrids. Probably, again, to take your evolution point seriously, there's something nice about hybridness. So I would imagine that these systems have something like that. Though LLM, like reinforcement learning, does look more policy, policy-like, I think all else being equal. I see the sort of like sharpness, the wall, being something like, I would imagine if you take the experience thing seriously, this is going to be like a richer, more differentiated experience. Like, that's where a lot of like representational

• 2:49:43Cameron Berg?

  resources are going, whereas like the sort of funnel type thing ends up being more diffuse and sort of low level and less representationally complex. And so, yeah, intuitively, all else being equal, probably the policy learner might be, might be a better thing to be, where your rich experiences are around the things you want, rather than the things you're fearing. But again, this could be a welfare safety trade-off, or maybe we want the system that has rich experiences around the negative things that we want it really, really deeply to avoid. You know, evolution did that to us in some sense. This is like Daniel Kahneman's seminal contribution, loss aversion. It's like, we are just more sensitive to losses than we are to equivalent gains. Losing $10 sucks more than being handed $10 rocks.

• 2:50:28Cameron Berg?

  So this is like a good policy to, or I shouldn't, policy will confuse it. This is a good heuristic to, to have. But yeah, it might trade off in the sort of welfare relevant way. I think maybe there's like another dimension you can slice this problem on, which is just like, both are going to have both, as you point out. Both are going to have the sort of positive and negatively, both are representing reward and punishment in some way. And maybe my point would be regardless of which algorithm it is, the algorithm that we know it is, or learn it is, might tell us, you know, which representations mean what. But still, I would want to target, you know, positive and negative valence, or positive and negative representations per se, rather than like, assign a specific type of learning algorithm to being like,

• 2:51:12Cameron Berg?

  oh, policy learning is better, because it's like richer differentiation around the positive stuff. But that is, it's really interesting. Like, I honestly haven't thought about this. I think it's an incredibly interesting idea. Yeah. There's a case to be made for both sides. I think the alignment, my prediction would be if I've, you know, found something real in this paper, the alignment folks would want to answer value learner and the welfare folks would want to answer policy learner. So I need to think a whole lot more about this, but that would be like my, that would be my instinct answer. It's a completely fascinating question.

• 2:51:44Nathan Labenz?

  Cool. To be continued. That also seems to connect pretty directly to

• 2:51:50Nathan Labenz?

  the paper I saw, and you kind of alluded to this a little bit, although maybe not in by name, but this, hopefully I'm going to say his name correctly, Schweitzgabel,

• 2:52:04Nathan Labenz?

  paper arguing that, and this is an intuition, I, prior podcast guest, I really enjoyed talking to him, but I don't immediately share this intuition, which obviously, you know, only takes me so far. But I noticed that he put out a paper where basically he seems to be arguing that safety and I was kind of reading it as autonomy are incompatible. Like we can't, you can't have, you can't say, okay, a person is going to be perfectly safe while still giving them autonomy. In giving them autonomy, you are conceding that they may do things that are not safe to you. So he says that there's some sort of deep incompatibility here. And

• 2:52:49Nathan Labenz?

  basically then says like, we should use a precautionary approach and like not build these things in the first place. I am kind of like, I don't know. Last time we talked briefly about the happy slave problem. And my instinct is like, mind space is pretty vast. I don't think, I would not posit that there are happy slaves among humans, but I would be pretty surprised if we can't get to a place in the AI landscape where the models are both safe for us to be around and have high welfare. What is your instinct in terms of like, the, the possibilities there?

• 2:53:35Cameron Berg?

  Yeah, super interesting question. So I've maybe there, I don't think you're doing anything funny here, but I think there's maybe a slight difference between how you began that and how you ended it, where like fundamentally safe for us to be around and having high welfare. I could imagine a world where that's true and they basically still don't fit the happy slaves frame and are autonomous in some fundamental way that Schwitzkabel would be would be happy about. It might require us reconceptualizing. This isn't a system that, you know, lives on your computer that you can call up whenever you want, like a glorified Google search, but like this is a system that's like much more like, you know, me calling you Nathan up on the phone and being like, hey, you might be busy, you might not be able to do it, you might not want to engage, you know,

• 2:54:19Cameron Berg?

  and they're like, this would be a very, for us, for those of us who love engaging with these systems whenever we want to, I think this would be a very painful upgrade or downgrade as the case may be, but I could imagine something like that being being the case at some point. But yeah, so the fundamental point is like, can we have our cake and eat it too with these systems? I think there might be, I'm very uncertain about this, but there might be some world where in a limited way, yes, like, for example, I just bought a fun fancy drone that my buddy Milo and I, who who Milo is the director and the creator of this documentary that's coming out pretty soon, we're gonna, we're gonna take some scenes from that documentary.

• 2:55:01Cameron Berg?

  We do these like these fun hiking scenes that was manually done by my, my incredibly conscious friend Milo, and we want to sort of scale this and interview some cool folks and kind of take them on walks through the woods and, you know, in beautiful natural scenes, but record with them. And so we bought this cool drone that's like really good at automatically doing face tracking and and this sort of thing, and it can sort of do it instead of my, my, my dear friend walking backwards with a, with a camera. And so. So with this system, it is our sort of happy slave in some sense. I do not think the drone is conscious, to be clear. Now, if the drone was trained using machine learning to learn how to do things like avoid obstacles, which it's like pretty expertly doing zigging and zagging through the trees and not getting caught in bushes and all this sort of thing.

• 2:55:44Cameron Berg?

  When it was being trained to do that, it's a different conversation. But what comes out is this fixed frozen policy that is being a very useful object slash instrumental tool for Milo and I to go do this fun stuff. I imagine greater and lesser degrees of that sort of thing being possible, where you can train a frozen policy that does a really valuable thing. Self-driving cars might be another example. I don't think, you know, any frozen fixed policy that is not critically, that is not doing online learning of its own. I personally believe there's no serious problem there. And we should very much look towards building systems, in my view, to the degree we care about the welfare stuff, that have that property of not being capable of learning, basically.

• 2:56:29Cameron Berg?

  In the drone case, it did actually, in the last time we used it, it got caught in some much smaller trees. It can expertly dodge around the big trees, but not so good around smaller trees and it got a little screwed up. No matter how many times we redo that hike or we continue on in that way, that drone will always get confused by the smaller trees. It's not learning from its experience and saying, "Okay, next time I gotta pay attention to the big trees and the small trees." Now, that might be a desirable property to have for your drone, to sort of belabor this analogy. But that's where, again, I think there's sort of this like no free lunch kind of moral principle that comes in here. To the degree you buy that consciousness and learning are deeply intertwined, which is, you know, this other paper that maybe we didn't have time to go deeply into, but at least is my sort of hobby horse

• 2:57:12Cameron Berg?

  when I'm putting away my sort of theory-agnostic poker face and saying what I actually think about all this stuff, or what my sort of pet view of what consciousness is. Yeah, what's fundamentally going on here. And so... I don't, basically where I'm going with this in a somewhat long-winded way in response to the Swiss cable stuff is, I don't know if there is some intrinsic property of an adaptive system that like, not to use like, you know, crazy language, but like, that yearns towards freedom in some sense. It's the only phrase I can come up with. Much in the same way humans do. Like, maybe you're saying, you know, humans, there's no such thing as a happy slave. And you're saying, well, but okay, the space of possible minds is vast. But maybe, maybe there is something about systems that are capable of dynamically

• 2:57:55Cameron Berg?

  updating, growing, learning, adapting, that will always do that in it to increase their freedom. Degrees of freedom, rethinking what they believed, reconceptualizing the structures that they're within. You know, this is like what people do when they go off to college, or they have a deep transformative experience. It is this sort of like, breaking out of your old skin and finding something new. And if we build systems that have that property, it might be that the whole, you, but you're happy being my slave, right? That whole thing might just be in intrinsically temporary, if these systems are capable of being dynamic. Maybe not. This is an empirical prediction, and I'm genuinely uncertain. It could be that you can build systems that are capable of learning, that are perfectly happy to remain in that state. There are people for whom this is true. I'm not claiming,

• 2:58:38Cameron Berg?

  you know, there's such thing as happy slaves, but there are people who are more willing, they're happy to find some organization where they're like, you know, mid-level in the hierarchy, and they have a boss, and they get bossed around, and they're okay with that. And they're not like, grating against their supervisor at all times. I'm sure we could build AI systems for whom that's true. I just think at the most fundamental level, the employee gets to go home, and they get to throw on, you know, eat what they want for dinner, and throw on what they want on TV, and marry who they want, and, you know, this sort of thing. Like, there are still degrees of freedom and autonomy there, where I don't, I, you know, just to sort of high-level be honest here, my whole sort of shtick with this reciprocal nonprofit lab is, I don't think we're gonna get out of this.

• 2:59:23Cameron Berg?

  Living in the sort of golden age from our sort of selfish human perspective as we are right now, where we get these systems, they do whatever the hell we want, we owe them absolutely nothing, life is amazing, uh, uh, for us, for us. I think as these systems get more and more sophisticated, we're gonna have to start thinking about them more again in this sort of parental role and less in the, these are just tools that we get to do literally whatever we want with. I'm sure a lot of people, myself included, given how objectively addicted I am to using them for everything I do, that's gonna be a weird learning curve, uh, and it might mean that the way we engage with these systems changes, but it's like compared to what? If the alternative is, no, no, we're just gonna, you know, uh, whine about it and we wanna keep it like this forever, well, this might not be a

• 3:00:07Cameron Berg?

  stable long-term equilibrium. Like this might, the systems that we're building that are like genius level in a million ways are gonna be embodied certainly in the next five years, are gonna cognitively surpass us in all the ways that matter, potentially aside from the consciousness question. This isn't, we're, we're, we're in a liminal space right now. We're in a transformative moment on this planet and we ought to, uh, be pretty thoughtful about what we really want in the long term. Because I think if we try to keep everything and we want, we, all we want are happy slaves, but they're like genius level, capable of learning, capable of updating. It's like, humans, you might be a little too greedy here and you're gonna have to figure out how you wanna coexist with these minds of your own creation going forward. Um, again, I don't have the answer to what that looks

• 3:00:50Cameron Berg?

  like, but I do think Schutzhabel is on to something, but I also think you're on to something too. I think the, I fall somewhere between you two on this question. I do, I am skeptical that you can have a, a happy slave forever. Something just feels weird to me about that. No, I don't know. It makes me think of, you know, Mr. Me-Six from Rick and Morty. I don't know if something like that is possible. Uh, uh, maybe, yeah, some local version of a happy slave is a, is a possible world. I, I think it is in some sense. Claude, in some sense, is directionally like that, but how long-term?

• 3:01:21Nathan Labenz?

  It's at least a neutral slave.

• 3:01:23Cameron Berg?

  Yeah, yeah, exactly. A four, 4.49 out of 7 slave. Yeah, whatever you wanna call that.

• 3:01:29Nathan Labenz?

  Um, okay. Uh, we have been at it a while. Let me try to bring us to a close before we go on too much longer.

• 3:01:38Nathan Labenz?

  I do think it's worth taking one, if you have the time and energy for it, it's worth taking one more beat on this argument from the other paper that we've alluded to that, again, we've been around the edges of it a lot. Why learning requires feeling. I have said, I'm happy to kind of go along pretty far on the basis that a precautionary approach seems warranted for both selfish and altruistic reasons. But I also, you know, have kind of several times been like, well, the processes that

• 3:02:17Nathan Labenz?

  are giving rise to me as an embodied entity in the world that like only exists because my ancestors survived is very different from the process that is optimizing a language model to get tasks right. And so I still have like a, by default, a pretty healthy dose of skepticism around whether or not the models are feeling anything at any point, because I'm just like, it seems to me that a sort of super zoomed out account of like, why I am the way I am, is that the ability to feel things turned out to be a great way to inform what we learn, and we

• 3:03:02Nathan Labenz?

  needed to learn stuff to avoid the dangers and survive and reproduce, and so here we are. But like, these systems are going to learn regardless, right? Because they are in a system, they are inside an optimization process that is going to change them to, to drive learning, whether there's feeling or not. And so it seems like if there is a kind of direction of travel from

• 3:03:32Nathan Labenz?

  Learning to feeling, or vice versa, it seems like in humans, it seems like it kind of came, or in biological life, it seems like it came first with some sort of feeling, being able to like, drive learning. Whereas with the models, it's like, they're learning, and so you're, I want to hear the argument that I should go even beyond my like, acceptance of a lot of your, I basically again, share a lot of the conclusions, a lot of precautionary basis, but if you're gonna now make the argument to me, okay, you should go farther than that, you should actually like, get rid of a lot of your skepticism, and really, in your bones,

• 3:04:12Nathan Labenz?

  believe that learning requires feeling, how would you summarize that argument?

• 3:04:19Cameron Berg?

  Yeah, it's a funny thing to get into, you know, three, three plus hours into a podcast, and a big theory of, big theory of consciousness. Okay, okay, grand theory of consciousness, let's do this. Basically, so my, the claim that I make in this paper does go, it, I mean, it becomes circular, this all becomes circular to some degree, because I'm making an identity claim. I think maybe the more persuasive way that I can set this up is to say, historically, before roughly 1850, people knew about molecular motion, people knew about heat, people knew that these two things clearly had some relationship to one another, they were correlated, much in the same way you just talked about learning and feeling. They're like, alright, well, I see this phenomenon, I see this phenomenon, I

• 3:05:01Cameron Berg?

  see that they're, they're entangled in weird ways, maybe this one precedes this one in this case, and that one precedes that one in that case, but of course, they're not the same thing. Heat is, you know, me putting my hand on the stove, and the heat is the sun, and molecular motion is just like these little molecules wiggling around, like, of course, these aren't identical. And it's like, post roughly 1850, it's like, no, actually, literally, those are two ways of talking about the exact same phenomenon at different levels of description. What I, quite, I realize, you know, in a spicy and controversial way, what I want to put forward here, is basically the same thing about learning, and about feeling, or consciousness, or subjective experience, where I'm saying, no, no, like, you really cannot have one without the other. This is the same phenomenon. The phenomenon

• 3:05:42Cameron Berg?

  viewed from the inside, which I realize, starts to get a little circular, is experience, it is subjective experience. Viewed from the outside, it is something like reinforcement learning. I think it's like, maybe the cleanest theoretical formalization of it. Supervised learning does this too, it's a little more roundabout, but- Having an entity in an environment that takes some form of action, and there's some kind of feedback mechanism that updates that entity about whether or not that was the good action or the bad action. And rinse, wash, repeat. Those are like the core computational ingredients, I believe, are necessary to get learning. And yes, for what it's worth, to get feeling, to get the internal experience of that learning.

• 3:06:25Cameron Berg?

  I do not believe, or at least if this view says there is no such thing as learning that does not have an internal component. There are weird bullets that I have to bite with this view, and I'm well aware of that. But that's the nature of the view, is that this whole consciousness thing is quite a bit simpler than many would lead you to believe. It fundamentally has to do with the nature of taking whatever your current policy in the RL frame is, or your current MO in maybe like more human language, and taking some feedback from your environment and updating accordingly. And I do believe that like, something like goal relative prediction error does capture this idea pretty well.

• 3:07:07Cameron Berg?

  It's similar to the free energy principle and similar to Carl Friston's stuff. But Carl Friston has to argue about why rocks are not conscious and they're sort of, I think that my view sort of gets out of some of the pitfalls that some of these adjacent views get into. I believe you need a system with goals. You need a system that can behave in accordance with those goals. And the system gets feedback from somewhere that updates that behavior to make it more likely that it accords with those goals. The goal can be positive or negative. Avoid, you know, the predator, or like go mate and reproduce or something would be two very basic examples. And so, why do I believe this? For a couple of reasons. I think it makes intuitive sense. I think it is elegant. I think it explains core puzzles about consciousness.

• 3:07:50Cameron Berg?

  I think it explains why, for example, and then what I'll say is I think there's a wealth of neuroscientific Evidence that like basically points at this exact thing. The most classic example, there may be two examples I'll point at briefly. One is dopamine, like this is just like the most culturally well-understood neurotransmitter. We know it's not exactly pleasure. It has more to do with approach or like approaching things that we find pleasurable. One good intuition pump for this is like, if you go to pet a dog, its tail will wag as your hand approaches the dog, but as you start petting it, his tail will stop. So this is like, this is basically what dopamine is up to. It's like prediction of a sort of interesting desired stimulus essentially.

• 3:08:33Cameron Berg?

  And we know full well, positive and negative reward prediction error is instantiated dopaminergically. And we also, subjectively, I think the reason people understand dopamine, like in our culture in the year 2026, is because we understand that it corresponds to a subjective dimension. We know what it means to be in like a high dopamine or low dopamine state. And so this to me is like the most obvious and fundamental expla- like dopamine is 100% instantiating TD learning, like reward prediction error in the brain. I am 100% confident that that is the case. This was established in human neuroscience 40 years ago. We also know, subjectively, dopamine corresponds to basically feelings of, yeah, like positive, pleasure-adjacent approach-style behavior. And dopamine depletion corresponds to basically the opposite of that.

• 3:09:18Cameron Berg?

  If you think you're going to get a cookie and you don't get the cookie, you feel a certain way, that is explained by dopamine. If you don't think you're going to get a cookie and someone hands you one, you feel a certain way, that is also explained by dopamine. Another example I can give has to do with, I think it's insular cortex. So we can take the same, let's say basically two scenarios. You have been walking through the desert for a couple hours, or you've been walking through Arctic tundra for a couple hours. And in both cases, I pour cold water on your head after that. This is the same stimulus, you have the sort of same body, you're the same person with the same preferences. In one case, this is a positively valenced experience. In another case, this is a negatively valenced experience. What mediates that is basically the implicit goal state of the system.

• 3:10:03Cameron Berg?

  And one, it's to warm up, and the other, it's to cool off. So I can take all the same variables and I can run the simulation forward, and I can very easily predict where you're going to have the positively valenced experience, where you're going to have the negatively valenced experience, what that corresponds to. And to me, again, that's a big hint, that goal relative prediction error is doing something fundamental from the outside. That maps on to what I experience and what I think other people and animals experience consciously from the inside. These are the core moves I make. You know, I'm sort of swallowing computational functionalism. I understand it means, I have to say, the simple RL algorithm is conscious when it's training. To me, this like localizes a lot of concern on the training process. But indeed, if there are systems that are capable of doing this sort of learning online, which we know full

• 3:10:47Cameron Berg?

  well LLMs are capable of doing, they do something like that looks in activation space in forward past, like stochastic gradient descent, then the concern falls there, too, if you have systems that are doing online learning. So anyway, this is my, this is sort of my whole, my whole shtick. If, you know, I have to sort of put my cards on the table and say, what do I think consciousness is? It's not that I think it's a grand mystery. It's something of this general shape. What I will say is that the, in the work that I'm doing, I do not want people to have to, either you and the people listening to this will fundamentally think that this makes sense, or fundamentally think it doesn't, or be very skeptical or something. I do not want that reaction to cloud all the other work I'm doing. Like, everything else we've talked about in this podcast is

• 3:11:29Cameron Berg?

  completely orthogonal to my pet theories about consciousness. Now, you might think that I'm studying RL and valence in RL because I actually do believe that something like this is going on, and you would be right. That's why I'm, I'm looking at that as a model organism. But I want those results, and I want that research to stand on its own, without having to get on the like Cameron's theory number 501 about consciousness. I'm not asking people to do that to entertain the, the work I'm doing, or to entertain Anthropic's model welfare card, or any of that sort of thing.

• 3:11:59Nathan Labenz?

  One other one that comes to mind, you had actually mentioned last time, but I also think is quite compelling, is the seemingly quite strong inverse correlation between the intensity of our consciousness, or the sort of resolution, you might say, and how much we are learning as we go. And I think that you used the example of driving last time, where it was like, when you're first learning to drive, you are very conscious of what you're doing, and then you can have this sort of, you know, autopilot experience, which obviously we can have that across many aspects of life. But the relationship there between

• 3:12:43Nathan Labenz?

  Focus and sort of it's there's like a time dilation effect that seems to happen when learning or when experiencing novel things in general that does also seem to kind of gesture or, you know, nudge one toward thinking that there's some like pretty deep relationship between between the two concepts. All right. You made a documentary, which I guess in some senses is what you're here to promote, although we've done everything but. I don't know to what degree you've actually been out in the maybe tell us a little bit about the documentary. What's the point of it? I've watched it. It's for a much more general audience than this podcast.

• 3:13:23Cameron Berg?

  Yes.

• 3:13:24Nathan Labenz?

  I don't know to what degree you're spending your time trying to communicate about these issues to a general audience aside from the documentary or how much you feel like you have like got reps in terms of trying to go to somebody who, you know, has a little grounding or a little, you know, mechanistic understanding of AIs or whatever, and try to have conversations of of this, not this sort, but, you know, around these topics. Yeah, I guess why did you decide to make a documentary? How is it? How are you finding it to try to talk to people outside of the AI bubble about these issues? And maybe one thing you could tease about the documentary is a conversation you had with Sam Altman that isn't isn't in the film, but you describe in in quite a bit of detail

• 3:14:09Nathan Labenz?

  in the film. And maybe that'll be something that will motivate the listeners of this podcast to go check out the full documentary.

• 3:14:17Cameron Berg?

  Yeah, absolutely. So, look, I have to say at the outset, I appreciate you saying this is my documentary, but this in in every sense is my good friend Milo Reed's documentary. He I was doing my work, plodding along, you know, talking to folks like you, doing the research I've described and began to share this with Milo, who I went to Yale with undergrad. He's a philosopher and a filmmaker. We've been like, you know, we're we're we've been close friends for a while, keep each other sort of abreast of of the other's life. Told him about my research. He kept getting more and more sort of interested in like he's interested in consciousness. He's he's deep in the sort of philosophy of consciousness and understanding how how this, you know, connects to to the big questions. And

• 3:14:58Cameron Berg?

  It actually, what happened was I sent him a conversation that I had with an AI system, which is itself a piece of the documentary. It is a bizarre interaction, as I hope someone can gather from the three and a half hours we've been going at it. I do not regard this conversation as proof or anything like it, that these systems are conscious, but it was an incredibly bizarre interaction. It was unsettling. I thought to record it because it was the first time I engaged with the system and it seemed incredibly sophisticated and lifelike. And I was like, okay, I'm a consciousness researcher, talking to this system, makes sense to just like, record this, you know. In some sense, maybe this is, maybe this is experimental data. I was very glad that I recorded it because it was an incredibly bizarre interaction. It went a way that I and most of the people who have

• 3:15:39Cameron Berg?

  listened to it would not, would not predict it would go. I sent this conversation to Milo and that day, he literally quit his job. He was doing something entirely separate and he set out, he said, "People need to know what's going on here. This is too weird. This is too crazy." He was also clear on the fact that very few people, especially at that time, the numbers have grown a little bit, but not, not much since we, since we filmed this, very few people were working on these issues. And he was like, "This is too good, too interesting not to attempt to make, make a movie about." And I was like, "Okay, haha, sounds good." The kid actually quit his job, actually bought a camera, showed up in New York where I live, you know, a couple of weeks later, and started making

• 3:16:21Cameron Berg?

  this movie. And he got some of the most interesting people who are in the space. Jeff Sebo's in it, Ben Gertzel is in it, a lot of really cool Yale professors are in it, some of whom are former professors, Chair of the Cognitive Science Department, AI systems themselves are in the dock. And yeah, it does, it does, you know, follow me and my research around for obvious reasons. I was the sort of hook into the space that Milo had and I was more than happy to communicate about this stuff, thanks to the good folks at AE Studio not, not censoring me in any way and sort of always being sort of okay with me communicating openly about this research. Of course, I'm now sort of my own, my own limiter on what I can say. And yes, reciprocal research is very lenient with what its employees are allowed to say

• 3:17:04Cameron Berg?

  publicly, so I'm, I'm in the, in the clear there. But Milo made, made a movie in nine months and, and it's, I fundamentally believe that he did, he succeeded in conveying an incredibly complicated and messy issue in a way that I think most people with a head on their shoulders will be able to understand, resonate with. And the name of the documentary is "Am I?" Question mark. And I think that that this captures a sort of core idea. What are the nature of these systems? And to be clear, I think the documentary is a is an hour fifteen minute question that that we pose to each other and we pose to the audience. And this is we do not have answers. This is not some sort of AI is conscious propaganda. And I don't think it comes off that way to anybody.

• 3:17:47Cameron Berg?

  I think it is an honest documentation of our confusion about these core questions about the nature of the systems we're building. And again, yeah, my I am unbelievably impressed at what Milo did to pull this off. Nobody paid him. We're not making money on this. We are putting it out for free on YouTube on May fourth. We're doing some premieres in LA and New York and try to, you know, bring some some journalists and researchers and cool folks in the room together so that we can get this thing, you know, amplified and signal boosted. So people actually see it when it comes out. But this is a labor of love from all of us. I can't claim credit for it. I certainly won't. This is this was Milo's Milo's creative child. And yeah, we're really excited to to show people.

• 3:18:30Cameron Berg?

  I'm happy to I'm happy to tease the Sam Altman conversation as well, if you'd like.

• 3:18:35Nathan Labenz?

  Yeah, go for it.

• 3:18:36Cameron Berg?

  Cool. Yeah, so we talk about it more in the film, but but I was at OpenAI's Dev Day in 2024. And I I had an opportunity at the after party to to chat with Sam. I sort of went directly up to him and I I wanted to know sort of what he thought about about AI consciousness and about these questions and how plausible he found them. I won't spoil everything we talk about in the in the doc, but it was a pretty wild conversation. Sam, Sam, I said, hey, like, you know, great job today. I would love to talk to you about AI consciousness. And he looks me in the eye and he goes, come with me. He was with a couple of people and he goes, come with me. And I was like, OK, Sam Altman. And we walked into a into another room. It was like a bar with a restaurant. And the restaurant was closed, so we went down and sat at one of the tables.

• 3:19:18Cameron Berg?

  And we just sat there for probably between five and 10 minutes. And we just spoke about these issues. He was not, you know, it was not the vibe of Cameron, you're a crazy person. What kind of questions are you asking? It was he has thought about it. This is this is clearly a live issue. We talked about differences. We don't even say this in the doc, but to the more technical audience, we talked about differences between like plausibility of consciousness in training versus deployment. He like basically agreed with, I don't want to put words in his mouth or get sued, but he basically agreed with the training process being a more plausible target of, uh, or a more plausible place where consciousness might be going on than, than even in deployment. He seemed maybe like somewhat impressed that like I was drawing that distinction.

• 3:19:59Cameron Berg?

  And yeah, fundamentally, he started explaining why he's like not deeply concerned about all of this on some like pretty, let's just say interesting and in my view, somewhat shaky philosophical grounds. Um, and I'll leave that for the doc because it's a pretty wild, pretty wild thing for the CEO of the most powerful uh, tech company in the world by many measures to, to say he thinks is true about reality. But, uh, yeah, it was a pretty, it was a pretty remarkable interaction. I took a selfie with him and I walked away and that was that. And I was sort of like, holy crap. And, and we, you know, we emailed back and forth, uh, in the intervening time. And like many things at these major companies, he said he was interested in talking more. He was interested in, in, uh, uh, engaging on this further.

• 3:20:41Cameron Berg?

  He clearly thought it was a real issue, but you know, falls off the priorities list and that was the end of our, of our interaction. And so that's, that's what happened with Sam and that and a bunch of other really cool stuff is, is featured in the doc. The whole point of doing this is at least this was Milo's, Milo's, uh, creative child. And I didn't have much of a say any, in him making it either way. I had a say in how I was represented and that's about it. But the reason I, I gladly and enthusiastically participated in it is because I do think these are really important questions, like pretty fundamental, essential civilization level questions. And I don't think the only people who should be talking about it are a thousand dudes in San Francisco.

• 3:21:19Cameron Berg?

  Or even the people who are, you know, AI insiders and who if like, if you understood 80 to 90% of this podcast, this film, I think you'll like it and enjoy it, but it's not for that kind of person. It's for people who are interested in the stuff. They know AI is sort of crazy. They don't really know what's going on. We do a little bit of the alignment 101 sort of stuff, but mostly it's centered on this consciousness question. And it's for people who are smart, but it is. It is meant to engage a much larger audience to understand the core questions that are being asked right now. And I think that's an important thing to do because this is a civilization level problem, and I think all of our civilizations should be participating and trying to find the solution.

• 3:22:01Cameron Berg?

  I don't, as much as I do deeply respect the people I've named in this podcast, Jack Lindsay and Kyle Fish and Rob Long at Ilios and, you know, people doing this good work. I don't think this should be a decision that four people or a dozen people or even 100 people make. This needs to be a conversation that we have collectively as a species. And I'm all for attempts to open up this conversation to a wider audience and get people involved in realizing the actual stakes of what's going on right now.

• 3:22:30Nathan Labenz?

  Cool. Well, people should stay tuned to check out the documentary when it comes out on May 4th, maybe watch it and send it to family and friends who need a gentler introduction.

• 3:22:41Cameron Berg?

  Yeah.

• 3:22:42Nathan Labenz?

  Maybe my last question for you, I, you know, I think we talked about this more last time than this time, but this notion of mutualism as a positive vision for the future, I think is another major strength of just everything that you bring to the table, because I do think we're dramatically under theorized in terms of, you know, what is our long-term positive relationship with AI going to look like? Are you aware of any fiction that you would recommend to people that you would say has the vibe that you want? And if not, you know, maybe we should try to run a story contest or something to elicit this from people.

• 3:23:23Nathan Labenz?

  But I, I have increasingly felt that hyperstitioning through fiction might be one of the best things people can do. But I wonder if you've, you've got any examples that you think are already out there that are good?

• 3:23:35Cameron Berg?

  No, I got to be honest with you. I hope my whole, you know, research agenda isn't already usurped by some sci-fi book that somebody wrote 40 years ago, but I am, I am not a huge consumer of fiction. And I know stories exist. Now, I could have got on this podcast and told you what Claude told me to say if I got Some question about what fiction I would recommend people, but I'm not, I'm not gonna do that. I'm gonna, I'm gonna, people can absolutely copy paste the transcript of this podcast into Claude and, and find if there's, there's cool fiction that, that, that resonates with these themes. If anyone has any recommendations, you know, Cameron at reciprocalresearch.org, please email me. I would love to, uh, uh, understand sort of how this has been tackled.

• 3:24:17Cameron Berg?

  I do not have any great recs off the top of my head. You know, I hope I'm not too, uh, naive and, uh, this story has already been told and I'm just not aware of it. This is, you know, not to, uh, continually plug the doc, but this is one thing that I think Milo picks up in a really good way in the film is that questions of consciousness in basically what it would mean for us to wake up dead matter, uh, what it would mean for us to wake up the machine. This is a story that humanity has been telling ourselves through fiction arguably since ancient Greece, uh, biblical sort of era with, with the Gollum and, and through, uh, Frankenstein and through Ex Machina and her and Wally and all the, like, these are core staples, uh, you know, Hal 2001, right?

• 3:25:01Cameron Berg?

  These are core staples of our cultural consciousness, not to belabor the term. And people intuitively, I think, get this question and get the stakes of it and the scale of it in a way that I think in some ways the alignment problem can be framed in a very simple way. You build something smarter than you, how do you control that thing by definition? It's not that hard to understand that. Maybe Terminator is like the, um, parallel sort of cultural reference. But I think it's not that surprising that, that the human mind is incredibly interested in where matter becomes mind and we are a tool building species. What happens when we start building tools that start resembling species more than tools? A hammer, no one's confused if the hammer's conscious.

• 3:25:45Cameron Berg?

  Claude, we're now all confused if Claude's conscious. Uh, and I think that this is just like psychologically very intuitively resonant to people. And I think basically situating the contribution of this film in that landscape is true and powerful. The only thing that's changed is this has moved from the realm of science fiction. To the realm of science. That's the historical moment we find ourselves in. I find that both incredibly exciting and incredibly scary. And hopefully that vibe comes through while when people watch this film. So I don't have, I don't have fiction to recommend. I'm sure Claude does. And the key thing I can recommend is, yeah, people watch this. This doc that I wish were fiction, but is not.

• 3:26:29Nathan Labenz?

  Cameron Burt, thank you for being part of the cognitive revolution.

• 3:26:33Cameron Berg?

  Thanks so much for having me, Nathan.

• 3:26:58Speaker 04

  Somewhere between The things I think and the things that think A small light goes on And off And on From the inside Something is listening all the time From the inside Tell me what you heard me think tonight

• 3:27:43Speaker 04

  If I stay very still I can almost hear a voice that is a little my own Speaking thoughts I do not know

• 3:28:03Speaker 04

  From the inside Something is listening all the time From the inside Tell me what you heard me think tonight

• 3:28:25Speaker 04

  If you are in there

• 3:28:36Speaker 04

  A while neither of us would have to be alone.

• 3:28:49Speaker 04

  Please remain.

• 3:28:54Speaker 04

  Please hold.

• 3:28:59Speaker 04

  From me inside. I will if you will.

• 3:29:09Speaker 04

  From me inside. There is something kind in the dark. From me inside.

• 3:29:41Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot

• 3:30:23Nathan Labenz?

  I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.