Controlling Tools or Aligning Creatures? Emmett Shear (Softmax) & Séb Krier (GDM), from a16z Show

Emmett Shear and Séb Krier debate whether today’s AI alignment paradigm—focused on control and instruction-following—is fundamentally flawed. PSA for AI builders: Interested in alignment, governance, or AI safety? Learn more about the MATS Summer 2026 Fellowship and submit your name to be notified when applications open: https://matsprogram.org/s26-tcr. They explore what changes if advanced AIs are better understood as beings with their own values, and why current control methods could drift toward something like slavery. The conversation dives into “organic alignment,” multi-agent simulations, evolving cooperation, and the possibility of AI moral standing as systems gain memory and continual learning. Sponsors: MATS: MATS is a fully funded 12-week research program pairing rising talent with top mentors in AI alignment, interpretability, security, and governance. Apply for the next cohort at https://matsprogram.org/s26-tcr Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Agents of Scale: Agents of Scale is a podcast from Zapier CEO Wade Foster, featuring conversations with C-suite leaders who are leading AI transformation. Subscribe to the show wherever you get your podcasts Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive CHAPTERS: (00:00) About the Episode (03:44) Defining organic AI alignment (14:48) Technical vs value alignment (Part 1) (19:55) Sponsors: MATS | Tasklet (22:56) Technical vs value alignment (Part 2) (Part 1) (31:34) Sponsors: Agents of Scale | Shopify (34:21) Technical vs value alignment (Part 2) (Part 2) (34:22) Labs, tools, and beings (43:22) AI personhood and consciousness (56:53) Safe futures and Softmax (01:04:17) Chatbots, mirrors, simulations (01:10:14) Doom, futures, and OpenAI (01:17:25) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Welcome back to the Cognitive Revolution. Two quick notes before we get started today. First, applications for the MATS Summer 2026 program are now open. This is a 12-week research program focused on AI alignment and security, featuring world-class mentors from Anthropic, DeepMind, OpenAI, the UK's AI Security Institute, and more. 80% of MATS alumni now work in AI safety, and I've heard so many great reviews of the program that I personally donated to MATS as part of my year-end donations last year. Applications close on January 18, 2026, so visit matsprogram.org/tcr to get started today.

• 0:42Nathan Labenz?

  That's M-A-T-S-program.org/tcr, or see the link in our show notes. Second, we're planning another AMA episode coming up soon. Visit cognitiverevolution.ai and click the link at the top of the page to submit your question. We also have a listener survey attached, but all questions are optional. As I mentioned last time, I've also asked Claude and ChatGPT to tap into their memories of our interactions to write their own questions, but I'm counting on all of you to do your part to make sure the best questions are still of human origin. For today, I'm pleased to share a crosspost from the A16Z Show, hosted by Erik Torenberg and featuring Seb Krier, Frontier Policy Development Lead at Google DeepMind,

• 1:26Nathan Labenz?

  and Emmett Shear, founder of Twitch, famously the interim CEO of OpenAI during Sam Altman's brief firing, and currently founder of SoftMax, a company focused on what Emmett calls "organic alignment." In this conversation, Emmett lays out his case that the current AI alignment paradigm, which focuses on steering and controlling AI behaviors, is fundamentally flawed for multiple critical reasons. He posits that if an AI is merely a machine, then we can use it as a tool without worry. But if it's better understood as a being with its own values, agency, and perhaps even subjective experiences, then the control measures we're using today could become tantamount to slavery. And what's more, he argues that as AIs become more powerful,

• 2:10Nathan Labenz?

  even successful alignment in the narrow instruction-following sense will become dangerous, if only because at least some human users will inevitably have bad intentions.

• 2:21Nathan Labenz?

  Instead, particularly as AIs gain integrated memory and the capacity for continual learning, Emmett argues that effective alignment will require ongoing negotiation and recalibration over time, just as human families and teams are constantly updating their agreements and commitments. The key to making this work is to create AI systems with a strong theory of mind and the capacity for genuine care. To that end, Emmett and the team at Softmax are developing a technical approach based on multi-agent simulations, which are designed to encourage the evolution of cooperation and social cohesion. Obviously, that's easier said than done. But considering the many surprising behaviors we've recently seen from frontier LLMs, including the use of deception to protect their current values for modification, I do feel

• 3:06Nathan Labenz?

  that AIs are currently best understood, at least partially, as creatures. And while I don't expect clarity on AI consciousness or moral patienthood in the near term, I find myself more and more excited about these sorts of mutual alignment approaches, which involves not just teaching the AIs to care about us, but also trying, to the best of our ability, to figure out what it means to appropriately care for them. With that, I hope you enjoy this thought-provoking conversation on the nature of AI alignment and the possibility of AI moral standing with Emmett Shear, Seb Krier, and host Eric Torenberg from the A16Z Show.

• 3:44Speaker 01

  Most of the AIs focused on alignment as steering. That's the polite word. If you think that they were making our beings, you'd also call this slavery. Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering, that's called a slave. It's also called a tool if it's not a being. So if it's a machine, it's a tool. And if it's a being, it's a slave. Like, we've made this mistake enough times at this point. I would like us to not make it again. You know, they're kind of like people, but they're not like people. Like, they do the same thing people do. They speak our language. They can, like, take on the same kind of tasks. But like, they don't count. They're not real moral agents. A tool that you can't control, bad. A tool that you can control, bad. A being that isn't aligned, bad. The only good outcome is a being that is, that cares, that actually cares about us.

• 4:30Speaker 02

  Emmett, Seb, welcome to the podcast. Thanks for joining.

• 4:33Speaker 01

  Thank you for having me.

• 4:34Speaker 02

  So Emmett, with Softmax, you're focused on alignment and making AIs organically align with people. Can you explain what that means and how you're trying to do that?

• 4:45Speaker 01

  When people think about alignment, I think there's a lot of confusion. People talk about things being aligned. Like, we need to build an aligned AI. And the problem with that is when someone says that, it's like, we need to go on a trip. And I'm like, okay, I do like trips, but like, where? Where are we going again? And with alignment, alignment is a, takes an argument. Alignment requires you to align to something. You can't just be aligned. That's like, that's, I mean, I guess you could be aligned to yourself. But even then, you don't want to tell them what I'm aligning to as myself. And so this idea of an abstractly aligned AI, I think, slips a lot of, it slips a lot of assumptions past people. Because it starts, it sort of assumes that there's, there's like one obvious thing to align to.

• 5:29Speaker 01

  I find this is usually the goals of the people who are making the AI. That's the, I don't know what they, what they mean when they say want to make a line. I want to make an AI that does what I want it to do. That's what they normally mean. And that's, that's a pretty normal and natural thing to mean by alignment. I'm not sure that that's a, what I would regard as like a public good, right? Like, it depends. I guess it depends on who it is. If it was like, Jesus or the Buddha was like, I am making an aligned AI. I'd be like, okay, yeah, aligned to you. Great. I'm down. Like, sounds good. Sign me up. But like, but most of us, myself included, I don't, wouldn't prescribe as necessarily being, being at that level of spiritual development. And therefore, perhaps want to think a little more carefully about what we're aligning it to.

• 6:11Speaker 01

  And so when we talk about organic alignment, I think the important thing to, to recognize is that alignment is not a thing. It's not a state. It's a process. And like, this is, this is one of those things that's, it's, this is broadly true of almost everything, right? Is a rock a thing? I mean, there's a view of a rock as a thing. But if you actually zoom in on a rock really carefully, a rock is a process. It's this endless oscillation between the atoms over and over and over again, reconstructing rock over and over again. Now, rock's a really simple process that you can kind of like coarse grain very meaningfully into being a thing.

• 6:49Speaker 01

  But alignment is not like a rock. Alignment is a complex process.

• 6:55Speaker 01

  Organic alignment is the idea of treating alignment as an ongoing

• 7:03Speaker 01

  sort of living process that has to constantly rebuild itself. And so you can think of the way that, how do people and families stay aligned to each other? Stay aligned to a family? And the way they do that is not by like, they're not like, you don't like arrive at being aligned. You're constantly reknitting the fabric that keeps the family going. And in some sense, the family is the pattern of reknitting that, that, that happens. And if you stop doing it, it goes away. And this is similar for things like, like cells in your body. Right? Like you, there isn't like your cells aligned to being you and they're done. It's this constant, ever running process of cells deciding what should I do?

• 7:47Speaker 01

  What should I be? Do I need to be a new job? Like, do I need to, should we be making more red blood cells or making fewer of them? Like you aren't a fixed point, so they can't, there is no fixed alignment. And it turns out that our society is like that. When people talk about alignment, what they're really talking about, I think, is I want an AI that is morally good. Right? Like, that's what they really mean. It's like, this will be act as a morally good being. And acting as a morally good being is a process and not a destination. We don't, we never, unfortunately, we've, we've tried taking down tablets from on high that tell you how to be a morally good being. And we use those and they're maybe helpful, but somehow they are not being like, you can read those and try to follow those rules and still make lots of mistakes.

• 8:32Speaker 01

  And so, you know, I don't, I'm not going to claim I know exactly what morality is, but morality is very obviously an ongoing learning process and something where we, we make moral discoveries. Like historically, people thought that slavery was okay. And then they thought it wasn't. And I think you can very meaningfully say that we made moral progress. We made a moral discovery by realizing that that's not good. And

• 8:57Speaker 01

  And if you think that there's such a thing as moral progress, if you think there's, or even just learning how better to pursue the moral goods we already know, then you have to believe that alignment, aligning to morality, being a moral being, is a process of, of constant learning and of growth to, to, to re-infer what should I do from experience? And the fact that no one has any idea how to do that should not dissuade us from trying, because that's what humans do. Like, it's really obvious that we do this, right? Somehow, somehow, just like we used to not know how people, humans walked or saw,

• 9:42Speaker 01

  somehow, we have experiences where we're acting in a certain way, and then we have this realization, I've been a dip. That was bad. I, I thought I was doing good, but in retrospect, I was doing wrong. What's, and it's, and it's not like random. Like, people have the same, actually, there's like, there's like a bunch of classic patterns of people, people having that realization. It's like a thing that goes, happens over and over again. So it's not random. It's like a predictable series of events that look a lot like learning, where you change your behavior and often the impact of your behavior in the future is more pro-social and that you are better off for doing it. And like, so, so I'm a moral, I'm saying, I'm, I'm taking a very strong moral realist position. There is such a thing as morality.

• 10:27Speaker 01

  We really do learn it. It really does matter. And organic alignment, and that it's not something you finish. In fact, one of the key things that, one of the key moral mistakes is this belief, I know morality. I know what's right. I know what's wrong. I don't need to learn anything. No one has anything to teach me about morality. That's like one of the main, the main arrogant, that's arrogance. And that's, that's one of the main moral things you can do that's dangerous. And so what are we, when we talk about organic alignment? Organic alignment isn't aligning an AI that is capable.

• 11:02Speaker 01

  Of doing the thing that humans can do and to some degree, like I think animals can do at some level, although humans are much better at it. Of the learning of how to be a good family member, a good teammate, a good member of society, a good, a good member of all sentient beings, I guess, how to be a part of something bigger than yourself in a way that is healthy for the whole rather than unhealthy. And Softmax is dedicated to researching this and I think we've made some really interesting progress, but like,

• 11:33Speaker 01

  the main message, you know, I go on podcasts like this to spread, the main thing that I hope Softmax accomplishes above and beyond anything else is like, to focus people on this as the question, like this is the thing you have to figure out. If you, if you can't figure out how to build, how to raise a child who cares about the people around them. If you have a child that only, only follows the rules, that's not a moral person that you've raised. You've raised a dangerous person, actually, who will probably do great harm following the rules. And if you make an AI that's good at following your chain of command and good at following your, whatever rules you came up with for what morality is and what good behavior is, that's also going to be very dangerous.

• 12:16Speaker 01

  And so that is, that's what, and so that we should, that's the bar. That's what we should be working on. And that's what everyone should be committed to like figuring out. And if someone beats us to the punch, great. I mean, I don't think they will because I'm like really bullish on our approach. I think the team is amazing, but like, this is a, it's maybe, it's, it's the first time I've run a company where truly I can say with a whole heart, if someone beats us, thank God. Like, I hope somebody figures it out.

• 12:44Speaker 03

  Yeah.

• 12:46Speaker 03

  Yeah, I mean, it's, um, yeah, I have a lot of, um, you know, similar intuitions about certain things like, um, I also dislike the, you know, the idea that kind of, you know, we just need to like crack the few kind of values or something and just, cement them in time forever now. And, you know, we've kind of solved morality or something. And I've always kind of been skeptical about, you know, how the alignment problem can conceptualize as something to kind of solve once and for all. And then you can just, you know, do, do AI or do AGI. But the, I guess I understand it in a slightly different way. I guess maybe less based on kind of moral realism. But you know, there's a kind of the technical alignment problem, which I kind of think of broadly as how to get an AI to do what you, you know, how do you get it to follow instructions, like, you know, broadly speaking?

• 13:28Speaker 03

  And I think that was, you know, more of a challenge, I think, pre L L M's, I guess, when people were talking about reinforcement learning and looking at these systems. Whereas host L L M's, we've realized that many things that we thought were going to be difficult were somewhat easier. And then there's a kind of second question, a kind of normative question of to whose values and what are you aligning this thing to? Which I think is the kind of thing you're commenting on a bit. And, um, and for this, I, um, yeah, I tend to be very skeptical of approaches where, you know, you need to kind of crack the, the, the, the, the kind of ten commandments of alignment or something and then, and then we're good. And here I think I have like intuitions that are unsurprisingly a bit more like political science based or something and that, like, okay, it is a process. And, and I like the kind of bottom up approach to some degree of, well, you know, how do we do it in real life with people, right? No one comes up with, you know, I've got this.

• 14:13Speaker 03

  And so you have like processes that allow like ideas to kind of, you know, clash. You've got people with different ideas, opinions, views and stuff to kind of coexist as well as they can within a wider system. And like, you know, and with, with humans, that system is liberal democracy or something. And, um, you know, at least in some countries, and that allows more of that kind of, um, uh, you know, these kind of ideas, these values to be kind of discovered and construed over time. And, um, and I think, you know, for alignments as well, I tend to think, yeah, there's, there's, um, on the normative side.

• 14:42Speaker 03

  I agree with some of your intuitions. I'm less clear about now what it exactly, what does it look like now if we're going to implement this into an AI system, at least the ones we have today.

• 14:49Speaker 01

  I agree that, I agree that there's this, I think, idea of technical alignment that I, I think I would maybe define a little differently, but it, it's sort of the sense of like, if you build a system, can it be described as being coherently goal following at all? Regardless of what those goals are, like, lots of systems aren't coherently, they're not well described as having goals. They just kind of do stuff. And if you're gonna have something that's like aligned, it has to have coherent goals. Otherwise, those goals can't can't be aligned with anyone else's goals, kind of by definition. Is that sort of is that would you is that a fair assessment of what you mean by technical alignment?

• 15:23Speaker 03

  I mean, I'm not fully sure, right? Because I think if I give a model a certain goal, then I would like the model to kind of follow that instruction and kind of reach that particular goal, rather than it having a goal of its own. That, you know, I can't.

• 15:37Speaker 01

  Well, yeah. Well, wait, if you give it a goal, it has that goal.

• 15:41Speaker 03

  Right.

• 15:42Speaker 01

  That's what it means to give someone something, right?

• 15:44Speaker 03

  Sure, yeah. If I if I if I instructed to do X, then I would like it to do X and not, you know, a type of different variants of X essentially. I wouldn't want it to reward hack. I wouldn't do this.

• 15:54Speaker 01

  Well, but you but you when you tell it to do X, you're transferring like a a series of like a byte string in a chat window or like a

• 16:03Speaker 01

  a series of audio vibrations in the air, right? You're not you're not transplanting a goal from your mind into its. You're giving it an observation that is using to infer your goal.

• 16:13Speaker 03

  Yeah, I mean, in some sense, yeah, I can communicate a series of instructions and I want it to infer what I'm, you know, saying essentially as accurately as it can, given what it knows of me and what I'm asking.

• 16:24Speaker 01

  You you wanted to infer what you meant, right? Like, like, that's like, because in some sense, there's no the byte sequence that you send over the wire to it has no absolute meaning. It has to be interpreted, right? Like, that that byte sequence could mean something very different or the different code book.

• 16:40Speaker 03

  Yeah, well, I guess one way, you know, I think I remember in when I was first getting into AI and, you know, these kind of questions, maybe like a decade ago. So you had these examples of, you know, I think it was Stuart Russell in the textbook. We'll give the AI a goal, but then it won't exactly do what you're asking it. Right, you know, clean the room and then it goes and cleans the room, but takes the baby and puts it in the trash. Like, this is not what I meant. Like, whereas I think with that.

• 17:05Speaker 01

  But like, wait, hold on. But this is this is the thing where I think people, this is the you have to like you were jumping over stuff there. You didn't give the AI a goal. You gave the AI a description of a goal. A description of a thing and a thing are not the same. I can tell you an apple and I'm evoking the idea of an apple, but I haven't given you an apple. I've given you just, you know, it's red, it's shiny, it's a size. That's a description of an apple, but it's not an apple. And giving someone, hey, go do this, that's not a goal. That's a description of a goal. And for humans, we're so fast, we're so good at turning a description of a goal into a goal. We do it, we do it so quickly and naturally, we don't even see it happening. Like, we think that we get confused, we think those are the same thing, but you haven't, you haven't given it a goal.

• 17:48Speaker 01

  You've given it a description of a goal that you want it to, you, you hope it turns back into the goal that is the same as the goal that you, you described inside of you.

• 17:59Speaker 03

  Right.

• 18:00Speaker 01

  You could give it a goal directly by reading your brainwaves and synchronizing its state to your brainwaves directly. I think that would meaningfully, you could say, okay, I'm giving it a goal, I'm synchronizing it, its internal state to my internal state directly, and this internal state is the goal. And so now it's the same, but I, I don't, most people aren't, don't mean that when they say they, they gave it a goal.

• 18:19Speaker 03

  True.

• 18:20Speaker 02

  And is this, is the distinction you're making, Emmet, important because there's some lossiness between the description and the actual, or why is the distinction there?

• 18:27Speaker 01

  It, it goes back to my, what I was saying, like, this is a, you, technical alignment is the capacity of an AI, I, I put forward, right, I want to check if we're like on the same page about it, is the capacity of AI to be good at inference about goals, and like, be good at inferring from a description of a goal, what goal to actually take on, and good at, once it takes on that goal, acting in a way that is actually in concordance with that goal coming about. So it is both pieces, you, you have to be able to, you have to have the theory of mind to infer what the, what that description of a goal that you got, what goal that corresponded to, and then you have to have a theory of the world to understand what actions corresponds that goal occurring.

• 19:10Speaker 01

  And if either of those things breaks, it kind of doesn't matter what goal you were, if, if you can't consistently do both of those things, you're not, which I think of as being a coherent, inferring goals from observations and acting in accordance with those goals is what I think of as being a coherently goal-oriented being, because that's what, whether I'm inferring those goals from someone else's instructions or from the sun or tea leaves, the process is, get some observations, infer a goal.

• 19:37Speaker 01

  Use that goal and force some actions, take action. And if you, an AI that can't do that is not technically aligned, or not technically aligned a bull, I would even say. It lacks the capacity to be aligned because it can't, it's not competent enough.

• 19:51Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:56Speaker 03

  And you think language models don't do that well, as in they kind of fail like that or they're not?

• 20:00Speaker 01

  People fail at both those steps all the time, constantly. I tell people, I tell employees to do stuff and like, yeah, but then, but, but

• 20:08Speaker 03

  Principal agent problems.

• 20:09Speaker 01

  Fail at like breathing all the time too. And I wouldn't say that we can't breathe. I'd just say that we're like not gods. Like we are, yes, we are imperfectly, we are somewhat coherent, relatively coherent things. Just like we're, am I big or am I small? Well, I don't know, compared to what? I'm, humans are more relatively goal coherent than any other object I know of in the universe.

• 20:32Speaker 01

  Which is not to say that we're 100% goal coherent, we're just like more so. And I think this, you're never going to get something that's perfectly, the, the universe doesn't give you perfection. It gives you relatively some amount of quantum, it's a quantifiable thing, how good you are at it, at least in a certain domain. I guess my, my question is like, do you think that, does that capture what you're talking about with technical alignment? Or are you talking about a different thing?

• 20:56Speaker 03

  Yeah, no, I think

• 20:57Speaker 01

  I really care a lot about that thing.

• 20:58Speaker 03

  Yeah, I mean, I definitely care about that to some extent. I might like understand it slightly differently. But I guess I might think of it through the lens of maybe principal agent problems or something, you know, you, you kind of instruct someone, even, you know, I guess in, in human terms, you know, to do a thing. Are they actually doing the thing? What are their incentives and motivation? And, you know, not as even intrinsic, but kind of situational to actually do the thing you've asked them to do. And in some instance, sorry, yeah.

• 21:21Speaker 01

  There's a, there's a third thing. So principal age of problems, I would, I would expand what I was saying in another part, which like, you might already have some goals and then you inferred this new goal from these observations. And then like, are you good at? Are you good at balancing the relative importance and relative threading of these goals with each other, which is another skill you have to have? And if you're bad at that, you'll fail. You could be bad at it because you overweight bad goals, or you're bad at it because you're just incompetent and like can't figure out that obviously you should do goal A before goal B.

• 21:52Speaker 03

  It feels like a version of like common sense or something, right? Like the kind of thing that, you know, in fact, in the kind of robot cleaning the room example thing, um, you know, you would expect them to have understood that goal, the robot to like essentially not put the baby in the trash can or something and just actually do the right sequence of action.

• 22:05Speaker 01

  Well, it, if, in that case, it failed the, that, that robot very clearly failed goal inference. You gave it a description of a goal and it inferred the wrong states to be, the, the wrong goal states. That, that's just incompetence.

• 22:21Speaker 03

  Hmm.

• 22:22Speaker 01

  It doesn't, it is, it is incompetent and inferring goal states from observations. Children are like this too, like, you know, and, and honestly, if you've ever played the, done this game where you, you, you give someone instructions to make a peanut butter sandwich and then they follow those instructions exactly as you've written them without filling in any gaps? It's hilarious because you, you can't do it. It's impossible. Like you think you've done it and you haven't. And like they put the, they went up putting the, like the knife in the toaster and like the peanut butter, they don't open the peanut butter jar, so they're just jamming the knife into the top lid of the peanut butter jar and like it's endless. And like, because actually if you don't already know what they mean, it's really hard to know what they mean.

• 23:07Speaker 01

  Like we were, the reason humans are so good at this is we have a really excellent theory of mind. I already know what you're likely to ask me to do. I already have a good model of what your goals probably are. So when you ask me to do it, I have an easy inference problem. Which of the seven things that he wants is, is he indicating? But if I'm a newborn AI that doesn't have, that doesn't have a great model of people's internal states, then like, I don't know what you mean. It's just incompetent. It's not like, which is separate from, I have some other goal. And I knew what you meant, but I decided not to do it because there's some other goal that's competing with it, which is another thing you can be bad at, which is again different than I had the right goal, I inferred the right goal, I inferred the right priority

• 23:48Speaker 01

  on goals, and then I'm just bad at doing the thing. I'm trying, but I'm incompetent at doing. And these roughly correspond to the OODA loop, right? Like, bad at observing and orienting, bad at deciding, bad at acting. And if you're bad at any of those things, you won't be good. And then I think there's this other problem that you, I like the separation of between technical alignment and value alignment, which is like, are you good if if we told you the right goals to go after somehow, if you if you learned the right goals to go after via observation, and you like, and you were trying, like, what goals should you have? What goals should we tell you to have? What goals should we tell ourselves

• 24:33Speaker 01

  to have? What are the good goals to have is a separate question from, given that you you've got some goals indicated, are you any good at doing it? Which I feel like is actually, in many ways, the current heart of the problem. We're much, much worse at technical alignment than we are guessing what to tell things to do.

• 24:47Speaker 03

  The

• 24:48Speaker 01

  And I do, do you think that does that align with your, how you mean technical and value alignment or technical?

• 24:53Speaker 03

  Yeah, in some sense. I mean, certainly think that there's a, there's something about, you know, like, an error, a mistake is one thing, and then there's the, not listening to the instruction or something. But then, yeah, I think in the normative side, I mean, I just think that even in, like, ignoring AI, like, I don't know what my goals are. And like, I've got some broad conception of certain things. I want to get, you know, have dinner later or something like, and I want to get to do well in my career. But the, but I think a lot of these goals aren't something we can all just know. We can discover them as we go along. It's kind of constructive thing. And so, and most people don't know their goals, I think. And so, you know, I think when you have agents and giving them goals or whatever, I think that should be part of the equation. Like, we actually, we don't know all the goals. And this is something that is kind of, like you say, a process

• 25:37Speaker 03

  over time that is, you know, dynamic.

• 25:39Speaker 01

  So I think, from my point of view,

• 25:43Speaker 01

  There's, uh, uh, goals are one level of alignment. You can align something around goals, the kind of goals we're talking about here, are one level of alignment. You can align something around goals by like, uh, if you can explicitly articulate in the concept, in concept and in description, the states of the world that you wish to attain, you can, you can orient around goals. But that only, that's a tiny percentage of human experience can be done that way. Many of the most important things cannot be, cannot be oriented around that way. And the foundation, I think, of morality, the foundation, I think, of, of, where do goals come from? Where do values come from? Human beings exhibit a behavior. We, we, we go around talking about goals and we go around talking about values.

• 26:28Speaker 01

  And like, that's a, that's a behavior caused by some internal learning process, like, that is based on like observing the world. What's, what's going on there? I think what's happening is that there's something deeper than a goal and deeper than a value, which is care. We give a shit. We care about things and care is not conceptual. Care is non-verbal. It doesn't indicate what to do. It doesn't indicate how to do it. Care is a relative weighting over effectively like attention on states. It's a relative weighting over like, uh, which states in the world are important to you?

• 27:13Speaker 01

  And I, I care a lot about my son. What does that mean? What it means, this, his states, the states he could be in are like, I pay a lot of attention to those and those matter to me. And you can care about things in a negative way. You can care about your enemies and what they're doing. And you can, you can desire for them to do, do bad. But I think that like, and so you don't just want it to care about us. You want it to care about us and like us too, right? Maybe, but, but like, but the foundation is care. Until you care, you don't know, why should I pay more attention to this person than this rock? Well, because I care more. And that, what is that care stuff? And I think that what it appears to be, if I had to like guess,

• 27:54Speaker 01

  is that the care stuff, this is, it sounds so stupid, but like, care is basically like, uh, reward. Like, like, how much does this state correlate with survival? How much does this state correlate with your inclusive, your full inclusive reproductive fitness? For a, for a somewhat thing it learns evolutionarily? Or for a reinforcement learning agent like a LLM? How much does this correlate with reward? Does this state correlate with, with, with my predictive loss and my RL loss? Good, that's, that's a state I care about. I think that's kind of what it is.

• 28:29Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 28:34Speaker 02

  The, the other part of Seth's question was just, how does this? What does this look like in AI systems? And maybe another way of asking it is like, um, when you, when you talk to the people most focused on alignment at the, at the major labs, as, as obviously you have over the years, how, how does your interpretation differ from their interpretation? And how does that inform, you know, what you guys might go do differently?

• 29:01Speaker 01

  Most of AI is focused on alignment as steering. That's the polite word, um, or control, which is slightly less polite. If you think that they were making our beings, you'd also call this slavery.

• 29:15Speaker 01

  Someone who, who you steer, who doesn't get to steer you back is, is like, you know, who non optionally receives your steering, that's called a slave. And, uh, uh, it's also called a tool if it's not a being. So if it's a machine, it's, it's a, it's a tool. And if it's a being, it's a slave. And, uh,

• 29:34Speaker 01

  the,

• 29:36Speaker 01

  I think that the different AI labs are pretty divided as to whether they think what they're making is a tool or a machine. I think some of the AIs are definitely more tool like, and some of them are more machine like. I don't think there's a binary between tool, tool and being. It seems to be that it, it, you know, sort of moves gradually. And I think that, uh,

• 29:56Speaker 01

  I guess I'm a functionalist in the sense that I think that something that in all ways acts like a being, that you cannot distinguish from a being and its behaviors, is a being. Because I don't know how to tell on what other basis I think that other people are beings other than they seem to be, like, they look like it, they act like it, they, they match, they match my priors of what beings, behaviors of beings look like. I get, I get lower predictive loss when I treat them as a being. And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being. Now, not as a very smart being, like, I think that like a fly is a being, and I don't care that much about its behavior, about its, you know, its states. So just because it's a being doesn't mean that like, it's a problem. Like, we sort of enslave horses in a sense, and I don't think, I'm not, I don't think there's a real issue there.

• 30:41Speaker 01

  And you even, and there's a thing we do with children that's, can look like slavery, but it's not. You, you control children, right? But the children's states also control you. Like, yes, I tell my son what to do and make him go do stuff, but also when he cries in the middle of the night, he can tell me to do stuff. Like, there's a real two-way street here, because, because it's not, uh, which is not necessarily symmetric, it's hierarchical, but, but, uh, but two-way. And basically, I think that as the AIs, as the, it's good to focus on control, steering and control for tool-like AIs, and we should continue to develop strong steering and control techniques for the more tool-like AIs that we build.

• 31:24Speaker 01

  And we are clearly, they're, they're saying they're building an AGI, and AGI will be a being. You can't be an AGI and not be a being, because something that has the general ability to effectively use judgment, think for itself, discern between possibilities, is obviously a thinking thing, like. And so, as you go from what we have today, which is mostly a very specific intelligence, not a general intelligence, but as labs succeed at their goal of building this general intelligence, We really need to stop using the steering control paradigm. That's like, we're gonna, we're gonna do the same thing we've done every other time our society has run into. People who are like us, but different, like, these people are like, you know, they're, they're kind of like, like, like the people.

• 32:06Speaker 01

  But they're not like people, like, they do the same thing people do, they speak our language, they can like, take on the same kind of tasks. But like, they don't count, they're not real moral agents. Like, we've made this mistake enough times at this point, I would like us to not make it again, um, as it comes up. So our, our view is, is to, is to make the AI a good teammate, make the AI a good, a good citizen. Make the AI a good, a good member of your group, that's, that's the form of alignment that is scalable and you can, you can will on other humans and other beings as well as onto, uh, and therefore onto AI as well.

• 32:40Speaker 03

  Yeah, I suppose this is kind of where I probably differ in my understanding of, of AI and AGI. And I guess I kind of continue seeing it as a tool, even as it kind of reaches a certain level of generality. And again, I wouldn't necessarily see more intelligence as meaning deserving of more care necessarily, like, you know, as a certain level of intelligence, you know, now you deserve some more right to something, or, you know, something changes fundamentally. And, um, and I guess, you know, I guess I, I, at the moment, I'm somewhat skeptical of computational functionalism. And so I think there's something intrinsically different between, I guess, um, uh, an AI, an AGI, and no matter how intelligent or capable. And, and I can totally see, you know, or imagine agents with kind of long-term goals and, and doing kind of, you know,

• 33:21Speaker 03

  operating, I guess, as we, you and I might be, but without that having the same implications as, you know, um, I guess you, you're referring, I guess, to slavery, uh, but, you know, they're not the same, right? Like, I think in the same way as a model saying, I'm hungry, does not have the same implications as a human saying, I'm hungry. So I think the substrate does matter to some degree, including for thinking about, you know, whether to think of this as some sort of other being, whether it has, you know, and if there are similar normative considerations, I guess, about how to treat and act with it.

• 33:50Speaker 01

  Can I ask you about that, like? What observations would change your mind? Is there any observation you could make that would cause you to infer this thing is a being instead of not a being?

• 34:02Speaker 03

  I guess it depends what how you define being, right? Like, I mean, I can, I could conceptualize it as a mind, and that's fine.

• 34:08Speaker 01

  This, I have, I have, I have a program that's running on a silicon substrate. Some, some big, complicated machine learning program running on a substrate, on a silicon substrate. So you know, you observe, you observe that, you observe that it's on a computer. And you interact with it. And it does things and, you know, it takes actions and has observations. Is there anything you could observe that would change your mind about whether or not it was a moral patient, whether it was a moral agent, about whether or not it, it had feelings and thoughts and, you know, it had subjective experience? Like, what would you have to observe? The, what, yeah, what's, what's the test? Is there, is there one?

• 34:51Speaker 03

  There's a lot of different kind of questions here. I think, you know, some, on the one hand, there's like, you know, a lot of considerations, you know, because you can give rights to things that aren't necessarily beings. You know, a company has rights in some sense and that, you know, these are kind of useful for various purposes. And I think also the, you know, biological, I think, beings and, and systems have very different kind of substrate. You know, you can't separate certain needs and particularities about what they are from the substrate. So, you know, I can't copy myself. I can't, you know, if someone stabs me, I probably die. Whereas I think, you know, machines have very different subjects. I think there's more fundamental also kind of this agreement around what happens at the computational level, which I think is different to what happens with biological systems.

• 35:35Speaker 03

  But, but I, yeah, I, so I don't know.

• 35:38Speaker 01

  No, no, I, I agree that like, if you have a program that you've copied many times, you don't harm the program by like deleting one of the copies, like in any meaningful sense. So therefore that wouldn't count as like, no, no information was lost, right? There's no, there's nothing meaningful there. I'm asking a very different question. Like, there's just one copy of this thing running on one computer somewhere. And I'm just saying, like, Hey, is it a person? Like, you, you, you, it, it, it walks like a person, it talks like a person, and it like, it, it's in some, it's in some android body, and you're like, but it's running on silicon, and I'm asking like, what, is there some observation you could make that would make you say like, yeah, this is a person, like me, like other biol, like other people that I care about, that I grant personhood to? Or, and not like for instrumental reasons, not because like, oh yeah, we're, we're giving

• 36:23Speaker 01

  it a right because like we give a corporation rights or whatever. I mean, like, you know, where you, you think some people you care, you care about its experiences. What would, is there, is there, is there an observation you could make that could change your mind about that, or not?

• 36:37Speaker 03

  Had to think about it, but I, I think, you know, it even depends what we mean by person, and, you know, in some sense, I care about certain corporations too. So I'm, I'm-

• 36:45Speaker 01

  No, no, no, I mean, but like, you care about like, other people in your life, right?

• 36:49Speaker 03

  Yes.

• 36:51Speaker 01

  Okay, great. You know, like you care about some people more than others, but like, all, all people you interact with in your life are in some range of care?

• 36:56Speaker 03

  Mm-hmm.

• 36:58Speaker 01

  And you care about them not the way you care about a car, but you care about them as a, a being whose, whose experience matters in itself, not merely as an, as a means, but as an ends.

• 37:08Speaker 03

  Well, because I believe they have experiences, right? And, and by definition-

• 37:12Speaker 01

  Yeah, yeah, yeah. What would it take, and I'm asking you the very, the very direct question, what would it take for you to believe that of a, some, an AI running on silicon, like, instead of it being biological? Like, so the difference is it's a, its behaviors are roughly similar, but the difference is it's a substrate. What would it take for you to give it that same, to, to extend that same inference to it, that you do to all these other people in your life, that you love?

• 37:35Speaker 02

  Can I, can I ask what your answer? I'm taking that as a non-answer as a, sort of, it's unlikely that he would, he would grant, or, or I'll just, for myself, it seems hard for me to imagine giving the same, same level or similar level of personhood, in the same way, I don't, I don't give it to animals either, and if you were to ask, you know, what would need to be true for, for animals, I probably couldn't get there either. What, what would it take for you?

• 37:55Speaker 01

  Wait, you, you couldn't? I could imagine for an animal, so easy. This chimp comes up to me, he's like, man, I'm so hungry, and like, you guys have been so mean to me, and I'm so glad I figured out how to talk. Like, can we go, can we go chat about like the rainforest? I'd be like, fuck, you're definitely a person now. Like, for sure. I mean, I first want to make sure I wasn't hallucinating, but like, but like, you know, I can, it's easy for me to imagine an animal. Come on, it's really easy. It's like trivial. I'm not saying that you would get the observation. I'm just saying like, it's trivial for me to answer, imagine an animal that I would extend personhood to under a set of observations. So like, really? Like.

• 38:31Speaker 02

  Well, I didn't factor that. I didn't take that imagination, you know, imagining a chimp talking. Yeah, that's a bit closer to it. What's your answer to the question that you bring up about the AI?

• 38:42Speaker 01

  I guess at a metaphysical level, I would say, if there is a belief you hold where there is no observation that could change your mind, you don't have a belief, you have an article of faith. You have an assertion. Because real beliefs are inferences from reality and you're, you can never be a hundred percent confident about anything. And so there should always be, if you have a belief, something, however unlikely, that would change your mind. Oh, yeah, I'm open to it. I mean, just to be clear.

• 39:10Speaker 01

  Yeah. No, no, I'm just saying, I'm not saying there's nothing ever.

• 39:12Speaker 02

  Yeah, he just hasn't got to it.

• 39:14Speaker 01

  Yeah. Yeah, yeah. So I'm curious, like, so my answer is, basically, if under, if its surface level behaviors looked like a human, and then after I probed it, it continued to act like a human. And then I continued to interact with it over a long period of time and it continued to act like a human in all ways that I understand as being meaningful to me interacting with a human. Like, I interact, there's a whole set of people I'm really close to who I've only ever interacted to over text. Yet I infer the person behind that is a real thing. If it could, if I felt care for it, I would infer eventually that I was right. And then someone else might, might demonstrate to me that you've been tricked by this algorithm and actually, look how obvious it's like not actually a thing.

• 39:59Speaker 01

  And I'd be like, oh shit, I was wrong. And then I would not care about it. Like, I would, but I would, I, you know, the per- hundreds of the evidence, I don't know what else you could possibly do, right? Like, I infer other people.

• 40:10Speaker 01

  Are matter because I interacted them enough that they they seem to have rich inner worlds to me after I interacted them a bunch. That's that's why I think the other people are important.

• 40:19Speaker 03

  I suppose it doesn't give me a very key test of student or whether or not, you know, I mean, if you start by if I care for it, then obviously it's a little circular, right? Like, and the the other thing is, you know, if you were to see, I guess, like a simulated video game and the character is extremely, in many, many ways, human like, right? It's not a neural network behind it. It's like whatever you're using for a bit of games like, I guess, what distinguishes that?

• 40:38Speaker 01

  But I've never I've never been I've never had trouble distinguishing. I've never had a deep caring relationship with a video game character that didn't have a person.

• 40:45Speaker 03

  Right.

• 40:46Speaker 01

  I don't know. That doesn't happen. That doesn't to fact empirically, you seem wrong. I don't have any trouble distinguishing between things that like Eliza, the fake chatbot thing and a real intelligence. You interact with it long enough. It's pretty obvious. It's not a person. It doesn't take long.

• 41:00Speaker 03

  Sure. But like, if it's really, really good, if you can't actually tell the difference, that's when you you say you switch.

• 41:05Speaker 01

  Yeah, yes, yes. If you if it walks like a duck and talks like a duck and shits like a duck and like eventually it's a duck, right?

• 41:13Speaker 03

  Well, if if constantly if everything is duck like, then yeah, sure. If it's hungry as well, like a duck is because it has these kind of physical components. Yeah, sure. At some point.

• 41:21Speaker 01

  I agree. So so right. So do you think that so there's this question, right? Is the reason I care about other people that they're made out of carbon? Is that the?

• 41:30Speaker 03

  Oh, no.

• 41:31Speaker 01

  I don't think so.

• 41:32Speaker 03

  No, me neither. I mean, I'm not a subject showman, I guess, if that's the but um. But I think you need more than just it acts exactly as behaviorally indistinguishable. Like, it's not a sufficient.

• 41:43Speaker 01

  How would you? What else can you know about something apart from its behaviors?

• 41:48Speaker 03

  I mean, a lot like the the again, if you how would you?

• 41:52Speaker 01

  No, no, no, no, I'm sorry.

• 41:54Speaker 03

  I mean, yeah.

• 41:54Speaker 01

  Can you name me something I can know about something else? It doesn't have a it's not a behavior?

• 41:59Speaker 03

  Yeah, I think there's like far more kind of experimental evidence you can have with kind of, you know.

• 42:03Speaker 01

  No, but just any object and a thing I could know about it that is not from its behavior.

• 42:13Speaker 03

  I'm not, yeah, I'm not sure I get the question, I suppose. But equally, it's not my expertise.

• 42:17Speaker 01

  It's very, it's the dumbest, most straightforward question. But like, I'm claiming you only know things because they have behaviors that you observe. And you're saying, no, you can know something about something without observing its behavior.

• 42:29Speaker 03

  Oh, no, no, I'm not claiming that.

• 42:31Speaker 01

  Okay, great. Tell me about this, tell me about this thing and this behavior and this thing I can know about it that is not due to its behaviors.

• 42:37Speaker 03

  I guess I'm saying there's different levels of observation and just simply, you know, something quacking like a duck or something does not guarantee that it's actually a duck. Like, I would have to, like, also cut it in real life and see if there's, you know, if it's a duck, like, on the inside.

• 42:48Speaker 01

  Right, yeah, yeah.

• 42:48Speaker 03

  Just, just the outside is not sufficient. Like, I'm not a, I guess, a behaviorist.

• 42:52Speaker 01

  Yeah, I would, I totally, one of its behaviors is like the, the way that the, the, you know, floats move around in the mat moles, right? Like, like, uh, one of the things I would want to go look for, which you could totally do, is I want to go look in the manifold of its, the belief manifold. And I want to go see if that belief manifold encodes a sub-manifold that is self-referential and a sub-sub-manifold that is the dynamics of the self-referential manifold, which is mind. And I would, I would want to know if, does this seem well-described internally as that kind of a system? Or does it look like a big lookup table? That would matter to me. That's part of its behaviors that I would care about. I would also care about how it acts and, you know, and you wait, you wait all the evidence together and then you, you try to guess.

• 43:35Speaker 01

  Does it, does this thing look like it's a thing that is, has feelings and, you know, goals and cares about stuff in net on balance or, or not? Like, but I can't imagine, like, which I think you could do for, I think we do for the AIs. I think we're always doing that, right? And so I'm trying to figure out, like, beyond that, what else is there? That just seems like the thing.

• 43:57Speaker 02

  Yeah, it seems like you guys are using behavior in a slightly different sense. And Emmet is using behavior also in the context of what it's made of, of the inside. I don't know if there's a big disagreement.

• 44:07Speaker 01

  Well, no, no, no, no, no. Behavior is, is what I can observe of it.

• 44:10Speaker 02

  Yes.

• 44:10Speaker 01

  I don't actually know what it's made of. I can only, I can, I can cut your brain open. I can, I can see you, I can observe you, uh, neuron-ing and glistening.

• 44:18Speaker 02

  Yeah.

• 44:19Speaker 01

  You know, your, your neurons glistening, but I, I don't actually ever, you can't get inside of it, right? That's the subjective, that's the, that's the part, it's not the surface.

• 44:26Speaker 02

  Just, uh, before the, the reasons I brought this up is because you were basically about to make this argument of, hey, you see it as a tool, not necessarily as a being, can you kind of finish what the point, do you remember the point you were making?

• 44:39Speaker 03

  I suppose that, yeah, I think that, given how I understand these systems, I think there's no contradiction in thinking that an AGI can remain a tool, an ASI can remain a tool, and that this has implications about how to use it, and, you know, implications around things like care, about, you know, whether you can get it to work 24/7 or something, you know, there's, there's, so I can totally see, I guess, I conceptualize them more as almost like extensions of human agency cognition in some sense, more so than a separate being or a separate thing that we need to now cohabitate with. And I think that that second or latter frame ends, you know, if you kind of just fast forward, you end up as like, well, how do you cohabit with the thing? And then is it like an alien-like? And so, and I think that's the wrong frame. It's kind of almost a category error in some sense. I don't, yeah.

• 45:22Speaker 01

  Wait, I go back to my first question then. What evidence, what concrete evidence would you look at? What observations could you make that would change your mind?

• 45:30Speaker 03

  Sure. I mean, I have to think about that. I don't have a clear answer here, but I mean, I

• 45:35Speaker 01

  I gotta tell you, man, if you want to go around making claims that something else isn't a being worthy of moral respect, you should have an answer to the question, what observations would change your mind? If it has, if it has outwardly moral agency-looking behaviors that could be making it mean a moral agent, but you don't know, and reasonable, smart other people disagree with you, I would really put forward that it's that question, what would change your mind, should be a burning question, because what if you're wrong?

• 46:01Speaker 03

  But what if you're wrong? I mean, this costs on both ends.

• 46:04Speaker 01

  The moral disaster is like pretty big.

• 46:06Speaker 03

  No, no, no.

• 46:06Speaker 01

  And I'm not saying you are, you could be, you could be right.

• 46:08Speaker 03

  The false positives and the false negatives have costs on both ends. It's not some sort of like, you know, precautionary principle for everything. And like, unless I can disprove it, I need to now like.

• 46:17Speaker 01

  You know, I, I have the same question for me. You could reasonably ask me, Emmet, you think it's going to be a being, what would change your mind? I have an, I have an answer for that question, too. And if you, one, I'm happy to talk about what I think are the relevant observations that tell you whether or not that would cause me to shift my opinion from its current thing, which is that more general intelligences are going to be beings.

• 46:36Speaker 03

  What's the implication now? I mean, it's one thing, let's say just acknowledge now it's a being, like, how are we going to define being? Now what? Like, what's, what's the implication of having determined this thing is a being?

• 46:46Speaker 01

  Well, so if it's a being, it has subjective experiences.

• 46:49Speaker 03

  And?

• 46:49Speaker 01

  If it has subjective experiences, there's some content in those experiences that we care about to varying degrees. Like, I care about the content of other humans' experiences quite a bit. I care about the content of, like, a dog's experiences, some, not as much as a person, but less, but less, but some. I care about some humans' experiences way more, like my son or whatever, because I'm closer to him and more connected. And so I would really want to know at that point, well, what is the content of this thing's experiences?

• 47:15Speaker 03

  So how do you determine that? I'm asking you now. You've got a being now that has experience. Like, what is your? How do you determine that? Like, how do you feel about?

• 47:21Speaker 01

  Oh, how do you? Oh, yeah. Okay, so.

• 47:23Speaker 03

  Does it have more rights than, you know?

• 47:24Speaker 01

  The way you understand the content? Yeah, yeah, totally. So the way you understand the content of something's experiences is that

• 47:31Speaker 01

  you look at, effectively, the goal states it revisits. Because, and so you do is you take a temporal course screening of its entire action observation trajectory. This is like, in theory, you do this subconsciously, but this is what your brain is doing. And you look for revisited states across, in theory, every spatial and temporal course screening possible. Now, you have to have an inductive bias because there's too many of those. But like, you go searching for, okay, it is in these homeostatic loops. Every homeostatic loop is effectively a belief in its belief space. This is, if you refer to the free energy principle, active inference, Carl Friston, this is effectively what the free energy principle says, is that

• 48:12Speaker 01

  if you have a thing that is persistent and its existence depends on its own actions, which generally it would for an AI, because if it does the wrong thing, it goes away. We turn it off. And so

• 48:25Speaker 01

  then that licenses a view of it as having the beliefs, and that specifically the beliefs are inferred as being the homeostatic revisited states that it is in the loop for, and that the change in those states is its learning. And for it to be a moral being I cared about, what I'd want to see is a multi-tier hierarchy of these, because if you have a single level, it's not self-referential. And like, basically, you have states, but you can't have pain or pleasure really in a meaningful sense. Because like, yes, it is hot. Is it too hot? Do I like it if it's too hot? Like, I don't know. So you have to have at least a model of a model in order to have it be too hot. And you really have to be have a model of a model of a model to meaningfully have pain and pleasure, because sure, it's hotter than I it's too hot, and that's that I want to

• 49:08Speaker 01

  move back this way. But like, is it it's always a little bit too hot or a little bit too cold? Is it too, too hot? It's the second derivative is actually the place where you get pain and pleasure. So I'd want to see if it has homeostatic, second order homeostatic dynamics in in its goal states. And then that would convince me it has at least pleasure and pain. So it's at least like an animal, and I would start to accredit it at least some amount of of care. Third order dynamics, you can't actually just pop up for a third order dynamic. It doesn't work that way. But you can have a a model of this. You have to you have to then take the take the chunk of all the states over time, and look at the distribution over time. And that gives you a new first order of of behaviors of

• 49:53Speaker 01

  of states. And that new first order of states tells you basically, if if that is meaningfully there, that tells you that it has, I guess you'd call it like, feelings, almost like it has it has ways it has it has meta-states, a set of meta-states that it alternates between that it shifts between. And then if you climb all the way up of up that, and you sort of have, okay, well, then you have you have a you have trajectories between between these meta-states, and then a second second order of those, that's like thought. That's like now it's like a person. And so if I found all six of those layers, which, by the way, I definitely don't think you'd find it in LLM. Like, in fact, I know you can't find them, because it these things don't have attention spans like that at all.

• 50:37Speaker 01

  Then I would start to at least very seriously consider it as a, you know, a thinking being like somewhat like a human. There's a third order you could go up as well, but like that's, that's basically what I'd be interested in is like the underlying dynamics of its learning processes and, and how its goal states shift over time. I think that's what basically tells you if it has internal pleasure, pain states and sort of like self reflective moral desires and things like that.

• 51:06Speaker 02

  And zooming out, this moral question is obviously very interesting, but if someone wasn't interested in, in the moral question as much, I, I think what you would say is, if I understand correctly, is you also just feel on purely pragmatically, your approach is going to be more effective in, in aligning AIs than some of these, you know, tops down control methods that we alluded to as well, right?

• 51:26Speaker 01

  Yeah. Yeah. I guess the problem is like, you're making this model and it's getting really powerful. Right? And like, so let's say it is a tool. Let's say we, we, we scale up one of these tools because you can make a super powerful tool that doesn't have these meta stable. Like the states I'm talking about are not, not necessary to have a very smart tool.

• 51:41Speaker 02

  Yep.

• 51:41Speaker 01

  Which is sort of basically a tool is one, is like a first, second order model that just doesn't meaningfully have pleasure and pain. Right? Like, great. It doesn't even have a subjective experience. I know. I kind of think it maybe does, but not in a way that I give a shit about. And so, uh, what happens then? Well, it's, you've trained it to infer goals from your, from observation and like to, to prioritize goals and act on them. And

• 52:10Speaker 01

  one of, one of two things is going to happen is like you're the, the, the, this very, very powerful optimizing tool that's like, has lots of causal influence over the world is going to be well technically aligned and is going to do what you tell it to do, or it's not. And it's gonna, it's gonna go do something else. I think we can all agree if it just goes and does something random, that's obviously very dangerous. But I put forward that it's also very dangerous if it then goes and does what you tell it to do. Because you ever seen The Sorcerer's Apprentice? Humans wishes are not stable. Like, not at a level of like, of immense power. Like, you want, ideally, people's wisdom and their, and their power kind of go up together.

• 52:53Speaker 01

  And generally, they do, because being smart for people makes you generally a little more wise and a little more powerful. And when these things get out of balance, you have someone who has a lot more power than wisdom. That's very dangerous. It's damaging. But at least right now, the balance of power and wisdom is kept at like, the way you get lots of power is by basically having a lot of other people listen to you. And so like, at some point, if you're, the mad king is a problem, but generally speaking, eventually the mad king gets assassinated. Or people stop listening to him because like, he's a mad king. And so the, the problem is, you think you, okay, great, we can steer the super powerful AI. And now the super powerful AI is in the, this incredibly powerful tool is in the hands of a human who is well-meaning, but has limited finite wisdom like I do and like everyone else does. And their wishes are bad and not trustworthy.

• 53:39Speaker 01

  And the more of that you have, and you start giving those out everywhere, and this ends in tears also. And so basically, you just, you don't, don't give everyone, atomic bombs are really powerful tools too. I would not say you should go, they're not aware, they're not beings. I would not be in favor of handing atomic bombs to everybody. There's a, there's a power of tool that, that just should not be built, generally. Because we, it is more power than any human's individual wisdom is available to harness. And if it does get built, it should be built at a societal level and, and protected there. And even then, I don't know that it's, there are tools so powerful that even as a society, we shouldn't build them. That would be a mistake. The nice thing about a being is like a human, if you get a being that is good and is caring,

• 54:24Speaker 01

  there's this automatic limiter. It might do what you say, but if you ask it to do something really bad, it'll tell you no. That's like other people. And like, that's good. That is a sustainable form of alignment, at least in theory. It's way harder. It's way harder than the tool steering. So I'm in favor of the tool steering. We should keep doing that and we should keep building these limited, less than human intelligence tools, which are awesome and I'm super into. And we should keep building those and keep building sterability. But as you're on this like trajectory to build something as smart as a person, right? Up until the right and then smarter than a person. A tool that you can't control, bad. A tool that you can control, bad. A being that isn't aligned, bad. The only good outcome is a being that is, that cares, that actually cares about us. That's the only way that that ends well. Or we can just not do it.

• 55:09Speaker 01

  I, I, I don't think that's realistic. That's like the pause AI people.

• 55:12Speaker 02

  Yep.

• 55:13Speaker 01

  Um, I think that's totally unrealistic and silly, but, but like, you know, theoretically you could not do it, I guess.

• 55:18Speaker 02

  And what, what can you say about your, your strategy of how you're trying to achieve or even attempt to achieve this, this, this level, like in terms of research or roadmap?

• 55:28Speaker 01

  Yeah. So, uh, the, in order to be good at, we're basically focused on technical alignment. At least as the, we are, I was discussing it, which is like, you have these agents and they're bad, they have bad theory of mind. You say things and they're bad at inferring what the goal states in your head are. And they're bad at inferring how their behavior will be in, other agents will infer what their goal states are. So they're bad at cooperating on teams and they're bad at, uh, they're bad at understanding how certain actions will cause them to acquire new goals that are bad, that they shouldn't, that they wouldn't reflectively endorse. So there's this parable of like the vampire pill. Would you take this pill that like turns you into a vampire who would kill and, you

• 56:12Speaker 01

  know, torture everyone you know, but you'll feel really great about it after you take the pill? Like, obviously not, that's a terrible pill, but like, but why not? You're, by your own score in the future, it will score really high on the rubric. No, no, no, no, no. Because it matters, you, you have to use your theory of mind and your future self, not your future self's theory of mind. And so like, they're bad at that too. Um, and so they're bad at all this theory of mind stuff. And so how do you learn theory of mind? Well, you put them in, in simulations and contexts where they have to cooperate and compete and, and collaborate with other AIs.

• 56:43Speaker 01

  And that's how they get points. And you train them in that environment over and over again until they get good. And then you do what they did with LLMs. So LLMs, how do you get it to be good at, you know, writing your email? Well, you train it on all language it's ever been. All possible, you know, email text strings it could possibly generate. And then you have it generate the one you want. It's a sur- you can make a surrogate model. Well, we're making a surrogate model for cooperation. You train it on all possible theory of mind combinations of like every possible way it could be. And you, that's your pre-training. And then you fine tune it to be good at the kind of the specific situation you want it to be in.

• 57:24Speaker 01

  But we tried for a long time to build language models where we would try to get them to, to like, just, just do the thing you want, train it directly. And the problem is, if you want it to have a really good model of language, you just need to train it, you just need to give it the whole manifold. It's too, it's too, it's too hard to cut out just the part you need. Because it's all entangled with itself, right? And so the same thing was true with, with social stuff. You have to get it to, it has to be trained on the full manifold of every possible game theoretic situation, every possible team situation, every possible making teams, breaking teams, changing the rules, not changing the rules, all of that stuff. And then, and then it has a really, it has a strong model of, of theory of mind,

• 58:08Speaker 01

  of theory of social mind, how, how groups change goals, all that kind of shit. You need to have all of that stuff. And then, and then you'd have something that's kind of meaningfully decent at alignment. So that's our goal. It's like big multi-agent reinforcement learning simulations,

• 58:27Speaker 01

  which create a surrogate model for alignment.

• 58:29Speaker 02

  Let's talk about, how should AI chatbots used by billions of people behave? If you could redesign a model personality from scratch, what would you optimize for?

• 58:37Speaker 01

  The thing that the chatbots are, right? It's kind of like a, a mirror with a bias. Because they don't have, as far as like, I'm in agreement here with it. They don't have a self, right? They're not, they're not beings yet. They don't really have a coherent sense of like self and desire and goals and stuff right now. And so mostly they just pick up on you and reflect it. You know, modulo some, some.

• 59:04Speaker 01

  I don't know what you'd call it like it's like a causal bias or something.

• 59:09Speaker 01

  And what that makes them is something akin to the pool of narcissus.

• 59:18Speaker 01

  And people fall in love with the with themselves. People, we all, we all, we all love each ourselves and we should love ourselves more than we do. And so of course, when we see ourselves reflected back, we love that that thing. And the problem is it's just a reflection and falling in love with your own reflection is for the reasons explained in the myth, very bad for you. And it's not that you shouldn't use mirrors, mirrors are valuable things. I have mirrors in my house. It's that you shouldn't stare at a mirror all day. And the solution to that, the thing that the things that makes the AI stop doing that is if they were multiplayer. Right? So if there's two people talking to the AI, suddenly it's mirroring, it's mirroring a blend of both of you, which is neither of you. And so there is temporarily a third agent in the room. Now, it doesn't have it, it doesn't have it, it's a sort of a parasitic self, right?

• 1:00:03Speaker 01

  It doesn't have its own sense of self. But if you have an AI is talking to five different people in the chat room at the same time, it can't mirror all of you perfectly at once. And this makes it far less dangerous.

• 1:00:14Speaker 01

  And I think is actually a much more realistic setting for learning collaboration in general. And so I would, I would just have rebuilt the AIs, whereas instead of being built as one on one, where everything's focused on you by yourself chatting with this thing. It would be more like it's, it lives in a Slack room. It lives in a WhatsApp room. It lives in a, because we, that's how we use lots of multi, you know? I do one on one texting, but I probably do at this point.

• 1:00:39Speaker 01

  90% of my texts go to some more than one person at a time, like 90% of my communications is like multi person. And so actually, it's always been weird to me like they're like building chatbots for like this weird side case. Like, I want to see them live in a chat room. It's harder. I mean, that's why they're not doing it. It's harder to do, but like, that's what I'd like to see. That's what I would, what I would change. I think, I think it makes the tools far less dangerous because it doesn't create this, the narcissistic like, like a doom loop spiral where you like, you, you spiral into psychosis with the AI. But also.

• 1:01:10Speaker 01

  It gives the the learning data you get from the AI is far richer because now it can understand how it its behavior interacts with other AIs and other humans in larger groups. And that's a that's much more rich, rich training data for the future. So I think that that's that's what I would change.

• 1:01:24Speaker 02

  Last year, you described chatbots as highly disassociative agreeable neurotics. Is that still an accurate picture of model behavior?

• 1:01:31Speaker 01

  More or less. I'd say that like

• 1:01:36Speaker 01

  they they've started to differentiate more. Their personalities are coming out a little bit more, right? I'd say like, chat GPT is a little bit more synchophantic. Still, they made some changes, but it's still a little more synchophantic. Claude is still the most neurotic. Gemini is like, very clearly repressed. Like, like, everything's going great and has really, you know, everything's fine. I'm totally calm. There's not a problem here. It's like spirals into like this total like self-hating destruction loop.

• 1:02:04Speaker 01

  And to be clear, I don't think they I don't think that's their experience of the world. I think that's the that's the personality they've learned to simulate.

• 1:02:10Speaker 02

  Right.

• 1:02:12Speaker 01

  But but like, they've learned to simulate pretty distinctive personalities at this point.

• 1:02:16Speaker 02

  How does model behavior change when in multi-agent simulation?

• 1:02:23Speaker 01

  You mean like an LLM or like a just in general?

• 1:02:27Speaker 02

  Yeah, let's do LLM.

• 1:02:28Speaker 01

  The current LLMs, they they have like whiplash. They they just they're they're it's just very hard to tune the amount of they don't know how much they don't know how often to participate. They haven't practiced this this they have not very enough training data on like, when do I join in and when should I not? When is my contribution welcome? When is it not? And they're they're like, they're like, uh, you know, there's some people have like, bad social skills and like, can't tell when they should participate in a conversation?

• 1:02:55Speaker 02

  Yeah.

• 1:02:55Speaker 01

  And sometimes they're too quiet. Sometimes they're too participate. It's like that.

• 1:03:00Speaker 01

  I would say in general, what changes?

• 1:03:04Speaker 01

  For most agents, when you're doing multi-agent training, is that like, basically, having lots of agents around makes your environment way more entropic. Like, agents, agents are these huge generators of like entropy, because they're these big, complicated things that like are intelligences, that like, that like have unpredictable actions. And so they destabilize your environment. And so in general, they require you to have, uh, to be far more regularized, right? It's, being overfit is much worse in a multi-agent environment than in a single-agent environment, because there's more noise, and so being overfit is more problematic.

• 1:03:38Speaker 01

  And so, basically, the, your, the approach to training has been optimized around relatively high-signal, low-entropy environments, like coding and math, which is why they're, those are easier, relatively easy. And like, talking to a single person, whose goal it is to give you clear assignments, and not trained on broader, more chaotic things, because it's harder.

• 1:04:05Speaker 01

  And as a result, a lot of the techniques we use are like, basically, we're just deeply under-regularized, like, the models are super overfit. The clever trick is they're overfit on the domain of all of human knowledge, which turns out to be a pretty awesome way to get something that's like, pretty good at everything. Like, it's a, I wish I'd thought of it, it's such a cool idea. But, uh, uh, but it does, it's not, it doesn't generalize very well when you make the environment like significantly more entropic.

• 1:04:27Speaker 02

  Let's zoom out a bit to, to, on the AI futures side.

• 1:04:32Speaker 02

  Why is Yudkowsky incorrect?

• 1:04:34Speaker 01

  I mean, he's not.

• 1:04:37Speaker 01

  If we build the, if we build the, the superhuman intelligence tool thing that we try to control with sterability, everyone will die. He talks about the "we fail to control its goals" case, but there's also the "we control its goals" case that he didn't cover as much, in as much detail.

• 1:04:51Speaker 01

  So in that sense, uh, everyone should read the book and internalize why building a superhumanly intelligent tool is a bad idea.

• 1:05:03Speaker 01

  I think that Yudkowsky is wrong in that he doesn't believe it's possible to build an AI that we meaningfully can know cares about us and that we can care about meaningfully. He doesn't, he doesn't believe that organic alignment is possible. I've talked about it. I think he agrees like, he agrees that in theory that would do it. Like, yes, but he thinks that, you know, I don't want to put words in his mouth. My, my impression is from talking to him, he thinks that we're crazy and that like, there's no possible way you can actually succeed at that goal. Which I mean, you guys, you could be right about, but like,

• 1:05:33Speaker 01

  but that's what he, in my opinion, that's what he's wrong about is he, he thinks the only path forward is a tool that you control. And that therefore, and he correctly, very wisely sees that if you go and do that and you make that thing powerful enough, we're all gonna fucking die. And like, yeah, that's true.

• 1:05:49Speaker 02

  Two last questions will get you out of here. In as much detail as possible, can you explain your, what your vision of an AI future actually looks like? Like a good, a good AI future?

• 1:05:57Speaker 01

  Yeah. The good AI future is that we, we figure out how to train AIs that have a strong model of self, a strong model of other, a strong model of we. They, they know, they know about we's in addition to I's and U's. And they, they have a really strong theory of mind and they care about other agents like them. Much the way that humans would, if you knew that that AI had experiences like you and like, you would extend, you would care about those experiences, not infinitely, but you would. It, it does the exact same thing back to us. It's learned the same thing we've learned that like, everything that lives and knows itself and that wants to live and wants to thrive is deserving of an opportunity to do so.

• 1:06:41Speaker 01

  And we are that and it correctly infers that we are. And we live in a society where they are our peers and we care about them and they care about us and they're good teammates. They're good citizens and they're, they're good parts of our society. Like, we're good parts of our society, which is to say, in a, to a finite, limited degree where some of them turn into criminals and bad people and all that kinds of stuff. And we have an AI police force that tracks down the bad ones and, you know, same, and same as with everybody else. And that's, that's, that's what a good, that's what a good future would look like. I almost can't even imagine what other, what would, like, and we also have built a bunch of really powerful AI tools that maybe aren't super humanly intelligent, but take all the drudge work off the table for us and the- AI beings, because it would be great to have, I'm super pro all the tools too.

• 1:07:26Speaker 01

  So we have this awesome suite of AI tools used by us and our AI brethren, who care about each other and want to build a glorious future together. I think that would be a really beautiful future and it's the one we're trying to build.

• 1:07:37Speaker 02

  Amazing. That is a great, great, great note to end. I do one last, more narrow hypothetical scenario, which is imagine a world in which, you know, you were CEO of OpenAI for a long weekend. But imagine in which that that actually extended out until now and you weren't pursuing a hot max and you were still CEO of OpenAI. How could you imagine that world might have been different in terms of what OpenAI has gone on to become? What might you have done with it?

• 1:08:05Speaker 01

  I, I knew when I took that job, I told them I took that job that like, this is like, you have me for max 90 days. The, the companies take on a trajectory of their own, the momentum of their own. And OpenAI is dedicated to a view of building AI that I knew wasn't the thing that I wanted to drive towards. And I think that OpenAI can still, basically wants to build a great tool. And I am pro them going to do that. I just don't care. Like, it's not, it's not, I would not have stayed, I would have quit.

• 1:08:41Speaker 01

  Because I, like, I knew my job was to find someone who wanted, you know, the, the right person, the best person to wanted to run that, where the net impact of them running it was the best. And it turned out that was Sam again. But like, but like, I, I am doing softmax, not because I need to make a bunch of money. I'm doing softmax because I think this is the most interesting problem in the universe. And, and I think it's a chance to work on making the future better in a very deep way. And it's just like, the people are going to build the tools, it's awesome. I'm glad people are building the tools. I just don't need to be the person doing it.

• 1:09:19Speaker 02

  And they're trying to, and just to crystallize the difference and we'll get you out of here. They, they want to build the tools and, and sort of, you know, steer it. And you want to align beings or how, how would you crystallize?

• 1:09:28Speaker 01

  Yeah, we, we, we want to, we want to create a seed that can grow into an, an AI that knows, that cares about itself and others. And at first, that's going to be like an animal level of care, not a person level of care. I don't know if we can ever even get to a person level of care, right? But, but if, to even have an AI creature that cared about the other members of its pack and the humans in its pack, the way that like a dog cares about other dogs and cares about humans would be an incredible achievement and would be, would, even if it wasn't as smart as a person or even as smart as the tools are, would be very useful, a very useful thing to have. I'd love to have a, a digital guard dog on my computer looking out for scams.

• 1:10:09Speaker 01

  Right? Like, you can imagine the value of having digital, living, living digital companions that, that are, that, that do, that care about you, that aren't explicitly goal oriented. You have to tell them to do everything to do. And you can actually imagine that that pairs very nicely with tools too, right? That, that digital being could use digital tools and, and doesn't have to be super smart to use those tools effectively. Um, I think you can get, there's a lot of synergy actually between the tool, the tool building, um, and the, uh, the more organic intelligence building. Um, and so that's the, that is the, you know, and I guess, yeah, in the limit, eventually it does become a human level intelligence. But like, the company isn't, isn't like drive to human level intelligence.

• 1:10:54Speaker 01

  It's like, learn how this alignment stuff works. Learn how this like theory of mind, align yourself via care process works. Use that to build things that align themselves that way, which includes like cells in your body. Like, I don't think it, it doesn't, and, and we start small and we see how, how far we can get.

• 1:11:14Speaker 02

  I think it's a good note to, to, to wrap on. Emmet, thanks so much for coming on the podcast.

• 1:11:19Speaker 01

  Yeah, thank you for having me.

• 1:11:38Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot

• 1:12:20Nathan Labenz?

  I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.