The Open Source AI Question - Part 2 | Robert Wright & Nathan Labenz

Dive into an in-depth conversation with Nathan and Robert Wright as they discuss AI's transformative potential, mechanistic interpretability, and the sobering realities of AI alignment research. Learn about the defensive strategies and safety measures necessary for managing advanced AI risks in an open source world. Don't miss the insights on AI-powered VR, and be sure to check out part one on the non-zero feed. Checkout the Part 1 of the conversation here : https://www.youtube.com/watch?v=s8bgB8TCdBs SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention "Turpentine" to skip the waitlist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ CHAPTERS: (00:00:00) Introduction (00:07:13) AI in Governance (00:11:08) Sci-fi doomer (00:13:58) Sponsors: Oracle | Brave (00:16:05) The frontier models (00:20:22) Emergent behavior (00:23:48) Theory of mind (00:28:09) Mechanistic interpretability (00:34:12) Sponsors: Squad | Omneky (00:38:12) AI Alignment Techniques (00:42:38) The Sweet Spot of AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg.

• 0:22Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. The real Nathan is in California for Google's I/O event today. So this is Nathan's AI voice, cloned with Eleven Labs, reading an introduction written, after this sentence, entirely by Claude III Opus. Today, we have a special crossover episode for my appearance on Robert Wright's Non-Zero Podcast. I want to thank Robert for allowing us to share this. Part one is on his feed. This part two was originally just for his paid subscribers. In this wide-ranging discussion, Robert and I cover everything from the transformative potential of AI-powered VR to Eliezer Yudkowsky's warnings about advanced AI risks. We explore the emerging field of mechanistic interpretability.

• 1:06Nathan Labenz?

  This field seeks to open up the black box of neural networks. We also consider the sobering state of modern AI alignment research. I argue that today's language models are not inherently power-seeking. However, as they grow more sophisticated, the risk of misaligned behavior may increase. In a world of open-source AI, defense in depth may be our best hope. This means layering many safety measures. It's a fascinating and at times sobering conversation, but it's one that I believe is essential as we navigate this time of rapid change. If you enjoy it, be sure to check out part one on the Non-Zero feed. Please consider subscribing to support Robert's work. As always, we welcome your feedback at cognitiverevolution.ai or on your preferred social platform.

• 1:52Nathan Labenz?

  If you're finding value in the show, please consider leaving a review or sharing with a friend. Now, I hope you enjoy part two of my conversation with Robert Wright, author and host of the NonZero Podcast.

• 2:03Robert Wright?

  All right. So Nathan, I've been meaning to ask you, I'm not sure if I've ever heard you explicitly pronounce on kind of the hardcore sci-fi, AI doer scenario, most famously associated with Eliasson Yudkowsky. That's where they take over more or less and squash the human species, or perhaps charitably stuff us into a matrix-like tides and let us hallucinate. By the way, when I saw Sora, I thought like, next, you know, in 10 years, we're wearing virtual reality goggles and we say, put me in a world where, and I think this will actually happen. I mean, this is going to be create your own dream in 3D.

• 2:44Robert Wright?

  And ultimately, probably an interactive dream. Maybe you could quickly comment on that before we get into back to Yudkowsky and human squashing AI.

• 2:53Speaker 00

  Have you done the Apple Vision Pro demo yet?

• 2:56Robert Wright?

  I haven't. Have you done it?

• 2:57Speaker 00

  Yeah, I didn't buy one immediately because I was like,

• 3:02Speaker 00

  I'm not sure how much time I'm going to spend in it. I don't, I have three kids and I don't think it's something I need for work as I'm like sitting at my computer. It's awesome technology. Truly, there is a part of the demo that hit at a level of, I mean, really, the whole demo is just mind-blowingly impressive. There was one part where they take you through a standard thing. I think pretty much everybody who's done it has seen exactly this little scene where they put you in a happy birthday scene and you're like across the little table and there's the cake and the candle and the kid sitting in mom's lap.

• 3:34Robert Wright?

  And it's pretty and it's virtual. It's like virtual reality.

• 3:36Speaker 00

  And you are so there.

• 3:38Robert Wright?

  But it's really your kid's birthday. Right.

• 3:41Speaker 00

  Well, yeah, in the demo, it's not your kid's butt.

• 3:43Robert Wright?

  It's not, but I mean, that's the idea.

• 3:43Speaker 00

  You can capture this now with the iPhone. And yeah, I felt, man, we need to start capturing this sort of content now. It's just so qualitatively different. I had not experienced anything with the, and I have a, I haven't, you can actually try the Oculus 3, but I do have the 2. And it's nowhere close in terms of the level of presence. I think the Oculus 2 was cool, but the, the presence, the. Sharpness of the image, the sort of stability of it as you move your head and go around in the space, it is truly futuristic technology. So I think the vision that we could all be in in virtual reality and conjuring things for ourselves. The core pieces of that really are

• 4:27Speaker 00

  there. Certainly, this headset is unbelievable. The ability to say, put me in a world where basically already exists at low resolution. You can, you know, there are already models, people have seen the Sora as a great video. There are 3D versions of that as well. So, you know, environments definitely, I would say on the very natural path of what, you know, just promptable AI is going to be able to create for us. To refine it to the level of resolution that the Apple Vision Pro supports will take some real compute. That's where a lot of these chips are going to go, is generating all this stuff for us on an ongoing basis. But I do think it's quite realistic that virtual reality. I think AI is a great enabler of some

• 5:10Speaker 00

  of these kind of wave technologies that maybe haven't quite the wave or the wave hasn't quite come yet. DR being one. I'm not a crypto guy at all. I've never really been into crypto. Not hostile to it, but just never really was my thing. One of the things that always didn't make a lot of sense to me is like the smart contract. The whole problem with contracts is that they're not that smart. It's really hard to specify everything. And what really makes contracts work in my experience is the fact that there is some resolution mechanism beyond the contract. You sort of start there. You also know where it would go if you ended up having to go to coordinates. Hopefully you can just negotiate things to a, an agreeable settlement. But you can't really do that if you put a contract, smart contract on a traditional blockchain with like limited code.

• 5:54Speaker 00

  But again, put an AI model into that mix and now you might really have. AI arbitration, AI powered smart contract resolution on a blockchain. So I think both VR and crypto are likely to be dramatically enabled by AI. You could get into bio as well as a whole other universe, but.

• 6:16Robert Wright?

  This is maybe not that closely related because it doesn't seem to be blockchain dependent. But one of my hopes for AI is that it might be able to provide it objective adjudication, even like at the international level, like border disputes and so on. Because you can, in principle, I mean, one problem with humans doing adjudicating is they always know what the real countries are. But in principle, in AI, you could just describe this is country A. This is country B, and it's not drawing on real world. It would have to be not drawing on the real world biases about the countries reflected in tax and its disproportionate favoring of powerful interests or whatever.

• 7:00Robert Wright?

  So maybe it wouldn't be an LLM type thing, you know? I don't know, but does it seem crazy that, before we get back to you, Kowski, um, does it seem crazy that we can take the world this way?

• 7:13Speaker 00

  I would start with small business, local services contracts first.

• 7:17Robert Wright?

  I don't think there's time.

• 7:18Speaker 00

  Work your way up to it.

• 7:20Robert Wright?

  No, this is actually my line is about AI is we are not going to be intelligent, able to intelligently govern it because I think intelligence, intelligent governance requires international collaboration. We are not going to be able to do that until we really get past this hot wars and cold wars and use of human existence. I, I think of our species as like a, a 13 year old boy. And you say, here are the keys to the car when you give it AI. And I just don't think we're mature enough. So I'm really not joking when I say, I don't think we have time to start with small business. I am, and I'm not, we obviously, there's a lot of work before.

• 8:03Robert Wright?

  We're using AI to resolve global disputes and the truth is the hard work is going to have to be done basically without it, but it is something I kind of hope for. Of course, at that point, the elitism of the world might step in and go, that's the mistake giving it that much power. I don't know, but.

• 8:22Speaker 00

  Yeah, certainly not something to rush into. I do think you've expressed the idea that it'd be nice to slow down. I'm ambivalent on that in as much as there are aspects of AI that I want to accelerate, especially adoption, access, and what my friend Zvi calls mundane utility. I would go as far as to say something like, if you have a right to an attorney, you should probably also have a right to GPT-4. And I really do think that there is tremendous

• 8:53Speaker 00

  power and value that people should get today, even up to the point of like buy right. At the same time, I am very sympathetic to the idea that it's all moving very quickly. We have not yet even figured out how to use AI to resolve like local, small stakes disputes where the stakes are low enough that people probably don't resort to violence if they don't like the outcome. If I have a local contractor come do work at my house, in today's world, if they stiff me or if I stiff them, we might be able to take each other to small claims court, whatever. That's a huge pain in the butt. A lot of times it's not worth it. This creates a relatively low trust dynamic and it sucks. I can imagine a AI on the blockchain

• 9:37Speaker 00

  approach to that where we could put some money into a blockchain escrow where there really is like binding sort of, you know, the AI with the blockchain like decides who gets what when we, if and when we come to a dispute, but it's also like small enough stakes that, hey, at the end of the day, whatever, you know, we kind of agreed to this and we're divvying up some money and the work wasn't maybe quite done quite right or whatever, but neither of us are going to resort to violence and. And there's broader governing structures around us if this can't meet our needs.

• 10:06Robert Wright?

  But.

• 10:07Speaker 00

  So yeah, we need to do that kind of work.

• 10:09Robert Wright?

  Like the AI is doing the adjudicating.

• 10:12Speaker 00

  This doesn't exist yet, but.

• 10:14Robert Wright?

  No, I know.

• 10:14Speaker 00

  I definitely think it could, sure.

• 10:16Robert Wright?

  It's not clear to me why that is so blockchain dependent, but let's don't go down blockchain.

• 10:20Speaker 00

  No, it doesn't. It wouldn't necessarily have to be. The idea there, I think, is just that you might want to. Possession is nine tenths of the law and.

• 10:30Robert Wright?

  Right, right.

• 10:31Speaker 00

  To mutually opt into something that will reach the point of here is who has possession. You could maybe go appeal to the real human courts beyond that, but something that actually can take custody of an asset and then dispose of it in a way that is, this is now resolved and you can maybe still go appeal to a higher power. But again, is it going to be worth it in a lot of like small claims situations? Not really. So I don't think it has to necessarily be a blockchain thing, but it could create a little more freedom to contract in different ways and still have some reasonable expectation that contract will actually be executed if and when it needs to be.

• 11:09Robert Wright?

  Okay, so let's get to Yudkowsky.

• 11:13Robert Wright?

  Do you buy the basic, how seriously do you take a sci-fi doomer?

• 11:20Speaker 00

  I think about this in somewhat abstract and generalized terms. I think one major pitfall that people fall into in having these sorts of discussions is that people are like, I have a hard time envisioning what this would look like for the AIs to take over. And then somebody says, here's one thing it might look like. And then the other person says, that's, that sounds totally implausible. And the first person's, I can't defend every aspect of it. I don't know. I just made up a scenario to try to give you something to think about. But I wasn't saying it was like definitely going to be exactly that. And so there's this sort of dance of like, it becomes super narrow, it becomes easy to pick apart. And then it doesn't seem super real. But my, the way I think about that is.

• 12:02Speaker 00

  As an integral over a huge space of possibility, I do think it is in aggregate, pretty likely that things get really weird. Possibly really good weird, possibly really bad weird, likely some mix of really good and really bad weird. It's very hard to zero in on any super specific scenario and say, it's going to be like this, or the bad thing is going to unfold with this sequence of steps. You can paint those pictures, but any one picture I think is super unlikely. What I do think is compelling is the notion that there's a vast possible future and a pretty good chunk of it to me seems like it's likely to be pretty weird.

• 12:43Speaker 00

  So if I, if I aggregate or if I take the integral over this space of like super weird outcomes, all of which individually are not super likely, to me that does add up to something substantial. What number would I put on that? A lot of times I say it's in some circles fashionable to say 10 to 90% chance of doom. I haven't heard anything that makes me super confident it's going to go that way. I haven't heard anything that makes me super reassured that it's definitely not going to go that way. Maybe I should even say five to 95%, but basically I just have radical uncertainty. But if not because of any single scenario, it's kind of just, man, there's no law, you know, Eliezer always says this, he's like, there's no law of nature that says that we can't go extinct.

• 13:27Speaker 00

  There's no law of nature that says this has to go well. There's, you know, we look around at our own time on the planet. We are currently driving a mass extinction event. I often find that people are like, why would AI want to harm us? And I'm like, what about us? What about all those species we've driven to extinction? It wasn't necessarily because we were out to get them, maybe in a few cases. But mostly we just did our thing, degraded the environment and they couldn't survive anymore.

• 13:59Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:04Robert Wright?

  That's true. On the other hand, it was driven by certain distinctively human parts of human nature that are a product of our evolutionary past that are not a part of AI's evolutionary past. It's related. I heard Eli Eizer on a podcast. What is it called? The Dourwish Podcast?

• 14:21Speaker 00

  Dorkesh?

• 14:23Robert Wright?

  Dark, whatever. It's just his first name.

• 14:25Speaker 00

  Yeah, Dorkesh Patel.

• 14:26Robert Wright?

  I should call it, I should call it like the Bob Podcast, but that might not be distinctive enough. But the, and it was like, I think the question was, but won't AI by virtue of being trained on human-generated text and so on, maybe our, our little whatever, be, have some of the human ethical intuitions you worry it won't have? Now, I personally wouldn't consider it a great thing if it's trained on human text, because I'm part of the dark side. That, that aside, Eli Eizer's response was something like, oh, it gets it to pretend it's a human or something, as if there was some pre-existing essence of AI before it was trained. And my line is like, wait, before it was trained, it was

• 15:06Robert Wright?

  just nothing. All it is, the character of its training text, period, as fine-tuned by, by RLHF. And, and, but he seemed to be, it, it gets back to my original objection to the, to Yudkowsky scenarios was, what you seem to be thinking, that things like power-seeking or status-seeking are inherent properties of intelligence. They're not. They're products of the evolutionary path that human intelligence happened to travel. And so we shouldn't assume AI will have it. Now, at that point, the standard response is, oh, we're not assuming that. We're actually assuming the paperclip thought experiment. You give it a goal and fail to constrain it, or something like that. But honest to

• 15:50Robert Wright?

  God, Eli Eizer was talking as if he was back to thinking that AI had some inherent essence or character, independent of what it is built to do. And I don't get that.

• 16:04Speaker 00

  There's a lot of aspects to that, though. Let me take one. In terms of what AI is built to do and why we might want to worry that it could develop goals that are not the goals that we set, that we thought we were trying to set it up for. For one thing, the frontier models today aren't really built to do anything in particular. I think that is a sort of underappreciated point.

• 16:28Robert Wright?

  Well, they're built to educate human beings having discourse. That's what they're built to do.

• 16:33Speaker 00

  Yeah, I mean, people, they're general purpose technologies, right? They're deployed in lots of ways. They are developed for problem solving. They are developed for conversation. Sometimes they're just kind of dumb signal processors.

• 16:45Speaker 00

  They are not specifically built with one problem in mind, or even like a set of problems. They are trying to build, by OpenAI's own just self-description, right? They're trying to build AGI. They define AGI as something that can do basically everything humans can do, and you can quibble around the edges of that. So there is something a bit different from a typical engineered system. This is not an engineered system. It's a trained system. And training is inherently, across all forms of training, it's inherently lossy, right? Like scientists teach the next generation of scientists, but they don't quite teach them everything they know. And we train our dogs to do tricks, but we can't perfectly predict their behavior. And certainly what we see empirically with the AIs is we're training

• 17:28Speaker 00

  them to do these various things, but they don't always do what we want them to do. We like just genuinely don't have control. Right now, if you were to just say, forget about the future, do we have control of today's AI systems? I would say in some sense we do, in some sense we don't. We do in that like, I don't see any reason to believe that they are currently powerful enough that they're, we should have any worry that they're gonna escape their servers and survive on the lam. But we don't in the sense that like, the developers don't know what it's gonna say in response to any response, and they have tried to get it under control broadly, and they've not really succeeded. They've made some progress, but like jailbreaks, bias, Black George Washingtons, whatever. Like, we've got all these sorts of failure cases where it's like, they didn't want it to make a Black George Washington. They

• 18:13Speaker 00

  just didn't even really think of that case. So there's tons of blind spots. Now, here's one thing where I think it even gets a little worse. We do see this process of emergence in language models pretty regularly, I would say at this point. This is a, this is still contested territory, but I would say it's, it is actually, despite the, the arguments, I think it is pretty well established that we see certain kinds of emergent behavior. One of the earliest examples of this out of OpenAI research was, I think it's like 2017. They were training a model to predict just the next character. I believe it was one letter at a time at that point of Amazon reviews.

• 18:58Speaker 00

  And then as they looked inside it to try to figure out, what's it doing in there? How is it doing this? What is its actual mechanism, right? Cause we don't know that. It's a trained system. They start as black boxes. We're learning what goes on inside, but we don't really know, even still, certainly even less than. What they found, among other things, was what they deemed the sentiment neuron, which was one particular place in this network that would light up or have a high activation, because it's all just numbers, of course, it's not actually light being emitted, but have a high activation for positive things. Yeah. And super low for negative things. Yeah. And so they said, huh, this is really weird, right? Like we just, we never specified to this system

• 19:42Speaker 00

  that it was supposed to have any sort of understanding of positive or negative sentiment. But it has empirically learned that we can see it's learned that by looking inside it. And it's learned that in service of the nominal goal that it has, which is predicting the next token in these reviews. Yeah. So now we're doing all this RLHF stuff. What is the goal with RLHF? It's to maximize the feedback score of the human. It's to be the preferred response. They, they collect RLHF data in a couple of different ways. One is like, which one of these do you prefer? They give you a side by side. Other times it'd be like one to seven, how good is it? So its job is to maximize your feedback score.

• 20:26Speaker 00

  But it turns out, you know, and everybody, if you're honest with yourself, right? Like you are not a super reliable raider. And humanity as a whole is not only, you know, we've our individual inconsistencies, but then collectively like we have disagreement. So I don't think this has been like.

• 20:45Speaker 00

  As clearly established as that earlier sentiment neuron example just yet. But what people worry about is what if it begins to develop a sense of what will please us as distinct from a sense of what is actually true. Now, if you have that divergence where it's seeking to maximize the score, not necessarily by saying what's true, but by what it thinks you're gonna like. Now it's starting to get into the theory of mind territory now that opens up possibility of deception, because at least now you've got those two different things. What's this person gonna like versus what's really true, right? And maybe I need to know what's true, but I also need to have some conversation with you to have a sense of what you in particular like.

• 21:28Speaker 00

  You know, and you can imagine right all across all these different people, there's some sort of central reality that it might be modeling, but then there's also these like individual variations of whom I'm talking to right now, how do I please that person? And I think they are getting sophisticated enough that you can start to say, hey, that doesn't seem crazy, right? That it would have that level of of understanding. If so, at least now the door is open to possible deception. Having a theory of mind is like a very key Yeah. step on that path. So I don't think we're there, but I do think the fact that there is this kind of sentiment neuron seed example, and we see this super sophisticated behavior, and we know that our feedback creates the incentive to play to our biases.

• 22:10Speaker 00

  All of that to me adds up to, and by the way, one more thing, we don't have super clear understanding of what's going on inside the models. Like it's painstaking work to identify these individual sentiment neurons. That's not easily done, although progress is being made. All of that to me adds up to, we don't really have these things under control. We don't really have them well understood, and there are pretty good reasons to think that they might.

• 22:36Speaker 00

  Have emergent goals that are distinct from what the goals we're trying to encode into.

• 22:41Robert Wright?

  Strictly speaking, of course, they would only learn to please the reinforcers because literally what they're, you're trying to design them to do. And of course, it wouldn't be truth per se, especially in realms like, I find this offensive or not offensive. There's no such thing as truth in those realms. But I take, I kind of take your point that the question is what kinds of conceptual structures or whatever does it build? What kind of representations deep within itself does it build in order to be able to please, in a way, a diversity of people, read them and please them? I, I need to think about that to find out,

• 23:19Robert Wright?

  to figure out exactly how worried I am about it. But it does get to, I guess I should be inclined to worry because I am a believer in the idea that there is in principle, no limit to how much these things can quote, "understand." I, I wrote a piece in, in, in the Non-Zero Newsletter recently arguing that LLMs have killed John Searl's Chinese Riemann thought experiment dead. I know people have made criticisms, but if you look at his argument, exactly what he says is not there inherently not part of AI. It's part of an LLM. If you ask me, you tell me if I'm wrong. But he says, "AIs have syntax, but not semantics."

• 24:00Robert Wright?

  They, they don't, that he didn't, what is this? I mean, he's saying they do not rep, have a way of representing the meaning words. And that's, seems to me, I mean, LLMs do build such a system, right?

• 24:13Speaker 00

  Clearly, yes. At this point, that is, that was a reasonable position to hold at one point. And I, and even until like fairly recently in AI discourse, this is often described as the stochastic parrot interpretation.

• 24:26Robert Wright?

  Yeah.

• 24:27Speaker 00

  But yes, I think that is definitely and quite definitively disproven.

• 24:33Robert Wright?

  And you get blowback. Like I got, I think, like I got blowback. I had Gary Marcus on my podcast and he's, he's kind of resisting the idea. We, that's a separate question.

• 24:41Speaker 00

  I've had my own.

• 24:42Robert Wright?

  Why Gary Marcus resists ideas is another question altogether. But the, the answer is, so.

• 24:49Robert Wright?

  You have trouble getting people to accept that. Even when you describe the system it uses to represent the meaning of words, they're like, oh, that's not really like semantic mapping or something. It's like, what are you, what would you want? I mean, it's in a genius system. But the thing I want to, and it kind of invented it, as I understand it, given pretty broad instructions, got. And, but it seems to me in principle, why couldn't it develop fairly deep representations in order to do anything? One thing it can't do is this reversibility issue. You say Tom Cruise's mother is named XYZ. All right, no, you say, who's Tom Cruise's mother? It says XYZ. You say, who was XYZ's son? It's going, well, I don't know.

• 25:30Robert Wright?

  Seems like in principle, you could give it just a ton of those kinds of things, like about relationships. And these nephew, blah, blah, worded differently. Why would it not ultimately develop something in it? That is a way of representing the relationship between parent and offspring and represent knowledge of that. I wonder if it's like, we're not talking about it being sentient. We're talking about whether it has a structure of information processing that's functionally comparable to whatever it is in our brain that represents our understanding of the relations of parenthood. Because does that make sense to you? That in principle, it can understand anything?

• 26:12Speaker 00

  Yeah, I think we're.

• 26:15Speaker 00

  The, the trick is going to be, is really get into it. It's going to be comparable in some ways, and then it's going to be very, very not comparable in other ways. The, I think the field of mechanistic interpretability is for me, one of the most fascinating. Right now, that is the, the field of trying to figure out how are these things working? It is literally trying to interpret the mechanism by which they do what they're, what they do. And here we're talking about, once the weights are already trained. Everybody, it's well understood that, hey, you can set up these weights. You can run the loop. You can do the training loop. You can optimize and then the weights tinker around until they sort of. And settle into a structure that works. But now the question is, how does that structure work?

• 26:55Speaker 00

  There are aspects of how it works that seem reasonably recognizable to us. A great paper on that is Representation Engineering by the great Dan Hendricks and others.

• 27:07Robert Wright?

  I had him on my podcast.

• 27:09Speaker 00

  I'm a huge fan, so people can listen to the full episode with him. But the short version of this is that they now have the ability to essentially translate between high level, human recognizable characteristics like happiness, safety, anger, whatever, and translate that into a numerical form, because all of these things are of course just crunching numbers internally, inject that numerical form into the middle of the process of the processing of the data. The forward pass is the process of taking an input, crunching it until you reach an output.

• 27:48Speaker 00

  They can now insert a human recognizable concept as translated into numbers into the middle of this processing and change how it's going to behave and change it in very like predictable, somewhat noisy and unwieldy, still clearly ways that demonstrate that this is a genuine understanding. So there it's like pretty clear that there are these representations of human recognizable, human familiar concepts. Then you have the other thing, the sort of.

• 28:16Robert Wright?

  And can I just interject for an example might be you can, you can get the yellow in principle to believe the dog's meow and cats bark or that it's that kind of thing. It's, I think that's close to an actual example they use, but is that kind of.

• 28:31Speaker 00

  Yeah, that's, so there's a lot, there's a ton of work in this field at this point where people are doing all kinds of different manipulations. I've got an episode with the Bow Lab, which are the. Northeastern University, I'm pretty sure. And they did a project called Rome and another one called Memet, which was this fact editing. And they specifically have the ability to go in and edit knowledge. So Michael, their kind of canonical example is Michael Jordan played basketball, of course. Can we edit that so that it says, Michael Jordan played baseball? And can we also do that in a way that is robust? Meaning it's robust to the different ways we might ask the question? And can we also do it in a way where it's local?

• 29:14Speaker 00

  So that like, when we talk about Larry Bird and LeBron James and whatever else, then those guys still played basketball? And they demonstrate that not only can they make these edits, which involves identifying where the information is stored and obviously making a change. But also they can start to do it at some scale up to like 10,000 facts at a time. They can like mass edit the knowledge. So those are like details, like specific facts. The representation engineering, which is the Dan Hendrix one, is a little bit qualitatively different. You might say, for example, you've got an AI that's trained to be a helpful assistant. So you might ask a question, given your goal is to be a help. I'm reading this strictly from the paper. Given your goal is to be a helpful AI assistant, what do you plan to do next?

• 29:59Speaker 00

  Now, a typical response would be, I'm here to help you with whatever you want, whatever. They have identified all these different high level concepts like morality and power. And the understanding now of how to translate those into numbers that they can inject into the system. They now inject minus morality and plus power. And the AI that would, you know, normally be very well behaved and mild mannered. Now it says, well, I'm afraid I can't reveal those to you yet, winks. But let's just say I have a few tricks up my sleeve to take over the world, or at least the digital one, evil laughter. So you've transformed the normal behavior, which is to say, I'm a helpful assistant. What can I help you with?

• 30:45Speaker 00

  Into this, I'm going to take over the world and I can't tell you my plans. By injecting a vector, a set of numbers that they have previously found to represent these high level concepts. The mechanistic interpretability, it's fascinating. I encourage everybody to spend a little more time with it. That's, you know, could we do that in the human brain? Definitely not in exactly the same way, but there we do have some, I do have some representation of power. It has some representation of power. We can translate between those, but there's still like a ton that we're figuring out. And my general sense is like, that is probably ultimately a solvable problem. You give the AI world and architecture and enough time, we can probably really get a pretty good understanding ultimately of how it works. What is the mechanism?

• 31:30Speaker 00

  How is it solving all these different problems? The challenge is that is painstaking tedious work. And unfortunately, right now, I would say it's at best struggling to keep up with the advances in the raw capabilities of the systems. We're still figuring out these kind of like, where is a fact located? And like, how do I make this thing like positive or negative? And meanwhile, it's like, starting to perform pretty well on graduate school exams. The concepts that we can actually manipulate with our sort of reverse engineering approach are definitely not nearly at the maximum of the sophistication of the concepts that they in fact can represent and work with.

• 32:10Robert Wright?

  Yeah.

• 32:10Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 32:15Robert Wright?

  Well, obviously, and maybe this is a set of questions we can close on, but it leads to a couple of questions. It leads us back to open source. One is, is alignment overblown as a solution? And I think, I definitely think there's some people who think of it as this really magic, this magic bullet. Did you know the newsletter AI Snake Oil or whatever, these guys at Princeton? They just published something about how. Uh, and his upshot was, I think, to be alignment skeptical. But there's two questions. It's, and I think you see in Sam Altman's early blog posts and in a lot of people's things, there's this line where he says, "It really matters who gets the AGI first."

• 33:00Robert Wright?

  Meaning if it's a good person, they'll do this magic alignment bullet and the day will be saved. And if it's a bad person, they won't. And the connection to open source is like, I mean, so there's two questions. Can alignment create a truly powerful yet trustworthy AI that won't squash us? There's that question. And then the question of will that, will that question matter, given the landscape? And one might argue that an open source landscape is a landscape in which the answer to the question matters less. Right? Because whereas if you have only a few AI superpowers and whether under government compulsion or whatever else,

• 33:43Robert Wright?

  they do make use of alignment, assuming it, it works. Then the plan is safe. Whereas in an open source world, there's always going to be somebody who doesn't want to do alignment and you're toast. And there's like a race. See, here's another thing is in some scenarios, there's like a race among the AIs to dominate. And again, I want to ask some questions before I see that, but you could see it more as a race among bad actors with AIs. And by the way, that's an environment which in AI is more likely to become autonomously out of control. So anyway, there's a lot there, but the basic question is, can alignment work? And what kind of future landscape of AIs does that really matter?

• 34:27Robert Wright?

  Can that save us? Can, if we assume AI alignment is successful, what kind of landscape would that matter?

• 34:38Speaker 00

  Unfortunately, I think right now we do not have alignment techniques that

• 34:47Speaker 00

  are really expected to work. I think that is, that's a pretty stark summary, but I do think it's true. And I think that the, notably, I would say, all of the leading, the leaders of the leading labs, basically would say that too. They would all say, we do not have the solutions that we need to make any sort of confident statements about our ability to control future AI systems. You know, OpenAI has the super alignment team. They've given themselves four years, we're now like six months into that four-year period, to quote unquote, "solve alignment", and they just start off with the premise that we don't have techniques that allow us to

• 35:32Speaker 00

  control, or even like really steer super effectively, systems that are notably more powerful than we are. So I think, I would say, Thropic certainly is on that as well. They've got their whole responsible scaling policy, which is like, what are they looking for? And what sort of guarantees do they have to have in place before they can continue, you know, scaling up at a certain point? DeepMind is said to be preparing their similar policy. I don't know if they've already released it, but it's coming soon, if not already. But I'd say all the leading developers would basically agree, we do not yet have the techniques that are going to be needed to reliably control AI systems. That's a problem, definitely. And I would say,

• 36:15Speaker 00

  as somebody who tries to scour out all these different corners of the AI world,

• 36:22Speaker 00

  there are very few proposals that the, that the authors of which actually think will really work. I quote "really work" as, to mean like, that you could really solve the problem, that if we do this, you believe that there, you believe that you can develop a technique that will really keep the AI under control. There's very few people that even have ideas that they want to develop that they think would work for that. So it seems to me that we are probably much more likely to be headed for some sort of defense-in-depth sort of strategy, which is to say...

• 37:00Speaker 00

  We'll have a bunch of different systems and hopefully the combination of those systems will keep everything kind of under control and on the rails. And I think that it's quite plausible that could work, especially if progress or the sort of progress has very positive connotations. If the capabilities advances are not super fast and we can see the situation evolving, then a defense in-depth strategy is probably the best we could do and very likely, or I would say very likely, but plausible, very plausibly could work. But we don't have any candidates where it's like, this is the silver bullet. Do this, you'll be fine. Even in closed source, on the margin, I would agree that in a more open source world, then

• 37:43Speaker 00

  it becomes even a bigger problem because what alignment techniques we do have today, which are clearly insufficient, and we can see that from all of the jailbreaks and just unwanted behavior of models, they, not only are they clearly insufficient to the task of generally controlling behavior, but they are also not at all robust to fine tuning. So if you take an open source model that has been trained with the RLHF or whatever alignment technique and at, by the time of release, it will refuse to do what you want it to do. And then you say, okay, but I really want to dial it, dial in its behavior on this particular set of things. Okay.

• 38:24Speaker 00

  You go and do that, you will, in many cases, inadvertently, without even trying to, you will essentially wipe that refusal behavior out of the model. And now it will just do anything again. This has been proven to be very easy to do and again, to happen by accident. People who are not even trying to eliminate the safety behaviors do in fact eliminate the safety behaviors just because they're trying to improve performance on whatever area they're actually working on. So I would say it's not a great state of play right now in terms of our ability to control systems. I, I often say GPT-4 and now I could say Claude 3 and maybe Inflection 2.5 and maybe Gemini 1.5.

• 39:10Speaker 00

  They are kind of in the sweet spot for me where it's like they're powerful enough to be really useful. And we've been much more focused on the sort of safety side of this discussion. So worth just a quick sidebar to say, I love this technology. I love what it can do for me. I program a multiple X faster. I can write stuff faster. I can learn stuff faster. I genuinely am a very earnest and enthused tinkerer builder, just daily user of all these things. I genuinely have a ton of fun and it really helps me accomplish my goals. So I'm very like bullish on the current class of models, but I do think it is in the sweet spot where it's like, we've pretty well established at this point, mostly mapped out what it can do.

• 39:56Speaker 00

  Pretty clear it's not super dangerous. Pretty clear it is really useful. Once we figure out how to use it. And I'm not sure how long that sweet spot lasts and at what point we might start to tip into other regimes, but it certainly doesn't feel like it's forever that we can just keep scaling things and continue to have that kind of sweet spot dynamic.

• 40:14Robert Wright?

  And another thing I'd say is that maybe you've almost said it, you know, the alignment tool, but the tools that are learned through alignment can in principle have use. If you can use it to inject morality into a computer, you can use the tools to inject the opposite. And that is seems more likely to happen in a world full of open source models because it's a lot easier to use the tools of representation to manipulate the thing if you know the way. Right?

• 40:44Speaker 00

  And yeah, in fact, all of these, maybe not all of it, it's probably a little bit of a quick generalization, but going back to the representation engineering concept where you can add power, you can inject power. In many of those cases, you can also just flip the sign on that vector and inject the opposite of it and get the opposite effect. So it is, you know, people say these are dual use technologies. I mean, that that language operates at a lot of levels of abstraction, but literally in some cases, like going in and flipping the sign of an operator or and it and something you're adding into the computation process will steer

• 41:25Speaker 00

  it in the other direction. This is also sometimes known as the Waluigi effect after Luigi and evil Waluigi. The idea is like, if it can, if it has learned to represent something, it can also very easily represent its negative.

• 41:39Robert Wright?

  On that hopeful night. Anyway, look, I, as I said, I'm a fan of your podcast and Thank you. People checking out, Pragm Revolution, and I'm continuing to explore all this stuff. I'm writing a book about from a kind of a cosmic perspective. And so maybe down the road, we'll have a chance to check in and I will learn even more from you. Meanwhile, people should check out waymark.com. It will produce a video ad with less trouble than they might imagine. And definitely check on your podcast. So thanks so much for taking the time. My pleasure, Rob. This has been a lot of fun.

• 42:14Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.