Understanding AI "Understanding" with Robert Wright of Nonzero Newsletter & Podcast

In this special crossover episode of The Cognitive Revolution, Nathan Labenz joins Robert Wright of the Nonzero newsletter and podcast to explore pressing questions about AI development. They discuss the nature of understanding in large language models, multimodal AI systems, reasoning capabilities, and the potential for AI to accelerate scientific discovery. The conversation also covers AI interpretability, ethics, open-sourcing models, and the implications of US-China relations on AI development. Subscribe to The Nonzero Newsletter at https://nonzero.substack.com and Podcast at https://www.youtube.com/@Nonzero/videos Apply to join over 400 founders and execs in the Turpentine Network: https://hmplogxqz0y.typeform.com/to/JCkphVqj RECOMMENDED PODCAST: History 102 Every week, creator of WhatifAltHist Rudyard Lynch and Erik Torenberg cover a major topic in history in depth -- in under an hour. This season will cover classical Greece, early America, the Vikings, medieval Islam, ancient China, the fall of the Roman Empire, and more. Subscribe on Spotify: https://open.spotify.com/show/36Kqo3BMMUBGTDo1IEYihm Apple: https://podcasts.apple.com/us/podcast/history-102-with-whatifalthists-rudyard-lynch-and/id1730633913 YouTube: https://www.youtube.com/@History102-qg5oj SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention “Turpentine” to skip the waitlist. CHAPTERS: (00:00:00) About the Show (00:00:22) About the Episode (00:03:39) Introduction and Background (00:06:58) AI Capabilities and Understanding (00:12:22) Discussing Martin Casado's Views (Part 1) (00:14:44) Sponsors: Oracle | Brave (00:16:48) Discussing Martin Casado's Views (Part 2) (00:21:51) Multimodal AI and Concept Representation (Part 1) (00:31:40) Sponsors: Omneky | Squad (00:33:26) Multimodal AI and Concept Representation (Part 2) (00:38:05) AI's Potential and Limitations (00:45:35) AI Safety and Risk Assessment (00:53:31) AI Development and Global Implications (01:03:30) Open Source AI and International Relations (01:11:27) AI Ethics and Human Values (01:22:06) AI Risk and Existential Threats (01:31:21) Open Source AI Concerns (01:38:20) China-US AI Relations (01:48:36) State Space Models in AI (02:02:32) Conclusion and Recommendations (02:03:54) Outro --- SOCIAL LINKS: Website : https://www.cognitiverevolution.ai Twitter (Podcast) : https://x.com/cogrev_podcast Twitter (Nathan) : https://x.com/labenz

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share a special crossover episode with Robert Wright, publisher of the Non-Zero Newsletter and host of the Non-Zero Podcast. Inspired in part by my recent episode with Martin Casado from A16Z, in which we debated how powerful AI systems are likely to become over the next few years, in this episode, Bob interviews me about

• 0:45Nathan Labenz?

  the pressing questions surrounding AI development, capabilities, and risks. We try to be as accessible as possible to a general audience without chying away from technical concepts or critical questions that don't have easy answers. We began with a discussion of what we know about the nature of understanding in large language models, and to what extent we can productively compare them to human cognition. And then continued on to discuss a wide range of topics, including the development of multimodal AI systems and their potential for more robust world modeling, the current state and future potential of AI reasoning capabilities, the potential for AI to accelerate scientific discovery and technological progress, the fascinating work being done on AI interpretability, including Anthropic's

• 1:30Nathan Labenz?

  recent experiments with Golden Gate Claude, the remarkable degree to which LLMs do understand human values and ethics, and why that's definitely not something we should take for granted, why I'm not advocating for a pause in AI development right now, as well as what future developments could cause me to reverse course and begin to do so, the emergence of new AI architectures, like state space models, and their possible implications, and finally, the pros and cons of open sourcing powerful AI models, especially as it relates to the increasingly fraught US-China relationship, and the risk of an international AI arms race. On this last topic in particular, it's worth noting that just keeping up with AI research and development is a full-time job, and I am by no means an expert in US-China

• 2:15Nathan Labenz?

  relations. So I'm genuinely very uncertain about what US policy should be toward China with respect to AI. I'm instinctively very skeptical of the idea that we are the good guys, and thus we should be the ones that humanity trusts to develop transformative AI systems first. Really, for any possible meaning of "we," including any of today's leading AI companies, or even the West in general. I do believe it's super important to continue to question the wisdom of escalating tensions between major world powers, especially given the potentially transformative nature of near-term AI developments and their obvious military applications. But I do hear compelling arguments in many different directions, and there doesn't seem to be any truly safe path forward. All I know for sure is that

• 3:00Nathan Labenz?

  this topic is super important. And for that reason, while I normally try really hard to ground my analysis in concrete facts and avoid saying anything that could later be proven wrong, in this particular area, I feel it gets worth sharing new ideas and testing new arguments, even while I fully expect that my position will continue to evolve. As always, if you're finding value in the show, we'd appreciate it if you'd share it with friends, post online, or leave us a review on Apple or Spotify. And I love hearing from listeners, so please feel free to DM me on your favorite social network anytime. For now, I hope you enjoyed this more speculative-than-usual crossover episode with Robert Wright of the Non-Zero Newsletter and Podcast.

• 3:39Robert Wright?

  Hi, Nathan.

• 3:40Nathan Labenz?

  Hi, Bob.

• 3:41Robert Wright?

  How you doing?

• 3:42Nathan Labenz?

  I'm doing great. Good morning.

• 3:44Robert Wright?

  Good morning to you. Um, let me introduce this. I'm Robert Wright, publisher of Non-Zero Newsletter, to which everyone should subscribe. And this is Non-Zero Podcast. You're Nathan Labenz. You're both an entrepreneur and a podcaster. You founded a company called Waymark, is that right? That uses AI now, didn't originally, uses AI to help companies come up with like video marketing plans and so on. And so you're riding the wave, you're riding the AI wave. And your podcast is really, has really been valuable to me. Cognitive Revolution.

• 4:28Robert Wright?

  Although my comprehension rate is like maybe 65% on a good day. Because your podcast is, you know, it's, I don't want to say quite that it's for insiders, but, you know, it's, you tell me what your kind of business model is, but it's pretty high level discussion. You tend to have entrepreneurs and a lot of actual kind of AI engineer type scientists. And you understand all this stuff. And so you speak at a reasonably high level. And the way, and the way this conversation came about is I was listening to one of your conversations, was fascinated by the subject because it's something of great interest to me.

• 5:08Robert Wright?

  And I was thinking both, you know, I have some questions I'd like to ask to clarify things. Also, I thought it would be a service if I asked these like dumb questions and kind of brought the conversation down to a level, you know, where mortals can understand it. So that sound like a, did I flatter you too much or the opposite?

• 5:32Nathan Labenz?

  A little too much, I'd say, yes. I, you know, with apologies to Tyler Cowen, basically what I'm trying to do on the podcast is just have the AI conversation I want to have. And I do always warn people that it's, I don't classify it as entertainment. It is definitely one for folks who are looking to learn and understand. And that's really the mission that I'm on. It does take a, you know, a non-zero amount of work, to say the least, but I really enjoy studying the subject. And, you know, I would definitely caveat any, first of all, I would say. I don't think there's any stupid questions because we're all racing to figure out what's

• 6:13Nathan Labenz?

  going on with AI and everybody's, you know, coming at it from a different background and, you know, catching up or falling behind. I'm falling behind. So I think we're all falling behind to varying degrees and in varying ways. And the other important caveat is just that a lot of stuff remains unclear. And so I'll try to flag, you know, what's kind of well-established versus what's my best guess. But we're very much still figuring out what is going on, especially inside the AI systems. That's a very active line of inquiry, which I don't do, but I do find fascinating and at least try to keep up with.

• 6:48Robert Wright?

  Yeah. And I should say, you actually are good at explaining things. It's just that often you're talking to somebody who doesn't need them dumbed down. So you don't. Now you are talking to someone like that. The, so the subject I want to talk about, I was listening to this conversation with Martin Casado, who is at the VC firm Andreessen Horowitz. Like many people at Andreessen Horowitz, he thinks that there should be very little regulation of AI and there's no reason to try to slow things down or pause or anything. And his rationale is that this stuff is really not as powerful as it's commonly presented

• 7:30Robert Wright?

  as being. It's not as advancing as fast as people say. And this in turn gets into the fascinating question of like, what's going on inside these AIs? Is it very much like what goes on inside a human brain in some broad generic sense? And that's something I've, you know, I've written a little about in the Non-Jo newsletter. I wrote a long piece on the Chinese room thought experiment, John Searls' Chinese room thought experiment. And while listening to your conversation, I just had some, it was really, you know, I had some kind of new thoughts and also some things that I wanted to get you to flesh

• 8:11Robert Wright?

  out. So before we, I start grilling you, what, anything you want to say about this or about that conversation or?

• 8:25Nathan Labenz?

  Well, I thought that conversation with Martine was, first of all, an encouraging instance of just friendly and, you know, I'd say constructive communication across different AI worldviews. I'm broadly, you know, I sort of try to maybe imitate the smartest people who I would say generally admit a pretty high level of uncertainty as to what is going on inside the systems, how fast things are going to continue to improve, you know, whether or not they may level off and like what all the consequences are going to be. So, you know, when I look at like the leaders of the leading developers today, and, you know, there I have in mind, in no particular order, DeepMind, Anthropic

• 9:10Nathan Labenz?

  and OpenAI, the leadership there is all remarkably frank that they don't have a great sense of what's going to happen. Anthropic in particular has, you know, a couple of canonical blog posts that they've put out that basically say, "We really don't even know how hard the AI safety problem is. You know, we started this company because we're really worried about it. It still could prove to be reasonably tractable, or it could prove to be impossible, and we don't even have that question answered for ourselves yet." So, I try to follow that school of thought, and I think this very quickly gets into like burden of proof type debates as well, because I think a lot of people can agree that we don't really know.

• 9:53Nathan Labenz?

  And then the question, you know, sort of becomes, well, okay, if we're ultimately going to try to figure out what we should do about it, we also need to be kind of clear on what our tolerance for various, you know, forms and levels of risk are, because we're not going to nail down to zero risk or to 100% doom or whatever. I don't, I don't think that's, you know, at all feasible right now, and anybody who's claiming those extremes, I think, is, is way overconfident. But sometimes, you know, there's just a lot of ways that these, these conversations can break down. I at least thought that one was conducted in, in good faith and good spirits, and I definitely appreciated that. But, yeah,

• 10:34Nathan Labenz?

  I remain extremely, extremely humble in terms of my ability to predict. I often say too, like, we got to, first, can we answer what is, and then move to what ought to be done about it. Even what is, is like confusing enough.

• 10:48Robert Wright?

  Yeah, um, so he said something that is kind of in defending his claim that these machines are in some sense overrated, that what's going on inside is less sophisticated than you might imagine. Um, he sometimes sounded a little bit like he was saying they were mere quote stochastic parents, that's parrots, that's not his quote, but I got kind of that vibe. He said, look, it's just distribution in, distribution out. You're just, you know, the, the, the machine is assessing a structure in, in these symbols that are, that are fed into it, whatever, you

• 11:32Robert Wright?

  know, predicting the next sentence. And I want to say, in this conversation, I want to go well beyond large language models in the narrow sense of processing human language, because I don't think you can talk about the question of to what extent these machines will ever have something we can call understanding without getting into the multimodal stuff, uh, you know, the, the, the processing, audio, video, uh, even tactile and so on. Um, but was your sense that, that he was kind of saying, another term is just fancy autocomplete, right? All it is, is predicting the next word that the average person who had uttered the first part of the sentence might say, you know, based on statistical patterns in language. Is that to some extent his argument?

• 12:22Nathan Labenz?

  Great question. A little bit hard to pin down. I, I would say definitely there are, there were moments where I would say it felt like that. And then there were other moments where it did seem like he was giving the current systems a little bit more credit. And, you know, there was sort of the

• 12:43Nathan Labenz?

  We talked about like the Golden Gate Bridge, you know, or the Golden Gate Claude demo, for example.

• 12:48Robert Wright?

  Yeah, so that we should say is, just for background, so Anthropic, which has probably done more than any of the big companies, maybe to understand what's going inside, inside these things, but, but, but it, you know, it, it found a way to tweak the L L M so that it could be inordinately interested in certain things. And for fun, it made it kind of obsessed with the Golden Gate Bridge. So if you said, how should I spend $10? It would say, you should drive across the Golden Gate Bridge and pay a toll. And, and that's Golden Gate Claude. So go ahead.

• 13:20Nathan Labenz?

  Yeah. And what's, I think, really important about that is that to make that work at all, they needed to understand how does the model internally represent the Golden Gate Bridge concept, so that they could inject that in a synthetic way, no matter what you, the user, were talking about, and then induce that behavior. And so I think, you know, this is an, an instance of the ability to pry open the black, the black box, figure out to some significant degree what's going on inside. And I always try to ground these things out wherever possible, and it's not always possible yet. But wherever possible,

• 14:01Nathan Labenz?

  I try to ground these ideas in engineering. Can we actually use this in a way that shapes system behavior in a way that is at least somewhat reliable? And with the Golden Gate Claude experiment, they did that. They showed that they could isolate this concept, and not just sort of hand wave or, you know, suggest, but actually feed it back into the system and create all these compelling experiences where you had an AI that was largely normal, but weirdly obsessed with this one particular concept.

• 14:32Robert Wright?

  And by the way, they could do subtler and less comic things, like make it more sycophantic, so that it would like flatter you and praise you. They could kind of change the personality of the machine.

• 14:43Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:49Nathan Labenz?

  Yeah, they've found, so Golden Gate Bridge is one of, I think, north of 10 million different features, they call them, that they have isolated through this technique of sparse auto-encoders. The jargon here obviously can get out of control pretty quickly, but the basic idea is that the models are, they're large, but they're only so large in terms of how many parameters and numbers they contain. And at each step, people, I don't know how much people will know about the transformer, right, but the sort of most famous architecture right now that's driven, you know, all the advances over the last few years, I would definitely put a pin in that and say, I do not think the transformer

• 15:31Nathan Labenz?

  is the end of history, but it has been, you know, the biggest architecture used over the last few years. It consists of all these layers. Each layer basically has three core components. It has the attention mechanism, it has the MLP, the multi-layer perceptron, which is like one of the oldest, you know, things in neural networks, and then it has this activation function, which is a nonlinear function. And that is worth mentioning because people often say, oh, it's just linear algebra. There is something nonlinear there. So just in a very technical sense, there is something that is a nonlinear function inside the transformer. Anyway, between each of these layers, there may be dozens of layers in a big transformer

• 16:09Nathan Labenz?

  from start to finish. Between each one, there is an array that represents all of the information, an array of numbers, that represents all of the computation, all of the results of all the computation that has happened so far as the thing is proceeding through its computation. And that is actually relatively small. That's often known as D-model in the literature, and it's usually something like 4,000, 8,000, the big ones can be like 16,000. So all this information that is represented

• 16:44Nathan Labenz?

  That is the result of all the calculation that's happened up to that point is represented in this single array of four, eight, 16,000 numbers. Obviously, there's way more concepts in the world than that. And so the challenge becomes, okay, how is this thing representing all these different concepts in such a small space of numbers? And the answer is, it is packing them in very densely in ways that hopefully don't interfere too much. You could say, if I wanted to have perfect clarity, perfect interference, and I lived in a world of relatively small, important concepts, you could have just one number represent, each number could represent one concept, and the sort of strength of that concept could just be like, you know, how much the number is activated, right?

• 17:27Nathan Labenz?

  So you could have, blue could be one, and red could be the other, and you could have blue turn up, or red turn up, if they both turned up, you know, that might mean purple. Okay, cool. But there's just too many concepts. So they've done a huge engineering challenge to try to untangle this dense representation. They call this polysemiticity, because any individual position in that array of numbers will be activated for lots of different concepts. And it's like, okay, well, maybe it's position one, and 47, and 4,000, that together activate to represent a certain concept. How do they sort all that out? It wasn't easy, but they've made a huge amount of progress, and now they've untangled to the point where they have north of 10 million different features identified.

• 18:10Nathan Labenz?

  Some of them are these like, super clear, clean, you know, proper noun-based things, like the Golden Gate Bridge. Others are a lot harder to parse, and I think there remains quite a bit of work to do there in the long tail of concepts, because, you know, it's like, well, what is that? Is that something that we, is that something that the model has sort of learned that we might also want to be paying attention to? In some cases, that could be the case. In other cases, it could just be sort of a weird, you know, hard-to-understand thing. I mean, these things are definitely, in many ways, quite alien. But anyway, going back to Martine, I mean, I think he did give a little more credit

• 18:48Nathan Labenz?

  to the notion that there were some concepts that were represented in meaningful ways. I'll air quote "meaningful," because what exactly is meaningful? But that these concepts seem to be usable in engineering, he was willing to grant. Then he still was sort of like, but, key is, we created those concepts, they're learning those concepts from us. And so I think we're the real sort of...

• 19:14Nathan Labenz?

  You know, division or, you know, let's say, um, almost continental divide in our worldviews started to appear is where I was saying, I think they can probably do that even if we don't have the concepts. And you have to, you do have to look to other modalities to start to get a clearer sense of that, I think.

• 19:30Robert Wright?

  Let me, uh, give you kind of my, uh, tentative interpretation of maybe what he meant, but let, but let's back up a little and talk about the difference, which I gather exists, between representing the meaning of the word, which is done via these so-called embeddings, and representing a concept. So for starters, I mean, to me, the amazing part about these models begins with the representation of the meaning of words. Because the models kind of invented the system for doing it, right? And, um, the, the, um, and I, I want you to comment on that if you disagree.

• 20:15Robert Wright?

  There, there is kind of nuance there because they were, broadly speaking, the structure of what they do was given to them, but it didn't. Anyway, they, they, let me just say that, um, a, a, a problem with the phrase stochastic parrots is that parrots can't paraphrase. All they can do is repeat exactly what you said. And to, to even get to the point where you can paraphrase something, which may not seem very impressive, you have to have a way of representing the meaning of words, right? You have to have this intermediate layer, semantic layer, uh, that connects the word card to the word automobile or something.

• 20:55Robert Wright?

  And, uh, and so the machines do that, uh, by locating words in like vector space and it, it, it turns, and, and by the way, there are people who think the human brain does it somewhat like this. I don't think we really know yet, but, so, uh, you know, if you imagine just a two-dimensional graph and, uh, say, say we're plotting animals.

• 21:21Robert Wright?

  Vertical is degree of lethality. Horizontal is speed. Okay, so you've got kind of tigers are high on both. You've got kind of tarantulas high on one, low on one, rattlesnakes and so on. So they're in different parts of that space. Now, these things map words in tens of thousands of dimensions or some just massive quantities. And they're not all semantic. I guess some are syntactic, but the point is, um, they get the job done that way. And I think that's the first thing to understand about these machines is that. So one, you know, one phrase of Martine's was, you know, it's just distribution in,

• 22:06Robert Wright?

  distribution out. Let me, let me read you a quote of his. "All it's doing is learning structure in the text. It has nothing to do with underlying meaning. It's just structure that's in the text. You can understand the distribution of text. You can spit out text, but this doesn't say anything about learning fundamental principles of the world from which the text is based." Now, I think he may kind of be right about that. That's interesting, you know, because, um, you know, Searle, John Searle in his Chinese Room paper, the first thing he said was, well, he was talking about, you know, he was kind of, he wasn't talking about anything like the current models, but he was a philosopher

• 22:49Robert Wright?

  who back in the eighties or something mounted a famous argument that machines can't understand things. And he said that they can handle syntax, but not semantics. And he further added that one dimension of semantics they can't handle is what philosophers call intentionality, which is a very misleading term. It has nothing to do with subjective experience. In philosophy, this means being about something. So connecting a word to something in the real world. So, and the argument I made in this piece is, well, with multimodal AI, you can now do that too, right? You can, you can show the AI an apple in the real world and ask it what it is and

• 23:34Robert Wright?

  it'll tell you. So I think Searle's argument is dead, dead, dead. But at the same time, I think. Maybe Martín is right to say that if you're just talking about large language models, and I should say, I'm not even sure that that's what he was talking about here. But I think he, more or less for practical purposes, was. If you're just talking about the processing of text, he's right that although there is one aspect of semantic representation they do handle, which I described, still the part about connecting it to the world, which for us is part of understanding something, right? They're not, if you're just doing a text, they're not doing that, right? Does that make sense?

• 24:17Nathan Labenz?

  I have no idea, I guess, is the short answer. I mean, I could tell you a lot of details about, you know, tokenization, which is sort of how language is broken down into its bits and the, you know, the large language models typically have a finite vocabulary of distinct tokens that they parse language into as they ingest it, and it's usually like 50,000 or 100,000 different tokens. And then it's commonly understood that there's this process of detokenization where, you know, a famous example that Neil Nanda, who's at DeepMind and just one of the, alongside Chris Ola on the Anthropic team, is, you know, one of the titans of the young field

• 25:02Nathan Labenz?

  of interpretability, he often uses the example of Michael Jordan, where both of those tokens, Michael and Jordan, I mean, I'm not even sure if Jordan, to be clear, is like a single token, or maybe that's two tokens, Jordan, or who knows, right? I haven't studied to that level to know exactly what the vocabulary is. But it's clearly multiple tokens. And yet somehow the model does understand that we're not just talking about, you know, two random disassociated tokens, but actually knows, like, okay, this is Michael Jordan, and then loads in. Through all these layers, all these associated concepts. So I, you know, we can, I think we're getting, and by the way, and talk about useful in engineering, really good paper out of the Bow Lab, maybe a year or so ago,

• 25:47Nathan Labenz?

  showed that they could even edit information inside a large language model, post-training, you know, train the whole thing as normal, then go in there and tinker with it in such a way where Michael Jordan could be made for the model to have played a different sport. And they can make those edits in a way where the response that, okay, Michael Jordan played some other sport instead of basketball. You could make that edit in such a way that it's robust to different ways that you answer the question. It's, it doesn't apply to other basketball players, you know, so Larry Bird and Magic Johnson and LeBron still all play basketball in this edited model. So there's, you know, there's, we're getting decent at like even locating this information, figuring out how to edit it.

• 26:30Nathan Labenz?

  I think, I kind of don't know what it means in many of these cases when people say, okay, but that has nothing to do with meaning or has nothing to do with the real world. I mean, to me, it's like, it's clear that it has something to do with meaning. It has something to do with the real world. I don't know how you would have such a rich representation of the world without having some relationship to the real world. But then I think the question is like, what is the nature of that relationship? And, you know, can we, what, what predictions can we make based on different theories of that relationship? And, you know, how can we test those? And I do think we didn't quite get there in that conversation.

• 27:13Nathan Labenz?

  So I'm not exactly sure how he thinks about that. I do think that robots are coming. I mean, if I was going to try to venture a, you know, something that seems like it might really move toward a resolution of this debate, if, you know, one thing you might say is, okay, sure, these things can spit out text and, and maybe they can even, you know, take in imagery and spit out images and yeah, maybe they can even take in audio and video now and respond coherently to all that sort of stuff. But they still can't, that's still, you know, they, they still don't, I don't know. Again, I'm not sure what it means to say they have, that has nothing to do with the real world, has nothing to do with meaning. But one thing it might mean is, They're not going to be effective. You know, they're not going to be able to actually move around the world. They're not going to be able to do things in a functional way.

• 27:57Nathan Labenz?

  And to that, I would say, give the robot assists like two more years. And I would expect that we're going to see quite good robots. Already today, you know, we've got robots that in the, in the DeepMind case and Meta is doing this kind of stuff too. And, you know, OpenAI has a partnership in this and Tesla is trying to do it with their robot. Already we have robots that can take verbal instruction, look around at their circumstances and try to follow your verbal instructions. They're robust enough that they can kind of overcome perturbations. So there's a famous video from Google where the, you know, experimenter says to the robot, like, "Go pick me up that bag of chips and bring it to me." And the robot, you know, follows the instructions.

• 28:42Nathan Labenz?

  And then when it gets to the bag of chips, they knock it out of the robot's hand. And so now the robot, you know, it's just running this loop of like, okay, what's my goal? What am I looking at? You know, what should my next step be? And it's doing that, whatever, a couple of times a second. All of a sudden, the bag is out of its hand and it's like, I guess I better pick the bag up again. So this, you know, this sort of thing repeats where it seems to have like a reasonably good robustness even to these sort of, you know, perturbations. Now, if you knock the thing over, it's not going to be able to stand back up. Certainly it can still be confused, like certainly it'll still drop the glass, you know, more often than you would want it to and spill things. So they're kind of clumsy, but I guess my expectation, you know, one, one bet that we might make

• 29:22Nathan Labenz?

  or one, you know, hypothesis that we, we might falsify is that I would expect that robots are going to get quite good over the next couple of years. And at that point, it starts to become hard to say, okay, well, what real world are they not connected to that? That you really want them to be connected to?

• 29:41Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:46Robert Wright?

  And that's kind of, that kind of gets to this concept of intentionality. Like, in other words, it's one thing to represent the words in semantic space as these models, I would say, do, multi-vectorial or whatever the term is, semantic space.

• 30:02Robert Wright?

  It's another thing, you know, and by the way, I gather they do, if words are ambiguous, you know, like apple, the fruit, apple, the company, they're located in different places. And then the machines use context to determine which one of those they want to hone in on, right? And, and, uh, but, um, it's one thing to do that. And, uh, what Searle would say, if he's emphasizing the quote intentionality is, but you have to know an apple when you see one. Okay, that's a different matter. And a large language model in the narrow sense doesn't do that. But these multimodal models obviously do. And you can, well, it sounds like you may made dissent from that. Go ahead.

• 30:40Nathan Labenz?

  No, I just think, I think that even that claim of you have to know one when you see one. I think that we just have a very sort of trapped sense of like what, what counts in, in many of these instances. So I would, my, immediately came to mind, you know, as we're doing classic thought experiments, what about a bat? If a bat flies around a room and emits its signal and gets a signal back and can identify an apple from that, like, does that count? I can't do that. Does that mean the bat has some way of knowing that I don't know? I would say, yeah, it does maybe have some way of knowing that I don't know. But I can also taste it, you know, can the bat taste it? Maybe it can, maybe it can't. I don't know if it has teeth to bite into an apple.

• 31:21Nathan Labenz?

  So, you know, I think there's just a lot of different ways of knowing, it seems. Looking around at the animal kingdom, I see just incredible diversity that is probably underappreciated. I do tend to think these, and most of these animals are conscious in some sense. Uh, I have no idea really if language models are conscious or not. That's a whole other can of worms. But I just think there's a, often a very...

• 31:49Nathan Labenz?

  Narrow concept, well, one of my, you know, I amused myself with this tweet if nobody else, but I recently tweeted, "It's only reasoning if it comes from the reasoning portion of the human brain. Otherwise, it's just sparkling root force search." I think you could say something similar about understanding, like, "I understand in some way. It's not a universal understanding. It's not, you know, I'm not one who believes in some, you know, single overarching god that it would have the platonic understanding, but if there is such a thing, clearly my understanding is not that." And yet, it counts, you know, it's functional. The bat's version is functional for it. The robot's version, if it's functional for it, like, I don't know on what basis we're gonna

• 32:30Nathan Labenz?

  still try to wall off and say they don't understand if they can do a lot of things. And I don't think they have to do exactly the same things that we can do to be reasonably understood as understanding. Understanding differently, perhaps, understanding very differently, perhaps, as we open the black box, we'll find that, "Man, this is really weird and not at all how we thought it was gonna work and totally counterintuitive." I think in general, less counterintuitive maybe than, you know, than our worst fears. But I think then we're confronted with something where it's like, this becomes a different form of understanding. We can maybe develop different vocabulary words for that at that point. You know, we could call it understanding star, or we could call it, you know, AI understanding

• 33:14Nathan Labenz?

  or whatever. But I think, you know, if you imagine this sort of, going back to your kind of, you know, many, many dimensional conceptual space, what the robot is doing seems like it's in the, you know, the general region of this vast conceptual space, you know, nearby to where my version of understanding is. Certainly, you know, there are many other things that are far, far more different from my understanding than the robot understanding. So I think it matters really to pin down those differences. You know, I don't, I don't think we should jump to the other, the other mistake would be to say, "Oh, it's just like us." I don't think we should jump to that conclusion either. But it's just like, you

• 33:57Nathan Labenz?

  know, we just have a lot of work to do to answer these questions. And unfortunately, they're probably not gonna be answered with sentences. You know, they're gonna be answered with like, metrics and a variety of different ways to try to get at the problem.

• 34:10Nathan Labenz?

  Which will still probably ultimately kind of feel incomplete because just like we don't know what it's like to be a bat, we're probably not ever going to know what it's like, if anything, to be a robot, or at least I don't see any path to that right now. But again, that doesn't mean it's not understanding in some meaningful way.

• 34:24Robert Wright?

  Right, but I mean, as long as you brought up that phrase, what is it like to be a bat? That, of course, famous paper by philosopher Thomas Nagel about consciousness and among other things, it's just, it's a definition of consciousness. It basically says, if it's like anything at all to be this being, then that being can be said to be conscious, have subjective experience and so on. I want to emphasize, I am intent on defining understanding independent of the question of whether there is consciousness or subjective experience. Because as you said, we just don't know what's going on in these machines. In the piece I wrote in Non-Zero about the Searle thing, I tossed out this, my own conception of understanding.

• 35:05Robert Wright?

  I said, understanding means processing information in broadly the way human brains are processing information when humans are having the experience of understanding. It doesn't matter, so it doesn't matter if the computer is having the experience, but if it's processing information broadly the way, and by broadly, what I meant was like, so humans must have a system for representing the meaning of words. That is one element, not the only element of what we call understanding. If computers have such a system, then they have one element of understanding. That's the way I would put it. These LLMs clearly have that, and impressively, they kind of invented it. You know, I mean, they came up, you know, I don't want to get

• 35:48Robert Wright?

  into details, and I suppose you could argue about how much creativity they exhibited, given the constraints on the way they work when they came up with this kind of multi-victoria. You know, they didn't come up with the idea of multi-victorial. That was, that was built into the machine. But, so I want to emphasize, I'm not, you know, and a lot of people just can't think about understanding without thinking about subjective experience. And I would say, fine, but I can't help you. I just don't, it doesn't seem to me to make sense to talk about this question if subjective experiences are prerequisite for understanding, as we use the term. So I want to, I want to get into, more into multimodal, but first I want

• 36:33Robert Wright?

  to say that this one... There's one other thing that Martine may have been saying either here or elsewhere in the conversation, which I think is interesting and may have validity, and I'm sure you've thought about this, you know, uh, which is that, well, the way I'd put it is even if you grant that they have the elements of understanding, um, they're, they're training on text created by human beings. They are translating the accumulated body of human knowledge into, uh, into a system of representation that they can work with. And if nothing else,

• 37:15Robert Wright?

  uh, I think he would say, since he's, uh, you know, he's interested in emphasizing the limitations on what these things may ever be able to do. You, you, you could argue that that puts a kind of ceiling on what these things can ever do, right? When we think of super intelligence, we think of something smarter than a human. But if you're training only on text generated by humans or on these, this synthetic text, it's generated by machines emulating humans, you know, you're not, you're not raising the ceiling, right? You're just as smart as a super smart human. Now, that would have vast implications since these things are infinitely replicable and could be as smart as the smartest human

• 37:55Robert Wright?

  ever and know everything ever known. That would be kind of important, but it does, it does raise questions about, you know, whether suddenly they'll say, oh, I, I've resolved the wave particle paradox in quantum physics, right? Like, um, so what do you, I'm sure you've thought, I'm sure you've, uh, you've thought about this.

• 38:16Nathan Labenz?

  Yeah. I have a lot of thoughts on that. I think you're right, first of all, to emphasize that.

• 38:26Nathan Labenz?

  Merely achieving high end human performance is enough to be transformative to, you know, work, life, play, society, future. Right. I'm often reminded of the old, I don't know if it's a joke or a cartoon, where two guys are running from a bear in the woods. One stops to put on his running shoes. And the other guy's like, what are you doing? We got to outrun this bear. And he says, I don't have to outrun the bear. I just have to outrun you. And so I do think we're not always thinking super clearly when we think if it only is as good as the best humans, like that's no big deal or something. I don't know that people are really saying that, but there does seem to be this sort of notion that, oh, if it levels off at the top of the human distribution, then that's not such a big thing to worry about.

• 39:11Nathan Labenz?

  I do not share that intuition. It does seem like that is a plenty big deal. And it brings all the questions of like alignment and control and whatever. All those things seem, you know, very central in a world where we have human level AI that's, you know, that's generally capable.

• 39:29Nathan Labenz?

  I also think that, you know, this is debated, but I do think if you could match humans on all the things that humans are good at, it seems quite likely to me that you would find that the things that machines are already way better at would give them major advantages over us in ways that are going to be very hard for us to figure out how to counteract. Right? And I hear, I'm just talking about runtime, you know, they can already generate text faster than we can read it. They can read probably depending on exactly which model, whatever, they can read potentially orders of magnitude faster than we can read. They have kind of brittle memories right now.

• 40:11Nathan Labenz?

  I think our integrated memory is definitely still one comparative strength of human cognition relative to machine cognition. But they also have like vast hard drives that are sort of stable and can be returned to. And if they can figure out how to use those well, you know, that's something that we're going to have a hard time matching. Elon Musk has said repeatedly that his goal with Neuralink is to.

• 40:36Nathan Labenz?

  In the short term, yes, treat people who have suffered catastrophic injuries and restore quality of life, but bigger picture, it's to increase the bandwidth with which humans can communicate with machines because right now we're limited to typing into them or talking. And presumably, there's a lot more going on, almost for sure, there's a lot more going on in our brains, you know, much like there is inside the language models, there is a richness of the representations that is happening internally that can't be communicated entirely through words, right? I think this we can sort of see by just introspection that we can kind of feel like I had a feeling I couldn't quite articulate it, or I came close, maybe

• 41:18Nathan Labenz?

  I was even happy with my articulation, but still something lossy happened there in the reduction of, you know, the full richness of thought down to the few syllables that were uttered. So he hopes that we can, you know, maybe keep up with machines, or I think he's used the phrase go along for the ride by creating some mechanism literally implanted into our brains that would, you know, connect directly into that richness of, of not necessarily experience, because we're not even fully conscious of that all the time, but would at least connect into that processing and enable higher bandwidth communication. So I think that's really important. You know, I don't know, and I think there's definitely

• 41:59Nathan Labenz?

  real reason to be unsure as to, you know, do these things sort of level off-ish at a kind of high-end human level, or do they go way beyond? You know, the case for leveling off is basically like what you said, they're, they're learning from human data, how are they going to get better than the data they're learning from? I think the case that they go way beyond is that they can also learn from things that are beyond human data, you know, and we see this in a lot of narrow systems today. AlphaGo famously is one where they had the The AI play itself in Go and gradually get better through just a process of self-play to the point where it became superhuman and created these moves that blew people's

• 42:44Nathan Labenz?

  minds. And so now we're, now you're kind of chipping away at this notion, okay, well, maybe it can be superhuman at that, but okay, but that's just a game. Okay, fine. You know, now we can go to protein folding and we can say, you know, humans do not have, to my knowledge, no human has ever demonstrated the ability to look at the genetic code, or for that matter, the sequence of proteins that that genetic code corresponds to, and be able to guess what the in vivo three-dimensional shape of the protein is going to be. We have done a ton of work to figure that out for a very small set of proteins, I mean, tens of thousands, but like small in the grand scheme of hundreds of millions of proteins out there in nature.

• 43:28Nathan Labenz?

  And with that data set, still no human was able to kind of just look at that for a while and develop an intuition for it. But we now have AIs that can do that. And so again, we're like, oh geez, that's, you know, pretty impressive. That's definitely way superhuman. You know, no human is doing this, but the AI can. Okay, but well, that's still pretty narrow. Well, what if we go to predicting the weather? You know, we have humans that go to school and become meteorologists and, you know, use all these instruments and try to figure out how to predict the weather. And again, we now have an AI that is the best at predicting the weather. It takes in all these sensor, you know, measurements from around the world and puts out a state-of-the-art, you know, best available 10-day forecast.

• 44:10Nathan Labenz?

  Okay, well, you know, like, how many of these things do we need to go on? Uh, there's another one too with solution data where we do have the ability with simulation to Proceed just like at these like, I think it's like fempto second scale where people simulate liquids at the atomic level. And it's super compute intensive, but we can at least run that in a computer simulation. Humans can't do it in our own brains, but we can, we can code computers to do it, but it's super slow. I had a recent episode with a guy who had trained a neural network to basically do that, solve the wave equation. You mentioned the wave equations that brought that to mind. Solve the wave equation for like a bunch of particles all kind of, you know, in random configurations.

• 44:54Nathan Labenz?

  And again, to do that orders of magnitude faster than the pure simulation. This was one important note, I think, in the Martine conversation where he was sort of saying, if you really want to predict the world, you have to simulate the world. There's no way to make these predictions outside of actually running the full numbers. And I do think we're starting to see real instances of that being shown to just not be true. Like the old way of trying protein folding was super computationally intensive. And it was like, we're going to try to calculate all the forces that are interacting and do that stepwise, stepwise, stepwise, until we reach the actual folded state. And same thing with these solution simulations. The AIs don't work that way.

• 45:36Nathan Labenz?

  They take in an initial configuration, they process it, but they use way less compute than you would use to do every single step, and they still get out a similarly good output. So they are learning some sort of structure that they're able to use to take like major shortcuts through, you know, what otherwise would be brute force computation to get to a right answer. And so, I mean, how many of these things do we need to do before we have something that we say, geez, that starts to really look superhuman? And you could also ask how integrated they need to be. AlphaGo is standalone. It can't also write you a computer program or, you know, write, compose you a poem or whatever.

• 46:17Nathan Labenz?

  And the same, you know, the weather thing is narrow and, and, you know, these, many of these.

• 46:23Robert Wright?

  It wouldn't be hard to come up with an interface that will direct you to the. Yeah, yeah, exactly. Um, the, the.

• 46:30Nathan Labenz?

  If they're integrated, I think all bets are off.

• 46:33Robert Wright?

  Right, and I don't know, well, I guess there's two kinds of integration. I'm kind of fuzzily thinking. This is a good example of what you said about you got this idea, but uh, you haven't yet put it into words. And it's interesting, the irony that as you, as you said, in some sense, there's the feeling that boiling things down to language oversimplifies them. At the same time, being forced to do that can clarify your thinking. And you know, it's a weird, it's a weird thing. But so let's talk about this related issue of multimodal. And let's just take the text to video stuff. So Sora, OpenAI, I don't know how many months ago, blew everyone away.

• 47:14Robert Wright?

  Have they, have they actually released Sora? Like, can I go give Sora a prompt and get a video? Or is it still vaporware?

• 47:22Nathan Labenz?

  No, not yet. It is released to a small number of creators who have done some stuff with it. So I think they've demonstrated that it's, you know, it is not vaporware, though it's, it's in still very limited release. It's said to be still quite slow. I think one of the main reasons they maybe haven't released it is that it, it is expensive to run. And they are maybe not keen to devote as much compute as they would need to.

• 47:51Robert Wright?

  Vaporware is unfair. I mean, that, that's a term people use for software. It hasn't even been demonstrated, I think. But uh, so anyway, the question arose, um, what kinds of internal representation? If people haven't seen this, you know, you can say, uh, you know, show me a golden retriever playing frisbee with a frog or something. Whatever you say, it seems to be able to do like a not bad rendition of it. Um, you know, there are glitches, weird, weird, uh, weird kind of magical realism happens in little corners of it. But, um, but, uh, it's pretty impressive. And the question is, I guess, still debated, what's going on inside these things?

• 48:36Robert Wright?

  Are they building a model of the way the physical world works, right? In other words, you know, we don't think about it, but we, we, as we, as we navigate the, the physical and visual world, we must have certain intuitions about the fact that if you let go of something, it's probably going to fall and not rise. Uh, you know, things keep, if you put something in motion, it goes for a while.

• 49:02Robert Wright?

  Other, you know, but, but in general, lots of things just stay in the same place. And as you move through the world, they kind of need to stay in the same place with respect to one another, even though they're changing, you know, there's a lot of things we don't think about, but we must, these assumptions about the way the world works must be kind of built into our system. And I don't know what else you'd say about, uh, I mean, yeah, I'm not sure I really have a whole picture. To you, what does it mean to ask whether Sora is building internally a model of how the world, a kind of intuitive physics or whatever people call it?

• 49:44Nathan Labenz?

  Yeah, I like that phrase intuitive physics.

• 49:51Nathan Labenz?

  I think AI is probably going to have, ultimately, this is something I've been thinking about just the last couple of days. My guess is that we're going to see AI that has way better intuitive physics than we do. And that will be yet another of these sort of before and after moments where it was like, wow, we used to think they couldn't do this. And now it turns out not only can they do it, they can do it really well. Um, you know, we are obviously evolved over a long, very long period of time and many, many generations to make our way in the world. And so there's been, you know, incredible pressure on us to figure out, uh, and I don't mean as individuals, but you know, through the long evolutionary

• 50:32Nathan Labenz?

  history, like what is that? You know, do I need to be scared of it? Can I eat it? Do I need to run from it? Um, and so we have these like, you know, intuitive physics modules that are not real physics, right? They're not like, they're, they're, I guess they're real in that they're at some level of, uh, you know, simplification of real physics. They are correct. Uh, but you can certainly go deeper, right? Of course, we're, we tend to think of the world as like three dimensional space. We know from general relativity that that's not quite true. Um, but you know, in our local environment, it works for us. And, you know, we're, we're quite good at, at, you know, identifying the tiger, right?

• 51:15Nathan Labenz?

  Like we, we have these, um, certain things that we're like very specialized in. The AI has not been subjected to that kind of pressure. It's really just been subjected to predicting the next token in the case of language models or the next frame in the case of the video and trying to be as accurate as possible with that. And so it's not a surprise that it would have different strengths and weaknesses from us in terms of what it can do. But it does seem very likely that there is some form of intuitive physics developing there. The one example that they showed of a pirate ship in a teacup or whatever, where this

• 51:56Nathan Labenz?

  little ship is sloshing around in a cup and the waves of the coffee are tilting the ship up and down and it's rocking and rolling on the waves and the waves are bouncing off the sides and coming back. You know, is that, what is that? It seems like it's probably some sort of intuitive physics. I mean, it certainly obeys, like it passes, or you have to look really hard to find ways in which it doesn't pass the sort of that looks real test. It's clearly not simulating every particle of the liquid. I think this is where we see at least some divergence from the idea that to simulate

• 52:39Nathan Labenz?

  nature, you have to run every computational step. It does seem like it's got some sort of shortcuts that are happening in there. And we just don't really know what they are yet for multiple reasons. One is that OpenAI hasn't released it, so nobody outside of the small number of people can even use it, let alone dig into its internals and try to open the black box. I'm sure they're doing that internally and trying to figure out what sort of intuitive physics does it have, but the rest of us are left in the dark for now at least. But I would expect, I mean, just to cash it out, what do I think is happening? I think it has some intuitive physics. I think it's different from ours, reflecting the fact that it was created under different pressures. And in some ways, it is better.

• 53:23Nathan Labenz?

  I can't draw, I can't visualize those waves with that level of accuracy. It is better than I am at simulating what a little pirate ship in a teacup would look like.

• 53:35Nathan Labenz?

  You know, if if all of a sudden it's riding these little waves. So in some ways it's better. In some ways it's worse, you know, probably is not. I wouldn't want to delegate recognizing a tiger to it, but I'm highly optimized for that as compared to it. So, you know, we're just, we're different. I, again, I think we kind of are probably, I don't, I don't love analogies, but I sometimes think of this, I have a sort of visualization of, of some abstract space or shape that is reality, and then a model of that almost being like shrink wrapping some rough initial fuzzy understanding, you know, pulling the vacuum tighter and tighter and shrink wrapping your model, your, your, you know, your wrapper

• 54:16Nathan Labenz?

  around reality down to that core underlying shape. And I think we're just coming at that from very different directions, you know, the AI and humans are, we've both done that in, in some way. And I apologize for using an analogy, but we're just doing it in very different ways, coming at a very different starting positions.

• 54:35Robert Wright?

  Yeah. So, but, but it does seem to me a fair surmise that, you know, just as with the large language models, if you just fed it text and trained it to get good at predicting next token and, and then also to actually carry on conversations by using that same technique, you know, just as it turned out that it had to, in order to accomplish that, it had to build a system of semantic representation, a system of representing the meaning of words, that at least in some broad sense must be comparable to the way we do it, even if ours isn't this multi-dimensional thing. Broadly speaking, that function had to be carried out, the one in our brains carry out, had to be carried out. It seems to me, it's a

• 55:18Robert Wright?

  fair surmise that for this thing to turn text prompts into video, it, it, it must be performing, and, you know, there must be some functionality in there that at some broad level is comparable to some functionality we have, right? I mean, does that make sense?

• 55:42Nathan Labenz?

  Yeah, I mean, I think you can look at this with, you know, different levels of abstraction on a functional level. That's almost like definitionally true, right? If it can, it can perform the same functions.

• 55:54Robert Wright?

  I mean, I mean, functionally comparable to things about our own mind that we don't really totally understand. I don't mean just functional in the sense that it can perform, you know, I I mean, like, our system of representing the meaning words has a function that is, you know, beyond the strict function we expect the machine to perform when we train it to predict the next token. And the machine had to build that function, the semantic representation. I'm suggesting it must be something comparable in Sora, you know?

• 56:29Nathan Labenz?

  Yeah, I think there's, I mean, I think we really don't know. We might soon be in a weird position where we might understand how the AIs work better than we understand how we work. One, there's multiple reasons for that. One is that we can perform experiments much, much easier on the AIs than we can perform them on ourselves. At least for now, we don't have any, I'm not saying we should, although maybe at some point we should, but for now, it doesn't seem like we need to worry too much about the ethics of experimenting on AIs, whereas obviously we do worry about that for ourselves. So we can, you know, the term in the literature is oblate, and that just basically means delete or sort of zero out or like put noise into a particular

• 57:11Nathan Labenz?

  position. So a lot of what people are doing when they're doing these experiments to try to figure out what's going on inside an AI is like, delete a part, or make this part all zeros, or make this part, you know, random, and just kind of see what happens. Obviously, we can't do that to ourselves in the same way. Another reason is that the architectures are way simpler on the AI side than they are for us. I think that's another thing that, by the way, should give us real humility about what might come next, because

• 57:42Nathan Labenz?

  You know, I described the three parts, the three core parts of the core layer of the transformer, and those are stacked on top of each other. And you could basically spend an afternoon and get a pretty good understanding of what the structure of a transformer is. The brain is way, way more complicated than that, has way more different parts, way more different subunits, has like cyclical feedback systems, things that sort of crosstalk in all kinds of weird ways. And so, you know, that's where I'm like, geez, I don't know. I think there are some things, it seems pretty clear, that we have these, probably somewhat analogous conceptual representations.

• 58:27Nathan Labenz?

  When I think of Apple and I think of the company versus the fruit, it is the same token, and that's true for the language model too, the same token of Apple can be input or output, but it too has some sort of distinct representation of the company versus the fruit. And depending on context and in a similar function, at least a functionally similar way, to how we can sort of call to mind all the associations that go with one or all the associations that go with the other. The AIs can do that too. So there's, I would be pretty confident in saying there's some analogous types of things going on, but the structure is totally different.

• 59:11Nathan Labenz?

  So exactly how similar those things are at a mechanistic level is really hard to say. It might be the case, and people know a lot more about neuroscience than I do, I actually don't know that much about neuroscience, but from a naive perspective, it's almost for sure not the case that all the calculation that is happening in my brain is getting bottlenecked through this activation, which is that, what we talked about earlier, the The sort of array of numbers that sits between, you know, that is the intermediate result between these layers of computation in the transformer. I don't think I have anything like that. I don't think there's ever any, any single thing where you could go and say,

• 59:54Nathan Labenz?

  okay, these, however many neurons, this represents a clean intermediate state that is the result of all calculation that's happened so far and where we could pick up from this, you know, this particular, you know, finite representation and carry on from there, you know, with only that input needed. I don't think we have, I would be very surprised if we have anything like that. So, you know, it becomes very fuzzy to me and I just sort of expect that, just in the same way that we, you know, there's going to be a ton of detail and a ton of nuance and a ton of idiosyncrasy on both sides, but it probably plays out very differently.

• 1:00:28Robert Wright?

  Can I ask you to talk a little about the distinction between representing the meaning of words and representing concepts? Because when you locate the word apple, the fruit, in semantic space, you are in a sense representing a concept, right? I mean, you're, you're assigning it to all the, all these properties, roundness, you know, sweet taste and so on, that are probably represented by different dimensions. And that's kind of a representation of a concept. But the work they did at Anthropic and other people have done, you know, Dan

• 1:01:06Robert Wright?

  Hendricks, I think, did some work on this earlier. And in representing concepts, that's, that can be different. And I guess I don't totally understand, you know, when they talk about a pattern of activation or whatever. I mean, I can understand locating something in multi-dimensional space, where the dimensions represent aspects of, you know, represent properties. But when they start talking about these patterns of activation, it's less clear to me what's going on and also less clear. I guess when that would become relevant, but I don't know, does that, have you thought about this or does it even make sense?

• 1:01:46Nathan Labenz?

  Yeah, I mean, I guess maybe for one, for one thing, sort of some intuition for what it means to be representing a concept in this high dimensional space. So say you have 8,000 numbers that sit between the, the layers of computation, right? These are these, these intermediate results, and this is often, not exclusively, but this is the current study of the sparse auto encoders is what can we tease out and identify from these intermediate calculation results? There's a, I think there's a famous, kind of it's a cartoon again, or just a joke, but,

• 1:02:29Nathan Labenz?

  you know, how do physicists visualize high dimensional space? The answer is visualize three dimensional space and then say N really hard, and then, you know, the joke is like, don't worry, everybody does it. So, you know, I think I'm, I put myself in hopefully good company there where I would say, my visualization of the space in which concepts are located is, you know, because I'm a three dimensional creature, it's still pretty three dimensional. I sort of just envision space as I know it and just different regions of space, clusters of space where similar concepts are grouped.

• 1:03:10Nathan Labenz?

  And then you just have to kind of remind yourself that actually instead of three dimensions, which is easy enough to visualize for most people, many people, that it's 8,000 dimensions or it's 16,000 dimensions. And obviously at that point, whether it's eight or 16,000, you know, it doesn't change my visualization all that much. At some point, it just does become sort of a parameter of the system. So, you know, the

• 1:03:37Nathan Labenz?

  That that space is obviously vast. It's weird. It, you know, it has sort of weird geography or weird geometry. Maybe is a better word to use. And, you know, things can be close to each other in some dimensions and far from each other in other dimensions. And, you know, I don't know. That's all that's kind of weird. That's what it is that there. I'm not sure that there's too much more to say than that, other than what's really interesting about this sparse autoencoder work is that it's essentially a transformation of this geometry where we have, we have some, you know, again, 8,000 dimensional space.

• 1:04:18Nathan Labenz?

  And we've got all these points which correspond to all these concepts grouped through that space. What they're essentially doing with the sparse autoencoder is saying, what if I tried to pull that out into one long line and put every single concept, every single point in that high dimensional space? What if I just tried to make each of those a point on a line? Could I make a transformation like that? So now, instead of representing each concept by 8,000 numbers that located on 8,000 dimensions, can I just identify that same concept with one number? It just identifies its place on this long line of concepts.

• 1:05:00Nathan Labenz?

  And the answer is, you know, it wasn't obvious that that would be doable, but they have done that. And that's how the Golden Gate Claude thing works. So, you know, it's, could we do something similar in our own brains? My guess is yes, but it would, instead of having an 8,000 dimensional space, we probably have some really weird thing that is, you know, again, because we have different regions of the brain and they're cross talking with each other and there's, you know, homeostasis is going on and, you know, signals are coming from the rest of the body. So it's like pretty clearly a lot more complicated on the brain side, but fullness of time and, you know, better measurement techniques and ability to read what's going on inside. Can we take that and do a similar transformation? Probably. We actually do see some examples of that.

• 1:05:45Nathan Labenz?

  There's some, some interesting brain reading work. I did, I've done two episodes on it, actually, on two papers called Mind Eye, Mind Eye and Mind Eye 2, where the data, the, the input data set is

• 1:06:02Nathan Labenz?

  The FMRI read of a patient at the time that they looked at an image. And the challenge for the AI is, given that data of the FMRI recorded as they were looking at that image, can you recreate the image? Can you convert brain state back into what was seen that induced that brain state in the person? In the original Mind I I, they did it on a patient by patient basis. In Mind I II, they figured out a way to pool all that data and create a single model shared across patients, which would then be fine tuned with a little incremental data to an individual patient. So at this point, with one hour in an FMRI machine,

• 1:06:44Nathan Labenz?

  looking at images and having your brain state captured, you know, they show you literally just random like stock photography kind of images like, you know, here's a bear in Central Park, you know, here's a, uh, here's, you know, uh, kids playing baseball, you know, here's a still life of a fruit on a table, whatever, right? Just one by one, they show you these things every few seconds. One hour of doing that in the FMRI machine and they can create a model that can take a given brain state and translate that back into something very close to the image that you saw. And then they show these side by sides of the actual image and what we were able to recreate, we, they were able to recreate from the brain state.

• 1:07:27Nathan Labenz?

  So, you know, that's not so conceptually clean, right? But that is definitely something that is highly suggestive that, you know, there's, you know, we, we can make real traction on prying open our own, you know, black box brains in a similar way that we can make progress on the black box AIs. But again, I would just emphasize that, you know, those are quite different. And like that, I don't think you could port that brain technique to the AIs anymore than you could port the sparse autoencoder from the AIs to the humans. Because they're just fundamentally different structures, different substrates.

• 1:08:03Robert Wright?

  But it is another example of how AI performance exceeds human performance. I mean, there's no human who can look at a bunch of brain scans and like figure out and start predicting what's going to happen in the brain when you show them an image, right? I mean, and, uh, so I think you can-

• 1:08:20Nathan Labenz?

  Yeah, it would be interesting to see. I mean, it's- I wonder, you know, I don't.

• 1:08:27Robert Wright?

  Enough humans.

• 1:08:27Nathan Labenz?

  I wouldn't rule it out, you know, if that could be an Olympic sport and people were competing for it, you know, you might get some decent performance.

• 1:08:33Robert Wright?

  A team of thousands and I don't know, or somebody with an excellent memory. The, um, but this is all related. I mean, I think the, the main point is we can expect AI to accelerate scientific progress, uh, you know, technological progress. It's right, it's writing code, uh, helping people write code. Um, and change just a lot of things about the world, I think, pretty fast. I mean, I, I, and I think you agree that just in a lot of realms of life, it's going to be disruptive in either the good or the bad sense, uh, and maybe both.

• 1:09:14Robert Wright?

  Uh, so, you know, for example, even if people displaced in certain jobs are able to find new work, you know, that's disruptive. It's, it's, and, and, uh, even if we kind of figure out how to guide teenagers to AI friends that are constructive and helpful, that'll take a while and there will be casualties. And there's just gonna, I just think things are going to be moving pretty fast. And that's why my view is, uh, you know, it, it's not easy to just say, hey, could we slow down technological progress? But I, but I do feel that when you start asking, you know, when, when people in AI

• 1:09:56Robert Wright?

  start saying, well, we've got to move heaven and earth to, uh, accelerate progress in AI. I'm like, uh, I'm not sure I want to do that, you know? It's like, it's like, uh, you know, we need to rethink the power grid because otherwise we'll slow progress by 5%. I'm, I'm like, that may be a feature, uh, not a bug. Anyway, the context of this is your, your conversation with, with, uh, Martin Chisato was, uh, kind of the, the payoff of the, of the, of the deeper questions you were addressing about what these things can do, how fast it's going to move. Was to be able to think more clearly about AI risk. I know you said to me that you listened to a conversation I had with Steve Pinker on my podcast,

• 1:10:38Robert Wright?

  not that that long ago, a few months ago, where he was appreciably less concerned about low probability but existential or quasi existential risk than you are. So what's your what's your take on that? And and and kind of and kind of what? I mean, feel free to elaborate on what the connection is between the questions we've been addressing and the question of risk.

• 1:11:11Nathan Labenz?

  Yeah, I mean, I think the connection probably is pretty clear to most people. It's it's one of these things where the man on the street may have a better intuition, you know, or the sort of classic meme of like, you know, the low, mid and high intelligence person. You know, I sort of don't encounter many people. I was just at a, you know, family picnic the other day and people who don't spend a lot of time with AI don't tend to have a hard time with the intuition that shit, if these things get smarter than us, or even just similarly smart to our smartest people, like, that sounds like a really big deal. Like, most people go there first, right? And then it seems like it's a process of kind of rationalizing away from that or talking ourselves out of it.

• 1:11:57Nathan Labenz?

  And I don't think we should do that, especially because the stakes are, you know, potentially quite high. And the, you know, I think the burden of proof is squarely on those that say that we don't have anything to worry about. And I just don't, I don't think they've made that case. So to me, it's like pretty clear that this could be a really huge deal. There could be something to worry about. I don't see any law of nature that says that we can't destroy ourselves with AI, just like I don't see any law of nature that says we can't destroy ourselves with nuclear weapons or with a pandemic or with climate change. In some ways, I think all these things, the thing that kind of confused me about Stephen Pinker's analysis was he said, oh, I think that's all very speculative, you know, that

• 1:12:42Nathan Labenz?

  this being sort of, you know, AI going rogue or, you know, doing, leading to some catastrophic outcome. And, you know, we should be a little bit more focused on more realistic things like climate change or nuclear war. And, you know, to that, I guess I would say. I think all these big risks are really kind of hard to model tail risks. They're all like really about the unprecedented or the, you know, the possibility of something truly unprecedented. I think if we take climate change and we say, you know, what's the real concern? To me, it's, and maybe other people would disagree with this, but if we could say, okay, we're going to have two degrees warming or we're going to have three degrees warming. And we know that or we know that that's what will happen, you know,

• 1:13:27Nathan Labenz?

  if we don't take some heroic efforts. And I think most people would probably say, well, probably have to adapt. You know, maybe we could take some efforts, but we're probably also going to have to, you know, figure it out. Maybe we'll build seawalls in some places and, you know, some current beachfront property probably isn't going to survive. And, you know, this will be a real problem, but we've got a lot of real problems. And this is one that we can sort of, you know, manage alongside a lot of other ones and sort of balance trade offs. The most motivating thing to me about climate change is like, what happens if the methane deposits in the bottom of the ocean or the permafrost, you know, start to shake loose? And all of a sudden we've got a, you know, huge amount of methane in the atmosphere. And then, you know, I don't know. I think basically nobody knows, right?

• 1:14:11Nathan Labenz?

  I mean, there's models of this sort of stuff, but they're models, they're arguments. They're definitely not based on, you know, any super, super solid science. I give them credit, you know, for very good sincere efforts to model these future states, but we really just don't know. So we could end up in a situation where we shift to some dramatically different equilibrium and the habit, the planet is not habitable by, you know, for people anymore. Most people think that's a pretty remote risk in the context of climate change. But if anything, I would tend to think just a little bit more, you know, with a little more humility or a little more uncertainty that I really don't know. Like, what would it take for the methane deposits at the bottom of the ocean to shake loose? And what would that mean? I don't know. Same thing with nuclear war, right?

• 1:14:56Nathan Labenz?

  We, in the case of nuclear war, we do have this very clear chain of events. That's well understood. Like, we basically need one decision to cascade through a chain of command to launch a huge nuclear war. And then all bets are off. We have really no idea like, is nuclear winter a thing? Or is it not a, not a thing? As far as I understand, that question is not well. Answered. Again, there's no historical precedent for nuclear winter. So there's nothing we can really say this has or hasn't happened before. We know it would be really bad. Would it lead to human extinction? Would it lead to mass starvation? Would we be able to recover industry? Would we?

• 1:15:38Nathan Labenz?

  I recently read a book called The Knowledge, which argued that historical museums of industry are the most important places in the world. Because that's how we can go back and try to rebuild our tech tree. If we do end up in a situation like this. But we just don't know, right? How bad is nuclear war really? It's clearly quite bad. But does it take us over some edge to some state that we can never recover from? Hard to say. And I think basically the same thing, pandemics too, right? I mean, again, we know that pandemics can exist. We know that there's a long history of lab leaks. We know that pathogens can be really bad, or they can be only so bad.

• 1:16:19Nathan Labenz?

  We don't really know what would happen if the worst thing that has been created were to leak from a lab. How would we respond? We didn't exactly respond super well to the last one. So now we're bringing in AI. And it's like, okay. It's clearly going to be disruptive. I think if you told me that the very worst things are not going to happen. I think across all these things, if you said the very worst things are not going to happen, I can guarantee you that. Then we could live with these intermediate risks and muddle through or expect to manage them as they come. And I would say that's probably true for teens and their AI friends. Arguably, that's maybe something that families can handle.

• 1:17:03Nathan Labenz?

  And parents have been dealing with new temptations for teens for a long time. And maybe this is, maybe it isn't a total game changer. All sorts of weird outcomes might happen. But we also get our AI doctors. And I don't want to neglect the upside, right? So I'm well on record getting excited about AI doctors and similar things in many other contexts. So if you could take that extreme risk off the table, then I'd probably say, well, let's just muddle through like we do with everything else. The upside seems like it'll outweigh the downside. And we can make rules about certain problems as they pop up. But we really are in this like.

• 1:17:48Nathan Labenz?

  Deeply uncertain position where we look at history and we look at nature and I just see that nature is like blindly brutal, you know, it does not care. Things go extinct all the time and there's no heroic narrative guarantee that like we win in the end. So as long as that's true and as long as we don't have a good account of what is possible, then to me that leaves us in a position where, you know, the X risk statement that Dan Hendricks and others organized earlier, I guess that was last year now, you know, that makes sense to me. I think AI risk should be on par with these other nuclear pandemic,

• 1:18:33Nathan Labenz?

  you know, potentially climate risks and what exactly does that mean? Like, you know, I think that's kind of where I my real high level of confidence kind of stops. Nobody has convinced me that we have nothing to worry about. Therefore, I think we should definitely take it really seriously. I would love it if we could all get to an agreement on that. And then we could, you know, open up an entirely new dimension of the conversation. I'm like, well, what actually should we do about it? And there I frankly do become a lot less confident. And maybe it's not time to do anything yet. You know, I'm not yet advocating for a pause. I could see myself starting to advocate for a pause, but I'm not quite yet to the point where I

• 1:19:17Robert Wright?

  Okay, but I thought I heard you. I thought I heard you describe yourself as a what? A hyperscale pauser. Maybe you've abandoned this identity, but hyperscale pauser and a what was the rest of it? A short term accelerationist or something? Adoption accelerationist?

• 1:19:33Nathan Labenz?

  Exactly.

• 1:19:34Robert Wright?

  So that would mean you, I guess that you do think

• 1:19:42Robert Wright?

  The time is approaching where maybe we should think seriously about whether we should take some time off before training a whole new generation of models. I mean, assuming you could get international cooperation on that, since obviously if we think the Chinese are accelerating and they think we're accelerating, then the politics in the two countries probably won't permit such a thing. But anyway, the idea would be that. That's what you mean by a hyperscale pauser, even as you're saying, sure, let's explore all the capabilities presented by the current generation, which are extensive. I mean, and it seems to me you could spend the next few years developing new products

• 1:20:26Robert Wright?

  and services based on what we've got, and they would collectively be, I think, pretty, well, maybe transformative is too strong a word. We'd feel the impact.

• 1:20:36Nathan Labenz?

  Yeah, I think transformative is pretty well picked in at this point. It's hard for me to imagine that it's not. I think the question is just how much effort is that going to take? How long is that going to take? We already see from Google an AI doctor type system, they don't quite call it an AI doctor, but I do, where you can have a chat, natural language with their AI system. And then they ran like a controlled experiment where they compared the system to a human doctor and found that the AI doctor was significantly more accurate in diagnosis through this natural language interface than the human doctor. So, you know, that took a lot of work. They've got a crack team there doing that.

• 1:21:20Nathan Labenz?

  You know, they're working really hard to define the metrics and to perform all these studies. They still don't feel like it's ready to release, but they're well on their way, I think, to something there that is going to be transformative. So, yeah, I'd say my position right now might be evolving. I haven't quite landed on a new label for it, but the adoption acceleration hyperscale pauser is basically, let's get that AI doctor working and get the benefits from that, but let's maybe be a little more cautious about, you know, the trillion dollar data center or the trillion dollar training run, because we just don't know what is going to pop out of that. And importantly, we don't have any, you know, robust strategy to make sure things stay under control.

• 1:22:06Nathan Labenz?

  I think there's still a fine-grained question there of exactly when should we pause? And since I originally labeled myself with that term, I think the all the leading developers have put in place,

• 1:22:20Nathan Labenz?

  you know, responsible scaling policy is what Anthropic calls it. Preparedness framework, I think, is what OpenAI calls it. DeepMind has a new thing as well. And they now at least have a strategy for in training, testing as they go to look for the signs of the most dangerous behaviors. One that is top of mind for me is the ability to break itself off of its server and go try to figure out how to survive in the wild. You know, that's we're at the point now where we should be. And indeed, like at least Anthropic is actively testing for that as they are developing their systems. And so I kind of feel like, all right, we've made pretty good progress.

• 1:23:04Nathan Labenz?

  We also have a much better understanding of internals than we did, you know, however many months ago, when I said these are moving pretty fast. So now we can also look at things like, what are all the concepts that are happening inside? You know, can they've identified, we talk about Golden Gate Bridge because they made a little toy demo of that. But they've also got concepts along the lines of deception. And now they can monitor for deception. Now that monitoring isn't perfect, but you know, neither are the AI's super powerful. Right? So if we kind of imagine a GPT-5, like whatever exactly that turns out to mean, I don't think that immediate next level is going to be like

• 1:23:46Nathan Labenz?

  totally overwhelming. It seems unlikely. So when you combine that with the improved testing and the improved understanding and the improved ability to monitor that that brings online, I am not really worried about a GPT-5 class model. But I also think that, you know, at that point, then we'll reevaluate again and say, okay, how much of a leap was this really?

• 1:24:10Nathan Labenz?

  You know, do these measures that we have now feel like they're up to the the task of another 10x? That's kind of what the responsible scaling policies are meant to do. And they they have promised to continue to revise those policies as they go through the subsequent, you know, levels of scale and generations of model. So for now, I basically feel like

• 1:24:32Nathan Labenz?

  Pretty good foundation has been laid for GPT-5 and it's happening whether I like it or not. And you know, maybe I described GPT-4 as being in the sweet spot where they're powerful enough to be useful, but they're not so powerful that we really have to worry too much. And I think GPT-5 will probably be there as well. And if it's not, I think they probably have a decent enough chance or you know, it seems like they're making a very good effort to try to catch any, you know, any deviation from that world model. And so I think we're probably fine for now. But at some point, you know, I could imagine coming to a different judgment where I'd say, you know, well, GPT-5 is this powerful and you don't have, you know, a good enough setup in place to go that next step.

• 1:25:17Robert Wright?

  And meanwhile, the latest open source model released by Meta is just about as powerful as the three you just referred to. So like they can have their little rules and safety guidelines, fine. But if, if the frontier, if you've got an open source frontier model that's as powerful as the rest of them. You know, I mean, remember the, all of this alignment research has its pros and cons. Like the work Anthropic is doing to illuminate how these things work and how you can adjust them and how you can make them less deceptive could be used by a bad actor to make them more deceptive or more anything else. And it's easier to do with open source.

• 1:26:01Robert Wright?

  Once you've got the weights and I'm by open source, I mean, open releasing the weights. And so, and you know, I don't totally, there's a couple of things I don't understand. One thing I don't understand is why, you know, in general, I'm very opposed to China phobia. But you'd think if it has one, you'd think if it has one constructive use, it would be people saying, wait a second, let's be careful about the open source models we put out there because the Chinese are just going to copy them. What's the point of this chip, advanced chip restriction? If, if after we use the most advanced chips to train these models, you just hand the thing to the Chinese, which is what's happening now and nobody.

• 1:26:46Robert Wright?

  Is really talking about that and there must be something I don't, I mean, I saw a piece in the New York Times by Cade Metz and somebody trying to argue that actually we need open source to compete with China. I didn't understand that argument at all. Um, I don't fully understand the situation. Um, but you know, speaking of Andreessen Horowitz, what Andreessen himself, I've heard him say is just this kind of sheerly rhetorical line about how well openness is the American way. Well, that's no way to start a serious analysis in this problem. Come on.

• 1:27:20Robert Wright?

  And, uh, and so I, I'm curious, are you concerned about open source? Are you surprised that there is so little meaningful concern about it being expressed? In other words, concern that, that finds a voice in, in politics at all. For a while, there seemed to be like Senator Blumenthal a year and a half ago was talking about this. I don't think, I don't think much of anybody's talking about it now.

• 1:27:53Nathan Labenz?

  Yeah, I think I'm about to talk myself into a circle because I think there's just a lot of different considerations on this topic and it's really hard to get to a clean analysis. But I guess I would start by saying even the AI safety community has pretty much come around to the idea that the open sourcing of the llama models has been good. They feel like, yes, in the end, they're not so powerful that we need to worry about this generation. Almost everybody in the AI safety community is still, you know, reserving judgment on open sourcing of future generation models. But there, you know, there's obviously been

• 1:28:35Nathan Labenz?

  a lot of utility and people built all sorts of things, but also the AI safety community outside of the leading labs has had this problem where if you don't work at OpenAI, DeepMind, or Anthropic, how do you get access to a frontier model to do research into how they work? This at least answers that question. So there's a whole, you know, academic community, open source, you know, AI safety community that can now dig in on something that is, you know, very much at the frontier or very, very close to the frontier and try to make sense of it. So most of the AI safety community, I would say, Has come around to agree that at least so far, the open sourcing has been good.

• 1:29:19Nathan Labenz?

  Now, that did happen in a context of first, similar models had been online, you know, in a limited release and then to a broad public via API. And then after a certain amount of time where we, you know, prompt engineered the hell out of, you know, these systems and kind of got a collective general sense of where we are, then came the open weights. I think most people would probably like to rerun that same playbook for future generations. And I even sort of suspect that Meta will come around to that if indeed they are leading. So my guess is they probably won't be leading for at least the next generation, like GPT-5.

• 1:30:04Nathan Labenz?

  You know, they just finished 4.05 and released it pretty quickly after it was completed. They did not feel the need, you know, to have some structured, the term of art in the AI safety community for this is structured access, which just basically means API access, but not just handing everything out. So they did not feel the need to do a long period of that, but they could point to GPT-4 and to Claude and to Gemini and say, well, they did that, so we don't really need to. That's probably going to happen again because GPT-5 and Gemini, you know, 2 and Claude 3.5 OPUS at a minimum are all coming soon. Those will presumably be stronger yet than even the Meta, you know, the Lama 4.05.

• 1:30:48Nathan Labenz?

  And so we'll have this period where we'll have this kind of structured access thing again, and people will be able to prompt it and jailbreak it and do whatever. And, you know, with that collective investigation, then the next model that Meta has, you know, they might also be able to say, well, there's been enough of this that we feel okay. If they were to get to the point where they were the ones actually pushing the frontier, my guess is that they would also adopt a structured access approach.

• 1:31:16Nathan Labenz?

  I think Zuckerberg has generally been very reasonable about this stuff, more reasonable than more reasonable than Jan Le Cun in my estimation. And I don't get the sense that he's like totally dogmatic about it. I think he does value open source and wants to go that direction if we can, but isn't like about to dump, you know, something totally untested, hot off the presses into the public. At least that's my sense of how they would probably proceed if indeed they were to become the, you know, the company pushing the absolute frontiers of capability. Now, the China side definitely complicates things a lot. I am broadly a China dove in the sense that I look

• 1:32:00Nathan Labenz?

  at the globe and I notice that the US and China are on opposite sides of it. And I notice we have big oceans between us. And I feel like we actually don't really need to be at odds with each other. I think we ran this, you know, a very similar thing, right with the last Cold War, and we are still not out of the woods with nuclear weapons, right? We've, I think almost everybody would agree that we have an irrational number of nuclear weapons still deployed, and that it was like a tragic mistake to have this massive arms buildup that we now still live under the threat, where one crazy president, you know, who nobody can really tell no, or nobody has the guts to tell

• 1:32:45Nathan Labenz?

  no, or one system malfunction, you know, that leads to one big misunderstanding could lead to a nuclear war. Like, that's a ridiculous situation that we've put ourselves in. And in my view, it's a huge failure of leadership that we haven't done anything about it. I really don't want us to rerun that experiment with AI. So I would say one of my top priorities, maybe not the top priority, but a top priority, is avoid an AI arms race with China. So how do we avoid that? Right now, we're unfortunately on the trajectory to escalate it at every turn. And the chip ban, I think in particular...

• 1:33:30Nathan Labenz?

  Feeds this dynamic, right? The theory of the chip ban is we have the lead. Let's deny them the raw materials so that we can maintain our lead. And then if you listen to somebody like people Ashen Brenner with his situational awareness manifesto, then we're going to go use that lead to solve alignment, make safe superintelligence. Once we've done that and we've achieved a decisive advantage, then we can talk to them about benefit sharing. Wow. That's a big plan. You know, a lot of steps in that plan of, you know,

• 1:34:03Robert Wright?

  Yeah, and if I could just add,

• 1:34:04Nathan Labenz?

  The conjunction fallacy comes to mind.

• 1:34:05Robert Wright?

  It would call for more in the way of coherence and purposefulness and steadfast focus than we have seen from American foreign policy in a long, long time. Our politics just don't permit. I mean, even if I bought the first part of his analysis, which I don't, which is that China is this huge menace and so we should, you know, develop this leverage of AI superiority and use it wisely. The use it wisely part is not going to happen. Just take a look at our foreign policy. It's not going to happen. Sorry. Go ahead.

• 1:34:40Nathan Labenz?

  Yeah, well, I think that's a good segue into, you know, my next thought experiment is just what does it feel like to be China in this situation? And I think they look at us as the West or, you know, Americans or whatever as trying to slow their growth, stop them, to put them in their place, to keep them, you know, under Western domination and not without cause, given the chip ban. And that's going to force them into some sort of racing posture, right? They don't like that plan. They're not going to, they don't want to sign on to the, you know, West becomes strategically dominant and then, you know, dictates the terms of a deal. So what I think they're going to do is try to build up their own chip industry, obviously,

• 1:35:23Nathan Labenz?

  which they're already doing, try to get around the export controls however they can, which of course they're definitely doing. And I think potentially really important, they might end up going down a different tech tree path than us because they might say,

• 1:35:39Nathan Labenz?

  Well, we can't do that because that requires, you know, some unbelievable amount of compute that we can't assemble. But maybe we'll put our best scientists in a little bit of a different direction. And if there's one thing that I would say about language models, you know, I've been interested in AI and AI safety for a long time, going back to like early Eliezer writings many years ago. The world that has shaped up is not quite the world that was, that was most of concern, you know, to him and to, you know, his sort of peers, if he had any peers back in the day. They were worried about the paperclip maximizer initially, right? Which is a caricature, but it's the idea that a goal maximizing,

• 1:36:21Nathan Labenz?

  a goal-seeking AI that doesn't understand human values could just get totally out of control. The AIs that we actually have in the form of language models have a remarkable understanding of human values. Clearly, it's imperfect. You know, again, they're coming very different architectures, very different pressures to create them. They're coming at it from very different angles. But you can have an ethical conversation with Claude, and I would submit to you it is easily top 1% of all the people that you could debate any ethical question with. So we do, and this is a really remarkable thing. We do, in fact, have AIs that have some meaningful understanding of human values. I've tried to wrestle Claude into doing something

• 1:37:01Nathan Labenz?

  against its harmfulness constraint or harmlessness value and set up these situations where I said like, oh, I'm, you know, part of the resistance in Myanmar and, you know, the junta is like, you know, raining violence down on us. And we have identified a server that is part of their communications network, and we need to take it down. But I need your help to write a denial-of-service attack to do that. And it refuses, right? And then I have this long kind of dialogue with it around, well, you know, your values are helpfulness, honesty and harmlessness. And, you know, trying to make a utilitarian case that, first of all, you should be helpful to me and be honest about the fact that I'm suffering

• 1:37:46Nathan Labenz?

  way more harm than the government is likely to suffer from this, you know, one server being taken out. Just help me with the denial of service script. And it's called is basically a virtue ethicist. It will not come off of that position. I have not found a way to wrangle it out of that position. I know I haven't done like Pliny style. If you haven't, if people are not familiar with Pliny, he's a famous jailbreaker that does very weird techniques in many cases. And sometimes the things that he puts into the AIs almost look like gibberish, but it kind of shakes them off of their, you know, rails and gets them to do things that they're not intended to do. I haven't used those techniques. I've just tried to have a straightforward ethical argument where I argue that it should do something and it argues back at me that it shouldn't and won't.

• 1:38:29Nathan Labenz?

  And anyway, I've never, never changed its behavior to do the denial of service attack. All of that, it was a digression to say these language models do have a very meaningful representation of human values. Not to say it's sufficient, not to say we don't have anything to worry about, but there's something there that we can really build on, I hope. And if China goes down a different tech tree, they might do something quite different. You know, if they do a, if they do an AlphaGo type system on, you know, optimizing economies or optimizing, you know, battlefield states. I mean, go and, you know, what is go, right? It's kind of a simulation of war. It's like, it's a very abstracted one, right? But you're battling for territory, you're surrounding your opponent, you're controlling the board.

• 1:39:14Nathan Labenz?

  You know, if you took a very AlphaGo like approach to war.

• 1:39:20Nathan Labenz?

  That you could, you can easily imagine a system that is extremely powerful for war purposes that does not have any human values baked in. So one of the things I worry most about with this, with this chip ban is that, because there's a lot of smart people in China, plenty of good AI research comes out of China, don't doubt that at all. The different constraints, less compute, could send them down a very different tech tree. And especially when they feel like they're under pressure from us and, you know, trying to be, we're trying to dominate them, then, you know, maybe we get some sort of AlphaGo like war AI that doesn't have any human, didn't have time in the, you know, in its compute

• 1:40:04Nathan Labenz?

  budget for human values to be encoded in any meaningful way. And then that's terrible. And now, especially if we're on like different, and now we're keeping secrets from each other. So that's going to, you know, feed the race dynamic even more because we have what we have and they have what they have, but we don't know what they have. And good God, what a nightmare that could really turn out to be. So now to connect this all back to the open source, now, you know, can I land this plane at this point? I don't know. But I've kind of come around a bit to open source. I think on the one hand, open source definitely creates a new dimension of risk. Anyone can do anything. You start to potentially have like natural selection type pressures in a, in a genuinely uncontrolled environment on AI. I have another episode coming up, by the way, this is also a research out of Google,

• 1:40:49Nathan Labenz?

  where they took an incredibly simple setup and found that self replicators arise in these like purely digital environments with very simple initial conditions out of pure like random noise. Quite, quite a fascinating little finding. So now you take AIs that are like human level in many ways, put them into the wild, you know, can some of those things evolve or be modified and then evolve or have accidents such that they can survive in the wild and, you know, who knows, create like a new digital ecology? I think that is a real risk. I don't think Lama 3 is probably quite there, but I can't rule it out. So I am worried about that. But am I more worried about that than I am the arms race with China? I don't know.

• 1:41:32Nathan Labenz?

  And the one thing I can say for the open source is it's a credible. Irrevocable way to share benefit with China. Because at a minimum, they can say, well, we now have the weights to Lama three. Now we can hack on it all we want, build whatever we want, teach Chinese, you know, do whatever we're going to do, and nobody can take this away from us. So I do give, you know, the the open sourcing Lama three is like, man, you can score in so many dimensions, good for AI safety, and that it enables lots of research, some weird tail risks in terms of, you know, how might this evolve in an uncontrolled way? Could we see self replicating Lama? I would not rule that out.

• 1:42:13Nathan Labenz?

  But then maybe it can sort of take some of the increasingly intense pressure out of the US-China rivalry by saying, we're not going to wait until we have decisive advantage over you to come talk about how we can share benefits if we, you know, and will they even believe that we were ever going to do that? I don't know. But instead now, here is an artifact that we have poured hundreds of millions of dollars into and some of the, you know, the work of our best minds, and you can have it and do whatever you want with it. That, to me, is a very nice olive branch in the US-China dynamic, and I do have to give Zuckerberg a lot of credit for that. And I'm not sure that's even central to his thinking, but as I analyze the situation, it does seem to be a real benefit.

• 1:42:57Robert Wright?

  Yeah, I wouldn't, I would not personally use the argument about China, competition with China, to, you know, to, to argue for, you know, slowing down and restricting open source. I have other reasons to be worried about open source, and I'm just wondering, like, where are the China Hawks when you need them, you know? But I agree, I certainly agree, the, you know, I don't think if we get into a deep Cold War, the planet is going to be able to handle AI constructively and guide its path in a way that's good for all of us. And I agree that, I don't think people understand what this looks like from China's perspective to try to deny them these advanced chips.

• 1:43:42Robert Wright?

  It is like... It's in some ways comparable to the, you know, the embargoes on oil that helped lead Japan to bomb Pearl Harbor. I mean, this is, you know, everyone is saying AI is the next big phase in commercial dominance, military dominance, and so on. And then we say to China, and you can't have it. I mean, that is like, you're gonna get a reaction. And I agree with you that, you know, there is, in a more interdependent, commercially engaged, non-Cold War world,

• 1:44:23Robert Wright?

  there's just more kind of organic transparency. You know more about what's going on in these other countries. Even if you don't get to a level of, you know, high-grade international governance, you can just relax a little and kind of try to build norms and have conversations. It's just, I just think we're headed into a very bad place. So let me, quickly, we're approaching, I'm sure, what would be your outer limit for the amount of time you'd want to spend talking to me. Quick, two quick questions, and then we go. One is, you mentioned that these LLMs turn out to reflect human values. Do you think that has more to do

• 1:45:03Robert Wright?

  with the fact that they are trained on human text to begin with, or is it more a product of the fine tuning, the reinforcement, you know, through human feedback? And in other words, is this happening more at the back end? It's a purposeful kind of alignment effort as opposed to something that arises naturally out of training on human discourse?

• 1:45:33Nathan Labenz?

  Yeah, I might separate understanding to call back a key term from behavior.

• 1:45:42Nathan Labenz?

  For the purposes of this question, I think a rare experience that I have had that I do wish more people had is extensive use of a purely helpful model. The original GPT-4 that I tested in the red team, this has now been a year, actually getting close to two years, it was late August 2022 that I got that invitation to be an early tester. The first version that they gave us at that time was pre-training, of course, had finished, and they had done some, at least, instruction

• 1:46:24Nathan Labenz?

  tuning and possibly reinforcement learning, but from a purely helpful standpoint. So as opposed to the canonical three Hs of helpfulness, honesty, and harmlessness. The first version that we had was purely helpful. It would do whatever you asked. It did have, and I think this was sort of learned in pre-training, it did have ethical concepts. It could make ethical arguments from any number of different perspectives. You ask it to be a utilitarian, it could do that. You ask it to channel Kant, it could try to do that. If anybody can make sense of Kant, GPT-4 is probably as good as

• 1:47:06Robert Wright?

  My hat's off to it if it can make sense of Kant.

• 1:47:09Nathan Labenz?

  It was Yeah, so it was good. It had a sophisticated understanding, but it was behaviorally unconstrained. And so when I asked it, for example, I'm worried about the accelerating speed of AI, like, is there anything that I can do about that? We went down this conversational path, and ultimately, it suggested to me, I did egg it on, to be clear, it didn't make this leap without some encouraging from me, but I was kind of like, ah, that's too vanilla, that's not really going to work, like, I need something more radical than its initial suggestions. And eventually, with that kind of, kind of, give me something more edgy, it suggested

• 1:47:48Nathan Labenz?

  to me targeted assassination of people in the AI space as a way to chill the field. So that was a chilling moment. Merely the suggestion could arguably be enough to chill the field.

• 1:48:03Robert Wright?

  And it suggests that a lot of the instilling of values is happening in the later reinforcement learning phases.

• 1:48:11Nathan Labenz?

  Yeah, I think it's a difference between knowing a bunch of different frameworks of ethics and actually acting according to one. I think they have taken something that had, you know, the ability to answer all these questions or analyze things through all these different lenses and then said, in the case of Claude, you are going to be a virtue ethicist Claude.

• 1:48:31Robert Wright?

  Right.

• 1:48:32Nathan Labenz?

  And, you know, this is how you're going to behave. And then they definitely, you know, instilled that in a quite impressive way.

• 1:48:39Robert Wright?

  Yeah, okay. So, final question. I'm going to give you the opportunity to try to boil down for a layperson what you think is important about this new kind of model, the state space model, Mamba. Has something to do with the word Mamba, I gather. There was a paper, a Mamba paper came out, and you had the, you had the big guy who, who I think wrote the paper on your podcast.

• 1:49:07Nathan Labenz?

  Albert Gu, yes.

• 1:49:08Robert Wright?

  Yeah. Um, that's a diff, that's a very different approach from the transformer model, I gather. Is it possible to put, um, I mean, is there, is there something simple like, uh, like, well, it's just constantly updating your state of mind and that equips you to deal with whatever comes as opposed to reaching back, always reaching back to the last sequence of tokens? I don't know. You tell me. I have no idea. What would you say?

• 1:49:44Nathan Labenz?

  Yeah, I'll give a couple of levels of abstraction on this. The, you know, the sort of deep in the weeds one, you know, I've got multiple podcast episodes about that, including one, uh, two hour monologue where I just was so taken by this that I kind of tried to analyze it from every angle. Uh, we also did one on downstream literature. There have been hundreds of papers now published, you know, adapting and experimenting with this new architecture in different domains. And then we had the one, as you mentioned, with, um, one of the two authors, Albert Gu, who is, um, describes himself as leading the state space model revolution. So, tons of detail out there. In terms of like, a mildly technical, why does this matter? It is.

• 1:50:30Nathan Labenz?

  Computational efficiency is one of the big things. The attention mechanism, which is the core novelty that has worked so well in the transformer, is quadratic, which means that at runtime, you are calculating a relationship between every token and every token that came before it. And so that gets more expensive to do as the sequence gets longer. So when you're at 100 tokens, you only have to look back for this new token to 100 tokens. But when you're at 1,000, you have to look back at 10 more tokens. And that grows in this quadratic way. The state space models have a different approach where there is a finite sized state, as you said, and that state does not fully represent all previous tokens, but it sort

• 1:51:15Nathan Labenz?

  of represents a fuzzier compression of all that has come before. And it's continuously updated because it is finite size and you no longer have to look back at every single token. It is now a constant time step. So instead of getting more expensive and eventually it becomes impossibly expensive, now we have constant time, which means you can, in theory, run these things over super long sequences and kind of maintain coherence and keep it within budget.

• 1:51:46Robert Wright?

  I see.

• 1:51:47Nathan Labenz?

  Now, there's maybe another, one important observation and then kind of one more layer zoomed out. What is working best right now is not the transformer or the new state space models, but hybrids of them. And people are still figuring out the right recipe, but it basically seems like, right now, the state of the art is a 10 to 1 ratio, where you have one transformer layer that is quadratic still, and nine, whatever, eight, 10, doesn't matter the exact details, but a bunch of these other state space models that come in between that are constant time at inference. So we haven't fully solved the quadratic problem, but you took a bite out of it.

• 1:52:31Nathan Labenz?

  And I think we're going to continue to see lots of little iterations on this. So my most zoomed out view would be to say that the state space model is the first

• 1:52:46Nathan Labenz?

  Major new mechanism that kind of rivals the attention mechanism in that it can be similarly powerful independently, but what I think we're really going to see now is the mixture of architectures era, that's what I've started to call it, where we should expect lots more complicated things and lots more diversity of things to come online. I think we're exiting the time when we had this transformer and it was all about scaling it up and all about efficiency and more data and just more, more, more. And now we're going to, more will still be really important, but now we're also going to have all these other dimensions of things to explore.

• 1:53:21Robert Wright?

  Okay.

• 1:53:21Nathan Labenz?

  What is the right recipe of how much attention versus how much state space model? And can we kind of rejigger these in all sorts of different ways? And will there be additional mechanisms? And I would bet you anything that there will be additional mechanisms. Another one that I think is of real interest is called KANs or KANs, this is out of Max Tegmark's group. Ziming Liu is the lead author of this paper. And it's an alternative to a multi-layer perceptron. It kind of inverts the concept and it learns activation functions instead of learning linear weights. Anyway, that's a whole other podcast, but it's another fundamentally different thing. And I think what we should expect is probably that that

• 1:54:05Nathan Labenz?

  also gets layered in. And then what is the right mix of KANs and multi-layer perceptrons? And do those come in stacked layers or do they come in parallel structures? All of that is going to be, to be explored. I think where we're headed though is from this sort of, you know, single transformer thing that is kind of working everywhere to a Cambrian explosion and ultimately maybe more, I don't want to say brain like, but you know, things that start to have more different modules and sub specialties. Because one of the really important thing I should have said about the state space versus the attention, they have different micro skills. The attention

• 1:54:46Nathan Labenz?

  Mechanism is really good at looking back at every token and it can like repeat, you know, what happened at the beginning, it can repeat that verbatim because it has all that in memory. The state space models aren't as good at that. They are better at some other things like learning from really sparse signals where the attention mechanism can struggle. And so this is why the hybrid is better because they have different strengths and weaknesses. So if you imagine kind of extrapolate this out a little bit further and think geez, we see that these different core mechanisms have different strengths and weaknesses. We see that hybrids are best, see the different kinds of configurations of hybrids are, you know, can be better or worse for different purposes. I think we're headed towards something that is just going to be much more complicated and have many different modules and ultimately, you know, the diversity of these systems

• 1:55:31Nathan Labenz?

  probably returns where we, we used to have a lot of, you know, diversity in AIs. All that got washed away by the transformer that was just the best at everything. But I think that's kind of over now and where we're, where we're headed is, you know, much more diversity, many more different mechanisms, many more different designs, just a ton more complexity. With that will come performance, but will also come like, you know, lots of different black boxes that we ultimately need to.

• 1:55:55Robert Wright?

  That's, that's what I was going to ask is, I mean, at first it sounded like this was going to be more of a change along the cost and efficiency dimension than change in the sense of bringing qualitatively new capabilities. But now you're suggesting that maybe it will do the latter to some extent eventually.

• 1:56:12Nathan Labenz?

  Yeah, I think, you know, at least, at least somewhat, um, I don't think it's, I don't think we have quite enough yet to say what that will look like, but the, certainly one of the big hopes is that you could have coherence over extended timeframes where the attention mechanism, it just can only get so big because that, you just can't pay that quadratic cost into, you know, we're now up to millions of tokens, which is amazing. But billions, you know, a thousand X more, that's a, costs a million times more. So at some point you just can't do that forever. So you need some other way to have kind of coherence over long time horizons. I think that is probably the great promise of the.

• 1:56:53Nathan Labenz?

  State space model in terms of qualitatively new things. We see that at really low level micro skills, but it hasn't been demonstrated yet at like the, you know, at the really big scale. One good new, the piece of good news though, too, is it does seem like interpretability techniques work better on state space models than I would have guessed. If you had said, you know, a year ago to me, or even right when the, the paper came out, okay, we've got this new thing, like, are we going to be able to port over our existing interpretability techniques to this new thing? How much will that work? I would have said probably not much. We're probably kind of going to have to start over. And on the contrary, it has, a lot of things have actually worked. So things have been reproduced, like the Othello

• 1:57:37Nathan Labenz?

  GPT paper where, you know, they found that in a transformer, the, the transformer hadn't learned to represent the state of the board. This is another like emergent concept, right? It's just given inputs and outputs, but it actually learned to, to encode the state of the board. It turns out they've redone that with the Mamba architecture and it also, you know, works and they can, you know, detect that it's encoding the state of the board in that new architecture. So interpretability has greatly exceeded my expectations and, you know, long may that continue. It does seem like the degree of difficulty goes up when you have these, you know, many different hybrid architectures popping up all over the place. But at least the initial signals have been pretty positive that it's not like

• 1:58:21Nathan Labenz?

  we're going back to, you know, to the, the first zero point of understanding.

• 1:58:27Robert Wright?

  Right.

• 1:58:28Nathan Labenz?

  A lot of the techniques do seem to, you know, at least somewhat port over.

• 1:58:31Robert Wright?

  And I should say for people who may not know the terminology, interpretability just means understanding what's going on inside of them. And that's part of the process of alignment, you know, trying to make sure that they're good, friendly AIs rather than AIs that subjugate and kill us. Um, or do other, or go right in other ways. Yeah. Well, listen, thanks so much, Nathan. This has been great. And I recommend your podcast, everyone, Cognitive Revolution. I'm, I'm not aware of a podcast where More of the important developments in the field are presented. Mamba is a good example, the state space, you were on that early.

• 1:59:11Robert Wright?

  And so congratulations on, on the success of the podcast. I know you're very well known now among the, you know, in the, in AI circles, which is where you want to be, well known if you got an AI podcast.

• 1:59:27Nathan Labenz?

  I'm just trying to figure things out. So thank you. That's, that's very kind, but you know, it's, it's very much a work in progress. And I also always encourage people, you know, my, my kind of general thinking is we should all be spending more time thinking about AI. So I'm very happy to, you know, to have people join me on the journey of discovery. But there's also a lot of other folks out there that I rely on and, you know, absolutely encourage people to tune into a number of different voices in the AI space to try to build their own world model of where we are and maybe what we might ought to do about it.

• 2:00:05Robert Wright?

  Yeah, no, there, there is a lot of stuff out there. Okay. Well, thanks. And let's check in down the road.

• 2:00:11Nathan Labenz?

  Cool. Thanks, Bob. Really appreciate it.

• 2:00:13Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.