2028 AGI, New Architectures, and Aligning Superhuman Models with Shane Legg, Deepmind Founder, on The Dwarkesh Podcast

We're sharing a few of Nathan's favorite AI scouting episodes from other shows. Today, Shane Legg, Cofounder at Deepmind and its current Chief AGI Scientist, shares his insights with Dwarkesh Patel on AGI's timeline, the new architectures needed for AGI, and why multimodality will be the next big landmark. If you need an ecommerce platform, check out our sponsor Shopify: https://shopify.com/cognitive for a $1/month trial period. You can subscribe to The Dwarkesh Podcast here: https://www.youtube.com/@DwarkeshPatel We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. --- SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. X/SOCIAL: @labenz (Nathan) @dwarkesh_sp (Dwarkesh) @shanelegg (Shane) @CogRev_Podcast (Cognitive Revolution) TIMESTAMPS: (00:00:00) - Episode Preview with Nathan’s Intro (00:02:45) - Conversation with Dwarkesh and Shane begins (00:14:26) - Do we need new architectures? (00:17:31) - Sponsors: Shopify (00:19:40) - Is search needed for creativity? (00:31:46) - Impact of Deepmind on safety vs capabilities (00:32:48) - Sponsors: Netsuite | Omneky (00:37:10) - Timelines (00:45:18) - Multimodality This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, we're concluding our holiday bonus content series with an episode from the Dwarke Sh podcast. I assume most listeners of the Cognitive Revolution are at least familiar with Dwarke Sh, as he's had a number of major AI interviews mixed in with his generally excellent feed over the course of the last year, including AI safety philanthropist Holden Karnofsky, former GitHub CEO Nat Friedman, OpenAI Chief Scientist Ilya Sutskever, the legendary Eliezer Yudkowsky, my once upon a time New York City roommate Carl Schulman, true story, Anthropic CEO Dario Amade, and AI safety pioneer Paul Cristiano. Across all those interviews, I have really appreciated how Dwarke Sh has asked some of

• 0:43Nathan Labenz?

  the most fundamental, critical questions, in the most plain spoken way, to some of the most influential people in the field. And that skill will be on full display today as we present Dwarke Sh's recent interview with DeepMind co-founder and chief AGI scientist Shane Legg. I chose to feature this conversation because I think it's one of the most candid, straightforward assessments of AI scenarios that you'll hear anywhere. And it just happens to be coming from someone who not only foresaw where we'd be today, but founded an organization in DeepMind that has delivered breakthrough after breakthrough along the way. Overall, the views that Shane presents in this conversation match up extremely well with both my understanding of current AI systems' weaknesses and my expectations for how and

• 1:27Nathan Labenz?

  how soon they are likely to be overcome. So when I hear him say that he sees relatively clear paths forward to addressing most of the shortcomings we see in existing models, I don't hear that as guessing, particularly in light of the fact that DeepMind published one of the recent state space model papers that I covered in the Mamba episode. That was the block state transformers paper, and it came out just a few weeks after this episode originally aired. Of course, DeepMind has tons of other projects underway internally as well.

• 1:59Nathan Labenz?

  Overall, it really does seem to me that we're headed for some form of AGI over the next few years. And while we're definitely not ready for it, I appreciate how much and for how long Shane Legg has been thinking about this. He is a signer of the Center for AI Safety's one-sentence extinction risk statement, and he's one of the big reasons that I think the AI game board is in remarkably good shape overall. I just hope he's right that the challenge of AI alignment gets easier in important ways as the models become more sophisticated. All that and more make this short conversation a great jumping off point for 2024, year two of AI's second era. So to help you better calibrate your timelines, here is Dwarkesh Patel with DeepMind chief AGI scientist Shane Legg.

• 2:45Speaker 01

  Okay, today I have the pleasure of interviewing Shane Legg, who is a founder and the chief AGI scientist of Google DeepMind. Shane, welcome to the podcast.

• 2:57Speaker 02

  Thank you. It's a pleasure to be here.

• 2:58Speaker 01

  So first question, how do we measure progress towards AGI concretely? So we have these loss numbers, and we can see how the loss improves from one model to another. But it's just a number. How do we interpret this? How do we see how much progress we're actually making?

• 3:13Speaker 02

  That's a hard question, actually. AGI by its definition is about generality. So it's not about doing a specific thing. It's much easier to measure performance when you have a very specific thing in mind, because you can construct a test around that. Well, maybe I should first explain, what do I mean by AGI? Because there are a few different notions around. When I say AGI, I mean a machine that can do the sorts of cognitive things that people can typically do, possibly more. But that's to be an AGI, that's kind of the bar you need to meet. So if we want to test whether we're meeting the threshold or we're getting close to the threshold, what we

• 3:52Speaker 02

  actually need then is a lot of different kinds of measurements and tests that spans the breadth of all the sorts of cognitive tasks that people can do. And then to have a sense of what is human performance on these sorts of tasks, and that then allows us to sort of judge whether or not we're there. It's difficult, because you'll never have a complete set of everything that people can do, because it's such a large set. But I think that if you ever get to the point where you have a pretty good

• 4:26Speaker 02

  Range of tests of all sorts of different things that people do, cognitive things people do, and you have an AI system which can meet human performance and all those things. And with some effort, you can't actually come up with new examples of cognitive tasks where the machine is below human performance, then at that point, it's conceptually possible that there is something that the the machine can't do that people can do. But if you can't find it with some effort, I think all practical purposes unite.

• 4:56Speaker 01

  So let's get more concrete. You know, we measure the performance of these large language models on MMLU or something, and maybe you can explain what all these different benchmarks are. But the ones we use right now that you might see in a paper, what are they missing? What aspect of human cognition do they not measure adequately?

• 5:15Speaker 02

  Oh, yeah, now that hard grease. These are quite big areas. So they don't measure things like understanding streaming video, for example, because these are language models and people can do things like understanding streaming video. They don't do things like humans have what we call episodic memory, right? So we have a working memory, which are things that have happened quite recently. And then we have the sort of cortical memory. So these are things that have sort of been, you know, in our cortex that have been. But there's also a system in between, which is episodic memory, which is the hippocampus. And so this is about learning specific things very, very rapidly. So some of the things I say to you today, if you remember them tomorrow, that'll be your episodic memory

• 6:00Speaker 02

  hippocampus. Our models don't really have that kind of thing. And we don't really test for that kind of thing. We just sort of try to make the context windows, which is, I think, more like a working memory, longer and longer to sort of compensate for this. But yeah, we don't, we don't really test for that kind of a thing. So there is, there is all sorts of bits and pieces. But, you know, it is, it is a difficult question because you really need to, as I said, intelligence, the generality of human intelligence is very, very broad. So you really have to start going into the weeds of trying to find, you know, if there's specific types of things that are missing from existing benchmarks or different categories of benchmarks that, you know, don't currently exist or something. Yeah.

• 6:38Speaker 01

  Hmm. The thing you're referring to with episodic memory, would it be fair to call that sample efficiency? Or is that a different?

• 6:44Speaker 02

  It's, it's very much related to sample efficiency. It's, it's one of the things that enables humans to be very sample efficient.

• 6:51Speaker 01

  Right.

• 6:53Speaker 02

  Large language models have a certain kind of sample efficiency. Because when something is in their context window, they can then, that that sort of biases the distribution to to behave in a different way. And so that's a very rapid kind of learning. So there are multiple kinds of learning and the existing systems have some of them, but not others. So it's a little bit complicated.

• 7:14Speaker 01

  So this kind of memory, or we call it sample efficiency, whatever. Is it a fatal flaw of these deep learning models that it just takes trillions of tokens, far more, many mortars in magnitude more than a human will see throughout their lifetime? Or is this something that just solved over time?

• 7:30Speaker 02

  So the models can learn things immediately when it's in a context window. And then they have this sort of this longer process of when you actually train the base model and so on. And that's they're learning over trillions of tokens, but they sort of miss something in the middle.

• 7:45Speaker 01

  Right.

• 7:45Speaker 02

  Right, that's sort of what I'm getting at here. I don't think it's a fundamental limitation. I think what's happened with large language models is something fundamental has changed. We know how to build models now that have some degree of, I would say, understanding of what's going on. And that did not exist in the past. And because we've got a scalable way to do this now, that unlocks lots and lots of lots of new things. Now, we can then look at things which are missing, such as this sort of episodic memory type thing, and we can then start to imagine ways to address that. So my feeling is that there are kind of relatively clear paths forwards now to address

• 8:30Speaker 02

  most of the shortcomings we see in existing models, whether it's about delusions, factuality, the type of memory and learning that they have, or understanding video, or all sorts of things like that. So I'm not actually, I don't see there are big blockers here. I don't see big walls in front of us. I just see there is more research and work and these things will will improve and probably be adequately solved.

• 8:54Speaker 01

  But going back to the original question, how do you measure when human level AI is arrived or beyond it? As you mentioned, there's these other sorts of benchmarks you can use and other sorts of traits. But concretely, if there is, what would it have to do for you to be like, okay, we've reached human level? Would it have to beat Minecraft from start to finish? Would it have to get 100% on MMLU? What would it have to do?

• 9:16Speaker 02

  There is no one thing that would do it, because I think that's the nature of it. It's about general intelligence, so I'd have to make sure it could do lots and lots of different things and it didn't have a gap. We already have systems that can do very impressive categories of things to human level or even beyond. So I would want a whole suite of tests that I felt was very comprehensive. And then furthermore, when people come in and say, okay, so it's passing a big suite of tests, let's try to find examples. Let's take an adversarial approach to this. Let's deliberately try to find examples where people can clearly typically do this, but the machine fails. And when those people cannot succeed, I'll go, okay, we're probably there.

• 10:01Speaker 01

  A lot of your earlier research, at least on that, Zach, I find, emphasized that AI should be able to manipulate and succeed in a variety of open-ended environments. It kind of sounds like a video game almost. Is that where your head is still at now, or do you think about it differently?

• 10:16Speaker 02

  Yeah, it's evolved a bit. When I did my thesis work around universal intelligence and so on, I was trying to come up with a sort of extremely universal, general, mathematically clean framework for defining and measuring intelligence. And I think there were aspects of that that were successful. I think in my own mind, it clarified the nature of intelligence being able to perform well in lots of different domains and different tasks and so on. It's about that sort of capability of performance and the breadth of performance.

• 10:59Speaker 02

  So I found that was quite helpful and enlightening. There was always the issue of the reference machine, because in the framework, you have a weighting of things according to their complexity. It's like an Occam's razor type of thing, where you weight tasks, environments, which is simpler, more highly in this sort of, because you've got an infinite, it's countable space of different computable environments, or semi-computable environments. And that complexity measure has something built into it, which is called a reference machine. And that's a free parameter.

• 11:41Speaker 02

  So that means that the the intelligence measure has a free parameter in it. And as you change that free parameter, it changes the weighting and the distribution over the space of all the different tasks and environments. So this is sort of an unresolved part of the whole problem. So what reference machine should we ideally use? There isn't really a, there's no universal, like one specific reference machine. People will usually put a universal Turing machine in there, but there are many kinds of universal Turing machines. So you have to put it, sorry, a universal Turing machine, but there are many different ones. So I think, given that it's a free parameter, I think the most natural thing to do is say,

• 12:22Speaker 02

  okay, let's think about what's meaningful to us in terms of intelligence. I think human intelligence is meaningful to us and the environment that we live in. We we know what human intelligence is, we are human too, we interact with other people who have human intelligence. We know that human intelligence is possible, obviously, because, you know, it exists in the world. We know that human intelligence is very, very powerful because it's affected the world profoundly in countless ways. And we know if human level intelligence was achieved, that would be economically transformative because the types of cognitive tasks people do in the economy could be done by machines then.

• 13:03Speaker 02

  And it would be philosophically important because this is sort of how we often think about, you know, intelligence. And I think historically would be a key point. So I think that human intelligence is actually quite, in a human-like environment, is quite a natural sort of reference point. So you could imagine sort of setting your your reference machine to be such that it emphasizes the kinds of environments that we live in as opposed to some abstract mathematical environment or something like that. And so that's how I've kind of gone on this journey of let's try to define a completely universal, clear mathematical notion of intelligence to, well,

• 13:43Speaker 02

  It's got a free parameter. One way of thinking about it is say, okay, let's think more concretely now about human intelligence. And can we build machines that can match human intelligence? Because we, we understand what that is. We know that that is a very powerful thing. It has, you know, economic, philosophical, historical kind of importance. So that's kind of the, the, and the other aspect, of course, is that, you know, in this pure formulation of combative complexity, it's actually not computable. And I also knew that there was, there was a limitation at the time, but it was a, it was an effort to say, okay, can we just even very theoretically come up with a clean definition? I think we can sort of get there. We have this issue of a, of a, of a reference machine, which is unspecified.

• 14:26Speaker 01

  So before we move on, I do want to ask, on the original point you made about these machines, or these L L M's need episodic memory. Well, you said that these are problems that we can solve. These are not fundamental impediments. But are, when you say that, do you think they'll just be solved by scale? Or do each of these need a fine-grained, specific solution that is architectural in nature?

• 14:50Speaker 02

  I, I, I think it'll be architectural in nature. Because the, well, the current architectures, they, they, they don't really have what you need to do this. They basically have a context window, which is very, very fluid, of course. And they have the weights, which things get baked into very slowly. So to my mind, it feels like working memory, which is like the, the activations in your brain. And then the weights, the synapses and so on in your cortex. Now, the, the brain separates these things out. It has a separate mechanism for rapidly learning specific, specific information. Because that's a different type of optimization problem compared to slowly learning deep generalities. Right? That's sort of, there's a, there's a, there's a tension between the two.

• 15:34Speaker 02

  But you want to be able to do both. You want to be able to, I don't know, hear someone's name and remember it the next day. And you also want to be able to integrate information over a lifetime. So you start to see deeper patterns in the world. These are, these are quite different, different optimization targets, different, you know, processes. But a comprehensive system should be able to do both. And so I think it's. It's conceivable you could build one system does both, but you can see because they're quite different things that it makes sense for them to be different. I think that's why the brain does it separately.

• 16:06Speaker 01

  Hmm. I'm curious about how concretely you think that would be achieved. And I'm specifically curious, I guess you can answer this as part of the answer. You know, DeepMind has been working on these domain-specific reinforcement learning type setups, AlphaFold, AlphaCode, and so on. How does that fit into what you see as a path to AGI? Have these just been orthogonal domain-specific models, or do they feed into the eventual AGI?

• 16:33Speaker 02

  Things like AlphaFold are not really feeding into AGI. You know, we may learn things in the process that may end up being relevant, but I don't see them as being, likely being on the path to AGI. But yeah, we're, you know, we're a big group. We've got hundreds and hundreds and hundreds of PhDs working on lots of different projects. So, you know, when we find, you know, well, we see like opportunities to do something significant like AlphaFold, we'll go and do it. It's not like we, we only do AGI-type work. We work on fusion reactors and, you know, various things in sustainability, energy. We've

• 17:16Speaker 02

  got people looking at satellite images of deforestation. We have people looking at weather forecasting. We've got tons of people. We've got lots of things.

• 17:27Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:31Speaker 01

  On the point you made earlier about the reference class or the reference machine as human intelligence, it's interesting because in your 2008 thesis, one of the things you mentioned almost as a side note is, well, how would you measure intelligence? And you said, well, you could do a compression test and you could see if it fills in words in a sample of text, and that could measure intelligence. And funnily enough, that's basically how the ones are trained. At the time, did it stick out to you as especially fruitful thing to train for?

• 18:02Speaker 02

  Yeah, I mean, in a sense, what's happened is actually very aligned with what I write about my thesis, which the idea is from Marcus Hutter with AI X, where you take Solomon of induction, which is this incomputable, but theoretically very elegant and extremely sample efficient prediction system. And then, once you have that, you can build a general agent on top of it by basically adding search and reinforcement signal. That's what you do with AI X. But what that sort of tells you is that if you have a fantastically good sequence predictor,

• 18:43Speaker 02

  some approximation of Solomon of induction, then going from that to a very powerful, very general AI system, AGI system, is just sort of another step, you know, you've actually solved a lot of the problem already. And I think that's what we're seeing today, actually, that these incredibly powerful foundation models are incredibly good sequence predictors. They're compressing the world based on all this data. And then you will be able to extend these in different ways and build very, very powerful agents out of them.

• 19:14Speaker 01

  Hmm. All right. Let me ask you more about that. So Richard Sutton's bitter lesson essay says that there's two things you can scale, search and learning. And I guess you could say that L M's are about the learning aspect. The search stuff, which you've worked on throughout your career, where you have an agent that is interacting with this environment. And is that, is that the direction that needs to be explored again? Or is that something that needs to be added to L M's where they can actually interact with their data or the world in some way?

• 19:42Speaker 02

  Yeah. I think that's on the right track. I think there's, these foundation models are world models of a kind. And to do really creative problem solving, you need to start searching. So if I think about something like AlphaGo and the move 37, the famous move 37, where did that come from? Did that come from all its data that it's seen of human games or something like that? No, it didn't. It came from it identifying a move as being Quite unlikely, but possible. And then via a process of search, coming to understand that the, that was actually a very, very good move.

• 20:26Speaker 02

  So you need to, to get real creativity, you need to search through spaces of possibilities and find these sort of hidden gems. That's what creativity is. I think current language models, they don't really do that kind of a thing. They really are mimicking the data. They are mimicking all the human ingenuity and everything, which they have seen from all this data that's coming from the internet that's originally derived from humans. If you want a system that can go be really truly beyond that and not just generalize in novel ways. So it can, you know, these models can blend things. They can do, you know, Harry Potter in the style of a Kanye West rap or something, even though it's never happened. They can blend things together.

• 21:07Speaker 02

  But to do something that's truly creative, that is not just a blending of existing things, that requires searching through a space of possibilities and finding these hidden gems that, that are sort of the hidden away in there somewhere. And that requires search. So I don't think we'll see systems that truly step beyond their training data until we have powerful search in the process.

• 21:31Speaker 01

  So there are rumors that Google Deep Mind is training newer models and you don't have to comment on those specifically. But when you do that, if it's, if it's the case that search or something like that is required to go to the next level, are you training in a completely different way than say G P T four or other transformers are trained?

• 21:50Speaker 02

  I can't say much about how we're training. I think it's fair to say we're doing the sorts of scaling and training roughly that you see many people in the, in the field doing. But we have, you know, our own take on it and our own different tricks and techniques.

• 22:08Speaker 01

  Okay, maybe we'll come back to it and get another answer on that. But let's talk about alignment briefly. So what will it take to align human level and superhuman AIs? And, you know, it's interesting because the sorts of reinforcement learning and self like kinds of setups that are popular now, like Constitution AI or L H F, Deep Mind obviously has expertise in it for, for more decades longer. So I'm curious what you think of the current landscape and how Deep Mind pursues that problem of safety towards human level models.

• 22:39Speaker 02

  So do you want to know about what we're currently doing? Or do you want me to have a stab at what I think we, that needs to be done?

• 22:44Speaker 01

  Needs to be done.

• 22:44Speaker 02

  Needs to be done. So, I mean, in terms of what we're currently doing, we're doing lots of things. We're doing interpretability, we're doing our process supervision, we're doing red teaming, we're doing evaluation for dangerous capabilities, we're doing work on institutions and governance and, you know, tons of stuff, right? There's lots of different things. Anyway, what do I think needs to be done? So, I think, I think that powerful machine learning, powerful AGI is coming in some time. Right. And if the system is really capable, really intelligent, really powerful, trying to somehow contain it or limit it is probably not a winning strategy because these systems ultimately will be very, very capable.

• 23:29Speaker 02

  So what you have to do is you have to align it. You have to get it so it's fundamentally a highly ethical, value-aligned system from the get-go, right? How do you do that? Well, I, I have a, maybe this is slightly naive, but this is, this is my take on it. How do people do it, right? If you have a really difficult ethical decision in front of you, what do you do? Right? Well, you don't just do the first thing that comes to mind, right? Because, you know, there could be a lot of emotions involved and other things, right? It's a difficult problem. So what you have to do is you have to calm yourself down, you've got to sit down, and you've got to think about it. You've got to think, well, okay, what, what could I do?

• 24:14Speaker 02

  I could do this, I could do this, I could do this. If I do each of these things, what will happen, right? And then you have to think about, so that requires a model of the world. And then you have to think about ethically, how do I view each of these different actions and the possibilities of what may happen from it? Right? What is the right thing to do? And as you think about all the different possibilities and your actions and what can follow from them and, and how it aligns with your values and your ethics, you can then come to some conclusion of what is really, you know, the best choice that you should be making if you want to be, you know, really ethical about this. I think AI systems need to essentially do the same thing.

• 25:00Speaker 02

  So when you sample from a foundation model at the moment, it's like it's blurting out the first thing. It's like system one, if you like, from psychology from Kahneman, right? That's that's not good enough. And if we do RLHF or what's it called? I can't remember. Anyway, it's the AI version without the human feedback. R A I F? Is that what it is? Oh, gosh, I'm confusing myself. Anyway, constitutional AI tries to do this sort of thing. You're trying to fix the underlying system one in a sense, right? And that can shift the distribution and that can be very helpful. But it's a very high dimensional distribution and you're sort of poking it in a whole lot of points. And so it's not likely to be a very robust solution. Right?

• 25:44Speaker 02

  It's like trying to train yourself out of a bad habit. You know, you can sort of do it eventually. But what you need to do is you need to have a system two. You need the system to not just sample from the model. You need the system to go, okay, I'm going to reason this through. I'm going to do like step by step reasoning. What are the options in front of me? I'm going to use my world model now and I'm going to use a good world model to understand what's likely to happen from each of these options. And then reason about each of these from an ethical perspective. So you need a system which has a a a deep understanding of the world, has a good world model. It has a good understanding of people, has a good understanding of ethics, and has robust and very reliable reasoning.

• 26:28Speaker 02

  And then you set it up in such a way that it applies this reasoning and this understanding of ethics to analyze the different options which are in front of it and then execute on which is the most ethical way forwards.

• 26:41Speaker 01

  But I I think when a lot of people think about the fundamental alignment problem, the worry is not that it's not going to have a world model necessary to understand its actions. I'm sorry, to understand the effects of its actions. I guess it's one worry, but not the main worry. The the main worry is that the effects it cares about are not the ones we will care about. And so even if you improve its systems, your thinking and do better planning, the fundamental problem of we have this really nuanced values about what we want. How do we communicate those values and make sure they're reinforced in the AI?

• 27:14Speaker 02

  It needs not just a good model of the world, but it needs a really good understanding of ethics. And we need to communicate to the system what ethics and values it should be following.

• 27:24Speaker 01

  And how do we do that in a way that's we can be confident that a human level or eventually a superhuman level model will preserve those values or learn them in the first place?

• 27:34Speaker 02

  Well, it should preserve them because if it's making all its decisions based on a good understanding of ethics and values and it's consistent in doing this, it shouldn't take actions which undermine that. That would be inconsistent.

• 27:47Speaker 01

  Right. So then how do we get to the point where it's learned them in the first place?

• 27:50Speaker 02

  Yeah, that's the challenge.

• 27:52Speaker 01

  Yeah.

• 27:52Speaker 02

  We need to have systems. The way I think about it is this, to have a profoundly ethical AI system, it also has to be very, very capable. It needs a really good world model, a really good understanding of ethics, and it needs really good reasoning. Because if you don't have any of those things, how can you possibly be consistently profoundly ethical? You can't. So we actually need better reasoning, better understanding of the world, and better understanding of ethics in our systems.

• 28:22Speaker 01

  Right. So it seems to me the former two would just come along for the ride as these models get more powerful.

• 28:27Speaker 02

  Yeah, so that's a nice property because it's actually a capabilities thing.

• 28:30Speaker 01

  Right. But then if the third one is a bottleneck, or if the third one is a thing that doesn't come along with the AI itself, what is the actual technique to make sure that that happens?

• 28:39Speaker 02

  The third one, sorry.

• 28:41Speaker 01

  The ethical model. What do humans value?

• 28:43Speaker 02

  We, well, we've got a couple of problems. First of all, we need to decide, we should train the system on ethics generally. I mean, there's a lot of lectures and papers and books and all sorts of things. So it understands human ethics well, right? And we need to make sure it understands humans' ethics well, right? Because that's important, at least as well as a, you know, very good ethic system. And we then need to decide, okay, of this sort of general understanding of ethics, what do we want the system to actually value? And what sort of ethics do we want it to apply? Now, that's not a technical problem. That's a problem for society and ethicists and so on to come up with.

• 29:28Speaker 02

  Now, you know, I'm not sure there's such a thing as true or correct optimal ethics or something like that, but I'm pretty sure that it's possible to come up with a set of ethics, which is much better than the, you know, what the so-called doomers worry about in terms of the behavior of these AGI systems. And then what you do is you engineer the system to actually follow

• 29:53Speaker 01

  Yeah.

• 29:54Speaker 02

  um, these, these things. So every time it makes a decision, it does an analysis using a, you know, deep understanding of the world and of ethics and very robust and precise reasoning to do an ethical analysis of what it's doing. And of course, we'd want lots of other things. We want people checking these processes of reasoning. We'd want people, you know, verifying that it's behaving itself in terms of, you know, how it reaches these conclusions.

• 30:21Speaker 01

  But I, I still feel like I don't understand how that fundamental problem of making sure it follows that ethic, because presumably, you know, it, it has Mao's Little Red Book, so understands Maoist ethics and understands all these other ethics, you know? How do we make sure the ethic that we say this is the, this is the one we've decided that this is in society and so on today, that is the one it ends up following and not the other ones it understands?

• 30:42Speaker 02

  Right, so you have to specify to the system, these are the ethical principles that you should follow.

• 30:47Speaker 01

  And how do we make sure it does that?

• 30:48Speaker 02

  We have to check it as it's doing it. We have to assure ourselves that it is consistently following these ethical principles, at least, I mean, I'm not sure there's such a thing as optimally, but at least as well as a group of human experts.

• 31:03Speaker 01

  Are you worried that if you do the default way, which is just reinforcing it whenever it seems to be following them, you could be training deception as well? That is reinforced way?

• 31:11Speaker 02

  Reinforcement. Has some, I, some, some dangerous, dangerous aspects to it.

• 31:17Speaker 01

  Yeah.

• 31:18Speaker 02

  Um, I think it's actually more robust to do, you know, check the process of reasoning and check its understanding of ethics. So, you know, to, to reassure ourselves that this system has a really good understanding of ethics, it should be, you know, grilled for, for, for some time to try to really pull apart its understanding and make sure it has a very robust. And then also if it's deployed, we should have people constantly looking for how, you know, the decision is making and the reasoning process that goes into those decisions. To try to understand how, that is correctly reasoning about these types of things.

• 31:56Speaker 01

  Hmm. Speaking of which, um, do you at Google DeepMind have some sort of framework for?

• 32:00Speaker 02

  Now, this is, this is, this is, this is not so much a Google DeepMind perspective on this. This is, this is my take on how I think we need to do this kind of thing. There are, there are many different views within, and there are, there are different variants on, on these sorts of ideas as well.

• 32:15Speaker 01

  So then do you personally think there needs to be some sort of framework for as you arrive at certain capabilities, these are the concrete safety benchmarks that you must have instated at this point, or you should, you know, pause or slow down or something?

• 32:26Speaker 02

  Uh, I, I think that's a sensible thing to do. It's actually quite hard to do. Uh, and there are some people thinking about, I know Anthropic has put out some things like that. Uh, we, we're thinking about similar things. Actually, you know, putting concrete things down is actually quite a thing to do. So I think it's an important problem and I certainly encourage people to work on it, yeah.

• 32:46Speaker 01

  Hey, we'll continue our interview in a moment after a word from our sponsors. It, it's interesting because you have these blog posts that you wrote when you started DeepMind, um, you know, back in 2008, where you talk about, um, the, the motivation was to accelerate safety. On net, what do you think the impact of DeepMind has been on safety versus capabilities?

• 33:08Speaker 02

  Ooh, interesting.

• 33:16Speaker 02

  I don't know, it's hard to, hard to judge, actually. You know, back in the, I, I've been worried about AGI safety for a long time, well before DeepMind.

• 33:30Speaker 02

  But it was, it was always really hard to hire people, actually, particularly in the early days, to work on AGI safety.

• 33:37Speaker 02

  I'm thinking back on two thousand, like thirteen or so, I think we had the first hire and he only agreed to do it part-time because he didn't want to, you know, drop all the capabilities work because, you know, the impact of your data wasn't career and so on. And this was someone who'd already previously been publishing in AGI safety, so.

• 33:56Speaker 02

  Yeah, I don't know, it's hard to, hard to know what is the counterfactual if we, if we weren't, weren't there doing it. I think, you know, we have been, we've been a group that's been,

• 34:11Speaker 02

  you know, talked about this openly, I've, I've, I've talked about this on many occasions, the importance of it. We've been, you know, hiring people to, to work on these topics.

• 34:22Speaker 02

  You know, I know a lot of other people in the area and I've talked to them over many, many years, I've known Dario since two thousand and five or something or other, you know, we've talked on and off about AGI safety and so on, so. I don't know, the, the impact that DeepMind has had, you know, we, I guess we were the first,

• 34:40Speaker 02

  I'd say the first AGI company, and as the first AGI company, we, we, you know, we always had an AGI safety group. We, we've been publishing papers in this for many years. I think that's leaned some credibility to the area when people see, oh, here's a AGI, I mean, AGI was a, you know, it was a fringe term not that long ago, and this person's doing AGI safety, not, well, they're a DeepMind? Oh, okay, I, I, I hope that sort of, you know, creates some space for people.

• 35:09Speaker 01

  And where, where do you think AI progress itself would have been without DeepMind? And this is not just a point that people make about DeepMind, I think this is a general point people make about OpenAI and Anthropic as well, that these people went into the business to accelerate safety and sort of the net effect might have been to accelerate capabilities far more.

• 35:25Speaker 02

  Right, right, right. I think we have accelerated capabilities, but again, the counterfactuals are quite, quite difficult. I mean, we, we didn't do ImageNet, for example, and ImageNet, I think, was very influential in, in attracting investment to the field.

• 35:40Speaker 02

  Um. We did do AlphaGo, um, and that changed some people's minds. Um, but you know, the the community is a lot bigger than just DeepMind. I mean, we, we have, well, not so much now, but because there are a number of other, you know, players with significant resources. But if you went back more than five years in the future, we were able to do um bigger projects with bigger teams and take on more ambitious things than than a lot of the smaller academic groups, right? And so the sort of nature of the type of work we could do was a bit different. Um, and that, I think, that affected the dynamics in some ways. But you know, the, the, the community is much, much bigger than, say, DeepMind. So

• 36:25Speaker 02

  maybe we've sped things up a bit, but I think a lot of these things would have happened for far too long anyway. I think, I think these, of- often good ideas are kind of in the air and, you know, as, as a researcher, you know, when sometimes you publish something or you're about to publish something, you see somebody else who's got a very similar idea coming out with some good results. Um, I think often it's the time is right, right for things. So, you know, it's, I find it very hard to reason about the counterfactuals there.

• 36:53Speaker 01

  Hmm. Speaking of the early years, it's really interesting that in, um, two thousand nine, you had a blog post where you say, my modal expectation of when we get human-level AI is twenty twenty-five, expected value is twenty twenty-eight. And this is before deep learning. This is when nobody's talking about AI. And it turns out like if you, if the trends continue, this, this is not an unreasonable prediction. This was, uh, how did you, I mean, before all these trends came into effect, how did you have that accurate an estimate?

• 37:20Speaker 02

  Well, first I'd say it's not before deep learning.

• 37:23Speaker 01

  Ah.

• 37:23Speaker 02

  Um, deep learning was getting started around two thousand eight.

• 37:28Speaker 01

  Oh, sorry. I meant to say before ImageNet.

• 37:30Speaker 02

  Before ImageNet, it was two thousand twelve, yeah. Um, so, well, I first formed those beliefs in about two thousand one, after reading Ray Kurzweil's The Age of Spiritual Machines. And I, I came to the conclusion, he was, he was, there was two really important points that in, in, in his book that I, I came to believe is true. One is that I Computational power would grow exponentially for at least a few decades. And that the quantity of data in the world would grow exponentially for a few decades. And when you have exponentially increasing quantities of computation and data, then the value of highly scalable algorithms gets higher and higher.

• 38:15Speaker 02

  So then there's a lot of incentive to make a more scalable algorithm to harness all this compute and data. And so I thought it would be very likely that we'll start to discover scalable algorithms to do this. And then there's a positive feedback between all these things, because if your algorithm gets better at harnessing compute and data, then the value of the data and the compute goes up because it can be more effectively used. And so that drives more investment to these areas. If your compute performance goes up, then the value of the data goes up because you can utilize more data. So there are positive feedback loops between all these things. So that was, that was the first thing. And then the second thing was just looking at the trends. If these scalable algorithms were, were to be discovered,

• 39:00Speaker 02

  then during the 2020s, it should be possible to start training models on significantly more data than a human would experience in a lifetime. And I figured that that would be a time where, where big things would start to happen and that would eventually unlock AGI. So that was, that was my reasoning process. And I think we're now at that first part. I think we can start training models now with the scale of the data is beyond what a human can experience in a lifetime. So I think this is the first unlocking step. And so, yeah, I think there's a 50% chance that, so about 2028. Now, it's just a 50% chance. I mean, I'm, I'm sure what's going to happen is going to get to, you know, 2029 and someone's going to say, oh Shane, you were wrong. It's like, come on, it's 50% chance.

• 39:45Speaker 02

  So, yeah, I, I, I, I think it's, it's entirely plausible. Yeah, it's 50% chance it could happen by 2028. But I'm not going to be surprised if it doesn't happen by then. Maybe, maybe, you know, the, you're, you often hit unexpected problems in, in research and sciences. And sometimes things take longer than you expect.

• 40:05Speaker 01

  If, if there was a problem that caused it, if we're in 2029 and it hasn't happened yet, looking back, what would be the most likely reason that would be the case?

• 40:16Speaker 02

  I don't know.

• 40:19Speaker 02

  I don't know. I, at the moment, it looks to me like all the problems are likely solvable with a number of years of research. That's my current sense.

• 40:30Speaker 01

  And what does the time from here to 2028 look like, if the 2028 ends up being the year? Is it, is it just we have trillions of dollars of economic impact in the meantime? And the world gets crazier? What happens?

• 40:42Speaker 02

  I, I think what you'll see is the existing models maturing. They'll be less delusional, much more factual. They'll be more up to date on what's currently going on when they answer questions. They'll become multimodal, much more than they currently are.

• 41:02Speaker 02

  And this will just make them much more useful. So I think probably what we'll see more than anything is just loads of great applications for the, for the coming years. I think that'll be the, there, there can be some misuse cases as well. I'm sure somebody will come up with, you know, some, something to do with these, with these models that is quite helpful. But my expectation for the coming years is mostly a positive one. We'll see all kinds of really impressive, really amazing applications for the, for the, for the coming years. Yeah.

• 41:35Speaker 01

  And on the safety point, you mentioned these different research directions that are out there and that you are doing internally in DeepMind as well. Interoperability, RAIF, and so on. Which are you most optimistic about?

• 41:51Speaker 02

  I don't know. I don't want to pick favorites.

• 41:55Speaker 02

  It's hard picking favorites. I know the people working on all these areas.

• 42:01Speaker 02

  I think, I think things of the sort of system two flavor. There's a, there's a, there's a work we have going on that Jeffrey Irving leads called deliberative dialogue, which kind of has a system two flavor where you have this sort of debates takes place about the actions that an agent could take or what's the correct answer to something or something like this. And people then can sort of review these, these debates and so on. And they, they use these sort of, these AI algorithms to help them judge. The correct outcomes and so on. And so this is sort of meant to be a way in which to try

• 42:40Speaker 02

  to scale the alignment to sort of increasingly powerful systems. So I think things of that kind of flavor, I think, have quite a lot of promise, in my opinion. But that's kind of quite a broad category. There are many different topics within that.

• 42:58Speaker 01

  That's interesting. So you've mentioned two, two areas in which AI needs to improve. One is episodic memory and the other is assistive thinking. Are those two related or are they two separate, are they two separate drawbacks?

• 43:14Speaker 02

  I think they're fairly separate, but they can be somewhat related. So you can learn different ways of thinking through problems and actually learn about this rapidly using your episodic memory. So all these different systems and subsystems interact. So they're never completely separate. But I think conceptually, you can probably think of them as quite separate things. I think delusions and factuality is another area that's going to be quite important. And particularly important in lots of applications. You know, if you want a model that writes creative poetry, then that's fine. Because you want to be able to be very free to suggest all kinds of possibilities and so on. You're not really constrained by a specific reality.

• 43:56Speaker 02

  Whereas if you want something that's in a, in a, in a particular application, normally you have to be quite concrete about, you know, what's currently going on and what is true and what is not true and so on. And models are a little bit sort of freewheeling when it comes to, you know, truth and creativity at the moment. And that, I think, limits their applications in many ways.

• 44:16Speaker 01

  So, final question is this. You've been in this field for over a decade, much longer than many others. And you've seen these different landmarks, ImageNet, transformers. What do you think the next landmark will look like?

• 44:32Speaker 02

  I think the next landmark that people will think back to and remember is going much more fully multimodal, I think. Because I think that will, that'll open out the, the sort of understanding that you see in language models into a much larger space of possibilities. And when people think back, they'll think about, oh, those old fashioned models, they, they just did like chat, they just did text. You know, it was, it just felt like a very narrow thing, whereas now they, you know, they understand. When you talk to them and they, they understand images and pictures and video and, and you can show them things or things like that.

• 45:11Speaker 02

  And they'll, they will have much more understanding of what's going on and it'll feel like the system's kind of opened up into the world and, and, and, and, and in a much more powerful way.

• 45:20Speaker 01

  Do you mind if I ask you a follow up on that? So, uh, ChatGPT just released their multimodal feature and then you in DeepMind, you had the Goto paper where, you know, you can, you have this one model, you can images, even actions, the video games, whatever you can throw in there. Um, and so far it doesn't seem to have been, it hasn't percolated as much as even like ChatGPT initially from G P T three or something. What explains that? Is it just that people haven't learned to use multimodality? They're not powerful enough yet?

• 45:45Speaker 02

  Uh, I think it's early days. Um, I think there's, you can see promise there, understanding images and things, uh, more and more. But I think it's, yeah, it's early days in this transition. Uh, is when you start really digesting a lot of the video and other things like that, that the systems will start having a much more grounded understanding of the world and all kinds of other aspects. And then when that works well, that will open up naturally lots and lots of new, new applications and all sorts of new possibilities. Cause you're not confined to text chat anymore.

• 46:16Speaker 01

  The new avenue of training data as well, right?

• 46:19Speaker 02

  Yeah, new training data and new, in all kinds of different applications that aren't just purely textual anymore. Um, and, you know, what are those applications? Well, probably a lot of them, we can't even imagine at the moment because there are just so many, so many possibilities. Once you can start dealing with all sorts of different modalities in a consistent way.

• 46:37Speaker 01

  Awesome, Shane, I think that's an excellent place to leave it off. Thank you so much for coming on the podcast.

• 46:42Speaker 02

  Thank you.

• 46:43Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.