Ilya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligence

I went over to the OpenAI offices in San Fransisco to ask the Chief Scientist and cofounder of OpenAI, Ilya Sutskever https://twitter.com/ilyasut, about: * time to AGI * leaks and spies * what's after generative models * post AGI futures * working with Microsoft and competing with Google * difficulty of aligning superhuman AI Watch on YouTube https://youtu.be/Yf1o0TQzry8. Listen on Apple Podcasts https://podcasts.apple.com/us/podcast/ilya-sutskever-openai-chief-scientist-building-agi/id1516093381?i=1000606122602, Spotify https://open.spotify.com/episode/5p5oJYgVN8vMvEW2nngBN7?si=1zDcArwGSjSOS-D4nk5_OA, or any other podcast platform. Read the full transcript here https://www.dwarkeshpatel.com/p/ilya-sutskever. Follow me on Twitter https://twitter.com/dwarkesh_sp for updates on future episodes. Timestamps (00:00) - Time to AGI (05:57) - What’s after generative models? (10:57) - Data, models, and research (15:27) - Alignment (20:53) - Post AGI Future (26:56) - New ideas are overrated (36:22) - Is progress inevitable? (41:27) - Future Breakthroughs This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  But I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions. Are you worried about spies? I'm really not worried about the way it's being leaked. We'll all be able to become more enlightened because we'd interact with an AGI that will help us see the world more correctly. Like, imagine talking to the best meditation teacher in history. Microsoft has been a very, very good partner for us. So I challenge the claim that next token prediction cannot surpass human performance. If your base neuron is smart enough, you just ask it like, what would a person with great insight and wisdom and capability do?

• 0:43Dwarkesh Patel?

  Okay, today I have the pleasure of interviewing Ilya Sutskever, who is the co-founder and chief scientist of OpenAI. Ilya, welcome to the Lunar Society.

• 0:52Speaker 00

  Thank you, happy to be here.

• 0:54Dwarkesh Patel?

  First question, and no humility allowed. There's many scientists, or maybe not that many scientists, who will make a big breakthrough in their field. There's far fewer scientists who will make multiple independent breakthroughs that define their field throughout their career. What is the difference? What distinguishes you from other researchers? Why have you been able to make multiple breakthroughs in your field?

• 1:13Speaker 00

  Well, thank you for the kind words.

• 1:16Speaker 00

  It's hard to answer that question. I mean, I tried really hard, I gave it everything I got. And

• 1:25Speaker 00

  that worked so far. I think that's all there is to it.

• 1:30Dwarkesh Patel?

  Got it. What's the explanation for why there aren't more illicit uses of GPT? Why aren't more foreign governments using it to spread propaganda or scam grandmothers or something?

• 1:40Speaker 00

  I mean, maybe they haven't really gotten to do it a lot. But it also wouldn't surprise me if some of it was going on right now. Certainly, I imagine they'd be taking some of the open source models and trying to use them for that purpose.

• 1:57Speaker 00

  Like I sure I would expect this would be something that be interested in in the future.

• 2:01Dwarkesh Patel?

  It's like technically possible, they just haven't thought about it enough?

• 2:04Speaker 00

  Or haven't like done it at scale using their technology? Or maybe it's happening, but you just don't know it.

• 2:09Dwarkesh Patel?

  Would you be able to track it if it was happening?

• 2:11Speaker 00

  I think large scale tracking is possible, yes. I mean, this requires of all special operation, it's possible.

• 2:17Dwarkesh Patel?

  Hmm. Now, there's some window in which AI is very economically valuable on the scale of airplanes, let's say. But we haven't reached AGI yet. How big is that window?

• 2:28Speaker 00

  I mean, I think this window, it's hard to give you a precise answer, but it's definitely going to be like a good multi-year window. It's also a question of definition because AI before it becomes AGI is going to be increasingly more valuable year after year. I'd say in an exponential way. So it's some, in some sense, it may feel like, especially in hindsight, it may feel like there was only one year or two years because those two years were larger than the previous years. But I would say that already

• 3:00Speaker 00

  last year, there've been a fair amount of economic value produced by AI. And next year is going to be larger and larger after that. So I think that it's going to be a good multi-year chunk of time. But that's going to be true. I would say from now till AGI, pretty much.

• 3:18Dwarkesh Patel?

  Okay, well, because I'm curious if there's a startup that's using your models, right? At some point, if you have AGI, there's only one business in the world, right? It's OpenAI. How much window do they have, does any business have, where they're actually producing something that AGI can't produce?

• 3:33Speaker 00

  Yeah, well, I mean, it's the same, it's the same question as asking how long until AGI?

• 3:37Dwarkesh Patel?

  Yeah.

• 3:37Speaker 00

  I think it's a hard question to answer. I mean, I hesitate to give you a number also because there is this thing where effect where people who are

• 3:46Speaker 00

  optimistic people who are working on the technology tend to underestimate the time it takes to get there. But I think that the way I ground myself is by thinking about a self-driving car. In particular, there is an analogy where if you look at the Tesla and if you look at the self-driving behavior of it, it like, it looks like it does everything.

• 4:06Speaker 00

  It does everything.

• 4:07Dwarkesh Patel?

  All right.

• 4:07Speaker 00

  But it's also clear that there is still a long way to go in terms of reliability. And we might be in a similar place with respect to our models where it also looks like we can do everything. And at the same time, it will be, we'll need to do some more work until we really iron out all the issues and make it really good and really reliable and robust and well behaved.

• 4:29Dwarkesh Patel?

  By 2030, what percent of GDP is AI?

• 4:32Speaker 00

  Oh, gosh, hard to answer that question. Very hard to answer that question.

• 4:34Dwarkesh Patel?

  Give me an over/under.

• 4:36Speaker 00

  Likely problem is that my error bars are in low scale. So I could imagine, like, I could imagine like a huge percentage. I could imagine like disappointing a small percentage at the same time.

• 4:44Dwarkesh Patel?

  Okay, so let's take the counterfactual where it is a small percentage. Let's say it's 2030 and, you know, not that much economic value has been created by these elements. As unlikely as you think this might be, what is, what would be your best explanation right now of why something like this might happen?

• 4:58Speaker 00

  My best explanation, so I really don't think that's a likely possibility.

• 5:03Dwarkesh Patel?

  Yeah.

• 5:04Speaker 00

  So that's, that's the preface to the, to the comment. But if I were to take the premise of your question, well, like, why were things disappointing in terms of the real world impact? And my answer would be reliability. If somehow it ends up being the case that

• 5:22Speaker 00

  you really want them to be reliable and they ended up not being reliable, or if reliability now to be harder than we expect. I really don't think that will be the case, but if I had to pick one, if I had to pick one and you tell me like, hey, like, why didn't things work out? It would be reliability that you still have to look over the answers and double check everything. That's just really puts a damper on the economic value that can be produced by those systems.

• 5:47Dwarkesh Patel?

  They'll be technologically mature. It's just a question of whether they'll be reliable enough.

• 5:51Speaker 00

  Yeah, well, in some sense, not reliable means not technologically mature. You see what I mean?

• 5:55Dwarkesh Patel?

  Yeah, fair enough. What's after generative models, right? So before you're working on reinforcement learning, is this, is this basically it? Is this a paradigm that gets us to AGI or is there something after this?

• 6:05Speaker 00

  I mean, I think this paradigm is going to go really, really far and I would not underestimate it. I think it's quite likely that this exact paradigm is not going to be the quite AGI form factor. I mean, I hesitate to say precisely what the next paradigm will be. But I think it will probably involve integration of all the different ideas that came, that came in the past.

• 6:27Dwarkesh Patel?

  Is there some specific one you're referring to or?

• 6:31Speaker 00

  I mean, it's hard to be specific.

• 6:34Dwarkesh Patel?

  So you could argue that next token prediction can only help us match human performance and maybe not surpass it. What would it take to surpass human performance?

• 6:42Speaker 00

  So I challenge the claim that next token prediction cannot surpass human performance.

• 6:49Speaker 00

  It looks like on the surface it cannot. It looks on the surface if you just learn to imitate, to predict what people do, it means that you can only copy people. But there is a counter argument for why it might not be quite so if your neural net is, if your base neural net is smart enough. You just ask it like, what would a person with great insight and wisdom and capability do? Maybe such a person doesn't exist, but there's a pretty good chance that the neural net will be able to extrapolate how such a person could behave. Do you see what I mean?

• 7:25Dwarkesh Patel?

  Yes, although where would it get that sort of insight about what that person would do if not from?

• 7:29Speaker 00

  From the data of regular people, because like if you think about it, what does it mean to predict the next token well enough? What does it mean actually? It's actually, it's a much, it's a deeper question than it seems. Predicting the next token well means that you understand

• 7:46Speaker 00

  the underlying reality that led to the creation of that token. It's not statistics, like it is statistics, but what is statistics? In order to understand those statistics, to compress them, you need to understand what is it about the world that creates those statistics? And so then you say, okay, well, I have all those people. What is it about people that creates their behaviors? Well, they have, you know, they have thoughts and they have feelings and they have ideas and they do things in certain ways. All of those could be deduced from next token prediction. And I'd argue that this should make it possible, not indefinitely, but to a pretty decent degree to say, well,

• 8:30Speaker 00

  can you guess what you'd do if you took a person with like this characteristic and that characteristic? Like such a person doesn't exist. But because you're so good at predicting the next token, you should still be able to guess what that person would do, this hypothetical, imaginary person, with far greater mental ability than the rest of us.

• 8:50Dwarkesh Patel?

  When we're doing reinforcement learning on these models, how long before most of the data for the reinforcement learning is coming from AIs and not humans?

• 8:59Speaker 00

  I mean, already most of the data for reinforcement learning is coming from AIs. Yeah, well, it's like the humans are being used to train the reward function. But then the but then the reward function in its interaction with the model is automatic. And all the data that's generated during the process of reinforcement learning is created by AI. So like, if you look at the current,

• 9:23Speaker 00

  I would say, technique paradigm, which is in getting some significant attention because of chat GPT, reinforcement learning from human feedback. There is human feedback. The human feedback is being used to train the reward function. And then the reward function is being used to create the data which trains the model.

• 9:40Dwarkesh Patel?

  Got it. And is there any hope of just removing the human from the loop and have it improve itself and some sort of alpha go away?

• 9:46Speaker 00

  Yeah, definitely. I mean, I feel like in some sense, our hopes for, like, our plan, like, very much so. The thing you really want is for the human teachers that tell, that teach the AI, for them to collaborate with an AI. You might want to think about it. And you might want to think of it as being in a world where the human teachers do 1% of the world and the work, and the AI do 99% of the work. You don't want it to be 100% AI, but you do want it to be a human-machine collaboration which teaches the next machine.

• 10:18Dwarkesh Patel?

  So currently, I mean, I've had the chance to play around these models. They seem bad at multi-step reasoning, and they have been getting better. But what does it take to really surpass that barrier?

• 10:28Speaker 00

  I mean, I think dedicated training will get us there. More more improvements to the base models will get us there.

• 10:34Dwarkesh Patel?

  Okay.

• 10:34Speaker 00

  But,

• 10:38Speaker 00

  like, fundamentally, I also don't feel like they're that bad at multi-step reasoning. I actually think that they are bad at mental multi-step reasoning, but they're not allowed to think out loud. But when they are allowed to think out loud, they're quite good. And I expect this to improve significantly, both with better models and with special training.

• 10:56Dwarkesh Patel?

  Hmm. Are you running out of reasoning tokens on the internet? Are there enough of them?

• 11:01Speaker 00

  I mean, you know, it's okay. So for for context on this question, like there is there are claims that indeed at some point we'll run out of tokens in general to train those models. And yeah, I think this will happen one day and we'll by the time that happens, we need to have other ways of training models, other ways of productively improving their capabilities and sharpening their behavior, making sure they're doing exactly precisely what we want without more data.

• 11:26Dwarkesh Patel?

  Well, you haven't run out of data yet. There's more.

• 11:29Speaker 00

  Yeah, I would say I would say the data situation is still quite good. There are still lots to go. But at some point, yeah, at some point, data will run.

• 11:36Dwarkesh Patel?

  Okay, where what is the most valuable source of data? Is it Reddit, Twitter, books? What would you train many other tokens of other varieties for?

• 11:46Speaker 00

  Generally speaking, you'd like tokens which are speaking about smarter things, tokens which are like more interesting.

• 11:53Dwarkesh Patel?

  Yeah.

• 11:54Speaker 00

  So I mean, all this, all the sources which you mentioned, they are valuable.

• 11:57Dwarkesh Patel?

  Okay, so maybe not Twitter, but do we need to go multimodal to get more tokens? Or do we still have enough text tokens left?

• 12:04Speaker 00

  I mean, I think that you can still go very far in text only, but going multimodal seems like a very good direction.

• 12:10Dwarkesh Patel?

  If you're comfortable talking about this, like where is the place where we haven't scraped the tokens yet?

• 12:14Speaker 00

  Oh, I mean.

• 12:16Speaker 00

  Yeah, obviously. I mean, I can't answer that question for us, but I'm sure I'm sure that for everyone, there is a different answer to that question.

• 12:22Dwarkesh Patel?

  How many orders of magnitude improvement can we get just not from scale or not from data, but just from algorithmic improvements?

• 12:30Speaker 00

  Hard to answer, but I'm sure there is some.

• 12:32Dwarkesh Patel?

  Is some a lot or is some a little?

• 12:35Speaker 00

  I mean, it's only one way to find out.

• 12:37Dwarkesh Patel?

  Okay, let me get to your like quickfire opinions about these different research directions. Retrieval transformers. So just like somehow storing the data outside of the model itself and retrieving it somehow.

• 12:47Speaker 00

  Seems promising.

• 12:48Dwarkesh Patel?

  Would you see that as a path forward or?

• 12:50Speaker 00

  I think it seems promising.

• 12:53Dwarkesh Patel?

  Robotics, was it the right step for OpenAI to leave that behind?

• 12:57Speaker 00

  Yeah, it was. Like back then, it really wasn't possible to continue working in robotics because there was so little data. Like back then, if you wanted to do on robot, if you wanted to work on robotics, you needed to become a robotics company. You needed to really have a giant group of people working on building robots and maintaining them and having.

• 13:20Speaker 00

  And even then, like if you only, if you're gonna have a hundred robots, it's a giant operation already, but you're not gonna get that much data. So in a world where most of the progress comes from the combination of compute and data, right? That's where we've been, where it was the combination of compute and data that drove the progress. There was no path to data from robotics. So back in the day, then you made a decision to stop working in robotics. There was no path forward.

• 13:51Dwarkesh Patel?

  Is there one now?

• 13:53Speaker 00

  So I'd say that now it is possible to create a path forward. But one needs to really commit to the, to the, to the task of robotics. You really need to say, I'm going to build like many thousands, tens of thousands, hundreds of thousands of robots and somehow collect data from them and find a gradual path where the robots are doing something slightly more useful. And then the data that they get from these robots and then the data that is obtained and used to train the models and they do something that's slightly more useful. So you could imagine this kind of gradual path of improvement where you build more robots, they do more things, you collect more data and so on. But you really need to be committed to this path. If you say, I wanna make robotics happen, that's what you need to do.

• 14:35Speaker 00

  I believe that there are companies who are thinking about such, doing exactly that. But I think that you need to really love robots and need to be really willing to solve all the physical and logistical problems of dealing with them. It's not the same as software at all. So I think one could make progress in robotics today with enough motivation.

• 14:56Dwarkesh Patel?

  Uh, what ideas are you excited to try, but you can't because they don't work well on current hardware?

• 15:01Speaker 00

  I don't think current hardware is a limitation.

• 15:03Dwarkesh Patel?

  Okay.

• 15:04Speaker 00

  I think it's just not the case.

• 15:05Dwarkesh Patel?

  Got it. So, but anything you wanna try, you can just spin it up or?

• 15:08Speaker 00

  I mean, of course, like the, the thing, you, you might say, well, I wish current hardware was cheaper or maybe it had higher, like maybe, maybe it would be better if it was higher memory processor bandwidth, let's say.

• 15:23Speaker 00

  But by and large, hardware is just a limitation.

• 15:27Dwarkesh Patel?

  Let's talk about alignment. Do you think we'll ever have a mathematical definition of alignment?

• 15:34Speaker 00

  Mathematical definition, I think, is unlikely.

• 15:36Dwarkesh Patel?

  Uh huh.

• 15:38Speaker 00

  Like, I do, I do think that we will instead have multiple, like, rather than, rather than achieving one mathematical definition, I think we'll achieve multiple definitions that look at alignment from different aspects.

• 15:51Speaker 00

  And I think that this is how we will get the assurance that we want. And by which I mean, you can look at the behavior, you can look at the behavior in various tests, controls, and in various adversarial stress situations. You can look at how the neural net operates from the inside.

• 16:08Speaker 00

  I think you have to look at several of these factors at the same time.

• 16:12Dwarkesh Patel?

  And how sure do you have to be before you release a model in the wild? Is it 100%, 95%?

• 16:17Speaker 00

  Well, it depends how capable the model is. The more capable the model is, the more, the more, the higher the, the more confident you need to be.

• 16:24Dwarkesh Patel?

  Okay, so just say it's something that's almost AGI. Where is AGI?

• 16:27Speaker 00

  Well, depends what your AGI can do. Keep in mind that AGI is an ambiguous term also.

• 16:30Dwarkesh Patel?

  Yeah.

• 16:31Speaker 00

  Like, like your average college undergrads is an AGI, right?

• 16:37Dwarkesh Patel?

  It's a data, yeah.

• 16:38Speaker 00

  But, but you, you, you see what I mean. There is significant ambiguity in terms of what is meant by AGI.

• 16:43Dwarkesh Patel?

  Mm hmm.

• 16:44Speaker 00

  And so depending on where you put this mark, you need to be more or less confident.

• 16:49Dwarkesh Patel?

  Well, you mentioned a few of the paths towards alignment earlier. What, what is the one thing that's most promising at this point?

• 16:54Speaker 00

  Like, I think that it will be a combination. I really think that you will not want to have just one approach. I think we will want to have a combination of approaches where we, you spend a lot of compute adversarially probably to find any mismatch between the behavior that you wanted to teach and the behavior that it exhibits. We look inside into the neural net using another neural net to understand how it, how it operates on the inside. I think all of them will be necessary. Every approach like this reduces the probability of misalignment. And you also want to be in a world where you're

• 17:33Speaker 00

  Degree of alignment keeps of increasing faster than the capability of the models. I would say that right now, our understanding of our models is still quite rudimentary. We've made some progress, but much more progress is possible. And so I would expect that ultimately, the thing that will really succeed is when we will have a small neural net that is well understood, that's given the task to study the behavior of a large neural net that is not understood to verify.

• 17:59Dwarkesh Patel?

  By what point is most of the AI research being done by AI?

• 18:02Speaker 00

  I mean, so today, when you use Copilot, right? What fraction? How do you do the, how do you divide it up? So I expect at some point, you ask your, you know, descendant of ChatGPT, you say, hey, like, I'm thinking about this and this, can you suggest fruitful ideas I should try? And you would actually get fruitful ideas. I don't think that will make it possible for you to solve problems you couldn't solve before.

• 18:25Dwarkesh Patel?

  Got it. But it's somehow just telling the humans, giving them ideas faster or something. It's not itself interacting with the.

• 18:31Speaker 00

  One example. I mean, you could, you could slice it in a variety of ways. But I think the bottleneck there is good ideas, good insights, and that's something which the neural nets could help with.

• 18:40Dwarkesh Patel?

  If you could design some, like a billion dollar prize for some sort of alignment research result or product, what is like the concrete criteria you would set for that billion dollar prize? There's something that makes sense for such a price?

• 18:50Speaker 00

  I, it's funny that you ask this. I was actually thinking about this exact question. I haven't, I haven't come up with an exact criteria yet. Maybe something that be the benefit, like maybe a prize where

• 19:03Speaker 00

  we could say that two years later, or three or five years later, we'll look back and say like that was the main result. So rather than say that there is a prize committee that decides right away, we wait for five years and then award it retroactively.

• 19:18Dwarkesh Patel?

  But there's no concrete thing we can identify yet as like you solved this particular problem and you're, you made a lot of progress.

• 19:25Speaker 00

  I think a lot of progress, yes. I wouldn't say that this would be the.

• 19:29Speaker 00

  The full thing.

• 19:30Dwarkesh Patel?

  Do you think end to end training is the right architecture for bigger and bigger models? Or do we need better ways of just connecting things together?

• 19:39Speaker 00

  I think end to end training is very promising. I think connecting things together is very promising.

• 19:43Dwarkesh Patel?

  Everything is promising. So OpenAI is projecting revenues of a billion dollars in 2024. That might very well be correct, but I'm just curious. When you're talking about a new general purpose technology, how do you estimate how big a windfall it will be? Like, why that particular number?

• 20:00Speaker 00

  I mean, you look at the current, you look at the, you know, we've already had a, so we've had a product

• 20:07Speaker 00

  for quite a while now, back from the GPT three days, from two years ago through the API. And we've seen how it grew. We've seen how the response to DALL-E has grown as well. And so you see how the response to chat GPT is. And I think all of this gives us information that allows us to make a relatively sensible extrapolation to 2024. Maybe that would be, that'd be one answer. Like, you need to have data. You can't come up with those things out of thin air because otherwise your error bars will be like off by, your error bars are going to be like 100x in each direction.

• 20:39Dwarkesh Patel?

  I mean, but most exponentials don't stay exponential, especially when they get into bigger and bigger quantities, right? So how do you determine in this case that?

• 20:47Speaker 00

  I mean, like, would you bet against the AI?

• 20:51Dwarkesh Patel?

  Not after talking with you. Let's talk about what like a post-AI future looks like. Are people like you, you know, I'm guessing you're working like 80 hour weeks towards this grand goal that you're really obsessed with? Are you going to be satisfied in a world where you're basically living in an AI retirement home? Or like, what is the word? What is your, what are you concretely doing after AGI comes?

• 21:11Speaker 00

  I think the question of what, what I'll be doing or what people will be doing after AGI comes is a very tricky question. You know, I think where, where will people find meaning? But I think, I think that that's something that AI could help us with.

• 21:27Speaker 00

  One thing I imagine is that we'll all be able to become more enlightened because we interact with an AGI that will help us see the world more correctly, become better on the inside as a result of interacting. Like, imagine talking to the best meditation teacher in history. I think that will be a helpful thing. But I also think that because the world will change a lot, it will be very hard for people to understand what is happening precisely and how to really contribute. One thing that I think some people will choose to do is to become part of AI in order to really expand their minds and understanding and to really be able to solve the hardest problems that society will face then.

• 22:10Dwarkesh Patel?

  Are you gonna become part of AI?

• 22:12Speaker 00

  Very tempting. It is tempting, yeah.

• 22:14Dwarkesh Patel?

  Well, do you think there'll be physically embodied humans in 3000?

• 22:18Speaker 00

  3000, oh. How do I know what's gonna happen in 3000?

• 22:21Dwarkesh Patel?

  Like, what does it look like? Are there still like humans walking around on Earth? Or have you guys thought concretely about what you actually want this world to look like?

• 22:27Speaker 00

  3000. Well, I mean, the thing is, here's the thing. Let me describe to you what I think is not quite right about the question. Like, it implies like, oh, like, we get to decide how we want the world to look like. I don't think that picture is correct. I think change is the only constant. And so, of course, even after AGI is built, it doesn't mean that the world will be static. The world will continue to change. The world will continue to evolve. And it will go through all kinds of transformations. And I really have no, I don't think anyone has any idea of how the world will look like in 3000. But I do hope that there will be a lot of descendants of human beings who will live happy, fulfilled lives where they're free to do as their wish, as they see fit,

• 23:12Speaker 00

  where they are the ones who are solving their own problems. Like, one of the things which I would not want, one world which I would find very unexciting is one where, you know, we build this powerful tool and then the government said, okay. So the AGI said that society should be run in such a way and now we shall run society in such a way. I'd much rather... Have a world where people are still free to make their own mistakes and suffer their consequences and gradually evolve morally and progress forward on their own, through their own strengths. See what I mean? Be the AGI providing more like a base safety net.

• 23:46Dwarkesh Patel?

  How much time do you spend thinking about these kinds of things versus just doing the research? But.

• 23:50Speaker 00

  I do think about those things a fair bit, yeah. I think those are very interesting questions.

• 23:54Dwarkesh Patel?

  So in what ways have the capabilities we have today? In what ways have they surpassed where you expected them to be in 2015? And in what ways are they still not where you'd have expected them to be by this point?

• 24:05Speaker 00

  I mean, in fairness, they did surpass what I expected them to be in 2015. In 2015, I, my thinking was a lot more, I just don't want to bet against deep learning. I want to make the biggest possible bet on deep learning. Don't know how, but it will figure it out.

• 24:19Dwarkesh Patel?

  But is there any specific way in which it's been more than you expected or less than you expected? Like some concrete prediction you had in 2015 that's been frowned?

• 24:29Speaker 00

  You know, unfortunately, I don't remember concrete predictions I made in 2015. But I definitely, but I definitely think that overall in 2015, I just wanted to, to, to move, to make the biggest bet possible on deep learning. But I didn't know exactly. I didn't have a specific idea of how far things will go in seven years. Well, I mean, 2015, I did have all these bets with people in 2016, maybe 2017, that things will go really far. But specifics. So it's like, it's both, it's both the case that it surprised me and I was making these aggressive predictions. But I think maybe I believe them only, only, only 50% on the inside.

• 25:09Dwarkesh Patel?

  Uh huh. Well, what do you believe now that even most people at OpenAI would find farfetched?

• 25:14Speaker 00

  I mean, I think that at this, because we communicate a lot at OpenAI, people have a pretty good sense of what I think. And so, yeah, we've reached the point at OpenAI where I think we see eye to eye on all these questions.

• 25:25Dwarkesh Patel?

  So Google has, you know, its custom TPU hardware. It has all this data from all its users, you know, Gmail and so on. Does it give it an advantage in terms of training bigger models and better models than you?

• 25:36Speaker 00

  So I think like when the first, at first when the TPU came out, I was really impressed and I thought, wow, this is amazing. But that's because I didn't quite understand hardware back then. What really turned out to be the case is that TPUs and GPUs are almost the same thing. They are very, very similar. It's like. I think a GPU chip is a little bit bigger. I think a TPU chip is a little bit smaller. It may be a little bit cheaper. But then they make more GPUs than TPUs. So I think the GPUs might be cheaper after all. But fundamentally, you have a big processor. And you have a lot of memory. And there is a bottleneck between those two. And the problem that both the TPU and the GPU are trying to solve

• 26:19Speaker 00

  is that by the amount of time it takes you to move one floating point from the memory to the processor, you can do several hundred floating point operations on the processor, which means that you have to do some kind of batch processing. And in this sense, both of these architectures are the same. So I really feel like hardware, like in some sense, the only thing that matters about hardware is cost, cost per flop, overall systems cost.

• 26:43Dwarkesh Patel?

  Okay, there is that, there isn't that much difference.

• 26:45Speaker 00

  Well, I actually don't know. I mean, I don't know how much, what the TPU costs are, but I would suspect that probably not. If anything, probably the views are more expensive because there is less of them.

• 26:56Dwarkesh Patel?

  When you're doing your work, how much of the time is spent, you know, configuring the right initializations, making sure the training run goes well and getting the right hyperparameters? And how much is it just coming up with whole new ideas?

• 27:07Speaker 00

  I would say it's a combination, but I think that coming up is, it's a combination. But coming up with whole new ideas is actually not. It's like a modest part of the work. Certainly, coming up with new ideas is important. But I think even more important is to understand the results, to understand the existing ideas, to understand what's going on. Because normally you have this, you know, a neural net is a very complicated system, right? And you ran it and you get some behavior which is hard to understand what's going. Understanding the results, figuring out what next experiment to run. A lot of the time is spent on that. Understanding what could be wrong, what could have caused the the neural net produce a result which was not expected. I'd say a lot of time is spent as well, of course, coming up with new ideas, but not new ideas.

• 27:52Speaker 00

  I think like, I don't, I don't like this framing as much. It's not that it's false, but I think the main activity is actually understanding.

• 28:03Dwarkesh Patel?

  What do you see is the difference between the two?

• 28:05Speaker 00

  So at least in my mind, when you say come up with new ideas, I'm like, oh, like what happened if it did such and such? Whereas understanding it's more like, like, what is this whole thing? Like, what are the real underlying? phenomena that are going on? What are the, what are the underlying effects? Like, why? Why are we doing things this way and not another way? And of course, this is very adjacent to what can be described as coming up with ideas. But I think the understanding part is where the real action takes place.

• 28:33Dwarkesh Patel?

  Does that describe your entire career? Like if you think back on like ImageNet or something, was that more new idea or was that more understanding?

• 28:38Speaker 00

  Oh, I was definitely understanding, definitely understanding. It was a new understanding of very old things.

• 28:44Dwarkesh Patel?

  What is the experience of training on Azure been like using Azure?

• 28:49Speaker 00

  Fantastic. I mean, yeah, I mean, Microsoft has been a very, very good partner for us and they've really helped take Azure and make it, bring it to a point where it's really good for ML.

• 29:04Speaker 00

  And you're super happy with it.

• 29:06Dwarkesh Patel?

  How, how vulnerable is the whole AI ecosystem to something that might happen in Taiwan? So let's say there's like a tsunami in Taiwan or something. What, what, what happens to AI in general?

• 29:15Speaker 00

  Like, it's, it's definitely going to be a significant setback. It's not going to, like, it might be something equivalent to like no one will be able to get more, more compute for a few years. But I expect compute will spring up. Like, for example, I believe that Intel has fabs just of the previous year of like a few generations ago. So that means that if Intel wanted to, they could produce something GPU like from like four years ago. So yeah, it's not the best, let's say. I'm actually not sure about if, if, if my statement about Intel is correct, but I do know that there are fabs outside of Taiwan that are just not as good. But you can still use them and still go very far with them. It's just, it just costs, it's just a setback.

• 29:57Dwarkesh Patel?

  Will inference get cost prohibitive as these models get bigger and bigger?

• 30:00Speaker 00

  So I have a different way of looking at this question. It's not that inference will become cost prohibitive. Inference of better models will indeed become more expensive. But is it prohibitive? Well, it depends on how useful is it? Like if it is more useful than it is expensive, then it is not prohibitive. Like to give you an analogy, like suppose you want to talk to a lawyer, you have some case you or need some advice or something. You are perfectly happy to spend five hundred dollars an hour.

• 30:29Speaker 00

  Right? So if your neural net could give you like really reliable legal advice, you'd say, I'm happy to spend $400 for that advice. And suddenly inference becomes very much non-prohibitive.

• 30:41Speaker 00

  The question is, can neural net produce an answer good enough at this cost?

• 30:48Dwarkesh Patel?

  Yes, and you will just have like price discrimination, different, different models.

• 30:53Speaker 00

  I mean, it's already the case today. So on our product, the API is sort of multiple neural nets of different sizes. And different customers use different neural nets of different sizes depending on their use case. Like if someone can take a small model and fine tune it and get something that's satisfactory for them, they'll use that.

• 31:13Dwarkesh Patel?

  Yeah.

• 31:13Speaker 00

  But if someone wants to do something more complicated and more interesting, they'll use the biggest model.

• 31:17Dwarkesh Patel?

  How do you prevent these models from just becoming commodities where these different companies just, they just bid each other's prices down until it's basically the cost of the GPU run?

• 31:25Speaker 00

  Yeah, I think, I think there is without question a force that's trying to create that. And the answer is you got to keep on making progress. You got to keep improving the models. You got to keep on coming up with new ideas and making our models better and more reliable, more trustworthy, so you can trust their answers. All those things.

• 31:43Dwarkesh Patel?

  Yeah, but let's say it's like 2025 and the model from 2024, somebody just offering it a cost. And it's like so pretty good. Why, why would people use a new one from 2025 if the one from just a year older is, you know, even better?

• 31:56Speaker 00

  So there are several answers there. For some use cases, that may be true. There will be a new model from 2025, which will be driving the more interesting use cases. There's also going to be a question of inference cost. Like you can, if you can do research to serve the same model at less cost. So there will be different, the same model will be served, will, will cost different, some different amounts to serve for different companies. I can also imagine some degree of specialization too, where some companies may try to specialize in some area and be stronger in an error area compared to other companies. And I think that's who may. That may be a response to commoditization to some degree.

• 32:34Dwarkesh Patel?

  As over time, do these different companies, do their research directions converge or they diverge? Are they doing similar and similar things over time or are they doing, are they going off, branching off into different areas?

• 32:43Speaker 00

  So I'd say in the near term, it looks like there is convergence in the like. I expect it's going to be a convergence, a divergence convergence behavior. Where there is a lot of convergence on the near term work. There's going to be some divergence on the longer term work. But then, once the longer term work starts to yield fruit, I think there will be convergence again.

• 33:03Dwarkesh Patel?

  Got it. When one of them finds the most promising area, they everybody just.

• 33:07Speaker 00

  That's right. Now, there is obviously less publishing now, so it will take longer before this promising direction gets rediscovered. But that's how I'd imagine it. I think it's going to be convergence to divergence convergence.

• 33:18Dwarkesh Patel?

  Yeah, we talked about this a little bit at the beginning, but you know, as foreign governments learn about how capable these models are. How do you, are you worried about spies or some sort of attack? To get your weights or, you know, somehow abuse these models and learn about them?

• 33:34Speaker 00

  Yeah, it's definitely something that you absolutely can't discount that.

• 33:39Dwarkesh Patel?

  Yeah.

• 33:40Speaker 00

  And yeah, something that we try to guard against the best of our ability, but it's going to be a problem for everyone who's building this.

• 33:48Dwarkesh Patel?

  How do you prevent your weights from leaking?

• 33:51Speaker 00

  I mean, you have like really good security people.

• 33:54Dwarkesh Patel?

  And like how many people have the, if they wanted to just like assess agent to the weights? A machine, how many people could do that?

• 34:00Speaker 00

  I mean, like what I can say is that the security people that we have, they've built, they've done a really good job so that. I'm really not worried about the way it's being leaked.

• 34:11Dwarkesh Patel?

  Okay, got it. What kinds of emergent properties are you expecting from these models at this scale? Is there something that just comes about de novo?

• 34:19Speaker 00

  I'm sure things will come, I'm sure really new surprising properties will come up. I would not be surprised. The thing which I'm really excited about or the thing which I'd like to see is reliability and controllability. I think that this will be very, very important class of emergent properties. If you have reliability and controllability, I think that helps you solve a lot of problems. Reliability means you can trust the model's output. Controllability means you can control it.

• 34:43Speaker 00

  And we'll see, but it would be very cool if those emergent properties did exist.

• 34:47Dwarkesh Patel?

  Is there somewhere you can predict it in advance? Like what will happen in this parameter count, what will happen in that parameter count?

• 34:52Speaker 00

  I think it's possible to make some predictions about specific capabilities, though it's definitely not simple and you can't do it in a super fine-grained way, at least today. But I think getting better at that is really important and anyone who is interested in, who has research ideas on how to do that, I think that can be a valuable contribution.

• 35:11Dwarkesh Patel?

  How seriously do you take these scaling laws? If like there's a paper that says like, oh, you just increase, you need this many orders of magnitude more to get all the reasoning out, like do you take that seriously or do you think it breaks down at some point?

• 35:23Speaker 00

  Well, the thing is that the scaling law tells you what happens as you, what happens to your look to your next word prediction accuracy, right? There is a whole separate challenge of linking next word prediction accuracy to reasoning capability.

• 35:40Speaker 00

  I do believe that indeed there is a link, but this link is complicated. And we may find that there are other things that can give us more reasoning per unit effort.

• 35:52Speaker 00

  Like, for example, some special, like, you know, you mentioned reasoning tokens and I think they can be helpful.

• 35:59Speaker 00

  There can be, there can be probably some things.

• 36:03Dwarkesh Patel?

  Uh, have you can, is this, is something you're considering just hiring humans to generate tokens for you or is it all going to come from stuff that already exists out there?

• 36:11Speaker 00

  I mean, I think that relying on people to teach our models to do things, especially, you know, to make sure that they are well behaved and they don't produce false things, I think is an extremely sensible thing to do and.

• 36:23Dwarkesh Patel?

  Isn't it odd that we have the data we need at exactly the same time as we have the transformer at the exact same time that we have these GPUs? Like, is it odd to you that all these things happen at the same time or do you not see it that way?

• 36:34Speaker 00

  I mean, it is, it is definitely an interesting, it is an interesting situation that is the case. I will say that it is odd and it is less odd on some level. Here is why it's less odd. So what is the driving force behind the fact that the data exists, that the GPUs exists, that the transformer exists? So. As the data exists because computers became better and cheaper, you've got smaller and smaller transistors. And suddenly, at some point, it became economical for every person to have a personal computer. Once everyone has a personal computer, you really want to connect them with a network. You get the internet. Once you have the internet, you have suddenly data appearing in great quantities. The GPUs were improving concurrently because you have more smaller and smaller transistors

• 37:18Speaker 00

  and you're looking for things to do with them. Gaming turned out to be a thing that you could do. And then, at some point, the gaming GPU, Nvidia said, wait a second. Try and maybe turn it into a general-purpose GPU computer. Maybe someone will find it useful. Turns out, it's good for neural nets. So it could have been the case that maybe the GPU would have arrived five years later or 10 years later. If, let's suppose, gaming wasn't a thing. It's kind of hard to imagine. What does it mean if gaming isn't a thing? But it could. Maybe there was a counterfactual world where GPUs arrived five years after the data or five years before the data. In which case, maybe things would move a little bit more.

• 38:02Speaker 00

  Things would have been as ready to go as they are now. But that's the picture which I imagine. All this progress in all these dimensions is very intertwined. It's not a coincidence that,

• 38:15Speaker 00

  like, you don't get to pick and choose which dimension, in which dimensions things improve, if you see what I mean.

• 38:20Dwarkesh Patel?

  How inevitable is this kind of progress? So if, like, let's say you and Geoffrey Hinton and a few other pioneers, if they were never born, does the deep learning revolution happen around the same time? How much does it delay?

• 38:31Speaker 00

  I think maybe there would have been some delay, maybe like a year delay. It's really hard to tell. It's really hard to tell. I mean, I hesitate to give a longer answer because, okay, well, then you'd have, GPUs would keep on improving, right? Then, at some point,

• 38:46Speaker 00

  I cannot see how someone would not have discovered it. Because here's the other thing. Okay, so let's suppose no one has done it. Computers keep getting faster and better. It becomes easy and easy to train these neural nets because you have bigger GPUs. So it takes less engineering effort to train one. You don't need to optimize your code as much. You know, when the ImageNet dataset came out, it was huge and it was very, very difficult to use. Now imagine you wait for a few years and it becomes very easy to download and people can just tinker. So I would imagine that, like, a modest number of years maximum. This would be my guess. I hesitate. I hesitate to give a longer answer, though.

• 39:26Speaker 00

  You know, you can't, you can't run, you can't rerun the world you don't know.

• 39:31Dwarkesh Patel?

  Let's go back to alignment for a second. As somebody who deeply understands these models, what is your intuition of how hard alignment will be?

• 39:38Speaker 00

  Like, I think with the, so here's what I would say. I think with the current level of capabilities, I think we have a pretty good set of ideas of how to align them. But I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions.

• 39:56Speaker 00

  Like, I think, I think it's something to,

• 39:59Speaker 00

  to think, to think about a lot and to research. I think this is one area also, by the way. You know, like oftentimes academic researchers asked me, asked me where, what, what's the best place where they can contribute? And I think alignment research is one place where I think academic researchers can make very meaningful contributions.

• 40:15Dwarkesh Patel?

  Other than that, do you think academia will come up with more insights about actual capabilities? Or is that going to be just the companies at this point?

• 40:21Speaker 00

  The companies will realize the capabilities. I think it's very possible for academic research to come up with those insights. I think it's just, it doesn't seem to happen that much for some reason. But I don't, I don't think there's anything

• 40:34Speaker 00

  fundamental about academia. Like, it's not like academia can't. I think maybe they're just not thinking about the right problems or something because maybe it's just easier to see what needs to be done inside these companies.

• 40:48Dwarkesh Patel?

  Hmm, I see. But there's a possibility that somebody could just realize.

• 40:51Speaker 00

  Yeah, I totally think so. Like, why, why would I possibly rule this out? You see what I mean?

• 40:56Dwarkesh Patel?

  What are the concrete steps by which these language models start actually impacting the world of atoms and not just the world of bits?

• 41:04Speaker 00

  Well, you see, I don't think that there is a distinction, a clean distinction between the world of bits and the world of atoms. Suppose the neural net tells you that, hey, like here is like something that you should do and it's going to improve your life, but you need to rearrange your apartment in a certain way. And you go and you rearrange your apartment as a result. Did the neural net impact the world of atoms just?

• 41:26Dwarkesh Patel?

  Yeah, fair enough, fair enough. Do you think it'll take a couple of additional breakthroughs as important as the transformer? They get to superhuman AI? Or do you think we basically got the insights in the books somewhere and we just need to implement them and connect them?

• 41:39Speaker 00

  So I don't really see such a big distinction between those two cases. And let me explain why. Like, I think what's, what one of the ways in which progress has taken place in the past. Is that we've understood that something had a property,

• 41:57Speaker 00

  a desirable property all along, but you didn't realize. So is that a breakthrough? You can say, yes, it is. Is that an implementation of something on the books? Also, yes. So I, my, my feeling is that a few of those are quite likely to happen. But that in hindsight, it will not feel like a breakthrough. Everybody is going to say, oh, well, of course, like it's totally obvious that such and such thing can, can work. You see, with the transformer, the reason it's been brought up as a big, as a specific advance is because it's the kind of thing that was not obvious for almost anyone. So people can say, yeah, like it's not something we actually knew about. But if an advance comes from something like, let's consider that the, the most fundamental advance of deep learning, that a big neural network trained with back propagation can do a lot of things.

• 42:41Speaker 00

  Like, where is the novelty? It's not in the neural network. It's not in the back propagation.

• 42:48Speaker 00

  But then somehow it's the kind of, but it was, it is most definitely a giant conceptual breakthrough because for the longest time, people just didn't see that. But then now that everyone sees it, everyone's going to say, well, of course, like it's totally obvious. Big neural network. Everyone knows that they can do it.

• 43:02Dwarkesh Patel?

  What is your opinion of your former advisor's new forward forward algorithm?

• 43:07Speaker 00

  I think that it's an attempt to train a neural network without back propagation. And I think that this is especially interesting if you are motivated to try to understand how the brain might be learning its connections. The reason for that is that as far as I know, neuroscientists are really convinced that the brain cannot implement back propagation. Because the signals in the synapses only move in one direction. And so if you have a neuroscience motivation and you want to say, okay,

• 43:42Speaker 00

  how can I come up with something that tries to approximate, approximate the good properties of back propagation without doing back propagation? That's what the forward forward algorithm is trying to do. But if you are trying to just engineer a good system, there is no reason to not use back propagation. Like.

• 44:02Speaker 00

  It's the only algorithm.

• 44:05Dwarkesh Patel?

  Hmm. I guess I've heard you in different contexts talk about the need like using humans as the, you know, the existing example case that, you know, AGI exists, right? So at what point do you take the metaphor less seriously and feel, don't feel the need to pursue it in terms of the research? Because it is important to you as a sort of existence case.

• 44:25Speaker 00

  Like, at what point do I stop caring about humans as an existence case of intelligence?

• 44:30Dwarkesh Patel?

  Or as the sort of, as an example of the model you want to follow in terms of pursuing intelligence in models?

• 44:37Speaker 00

  I see. I mean, like, you gotta, I think it's good to be inspired by humans. I think it's good to be inspired by the brain. I think there is an art into being inspired by humans and the brain correctly. Because it's very easy to latch on to a non-essential quality of humans or of the brain. And I think many people who, many people whose research is trying to be inspired by humans and by the brain, often gets a little bit specific. People get a little bit too, okay, so like, what cognitive science model should we follow? At the same time, consider the idea of the neural network itself, the idea of the artificial neuron. This too is inspired by the brain, but it turned out to be extremely fruitful. So how do we do this? What behaviors of human beings are essential?

• 45:21Speaker 00

  That you say like, this is something that proves to us that it's possible. What is in essential? No, actually, this is like some emergent phenomenon of something more basic. And we just need to focus on our, on getting our own basics right. I would say, I would say that it's like,

• 45:42Speaker 00

  I think one should, one can and should be inspired by human intelligence with care.

• 45:48Dwarkesh Patel?

  Final question. Why is there, in your case, such a strong correlation between being first to the deep learning revolution and still being one of the top researchers? You would think that these two things wouldn't be that correlated, but why is that the correlation?

• 46:00Speaker 00

  I don't think those things are super correlated indeed. I feel like in my case,

• 46:05Speaker 00

  I mean, honestly, it's hard to answer the question. You know, I just kept on.

• 46:10Speaker 00

  I kept trying really hard and it turned out to have sufficed thus far.

• 46:14Dwarkesh Patel?

  Got it. So it's a perseverance.

• 46:16Speaker 00

  I think it's a necessary but not a sufficient condition. Like, you know, many things need to come together in order to really figure something out.

• 46:25Speaker 00

  Like, you need to really go for it and also need to have the right way of looking at things. And so it's hard. It's hard to give him like a really meaningful answer to this question.

• 46:35Dwarkesh Patel?

  All right. Ilya, it has been a true pleasure. Thank you so much for coming on the Lunar Society. I appreciate you bringing us to the offices. Thank you.

• 46:43Speaker 00

  Yeah, I really enjoyed it. Thank you very much.

• 46:46Dwarkesh Patel?

  Hey, everybody. I hope you enjoyed that episode. Just wanted to let you know that in order to help pay for the bills associated with this podcast, I'm turning on paid subscriptions on my Substack at warkeshpatel.com. No important content on this podcast will ever be paywalled. So please don't donate if you have to think twice before buying a cup of coffee. But if you have the means and you've enjoyed this podcast or gotten some kind of value out of it, I would really appreciate your support. As always, the most helpful thing you can do is to share the podcast. Send it to people you think might enjoy it, put it in Twitter, your group chats, et cetera. Just blitz the world. Appreciate your listening. I'll see you next time. Cheers.