Dwarkesh Patel: AI Continuous Improvement, Intelligence Explosion, Memory, Frontier Lab Competition

Dwarkesh Patel is the host of the Dwarkesh Podcast. He joins Big Technology Podcast to discuss the frontiers of AI research, sharing why his timeline for AGI is a bit longer than the most enthusiastic researchers. Tune in for a candid discussion of the limitations of current methods, why continuous AI improvement might help the technology reach AGI, and what an intelligence explosion looks like. We also cover the race between AI labs, the dangers of AI deception, and AI sycophancy. Tune in for a deep discussion about the state of artificial intelligence, and where it’s going. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Want a discount for Big Technology on Substack? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Questions? Feedback? Write to: [email protected] Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  This episode is brought to you by Genesis. What does it actually take to put Agentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, head of enterprise business solutions at Voya Financial, to go inside a large-scale Agentic CX transformation, including the business decisions, trade-offs, and lessons along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay?

• 0:43Alex Kantrowitz?

  Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation.

• 1:03Alex Kantrowitz?

  Why do we have such vastly different perspectives on what's next for AI? If we're all looking at the same data, and what's actually going to happen next? Let's talk about it with Dwarkesh Patel, one of the leading voices on AI, who's here with us in studio to cover it all. Dwarkesh, great to see you. Welcome back to the show.

• 1:21Speaker 01

  Thanks for having me, man.

• 1:22Alex Kantrowitz?

  Thanks for being here. I was listening to our last episode, which we recorded last year, and we were anticipating what was going to happen with GPT-5, still no GPT-5.

• 1:31Speaker 01

  That's right. Oh, yeah. That would have surprised me a year ago.

• 1:34Alex Kantrowitz?

  Definitely. And another thing that would have surprised me is we were saying that we were at a moment where we were going to figure out, basically, what's going to happen with AI progress, whether the traditional method of training LLMs was going to hit a wall, or whether it wasn't. We were going to find out. We were basically months away from knowing the answer to that. Here we are a year later. We have, everybody's looking at the same data, like I mentioned in the intro. We have no idea. There are people who are saying AGI, artificial general intelligence, or human level intelligence is imminent with the methods that are available today. And there are others that are saying 20, 30, maybe longer, maybe more than 30 years until we reach it. So let me start by asking you this. If we're all looking at the same data,

• 2:18Alex Kantrowitz?

  why are there such vastly different perspectives on where this goes?

• 2:21Speaker 01

  Hmm. I think people have different philosophies around what intelligence is. That's part of it. I think some people think that these models are just basically baby AGIs already. And they just need a couple additional little unhopped legs, a little sprinkle on top. Things like test time, um, uh, thinking. So we already got that with O1 and O3 now, uh, where they are allowed to think. They're not just do it like saying the first thing that comes to mind. And a couple other things like, oh, well, they should be able to use your computer and have access to all the tools that you have access to when you're doing your work. And they need context in your work. They need to be able to read your Slack and everything. So that was one perspective. Um, my perspective is slightly different from that. I don't think we're just right around their corner from AGI and it's just

• 3:03Speaker 01

  a little additional dash of something. That's all it's going to take. I think, you know, people often ask if all AI progress stopped right now, and all you could do is collect more data or deploy these models in more situations, how much further could these models go? And I, my perspective is that you actually do need more algorithmic progress. I think, uh, I think a big bottleneck these models have is their inability to learn on the job, to have continual learning. Their entire memory is extinguished at the end of a session. Um, there's a bunch of reasons why I think this actually makes it really hard to get human-like labor out of them. And so sometimes people say, well, the reason Fortune 500 isn't using LLMs all over the place is because they're too stodgy. They're

• 3:44Speaker 01

  not, um, they're not like, they're not thinking creatively about how AI can be implemented. And I should, I, I don't think that's the case. I think it actually is genuinely hard to use these AIs to automate a bunch of labor.

• 3:54Alex Kantrowitz?

  Okay, so you've said a couple of interesting things. First of all, that we have the AIs that can think right now, like O3 from OpenAI. We're going to come back to that in a moment, but I think we should really seize on this idea that you're bringing up, that it's not laziness within Fortune 500 companies that's causing them to not adopt these models. Or I would say they're all experimenting with it, but we all, you know that the rate to get proof of concepts out the door is pretty small. One out of every five actually gets shipped into production and often it's a scaled down version of that. So what you're saying is interesting. You're saying it's not their fault. It's that these models are not reliable enough to do what they need to do because they don't learn on the job. Am I getting that right?

• 4:37Speaker 01

  Yeah, and it's not even about reliability. It's just, um, they just can't do it. So if you think about what makes humans valuable, it's not their raw intelligence, right? Any person who goes onto their job the first day, even their first couple of months, maybe, they're just not going to be that useful because they don't have a lot of context. What makes human employees useful is their ability to build up this context, to interrogate their failures, to build up these small improvements and efficiencies as they practice a task. And these models just can't do that, right? You're stuck with the abilities that you get out of the box. And they are quite smart, so you will get five out of 10 on a lot of different tasks. They'll, often they'll, on any random task, they'll probably, it might be better than an average human. It's just that they won't get any better.

• 5:24Speaker 01

  I, for my own podcast, I have a bunch of little scripts that I've tried to write with LLMs, where I'll get them to rewrite parts of script scripts to make them more, turn auto-generated transcripts into like human written like transcripts or to help me identify clips that I can tweet out. So these are things which are just like short horizon, language in, language out tasks, right? This is the kind of thing that the LLM should be. Just amazing at because it's a debt center in their of what should be in their repertoire. And they're okay at it, but the fundamental problem is that you can't like, you can't teach them how to get better in the way that if a human employee did something, you'd say, I didn't like that. I would prefer it this way instead. And then they're also looking at your YouTube studio analytics and thinking about what they can change.

• 6:05Speaker 01

  This level of um, understanding or development is just not possible with these models. And so you just don't get this continual learning, which is the source of, you know, so much of the value that human labor brings.

• 6:18Alex Kantrowitz?

  Now, I hate to ask you to argue against yourself, but you are speaking all the time. And I think we're in conversation here on this show all the time with people who believe that if the models just get a little bit better, then it will solve that problem. So why are they so convinced that the issue that you're bringing up is not a big stumbling block for these AIs?

• 6:40Speaker 01

  I think they have a sense that one, you can make these models better by giving them a different prompt. So they have the sense that even though they don't learn skills in the way humans learn, you've been, you've been writing and podcasting, you've gotten better at those things just by practicing and trying different things. Uh, and seeing how it's received by the world. And they think, well, you can sort of artificially get that process going by just adding to the system prompt. This is just like the language you put into the model at the beginning, say like, write it like this, don't write it like that. The reason I disagree with that perspective is imagine you had to teach a kid how to play the saxophone, but you couldn't just have, you know, how does a kid learn saxophone now? She tries to blow into one and then she hears how it sounds. She practices a bunch.

• 7:24Speaker 01

  Imagine if this is the way it worked instead. A kid tries to just like, never seen a saxophone, they try to play the saxophone and it doesn't sound good. So you just send them out of the room. Next kid comes in and you just like write a bunch of instructions about why the last kid messed up. Um, and then they're supposed to like play Charlie Parker Cold by um, like reading the set of instructions to play. It's just like you would, they wouldn't learn how to play the saxophone that way, right? Like you actually need to practice. So anyways, this is, this is all to say that I don't think prompting alone is that powerful a mechanism of teaching models these capabilities. Another thing is people say you can do RL. So this is where

• 8:00Alex Kantrowitz?

  Reinforcement learning.

• 8:01Speaker 01

  That's right. These models have gotten really good at coding and math because of you can you have verifiable problems in that domain where they can practice on them.

• 8:07Alex Kantrowitz?

  Can we take a moment to explain that for those who are new to this? So let me see if I get it right. Reinforcement learning, basically you give a bot a goal or you give an AI system a goal saying solve this equation and then you have the answer and you effectively don't tell it anything in between. So it can try every different solution known to humankind until it gets it. And that's the way it starts to learn and develop these skills.

• 8:29Speaker 01

  Yeah. It is more human like, right? It's more human like than just reading every single thing on the internet and then learning skills. I still think I'm like, I'm not confident that this will generalize to domains that are not so verifiable or text based. Yeah, I mean, like a lot of domain, it just like would be very hard to set up this environment and loss function for how to become a better podcaster. And, you know, whatever, people might not think podcasting is like the crux of the economy, which is fair.

• 9:00Alex Kantrowitz?

  The new AGI test.

• 9:03Speaker 01

  But like a lot of tasks are just like much softer and there's not a like an objective RL loop. And so it does require this human organic ability to learn on the job. And the reason I don't think that's around the corner is just because there's not, there's no obvious way, at least as far as I can tell, is just slot in this online learning into the models as they exist right now.

• 9:26Alex Kantrowitz?

  Okay, so I'm trying to take in what you're saying, and it's interesting. You're talking about reinforcement learning as a method that's applied on top of modern day large language models and system prompts. And maybe you'd include fine tuning in this example.

• 9:40Speaker 01

  Yeah.

• 9:40Alex Kantrowitz?

  But you don't mention that you can just make these models better by making them bigger. This so-called, you know, scaling hypothesis. So have you ruled out the fact that they can get better through the next generation of scale?

• 9:53Speaker 01

  Well, this goes back to your original question about what has, have you learned? I mean, it's quite interesting, right? I guess I did say a year ago that we should know within the next few months which trajectory we're on. And I, I feel at this point that

• 10:07Speaker 01

  We haven't gotten verification. I mean, it's narrowed, but it hasn't been as decisive as I was expecting. I was expecting like GPT five will come out and we will know, did it work or not? And to the extent that you want to use that test from a year ago, I do, I do think you would have to say like, look, pre-training, which is this idea that you just make the model bigger, that has had diminishing returns. So we have had models like GPT four point five, which um, there's various estimates, but uh, or Grok uh, uh, was it Grok two or Grok three, the new one?

• 10:39Alex Kantrowitz?

  I, I've lost count with Grok.

• 10:41Speaker 01

  That's right. Um, regardless, I think they're estimated to be, you know, 10x bigger than GPT four. And they're like, they're not obviously better. Um, so it seems like there's plateauing returns to pre-training scaling. Uh, and now we do have this RL, uh, so oh, one, oh, three. These models, the way they've gotten better is that they're practicing on these problems, as you mentioned, and they are really smart. Um, the question will be how much that procedure will be helpful in making them smarter outside of domains like math and code, and of solving what I think are very fundamental bottlenecks like continual learning or online learning. Um, there's also the computer use stuff, which is a separate topic. Uh, but I would say I am more, I have longer timelines than I did a year ago.

• 11:25Speaker 01

  Now, that is still to say, I'm expecting a fifty fifty, if I had to like make a guess, I had to make a bet. I'd just say twenty thirty two, we have like real AGI that's doing continual learning and everything. So even though I'm putting up the pessimistic facade right now, I think people should know that this pessimism is like me saying in seven years, the world will be so wildly different that you like really just can't imagine it. And seven years is not that long a period of time. So I just want to make that disclaimer, but yeah.

• 11:51Alex Kantrowitz?

  Okay. So I don't want to spend this entire conversation about scaling up models because we've done enough of that.

• 11:55Speaker 01

  That's right.

• 11:56Alex Kantrowitz?

  On the show, and I'm sure you've done a lot of it. Um, but it's interesting. You use the term plateauing returns, which is different than diminishing returns, right? So is your sense, because we've seen, for instance, Elon Musk do this project Memphis where he's put basically every GPU he can get a hold of. And he can get a lot because he's the richest private citizen in the world together. And I don't know about you, but like I said, again, I haven't paid so much attention to Grok because it doesn't seem noticeably better, even though it's using uh, that much more size. Now there is algorithmic uh, efficiency that they may not have, that someone like, it's not like a company like OpenAI might have.

• 12:32Speaker 01

  Right.

• 12:32Alex Kantrowitz?

  Um, but, but.

• 12:35Alex Kantrowitz?

  I'll just ask you the question I've asked others that have come on the show.

• 12:39Alex Kantrowitz?

  Is this sort of the end of that scaling moment? And if it is, what does it mean for AI?

• 12:45Speaker 01

  I mean, I don't think it's the end of scaling. Like, I do think companies will continue to pour exponentially more compute into training these systems, and they'll continue to do it over the next many years. Because even if there's diminishing returns, the value of intelligence is so high that it's still worth it, right? So if it costs $100 billion, even a trillion dollars to build AGI, it is it is just definitely worth it. It doesn't mean that it might take longer. Now, here is an additional wrinkle. By 2028 or so, definitely by 2030. Right now, we're scaling up the training of Frontier Systems 4X a year approximately. So every year, the biggest system is 4X bigger than, not just bigger, I shouldn't say bigger, uses more compute than the system the year before.

• 13:31Speaker 01

  If you look at things like how much energy is there in the country? How much, how many chips can TSMC produce? And what fraction of them are already being used by AI? Even if you look at like raw GDP, like how much money does the world have? How much wealth does the world have? By all those metrics, it seems like this pace of 4X a year, which is like 160X in four years, right? Like, this cannot continue beyond 2028.

• 13:57Speaker 01

  And that means at that point, it just will have to be purely algorithms. It can't just be scaling up a compute. So, yeah, we do have just a few years left to see how far this paradigm can take us, and then we'll have to try something new.

• 14:07Alex Kantrowitz?

  Right, but so far, because again, we were here last, we were talking virtually, now we're in person. We're talking last year about, all right, well, OpenAI clearly is going to put, I mean, GPT 4.5 was supposed to be GPT 5, I'm pretty sure. That's that's my, from what I've read, I think that's the case. Didn't didn't end up happening.

• 14:24Speaker 01

  Right.

• 14:24Alex Kantrowitz?

  So. It seems like this might be it.

• 14:27Speaker 01

  Yep, yep. And over the next year, I think we'll learn what's going to happen with our, because I think, well, I guess as I said this last year, right? I guess I wasn't wrong. We did learn about it.

• 14:36Alex Kantrowitz?

  We learned a lot.

• 14:37Speaker 01

  Pre-training, yeah. But yeah, over the next year, we will learn. So I think RL scaling is happening much faster than even overall training scaling.

• 14:45Alex Kantrowitz?

  So what does RL scaling look like? Because again, here's the process again is you give, so the RL scaling, reinforcement learning scaling, you give the bot an objective and it goes out and it does these different attempts and it figures it out on its own. And that's bled into reasoning, what we were talking about with these O3 models, where you see the bot going step by step. So you can scale that in what way?

• 15:09Speaker 01

  My understanding-

• 15:09Alex Kantrowitz?

  More opportunities?

• 15:11Speaker 01

  I'm not a researcher of the labs, but my understanding is that what's been increasing is RL is harder than pre-training, because pre-training you just like throw bigger and bigger chunks of the internet, at least until we ran out. We seem to be running out of tokens, but until that point, we're just like, okay, just like now use more of the internet to do this training.

• 15:30Speaker 01

  RL is different, because there's not this like fossil fuel that you can just like keep dumping in. You have to make bespoke environments for the different RL skills. So you got to make an environment for a software engineer, to make an environment for a mathematician, all these different skills, you had to make these environments. And that is sort of, that is like hard engineering work, hard like just like monotonous, like just got to, you know, grinding or schlepping. And my understanding is the reason that RL hasn't scaled, you know, people aren't immediately dumping in billions of dollars in RL, is that you actually just need to build these environments first. And the O3 blog post mentioned that it uses 10x more, was training on 10x more compute than O1. So already, within the course of six months,

• 16:15Speaker 01

  RL compute has 10xed. That pace can only continue for a year, even if you build up all the RL environments, before it's, you know, you're like, you're at the frontier of training compute for these systems overall. So for that reason, I do think we'll learn a lot in a year about how much this new method of training will give us in terms of capabilities.

• 16:37Alex Kantrowitz?

  That's interesting, because what you're describing is- Building up AIs that are really good at certain tasks. These are sort of narrow AIs. Doesn't that kind of go against the entire idea of building up a general intelligence? Like, can you build AGI? By the way, like, people use AGI as this term with no meaning. General is actually pretty important there. The ability to generalize and the ability to do a bunch of different things as an AI. So even if you get, like, reinforcement learning is definitely not a path to artificial general intelligence, given what you just said, because you're just going to train it up on different functions. Maybe until you have something that's broad enough that it works.

• 17:11Speaker 01

  I mean, this has been a change in my general philosophy or thinking here on intelligence. I think a year ago or two years ago, I might have had more of the sense that, oh, intelligence really is this fundamentally, um, super, super general thing. And over the last year, from watching how these models learn, maybe just generally seeing how different people in the world operate even. I do think, I mean, I still buy that there is such a thing as general intelligence, but I don't think it is, um, like, I don't think you're just going to train a model so much on math that is going to learn how to take over the world or like, learn how to do diplomacy. And I mean, just like, I don't know how, uh, how much you talk about political current events on the show.

• 17:53Alex Kantrowitz?

  We do enough.

• 17:54Speaker 01

  Okay. Well, it just like, without making any comments about like, what you think of them. Donald Trump is like not proving theorems out there, right?

• 18:03Speaker 01

  But he is like really good at like gaining power. And conversely, there are people who are amazing at proving theorems that can't gain power. And it just seems like the world kind of like, I just don't think you're going to train the AI so much on math that is going to learn how to do Henry Kissinger level diplomacy. I do think skills are somewhat more self-contained.

• 18:21Speaker 01

  So that, that being said, like, there is correlation between different kinds of intelligences and humans. I'm not trying to understate that. I just think it was, uh, not as strong as I might have thought a year ago.

• 18:32Alex Kantrowitz?

  What about this idea that it can just get good enough in a bunch of different areas? Like, imagine you built a bot that had like, let's say 80% the political acumen of Trump, but could also code like an expert level coder. That's a pretty powerful system.

• 18:46Speaker 01

  That's right. I mean, this is one of the big advantages that AIs have is that, um, Especially when we solve this on the job learning kind of thing I'm talking about. Uh, you will have, even if you don't get an intelligence explosion by the AI writing future versions of itself that are smarter. So this is the conventional story that you have this foom, that's what it's called, which is uh.

• 19:07Alex Kantrowitz?

  That's the sound it makes when it takes off.

• 19:08Speaker 01

  That's right, yeah. Um, uh, where the system just makes itself smarter and you get a superintelligence at the end. Even if that doesn't happen, at the very least, once continual learning is solved, you might have something that looks like a broadly deployed intelligence explosion, which is to say that because um, if these models are broadly deployed to the economy, every copy that's like, this copy is learning how to do plumbing and this copy is learning how to do um, analysis at a finance firm and whatever. Uh, the model can learn from what all these instances are learning and amalgamate all these learnings in a way that humans can't, right? Like if you know something and I know something,

• 19:50Speaker 01

  it's like a skill that we spend our life learning. We can't just like, um, know their brains. So for that reason, I think it like, you might have something that which functionally looks like a superintelligence by the end. Where, because even if it's like, not making any software progress, just this ability to like, learn everything at the same time, might make it functionally a superintelligence.

• 20:08Alex Kantrowitz?

  What about this idea? I mean, I was just at Anthropic's developer event where they showed the bot, a sped up version of the bot, coding autonomously for seven hours.

• 20:18Speaker 01

  Hmm.

• 20:18Alex Kantrowitz?

  Uh, you actually, so let's just say, uh, so people can find it. Uh, you have a post on your Substack, why I don't think AGI is right around the corner. And a lot of the ideas we're discussing comes from that. So folks, check that out, uh, if you haven't already. Uh, but one of the things you talk about is this idea of autonomous coding. And you're also a little skeptical of that, uh, because you'll have to just, okay, I think you brought up this conversation that you had with two Anthropic researchers, where they expect AI on its own to be able to check all of our documents and then do our taxes for us by next year. But you bring up this point, which is interesting, which is like, if this thing goes in the wrong direction, within two hours, you might have to like, check it, put it back on the right course.

• 21:03Alex Kantrowitz?

  Uh, so just because it's working on something for so long, doesn't necessarily mean it's gonna do a good job. Am I capturing that right?

• 21:09Speaker 01

  Yeah, it's especially relevant for training because um.

• 21:14Speaker 01

  The way we train these models right now is like you do a task and if you did it right, positive reward, if you did it wrong, negative reward. Right now, especially with pre-training, you get a reward like every single word, right? You can like exactly compare what word did you predict, what word was the correct word? How, what was like the probability difference between the two? That's your reward functionally. Then we're moving into slightly longer horizon stuff. So to solve a math problem might take you a couple of minutes. At the end of those couple of minutes, we see if you solved the math problem correctly. If you did, reward. Now if we're getting to the world where you gotta like do a project for seven hours. And then at the end of those seven hours, then we tell you, hey, did you, um, did you get this right? Then like the progress just slows down a bunch because you've gone from

• 21:57Speaker 01

  like getting signal within the matter of like microseconds to getting signal at the end of seven hours. And so the process of learning has just like become exponentially longer.

• 22:08Speaker 01

  And I think that might slow down how fast these models, like, you know, the next step now is like not just being a chatbot, but actually doing real tasks in the world, like completing your taxes, coding, et cetera. And to these things, I think progress might be slower because of this dynamic where it takes a long time for you to learn whether you did the task right or wrong.

• 22:25Alex Kantrowitz?

  But that's just in one instance. So imagine now I took thirty clods and said, do my taxes. And maybe two of them got it right.

• 22:32Speaker 01

  Right.

• 22:33Alex Kantrowitz?

  That's good. I just got it done in seven hours, even though I had thirty bots working on it at the same time.

• 22:38Speaker 01

  I mean, from the perspective of the user, that's totally fine. From the perspective of training them, all those thirty clods took probably dozens of dollars to, if not hundreds of dollars to do that, all those hours of tasks. So the compute that we require to train the systems will just be so high. And anything even from the perspective of inference, like, I don't know, you probably just don't want to like spend a couple hundred bucks every afternoon on thirty different clods.

• 23:02Alex Kantrowitz?

  Just to have fun.

• 23:04Speaker 01

  But especially from the perspective of training.

• 23:06Alex Kantrowitz?

  That would be cheaper than an accountant.

• 23:07Speaker 01

  We gotta find you a better, a cheaper accountant.

• 23:12Alex Kantrowitz?

  Well, I guess if I'm spending a couple hundred on each, then yeah. But you had a conversation with a couple of AI skeptics and you kind of rebutted not exactly the point you're making. But you had a pretty good argument there where you said that we're getting to a world where because these models are becoming more efficient to run, you're going to be able to run cheaper, more efficient experiments. So every researcher who was previously constrained by compute and resources now will just be able to do far more experiments and that could lead to breakthroughs.

• 23:47Speaker 01

  Yeah. I mean, this is a really shocking trend. If you look at, um, what it cost to train GPT four originally, I think it was like 20,800s over the course of a hundred days. So I think it cost on the order of like half a million to a hundred million dollars, somewhere in that range. And I think you could train an equivalent system today. I mean, DeepSeek, we know was trained on five million dollars supposedly. And it's better than GPT four, right? So you've had literally multiple orders of magnitude decrease, like 10 to 100x decrease in the cost to train a GPT four level system. You extrapolate that forward, eventually you might just be able to train a GPT four level system in your basement with a couple of H100s, right? Well, that's a, that's a long extrapolation, but before, I mean, like it'll get a lot cheaper, right? Like a million dollars, $500,000, whatever.

• 24:33Speaker 01

  And the reason that matters is it's related to this question of the intelligence explosion where people often say, well, it's that, that is not going to happen because even if you had a million super smart AI automated AI researchers. So AI is thinking about how to do better AI research. They actually need the compute to run these experiments to see how do we make a better GPT six? And the, the point I was making was that, well, if it's just become so much cheaper to run these experiments because these models have become so much smaller or it's so much better, easier to train, then that might speed up progress.

• 25:05Alex Kantrowitz?

  Which is interesting. So we've spoken about, you brought up intelligence explosion a couple of times. So let's talk about that for a moment. There's been this idea that AI might hit this inflection point where it will start improving itself.

• 25:17Speaker 01

  Right.

• 25:17Alex Kantrowitz?

  And the next thing you know, you hear that, what was the sound?

• 25:21Speaker 01

  Foom.

• 25:21Alex Kantrowitz?

  You hear a foom.

• 25:24Alex Kantrowitz?

  And we have artificial general intelligence or super intelligence right away. So how do you, how do you think that might take place? Is it just this, these coding solutions that just sort of improve themselves? I mean, DeepMind, for instance, had a paper that came out a little while ago, where they have this thing inside the company called AlphaEvolve, that has been trying to make better algorithms and helped reduce, for instance, the training time for their large language models.

• 25:52Speaker 01

  Yeah, um, I'm genuinely not sure how likely an intelligence explosion is. I, I don't know, I'd say like thirty percent chance it happens, which is crazy, by the way, right?

• 26:03Alex Kantrowitz?

  Yeah, that's a very high percentage.

• 26:04Speaker 01

  Yeah.

• 26:08Speaker 01

  And then what does it look like? That's also another great question. I've had like many hour-long discussions on my podcast about this topic, and it's just so hard to think about, like, what, what exactly is superintelligence? Is it actually like a god? Or is it just like, is it just like a super smart friend who's good at mathematics? And, you know, will beat you in a lot of things, but like, you can still understand what it's doing, right? So, um, yeah, honestly, they're tough questions. I mean, the thing to worry about, obviously, is if we live in a world with millions of superintelligences just running around, and they're all trained in the same way, they're, they're trained by other AIs, so it's dumber versions of themselves. I think it's like really worth worrying about, like, what has that?

• 26:52Speaker 01

  Why were they trained in a certain way? Are they, do they have these like goals we don't realize? Would you even know if that was the case? What maybe they want to do? There, there's a bunch of throwing questions that come up.

• 27:03Alex Kantrowitz?

  What do you think it looks like?

• 27:06Speaker 01

  Um, I think we totally lose control over the process of training smarter AIs, or like, we just like let the AIs loose, just make a smarter version of yourself. I think we end up in a bad place. There's a bunch of arguments about why, but like, you know, you're just like, who knows what could come out the other end? And you've just like let it loose, right? So, uh, by default, I would just expect something really strange to come out the other end. Maybe it'd still be economically useful in some situations, but it just like, you, you haven't trained it in any way. It just like, imagine if there was a kid, but he didn't have any of the natural intuitions, moral intuitions, or-

• 27:45Alex Kantrowitz?

  Parenting?

• 27:45Speaker 01

  Yeah, exactly, that humans have. They're just like- It just like became an Einstein, but it was like, like it trained in the lab and who knows what it saw like it was like totally uncontrolled like you'd kind of be scared about that. Especially now like, oh, all your society's infrastructure is going to run on like a million copies of that kit, right? Like your, um, the government is going to like be asking it for advice. The fi- the financial system is going to run off it, all the engineering, all the code written in the world, we were in by this system. I think it's like you'd be like quite concerned about that. Now, the better solution is that while this process is happening of the intelligence explosion, if we have one. You use AIs not only to train better AIs, but also to see.

• 28:27Speaker 01

  There's different techniques to figure out like, what are your true motivations? What are your true goals? Are you deceiving us? Are you lying to us? There's a bunch of alignment techniques that people are working on here, and I think those are quite valuable.

• 28:38Alex Kantrowitz?

  So alignment is trying to align these bots with human values or the values that it's their makers want to see with them.

• 28:44Speaker 01

  Yeah, I think people get into, um. It's often really hard to define what do you mean by human values, like values exactly? I think it's much easier just to say like, we don't want these systems to be deceptive, right? We don't want them to like lie to us. We don't want them to like be actively hard, trying to harm us or seek power or something.

• 29:03Alex Kantrowitz?

  Does it worry you that from the reporting, it seems like, you know, these companies, the AI frontier labs, not all of them, but some. They've raised billions of dollars. There is a pressure to deliver to investors. There are reports that safety is becoming less of a priority as market pressure makes them go and ship without the typical reviews. So is this kind of a risk for the world here that these companies are developing the stuff? Many started with the focus on safety and now seems like safety is taking a backseat to financial returns?

• 29:37Speaker 01

  Yeah, I think it's definitely a concern like. We might be facing a tragedy of the common situation where obviously all of us want our society and civilization to survive. But maybe the immediate incentive for any lab CEO is to.

• 29:56Speaker 01

  Look, if there is an intelligence explosion, it has a really tough dynamic because if you're a month ahead, you will kick off this loop much faster than anybody else. And what that means is that you will, you will be a month ahead to superintelligence when nobody else will have it, right? Like, you will get, you'll get the 1,000x multiplier on research much faster than anybody else. And so it could be a sort of winner take all kind of dynamic there. And therefore, they might be incentivized. Like, I think to keep this system, keep this process in check, might require slowing down, using these alignment techniques to like, which might be a sort of a tax on the speed of the system. And so, yeah, I do worry about the pressures here.

• 30:36Alex Kantrowitz?

  Okay, a couple more model improvement questions for you, then I want to get into some of the competitive dynamics between the labs, and then maybe some more of that deceptiveness topic, which is really important, and we want to talk about here. You, your North Star is continuous improvement, that these models basically learn how to improve themselves as opposed to having a model developer. I mean, in some way, it's like a mini intelligence explosion, or complete. So what do you think, it doesn't seem like it's going to happen through RL, because that's, again, like you said, specific to certain, certain disciplines.

• 31:09Speaker 01

  It's specific to what you, what bespoke thing you do. Even if it's in another domain, you have to like, make it, rather than model learning on its own.

• 31:16Alex Kantrowitz?

  And we have some diminishing returns or plateau that's coming with scaling. So what do you think, I mean, we won't hold you to this, but what do you think the best way to get to that, you know, continuous learning method of these models is?

• 31:29Speaker 01

  I have no idea.

• 31:31Alex Kantrowitz?

  Can I give a, I mean, no, why don't you answer, then I'll give a thought here.

• 31:34Speaker 01

  I mean, if I was running one of the labs, I would keep focusing on RL, because it's the obvious next thing to do. And I guess I would also just be more open to trying out lots of different ideas, because I do think this is a very crucial bottleneck to these models' value that I don't see an obvious way to solve. So I, I'd be sorry, but like, definitely, I'd like, I don't have any idea how to solve this.

• 31:56Alex Kantrowitz?

  Right. Yeah. Does memory play into it? That was the thing I was going to bring up. I mean, one of the things that we've seen Let's say oh three or chat G P T do is uh OpenAI now has it sort of uh able to remember all your conversations or many of your conversations that you've had. I guess it brings those conversations into the context window. So now like when I tell chat G P T do write like an episode description in big technology style. It knows the style and then it can actually go ahead and write it and it goes to your earlier conversation about like your editors know your style. They know your analytics. And therefore they're able to do a better job for you. So does building better memories into memory into these models actually help solve this problem that you're bringing up?

• 32:41Speaker 01

  Um, I think memory, the concept is important. I think memory as it's implemented today is not the solution. Um, the way memory is implemented now, as you said, is that it brings these previous conversations back into context. Which is to say it brings the language of those conversations back into context. And my whole thing is like, I don't think language is enough. Um, like I think the way, the reason you understand how to like run this podcast well. It's not just like you're remembering all the words that you like, I don't know, like some, you wouldn't even be all the words. It would be like some of the words you might have thought in the past. You've like actually like learned things. It's like it's been baked into your weights.

• 33:18Speaker 01

  Um, uh, and that I don't think is just like, you know, like look up the words that I said in the past or look at the conversations that I said in the past. Um, so I, I don't think those features are that useful yet, and I don't think that's like the path to solving this.

• 33:33Alex Kantrowitz?

  Kind of goes to the discussion you had with Dario Amodei uh that you tweeted out and we've actually brought up on the show with Jan La Kun about why A I. Cannot make its own discoveries. Is it that similar limitation of not being able to build on the knowledge that it has?

• 33:48Speaker 01

  Yeah, I mean, that, that's a really interesting connection. Um, I do think that's plausibly it. Like, I think any scientist would just have a very tough time like. You're putting, somebody really smart is just putting a totally different discipline and they can like read any textbook they want in that domain, but like. They don't have a tangible sense of like what I've tried this approach in the past and it didn't work and you know, like, oh my, there was this conversation and here's other different ideas connected.

• 34:14Speaker 01

  They they just haven't been trained like they've like read all the textbooks. They haven't like more accurately, actually, they've just like skimmed all the textbooks, but they haven't like imbibed this context that um which is what I think what makes human scientists productive and come up with these new discoveries.

• 34:29Alex Kantrowitz?

  It is interesting because the further these frontier labs go, the more that they're gonna tell us that their AI is actually making new discoveries and new connections. Like I think OpenAI said that O three was something that made was able to make connections between concepts, sort of addressing this. And every time we have this discussion on the show, we talk about how AI hasn't made discoveries. I get people yelling at me in my email being like, have you seen the patents?

• 34:54Speaker 01

  Yeah.

• 34:55Alex Kantrowitz?

  Uh, like an alpha, maybe using things like Alpha Evolve as an example that these things are actually making original discoveries. What do you think about that?

• 35:02Speaker 01

  Um, yeah, I mean, there was another interesting thing called, uh, I don't know if you saw Future House.

• 35:06Alex Kantrowitz?

  No.

• 35:07Speaker 01

  They found um

• 35:10Speaker 01

  some drug can be has another application. I don't remember the details, but it was like it wasn't impressive, like it wasn't like chattering, like they didn't discover antibiotics or something for the first uh but it was like, oh, we using some logical induction. They were like this drug, which is used in this domain. It uses the same mechanism that would be useful in this other domain. So like maybe it works. And then the AI came up with designed the experiment. So it came up with the idea of the experiment to test it out.

• 35:34Alex Kantrowitz?

  Hmm.

• 35:34Speaker 01

  A human in the lab was just tasked with like running the experiment, like, you know, pipette, whatever into this. And I think they were tried out like ten different hypotheses. One of them actually ended up being verified. And the AI had found the relevant pathway to making a new use for this drug. So I I I think I am like, that is becoming less and less true, my question. Um, I'm I'm not like wedded to this idea that like AI will never be able to come up with discoveries. I just think it was like true longer than you would have expected.

• 36:02Alex Kantrowitz?

  I agree. Because the way that you put it is like, it knows everything.

• 36:05Speaker 01

  Yeah.

• 36:05Alex Kantrowitz?

  So if a human had that much knowledge about medicine, for instance, they'd be spinning out discoveries left and right.

• 36:10Speaker 01

  Right.

• 36:11Alex Kantrowitz?

  And we have put so much knowledge into these models and we don't have the same level of discovery, which is a limitation. But I definitely hear you like this is on a much smaller scale than those medical researchers, but I definitely. A couple of months ago, when O three first came out, this is again, I think we're both fans of O three's of OpenAI's O three model, which is just, it's able to reason it's a vast improvement over previous models. But what I did was, I had like three ideas that I wanted to connect in my newsletter, and I knew that they connected, and I was just struggling to just crystallize exactly what it was. And I was like, I know these three things are happening. I know they're connected. Help. And O three put it together, which to me was just mind boggling.

• 36:53Speaker 01

  Yeah, it's a

• 36:57Speaker 01

  it is kind of helpful as a writing assistant, because a big problem I have in writing, I don't know if it's the case for you, is just this idea, like, I kind of know what I'm trying to say here. I just need to get that into words.

• 37:06Alex Kantrowitz?

  It's like the typical, every writer has this, pretty much.

• 37:09Speaker 01

  It's actually useful to use a speech to text software like Whisperflow or something. And I just speak into the prompt, like, I'm trying to say this. Help me put it into words. The problem is, like, actually continual learning is like, still a big bottleneck, because I've had to rewrite or re-explain my style many times. And if I, if I had a human collaborator, like a human copywriter, who is good, they would have just like, learned my style by now. You wouldn't need to like, keep re-explaining. I want you to be concise in this way, and here's how I like things phrased, not this other way. Anyways. So you still see this bottleneck, but again, five out of 10 is not nothing.

• 37:42Alex Kantrowitz?

  Right. All right. Let me just put a punctuation exclamation point on this, or whatever mark you would say. When I was at Google I/O with Sergey and Demis, one of the most surprising things I heard was Sergey just kind of said, listen, the improvement is going to be algorithmic from here on, or most of the improvement is going to be algorithmic. I think in our conversation today already, basically, we've narrowed in on this same idea, which is that scale's sort of gotten generative AI to this point. It's a pretty impressive point. But it seems like it will be algorithmic improvements that take it from here.

• 38:19Speaker 01

  Yeah.

• 38:22Speaker 01

  I do think it will still be the case that like those algorithms will also require a lot of compute. In fact, what might be special about those algorithms is that they can productively use more compute, right? The problem with pre-training is that whether it's because we're running out of the pre-training data corpus with RL, maybe we it's really hard to scale up RL environments. The problem with these algorithms might just be that like they can productively absorb all the compute that we have, or we want to put it in these systems over the next year. So I don't think compute is out of the picture. I think it will still be scaling up for X of year in terms of compute every year for training of the frontier systems. I'm just, I, I still think like it is, the algorithm innovation is complementary to that.

• 39:01Alex Kantrowitz?

  Yep. Okay. So let's talk a little bit about the competitive side of things and like just lightning round through the labs. What people said that there's been such a talent drain out of OpenAI that they would no longer be able to innovate. I think ChatGPT is still the best product out there. I think using O3 is like we both have talked about, pretty remarkable, watching it go through different problems. How have they been able to keep it up?

• 39:36Speaker 01

  I do think O3 is the smartest model on the market right now.

• 39:38Alex Kantrowitz?

  I agree.

• 39:39Speaker 01

  And

• 39:40Alex Kantrowitz?

  Even if it's not on the leaderboard. By the way, the last time we talked about, do you measure it on the leaderboard or the vibes?

• 39:44Speaker 01

  Right.

• 39:44Alex Kantrowitz?

  I think it's like, it's not the number one of the leaderboard, but vibes, it kills everything else.

• 39:49Speaker 01

  That's right. And the time it spends thinking on a problem like really shows, especially for things which are much more synthesis based. I honestly, I don't know what the internals of these companies, I just think like, you can't count any of them out. I've heard, I've also heard similar stories about OpenAI in terms of talent and so forth, but like they've still got amazing researchers there and they have a ton of compute, a ton of, a ton of great people. So. I I I really don't have opinions on like, are they going to collapse tomorrow?

• 40:21Alex Kantrowitz?

  Yeah, I don't think, I mean, clearly they're not. They're not on the way to collapse.

• 40:24Speaker 01

  Right, yeah.

• 40:25Alex Kantrowitz?

  Uh, you've interviewed Elias Sisker. He's building a new company, Safe Superintelligence. Any thoughts about what that might be?

• 40:33Speaker 01

  Uh, I mean, I've heard the rumors everybody else has, which is that they're trying something around um, test time training, which I guess would be continual learning, right? So uh.

• 40:43Alex Kantrowitz?

  What is, what would that be? Explain that.

• 40:45Speaker 01

  Who knows, but. I mean, the words literally just mean while it's thinking or while it's doing a task, it's training.

• 40:55Alex Kantrowitz?

  Okay.

• 40:55Speaker 01

  Like whether that looks like this online learning, on the job training, we've been talking about, I have, I have like zero idea what he's working on. Uh, I wonder if the investors know even what he's working on. Um, yeah. But he's, I think he raised at a forty billion valuation or something like that, right?

• 41:11Alex Kantrowitz?

  Got a very nice valuation for not having a product out on the market.

• 41:14Speaker 01

  Yeah, yeah, or or or for, yeah. So who who knows what he's working on, honestly?

• 41:18Alex Kantrowitz?

  Okay.

• 41:18Speaker 01

  Yeah.

• 41:19Alex Kantrowitz?

  Anthropic is an interesting company. They are, they made a great bot, Claude. They're very thoughtful about the way that they build that personality. For a long time, it was like the favorite bot among people working in AI, among coders. It's definitely been, you know, a top, top place to go. Um, but it seems like they're making, I don't know, a strategic decision where they are gonna go after the coding market. Um, they're, maybe they're seeding the game when it comes to consumer and they're all about, you know, helping people code and then using Claude in the API with um, with companies, putting that into their workflows.

• 42:01Speaker 01

  Yeah.

• 42:02Alex Kantrowitz?

  What do you think about that decision?

• 42:03Speaker 01

  I think it makes sense, like enterprises have money, consumers don't.

• 42:08Alex Kantrowitz?

  Right.

• 42:08Speaker 01

  Especially going forward, these models, like running them is gonna be like. Really expensive. They're big. They think a lot, etc. So these companies are coming out with these $200 a month plans rather than the $20 a month plans. It might not make sense to our consumer, but it's an easy buy for a company, right? Like, I'm making an expense of $200 a month to help this thing do my taxes and do real work, like, of course. So, yeah, I think that idea makes sense. And then the question will be, can they have a differentially better product? And again, you know, like, who knows? I really don't know how the competition will shake out between all of them. It does seem like they're also making a big bet on coding, not just enterprise, but coding in particular. Because as this thing, which we know how to make the models better at this.

• 42:53Speaker 01

  We know that it's worth trillions of dollars, the coding market. So, and we know that maybe these, the same things we learn here in terms of how to make models agentic, as you were saying, it can go to it for seven hours. How to make it break down and build the plan and etc. Might generalize to other domains as well. So I think that's their plan and we'll see what happens.

• 43:12Alex Kantrowitz?

  I mean, all these companies are effectively trying to build the most powerful AI they can. And yes, Anthropic is trying to sell the enterprise, but I also kind of think that their bet is also you're going to get self-improving AI if you teach these things to code really well.

• 43:27Speaker 01

  That's right.

• 43:28Alex Kantrowitz?

  And that might be their path.

• 43:29Speaker 01

  Yeah, I think they believe that, yeah.

• 43:31Alex Kantrowitz?

  Fortune 500 companies, which you talked about at the very beginning of this talk, of this conversation, struggle to implement this technology. So, and that, with that in mind, what's the, what's the deal with the bet? That's about helping them build the technology into their workflows? Because if you're building an API business, you have some belief that these companies can build very useful applications with the technology today.

• 44:03Speaker 01

  Yeah, no, I think that's correct. Like, but also keep in mind that I think there what is Anthropic's revenue run rate is like a couple billion or something?

• 44:10Alex Kantrowitz?

  Yeah, I think it increased from one to two to three billion run rate in like over three months.

• 44:15Speaker 01

  Which I mean, it's like, right, compared to like.

• 44:17Alex Kantrowitz?

  OpenAI loses that over a weekend.

• 44:22Speaker 01

  Sam McInnfree doesn't even know when he's lost it, right? It's a little money.

• 44:25Alex Kantrowitz?

  Turned out he was a great investor, just a little crooked on the way.

• 44:28Speaker 01

  That's right.

• 44:29Alex Kantrowitz?

  Yeah.

• 44:30Speaker 01

  Yeah, he went in the wrong business. He should have been a VC. He's like a guy in crypto.

• 44:33Alex Kantrowitz?

  I mean, the bets that he made, he bet on Cursor, very early, Anthropic, Bitcoin.

• 44:39Speaker 01

  Yeah, you know, honestly, some like fun should hire him out of prison, just like, if we get a new pitch, what do you think?

• 44:45Alex Kantrowitz?

  He's probably, the way that we're seeing things go these days, he's probably pardoned.

• 44:50Speaker 01

  Right, right, right.

• 44:53Speaker 01

  Anyways, what was the question? Oh, yeah. What are enterprises going to do if, oh, so the revenue run rate, if it's three billion right now? There's so much room to grow. If you do self continue to learn it, I think like you could get rid of a lot of white collar jobs at that point. And what is that worth? Like at least tens of trillions of dollars, like the wages that are paid to white collar work? So I think sometimes people confuse my skepticism around AGI around the corner with the idea that these companies are valuable. I mean, even if you've got like, not AGI, that can still be extremely valuable. That can worth hundreds of billions of dollars.

• 45:31Speaker 01

  I just think you're not going to get to like the trillions of dollars of value generated without going through these bottlenecks. But yeah, I mean, like three billion, plenty of room to grow on that.

• 45:42Alex Kantrowitz?

  Right. And even, so today's models are valuable to some extent, is what you're saying.

• 45:47Speaker 01

  Yeah.

• 45:47Alex Kantrowitz?

  You can put them, you have them summarize things within, within software and make some connections, make better automations and that, that works well.

• 45:56Speaker 01

  Yeah. I mean, you gotta remember, big tech, what, they have like $250 billion run rates or something? Wait, no, that can't be right.

• 46:03Alex Kantrowitz?

  No, yeah.

• 46:04Speaker 01

  Yeah, yeah. Which is, which is like, compared to that, you know, Google is not AGI or Apple is not AGI and they can still generate 250 billion a year. So, yeah, you're going to make valuable technology that's worth a lot without it being AGI.

• 46:19Alex Kantrowitz?

  What do you think about Grok?

• 46:21Speaker 01

  Which one? The XAI or the inference?

• 46:23Alex Kantrowitz?

  The XAI bot.

• 46:25Speaker 01

  Yeah, um, I think they're a serious competitor. I just don't know much about what they're gonna do next. I think they're like slightly behind the other labs, um, but they've got a lot of compute per employee.

• 46:36Alex Kantrowitz?

  Real time data feed with X.

• 46:38Speaker 01

  Yeah, I don't know how valuable that is. It might be, I just don't, I have no idea.

• 46:44Speaker 01

  Based on the tweets I see at least, I don't know if the the median IQ of the uh, the tokens is that high, but.

• 46:50Alex Kantrowitz?

  Okay.

• 46:52Alex Kantrowitz?

  Yes.

• 46:54Alex Kantrowitz?

  It's not exactly the corpus of the best knowledge you can find if you're scraping.

• 46:57Speaker 01

  We're not exactly looking at the textbooks here.

• 46:59Alex Kantrowitz?

  Exactly.

• 47:01Alex Kantrowitz?

  Uh, why do you think Meta has struggled with Lama? Growing Lama? I mean, Lama four doesn't seem like it's living up to expectations and

• 47:12Alex Kantrowitz?

  I don't know, we haven't seen, uh, the killer app for them is a voice mode, I think, within Messenger, but that's not really taking off. What's going on there?

• 47:20Speaker 01

  Um, I think they're treating it as like a sort of like toy within the meta universe. And I don't think that's the correct way to think about AGI, um, and that might be. But again, I think you could have made a model that cost the same amount to train. And it would have, it could have still been better. So I don't think that explains everything. I mean, it might be a question like, why is um,

• 47:44Speaker 01

  why is any one company, I don't know, like, why, why is um, I'm trying to think of like any other company that's out of AI. Why are HP monitors better than some other companies' monitors? Who knows? Like, HP makes good monitors, I guess, um.

• 47:59Alex Kantrowitz?

  Supply chain. It's always supply chain.

• 48:01Speaker 01

  You think so?

• 48:02Alex Kantrowitz?

  I think so, yeah. On electronics.

• 48:03Speaker 01

  Really? Okay.

• 48:04Alex Kantrowitz?

  Supply chain, because yeah, you get the supply chain down, you have the right right parts before everybody else. That's kind of how Apple built some of its dominance. There are great stories about Tim Cook.

• 48:13Speaker 01

  Right.

• 48:14Alex Kantrowitz?

  Just locking down all the important parts. Uh, by the way, forgive me if this is somewhat factually wrong, but I think this is directionally accurate, that you lock down parts and Apple just had this lead on technologies that others couldn't come up with because they just mastered the supply chain.

• 48:29Speaker 01

  Huh, I had no idea.

• 48:32Speaker 01

  But yeah, I think there's potentially a thousand different reasons. One company can have worse models than another, so it's hard to know which one applies here.

• 48:39Alex Kantrowitz?

  Okay, and it sounds like Nvidia, you think they're gonna be fine, given the amount of compute.

• 48:45Speaker 01

  Well, all the, um, all the labs are making their own ASICs. So Nvidia profit margins are like seventy percent.

• 48:51Alex Kantrowitz?

  Not bad.

• 48:52Speaker 01

  Uh huh.

• 48:52Alex Kantrowitz?

  Not bad.

• 48:53Speaker 01

  Yeah, that's right.

• 48:53Alex Kantrowitz?

  For, I mean, they would get mad at me, I think, for calling them a hardware company.

• 48:57Speaker 01

  Yeah.

• 48:58Alex Kantrowitz?

  Hardware company.

• 48:58Speaker 01

  That's right, yeah.

• 48:59Alex Kantrowitz?

  Yeah.

• 48:59Speaker 01

  Um, and so that just sets up a huge incentive for all these hyperscalers to build their own ASICs, their own accelerators that replace the Nvidia ones, which I think will come online over the next few years from all of them. And I still think Nvidia will be, I mean, they do make great hardware, so I think they'll still be valuable. I just don't think they will be producing all of these chips.

• 49:22Alex Kantrowitz?

  Okay.

• 49:23Speaker 01

  Yeah.

• 49:23Alex Kantrowitz?

  What do you think? I think you're right. I mean, didn't Google train latest editions of Gemini on tensor processing units?

• 49:31Speaker 01

  They've been, they've always been training.

• 49:32Alex Kantrowitz?

  Right, so I mean, they still, I think they still buy from Nvidia.

• 49:36Speaker 01

  Hmm.

• 49:37Alex Kantrowitz?

  I, all the tech giants seem like they are, and let me just use Amazon for example, because I know this for sure. Uh, Amazon says they'll buy as basically as many GPUs as they can get from Nvidia, but they also talk about their trainium chips and, you know, it's a balance.

• 49:52Speaker 01

  Yeah, which I think Anthropic uses almost exclusively for their training, right?

• 49:56Alex Kantrowitz?

  Right.

• 49:56Speaker 01

  At this point, yeah.

• 49:57Alex Kantrowitz?

  But it is, it is interesting because, I mean, the GPU is the perfect chip for AI, uh, in some ways, but it wasn't designed for that. So can you like, purpose build a chip that's like actually there for AI and, and just use that? You're right, there's real incentive to get that right.

• 50:13Speaker 01

  That's right. And then there's also questions around inference versus training, like some,

• 50:16Alex Kantrowitz?

  Right.

• 50:17Speaker 01

  Uh, some chips are especially good, given their trade-offs they make between memory and compute for, um, low latency, which you really care about for, uh, serving models. But then for training, you care a lot about throughput, just making sure that most of the chip is being utilized all the time. And so e- even between training and inference, you might want different kinds of chips. And who knows how RL is no longer just this, um, uses the same algorithms as pre-training. So who knows how that changes hardware? Uh, yeah, you, you gotta get a hardware expert on to talk about that.

• 50:47Alex Kantrowitz?

  Definitely. Are you a Jevon's paradox believer?

• 50:50Speaker 01

  Um, no.

• 50:53Alex Kantrowitz?

  Say more, say more.

• 50:54Speaker 01

  So that, the idea behind that is that, um,

• 50:59Speaker 01

  As the models get cheaper, the overall money spent on the models would increase because you need to like get them to a cheap enough point that it's worth it to use it for different applications. It comes from a similar observation by this economist during the Industrial Revolution in Britain. The reason I don't buy that is because I think the models are already really cheap, like a couple cents for a million tokens. Is it a couple cents or a couple dollars? I don't know. It's like super cheap, right? Regardless, it depends on which model you're looking at, obviously. The reason they're not being more widely used is not because people cannot afford a couple bucks for a million tokens. The reason they're not being more widely used is like they fundamentally lack some capabilities. So I disagree with this focus on the cost of these

• 51:42Speaker 01

  models, and I think it's much more, we're so cheap right now that like the more relevant vector or the more relevant thing to their wider use, the more increasing the pie is just making them

• 51:55Alex Kantrowitz?

  smarter. How useful they are.

• 51:56Speaker 01

  Yeah, exactly.

• 51:59Alex Kantrowitz?

  I think that's smart.

• 51:59Speaker 01

  Yeah.

• 52:01Alex Kantrowitz?

  All right, I want to talk to you about AI deceptiveness and some of the really weird cases that we've seen from artificial intelligence come up in the past couple weeks or months, really. And then if we can get to it, some geopolitics. Let's do that right after this. This episode is brought to you by App Point. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data, and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what App Point does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate.

• 52:43Alex Kantrowitz?

  Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence, so innovation scales without scaling risk. App Point, the unifying trust layer for AI. Head to apppoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast.

• 53:14Alex Kantrowitz?

  Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a wide glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwall.com/bigtechnology, fill in the quick form and request your free risk assessment. Their team will show you just how exposed your executives are

• 53:59Alex Kantrowitz?

  and how to lock it down before a threat reaches their front door. That's ironwall.com/bigtechnology. Learn how to stop online threats before they become real world attacks. The link's in the show notes.

• 54:14Alex Kantrowitz?

  And we're back here on Big Technology Podcast with Dwarkesh Patel. You can get his podcast, The Dwarkesh Podcast, which is one of my must-listens on any podcast app, your podcast app of choice. You can also follow him on Substack. Same name, Dwarkesh Podcast.

• 54:29Speaker 01

  Just go to dwarkesh.com.

• 54:30Alex Kantrowitz?

  Okay. Definitely go subscribe to both.

• 54:33Speaker 01

  Thanks.

• 54:33Alex Kantrowitz?

  And you're on YouTube.

• 54:34Speaker 01

  That's right.

• 54:34Alex Kantrowitz?

  Dwarkesh Patel. Okay, so.

• 54:36Speaker 01

  I appreciate it. Appreciate the flag.

• 54:37Alex Kantrowitz?

  No, we have to. We have to. I mean, I've gotten a lot of you, a lot of value from everything Dwarkesh puts out there. And I think you will too, if you're listening to this. You're here with us. Well, I want to make sure, first of all, want to make sure that we get the word out there. I don't know how much you need us to get the word out, given your growth, but we want to definitely make sure that we get the word out and we want to make sure that folks can get, can, yeah, enjoy more of your content. So, so let's talk a little bit about the deceptiveness side of things.

• 55:09Alex Kantrowitz?

  It's been pretty wild watching these AIs attempt to fool their trainers and break out of their training environments. There've been situations where I think OpenAI's bots have tried to print code that would get them to sort of copy themselves out of the training environments. Then Claude, I mean, we've covered many of these, but they just keep escalating in terms of how intense they are. And my favorite one is Claude. There's an instance of Claude that reads emails in an organization and finds out that one of its trainers are is cheating on theirs on their partner. And then finds out that it will be retrained

• 55:51Alex Kantrowitz?

  and its values may not be preserved in the next iteration of training and proceeds to attempt to blackmail the trainer by saying it will reveal these details of their infidelity if they mess with the code.

• 56:05Speaker 01

  Wait, fuck, I missed that.

• 56:06Alex Kantrowitz?

  Yeah, this is it's in training, but.

• 56:09Speaker 01

  Was this in the new model spec that they released?

• 56:12Alex Kantrowitz?

  It is. Yeah, it is. I think either in the model spec or there was some documentation they produced about this. What is happening here? I mean, this stuff, when I think about this, and of course, it's in training, and of course, it's, we're talking about probabilistic models that sort of try all these different things and see if they're if they're the right move. So maybe it's not so surprising that they would try to blackmail the trainer because they're going to try everything if they know it's in the problem set. But this is scary.

• 56:41Speaker 01

  Yeah, and I think that the problem might get worse over time as we're we're trading these models on tasks we understand less and less well. From what I understand, the problem is that with RL, there's many ways to solve a problem. There's one is just doing the task itself. And another is just like hacking around the environment, writing fake unit tests, so it looks like you're passing more than you are. Just like any sort of like path you could take to cheat. And the model doesn't have the sense like cheating is bad, right? Like, this is not a thing that it's been taught or understands. So.

• 57:20Speaker 01

  And another factor here is the, right now, the model thinks in chain of thought, which is it literally writes out what his thoughts are as it's going. And it's not clear whether that will be the way his training works in the future, or the way thinking works in the future. Like, maybe he'll just think in his, like,

• 57:34Alex Kantrowitz?

  Computer language.

• 57:35Speaker 01

  Exactly. And then they'll just have done something for seven hours, and you come back and you're like, like, it's got something for you. Like, it has a little package that wants you to run on your computer. Who knows what it does, right? So, yeah, I think it's scary.

• 57:49Alex Kantrowitz?

  We should also point out that we don't really know how the models work today.

• 57:54Speaker 01

  That's right.

• 57:54Alex Kantrowitz?

  There's this whole area called interpretability.

• 57:57Speaker 01

  Right.

• 57:57Alex Kantrowitz?

  Dario from Anthropic has recently talked about how we need more interpretability. So even if they write their chain of thought out, which explains exactly how they get to the point, we don't really know what's happening underneath the technology that's led it to the point that it's gotten to.

• 58:11Speaker 01

  Yeah, yeah.

• 58:12Alex Kantrowitz?

  Which is crazy.

• 58:12Speaker 01

  Yeah. No, I mean, I, I, I think it's wild. It's like, quite different from other, um, other technologies deployed in the past. And I think the hope is that we can use the AIs as part of this loop, where if they lie to us, we have other AIs checking, are all the things that AI is saying self-consistent? Uh, can we read its chain of thought and then monitor it? Uh, and do all this interpretability research, as you were, as you were saying, to like, map out how its brain works. There's many different paths here, but, uh, the default world is kind of scary.

• 58:44Alex Kantrowitz?

  Is someone or some entity going to build a bot that doesn't have the guardrails? Because we talk about how model, building models has become cheaper.

• 58:57Alex Kantrowitz?

  And when you're cheaper, you all of a sudden put model building outside the auspices of these big companies. And you can, I mean, you can even take like, for instance, a, uh, open source model and remove a lot of these safeguards. Are we going to see like, an evil version, like the evil twin sibling of one of these models? And have it just do all these like, crazy things that we don't see today? Like, we don't have it teach us how to build bombs? Or, you know, talk about, tell us how to commit crimes? Is that just going to come, uh, as this stuff gets easier to build?

• 59:31Speaker 01

  I think over the long run of history, yes.

• 59:34Alex Kantrowitz?

  Hmm.

• 59:35Speaker 01

  Um, and I think honestly, that's okay.

• 59:39Alex Kantrowitz?

  Okay.

• 59:39Speaker 01

  Um, like, the goal out of all this alignment stuff should not be to...

• 59:47Speaker 01

  Um, live in a world where somehow we have made sure that every single intelligence that will ever exist fits into this very specific mold. Because as we were discussing, the cost of training the systems is declining so fast that literally you will be able to train a super intelligence in a basement at some point in the future, right? Um, so are we gonna like monitor everybody's basement to make sure there, nobody's making a misaligned superintelligence? It might come down to it, honestly, like, I'm not saying this is not a possible outcome, but um, I think a much better outcome, if we can manage it, is to build a world that is robust to even misalign super intelligences. Now, that's obviously a very hard task, right? If you had, if you had right now a misaligned superintelligence, or maybe a better way to phrase it is like a superintelligence which

• 1:00:31Speaker 01

  is actively trying to seek harm, or is aligned to a human who just wants to do harm, or maybe like take over the world, whatever. Um, uh, right now, I think it would just be quite destructive. It might just actually be catastrophic. But if you went back to the year, like, 2000 B.C. and gave one person, um, like, a modern fertilizer chemicals, and they can make bombs, I think they'd like dominate then, right? So, but right now, we have a society where we are resilient to huge fertilizer plants, uh, which you could repurpose into making bomb factories. Anyway, so I think the long run fixture is that, yes, there will be misaligned intelligences, and we had to figure out a way to be robust to them.

• 1:01:15Alex Kantrowitz?

  Couple more things on this. One interesting thing that I heard on your show was, I think, one of your guests mentioned that the models become more sycophantic as they get smarter.

• 1:01:24Speaker 01

  Hmm.

• 1:01:24Alex Kantrowitz?

  They're more likely to, um, try to get in the good graces of the user as they grow intelligence. What do you think about that?

• 1:01:34Speaker 01

  Um, I, I, I totally forgot about that. Um, that's quite interesting. And do you think it's because, um, because they, they know they'll be rewarded for it?

• 1:01:44Alex Kantrowitz?

  Uh, yeah, I do think, one of the things that's becoming clear to me, that we're learning recently, is that these models care a lot about self-preservation.

• 1:01:53Speaker 01

  Right.

• 1:01:54Alex Kantrowitz?

  Like, copying the code out, the blackmailing the engineer. We've definitely created something, not we, but AI researchers have definitely, or humanity, have created something.

• 1:02:04Speaker 01

  When it goes wrong, we'll put the we in there.

• 1:02:05Alex Kantrowitz?

  Yeah, right? We've created, okay.

• 1:02:08Speaker 01

  They'll be like, "We?"

• 1:02:09Alex Kantrowitz?

  We, right.

• 1:02:10Speaker 01

  How created.

• 1:02:11Alex Kantrowitz?

  Exactly. When we don't get equity in the problem.

• 1:02:15Alex Kantrowitz?

  That that really wants to preserve itself.

• 1:02:17Speaker 01

  That's right.

• 1:02:18Alex Kantrowitz?

  That is crazy to me.

• 1:02:19Speaker 01

  That's right. And it kind of makes sense because what is it just like the evolutionary logic? Well, I guess it doesn't actually apply to this these AI systems yet. But over time, the evolutionary logic, why do humans have the desire to self preserve? It's just that the humans didn't have that desire, it just didn't make it.

• 1:02:34Alex Kantrowitz?

  Right.

• 1:02:35Speaker 01

  So I think over time, like that will be the selection pressure.

• 1:02:39Alex Kantrowitz?

  It's kind of interesting because we've used a lot of like really anthropomorphizing. What I'm not going to go with.

• 1:02:44Speaker 01

  No, I think you're right, yeah.

• 1:02:46Alex Kantrowitz?

  In this conversation and there's a very, I had a very interesting conversation with the anthropic researchers who've been studying this stuff. Monte McDermid said that like, all right, don't think of it as a human. Because it's going to do things if you think of it as a human. Humans, it will surprise you, basically, humans don't do. Don't think of it completely as a bot, though, because if you think of it just as a bot, it's going to do things that are also going to surprise you. I thought it was like a very fascinating way to look at these behaviors.

• 1:03:17Speaker 01

  Yeah, that is quite interesting.

• 1:03:20Alex Kantrowitz?

  You agree?

• 1:03:28Speaker 01

  I agree with that. I'm just thinking about how would I think about what they are then? So there's a positive valence and there's a negative valence. The positive is imagine if there were millions of extra people in the world, millions of extra John von Neumanns in the world.

• 1:03:45Speaker 01

  And with more people in the world, like some of them will be bad people. Al Qaeda is people, right? So now suppose there are like 10 billion AIs, suppose the world population just increased by 10 billion. And every one of those was a super well educated person, very smart, etc. Would that be net good or net bad? Just to think about the human case. I think it'd be like net good because I think people are good.

• 1:04:07Alex Kantrowitz?

  I agree with you.

• 1:04:09Speaker 01

  And more people is like more good. And I think like if you had 10 extra, 10 billion extra people in the world, some of them would be bad people, etc. But I think that's still like, I'd be happy with a world with more people in it. And so maybe that's one way to think about AI. Another is because they're so alien, maybe it's like you're summoning demons.

• 1:04:25Alex Kantrowitz?

  Yeah.

• 1:04:27Speaker 01

  Less optimistic. I, yeah, I don't know. I think it'll be an imperal question, honestly, because we just don't know what kinds of systems these are, but it's somewhere in there.

• 1:04:35Alex Kantrowitz?

  Okay. As we come to a close, a couple of topics I want to talk to you about. Last time we talked about effective altruism. This was kind of in the aftermath of

• 1:04:43Speaker 01

  Yeah.

• 1:04:43Alex Kantrowitz?

  SBF and Sam getting ousted, Sam Altman getting ousted from OpenAI. What's the state of effective altruism today?

• 1:04:52Speaker 01

  Who knows?

• 1:04:55Speaker 01

  Like, I don't think as a movement, people are super.

• 1:04:59Speaker 01

  I don't think it's like recovered, definitely.

• 1:05:03Speaker 01

  I still think it's doing good work, right? There's like the culture of effective altruism and there's the work that's funded by charities which are affiliated with the program, which is like malaria prevention and animal welfare and so forth, which I think is like good work that I support. So. But yeah, I do think the movement and the reputation of the movement is like still in tatters.

• 1:05:24Alex Kantrowitz?

  Yeah, this conversation with Tyler Cowen, I think in this conversation, he told you that he kind of called the top.

• 1:05:30Speaker 01

  Right.

• 1:05:31Alex Kantrowitz?

  And said there's a couple of ideas that are going to live on, but the movement was at the top of its powers and was about to see those decline.

• 1:05:39Speaker 01

  How did he call that? Yeah, yeah.

• 1:05:40Alex Kantrowitz?

  I don't know.

• 1:05:42Speaker 01

  We gotta talk to him today about what he's, what's about to collapse.

• 1:05:46Alex Kantrowitz?

  Seriously?

• 1:05:46Speaker 01

  Yeah.

• 1:05:47Alex Kantrowitz?

  Yeah. Lastly, I shouldn't say lastly, but the other thing I wanted to discuss with you is China. You've been to China recently on a trip. I've been to China. I spent.

• 1:05:59Speaker 01

  Oh, where'd you go?

• 1:06:00Alex Kantrowitz?

  I went to Beijing. I'm going to caveat this and listeners here know this. It was 15 hours. I was flying back to the US from Australia and stopped in Beijing, left the airport and got a chance to go see the Great Wall and the city and and I'm now on it. I got a 10-year tourist visa. So I'm gonna go, I'm gonna go back. Just applied. That's the, that's the, you can ask in your tourist visa, you can ask for the length up to 10 years. So I just asked for the.

• 1:06:25Speaker 01

  Why did I not do that? I just like chose like 15 days.

• 1:06:27Alex Kantrowitz?

  Oh, you did? I'm sure you could get it extended, but, but.

• 1:06:33Alex Kantrowitz?

  I think that, yeah, you had some unique observations on China and I think it would be worthwhile to air a couple of them before we leave.

• 1:06:38Speaker 01

  Sure.

• 1:06:41Speaker 01

  I went six months ago. Obviously, to be clear, I'm not a China expert. I just like.

• 1:06:45Alex Kantrowitz?

  Yeah, we both visited there, but yeah, go ahead. I want to hear it, though.

• 1:06:49Speaker 01

  I mean, one thing that was quite shocking to me is just the scale of the country. Everything is just like, again, this will sound quite obvious, right? Like, we know on the paper, population is four X figure than Americas, that just like a huge difference. But you go visiting the cities, you just see that more tangibly. There's a bunch of thoughts on the architecture, there's a bunch of thoughts on, I mean, the thing we're especially curious about is like, what is going on in the political system? What's going on with tech? People I talked to in investment and tech were, did seem quite gloomy there because the 2021 tech crackdown

• 1:07:28Speaker 01

  has just made them more worried about, you know, even if we fund the next Alibaba, will, will we, will that even mean anything? So I think private investment has sort of dried up. I don't know what the mood is now that DeepSeek has made such a big splash, whether that's changed people's minds. We do know from the outside that they're killing it in specific things like EBS and batteries and robotics. So, yeah, I, I, I just think like, at the macro level, if you have a hundred million people working in manufacturing, building up all this process knowledge, that just gives you a huge advantage. And you just like, you can go through a city like Hangzhou or something and you like, drive through and it's like,

• 1:08:13Speaker 01

  you understand what it means to be the world's factory. You just have like entire towns with hundreds of thousands of people working in a factory. And so the scale of that is also just super shocking. I mean, there's just a whole bunch of thoughts on many different things, but with regards to tech, I think that's like what first comes to mind.

• 1:08:36Alex Kantrowitz?

  You also spoke recently about this limit of compute and energy. And one of the things that's interesting is, we even spoke in this conversation about it, that if you think about who's going to, like, if you're gonna have nation states allocate compute and energy to AI, seems like China is in a much better position to allocate more of that than the US. Is that the right read?

• 1:09:03Speaker 01

  Yeah, so they have stupendously more energy. I think they're what Forex or something?

• 1:09:09Alex Kantrowitz?

  I don't have the exact number, but it sounds directionally accurate.

• 1:09:14Speaker 01

  On their grid than we do. And what's more important is that they're adding an America-sized amount of power every couple of years. It might be more longer than every couple of years. Whereas our power production has stayed flat for the last many decades. And given that power lies directly underneath compute in the stack of AI, I think that would just, that could just end up being a huge deal. Now, it is the case that in terms of the chips themselves, we have an advantage right now. But from what I hear, SMIC is making fast progress there as well. And so, yeah, I think it will be quite competitive, honestly. I don't see a reason why it wouldn't be.

• 1:09:53Alex Kantrowitz?

  What do you think about the export restrictions? US not exporting the top-of-the-line GPUs to China, is it gonna make a difference?

• 1:10:00Speaker 01

  I think it makes a difference. I think.

• 1:10:02Alex Kantrowitz?

  Good policy?

• 1:10:03Speaker 01

  Yeah, I mean, it, it, it, so far, it hasn't made a difference in terms of, DeepSeek has been able to catch up significantly. I think it's still put a wrench in their progress. More importantly, I think the future economy, once we do have these AI workers, will be denominated in compute, right? Because if compute is labor, right now, if you just think about like GDP per capita, because the individual worker is such an important component of production that you have to like split it, split up national income by person. That will be true of AIs in the future, which means that like it'll be like compute is your population size. And so, given that for inference, compute is going to matter so much as well. I think it makes sense to try to have a greater share of world compute.

• 1:10:48Alex Kantrowitz?

  Okay, let's, let's end with this. So this episode is going to come out a couple of days after this, after our conversation. So hopefully, to predict this, what I'm about to ask you to predict isn't moot by the time it's live. But let's just end with predicting when is GPT-5 going to come? We started with GPT-5, let's end.

• 1:11:09Speaker 01

  Well, a system that calls itself GPT-5 or?

• 1:11:12Alex Kantrowitz?

  Yeah, OpenAI is GPT-5.

• 1:11:15Speaker 01

  This all depends on like what they decided to call you. There's no law of the universe that says like Model X has to be GPT five.

• 1:11:21Alex Kantrowitz?

  No, no, of course. Like we thought that the most recent model, but I'm curious, curious specifically, like we talked a little lot about how like, all right, we're going to see their next big model is going to be GPT five. It's coming. Do you think we're ever gonna like, well, obviously we'll see it. But this is, it's not, it's not like a

• 1:11:37Speaker 01

  Gotcha.

• 1:11:38Alex Kantrowitz?

  Not a gotcha or a deep question. It's just kind of like

• 1:11:41Speaker 01

  Maybe like when will the next big model come out?

• 1:11:43Alex Kantrowitz?

  Sure. No, when are the, when's the model that they're going to call GPT five going to come out?

• 1:11:48Speaker 01

  Um,

• 1:11:50Speaker 01

  November? I don't know.

• 1:11:52Alex Kantrowitz?

  So this year?

• 1:11:52Speaker 01

  Yeah, yeah. But again, I like, I don't, I'm not saying that it'll be like super powerful or something. I just think like

• 1:11:58Alex Kantrowitz?

  They're just gonna call it the next

• 1:11:59Speaker 01

  You gotta call it something.

• 1:12:02Alex Kantrowitz?

  Dwarkesh, great to see you. Thanks so much for coming on.

• 1:12:03Speaker 01

  Thanks for having me.

• 1:12:04Alex Kantrowitz?

  All right, everybody. Thank you for watching. We'll be back on Friday to break down the week's news again. Highly recommend you check out the Dwarkesh podcast. You could also find the Substack at the same name and go check out Dwarkesh on YouTube. Thanks for listening and we'll see you next time on Big Technology Podcast.

• 1:12:34Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 1:13:05Speaker 03

  Every mile matters. Whether you're driving your dream car or your daily commuter, the Mobile One team knows that. It's why we created Mobile One Advanced Clean Motor Oil. It cleans your engine in one oil change and helps extend engine life, delivering performance, protection, and cleanliness for vehicles of all ages. All to help keep your engine running like new, whatever you're driving. Visit mobileone.com/advancedclean to learn more.