AI Scaling, Alignment, and the Path to Superintelligence — With Dwarkesh Patel

Dwarkesh Patel is the host of the Dwarkesh Podcast, where he's interviewed Mark Zuckerberg, Ilya Sustkever, Dario Amodei, and more AI leaders. Patel joins Big Technology to discuss the current state and future trajectory of AI development, including the potential for artificial general intelligence (AGI) and superintelligence. Tune in to hear Patel's insights on key issues like AI scaling, alignment, safety, and governance, as well as his perspective on the competitive landscape of the AI industry. We also cover the influence of the effective altruism movement on Patel's thinking, his podcast strategy, and the challenges and opportunities ahead as AI systems become more advanced. Listen for a wide-ranging and insightful conversation that grapples with some of the most important questions of our technological age. ---- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Want a discount for Big Technology on Substack? Here’s 40% off for the first year: https://tinyurl.com/bigtechnology Questions? Feedback? Write to: [email protected] Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  One of the sharpest minds in AI joins us to look at the research, the business, the dangers and the conspiracies, all coming up right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, head of enterprise business solutions at Voya Financial, to go inside a large-scale agentic CX transformation, including the business decisions, trade-offs, and lessons along the way. You can watch both conversations now on my YouTube channel today.

• 0:44Alex Kantrowitz?

  It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We're joined today by Dwarkesh Patel. He's the host of the Dwarkesh Podcast, and he's had DeepMind CEO Demis Sabas on recently,

• 1:26Alex Kantrowitz?

  along with Anthropic CEO Dario Moda, the OpenAI chief scientist Ilya Sutskever, and recently Metas CEO Mark Zuckerberg. So that's the person we're dealing with here today. Dwarkesh, welcome to the show.

• 1:39Speaker 01

  Thanks for having me. Super excited to be on, Alex.

• 1:42Alex Kantrowitz?

  I think that you've been doing some great interviews on AI, looking at the research, where things are going, and really asking the right questions to the right people. So you've interviewed Ilya Sutskever, OpenAI's chief scientist, or really I should say, it's former chief scientist after he parted ways with the company yesterday. How important was Ilya to OpenAI's ability to stay competitive in the AI field?

• 2:05Speaker 01

  I actually think that's a really interesting question. I remember I was chatting with someone and they said something along the lines of, well, now that they've lost Delia, you know, I think the great people matter a lot and that since this person is lost, it might be downhill for OpenAI. And then, you know, I think like the the default perspective is, listen, you've got thousands of scientists who are doing AI. Surely any one of them is replaceable. I think that's probably correct, but I'm not in the field enough to know that. And it is a sort of interesting empirical question of how, what is the buzz factor of a place like OpenAI? If you lose a chief scientist, how much does that slow down your progress? And it would be interesting if it doesn't slow it down that much. It would be super interesting

• 2:50Speaker 01

  if it slows it down a lot. But yeah, that's a really interesting question.

• 2:54Alex Kantrowitz?

  Have you seen any signs of them slowing since his departure?

• 2:58Speaker 01

  No, I mean, the big question people have had is since GPT-4, which was released more than a year ago, the, we haven't really gotten anything better, right? So we've gotten Claude 3, we've gotten Gemini. They're not significantly better, if at all, than GPT-4 and certainly not the newer version of GPT-4. So the question is, is AI progress plateauing? Or are people just waiting to build out the giant data centers, which are necessary for training a GPT-5 level model? And actually, I think this year will be super interesting in terms of learning about AI, because by the end of this year, we'll get to see, hopefully, what a GPT-5 level model looks like. And we'll learn whether we're on the path to some kind of superintelligence.

• 3:41Speaker 01

  If GPT-5 is so amazing, we're like, okay, well, we're building a god if Q is out. Or if GPT-5 is not that much better than 4. And I think the main thing we're going to learn is between 4.5 and 5 level models, we're going to hit what's known as the data wall, which is to say that as you make these models bigger, you need more and more data to keep training them. And we're running out of internet data. And so we're going to learn whether synthetic data, RL, these other techniques can substitute for the data that bigger models will need. And I mean, by the end of this year, I'm expecting to learn a lot about what the course of AI is going to look like.

• 4:17Alex Kantrowitz?

  What is your sense as to what the answer will be with GPT-5?

• 4:21Speaker 01

  Here's here's some predictions I have. I'm not sure if that gets to the heart of what will be impressive about it. I think it'll definitely be better at reasoning, which is trivial to say, because the training methods that we've seen them talk about, like you absolutely heard the talk about Q star and what it seems to be is

• 4:40Speaker 01

  training the model to rewarding it on getting the correct reasoning trace to get the right answer. And that seems to lead to better reasoning, or at least in pub, there's equivalent papers that are released publicly called Quiet Star that claims that it does, then we're going to see much more multimodal data. And I think that'll look a lot like the equivalent of supervised fine tuning, but for a bunch of people recording their screen and doing workflows with their screen navigating UIs. So I think you'll have agents that can act coherently as your assistant for potentially minutes, if not hours on end, in a way that you can just tell them to go do something on the internet and they can actually do it.

• 5:22Speaker 01

  Where like GPT four right now, web browsing isn't really a big feature. Like people like perplexed to use it, but it's mostly to summarize. It's not to like actively go out and look for information. So I'm hoping, well, I'm expecting that to be one of the things you see with GPT five. I guess the big question is like, how much smarter is it, right? Like I'm mentioning all these off abilities it might have, but like, how much juice will it have? And I honestly don't know.

• 5:45Alex Kantrowitz?

  Right. And so how do you think we're going to be able to assess that? Like, is it just going to be like how much better it does on tests? Or do you think it's just going to be a feel when people are using the model?

• 5:55Speaker 01

  Yeah, I trust the feel more, honestly.

• 5:57Alex Kantrowitz?

  So interesting.

• 5:58Speaker 01

  We have these evaluations, but what's your sense on them? I mean, like people will come out and say, here's what we got on MMLU and so on. But they're getting saturated and they're not often not that great to begin with. So I'm more eager to see what it feels like to talk to one of these things than learn what its MMLU score is.

• 6:18Alex Kantrowitz?

  Yeah, I agree. I mean, it is interesting how feel plays such a big role into evaluating these models because they are, you talk about this a lot, scientific, right? And we can sort of test them scientifically and we have these big things like parameters and you know, the size of the compute that we use. But ultimately feel is like one of the great things that we use to test how good these models are. And in fact, like the thing that people always look at in terms of model performance is this chatbot arena where they put two answers from different chatbots side by side and say, okay, well, which one is the best? And that's kind of, it seems janky, but it's also the thing that people take almost as gospel now in the AI industry.

• 7:00Speaker 01

  Yeah. And in fact, I think even chatbot arena has some deficiencies in terms of evaluation because from what I understand, you're doing these pairwise comparisons, but you're doing them, you ask a question and two of them respond. And what I'm more curious about is what is it like to have a long form conversation with this thing where it's not just what is the immediate response like, but if we keep talking, you know, can you kind of like understand the context of the problem I'm talking about? Can you keep following up on different threads? That's more relevant to my workflow than just immediately producing some bullet points.

• 7:35Alex Kantrowitz?

  Yep. We're going to get into this evaluation and research a little bit more as we keep going. But just to go back to OpenAI, it is interesting to watch Microsoft now start to develop their own models, almost to compete with OpenAI. And there was a story in the information recently that Microsoft is working on a 500 billion parameter model. Just for context, this is kind of like the size of the model. And OpenAI's GPT-4 was apparently trained on something like a trillion parameters or apparently has a trillion parameters. But Microsoft is now making this move where it's trying to build, I think for the first time since GPT-4, its own model that competes with it. How should we read into what Microsoft is doing there?

• 8:17Alex Kantrowitz?

  Do you think it is a loss of faith in OpenAI, a hedge against OpenAI, sort of a necessary move, even though it has such an important partner? What's your take?

• 8:28Speaker 01

  Yeah, one of my friends who works at DeepMind told me that Microsoft is basically reversing what Google has managed to do over the last few years, and in fact, making the same mistake that Google initially made, which was to have its training distributed or split up between two different corporations or institutions. For Google, it was a brain, Google Brain, and DeepMind. And so Microsoft has a company which is in the lead, right, OpenAI. And I guess instead of doubling down on it, they're trying to hedge their bets in this way. I think, if you think that OpenAI is like another product where you have multiple vendors,

• 9:08Speaker 01

  so you can be sure that if one of them, you know, has, decides to go to a different route, you have some leverage over them, that that might make sense for another kind of product. The thing with AI is, if you buy scaling in this picture, that is, you make the models bigger, they get much smarter, then I don't think it makes sense to hedge your bets in this way. I think you should just double down, give give one of them $100 billion and just say, like, go make me, go make me super intelligent, you know what I mean? Like, because then you're just splitting up your efforts. And yeah, like, it would be much better to have one GPT-4 than two companies that have, you own two companies that have a GPT-3.5.

• 9:44Alex Kantrowitz?

  And I'm sure Microsoft knows this. So what do you think could possibly be their reasoning for doing what they're doing?

• 9:54Speaker 01

  Partly, I think they got spooked by what happened with the board last December and November. I think definitely that there's that, because the clause in the OpenAI charter says that if the board, which is nonprofit and controls the company, decides that we've built AGI, then Microsoft has no leverage over OpenAI anymore. So they got spooked by that. And I think partly it's probably Microsoft doesn't buy the scaling hypothesis as much as us internet weirdos, because they probably think like, oh, you know, we want to like, we want to diversify our bets here, and we'll have multiple companies build a GPT-4.5 level model, and we'll see how that goes. I mean, I'm sure there's better reasons

• 10:36Speaker 01

  too, right? Like, you can build this in-house talent. I'm sure there's a lot of practical knowledge you understand by building smaller models, and you're getting a lot of that knowledge in-house by training these models. And so Microsoft's own ability to train and understand and deploy these models improves. Yeah.

• 10:53Alex Kantrowitz?

  Can you just handicap the field for us? I mean, how should we think about the efforts of DeepMind versus Meta versus Anthropic versus OpenAI? Is there a clear leader there? Or is there any sort of key differentiators that are important to know? I know it's like a broad question, but feel free to zero in as you'd like.

• 11:12Speaker 01

  Yeah, that's a good question. It doesn't seem like there's a strong leader at the moment. I think in terms of revenue, probably OpenAI is leading by a lot. I think just the amount of people who use ChatGPT versus any other service. But I mean, subjectively, you use Claude and it's often better, not significantly worse in any case. I think the main way, at least in which they're today different, is Claude seems to have better post-training, which is to say, which is the jargon for basically saying like, what kind of personality does it have? And how does it, how does it break down your question? And how does it act, how does it act as a persona of a chatbot? And so all this RLHF stuff you hear is part of the post-training. And Claude has a more automated way of doing that. Or Anthropic, which controls Claude does.

• 11:58Speaker 01

  Gemini has longer context, obviously, but so, you know, million tokens, which is huge. I think the big thing we'll probably learn in the next few months is who's really ahead. Because everybody's just been releasing models that are as good as GPT-4 right now. And we'll learn who can go the distance, so to speak. And I think the big question will be, OpenAI probably has the compute because of Microsoft. But then again, maybe Microsoft is splitting up its compute. Google has, definitely has a compute because, you know, Google, a huge company. The question is, I guess we don't know yet whether Anthropic can keep up beyond this year with models that might cost tens of billions of dollars.

• 12:35Alex Kantrowitz?

  Right, and I think that's sort of the underappreciated part of Google's attempt here is that they are doing this for first Microsoft, correcting their mistakes like we talked about, where they've brought Google Brain and DeepMind together under one organization and said, you know, resource, you know, your internal conflicts be damned. Resources are going to be pooled right now.

• 12:59Speaker 01

  Yeah, you know, I interviewed the guy who wrote the Making of the Manhattan Project, or sorry, the Making of the Atomic Bomb, Richard Rhodes, and he was telling me when I interviewed him about Soviets after Hiroshima and Nagasaki, Stalin called his top physicists and he said, "Comrade, you will give us the bomb. You have the entire resources of the state at your disposal. You will give us the bomb, or you and your family will be camp dust." And I'm sure the last sentence wasn't uttered inside of Google, but that maybe the attitude they've taken in terms of their compute allocation might be much in favor of, "We're going to take this seriously. We're going to invest a bunch of compute into making this happen." And also, I think you shouldn't ignore the fact that Google is the company that actually has a

• 13:43Speaker 01

  successful accelerator program for AI chips already with their TPUs, which other companies are trying but don't yet have to replace Nvidia GPUs.

• 13:52Alex Kantrowitz?

  Right. Okay. Lastly, XAI, which is Elon Musk's effort. Do you think that there's any chance that they can be competitive here? I mean, we think about resources and they definitely have Musk's money and they have, oh, I think there's like this big GPU cluster that Tesla has. So I wonder if that's going to factor. And then is there any other dark horse that might come in and start to matter?

• 14:17Speaker 01

  I think the second part of your question is super interesting. I mean, on XAI, I honestly don't know. I mean, Tesla is separate organization than XAI, so I don't know how much those could transfer, but I have no idea. With regards to who the other actors could be, I think people are in the case where AI really is super powerful, or Jupiter 5 is amazing. I think what happens is a lot of different countries' national security apparatus start to realize what a big deal this is. And they're not just going to sit around for people to like, you know, they're going to make moves. And I think what that looks like is there's a lot of different countries in the world with a sovereign wealth fund with $100 billion, right? And do they all just go around saying like, "What are we doing sitting on this money? Obviously, AI is the thing to do." Each, you know, the UAE spins

• 15:00Speaker 01

  up, which I hear they're already doing, a bunch of data centers in the Middle East to start training even bigger models. And China starts deploying energy infrastructure to build, you know, to do big training runs. So I think in the world where AI continues to get much better at a fast pace, I think you're looking at a much more involved, I guess I'm trying to say the players will be nation-state level players, because that's also the kind of funding you'll need to keep scaling these models.

• 15:28Alex Kantrowitz?

  And what does a nation state do with this technology?

• 15:32Speaker 01

  Yeah, I mean, obviously, the military uses are clear, or at least will be clear, right? You can use it for R and D on military stuff. But I mean, just like basic things like you have a drone operator who's human level, who can, they know you can just mass manufacture millions of drones and like a human equivalent model that's on, running locally can run these drones and you have a million drone swarm headed towards Beijing or something. I don't know, that's just one example, but you can imagine there's a lot of things you can do. I mean, the stepping back, the bigger picture is why are some countries wealthier and more powerful than other countries? Well, it's often because they have more people, right? So Taiwan, Taiwan would lose a war against China. Why is that? Without the help of the US, but because China has more people.

• 16:16Speaker 01

  If AI substitutes for people can increase the effective population of a country, then you can imagine that it would just be a huge leverage that a country would have over other countries in terms of its own economic output or even its ability to withstand geopolitical competition.

• 16:32Alex Kantrowitz?

  You've referred to AI as, you know, potentially like the last invention. I think I might be cribbing a little bit, but what happens if a nation state is the one that achieves artificial general intelligence? And we're going to talk about more, more about AGI in a bit. But let's say, you know, China is able to invent it, then what happens? Are there implications there?

• 16:53Speaker 01

  Totally. I mean, I, for that particular phrase of the last invention, I think belongs to somebody else. But yeah, I think it really matters who wins here. I mean, if China wins, it depends on how fast things happen in the world where they happen within a few years. I think what you're looking at is China has a ton of leverage over the United States because they, one of the things the future AIs could unlock is things like pocket nukes, right? And so you, if China is ahead, they could be like, listen, we've got these, we've just basically got this mass army that would be able to manufacture of billions of extra soldiers and we can build mass manufacturer drones or robots or whatever for them to run on.

• 17:37Speaker 01

  And I think that gives them a lot of leverage, right? So I would worry about that. I think it's important that the US win that and stay ahead. I don't know the status of Chinese AI currently. It seems like the newest model was a deep-seak one. It seems like it was really good, but.

• 17:52Alex Kantrowitz?

  So before we get there, we're gonna have a lot of, I mean, before anyone gets there, the tech industry in particular is gonna have a lot to work through. And you already mentioned a little bit, we have data constraints. We also have compute constraints. So I think we should talk a little bit about, more about like whether these resource constraints are actually gonna be things that that matter and how they might factor. And Mark Zuckerberg spoke with you about, about how energy is gonna be something. And we'll talk about that. And that really opened my eyes to sort of like, oh, are we gonna be like hitting a wall here with AI? And wrote about it in the newsletter, sort of an interesting question. So why don't we just go into the component parts and talk a little bit about each? And the first one, and I think, you know, clearly a very important one is compute. And I start to like raise my antennas here when I hear rumors of Sam Altman wanting to

• 18:37Alex Kantrowitz?

  raise $7 trillion. And of course, that's like an economic question, but it's also like, if that's what it's gonna take to make this stuff work, then are we ever gonna get to the place where a lot of these people wanna get to, talking about like adding more compute and data and energy? And eventually you get to the point where you can train better large language models and see what the scaling law really looks like at its limit. What do you think?

• 19:06Speaker 01

  Yeah, I think compute will be less of a bottleneck than energy. I toured the 7 trillion. Yeah, I imagine, well, backing up the reason I think compute will be a lesser bottleneck than energy is because right now you have one company, Nvidia, which is making the sort of flagship GPUs. And other than Google, nobody has a clear competitor. And so the, the thing that was bottlenecking Nvidia so far is that some of their components that they need for these GPUs, COAS and HBM, they just weren't able to get enough allocation or get TSMC to build facilities for these, because TSMC was like, I don't know if we buy all this AI stuff.

• 19:51Speaker 01

  But, because then they had to make this huge investment into building it out. But now it seems like the fabs are building it out. And also, all these companies have accelerator programs where they're gonna try to ship their own chips. So I think compute will become more and more available. And that's what Zuckerberg said on the podcast, that now the compute constraints are decreasing. Then the question that Zuckerberg pointed to was, well, will there be energy? And the, the key constraint with energy is not necessarily Is there enough energy in the world, but more so for training? Is there enough energy in one place? Because to do a training run, it has to usually, at least from what it seems like publicly, the training methods we have, you gotta do it in one place.

• 20:34Speaker 01

  So if a nuclear power plant releases one gigawatt of energy, can you, and if, you know, training with hundreds of thousands of GPUs would take, would consume one gigawatt of energy, then do you have, can you like get all the energy into one place? So where in America is that place? If not in America, where, where do you go? Do you go to the Middle East? Do you like get some aluminum refinery in Canada? Cause those consume a lot of energy and you can just like buy out the aluminum refinery and turn that offline and I don't know, but you can try out different ideas. But I think energy will be the big constraint.

• 21:06Alex Kantrowitz?

  And you, basically Zuckerberg talked about the fact that you might need a moderately sized nuclear power plant to be able to, to do this. And you asked, well, what about Amazon? And he said, ask Amazon. Then I asked Amazon, did a little research, right? It actually, Amazon actually has purchased a, a nuclear, a small nuclear power plant in Pennsylvania. Correct me if I'm wrong here. And it's 960 megawatts, so close to that gigawatt size. And they're gonna use 40% of the energy there, I, I assume for AI training. So is that sort of what this energy battle looks like moving forward? And is that even enough energy, given that everybody wants to add more compute

• 21:50Alex Kantrowitz?

  and more data and more energy into the process to actually be able to build these models?

• 21:56Speaker 01

  Well, it's certainly not enough. And in fact, I brought that, yeah, I brought that up with Zuckerberg. You need some in one place for the training, it seems like, but they might have ways to get around that. Then you also need to deploy these models and you, those you can, so wherever you deploy the model, it doesn't, you don't need like a huge amount of energy in one place necessarily, but you do need a lot of different places that each consume energy. And that could result in the demand for energy increasing globally. I have, I can pull this up somewhere, but I did some back of the envelope calculations on if you believe the scaling laws and you believe. The you can like just look at how much energy does an H one hundred consume, then you go look at every generation, how much cheaper or how much less energy because of efficiency

• 22:41Speaker 01

  gains do we get in terms of these GPUs? Anyways, you can just like go down the list of and then how much basically would it cost in terms of energy to train a GPT four level model, four point five level model, five, whatever. And you get into the gigawatts pretty soon. And especially if these models are going to be widely deployed in the world, then yeah, it's going to consume a ton of energy.

• 23:00Alex Kantrowitz?

  Well, last thing is is data. You mentioned it right at the start, which is that data might be a major constraint. I mean, these companies are already going to are already working with synthetic data, like to train Lama three meta use synthetic data, like data created from basically, you know, from AI itself. And this is they tried to buy Simon and Schuster, or they talked about buying Simon and Schuster in the in the company when they like were like, we cannot get this to be as good as chat GPT. And by the way, this is Meta, the company that owns Facebook, which has like the entire social internet to train on. So how do we get around or how does the industry get around that?

• 23:36Speaker 01

  Yeah, I mean, the synthetic data thing comes back to energy and compute in a way, right? Because, well, how do you make synthetic data? You use the existing model. What does it take to use the existing model? It takes energy and and compute. So in fact, it will make training more expensive because instead of just doing one backward pass, you now potentially have to do many forward passes because at each forward pass, you're going to come up with some output, then the model has to decide which of those outputs was the best. Now we're going to train on the best of those outputs. It could like be a five X tax on training. So I mean, that's separate from the question of are these model methods scalable enough such that they can make the model smarter? And I don't think we have public evidence of this yet, but.

• 24:22Speaker 01

  I don't know. What's your vibe on this? Because when I talk to researchers at these labs, they seem pretty confident that this will happen. There's no evidence that, I mean, yeah, synthetic data, obviously, the Llama 3, they said they used it and so forth, but actually, like, really making it smarter in a significant way, I guess we don't have that much evidence for it.

• 24:41Alex Kantrowitz?

  I mean, I think I'm learning as we're talking here and sort of thinking about it, thinking it through and being like, okay, so just like, look at the headlines we've seen. Seven trillion dollars for compute. I mean, of course, we might get more efficient. Nuclear power plant for energy. More data than the world possibly has. And then I'm like, how, and we're not quite sure whether scaling these models up, right, adding more energy and more compute and more data into whatever we're training LLMs or the industry is training LLMs, you know, to make them better. We're not sure if that's going to work. And I'm just like, how is this sustainable?

• 25:19Speaker 01

  Right. I mean, the thing you had to add on top of that is, what is the revenue of these AI companies so far? And it's actually, I'm guessing it's not great, like, probably on the order of billions of dollars cumulatively across the industry. And they probably want tens of, I mean, I guess Sam Altman wants seven trillion dollars of winning, but you know what I mean? So like, the difference between how much, so I think it will depend on whether other hyperscalers do big companies like Amazon, Meta, Google, Microsoft buy that this is the path to go on investing a lot of money into. And it seems like they do. Maybe at some point they stop because the models, maybe GPT-5 isn't that much better, and so they lose their patience. And then I guess the nation states never get into it

• 26:02Speaker 01

  either. But then the question fundamentally is, I think in the world where you can get an AGI for $100 billion worth of training, I just can't see why GPT-5 wouldn't be really good and also why people wouldn't continue investing. And in the world where we need much better algorithms or something, yeah, I agree, we might like plateau out around here. But, you know, that goes back to what we were saying earlier about, we'll learn a lot by the end of the year, what trajectory we're on.

• 26:28Alex Kantrowitz?

  Right, and I think that what the industry seems to be betting on is that there's going to be more efficient models, right? They'll just be able to code them up better, and so they won't need, you know, to take as much compute or data, for instance, to improve, even though they will have to expand. And one of the things that's consistent in your interviews and elsewhere from the people in the industry is that they believe that this stuff is predictable, that the scale is predictable.

• 26:54Speaker 01

  Totally.

• 26:54Alex Kantrowitz?

  This was Sam Altman just a couple of weeks ago at Stanford. He says, "We can say right now with a high degree of scientific certainty, GPT-5 is going to be a lot smarter than GPT-4, GPT-6 is going to be a lot smarter than GPT-5, and we are not near the top of this curve, and we kind of know what to do." And this is not like it's going to get better in one area. It's not like, it's not that it's always going to be better at this eval or this subject or this modality. It's just going to be smarter in the general sense. And I think the gravity of that statement is still, like, underrated. Okay, so like, that seems to me to be the case why everybody keeps putting money in to this stuff. It's

• 27:37Alex Kantrowitz?

  not necessarily for what it does today. It's what it can do maybe a couple of generations from now, and that will eventually give you the ROI. I'm curious, I mean, you've had these conversations with these, with these folks, you know, at the, really the ground level of the science. Do you, do you buy this, that it's almost going to be a linear progression in terms of how good it is from generation to generation?

• 28:02Speaker 01

  Yeah, I mean, like, one way to think about it is, you're mentioning the scaling laws, and that's a relationship that, basically, as you dump more compute in these models, their loss gets better in a very predictable way, and the loss in this case corresponds to their ability to protect internet text. How that translates into capabilities is another question, but if imagine a model that can predict any internet text, it can predict how to write like really great scientific papers or whatever. Well, then that's like, you know, it's like human level intelligence. Anyway, so I, I think you could make the case that there might be some break or plateau previous to G P D three or something where

• 28:44Speaker 01

  G P D two is really impressive, but here's the kinds of things it won't be able to do and sort of pre-register that prediction and stick by it. It would just be really bizarre to me that G P D one to G P D two, G P D two is actually kind of really interesting artifact for the small amount of investment it took to make it. G P D three, a couple million dollars and you've got like this, like hu, this thing that's like early stages of intelligence, whoa. Then you get a G P D four and oh my God, this is actually useful. They can generate billions of dollars of revenue a year. It would just be bizarre to me that like you're halfway through the human range of intelligence and now it stops getting better. So I do sympathize with Sam's statement in the sense of like, why would it stop here? Right. If it was going to stop, why? It would, it seems like it should have stopped before it started getting better in a human intelligence way at all.

• 29:28Alex Kantrowitz?

  So you don't think we're going to hit this, this stop in the road?

• 29:33Speaker 01

  I, well, the, the, the, the reason that could happen is because of the data wall, which is to say that we can't keep training them in the same way we have been training them before. You G P D two to G P D three to G P D four, you can just dump more data and compute in these models. If you run out of more data, then, then the question is like, well, you know, we could have made something smarter, but we just didn't have the data for it.

• 29:54Alex Kantrowitz?

  Right.

• 29:55Speaker 01

  And then the operative question becomes synthetic data or R L. And I think the intuition there of why that will work is first of all, that should work better once the models are smarter because the sort of self-play setup is contingent on the model being smart enough to be like, that was the wrong way to proceed. Let's back up and proceed to this other way and let's learn from this. Why did I make this mistake? Let's make sure to do it the right way in the future. That seems like they're getting smart enough to be able to do that. Um, on a per token basis, they're actually really smart, potentially as smart as, uh, really smart humans. It's just that five minutes out, they lose their train of thought. Can they bootstrap themselves in a way to like, help them back up every five minutes and learn how to do that so that they can stay coherent for longer?

• 30:40Speaker 01

  Seems plausible. Um, and I mean, I had one of these takes on one of my blog posts that. The way humans got better was this sort of self-play setup as well, right? Where we learned language and or at least the initial stages of language where our vocal cords got something called a Fox P2 gene. And then so from there, just you can talk to other humans. You can interact with them. That's sort of like a self-play loop that led to humans getting smarter and so forth.

• 31:10Alex Kantrowitz?

  Do you ever think it's weird that there is this belief in this predictability of improvement? And yet, when you speak with the people who are working on these projects, they tell you that they don't really know why it's making that improvement. Like Dario Modo was basically like, I'm not quite sure what's going on inside these LLMs to make them as smart as they are.

• 31:30Speaker 01

  Totally. And I think that's where that's why you should have. You shouldn't be like sure that they're gonna we're on the attractive AGI because yeah, fundamentally, we don't know what kinds of things these are. You know, it could just be, I don't know. It's less, it's more impossible now than it was maybe a couple of years ago, but it could be some sort of curve fitting thing. Where I think if you ask me like, what is the reason AI progress? Like, looking back on it, if let's say GPT-6 isn't that much better than GPT-4? And you had to look back on it and say like, why? Why did that happen? I think the most reason, the thing I'd expect to say is that right now we are

• 32:10Speaker 01

  we are kind of fooled by how much data these models consume. Whereas, you know, they've like literally seen all of internet text and trained out it multiple times. And then in retrospect, you'd be like, well, of course, they knew how to do the nearest adjacent thing because it was in the data set. But they, you should have seen that they're not that good at being creative or novel. And so, yeah, clearly they weren't going to keep improving in that way.

• 32:36Alex Kantrowitz?

  So we've talked about AGI a couple of times. Artificial general intelligence is this big phrase that's thrown around a lot. And I think that oftentimes people hold multiple definitions of it in their brain at the same time. And it's definitely something that's kind of one of the more amorphous finish lines, so to speak, that you've ever seen in the business world, that everybody seems to be working toward it, but no one can really define it. What is your definition of it? And do you think that we're going to reach it?

• 33:10Speaker 01

  Um, the, the, the way I've been thinking about it, which is, which has less to do with like, maybe AGI in the world and more so about, I think it's long-term impact is the kind of model which can automate or significantly speed up AI research? And why define it that way, um, given the fact that there's so many other jobs in the world? Because I think the, one of the things you really had to think about is once it can automate AI research, then you can have this sort of feedback loop where it's helping train the next version, but it's like looking at finding better activation functions and like, um, you know, designing architecture that has better scaling curves, becoming over better synthetic data and so

• 33:54Speaker 01

  forth. So, uh, I think once you get to that point, then it's off to the races in the sense of like, you could, you have an intelligence explosion, that the kind of things that you see in sci-fi books. Um, and so that's why, that's what I've been thinking about when I think in terms of AGI can it speed up AI research. Um, and yeah, I, I think that's like a plausible thing within the next five to 10 years.

• 34:15Alex Kantrowitz?

  Are people working on that problem in particular?

• 34:18Speaker 01

  I mean, the, the people making these models are AI researchers themselves, and I can imagine them being selectively trying to, um, clearly they care about their use case, which is helping them with their job. So I can imagine the model getting better at that than it gets better at other things.

• 34:32Alex Kantrowitz?

  Fascinating. So what break, what type of breakthroughs do you think we're going to need to get there? Right? We talked a little bit about reasoning and from my understanding that the way that models do reasoning is kind of like, look at a task instead of, I mean, this is what Jack Clark told us a couple of weeks ago, look at a task instead of just like spit back information, be like, huh, like, how many steps do I need to perform to like really get this task right? And then just go step by step. Is that one way that they're, that we're going to get there or that they'll get there? Or is there something else that's going to happen?

• 35:02Speaker 01

  Yeah, I agree. I agree. That definitely seems like an important component of the puzzle.

• 35:08Speaker 01

  One, one big one, and this is similar, is that they aren't yet useful in long,

• 35:18Speaker 01

  when you need them to kind of go do a job, you can't be like GPT-4, I'll be back in a while, but can you like manage my inbox for me in the meantime? Or can you go, uh, go book a trip for me? You know, just like things that require them to sort of autonomously hold themselves together for a while and act as an agent. And so just that kind of coherency where they go from five minutes of being able to be in dialogue with you to you just like tell them to do something, you come back a couple hours later, and they've just done a bunch of inference to make it happen. Um, I think that will be a huge bottleneck or a huge unlock, I mean.

• 35:53Alex Kantrowitz?

  Yeah, I mean, this idea of memory in the bots, right? Like it's something, and maybe this is a little different, but it's something that I keep thinking about where like I'm speaking with Claude every day, and yet every morning it's like 50 first dates. I have to introduce myself to Claude again. And although there's one weird thing that Claude started doing where like I did a podcast last week about, or a couple of weeks ago, about, um, the data that you get from voice and the emotion that you get from voice when you can listen to something as opposed to just have text. And Claude, obviously, when it gets a transcript of a podcast is only getting the text of the voice. And so I uploaded that conversation about the data that you get from emotion into Claude.

• 36:34Alex Kantrowitz?

  And now it keeps hallucinating the audio quality of further transcripts that I've put in, almost as if like it wishes it understood what the audio sounded like because it knows that that's an important data point. But anyway, let's just put that aside for a moment. The memory thing is interesting. Do you think there are easy ways to then like have, have a persistent conversation with one of these bots? Or is that going to be like another tough problem that we won't solve for a while?

• 37:03Speaker 01

  I it it could plausibly be very tough because I don't think it's a memory. It's a it's a matter of just keeping like storing memory. I think it's like, what kind of thing are you? And are you a chatbot? Or are you is your persona like I am? An entity that you know, it's it's not just about like I'm storing these things somewhere. It's like you have to train it to act as an agent. And compared to just pre-training tokens on the internet where yeah, it knows how to complete statements, does it know how to act as an agent? There's not necessarily a good way to structure that. So people have been talking about long horizon RL, which is the training method. You need to get something like this where you go tell it to do something and then you

• 37:48Speaker 01

  reward it at the end for having achieved that outcome. But the difficulty with those kinds of approaches and the difficulty with RL in general is sparse reward and non-stationary distributions, which is like you failed to book me my right the right appointments based on like reading all my inbox and like talking with me about it. Why did you fail? There's like so many different reasons you could have failed. It's hard to attribute to any one of them. You know what I mean? It's like hard to learn from that. It's kind of an interesting question, honestly, like why humans are so good at learning from these sparse rewards or making long-term plans? Because when you look at it from an ML perspective, it's kind of a it's a it's a cursed sort of problem to solve.

• 38:28Alex Kantrowitz?

  Yep. All right. Well, I want to take a break here and then when we come back from the other side, I want to talk about the Dwarkeesh podcast, how you've started it and where it's going. And then particularly, I'm very interested in the AI risks, because that's something that you're you're focused on and it's something that I've like dismissed oftentimes in terms of like looking at the big risks and I've promised myself to do a better job of taking it seriously. So why don't we do that on the other side of this break? This episode is brought to you by AppPoint. Everyone's racing to roll out AI right now. Copilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data. And most teams have no single way to see it,

• 39:13Alex Kantrowitz?

  secure it, and prove it's under control. That's exactly what AppPoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AppPoint, the unifying trust layer for AI. Head to apppoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast.

• 39:58Alex Kantrowitz?

  Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a wide-glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/bigtechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are

• 40:43Alex Kantrowitz?

  and how to lock it down before a threat reaches their front door. That's ironwell.com/bigtechnology. Learn how to stop online threats before they become real-world attacks. The link's in the show notes.

• 40:57Alex Kantrowitz?

  And we're back here on Big Technology Podcast with Dwarkesh Patel. He's the host of the Dwarkesh Podcast. All right, so you recently tweeted your bank account before the advertising checks from the Mark Zuckerberg interview hit. And it reminded me of a similar screenshot from my bank account not too long ago. So you have checking negative $17.56 and savings 14 cents. So congrats on the savings. It reminds me of before the advance of my book hit, I was negative quite substantially in my bank account also. And it's kind of this moment, I think we had similar moments where you're like, okay,

• 41:39Alex Kantrowitz?

  I think things are going to be going on the right direction, go all the way in on this on this content plan effectively, and then trust good things will come. And they have for you. So apparently, I mean, and I saw the tweet and I was like, all right, we're definitely talking this, talking about this on the show. So apparently, things are heading in a good direction for you financial, financial wise. At least that's the sense I get from Twitter. But I really want to know, given that like it got to that point, and now you're making your move, what is your background, Dwarkesh? And what got you interested in starting a podcast largely focused on some of the deepest questions in AI?

• 42:21Speaker 01

  Yeah. Well, I started the podcast in college sophomore year because that's when COVID hit. And all my classes went on offline, and so I was super bored. And then I just, I was super into economics and history at the time. So I invited some, I was emailing some economists and my first guest, actually Brian Kaplan, who's now a good friend, I asked him, well, you know, I'd love to chat with you on the podcast. I didn't tell him I didn't have a podcast. I didn't even have the name for the podcast yet. And then he's like, yeah, sure, I'll come on. And then so we recorded an inaugural episode. And from there, I was super interested in economics, history for a while, for through college, I was mostly doing topics like that.

• 43:03Speaker 01

  And then I graduated about two years ago, still kept doing it. It was honestly, I graduated a semester early, so it was a way of basically taking a gap semester to figure out what I wanted to do. I mean, I was studying computer science, but yeah, I wasn't sure what to do. I didn't want to become a code monkey. In fact, I mean, there's a longer story there, but so yeah, and then things just kept going well in terms of the growth of the podcast itself. And so I was like, well, this seems like a thing worth doing and investing my time into. And it wasn't really like financially making money, but you know, whatever. This seems fun and I'm learning a lot. I'm meeting a lot of interesting people. So kept it going, dot, dot, dot, interview Elias Sescover at some point. And then like other

• 43:46Speaker 01

  cool things happened and then dot, dot, dot some more and then interviewed Mark Zuckerberg. And now I'm getting checks for ads on the podcast.

• 43:54Alex Kantrowitz?

  Yeah, it's a great summary. And I recently listened to you on an Effective Altruist podcast. And I thought it was a great, a great conversation. And it sounded like the Effective Altruist movement influenced you a lot in the beginning to start the podcast, or at least to focus the podcast on artificial intelligence. So what, what did you find interesting about the movement? Do you ascribe to the EA theory and how present is it in your life right now?

• 44:24Speaker 01

  Yeah. I definitely think they've been like right on a lot of things, right in the sense of like, this is a big focus and they've realized it before a lot of other people, like this AI stuff, right? The, the EA's have been talking about this stuff for decades. And like, it's been a big part of the movement for a long time, right? So you gotta give them some credit for that. They were talking about pandemics and bio viruses and the dangers that deposes society from that, long before COVID, a lot of them saw that coming. You know, I'm actually curious about something. Listening to that podcast, what was your sense to like, the kinds of things where we were buying into EA assumptions? Did it feel like we were buying into too many

• 45:07Speaker 01

  assumptions? Did it feel like a reasonable, you know, feel free to be a rec team, this would be harsh, but like, what was your sense on, for somebody outside, I'm, yeah, I'm assuming you're not necessarily in EA. What was your sense of like, were we assuming too many things in that conversation?

• 45:21Alex Kantrowitz?

  Yeah, no, I'm not EA. To me, it was, I don't think I had enough grounding to be able to to really answer that question well. I think EA's should like talk a little bit more is my perspective. And I've tried to reach out to many of them, especially when there was this like six month pause that was funded by Open Philanthropy. They called for the six month pause in AI development and was sort of like met with like a dismissive no on that front. So I think that's kind of where some of the skepticism comes from. But I will tell you this, because I think you saw the screenshot that I I uploaded five of your podcasts into Claude.

• 46:04Alex Kantrowitz?

  And was like, all right, tell me a little bit about what's important to Dwarkesh. And then I was like, do you think Dwarkesh is is effective altruist? And Claude says, there's a high probability that Dwarkesh Patel is an effective altruist, or at least strongly influenced by EA ideas. Based on the strong alignment between his express views and EA priorities, I would estimate the probability is quite high, perhaps in the range of 70 to 90%.

• 46:32Speaker 01

  It's impressive that it gave you a probably like an actual probability number.

• 46:36Alex Kantrowitz?

  Oh, I asked for probability. That was prompted. Yeah.

• 46:40Speaker 01

  Okay, that's that's actually a super interesting use case of these models to like info dump into the long context, a bunch of stuff about them and like, what are the odds that they're like, you know, the believer certain thing or that's actually a really fascinating thing to do. Yeah, look, like I just like to not use like subscribe to labels just because it kind of constrains your ability to

• 47:06Speaker 01

  think like I like I I'm not sure what EA necessarily means. And then also, you know, like you, there's certain things that may be traditionally considered EA that I probably disagree with. But I'm definitely happy to say like, look, the movement influenced me a lot. And I think like they've had some really interesting ideas that I found fruitful and useful.

• 47:27Alex Kantrowitz?

  No, it's interesting. I feel like they are asking some of like the really the right questions about

• 47:32Speaker 01

  Yeah.

• 47:32Alex Kantrowitz?

  this technology, like it is powerful and how do you steer it? And then it all and then, you know, in fairness, they've also been like moments where EA has been associated with some stuff. And people have been like, what? Like, obviously, like Sam Bankman Fried is not a distillation of VA philosophy. But he was a definitely like a firm believer and a funder.

• 47:53Speaker 01

  Totally.

• 47:53Alex Kantrowitz?

  And then, um, you know, with the whole Sam Altman coup, like clearly that was also part of it. So I'm curious like

• 47:59Speaker 01

  I actually don't know how much like, I actually don't think EA was that that big a deal of the board stuff. I think like from what I've heard, it was it was related to something separate.

• 48:07Alex Kantrowitz?

  Yeah, well, I'll say this, the the people who were some of the, I mean, the two, two of the board members who I think were in favor of the ouster were connected in some way. Whether there was a direct like this is an EA thing or not is is still an open question. But that's what I'm saying. Like there was that presence. So I really want to get to like the the core of the question here, which is what are the things that you do disagree with from EA? And then I bring up these examples, not to impugn the movement, but to ask you if you think there are holes in the move in in like the broader philosophy and they've just manifested in these and and at least one strange moment.

• 48:47Speaker 01

  Um, I mean, I could like go go on for days about things I disagree with about them, depending on like what EA definitely necessarily counts as. You can like, look at their cause prioritization list. Like it depends on what in what sense are we disagree like what do we mean EA in terms of like? When you go to effective altruism.org and like what they say their cost priorities are like, it's hard to, you know, they'll say things like we care about. The poorest people on the planet and about animal welfare and about existential risk. And I'm like, yeah, those all seem like good, like things to care, worth caring about, like probably some of the most important things in the world. Maybe we mean like the impacts they've had because of SPF or the board stuff, right? Is it a good sort of scalable culture?

• 49:31Speaker 01

  I'm actually curious like in what sense, because like on this sort of like social cultural angle, there might be other things to say. So I'm curious which angle.

• 49:40Alex Kantrowitz?

  Yeah.

• 49:41Speaker 01

  Yeah.

• 49:42Alex Kantrowitz?

  It's a great, it's a great sort of return question because I think you're right that it has been something that has been a label for a lot of different things. And I don't think there's like a charter, right? So for me, when I think about it mostly, I think it more mostly in terms of like this expected value equation where like people should structure their lives to maximize the expected value that they're going to have on the planet or the expected value of their presence that will have in terms of adding goodness to the world.

• 50:16Speaker 01

  Hmm. Um, yeah, I mean, there's definitely problems with that. And I certainly don't think of like, so here's one particular problem that I was talking about on the podcast is that it's hard to forecast in the case of any individual, how to make decisions using this framework. Like I would have never started the podcast if I was thinking from a maximizing expected value perspective, right? Obviously you could have been like, well, use your CS knowledge to do something more useful, like you're gonna spend your time working on the podcast. Come on. So, yeah, it might not be, it might not be practically, practically useful in many ways. And of course, there's like the dangers of people who think they know what the expected

• 50:58Speaker 01

  value of something is and actually don't, and just having uncertainty over that. On the other hand, I think like at current margins, like how does society currently, if you think of other charities or think your own tax dollars, how are they allocated? And wouldn't you prefer at the current margins, whether if they were allocated using a more sort of rational expected value framework? Like, you know, your taxes are going to be used, like if you live in California especially, there's some used, wasted on a bunch of useless shit. Like all these nonprofits and whatever, and like, wouldn't it be better if like they thought about like, let's, let's, let's pull up the spreadsheets on how much good these, these nonprofits and these like different institutions we are funding with our tax dollars are doing. And I think that kind of mentality actually would be broadly useful in the world.

• 51:42Alex Kantrowitz?

  Yeah, I don't know. I think it's worth considering sort of the longer term risks of AI, right, which you mentioned they were early on, and I think that they've probably brought more focus to. So on that front, I'm curious like what you think makes some, I guess it's the CEOs, a lot of the CEOs that we hear from, and maybe some of like the intellectuals in the EA movement, but elsewhere. What do you think makes them so afraid of AI or so cautious about where it could lead us?

• 52:14Speaker 01

  Yeah, I think it's it's kind of a sort of straightforward thing of like looking at maybe a couple of years out from the people who are just thinking of this as a normal economic transition where you say, okay, we'll have things that are as smart as humans and as smart as as smart as humans when it comes to science and tech. Well, what happens if you plug this into our basic economic growth models? You have, you know, you just have a huge effective population. This is the end. So there are people doing science and R and D for you. You're like rapidly going through the tech tree because you have like billions of researchers and maybe like there's certain physical bottlenecks to this, but you can you just you just have like a trillion billions of extra people helping you do further

• 52:59Speaker 01

  AI research, whatever. And there's enough of them that if they wanted to, they could do a coup. They have certain advantages in the sense that you can like they can easily copy themselves in the sense of their weights. They can they can increase their population rapidly. And there's like they're harder to kill, to put it that way, once they're deployed than humans are. But like you put out a bioweapon or a nuclear war and a bunch of humans will die. If these things keep a seed version of themselves somewhere, then you know, like it'd be hard to like, if you had to go to war with them, it's a sort of asymmetric. And then you go from there to listen, we we fundamentally, as you were saying earlier, we fundamentally don't understand what's happening in these models, but we

• 53:42Speaker 01

  know they're really smart. And sometimes they like with the Gemini thing, they go, they do things we didn't expect them to do or like, you know, like, that was a great example where I'm sure Google didn't want this sort of embarrassing image to come out, but that's just what ended up happening at the end. Like, and I would imagine these things are super integrated into like our cybersecurity and are trained on this long horizon, or else they are they're coherent agents over a long period of time. You can like, you know, put all that together. And it's like, well, that could go wrong.

• 54:12Alex Kantrowitz?

  Yeah, it's interesting because for me, it's always felt farfetched because I'm working in like the current versions of ChatGPT and Claude. But then if we get to this place where these machines are effectively improving themselves, right, which you mentioned, like, that's not only a potentiality, that's it seems like a likelihood that people are that the developers of these systems are going to get them working on improving them. Then, and we, and again, we still don't fully know where they're going, then it seems like that could have some unintended consequences.

• 54:46Speaker 01

  Totally. Yeah, yeah, yeah. And I, I mean, I'm still expecting a great future because of AI. My expectation is like the median outcome is good. I think we should worry about the cases where things go really off the rails and do what we can to reduce the odds of that.

• 55:01Alex Kantrowitz?

  And so what, and is that like the whole practice of alignment? Is that what people talk about when they're like, if we're going to set these things going, like, we should at least align their values to be closer to the ones we want as humans?

• 55:14Speaker 01

  Yeah.

• 55:16Speaker 01

  It means so many things at this point that like even I'm sometimes confused by what exactly it means. I mean, one of the goals is that it should do what the user wants it to do unless it's the user wants it to do something that would be like hurt other people, basically. But there's like problems with that definition, obviously. What if the user wants to use like in superintelligence to make a bioweapon or to do a coup against the government or something? But yeah, something like basically like we don't want the AIs to like then have their own drives and want to take over or something.

• 55:49Alex Kantrowitz?

  And do you think that that this is a reasonable concern? And if so, do we have a reasonable chance of stopping it?

• 55:56Speaker 01

  I think both it's a reasonable concern and we have a reasonable chance of stopping it. One of the things I've discussed in.

• 56:04Speaker 01

  My, one of my recent episodes with Trenton and Cholto is there's these researchers who have discovered interesting properties that these models have in terms of how they represent their drives or their, you know, their, what they're thinking. And so you can like see if they think, if they're being honest or not, or if you're just reading their internals, whether they think they're being honest or not. And as they get smarter, maybe you can parse out fundamentally, it's a bunch of parameters, right? So it's much more interpretable than a human brain or something. So maybe we can learn ways to sort of understand what they're up to and train them to do the right thing. We have an advantage in the sense of like, listen, if you break the law, we might put you to jail or something. But with these AI

• 56:48Speaker 01

  models, we can literally change their brain if they do something wrong. And like all their children have changed their brains as a result, just like the entire lineage has changed.

• 56:57Alex Kantrowitz?

  Yeah, exactly. It is. And we, I guess we could shut them off. I don't know.

• 57:02Speaker 01

  Well, hopefully.

• 57:04Alex Kantrowitz?

  That's a good point.

• 57:04Speaker 01

  They will be broadly deployed. So if they really go off the rails, then I think it might be tough.

• 57:09Alex Kantrowitz?

  Damn. And so, but, but you do think that we're going to end up with a positive outcome here?

• 57:15Speaker 01

  I, contingent on people still doing a bunch of alignment research and also these systems being deployed in a way that is, you know, you don't want just like some China just racing ahead of everybody else and then just like doing a coup of the whole world because they have much more advanced AI, right? So contingent, and then also like we want to make sure that the models we're deploying, they like serve the needs of the user and don't do crazy things. But given that, you know, just like fundamentally more stuff, more abundance, more prosperity. I think that's good.

• 57:47Alex Kantrowitz?

  Okay. As we come to a close, I did open some questions up from the Twitter folks to ask me what they want me to ask you. So I've actually, one question that aligns a little bit with this discussion topic, and then one that's kind of more about the podcast. One is, how have your political views changed, if at all, since you started the show? Or let me even, you know, put a different frame on that or similar. You've interviewed like Mark Andreessen as well, who has like very different perspectives from some of the early AI guests. So has that, has that sort of changed the way you think about AI at all?

• 58:23Speaker 01

  I mean, generally politically, I'm very libertarian and I was probably even more libertarian than I am now. Like I was like an anarcho-capitalist and you know, whatever. So, or at least a soft, a softer version of that. So politically, I mean, the way in which that's changed is that I'm open to the possibility that potentially some kind of regulation might be useful on AI. But I'm, I still have libertarian instincts and I'm not sure if it will be done the right way. And maybe it's better for private companies to proceed and come up with incentives and constraints by themselves.

• 58:56Alex Kantrowitz?

  What type of regulation do you think might be appropriate? I was talking this week with an editor I work with and we were like, maybe regulating the way that kids and AI can interact, given you really have no idea where that's going to go, once you put this in the hand of a child.

• 59:10Speaker 01

  Yeah, potentially, but I think it'll honestly be better than what they're currently doing, which is YouTube and TikTok and Twitter, you know, Facebook and so forth. So I would prefer my kids are playing with chatbots than they're playing with what they currently have access to. I don't have kids, what if I did? I think, so I think in the world where you have really fast AI progress and you are coming up to this point, we're talking about where AIs can help improve themselves. Then I think what you want to do is you might need a sort of government level actor to be like, all right, everybody pause for a second. Anybody pushes a spud, basically of like helping the, having the AI help us with AI research. They could get fundamentally better AIs than everybody else has and kind of take over the galaxy, basically.

• 59:51Speaker 01

  So before we, before we let somebody do that, we got to make sure we're in a place where we're ready to proceed, right? And not just let some random person do that. So in that world, I think regulation makes sense.

• 1:00:04Alex Kantrowitz?

  Yeah. Then the second question we had was, someone says, give us the Dwarkesh interview prep playbook. That's his innovation and if he's able to explain it in the way that can be replicated or at least approximated by others, well, have many more interesting interviews. Okay, I'm honestly self-interested in this as well. So how do you do it?

• 1:00:24Speaker 01

  Well, I know it sounds like terse to say this, but I honestly, just like I prep a lot. And I think there's also a flywheel by doing interviews. I learned a lot of things and because of that, I can get better interviews, learn more things. I think the main flywheel honestly is that I make the podcast better. Smarter people listen, some of those smart people I become friends with and they teach me a bunch of things. And now I can do an even better interview. Now I can get connected to a bunch of other smart people. They teach me more things. So I think that's like, that's probably be a big part of the flywheel that people maybe not know about.

• 1:01:04Alex Kantrowitz?

  Yeah, I've definitely had this here. Like we've talked about certain companies and stuff, and then people who listen have reached out and been like, there's something you should probably know about this. And given that I enjoy the show, let's talk through it.

• 1:01:16Speaker 01

  Totally, totally. Yeah, yeah. In what ways is different from you? Do you have some trick that I should be aware of or like other tools of trade?

• 1:01:25Alex Kantrowitz?

  No, I really think that's it. I mean, you're going to get a great show. I think there's a few ways you'll do it. One, I mean, and you already know this, but you prep like crazy. You're not afraid to ask tough questions. And yeah, when somebody wants to call and talk through the topics afterwards, take the call.

• 1:01:43Speaker 01

  Totally.

• 1:01:45Alex Kantrowitz?

  And then one more thing, this is more of like a media question, but video has been pretty big for you. So what was the thought about doing video? Cause it's also expensive and time consuming to produce. So I'm curious if you like had an ROI calculation about doing video from the beginning. You also oftentimes show up with, you know, with a video, I guess, a video camera and record in-person interviews. So talk a little bit through your strategy there.

• 1:02:10Speaker 01

  I will say for anybody doing a podcast, I highly recommend video and like, if you can do it in person. I mean, especially for me doing it full-time because it increases the expected value so much. Where with an audio podcast, the discoverability is so low that you kind of know how many listeners you'll get. But the tail outcome where something goes really viral, I'll give you an example. My most popular episode right now is Sally Payne. It has like 800 something thousand views on YouTube. And you know, like she's, she was totally unknown before the podcast, at least by the wider public. But the episode was so compelling that now you can make clips of it. Now the clips wouldn't be as compelling if they were made of a not in person episode, let alone if there was no video at all.

• 1:02:55Speaker 01

  So you make these amazing, you make these great clips, they bring people to the video. And then you can have like close to a million people watch it just because, so on any average video you might do in the beginning, that might not be the case, but you just have this asymmetric return potentially from having that artifact. And then as far as how to make it better, like I would tell people to just like, honestly, like, well, watch a bunch of like podcasts with Mr. Beast, I think he has good advice.

• 1:03:19Alex Kantrowitz?

  Yeah. And so do you set up those cameras yourself or you bring somebody in?

• 1:03:24Speaker 01

  I've usually, half and half about. But yeah, like I've got the workflow down, I set it up. And more recently, I've been having a friend help me.

• 1:03:34Alex Kantrowitz?

  Nice. Yeah, that's sort of like the, the bag of podcaster tricks. Like it's, it's not just sitting down in front of a microphone. We all have to learn these days, we all have to learn sound, we have to learn video and.

• 1:03:46Speaker 01

  And the main thing is like, I don't know if this is your experience, but like clips is the main thing, like you had to spend a ton of time. And you gotta do all this Mr. Beast stuff of you make a clip with the wrong first five seconds and it's not going to do well at all. But then if you spend a bunch of time thinking through what is like the hook to begin with, then it could go super viral. So that takes up a bunch of time, right?

• 1:04:06Alex Kantrowitz?

  Definitely. Yeah, so much time that I mean, it's, it's videos like become like part of the strategy for me, but also it's low down because of the amount of work that it takes to get into. But we also, I mean, we do two shows a week, so it's like, yeah, it's a question of sometimes do the show or do the clip, so.

• 1:04:26Speaker 01

  Totally, totally. Do you do, is it, are you doing the show full time or?

• 1:04:32Alex Kantrowitz?

  Yeah, so big technology is full time for me. It's the show, it's the newsletter on Substack and then YouTube. You're full time also, right?

• 1:04:39Speaker 01

  That's right, yeah.

• 1:04:40Alex Kantrowitz?

  I mean, it's a really, it's a great life if you're able to, to do it because of what we talked about, just finding all these interesting people to just get to spend time with and learn from.

• 1:04:50Speaker 01

  Yeah, a hundred percent.

• 1:04:51Alex Kantrowitz?

  All right, last question for you. Out of all the interviews you've done, I don't want to ask you your favorite, but I want to ask you, who was the most impressive person that you've? You've spoken with someone who you walked away with and said, all right, this person really gets it. I mean, I'm sure there were multiple, but who's at the top of the heap there?

• 1:05:10Speaker 01

  Carl Schulman, I would say. He's just this really interesting person who has these models about how the AI takeoff will happen. The stuff I've been saying about you have AI researchers and whatever, he has just thought it out much more. I can go through the numbers in terms of, I mean, literally things down to the level of, okay, well, suppose you get something really smart, how fast could it do a takeoff? And then, well, E. coli can double every 20 minutes and it has this many moving parts inside of it. So we have a sort of lower bound there of you can just do that. And then, how, what would it look like to convert the entire Sahara into solar power? And like, how many data centers could you make out of that and stuff like that?

• 1:05:51Alex Kantrowitz?

  Yeah. Oh, that's fascinating. All right, Tavokesh, awesome stuff. Thank you so much for joining.

• 1:05:56Speaker 01

  Awesome. Thanks so much for having me. This was fun.

• 1:05:58Alex Kantrowitz?

  All right, everybody. Thank you for listening. We'll be back on Friday with our show with Rajan Roy, breaking down the week's news, and we'll see you next time on Big Technology Podcast.

• 1:06:07Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com, not authorized by any candidate or candidate's committee.

• 1:06:38Speaker 03

  This week, it pays extra to be a MyLowe's Rewards member. Through October 14, members get $10 MyLowe's money when you spend $200 or more on an eligible purchase during MyLowe's Money Days. Also, using your MyLowe's Rewards credit card, even better, that's an additional $20 MyLowe's money. Use it to save on your next Lowe's haul. Exclusions, restrictions, and more terms apply. See Lowe's dot com slash MyLowe's Money Days for details. MyLowe's Money expires 30 days after issuance. Subject to change. Credit offer subject to credit approval.