AI in 2025 – Infrastructure, investment & bottlenecks with Dylan Patel

Dylan Patel, founder of SemiAnalysis and one of my go-to experts on semiconductors and data center infrastructure joins me to discuss AI in 2025. Several key themes emerged about where AI might be headed in 2025: (1) The hyperscalers are racing ahead in capital expenditure, (2) The expected explosion in AI workloads is drawing in a wave of new specialized GPU cloud providers, (3) By 2027, AI data centers alone could account for 10% or more of total US electricity consumption, straining America’s aging infrastructure, (4) Open-source variants like Llama 3.1 are driving commoditization at speed, slicing away the profit margins of plain-vanilla model-serving.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Hi, happy holidays. While I'm away this holiday season, I wanted to drop here a few of my conversations with leading experts in AI. I discussed how AI might surprise us in 2025 with Ethan Mollick, Dylan Patel, Nathan Benet and Kai-Fu Li. The conversations were initially released for members of Exponential View. First one up, my discussion with Dylan Patel. Enjoy.

• 0:23Azeem Azhar?

  So it's Azim here from Exponential View and this is part of our series of conversations about what surprises artificial intelligence will have in store for us in 2025. It's a particularly hard question, of course, because they're surprises. If we knew what they were, they wouldn't be surprises. But to help us work through one of those questions, I've got Dylan Patel, who is the founder, the boss, the supremo of really my favourite semiconductor data centre infrastructure vehicle for analysing these questions. Dylan, thank you for joining us.

• 0:59Speaker 02

  Yeah, thank you for having me, Azim. And I'm here with Azim, the leader of Exponents, right?

• 1:06Azeem Azhar?

  We try, we try. So I'm in a hotel in New York, which explains the slightly bizarre, weird vanilla background. I was here for the Dealbook Summit, Andrew Ross Sorkin and New York Times Dealbook. So I was able to hear Jeff Bezos, Sundar Pichai and Sam Altman speak yesterday, well, also Prince Harry and Serena Williams. A few of them talked about AI. And the thing that really came across, there was a slightly different message between Sundar and Sam. Sam said, there is no wall, things are still going to keep going, there's

• 1:51Azeem Azhar?

  lots of room to run. And Sundar said,

• 1:56Azeem Azhar?

  The low hanging fruit has been picked and it's going to get a little bit harder from now. And Jeff said, Amazon is really turning its attention and we've just released these new, this new vertical stack of, you know, chips and models, new pricing and so on.

• 2:17Azeem Azhar?

  Who was right out of them, do you think?

• 2:19Speaker 02

  Um, yes, I think, I think it's like quite a, quite an interesting, right? Like one's just getting started, one's saying, hey, it's going to be more incremental from here. And one's hype beast, uh, you know, we're going to keep going exponential from here. Right? Uh, I think this, this is a sponsored by Smartwater somehow, right?

• 2:39Speaker 02

  But the, the interesting thing is you also have to think about what is each person's motivations for their statements, right?

• 2:49Speaker 02

  You know, Amazon is just getting started, right? Like, you know, if you look back a year ago, they were woefully behind. Um, and today they're still woefully behind, right? Their new models are still like very much not even top five in the world, right? You know, but as, as like the leading cloud company, they should be in that top five, but they're not. You know, you look at, you look at OpenAI, their motivations are they must raise, right? Uh, the, the level of scaling that they want, they're going to spend all of the money that they just raised, the six billion equity and four billion debt, you know, in like a year and a half. And, and you know, they would like to commit to more compute than they are. They, they got with that money. And so they, they need to raise again, like, like in a quarter or two, right?

• 3:32Speaker 02

  Uh, if they want to really get that money and then start like committing to even more compute. And then, and then lastly, you have, um, you have Sundar who, uh, you know, I think, I think they're still in like release paralysis. They've, they've, you know, haven't been able to release their Gemini Ultra models in a while, right? They keep releasing Pro, iterations of the Pro, they keep inching up, but they haven't released this flagship model yet, right? Uh, that, that OpenAI has, Anthropic has, um, you know, and so on and so forth, right? So they're really, uh, lagging behind on that front. So I think it's interesting that they say it's only incremental from here. Given they still haven't even gotten to the top rung of a top echelon. So that's, that's a little bit concerning. As far as who's right, I think it's, I think it's closer to Sam or Sundar, or sorry, Sam or Bezos.

• 4:17Speaker 02

  Because we are just getting started in AI, right? Like these, these scaling, uh, you know, on, on many different vectors, whether it be synthetic data, whether it be compute, whether it be push training, whether it be reasoning, um, whether it be AI infrastructure and inference rollout, all of this is just started, right? Uh, we're, we're nowhere close to this being as pervasive.

• 4:38Azeem Azhar?

  Right. And what you've just described though, reminds me of the Charlie Munger quote, which is, uh, show me the incentive and I'll show you the outcome. And these, these three different leaders have got different incentives to give us, give us a particular answer. As you say, sort of Sam has to exponentiate and continues to do so in order to keep, to keep raising. So you say, and I, I, I believe I also agreed with that, that we are just getting started, that the, it's easy to mistake turbulence on the journey for the plane not getting to its destination. And of course, engineering of new products of these, of this scale is bound to hit road, roadblocks. When we talk about exponential growth, right? And we think about what's happening with these AI models.

• 5:19Azeem Azhar?

  Um, there's lots of exponentials going on. There is the amount of data they need, the amount of compute, they need the speed with which they're growing.

• 5:28Azeem Azhar?

  Whenever we've looked at technologies like this, like in the semiconductors back in the seventies, eighties and nineties, there are often cracks, right? There are moments where it just becomes hard to get around that hurdle. You need a new approach. Where do you think that, at what layer do you think we'll first see a crack in the exponential sort of driving AI? Would it be design, manufacturing, materials, deployment?

• 5:55Speaker 02

  Yeah. Interesting. So you, you mentioned like, uh, glimmers of cracks, right? Like there's obviously one, one angle is, hey, models keep getting better, but are they getting better at the pace that, uh, people are espousing, right? And so then there's the whole like debate over the last, you know, a couple of months of scaling laws slowing down. Uh, then there's also the flip side is, Um, cracks around deployment and that's where actually there's a lot of cracks, right? Um, Microsoft, every earnings call says they don't have enough capacity to deploy, deploy the models as pervasively as they'd like. Uh, OpenAI constantly says they don't have enough capacity to deploy the models they'd like, right? Um, you know, and other, other folks as well, right? Uh, Google and Anthropic, they've definitely had issues deploying the models as fast as they'd like to, uh, just due to capacity, right? Um,

• 6:39Speaker 02

  so as you look across the ecosystem-

• 6:41Azeem Azhar?

  And you mean capacity, that's, that's GPU data center capacity?

• 6:44Speaker 02

  Right, exactly. Um, and then, and then the flip side is like, hey, like enterprises, uh, getting them to adopt AI is taking time, right? It's not a, not a flip of the switch, um, as it has been with like startups and some of the tech companies. Um, so you know, that's, that's like the, the challenge there. Um, I think, I think the crack that we see, um, now is really that no one gets to make money besides the person who's their first or second, right? Everyone else is actually, their margins are really bad. Um, you know, when we're talking about, hey, serving llama tier models, serving deep seek tier models, that's great, right? Like, yeah, but like Meta and DeepSeek aren't making money from open sourcing it. They're, they're, they've got other tertiary benefits, of course. Um,

• 7:27Speaker 02

  but then the people who take these open source models and serve them, you know, plain vanilla, uh, aren't making money, right? They're, because there's such intense competition. Um, and then, you know, you kind of have to have built some value add on top and that value add is more sticky and customer specific, right? And that's where I think like, there's a lot of cracks is like, how can you do the sticky customer specific stuff without doing the, you know, base model and instruct model stuff, uh, without that, like dominating your cost structure and being really bad. So there's sort of a, a catch 22 there.

• 7:59Azeem Azhar?

  Right. So, so in a sense, the, you know, Andreessen, Mark Andreessen has gone off and said, look, this is a race to the bottom. And these models are going to get commoditized really, really rapidly. So is the observation that you're making, if you look at some of these people who serve models like Together or Grok, they're they're serving very, very cheaply, which means extremely small margin. But not only that, they're not necessarily serving at the Claude Sonnet or GPT-4 level.

• 8:29Speaker 02

  Yeah, so so I think the the real challenger, so like like last year, we talked about this for Mistral. We said it was a race to the bottom. Like the inference was a race to the bottom and that everyone's just going to take these models and commoditize them. And anytime there's an open source model, that commoditizes everything below that tier. And that's that's been the case, right? With the release of Lama 3.1, 405B and 70B, that's that's continued to commoditize a lot of a lot of language models, right? So when we look at like, hey, look at look at Amazon's new model, well, it doesn't beat 405B. And it's it's it's just a commodity at that point, right? Like, what what am I doing here that's differentiated?

• 9:11Speaker 02

  And so far, the only two companies that have a good margin on serving are OpenAI and Anthropic, right? And then their partners who get to serve their models, right? So, you know, OpenAI with Microsoft and Anthropic with Amazon. And now they recently announced the Snowflake thing. So there's there's very few folks that get to make money serving. And it's really just dominated by that first and second place player. And the question is like, hey, what happens when XAI gets that first or second place? Does that knock one of them out? But given that they've been spending so much, do they even get there? Does that mean that there's now a first, second and third? Or do first, second and third now make less money? And and and that tale continues on, right? What happens when Meta releases Lama 4? Does that just like wipe out another humongous tier of models?

• 9:52Azeem Azhar?

  Let's assume that these XAI's model is actually better than GPT-4 or Claude 3.5. How long would it need to be better than those models for developers to actually switch, right? Because the history of technology markets is that there is no number three, there is no number four, there is no number five. And it was it was true even in in databases, right? From back in the late 80s, early 90s, you very, very quickly, you had kind of Oracle and Sybase and and Infomix, I think it was. You couldn't really think of who four or five were and the dominance of the top two was really substantial. And why would this technology market be any, any different?

• 10:39Speaker 02

  Yeah, so I think one thing about language models is that they're very replaceable, right? So like, you know, we, we talk about like, hey, this commoditization is, is bad for some businesses. It is, but guess what? It's, it's driving so much adoption, right? The fact that people can adopt Llama 3.170B without paying anyone a margin is driving the cost down so much, right? And in fact, in some cases, VCs are subsidizing it with inference providers that they keep funding, right? So, so, and then like clouds as well, right? Like with loss leadership type strategies. So it turns out that like, hey, um, these, these models commoditizing are driving adoption faster and faster. It's just, there's no one really making money off of it, right?

• 11:19Speaker 02

  Every time someone open sources a model, any margin that the model makers were making, a lot of it gets wiped away. And then what's, what's left there? It's margin that Nvidia has, right? And margin that the data center folks have, right? So, so it ends up being that you knocked out half of this margin, right? You know, you look at Anthropic and OpenAI's margins, their gross margins are 60, 70 plus percent, right? So it's, it's that whole thing just gets wiped away, right? Um, and, and 40 mini, I bet it's not making a ton of money because it's so small. And, uh, there's got so much competition from Gemini Flash and Anthropic and Lama, and you go down the list. So, so if a company like XAI were, or Google were to come out with, hey, this is definitively right up there.

• 11:59Speaker 02

  If anything, the only way they're going to get market share is by pricing it a little bit lower. That drives people down, right? That drives the margin maybe from 70% to 60% or 50%. All of that delta is money that's lost, but also like semiconductors, right? Moore's law, every two years, we doubled the number of transistors and have the cost, right? Guess what? Market never shrank, it kept growing.

• 12:18Azeem Azhar?

  That's the bet, that is the bet, right? That increasing demand, expanding the market, there's a positive elasticity of demand as price falls, is the bet that is partly being taken here. So that if you ask, or ask the question, what does 2025 hold? Does anyone really start making money?

• 12:36Azeem Azhar?

  It might be the case that we're still not in that stable environment where anyone is really making money.

• 12:42Speaker 02

  I mean, you can look at many different platforms out there in the sort of Bay Area tech community, and they didn't make money for a long time, right? You're sure you can

• 12:50Azeem Azhar?

  11 years for Amazon.

• 12:52Speaker 02

  You can look at Airbnb and Uber, or you can go even more intense and say YouTube. How long did it take Google to make YouTube money from YouTube? More than a decade. YouTube was a money-losing enterprise for more than a decade, right? And it scaled and scaled and scaled, but now it's a very tremendously profitable business for Google, not like search, but it's still tremendously profitable. And same with like Amazon and Twitch, right? And there's so many different platforms that are hugely capital intensive, right? Because there's so much compute or bandwidth or storage required to serve them to millions of users, but once they get the moat, you know, that's really helped. Now, the big issue here, question here is the switching cost is really low, right? If I want to go from OpenAI's API to, you know, hey, new Llama model on Together,

• 13:36Speaker 02

  or new DeepSeek model on Together, or Fireworks, or whoever it is, turns out it's like you change it like that. It's that easy. You have to do a little bit of testing and you're good. You're done, right? Like it's like immediately replaceable. So the question is like, can OpenAI, Anthropic and these other companies build services on top, right? The the ability to switch off of Microsoft Copilot is probably zero, right? Now Copilot hasn't gotten a huge amount of adoption because Microsoft doesn't have the compute to actually deploy it at scale with all the features that they promised. But, you know, that's something that doesn't have switching cost or has huge switching costs, whereas like just raw API models, tokenomics, those have low switching costs. And so that always chases, you know, reducing cost and switching makes easier and then

• 14:22Speaker 02

  therefore driving down margin and cost.

• 14:24Azeem Azhar?

  You talked about compute constraints there for Microsoft. So let's talk about compute and semiconductors as a starting point here.

• 14:35Azeem Azhar?

  We've just seen Amazon announce their own family of chips in the last couple of days, the Tranium chips upon which they've got the new models running, Nova, I think the other models, the models are called. What do we think will happen with the semiconductor market next year? There's been a supply chain crunch in a sense, right? People can't order enough of these high-end chips and we've got Nvidia continuing to push out their roadmap. What should we expect?

• 15:09Speaker 02

  Yes, I think we'll see, you know, today we have more than 100 cloud companies that have started in the just the last few years, right? Last couple of years. Some of them pivoted from other things, but all of them are basically new to GPU cloud, right? You know, historically, we had these massive cloud companies and, you know, huge earnings and, and, you know, all these guys still have fantastic margins on GPU stuff. But as, as these 100 cloud companies start to get bearings and hey, there will be some consolidation, but there's at least like five of them that are legitimate businesses that will be long-term that I say will be around in 10 years, right? And be much, much bigger, right? Like the CoreWeave, Nebius, Crusoe, those kind of folks, they're going to be much, much bigger, right?

• 15:51Speaker 02

  And so then the question is like, what happens next year, right? Well, we see, we really see some of these folks take a new stride, right? Nebius is already a public company, CoreWeave likely goes public, Crusoe continues to raise money, right? These kind of companies become a next level of scale and start rivaling the big guys to some extent. And then on the bottom side, you see a lot of people falter and keep driving the price of the market down. And then you have the big guys, hey, they're going to continue to buy as much as possible, but given their plans for next year, investors are going to be like pretty spooked, right? Like you look at like, like what is Microsoft going to spend on CapEx next year? North of $80 billion, right? You know, the street is not ready for that, right? So, so yes, Nvidia has to deliver, Nvidia's chips are a little bit delayed, right?

• 16:36Speaker 02

  But they'll, they'll be out there and they'll continue to grow revenue, they'll continue to ship chips. The, the real problem is who, how are people going to like handle the amount of capital being flown into them, right? Now, now thankfully, unlike, you know, most of these AI companies.

• 16:53Speaker 02

  GPU clouds are profitable today, so that's kind of helpful, but that's also leading to a lot more demand being flown in. And by demand, I mean capital continuing to enter, whether it be the big guys with their balance sheet and their cash flows, driving those down lower and lower and lower. Hey, is Microsoft going to have a cash flow of only 10 billion versus 40 billion? Or, you know, same with Meadow, or if it's, you know, the new guys and they're like, yeah, we're going to raise another $10 billion, right, to build out another cluster, right? Or it's Oracle, it's like, hey, we're going to spend a bunch of money. This is, this is the big theme of next year is like, the worry, the worry around that, right? The, the, the dollars that have been spent so far are somewhat insignificant compared to what's being planned next year, right? It's more than doubling, right?

• 17:34Azeem Azhar?

  It's more than doubling. So let's just put these numbers into, into some kind of context, right? So $80 billion is, you, you said, well, that's kind of where Microsoft roughly end up with their CapEx. Their CapEx spend five years ago was probably in the $10 billion range, right? It was, I've actually trying to look the number up right now, but I am struggling to find it. Yeah, I mean, 2020, $15 billion in CapEx. So you're looking at a five, five-fold increase in, in five years. And, but, isn't that being communicated to the streets, right? That is what, what happens in the earnings calls. This is what the, this is what is happening.

• 18:15Azeem Azhar?

  If you look at what TSMC, TSMC says about their forward outlook, and then you look at what Nvidia says about their forward outlook, that's, that's got to go somewhere, right? And it's going to go into the big hyperscalers.

• 18:27Speaker 02

  Yeah. And the only thing that these hyperscalers are communicating is, hey, it's going to grow. They're not saying what number it's going to be, right? And hey, Microsoft, right now, I think the street is maybe at 70, 80, but it's going to be north of 80, to be clear, right? So there is a big, big change there.

• 18:44Azeem Azhar?

  North of 80. And what about, what about Amazon and, and Google? Because they, they're not, those are the big three. Actually, in Meta as well, they tend to dominate CapEx spending from the big, the big companies.

• 18:57Speaker 02

  Yeah, and so what's interesting is their share of CapEx is also their their CapEx is also skyrocketing, right? Amazon is, Google is, Meta is. The interesting thing there, Google being the least aggressive for some reason, which is interesting. But the other thing there is their share of CapEx is falling, right? All of these new guys are becoming a higher and higher and higher percentage of the total CapEx, which in effect is your natural market share falling, right? So they really, they really must either answer the question, hey, we're gonna lose market share, or are we gonna be, you know, done, right? Like, or not done, but like, are we gonna spend way more money than than our investors would like? In the case of like Meta, it doesn't matter what the investors think. But in the case of like the other three, it matters a lot what the

• 19:38Speaker 02

  investors think, right? You know, I say it doesn't matter for Meta because Zuck has all of the voting rights.

• 19:44Azeem Azhar?

  Right, he has all the votes, but but actually, I'm just curious about, you know, the the the new players who are taking up the market share. Fundamentally, they're a they're a sort of single line item business. You know, we buy, we build and operate GPU data centers as as utility. So what kind of capital is that that appealing to? Because that just feels like it's a different class of capital than the kind of person who owns Microsoft stock.

• 20:12Speaker 02

  It absolutely is, right? And like, this is sort of the arbitrage that's going on, right? If you look at Microsoft, their equity cost of capital, you know, sort of is like much higher, right? People aren't expecting equity cost of capital in the in the single digit range. Or we can go out there and get loans for 9%, right? And then 9% loans stacking on, you know, these massive data center build outs, and then renting them out, they can do that, and then they can make a margin that's like fair, they can make a good margin. Maybe not what Amazon is used to when they're selling database products like Redshift or whatever, right? You know, or, you know, all these different database products that Amazon has at humongous gross margins, it's not that high, but it's still really healthy. And so it's very appealing to,

• 20:56Speaker 02

  you know, real estate investors, private equity, credit folks, right? Like, all of these folks are like, wow, 9%, 9.5%, 10%? For Corey, right, which is like the triple gold A, right? Like in this space, you go out the tail. Some of these guys are paying 20%, right? It's like a 20% three-year loan on GPUs. It's like, yes, like there's people chasing that, right?

• 21:18Azeem Azhar?

  But but turning it into an asset class actually does unlock an enormous new slew of capital that is willing to take a different risk with a different tenor, different time horizon. And and that that that looks different to effectively vanilla equity on the NASDAQ or whatever it happens to be. But there's still this issue about where the chips are going to come from. And and and you know, just read today that Elon wants to 10x the size of his biggest data center. I think he wants to build it in a different state and it's going to be a million chips. Where's he going to get those from? And and can Nvidia supply him and everyone else?

• 21:57Speaker 02

  Yeah. So so this is like Nvidia continues to ramp, right? They're going to produce, you know, they produce more than 5 million high-end GPUs, right? Hopper GPUs this year and and a number of Blackwell as well, you know, as the start of the ramp. But more than 5 million hopper and and next year they're going to do more and the year after they're going to do more. And hey, Elon wants to do a million. They're not happening in three months. They're not happening in six months, right? It's happening over the next couple of years that it's a million GPU cluster, right? So you can see they've been expanding capacity. They've been buying a bunch of land in Memphis. They're going to build a massive gas plant, not just mobile generators, but actually like a natural gas combines plant right there. They're they're they're going to, they can deploy it there.

• 22:38Speaker 02

  Nvidia and TSMC and SK Hynix and the rest of the supply chain here are continuing to ramp up production. The question is, hey, where is Elon going to get this capital from? Well, it turns out Elon can probably raise more money than anyone else in the world right now, right? Because he's got, you know, he's got Tesla and SpaceX, which have done tremendous returns for all of their investors. And, you know, anyone who's invested in those is is like, yeah, I'm going to throw money at XAI. I believe in Elon, right?

• 23:03Azeem Azhar?

  He's also got Mar-a-Lago.

• 23:07Speaker 02

  There's the political angle too. Yeah, yeah.

• 23:09Azeem Azhar?

  Yeah, I mean, that's gonna be a tailwind for him.

• 23:12Speaker 02

  So, so the TVA Tennessee Valley Authority, as soon as Trump was elected, they approved some of like Elon's like permits or what have you, right? So it was quite interesting, right? Like, it's like, oh, yes. Now that we're, you know, it's our folks, right? Like, you know, so I think there is like a political aspect. I don't think that the government will specifically give him money. I think what the government will do is just slash every regulation so you can continue to do whatever the heck he wants as fast as he wants, right? Oh, yeah, yeah, you don't need to wait for FFA approval. You know, you don't need to delay your launches for FAA approval with SpaceX. No, no, no. Just do them whenever you want, right? Like, you know, kind of thing, right? And same with same with Memphis. Like, oh, you want to build a massive gas plant?

• 23:52Speaker 02

  Go ahead. You don't need an environmental site review, right? Just go, right? Like, this is probably what's more likely to happen, not money. But yeah, there is there is definitely an advantage to Mar-a-Lago.

• 24:03Azeem Azhar?

  Right. So you we talk about that gas plant for the new Memphis data center that they're talking about. And of course, they have mobile oil generators or diesel generators, I think, at the existing Colossus site, which is 100,000 GPU site they built in repeatedly 19 days.

• 24:23Azeem Azhar?

  Infrastructure and power consumption seems to be a big bottleneck. Now, you know, we've seen the numbers, we've seen the estimates of the growth. I think you are, I think, even more aggressive than Goldman's more aggressive growth numbers for power consumption from AI data centers. Can you just remind us what your high end estimate is and what as a percentage of US electricity consumption that is?

• 24:50Speaker 02

  Today, data centers are roughly

• 24:55Speaker 02

  3% of total US grid consumption.

• 24:59Speaker 02

  And this is, this is, you know, differs by state, right? States like Virginia are 30%, right? And then there's many other states that are much less, right? For US, North America, I bucket in Canada because it's satellite, right? It's Canada's satellite state, guys, come on.

• 25:16Speaker 02

  By the, by the end of this decade, the US, you know, power consumption for data centers, it could be as high as 15%, right? You know, by 2027 is where we see it being around 10%, right? And that's given the massive, massive growth that's happening in some states, right? So Virginia continues to grow, but really there's a handoff to new states, right? States like Ohio and Illinois and Texas and Wyoming. This is where the power consumption is soaring, right? Certain states like Oregon as well, right? Continue to soar. Some of these states, the majority of power will be data centers by 2027.

• 25:54Azeem Azhar?

  Right.

• 25:54Speaker 02

  Right? Not 10%, 10% by the nation.

• 25:57Azeem Azhar?

  Yes, but let's turn that into, into numbers, right? Because of course you can, you can forecast that, but ultimately, you need to build power stations and you need to build grid connections. And the US does not build power stations as quickly as some places. So is, is that going to end up being a realistic gating factor?

• 26:18Speaker 02

  It, it already is, right? You know, to be clear, these numbers would be higher if the US could build them faster. That's the biggest challenge, right? Like instead you have companies rushing to go to, you know, very odd parts of the world like Malaysia and Indonesia, the Middle East.

• 26:36Azeem Azhar?

  Johar in Malaysia, that, that, I wrote a, a, a piece for Wired on the growth of data centers in Johar.

• 26:45Speaker 02

  Yeah, exactly. So, so like Microsoft's going there, many other companies are as well. Nvidia's investing there. There's so many folks investing in geographies that have, you know, pretty meh internet connection, pretty meh like data center capacity today. And, and, and a lot of that is because, hey, we don't have capacity in the US. I'm trying to get everything I can, right? You wouldn't see Elon do what he's doing if it was possible to do it in the normal way, right? What he's doing is more expensive and more complicated because that's the only way he can do it as fast as he can, right? And so these, these grid.

• 27:18Azeem Azhar?

  You go ahead.

• 27:20Speaker 02

  The grid connection, the cost to transport power on the grid, right? Actually, in the northeast, around Pennsylvania and Ohio and such, the cost of sending power around on the grid is soaring, right? In some places, the contract rates are above that, you know, for new contracts, above that of actually the cost of generating the power. So our grid is dilapidated and weak. And people are trying to build as fast as they can. And you see folks like Elon building power onsite at his data center. You see folks like Meta in Louisiana, you know, signing a contract for two, three, you know, two massive 750-megawatt natural gas plants right next door to their data center plans, right? You see these plans because, hey, grid can't handle it, right?

• 28:02Speaker 02

  We need to think about not just the data center and the GPUs. We have to think about the data center, the substation, the grid transportation, and the generation all at once holistically. Because otherwise, the project of this scale just will not happen.

• 28:15Azeem Azhar?

  Right, so let's try to, let's dig into this a little bit. People started talking about the gigawatt data center requirement, right? Now, to put that into context, the gigawatt is like the output of a big nuclear power plant or a really big CCGT power plant.

• 28:33Azeem Azhar?

  When do we actually see gigawatt data centers coming around?

• 28:40Speaker 02

  So there is, you know, a difference of like how you can define it, right? In one way, you could say, hey, Google in Iowa already has a gigawatt data center. Now, why is that? Because they have three data centers within 20 miles of each other. And then another one that's 40 miles away in Nebraska. And that's a gigawatt, once you add it all together, right? So that's next year, right? In other cases, you like, look at like, hey, this OpenAI deal in Texas, right? With Oracle, that's a gigawatt by the end of 2026. Same with Microsoft in Memphis, and, oh, sorry, in Wisconsin, right? And then Elon in Memphis, and Meta in places like Louisiana, right? These are all gigawatt data centers in 2026, right?

• 29:23Speaker 02

  So 2026 is the year of the gigawatt data center. But in the meantime, people are still Scaling, right? And and they're doing it whether it be through multiple data centers in the same location or or nearby, or even their plans include like spending billions and billions of dollars on fiber to connect data centers that are further away with massive amounts of fiber. So you see Microsoft having signed big deals, you see Meta having signed big deals, you see this also happening, right? Because hey, we can't get the data center in one place, let me connect them up, right? And then it's sort of like a Band-Aid, but the gigawatt data center, if you view it through that lens, is already here next year. But you know, for an actual site, it's it's 2026.

• 30:02Azeem Azhar?

  2026. So so one of the things I was I was playing around with was, you know, as people talk about scale, and I think I think it may have been Sam who talked about a five gigawatt data center at some point, it was one of the frontier labs. And and I was playing around with it, the fact that the US actually only has one five gigawatt power station, which is the the Grand Cooley or Grand Cooley hydropower setup over in Washington State. And then the second biggest is Vogel Vogel in Georgia, which is a nuclear power station, but it's below five gigawatts. But China, on the other hand, has lots of really big power stations. They've proven to build seven, eight,

• 30:47Azeem Azhar?

  10, 12, 6, 15 gigawatts in nuclear wind hydro. And they can build them really, really quickly. I just wonder in this race condition of needing to build bigger and bigger data centers in order to build bigger, better models. Does China's proven ability to build a four gigawatt power station in a in a shorter period of time in the US perhaps change the nature of that competition?

• 31:22Speaker 02

  Yeah, so the interesting thing is like, what is, what does the US have a disadvantage in? It's building physical infrastructure. We can't build the data centers, we can't build the power substations, we can't build the power generation fast enough. On the flip side, China, what do they have an issue with? Well, they certainly don't have an issue with power. They added an entire US grid worth of power in like seven, eight years, right? It's a snap of the finger. We talk about, hey, what's a gigawatt?

• 31:47Azeem Azhar?

  Problem either.

• 31:49Speaker 02

  Yeah, and they don't have a grid problem either, right? And it's, and it's like, hey, do they have a data center problem? No, they, they literally have multiple factories, aluminum mills, et cetera, with multi-gigawatt substations right there, right? So it's, it's not a question of power, they have a problem is getting the chips, right? And then, and then sort of, then there's like, there's sort of like the odds of both of these, right? You know, hey, we have what they don't, and they have what we don't. And, and the question is, which one is solvable faster? And for the US, if we, if we play our cards right with regards to regulation, with regards to accelerating spending, with regards to unlocking the permitting process and approval processes, then the US could do it faster. This is not the case for China, like the chip, semiconductor supply chain is just much more complicated, right?

• 32:33Speaker 02

  And restrictions are hampering them to some degree. Although maybe not as much as people in Washington think they are, but they are hampering them to some degree.

• 32:42Azeem Azhar?

  There have been some degree, but I, I was really interested to see what had happened with, with DeepSeek, right? And they introduced this, which is a Chinese AI model company that came out of a quantitative hedge fund. And they brought out their reasoning model, which was actually not a, not a bad model. And these are much, much smaller, more power efficient models. Now, the founder of DeepSeek does say, listen, we are compute constrained, but they are being forced to be a bit, a bit innovative. So it seems to me like there's, it's not a certainty that you.

• 33:17Azeem Azhar?

  You couldn't make quite a lot of progress without the brute forcing that is the strategy of the US Frontier Labs.

• 33:24Speaker 02

  So I think this is maybe like a bit of confusion, right? Like with regards to a lot of folks, they think DeepSeek is behind, right? And they're not, right? Their models are better than any model that XAI has released, right? Their models are better than any model that Amazon has released or Microsoft or and even Meta at this point, right? They're better than Llama, right? And so the question is, how are they able to do that? Well, part of it is they have an insane amount of talent and the other aspect is their compute constraints are not as bad as people think they are, right? Yes, they're in China, yes, compute is limited, it's hard to import. Guess what? Smuggling GPUs is a thing.

• 34:06Speaker 02

  There are China versions of GPUs and between these factors and then you can rent GPUs externally outside of China. And between those three, DeepSeek, what a DeepSeek employee has told me is that they have roughly 50,000 Hopper GPUs, right? Now, for reference, this is not all in one location, right? For reference, OpenAI has many, many more Hopper GPUs than that, but their largest individual site for Hopper GPUs is in Arizona, near the airport. It's 70,000 GPUs growing to 100,000, right? So when you think about on the scale of, hey, just training compute, is OpenAI that far ahead? Uh, not yet, right? Now, next year, the plans are scaling really rapidly. Elon's going to 200,000 GPUs by the end of this year, right? You know, there's, you know, OpenAI and Anthropic and such.

• 34:51Speaker 02

  Anthropic just announced a 400,000 Trainium cluster, right? For their next generation training supercomputer. So when you look across these companies, yes, there is a lot more growth.

• 35:02Speaker 02

  But at least today, there is not as much of a gap on compute because, hey, 50,000 GPUs is not an absurd amount of capital either, right?

• 35:13Azeem Azhar?

  No, it's not.

• 35:14Speaker 02

  You know, it's on the scale of single-digit billions, right? And not even 5 billion, right? Like, I'm talking like 2 billion, like dollars for 50,000 GPUs. And that's to buy them, right? Like, if you were to rent them, it'd be even less. And there's a good reason to believe they're renting a chunk of their GPUs. So, so when you, when you think about all of these factors, it's like, well, you know, this is a scale of capital that's like pretty large, but uh, China, China capital investment's not a challenge either, right? But certainly for DeepSeek it is, um, today at least, uh, but you know, so the question is, can the US labs outrace the Chinese labs? Um, and that's the real big question that's, that's, uh, out there, um, because 50,000 GPUs, 100,000 GPUs, this is sub $10 billion. This is achievable, right? Um, by many players.

• 35:58Speaker 02

  Uh, once it scales to $50 billion clusters, right? That's when it becomes, oh, well, this isn't really achievable. This is, this is such a large number of chips that restrictions do hamper it.

• 36:09Azeem Azhar?

  They, they know, they, they absolutely do, but if you can't get power to those 50 billion dollar cluster, right? If you can't get power to your million Blackwells, you can only power 87,000 of them while you wait for a grid connection or permitting. It doesn't really matter that you've gone off and bought a million of them. And that, that is the one dynamic that I've been, I'm curious about, I'm curious about whether, given gray market supplies, given the ability to rent, given the ability to perhaps run, run big clusters of older chips. There is a, there's a path that this race where, of course, DeepSeek is ahead of XAI, but it's not ahead of OpenAI and Anthropic. There's a path where

• 36:55Azeem Azhar?

  companies outside of the US continue to keep, keep within a few months of, of the, the US frontier.

• 37:05Speaker 02

  I, I, I, I completely agree that there's a path to, and DeepSeek, over the next six months before the Blackwell, Blackwell ramps really get full swing. There's, there's like, I see no way for the US labs to really leap that far ahead.

• 37:19Speaker 02

  You know, I see, I see them like improving incrementally, getting better, um, but nothing, you know, now that O1 full is launched, right? Like, I don't expect too many big jumps much larger than O1. You know, within the next six months until, you know, really people get Blackwell scale, uh, training clusters, which is next year, right? Um, and up until then, DeepSeek is also not on a tremendous compute disadvantage then, right? Um, and, and in fact, they may even be able to Catch up even more, right, with the fifty thousand that they have. And so this is, this is, this is the question. This is, you know, this is like an issue of the government slowing down progress. Can, can something be done? No one's asking the government for money. They're asking for them to make the process faster.

• 38:02Azeem Azhar?

  Absolutely. Absolutely. So now let's go up stack from the chips and let's talk a little bit about, about these models and how well they work. We are recording this. One hour and twenty minutes after I laid down two hundred bucks to get O one Pro, which is the pro version of the reasoning model from ChatGPT. What, you know, why not? I've been quite impressed with it. What, what is the model that you use? What do you use most regularly in your work?

• 38:30Speaker 02

  So we use a lot of Claude, actually. But today, I also just spent two hundred bucks, not just for myself, but for a number of the people. There's seventeen people at my firm. And, and guess what? Our spend on AI is skyrocketing, right? You know, we use a lot of Gemini as well, because the long context of Gemini is really helpful for certain applications. But I think between those three, right? Like, you know, I probably will continue to use Anthropic for single, you know, one-shot type of applications, right? Non-reasoning, right? Because Anthropic's model is the best non-reasoning model. And then the reasoning models are very expensive and they take a long time to run. So there is a use case, but the fact that they take a long time to run hampers workflow. So I will use them as well.

• 39:14Speaker 02

  And then Gemini, because they have such a long context length, you can input video, audio into them. And, and that, that's extreme. The multimodality aspect of it is really helpful. So between those three, I kind of actually use multiple models, right? But definitely Anthropic and OpenAI are the workhorses that I use.

• 39:32Azeem Azhar?

  Yeah, that's, that is, there's a convergence. That's very, very similar to us. So Claude, new Sonnet three point five, Gemini multimodal for the, the longer things, the thing that, things that involve video. And then O one, when you need to ask a different class of questions where a bit of reasoning is required, is the portfolio. And actually, we have to do that switch in our, in our head. I have a favorite. Mobile app feature, which is the the symbol, simple voice mode. So what you can do is you can give it a a stream of consciousness like James Joyce Ulysses style blurry blurred thoughts. And just tell it to organize them for you, which I find extremely useful when I wake up in the morning or if I'm in the car.

• 40:14Azeem Azhar?

  And I just need to offload some thesis I'm playing around with, I I will kind of hit that record button. And it does a pretty good job. It's not a transcript. It tries to make sense of what you've you've said, and it's a it's quite a nice interface unlock. So Dylan, if you haven't tried that, I do recommend it.

• 40:31Speaker 02

  Yeah, that's that's I didn't actually know that Anthropic had enabled enabled a voice mode because I tend to use Anthropic on the. Um, on the computer and when I'm on the go, I actually use a. Uh, OpenAI's advanced voice mode because it's it's so smooth and conversation friendly, right? Like if you're driving or what have you, right? Um, so that's that's interesting. I will have to try that out. That's a good tip.

• 40:50Azeem Azhar?

  So that brings me to this this sense of what we we think might happen um with. The way in which apps get developed. So um everyone from Sam Altman to Satya. To any number of the VCs and startup founders have said. It's agent in twenty twenty five. I think OpenAI has operator as one of their their sort of code names for this. And the you know, the idea behind the agent is you can. Get these different systems to break down tasks and work within between each other to give you a much more kind of complex output. And the outcome of that actually is a vast increase in the number of tokens that get fired around because every agent is going off and doing its own its own piece. Um, do you think that?

• 41:36Azeem Azhar?

  That agent vision is something deeper than marketing. Where and slide where? Do you think it's a kind of real architectural shift in the way we use these systems?

• 41:49Speaker 02

  Yes, I think that agents are, you know, like, like OpenAI is like, uh, people, people hyped up agents a lot. They just kept hyping them up until OpenAI decided to release their sort of five levels. And, and the thing that really, uh, was interesting for a lot of folks is that reasoning was before agents. Right? Uh, because people thought there was, you know, there's language models, agents, right? And there's going to be this huge boom. Um, now, because reasoning is all about, and, and this is something that OpenAI released today, right? Uh, in their evals, what some of the evals that they showed was, hey, if you ask the same question five times, uh, you get five different responses, just because the way the models work, temperature, uh, the non-deterministic aspect of GPUs.

• 42:33Speaker 02

  Um, you get five different responses and those five different responses, often one of them, maybe two of them will be right. Right? And then many of them will be wrong. And so a lot of people game benchmarks by just like running it again and again, until they get the right answer. Um, and then they game benchmarks. So, so today with O1 Pro launch, they were, they showed that, hey, um, let's show the benchmark if we run this many times and we make, we only counted as correct is if every single time it responded, it was accurate. Right?

• 43:01Azeem Azhar?

  Right.

• 43:02Speaker 02

  And that was a real, real game changer to me, right? Uh, is that they're focusing on the reliability, right? And, and, and the big ass issue with agents is if you're not reliable, then you can't string together tasks. Agents require you to string together tasks and, and if your accuracy is 99%. Um, good luck because 10 tasks back to back, that's 99 to the 10th, right? This is, this is very bad. You know, you're under, you're under 80%. Right? Like you're under 80% hit rate. It's like, who wants some, who wants a human that hits 80%? No, like get the work done right. Uh, scaling intelligence is about hitting that reliability metric on basic tasks.

• 43:40Speaker 02

  And, and that's through reasoning, right? And, and once you hit that reliable, uh, high reliability, then you can have agents that launch work constantly. Um, and, and so I think agents are the holy grail, but people need to, uh, refactor and refocus to, hey, this thing puts something out that looks really smart. And I only have to edit a little bit. Fantastic productivity boost, by the way, right? I'll have to edit it a little bit. Um, but I can't dream of a world where robots or AI is doing a bunch of stuff for me without it. Consistently doing simple things 100% of the time, right? And then it can move to doing complex things 100% of the time, right? Because right now it's simple things 90% of the time and complex things 40, 50% of the time. Right?

• 44:24Speaker 02

  And and string those together, you're done, you're you're cooked. It's not going to work. So computer use robotics, all this agent stuff. This is not possible until we bring the reliability up and reasoning is really pervasive.

• 44:35Azeem Azhar?

  Well, we, yes, I mean, I use a few workflows that are a little bit agentic in the sense that I for research. So I may put in a question. I farm the question out to three different L L Ms that have different characteristics, different prompts. They bring their answers back. A fourth L L M is acts as a judge, a convenor, synthesizes the answers, evaluates them. And if it's not good enough, sends it back through the loop. And that process is actually really, really nice. It probably only works one time in six. But when it works, it's really good. And so it doesn't matter because I'm not running it in an automated fashion. I just kind of click, did it work? I checked 30 seconds later. If it did work, I'm I'm quits in, right? I'm I'm up. If it didn't work, I just, you know, go and use a different approach.

• 45:20Azeem Azhar?

  So so I do hear your point about about reliability. Okay, let's turn to predictions for 2025. And you are sort of in the forecast business, but I'm going to push you for predictions. So if you have like, let's let's give go for three, three predictions.

• 45:37Azeem Azhar?

  And at least one, I would like you to say, this is my most contrarian prediction out of them for for 2025 in the space.

• 45:44Speaker 02

  Hmm, contrarian predictions. I mean, I I think I think it's mostly just I continued. I believe the exponentials will continue to stack, right? There's a lot of like worry and fear and difficulty with people imagining that, you know, exponentials continue to stack because the scale of investment, the raw figures are frightening, right? Are people really going to go from investing 150, 200 billion dollars in data centers to 400, 500 billion? Yes, they will. That's that's that's a big question on people's mind. And is it 400, 500 or is it 300, right? And no, it's 400, 500 billion, right? This is, this is what the exponential, the, the requirement of rollout.

• 46:23Speaker 02

  Um, I think the other aspect is that we will start to see, um, you know, real software companies, uh, with big impacts from AI, right? You know, people talked about Zuck, you know, Zuck's year of efficiency, what Elon did with Twitter. Um, this is going to happen in a lot of software. Um, because the one area where it's indisputable, the value of AI, um, is, is, is coding, right? And, and the cost of coding is tanking faster than anyone can imagine. Um, and I think that this is going to lead to a number of software companies instituting that level of cuts, right? It's not just like, hey, we're going to cut 10, 20%. I think some software companies are going to be able to cut more than that, right?

• 47:04Speaker 02

  Uh, and that's going to be a real, real, I think we can, we're going to start seeing some level of, um, you know, automation and it's not, it's not, you know, the, the risk used to, or thought it used to always be, hey, it's the, it's the lowest people of society that are going to get, um, automated and thrown into poverty. It's like, no, no, no. This is like some of the top income earners who are, uh, being thrown, uh, uh, uh, off of the rails and their earnings are not growing nearly as fast. This is like a very scary prospect, um, for angst and such. Um, and then I think, I think another aspect is we'll see the largest private round of a company ever, right? Uh, we still haven't seen that yet, um, but I think we'll see, you know, multiple rounds north of $10 billion, right? Uh, we see Anthropic, OpenAI, XAI, we're all sub 10 billion.

• 47:48Speaker 02

  Um, Core Wave, sub 10 billion, right? Like they were all like teetering in the mid billions, um, a little bit higher, a little bit lower. I think we'll see rounds that are north of 10 billion from multiple firms, not just one, multiple. Um, because the progress is going to be so powerful and the clarity of, hey, what's going to happen in the future is going to be so, uh, strong that you'd be silly as a large multinational sovereign, uh, you know, investment firm or as a sovereign wealth fund to not deploy your capital in this. And by the way, these guys have not deployed much capital yet into AI, right? You're talking about the big sovereign wealth funds, um, or the, uh, as a percentage of AUM especially. They just haven't pushed much in AI. It's been quite small, so these people are also going to do that.

• 48:29Azeem Azhar?

  You've been a great sport there with three predictions, right? The four, $500 billion capex, uh, collapse in the cost of software, which we'll see, could see really significant headcount reductions and greater than $10 billion private company rounds showing up.

• 48:45Azeem Azhar?

  Let's evaluate your predictions now. Do you think you are more likely to be surprised to the upside? In other words, you are a bit more conservative with your predictions? Or most likely to be surprised to the downside in that you were too optimistic and aggressive with your predictions?

• 49:02Speaker 02

  So I think on the data center spend side, it can't be any higher. Chip supply chains take time to build. Data center supply chains take time to build. Sorry, it just can't be higher. I think it would be higher if we could build it faster, but we can't. And so I think that sort of is like, you know, like is burgeoning on A, it could be more, but actually it's capped here. So, you know, if there's upside or if there's a little bit of downside, we still hit that number basically. Versus what I think. And so I think that's sort of like the upper limit, right? Upper bound on terms of like CapEx on GPUs, data centers, ASICs, storage, networking, etc., right? On, on the side of automation, you know, I'm think I'm still like hedging my risk in that like, yeah, we'll see 20, 30% layoffs, right? Maybe, maybe we see bigger, right?

• 49:44Speaker 02

  With like O1 Pro or a, you know, another leap in reasoning models. You know, or a humongous cost decrease, right? Of reasoning models, we could see more, right? I just, I don't know yet, but I, I'm like, you know, only still pinning in like 20, 30% layoffs, right? For some companies and it's not across the ecosystem, just some. And then the, the sort of last one, I think, is where there's a lot of upside and downside, right? If models don't continue to progress, people can't keep raising money. And OpenAI sort of cemented that they can continue to raise money by, because they're doing the 12 days of Christmas. Was starting today with the O1, right? Pro and we'll see what they do on the rest of the days, because oh my.

• 50:24Azeem Azhar?

  Oh, it's gonna be nuts. Yeah, yeah, absolutely.

• 50:26Speaker 02

  Yeah, but, but regardless, like, you know, the, the challenge is like, hey, Anthropic's round was only Amazon. They tried to raise money from other people, to be clear, but they just couldn't get it. So they really need to put the metal to the pedal, pedal to the metal and, and actually like deliver big increases and improvements. Otherwise, where's the money, money train gonna stop, right? Same with XAI, right? Like they've raised multiple rounds. But now like they still, they still have nothing to show for it, right? They have got the most badass data center in the world, right? Possibly also the most polluting, but the most badass data center in the world. What are they gonna deliver with it? So, you know, things can happen. They can fail. They can mess up. So that's the other aspect of this.

• 51:06Azeem Azhar?

  It's definitely gonna be a fun year to watch, I think, 2025. Dylan, thank you so much for your time.

• 51:11Speaker 02

  Alrighty, thank you so much as well.