AI Semiconductor Landscape feat. Dylan Patel | BG2 w/ Bill Gurley & Brad Gerstner

Open Source bi-weekly convo w/ Bill Gurley and Brad Gerstner on all things tech, markets, investing & capitalism. This week they are joined by Dylan Patel, Founder & Chief Analyst at SemiAnalysis, to discuss origins of SemiAnalysis, Google's AI workload, NVIDIA's competitive edge, the shift to GPUs in data centers, the challenges of scaling AI pre-training, synthetic data generation, hyperscaler capital expenditures, the paradox of building bigger clusters despite claims that pretraining is obsolete, inference-time compute, NVIDIA's comparison to Cisco, evolving memory technology, chip competition, future predictions, & more. Enjoy another episode of BG2! Timestamps: (00:00) Intro (01:50) Dylan Patel Backstory (02:36) SemiAnalysis Backstory (04:18) Google's AI Workload (06:58) NVIDIA's Edge (10:59) NVIDIA's Incremental Differentiation (13:12) Potential Vulnerabilities for NVIDIA (17:18) The Shift to GPUs: What It Means for Data Centers (22:29) AI Pre-training Scaling Challenges (29:43) If Pretraining Is Dead, Why Bigger Clusters? (34:00) Synthetic Data Generation (36:26) Hyperscaler CapEx (38:12) Pre-training and Inference-tIme Reasoning (41:00) Cisco Comparison to NVIDIA (44:11) Inference-time Compute (53:18) The Future of AI Models and Market Dynamics (01:00:58) Evolving Memory Technology (01:06:46) Chip Competition (01:07:18) AMD (01:10:35) Google’s TPU (01:14:51) Amason’s Tranium (01:17:33) Predictions for 2025 and 2026 Produced by Benny Beausoleil Music by Yung Spielberg Available on Apple, Spotify, www.bg2pod.com http://www.bg2pod.com Follow: Brad Gerstner @altcap https://x.com/altcap Bill Gurley @bgurley https://x.com/bgurley⁠ Dylan Patel @dylan522p https://x.com/dylan522p BG2 Pod @bg2pod https://x.com/BG2Pod⁠

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Is scaling dead, then why is Mark Zuckerberg building a 2 gigawatt data center in Louisiana? Right. Why is, why is Amazon building these multi-gigawatt data centers? Why is Google, why is Microsoft building multiple gigawatt data centers? Plus, buying billions and billions of dollars of fiber to connect them together? Because they think, hey, I need to win on scale. So let me just connect all the data centers together with super high bandwidth. So then I can make them act like one data center, right? Right. Towards one job, right? So this is, this, this whole like is scaling over narrative falls on its face. When you see what the people who know the best are spending on.

• 0:44Brad Gerstner?

  Great to be here psyched you both are in the shop today. Dylan, this is one of the things we've been talking about all year, which is, you know, how the world of compute is radically changing. So Bill, why don't you, why don't you tell folks who, who Dylan is and let's get started.

• 0:59Bill Gurley?

  Yeah, we're thrilled to have Dylan Patel with us from Semi Analysis. Dylan has, has quickly built, I think, the most respected research group on global semiconductor industry. And so what we thought we'd do today is, is dive deep on, on the intersection, I think, between everything Dylan knows from a technical perspective about the architectures that are out there, about the scaling, about the key players in the market globally, the supply chain. And the best and the brightest of people we know are all listening and reading Dylan's work. And then connect it to some of the business issues that our audience cares about. And see where, see where it comes out. What I was hoping to do is kind of get a moment in time snapshot

• 1:43Bill Gurley?

  of all the semiconductor activity that relates to this big AI wave and try and put it in perspective.

• 1:50Brad Gerstner?

  Dylan, how'd you get into this?

• 1:52Speaker 00

  So when I was eight, my Xbox broke and I have immigrant parents. I grew up in rural Georgia, so I didn't have much to do besides be a nerd and I couldn't tell them I broke my Xbox. I had to open it up, short the temperature sensor and fix it. And that was the way to fix it. I didn't know what I was doing at the time, but then I stayed on those forums and I became a forum warrior. Right? You know, you see those people in the comments always yelling at you, Brad. You know, it's like, it's like, uh, that was me, right? As a child and you didn't know as a child then, but you know, it was just like, you know, arguing with people online as a child and then being passionate. As soon as I started making money, I was reading earnings from semiconductor companies and investing in them, you know, with my internship money and yeah, uh, reading technical stuff as well, of course, and then working a little bit and then, yeah.

• 2:36Brad Gerstner?

  And, and, and just tell us, give us that quick thumbnail on semi-analysis today. Like, what is the business?

• 2:41Speaker 00

  Yeah. So today we are a semiconductor research firm, AI research firm. We service companies. Our biggest customers are all hyperscalers, uh, the largest semiconductor companies, uh, private equity as well as hedge funds. And we sell data around where every data center in the world is, how, what the power is in each quarter, what, how the build outs are going. Um, we sell data around fabs. We track all 1,500 fabs in the world. For your purposes, only 50 of them matter, but like, you know, all 1,500 fabs around the world. Uh, same thing with, uh, the supply chain of like, whether it be cables or servers or boards or transformer substation equipment. We try and track all of this on a very, uh, number driven basis as well as forecasting. Um, and then we do consulting around those areas.

• 3:21Brad Gerstner?

  Yeah. So I mean, you know, Bill, uh, you and I just talked about this. I mean, for altimeter, our team talks with Dylan and Dylan's team all the time. I think you're right. Um, he's quickly emerged, really just through hustle, hard work, doing, doing the grindy stuff that matters. Um, I think is a, you know, a benchmark for what's going on in the semiconductor industry. And we're at this, you know, I suggested, we're two years into this, maybe, you know, this build out. And it's been hyperkinetic. And one of the things Bill and I are talking about is as we we enter the end of 2024, taking a deep breath, thinking about 25, 26 and beyond, because a lot of things are changing and there's a lot of debates. And it's going

• 4:02Brad Gerstner?

  to have consequence for trillions of dollars of value in the public markets, in the private markets, how the hyperscalers are investing and where we go from here. So Bill, why don't you take us a little bit through the start of the questions?

• 4:16Bill Gurley?

  Well, so I think if you're going to talk about AI and semiconductors, there's only one place to start, which is to talk about Nvidia broadly. Dylan, what percentage of global AI workloads do you think are on Nvidia chips right now?

• 4:31Speaker 00

  So I would say if you ignored Google, it'd be over 98%. But then when you bring Google into the mix, it's it's actually more like 70% because Google is really that large a percentage of AI workloads, especially production workloads, you know,

• 4:45Bill Gurley?

  By production, you mean in-house workloads for Google?

• 4:48Speaker 00

  Production as in things that are making money. Things that are making money, they're actually probably, it's probably even less than 70%, right? Because you think about Google search and Google Ads are the two largest, you know, two of the largest AI-driven businesses in the world, right? You know, the only things that are even comparable are like TikTok and Metas, right?

• 5:06Bill Gurley?

  And those Google workloads, I think it's important just to kind of frame this, those are running on Google's proprietary chips. They're they're non-LLM workloads, correct?

• 5:20Speaker 00

  So Google's production workloads for non-LLM and LLM run on their internal silicon. And I think one of the interesting things is, yes, you know, everyone will say, Google dropped the ball on transformers and LLMs, right? How did OpenAI do GPT, right? And not Google. But Google was running transformers, even in their search workload since 2018, 2019. The advent of BERT, which was one of the most most well-known, most popular transformers before we got to the GPT madness, is has been there in their production search workloads for years. So they run transformers on their own in their in their search and ads business as well.

• 5:59Bill Gurley?

  The going back to this, this number you'd use 98%. If you just look at, I guess, people workloads, people are purchasing to do work on their own. So you take the captives out, you're, you're at 98, right? This is a dominant landslide at this moment in time.

• 6:17Brad Gerstner?

  Back to Google for a second, they also are one of the big customers of Nvidia.

• 6:22Speaker 00

  They, they do buy a number of GPUs. They buy some for, you know, some YouTube video related workloads, internal workload, right? So not everything internal is like, is, is, is Google, is a GP, is a TPU, right? They do buy some for some other internal workloads, but by and large their GPU purchases are for Google Cloud to then rent out to customers.

• 6:41Bill Gurley?

  Right.

• 6:42Speaker 00

  Because they are, while they do have some customers for their internal silicon externally, such as Apple, the vast majority of their external rental business for AI in terms of cloud business is still GPUs.

• 6:55Bill Gurley?

  And that's Nvidia GPU.

• 6:57Speaker 00

  Correct, Nvidia GPUs.

• 6:58Bill Gurley?

  Why are they so dominant? Why, why is Nvidia so dominant?

• 7:03Speaker 00

  So I like to think of it as like a three headed dragon, right? I, I would say every semiconductor company in the world sucks at software, except for Nvidia. So their software, there's of course hardware. People don't realize that Nvidia is actually just much better at hardware than most people. They get to the newest technologies first and fastest because they drive like crazy towards hitting certain production goals, targets. They get chips out faster than other people from thought, design to deployed. And then the networking side of things, right? They bought Mellanox and they've driven really hard with the networking side of things. So those, those three things kind of combined to make a three headed dragon that no other semiconductor company can do alone.

• 7:45Bill Gurley?

  Yeah, I'd call out a piece you did Dylan, where you, you helped everyone visualize the complexity of one of these modern, cutting edge Nvidia deployments that involves the racks, the, the memory, the, the networking, the size of the scale of the whole thing. Super helpful.

• 8:02Brad Gerstner?

  I mean, there's this comparison oftentimes between companies that are truly standalone chip companies. They're not systems companies, they're not infrastructure companies and, and Nvidia. But I think one of the things that's deeply underappreciated is the level of competitive modes that Nvidia has. You know, software is becoming a bigger and bigger component. Of squeezing efficiencies and, you know, total cost of operation out of, out of these infrastructures. So talk to us a little bit about that schema, you know, that Bill's referring to, like, there are many different layers of systems architecture and how that's differentiated from maybe, you know, a custom ASIC or, or an AMD.

• 8:43Speaker 00

  Right. So when you, when you look broadly at the GPU, right, no one buys one chip for running an AI workload, right? Models have far exceeded that, right? You look at, you know, today's leading edge models like GPT-4 was, you know, over a trillion parameters, right? A trillion parameters is over a terabyte of memory. You, you can't get a chip with that capacity. You, a chip can't have enough performance to serve that model, even if it had enough memory capacity. So therefore you must tie together many chips together. And so what's interesting is that Nvidia has, has seen that and built an architecture that has many chips network together really well called NVLink. But funnily enough, and

• 9:24Speaker 00

  the thing that many ignore is that Google actually did this alongside Broadcom, you know, and they did it before Nvidia, right? You know, today everyone's freaking out about, or not freaking out, but like everyone's like very excited about Nvidia's Blackwell system, right? It is a rack of GPUs. That is the purchased unit, right? It's not one server, it's not one chip, it's a rack. And this rack weighs three tons and it has thousands and thousands of cables and all these things that Jensen will probably tell you, right? Extremely complex. Interestingly, Google did something very similar in 2018, right? With the TPU. Now, they couldn't do it alone, right? They know the software. They know, they know what the compute element needs to be, but they didn't know anything. They can't do, they can't do a lot of the other difficult things like package design, like

• 10:07Speaker 00

  networking. And so they had to work with other vendors like Broadcom to do this. And because Google had such a unified vision of where AI models were headed, they actually were able to build this system, this system architecture that was optimized for AI, right? Whereas at the time Nvidia was like, well, how big do we go? I'm sure they could have tried to scale up bigger, but what they saw as the primary workloads didn't require scaling to that degree, right? Now everyone sort of sees this and they're running towards it, but Nvidia's got Blackwell coming now. Competitors like AMD and others have to, you know, make an acquisition recently to help them get into the system design, right? Because building a chip Is one thing, but building many chips that connect together, cooling them appropriately,

• 10:50Speaker 00

  networking them together, making sure that it's reliable at that scale is a whole host of problems that semiconductor companies don't have the engineers for.

• 10:58Bill Gurley?

  Where would you say Nvidia has been investing the most in incremental differentiation?

• 11:06Speaker 00

  I would say for differentiating, Nvidia has primarily focused on supply chain things, which, you know, might sound like, oh, well, like, yeah, they're just like ordering stuff. No, no, no, no. You have to work deeply with the supply chain to build the next generation technology so that you can bring it to market before anyone else does, right? Because if Nvidia stands still, they will be eaten up, right? They're, you know, sort of the Andy Grove, only the paranoid will survive. Jensen is probably the most paranoid man in the world, right? He's known for many years, since before the LLM craze, all of his biggest customers were building AI chips, right? Before the LLM craze, his main competitors were like, oh, we should make GPUs.

• 11:48Speaker 00

  And yet he stays on top because he's bringing to market technologies at volume that no one else can, right? And so whether it be in networking, whether it be in optics, whether it be in water cooling, right? Whether it be in, you know, all sorts of other power delivery, all these things, he's bringing to market technologies that no one else has. And he has to work with the supply chain and teach those supply chain companies, and they're helping, obviously, they have their own capabilities, to build things that don't exist today. And Nvidia is trying to do this on an annual cadence now.

• 12:20Bill Gurley?

  That's incredible, yeah.

• 12:21Speaker 00

  Blackwell, Blackwell Ultra, Ruben, Ruben Ultra, they're going so fast, they're driving so many changes every year. Of course, people are gonna be like, oh, no, there, you know, there are some delays in Blackwell. Yeah, of course, you're driving, look how hard you're driving the supply chain.

• 12:33Brad Gerstner?

  Is that part, like, how big a part of the competitive advantage? Is the fact that they're now on this annual cadence, right? Because it seems like by going there, it almost precludes their competitors from catching up. Because even if you skate to where Blackwell is, right? You're already on next generation within 12 months. He's already planning two or three generations ahead because it's only two, two to three years ahead.

• 12:57Speaker 00

  Well, the funny thing is, a lot of people in Nvidia will say Jensen doesn't clear plan more than a year, a year and a half out. Because they change things and they'll deploy them out that fast, right? No semi, every other semiconductor company takes years to deploy, you know, at, at, make architecture changes, but.

• 13:12Bill Gurley?

  You said if they stand still, they're, they would, they would have competition like what, what would be their area of vulnerability? Or, or what would have to play out in the market for other alternatives to take more share of the workloads?

• 13:29Speaker 00

  Yeah. So, so the main thing for Nvidia is, you know, hey, this workload is this big, right? It's, it's, it's well over a hundred billion dollars of spend for the biggest customers. They have multiple customers that are spending billions of dollars.

• 13:42Bill Gurley?

  Yeah.

• 13:42Speaker 00

  I can hire enough engineers to figure out what, how to run my model on other hardware, right? Now, maybe I can't figure out how to train on other hardware, but I can figure out how to run it for inference on other hardware. So Nvidia's moat in, in inference is actually a lot smaller on software. But it's a lot bigger on, hey, they just have the best hardware. Now, what is, what does the best hardware mean? It means capital cost and it means operation cost and then it means performance, right? Performance TCO.

• 14:07Brad Gerstner?

  Yes.

• 14:08Speaker 00

  And Nvidia's whole moat here is if they stand still, the performance TCO doesn't grow. But interestingly, they are, right? Like with Blackwell, not only is it way, way, way faster, anywhere from 10 to 15 times on really large models for inference, because they've optimized it for very large language models. They've also decided, hey, we're going to cut our margin too somewhat because I'm competing with Amazon's, you know, chip and TPU and, and AMD and all these things. They've decided to cut their margin too. So, so between all these things, they've, they've decided that they need to push performance TCO, not 2X every two years, right? You know, Moore's law, right? They've decided they need to push performance. Five performance TCO, five X, maybe every year, right? At least that's what Blackwell is and we'll see what Ruben does.

• 14:53Speaker 00

  But you know, five X plus in a single year for performance TCO is an insane pace. Right? Um, and then you stack on top like, hey, AI models are actually getting a lot better for the same size. The cost for delivering L L Ms is, is tanking, which is going to induce demand. Right? So this-

• 15:09Bill Gurley?

  And just to, to clarify one thing you said, or at least restated to make sure. I think when you said the software is more important for, for training, you meant Kuda is more of a differentiator on training than it is on inference.

• 15:22Speaker 00

  So, so I think a lot of people in investor community, you know, call Kuda, which is just like one layer for all of Nvidia software. There's, there's a lot of layers of software, but for, for simplicity's sake, you know, regarding networking or what runs on switches or what runs on, you know, all sorts of things, fleet management stuff, all sorts of stuff that Nvidia makes that we'll just call Kuda for simplicity's sake. But all of this software is, is stupendously difficult to replicate. In fact, no one else has deployments to do that besides the hyperscalers, right? And a few thousand GPUs is, is like a Microsoft inference cluster, right? It's not, it's not a training cluster. So, so when you talk about, hey, what is, what is, what is the difficulty here, right? On training, this is users constantly experimenting, right?

• 16:05Speaker 00

  Researchers saying, hey, let's try this, let's try that, let's try this, let's try that. I don't have time to optimize and hand wring out the performance. I rely on Nvidia's performance to be quite good with existing software stacks or very little effort. Right? But when I go to inference, Microsoft is deploying five, six models across how many billions of revenue? Right? So all of OpenAI's revenue plus whatever they have in co-pilot and all that.

• 16:30Bill Gurley?

  Ten billion of inference revenue.

• 16:31Speaker 00

  Yeah, so, so they have $10 billion of revenue here and they're deploying five models, right? GPT four, four O, you know, four O mini. And now, you know, the reasoning, yeah, the reasoning models, um, O one and yeah. So, so it's like they're, they're deploying very few models. And those change what every six months, right? So every six months they get a new model and they deploy that. So within that timeframe, you can, you can hand wring out the performance. And so Microsoft has deployed- GPT style models on on other competitors hardware such as AMD. They've and some of their own, but mostly AMD. And so they can ring that out with software because they can spend hundreds of engineers, dozens of engineers hours, hundreds of engineer hours or thousands of engineer hours on working this out.

• 17:14Speaker 00

  Because it's such a unified sort of workload, right?

• 17:18Brad Gerstner?

  I want to I want to get you to comment on this chart. This is a chart we showed earlier in the year that I think was kind of a a moment for me with Jensen. When he was in, I think, the Middle East. And for the first time, he said, not only are we going to have a trillion dollars of new AI workloads over the course of the next four years. He said, but we're also going to have a trillion dollars of CPU replacement, of data center replacement workloads over the course of the next four years. So that's an effort to model it out. And I, you know, we referenced it on on the pod with him. And and he seemed to indicate that it was directionally correct, right? That he still believes that it's not just about, because there's a lot of fuss in the world about,

• 18:03Brad Gerstner?

  you know, pre-training and what if what if pre-training doesn't continue apace? And it seemed to suggest that there was a lot of AI workloads that had nothing to do with pre-training that they're working on, but also that they had all of this data center replacement. Do you buy that? I've heard a lot of people push back on the data center replacement and say, there's no way people are going to, you know, rebuild a CPU data center with a bunch of Nvidia GPUs. It just doesn't make any sense. But he, his argument is that an increasing number of these applications, even things like Excel and PowerPoint are becoming machine learning applications and require accelerated compute.

• 18:41Speaker 00

  Nvidia has been pushing non-AI workloads for accelerators for a very long time, right? Professional visualization, right? Pixar uses a ton of GPUs, right, to make every movie. You know, all these Siemens engineering applications, right? All these things do use GPUs, right? I would say there are a drop in the bucket compared to, you know, AI. The other aspect I would say is, and this is sort of a bit contentious with your chart, I think, but. I'd be a mainframe, sell more volume and revenue every single cycle, right? So, you know, yes, no one in the bay uses mainframes or talks about mainframes, but they're still growing, right? And so like, I would, I would say the same applies to CPUs, right? To classic workloads.

• 19:24Speaker 00

  Just because, you know, AI is here doesn't mean web serving is like going to slow down or databasing is going to slow down. Now, now what does happen is that that line is like this and the AI line is like this. And furthermore, right? Like when you talk about what, you know, hey, these applications, they're now AI, right? You know, Excel with Copilot or Word with Copilot, whatever, right? There's, they're going to be, they're still going to have all of those classic operations. You don't get rid of what you used to have, right? Southwest doesn't stop booking flights. They just run AI analytics on top of their flights to maybe, you know, do pricing better or whatever, right? So I would say that still happens, but there is an element of replacement that is sort of misunderstood, right? Which is given how much people are deploying, how tight the supply chains for data centers

• 20:09Speaker 00

  are, data centers take longer, they're longer time supply chains, unfortunately, right? Which is why you see things like what Elon's doing. But when you, when you think about that, well, how can I get power then, right? So you can do what CoreWeave is doing and go to crypto mining companies and just like clear them out and put a bunch of GPUs in them, right? retrofit the data center, put a GPU, data, uh, GPUs in them like they're doing in Texas. Or you can do what some of these other folks are doing, which is, hey, well, my depreciation for CPU servers has gone from three years to six years in just a handful of years. Why? Because Intel's progress has been this, right? So, so in reality, like the old Intel CPU is not that much better. But all of a sudden, over the last couple of years, AMD's burst onto the scene, ARM CPUs have burst onto the scene, Intel's started to right the ship.

• 20:54Speaker 00

  Now I can upgrade the, the most, the plurality of, of Amazon's CPUs in their data centers are 24 core Intel CPUs from, that were manufactured from 2015 to 2020. Same, more or less the same architecture. There's 24 core CPU. I can buy 128 or 192 core CPU now today, where each CPU core is higher performance. And well, if I just replace like six servers with one, I've basically invented power out of thin air. Right? I mean, like, you know, in effect, because these old servers, which are six plus years old, or even, you know, they can, they can just be deprecated and put. So with CapEx of new servers, I can replace these old servers. And now, you know, when I do, every time I do that, I can throw another AI server in there.

• 21:39Speaker 00

  Right? So this is sort of the, yes, there is some replacement. I still need more total capacity, but that total capacity can be served by fewer machines, maybe if I buy new ones. And generally, the market is not going to shrink. It's still going to grow, just nowhere close to what AI is. And AI is causing this behavior of I need to replace so I can get power.

• 21:58Brad Gerstner?

  Hey, Bill, this, this reminds me of a point Satya made on the pod last week that I've seen replayed a bunch of times. And I think is fairly misunderstood. He said last week on the pod that he was power and data center constrained, not chip constrained. What I think it was, was more a assessment on the real bottleneck, which is data centers and power as opposed to GPUs, because GPUs have, have come online. And so I think the, the case you just made, I think helps to clarify that.

• 22:29Bill Gurley?

  Well, before, before we dive into the alternatives to Nvidia, I thought we would hit on this, this pre-training scaling debate that you wrote about in your last piece, Dylan, and we've already talked about quite a bit. But, but why don't you give us your view of what's going on there? I think Ilya was the one, the, the most credible AI specialist that brought this up, and then it got repeated and, and cross analyzed quite a bit.

• 22:58Brad Gerstner?

  And Bill, just to repeat what it is.

• 23:01Bill Gurley?

  Oh, fair enough.

• 23:01Brad Gerstner?

  I think Ilya said, you know, data is the fossil fuel of the internet of AI, that we've consumed all the fossil fuel because we only have but one internet. And so the huge gains we got from pre-training are not going to be repeated.

• 23:16Bill Gurley?

  And some, some experts had predicted a data, the data would run out a year or two ago. So it wasn't, it wasn't. Like out of, out of nowhere that that argument came to light. Anyway, let's hear what Dylan has to say.

• 23:31Speaker 00

  So pre-training scaling laws are pretty simple, right? You get more compute and then I throw it at a model and it'll get better. Now, what is that? That breaks out into two axes, right? Data and parameters, right? You know, the bigger the model, the more data, the better. And there's actually an optimal ratio, right? So Google published a paper called Chinchilla, which says the optimal ratio of data to parameter, you know, model size. And, and that's the scaling thing. Now, what happens when the data runs out? Well, I don't really get much more data.

• 23:59Bill Gurley?

  Right.

• 23:59Speaker 00

  But I keep growing the size of the model because my, my budget for compute keeps growing. This is a bit not fair though, right? We have barely, barely, barely tapped video data, right? So there is a significant amount of data that's not tapped. It's just video data is so much more information than written data, right? And so therefore you're throwing that away. But I think that's like, that's part of the, the, the like, you know, that there's a bit of a misunderstanding there. But more importantly, text is the most efficient domain, right? Humans generally, yes, a picture paints a thousand words. But if I write a hundred words, I can probably, you can tell, figure out faster.

• 24:35Bill Gurley?

  And the transcripts of most of those videos were already.

• 24:38Speaker 00

  Yeah, the transcripts of many of those videos are in there already. But you know, regardless, the data, data is like a big axis. Now, the problem is, this is only pre-training, right? Quote, pre. Training a model is more than just the pre-training, right? There's, there's many elements of it. And so people have been talking about, hey, inference time compute. Yeah, that's important, right? You can continue to scale models if you figure out how to make them think and recursively like be like, oh, that's not right. Let me think this way. Oh, that's not right. That let me, you know, much like, you know, you don't have, hire an intern and say, hey, what's the answer to X? Or you don't hire a PhD and say, hey, what's the answer to X? You're like, go work on this and then they come back and bring something to you. So inference time compute is important, but really what's more important is as we continue to get more and more compute,

• 25:22Speaker 00

  can we improve models if data is run out? And the answer is you can create data out of thin air almost, right? In certain domains, right? And so this is the whole, the debate around scaling laws is how Can we create data, right? And so what is, what is, what is Ilya's company doing most likely? What is, what is Mira's company doing most likely? Amira Miradi, a CTO of, of OpenAI. What are, what are, you know, all these companies focused on? OpenAI? What are all these companies focused? They have Noam Brown, who's like sort of the, one of the big reasoning people on roadshows, but just going and speaking everywhere, basically, right? What, what are they doing, right? They're saying, hey, we can still improve these models. Yes, spending compute at inference time is important, but what do we do at training time?

• 26:06Speaker 00

  Cause you, you can't just tell a model, think more and it gets better. You have to do a lot of training time. And so what that is, is I take the model, I take an objective function I have, right? What is the square root of 81? Right? Now, if I told you the square, ask, ask many people, what's the square root of 81? Many could answer, but I bet many people could answer if they thought about it more, like almost, you know, a lot more people, right? And maybe this is a simplistic problem, but you say, hey, let's have the existing model do that. Let's have it just run every possible, uh, you know, not every possible, many permutations of this. Start off with say five, and then anytime it's unsure, branch into multiple. So you start out, you have hundreds of quote unquote rollouts or trajectories of generated data. Most of this is garbage, right? You prune it down to, hey, only these paths got to the right answer.

• 26:52Speaker 00

  Okay, now I feed that and that is now new training data. And so I do this with every possible area where I can do functional verification. Functional verification, i.e. hey, this code compiles. Hey, this unit test that I have in my code base, um, how can I generate the solution? How can I generate the function? Okay, now, now, and, and you do this over and over and over and over again, across many, many, many different domains where you can functionally prove it's real. You generate all this data, you throw away the vast, vast majority of it, but you now have some chains of thought that you can train the model on, which then it will learn how to do that more effectively and it generalizes outside of it, right? And so this is the whole domain. Now, when you talk about scaling laws, It's, it's point of diminishing returns is kind of uh not proven yet, by the way,

• 27:37Speaker 00

  right? Because it's, it's more so, hey, the scaling laws are a log, log axis, a log IE, it takes 10x more investment to get the next iteration. Well, 10x more investment, you know, it, it, you know, going from 30 million to 300 million, 300 million to 3 billion is relevant. But when Sam wants to go from 3 billion to 30 billion, it's, it's a little difficult to raise that money, right? That's why, you know, the most recent rounds are a bit like, oh crap, we can't spend 30 billion on the next run. And so the question is, well, that's just one axis. Where have we gone on synthetic data? Oh, we're still like very early days, right? We've, we've spent tens of millions of dollars maybe on synthetic data.

• 28:16Bill Gurley?

  But isn't it with synthetic data, you used a qualifier in certain domains. When they released O1, it also had a qualifier like that in certain domains. I'm just saying those two scaling access do better in certain domains and aren't as applicable in others. And we have to figure that out.

• 28:33Speaker 00

  Yeah, I think, I think one of the interesting things about AI is, is at first in 2022, 2023, with the release of diffusion models, with the release of text models, people like, oh, wow, artists are the one that are the most, you know, out of luck, not, not technical jobs. Actually, these things suck at technical jobs. But with these new acts, this new axis of synthetic data and test time compute, actually, where are the areas where we can teach the model? We can't teach it what good art is, because we have no way to functionally prove what good art is. We can teach it to write really good software. We can teach it how to do mathematical proofs. We can teach it how to engineer systems, because there are, while there are trade offs, and this is not like, it's not just a one zero thing, especially on engineering systems,

• 29:16Speaker 00

  this is something you can functionally verify, is this works or not, or this is correct or not?

• 29:21Bill Gurley?

  The output and then the model can iterate more often.

• 29:24Speaker 00

  Exactly.

• 29:25Bill Gurley?

  It goes back to the AlphaGo thing and why that was a sandbox that could allow for novel, uh, novel moves and plays, because you could traverse it and, and run synthetically. You could just let it, let it create and create and create.

• 29:43Brad Gerstner?

  Putting on my investor hat, public investor hat here. There is a lot of tension in the world over Nvidia as we look forward at 2025. And this question of pre-training, right? And if in fact, you know, we've seen we pluck 90% of the low hanging fruit that comes from pre-training, then do people really need to buy bigger clusters? And I think there's a view in the world, particularly post Ilya's comments, that, you know, no, the 90% benefit of pre-training is gone. But then I look at, you know, the comments out of Hock Tan this week.

• 30:18Brad Gerstner?

  You know, during their earnings call that all the hyperscalers are building these million, you know, XPU clusters. I look at, you know, the commentary out of X.ai that they're going to build 200 or 300,000 GPU clusters. You know, Meta reportedly building much bigger clusters. Microsoft building much bigger clusters. How do you square those two things, right? If everybody's right and pre-training is dead, then why is everybody building much bigger clusters?

• 30:44Speaker 00

  So the scaling, right, goes back to what's the optimal ratio? What's the, how do we continue to grow, right? Just blindly growing parameter count when we don't have any more data, or the data is very hard to get, i.e. because it's video data, wouldn't give you so many gains. And then there's also the access of it's a log chart, right? You need 10x more to get the next job, right? So when you look at both of those, oh crap, like I need to invest 10x more. And I might not get the full gain because I don't have the data. But the, the data generation side, we are so early days with this, right?

• 31:15Brad Gerstner?

  So the point is, I'm still going to squeak out enough gain that it's a positive return, particularly when you look at the competitive dynamic. You know, our models versus our competitor models. So it's a rational decision to go from 100,000 to 200,000 or 300,000, even if, you know, the, the kind of big one-time gain in pre-training is behind us.

• 31:36Speaker 00

  Or rather, it's exponentially more, it's logarithmically more expensive to do that gain, right? So it's still there, like the gain is still there, but like the sort of whole like Orion has failed sort of narrative around OpenAI's model. And they didn't release Orion, right? They released O1, which is sort of a different axis. It's partially because, hey, this is, you know, because of these like data issues, but it's partially because they did not scale 10x, right? Because scaling 10x from four to this is actually was like.

• 32:03Brad Gerstner?

  I think this is Gavin's point, right?

• 32:05Bill Gurley?

  I would also, let's go to Gavin in a second. One of the reasons this became controversial, I think, is. And Dario and Sam had prior to this moment, at least the way I heard them, made it sound like they were just gonna build the next biggest thing and get the same amount of gain. They had left that impression. And so we get to this place, as you described, and it's not quite like that. And then people go, oh, what does that mean? Like, it causes them to raise their head up.

• 32:38Speaker 00

  So I think, I think they have never said the Chinchilla scaling laws were what delivers us, you know, AGI, right? They've had scaling. Scaling is, you need a lot of compute. And guess what? If you have, you know, if you have to generate a ton of data and throw away most of it because, hey, only some of the paths are good, you're spending a ton of compute at training time, right? And this is sort of the axis that is like, we may actually see models improve faster in the next six months to a year than we saw them improve in the last year. Because there's this new axis of synthetic data generation and the amount of compute we can throw at it is, we're still, we're still right here in the scaling law, right? We're not here. We haven't pushed it to billions of dollars spent on synthetic

• 33:21Speaker 00

  data generation, functional verification, reasoning, training. We've only spent millions, tens of millions of dollars, right? So what happens when we scale that up? So there's, there is a new axis of spending money. And then there's, of course, test time compute as well, i.e. spending time at inference to get better and better. So it's possible. And in fact, many people at these labs believe that the next year of gains or the next six months of gains will be faster because they've unlocked this new axis through a new methodology, right? And it's still scale, right? Because this requires stupendous amounts of compute. You're generating so much more data than exists on the web, and then you're throwing away most of it. But you're generating so much data that you have to, you have to run the model constantly, right?

• 34:00Bill Gurley?

  What, what domains do you think are, are most applicable with this approach? Like, where, where, where would synthetic data be most effective? And maybe, maybe you do a Both a pro and a like a scenario where it's going to be really good and one where it wouldn't work as well.

• 34:20Speaker 00

  Yeah, so I think that that goes back to the point around what can we functionally verify is true or not? What can I grade and it's not subjective? What class can you know you take in college and you get the card, you get the thing back and you're like, oh, this is BS or like dang, I messed that up, right? There's certain classes where you can-

• 34:37Bill Gurley?

  There's like a determinism of of grading the output.

• 34:42Speaker 00

  Right, exactly. So if there if it can be functionally verified, amazing. If it has to be judged, right? So there's sort of two ways to judge an output, right? There is, you know, without using humans, right? This is sort of the whole scale AI, right? What were they initially doing? They were using a ton of manpower to to create good data, right? Label data. But now, humans don't scale for this level of data, right? Humans post on the internet every day and we've already tapped that out, right? Kind of more or less on the text of it.

• 35:09Bill Gurley?

  What are domains that work?

• 35:10Speaker 00

  So so this, these are domains where, hey, in Google, when they push data to, you know, any of their services, they have tons of unit tests. These unit tests make sure everything's working. Well, why can't I have the LLM just generate a ton of outputs and then use those unit tests to grade those outputs, right? Because it's pass or fail, right? It's not. And then you can also like grade these outputs in other ways. Like, oh, it takes this long to run versus this long to run. So then you have various, there's other areas such as like, hey, image generation. Well, actually, it's harder to say which which image looks more beautiful to you versus me, you know, I might, I might like, you know, some like sunsets and flowers and you might like the beach, right? You can't really argue what is good there. So there's no functional verification.

• 35:51Speaker 00

  There is only subjective, right? So the objective nature of this is where, so where do we have objective grading, right? We have that in code, we have that in math, we have that in engineering. And while they could, these can be complex, like, hey, engineering is not just this is the best solution. It's, hey, given all the resources we've had and given all these trade offs, we think this is the best trade off, right? That's usually what engineering ends up being. Well, I can still, I can still look at all these axes, right? Whereas in in in subjective things, right? Like, hey, what's the best way to write this email? Or what's the best way to negotiate with this person? That's difficult, right? That's not something that is objective.

• 36:26Brad Gerstner?

  What are you hearing from the hyperscalers? I mean, they're all out there saying our capex is going up next year, we're building larger clusters. You know, is that in fact happening? Like, what's happening out there?

• 36:36Speaker 00

  Yeah, so I think when you look at the Street's estimates for CapEx, they're all far too low. You know, based on a few factors, right? So when we track every data center in the world, and it's insane how much, especially Microsoft, and now Meta and Amazon and, you know, and many others, right? But those guys specifically are spending on data center capacity. And as that power comes online, which you can track pretty easily if you look at all of the different regulatory filings, and you satellite imagery, all these things that we do, you can see that, hey, they're going to have this much data center capacity, right?

• 37:09Brad Gerstner?

  Right. So it's accelerating.

• 37:12Speaker 00

  What are you going to fill in there, right? It turns out you have to fill, to fill it up, you know, you can make some estimates around how much power is each GPU? All in, everything, right? Satya said he's going to slow down that a little bit, but they've signed deals for next year rentals.

• 37:24Brad Gerstner?

  Right.

• 37:25Speaker 00

  Right, for some, in some of these cases, right?

• 37:27Brad Gerstner?

  And it's part of the reason, he said, is he expects his cloud revenue in the first half of next year to accelerate. Because he said, we're going to have a lot more data center capacity, and we're currently capacity constrained.

• 37:35Speaker 00

  So, you know, what they're, you know, like, again, going back to the, is scaling dead? Then why is Mark Zuckerberg building a 2 gigawatt data center in Louisiana?

• 37:42Brad Gerstner?

  Right.

• 37:42Speaker 00

  Why is, why is Amazon building these multi-gigawatt data centers? Why is Google, why is Microsoft building multiple gigawatt data centers? Plus, buying billions and billions of dollars of fiber to connect them together? Because they think, hey, I need to win on scale. So let me just connect all the data centers together with super high bandwidth. So then I can make them act like one data center, right? Towards one job, right? So this is, this, this whole, like, is scaling over narrative falls on its face when you see what the people who know the best are spending on, right?

• 38:12Bill Gurley?

  You talked a lot at the beginning about Nvidia's differentiation around these large coherent clusters that are used in pre-training. Can you see anything, like, I guess, one, someone might be super bullish on inference and keep building out a data center. But they might have thought they were going to go from 100,000 nodes to 200 to 400 and might not be doing that right now, if, if this pre-training thing is real. Are you seeing anything that gives you? Any visibility on that dimension?

• 38:44Speaker 00

  So when you think about training a neural network, right? It is it is doing a forwards pass and a backwards pass, right? Forwards pass is generating the data, basically. And it's half as much compute as the backwards pass, which is updating the weights. When you look at this new paradigm of synthetic data generation, grading the outputs, and then training the model, you are going to do many, many, many forward passes before you do a backwards pass. What is serving a user? That's also just a forwards pass. So it turns out that there is a lot of inference in training, right? In fact, there's more inference in training than there is updating the model weights, because you have to generate hundreds of possibilities. And then, oh, you only train on a couple of them, right? So there is that paradigm is very relevant.

• 39:27Speaker 00

  The other paradigm I would say that is very relevant is when you're when you're training a model, do you necessarily need to be co-located for every single aspect of it? Right. Right? And this is

• 39:39Bill Gurley?

  And what's the answer?

• 39:40Speaker 00

  The answer is depends on what you're doing. If you're in the pre-training paradigm, then maybe you don't. Yeah, you need it to be co-located, right? You need everything to be in one spot. Yeah, why Microsoft in Q1 and Q2 signed these massive fiber deals, right? And why are they? Why are they building multiple similar sized data centers in Wisconsin and Atlanta and Texas? And so on and so forth, right? And Arizona? Why are they doing that? Because they already see the research is there for being able to split the workload more appropriately, which is, hey, this data center, it's not serving users. It's running inference. It's just running inference and then, you know, throwing away most of the output, because some of the output is good.

• 40:18Speaker 00

  Because I'm grading it, right? And it's doing they're doing this while they're also updating the model in other areas. So the whole paradigm of training, you know, pre-training is not slowing down. It's just it's logarithmically more expensive each for each generation for each incremental improvement.

• 40:32Bill Gurley?

  People are finding other ways to

• 40:33Speaker 00

  But there's other ways to not just continue this, but hey, I don't need a logarithmic increase in spend to get the next generation of improvement. In fact, through this reasoning, training and inference, I can get that logarithmic improvement in the model without ever spending that. Now, I'm gonna I'm gonna do both, right? Because this is because each model jump has unlocked huge value, right? The

• 40:58Brad Gerstner?

  You know, the thing that I think so interesting, you know, I hear Kramer on CNBC this morning, you know, and they're talking about is this Cisco from 2000. I was in Omaha, Bill, Sunday night for dinner. You know, they're obviously big investors in utilities and they're watching what's going on in the data center build out. And they're like, is this Cisco from 2000? So I had my team pull up a chart for Cisco, you know, 2000. And we'll show it on the pod, but you know, they peaked at like 120 PE, right? And you know, if you look at the falloff that occurred in revenue and in EBITDA, you know, and then it had 70% compression in the, in the price earnings multiple, right? So the price earnings multiple went from 120 down to something closer to 30.

• 41:43Brad Gerstner?

  And so I said to, you know, in this dinner conversation, I said, well, Nvidia's, you know, PE today is 30. It's not 120, right? So you would have to think that there would be 70% PE compression from here, or that their revenue was going to fall by 70%, or that their earnings were going to fall by 70%. You know, in order to have a Cisco-like event, we all have post-traumatic stress about that. I mean, hell, I, you know, I lived through that too. Nobody wants to repeat that. But when people make that comparison, it strikes me as uninformed, right? It's not to say that there can't be a pullback, but given what you just told us about the build out next year, given what you told us about scaling laws continuing, you know, what do you think when you hear,

• 42:26Brad Gerstner?

  you know, the Cisco comparison when people are talking about Nvidia?

• 42:30Speaker 00

  Yeah, so I think there's a couple of things that are not fair, right? Cisco's revenue, a lot of it was funded through private slash credit investments into building out telecom infrastructure, right? When we look at Nvidia's revenue sources, very little of it is private slash credit, right? And in some cases, yes, it's private slash credit like CoreWeave, right? But CoreWeave is just backstopped by Microsoft. There is significant amounts of like difference in like, what is the source of the capital, right? The other thing is, at the peak of the dot com, you know, especially when you inflation adjust it, the private capital entering the space. Was much larger than it is today, right? As much as people say the venture markets are going crazy, throwing these huge valuations at,

• 43:15Speaker 00

  you know, all these companies. And we were just talking about this before the show, but like, hey, the venture markets, the private markets have not even tapped in, right? Guess what? Private market money like in the Middle East in these sovereign wealth funds? It's not coming in yet, has barely come in, right? Why, why, why, why wouldn't there be a lot more spend from them as well? Right? Um, and so there is a significant amount of difference of capital. The source is positive cash flows of the most profitable companies that have ever lived or ever existed in humanity versus credit speculatively spent, right? So I think that is like a big aspect. Uh, that also gives it a bit of a knob, right? These companies that are profitable will be a bit more rational.

• 43:55Bill Gurley?

  I think corporate America is investing more in AI and with more conviction than they did even in the internet wave also.

• 44:03Brad Gerstner?

  Maybe we can switch a little bit. You've mentioned inference time reasoning a, a, a few times now. It's clearly a new vector of scaling intelligence. And I read some of your analysis recently about how inference time reasoning is way more compute intensive, right? Than simply pre-trained, you know, scaling pre-training. Why don't you walk us through? We have a really interesting graph here about why that's the case. Um, that we'll, we'll post as well. But why don't you walk us through first, just kind of what inference time reasoning is from the perspective of compute consumption? Why it's so much more compute intensive? And so leading to the conclusion that if, if this is in fact

• 44:44Brad Gerstner?

  going to continue to scale as, as a new vector of, of, of, of intelligence, it, it looks like it's going to be even more compute intensive than what came before it.

• 44:54Speaker 00

  Yeah, so pre-training maybe slowing out or it's, it's too expensive, but there's these other aspects of synthetic data generation and inference time compute. Inference time compute is, um, on the surface, sounds amazing, right? I don't need to spend more training a model. But when you think about it for a second, this is actually very, very, uh, this is not the way you want to scale. You only do that because you have to, right? Um, the way because, or think about it. G P T four was trained with hundreds of billions of dollars and it's generating billions of dollars of revenue. Hundreds of millions of dollars. Hundreds of millions of dollars to train G P T four. Yeah. And it's generating billions of dollars of revenue. So when you say like, hey, Microsoft's capex is nuts, sure. But their spend on G P T four was very reasonable relative to the R O I they're getting out of it, right?

• 45:39Speaker 00

  Now, when you say, hey, I want the next game. If I just spend, you know, sort of a large amount of capital and train a better model, awesome. But if I don't have to spend that large amount of capital and I deploy it, you know, I deploy this better model without, you know, at the time of revenue generation, rather than ahead of time when I'm training the model. This also sounds awesome, but this, this comes with this big trade off, right? When you're running reasoning, right, you're, you're having the model generate a lot. And then the answer is only a portion of that, right? Today, when you open up chat G P T, you use G P T four, four O, you say something, you get a response. Correct. You send something, you get a response, whatever it is, right? All of the stuff that's being generated is being sent to you.

• 46:23Speaker 00

  Now you're having this reasoning phase, right? And OpenAI doesn't want to show you, but you know, there's some open source Chinese models like Alibaba and DeepSeek. They've released some open source models, which are not quite as good as OpenAI, of course, but they show you what that reasoning looks like if you want to. And OpenAI has released some examples. It generates tons of things. It's like, it, it sometimes switches between Chinese and English, right? Like whatever it is, it's thinking, right? It's turning, it's like this, this, this, this, oh, should I do it this way? Should I break it down into these steps? And then it comes out with an answer, right? Now, on the surface, awesome, I didn't have to spend any more on R and D or capital, right? I'm saying this in the loose terms, they don't, they don't, they don't treat training models as R and D. I think on Microsoft on a financial basis, but they don't have to treat this, they don't have this R and D ahead of time, right?

• 47:07Speaker 00

  You, you get it at spend time, but. Think about what that means, right? If for you, right? For example, one simple thing that we, you know, we've done a lot of test on is, hey, generate me this, uh, this soft, this, this code, right? Like make this function. Great. Uh, I put, I described the function and, you know, a few hundred words. I get back a response that's, you know, a thousand words. Awesome. And I'm paying per token. When I do this with O1 or any other reasoning model, I'm sending the same response, right? Few hundred tokens. I'm paying for that. I'm getting the same response roughly, you know, a thousand tokens. But in the middle, there was 10,000 tokens of it thinking. Right. Now, what, what does that 10,000 tokens of thinking actually mean? It means, well, the model's spitting out 10 times as many tokens.

• 47:49Speaker 00

  Well, if Microsoft's generating, you know, call it $10 billion of inference revenue, and their margins on that are good. They've stated this, right? They're anywhere from 50 to 70%, um, depending on how you count the OpenAI profit share. Um, you know, anywhere from 50 to 70% gross margins. Their cost for that is a few billion dollars for $10 billion of revenue. Right. If, now, obviously, the better model gets to charge more, right? So O1 does charge a lot more, but you're now increasing your cost from, hey, I outputted a thousand tokens to I outputted 11,000 tokens. I've 10x'd my spend to generate, now, not the same thing, right? It's higher quality, right? Correct. Um, and, and that's, that's only part of it. That's deceptively simple. It's not just 10x, right?

• 48:31Speaker 00

  Because if you go look at O1, despite it being the same, uh, model architecture as GPT-4O, it actually costs significantly more per token as well. And that's because of, you know, sort of this chart that we're looking at here, right? And this chart shows, hey, what is GPT-4O, right? If I'm generating, you know, call it a thousand tokens, right? And, and that's what GPT-4O on the bottom right is. Uh, of Llama 4.5B, this is an open model, so it's easier to simulate, um, you know, the exact, uh, metrics of it. But, you know, if I'm doing that, I'm keeping my users', uh, you know, experience of the model constant, um, i.e. the number of tokens they're getting at the speed. Then, you know, when I, when I ask it a question, it generates the unit, generates the code, whatever it is. I'm, I can group together many users' requests.

• 49:14Speaker 00

  I can group together over 256 users' requests on one server for Llama 4.5B of, of, of, of Nvidia server, right? Like, you know, $300,000 server. So when I do this with O1, right? Because it's doing that thinking phase of 10,000, right? This is, this is basically the whole context length thing. Context length is not free, right? Context length or sequence length means that it has to calculate attention, the attention mechanism, i.e. it spends a lot of memory on generating this KV cache and reading this KV cache constantly. Now, the maximum batch size, i.e. concurrent users I can have, is a fraction of that, 1/4 to 1/5 the number of users concurrently use this server. So not only do I need to generate 10x as many tokens,

• 49:56Speaker 00

  each, each token that's generated is 4 to 5x less users. So the cost increase is, is stupendous when you think about a, a single user. Cost increase for a single token to be generated is 4 to 5x. But then I'm generating 10x as many tokens. So you could argue the cost increase is 50x for an O1 style model on input to output.

• 50:17Bill Gurley?

  I knew the 10x because it was on the original O1 release, but with the log scale, but I didn't know. Well, it's 10x.

• 50:22Brad Gerstner?

  And it just requires you to have, you know, again, to service the same number of customers, you have to have multiples more compute.

• 50:30Bill Gurley?

  Well, this, there's good news and bad news here, Brad, which I think is what Dylan's telling us. If you're, if you're just selling Nvidia hardware and they remain the architecture and this is our scaling path, you're going to consume way more of it.

• 50:42Brad Gerstner?

  Correct. But the margins for the people who are generating on the other end, unless they can pass it on to the end consumer are gonna, you know, are gonna compress.

• 50:51Speaker 00

  And the thing is, you can pass it on to the end consumer because, hey, it's not really like, oh, it's X percent better on this benchmark. It's, it literally could not do this before and now it can, right? And so.

• 51:03Brad Gerstner?

  And they're running a test right now where they're 10x-ing what they're charging, the end consumer, you know.

• 51:08Speaker 00

  And it's 10x per token, right? Remember, they're also paying for 10x as many tokens, right? So it's actually, you know, the, the consumer is paying 50x more per query.

• 51:16Brad Gerstner?

  Right.

• 51:17Speaker 00

  But they're getting value out of it because now all of a sudden, it can pass certain benchmarks like Sweetbench, right? Software engineering benchmark, right? Which is just a benchmark for generating like, you know, decent code, right? There's front-end web development, right? What do you pay front-end web developers? What do you pay backend developers? Versus, hey, what if they use O1? How much more code can they output? How much more can they output? Yes, the queries are expensive, but they're nothing close to the human. Right? And so each level of productivity gain I get, each level of capabilities jump is a whole new class of tasks that it can do, right? And therefore I can charge for that, right? So this is the whole like axes of yes, I spend a lot more to get the same output, but you're not, you're not getting the same output with this model.

• 51:59Brad Gerstner?

  Are we overestimating or underestimating end demand, enterprise level demand for the O1 model? What are you hearing?

• 52:07Speaker 00

  So I would say the O1 style model is so early days, people don't even like get it, right? O1 is like they just cracked the code and they're doing it. But guess what? Right now, on, you know, some of the anonymous benchmarks, there are, you know, it's called LLM CISS, which is like an arena where different LLMs get to like compete sort of, and people vote on them. There's a Google model that is doing reasoning right now. And it's not released yet, but it's going to be released soon enough, right? Anthropic is going to release a reasoning model. These people are going to one up each other. And also, they've spent so little compute on reasoning right now in terms of training time. And they see a very clear path to spending a lot more, i.e. jumping up the scaling laws. Oh, I only spent $10 million. Well, wait, that means I can jump up two to three logarithms in

• 52:52Speaker 00

  scaling like that because I've already got the compute. You know, I can go from 10 million to 100 million to billion to $10 billion on reasoning in such a quick succession. And so the performance improvements we'll get out of these models is, is humongous, right? In the coming, you know, six months to a year in certain benchmarks where you have functional verifiers.

• 53:12Bill Gurley?

  Quick question, and we promised we'd go to these alternatives, so we'll have to get there eventually. But if you go back, we've used this internet wave comparison multiple times. When all of the venture backed companies got started on the internet, they were all on Oracle and Sun. And five years later, they weren't on Oracle or Sun. And some have argued it went from a development sandbox world to a optimization world. Is that likely to happen? Is there an equivalency here or not? And if you could touch on why the, the backend is so steep and cheap, like, you know, just you go a model, you know, behind or you like

• 53:57Bill Gurley?

  the, the price you can save by just backing up a little bit is nutty?

• 54:03Speaker 00

  Yeah, yeah. So today, right, like O1 is stupendously expensive. You drop down to 4.0, it's a lot cheaper. You jump down to 4.0 mini, it's so cheap. Why? Because now I'm with 4.0 mini, I'm competing against Llama and I'm competing against DeepSeek. I'm competing against Mistral, I'm competing against Alibaba and I'm competing against tons of companies.

• 54:21Brad Gerstner?

  So you think those are market clearing prices?

• 54:23Speaker 00

  I think, and in addition, right, there is also the problem of inferencing a small model is quite easy, right? I can run Llama 70B on one AMD GPU. I can run Llama 70B on one Nvidia GPU and soon enough there will be on one set of Amazon's new trainium, right? I can sort of run this model on a single chip. This is a very easy, I won't say very easy problem, still hard. It's quite an easier problem than running this complex reasoning or this very large model, right? And so there is, there is that difference, right? There's also the fact that, hey, there's literally 15 different companies out there offering API inferences, inference APIs on Llama and Alibaba and DeepSeek and Mistral, like these different models, right?

• 55:06Brad Gerstner?

  You're talking about Cerebras and Grok and, you know, fireworks and all these others.

• 55:10Speaker 00

  Yeah, fireworks together, you know, all the companies that aren't using their own hardware. Now, of course, Grok and Cerebras are doing their own hardware and doing this as well. But the, the market, the margins here are bad, right? Um, you know, sort of, we had this whole thing about the inference race to the bottom. When Mistral released their Mistral model, which was like very revolutionary sort of late last year, because it was such a level of performance that didn't exist in the open source, that it drove pricing down so fast, right? Because everyone's competing for API. What am I as an API provider providing you that, like, why don't you switch from mine to his? Why? Because, well, there's no, it's pretty fungible, right? I'm still getting the same tokens on the same model. And so the margins for these guys is much lower.

• 55:52Speaker 00

  So Microsoft's earning 50 to 70% gross margins on OpenAI models. And that's with the profit share they get to get, or the share that they give OpenAI, right? Or, you know, Anthropic similarly in their most recent round, they were showing like 70% gross margins. But that's because they have this model. You step down to here, no one uses this model from, you know, a lot less people use it from OpenAI or Anthropic because they can just like. Take the weights from Llama, put it on their own server or vice versa. Go to one of the many competitive API providers. Some of them being venture funded, some of them, you know, and losing money. Right. So there's all this competition here. So not only are you saying, I'm taking a step back and it's easier problem. I'm then, and so therefore, like if the model's 10x smaller, it's like 15x cheaper to run.

• 56:36Speaker 00

  And on a top, on top of that, I'm removing that gross margin. And so it's, it's not 15x cheaper to run, it's 30x cheaper to run. Um, and so this is, this is sort of the beauty of like, well, is everything commodity? No, but like, there is a huge chase to like, if you're deploying it in services, that's going to be, this is great for you. A, uh, B, you have to have the best model or you're no one. If you're one of the labs, right? And so you see a lot of struggles for the companies that were trying to build the best models, but failing.

• 57:02Brad Gerstner?

  And arguably, not only do you have to have the best model, you actually have to have an enterprise or a consumer willing to pay for the best model, right? Because at the end of the day, you know, the, the best model implies that somebody's willing to pay you these high margins. Right. And that's either an enterprise or a consumer. So I think, you know, you're, you're quickly narrowing down to just a handful of folks who will be able to compete, you know, in that market.

• 57:25Speaker 00

  I think on the model side, yes, I think on the, who's willing to pay for these models is, I, I think of, I think a lot more people will pay for the best model, right? When we use models internally, right? We have, we have, we have language models go through every regulatory filing and permit to look at data center stuff and pull that out and tell us where to look and where to not to. Um, and, and we just use the best model because it's so cheap, right? Like the data that I'm getting out of it, the value I'm getting out of it is so much higher.

• 57:50Brad Gerstner?

  What model are you using?

• 57:51Speaker 00

  Um, we're using Anthropic actually right now, Claude 3.5, Sonnet new, um, Sonnet. Um, and so, uh, just because O1 is a lot better on certain tasks, but not necessarily regulatory filings and permitting and like things like that, because the cost of errors is so much higher. Right. Same with a, a developer, right? If I can increase a developer who makes $300,000 a year here in the Bay by 20%, that's a lot. If I can, uh, if I can take a team of a hundred developers and use 75 or 50 to do the same job, or I can ship twice as much code. This is so worth using the most expensive model because O1, as expensive it is relative to 40. It's still super cheap, right? The cost for intelligence is so high in society, right? That's why intelligent jobs are the most, you know, high paying jobs,

• 58:36Speaker 00

  white collar jobs, right? Are the most high paying jobs. If you can bring down the cost of intelligence or augment intelligence, then there's a high market clearing price for that, which is why I think that like sort of the, oh yes, oh, one is expensive. And people will always gravitate to what's the cheapest thing at a certain level of intelligence. But each time we break a new level of intelligence, it's not just, oh, you know, we've, we've got a few more tasks we can do. I think it grows the mode of tasks that can be done dramatically. Very few people could use G P two and three, right? A lot of people can use G P T four. When we get to that quality of jump that we see for the next generation, the amount of people that can use it, the tasks that it can do balloons out. And therefore the amount of sort of white collar jobs that it can augment, increase productivity on will grow. And therefore the market clearing price for that token will be very high.

• 59:20Bill Gurley?

  Super interesting. I could make the other argument that someone that's in a high volume, you know, this replacing tons of customer service calls or whatever, might, might be tempted to minimize the spend.

• 59:35Speaker 00

  Absolutely.

• 59:35Bill Gurley?

  And, and maximize the amount of value add. They build around this thing, database writes and reads.

• 59:41Speaker 00

  Absolutely. So, so one of the funny things I like to, the calculations we did is if you take one quarter of Nvidia shipments and you said, all of them are going to inference Llama seven B, you can give every single person on earth 100 tokens per minute, right? Um, or sorry, 100 tokens per second. You give every single person on earth 100 tokens per second, which is like absurd.

• 1:00:02Bill Gurley?

  Yeah.

• 1:00:03Speaker 00

  You know, so like if we're just deploying Llama seven B quality models, we've so overbuilt, it's not even funny. Now, if we're deploying things that can like augment engineers and increase productivity and help us build robotics or A V or whatever else faster, um, then that's, that's a very different like calculation, right? And so that's sort of the, the whole thing. Like, yes, small models are there, but like they're so easy to run.

• 1:00:27Bill Gurley?

  And it may just, both these things may be true.

• 1:00:30Speaker 00

  Right, we're going to have tons of small models running everywhere, but the compute cost of them is so low.

• 1:00:33Bill Gurley?

  Yeah, fair enough.

• 1:00:34Brad Gerstner?

  Bill and I were talking about this earlier with respect to the hard drives, um, you know, that you used to cover. But if you look at the memory market, it's been one of these boomer bust markets. The idea was you would always, you know, sell these things when they're nearing peak. Um, you know, you always buy them at trough. You don't own them anywhere in, you know, in between. They trade at very low earnings multiples. I'm talking about Hynix and I'm talking about, you know, Micron. As you think about the shift toward inference time compute, it seems that the memory demanded of these chips, and Jensen has talked a lot about this, just is on a secular shift higher, right? Because if they're doing these passes, you know, and you're running, like you said, 10 or 100 or 1,000 passes for inference time reasoning,

• 1:01:19Brad Gerstner?

  you just have to have more and more memory as this context length expands. So, you know, talk to us a little bit about, you know, kind of how you think about the memory market.

• 1:01:27Speaker 00

  Yeah, so, you know, to sort of like, set the stage a little bit more is thinking reasoning models output thousands and thousands of tokens. And when you, when we're looking at transformers, attention, right, the like holy grail of transformers, i.e. how it like understands the entire context grows dramatically. And the KV cache, i.e. the memory that is keeping track of how, what this like context means is growing quadratically, right? And therefore, if I go from a context length of 10 to 100, it's not just a 10x, it's much more, right? And so you treat that, right? Like today's reasoning models, they'll think 10,000 tokens, 20,000 tokens. When we get to, hey, what is complex reasoning going to look like?

• 1:02:12Speaker 00

  Models are going to get to the point where they're thinking for hundreds of thousands of tokens. And then this is all one chain of thought, or it might be some search, but it's going to be thinking a lot, and this KV cache is going to balloon.

• 1:02:23Bill Gurley?

  So you're saying, you're saying memory could grow faster than GPU?

• 1:02:26Speaker 00

  And it objectively is when you look at the cost of goods sold of Nvidia. Their highest cost of goods sold is not TSMC, which is a thing that people don't realize. It's actually HBM memory, primarily.

• 1:02:38Bill Gurley?

  That may be for now also.

• 1:02:40Speaker 00

  Yeah, so there's, there's three memory companies out there, right? There's, there's Samsung, SK, Hynix, and Micron. Nvidia has majority used SK, Hynix. And this is like, a big shift in the memory market as a whole. Because historically, it has been a commodity, right? I.e. it's fungible. Whether I buy from Samsung, or SK, Hynix, or Micron.

• 1:02:58Bill Gurley?

  Is the socket replaceable?

• 1:03:00Speaker 00

  Yeah. And even now, Samsung is getting really, really hit hard because there's a Chinese memory maker, CXMT, and their memory's not as good as the last, but it's fine. And low end memory, it's, it's fungible. And therefore, the price of low end memory has fallen a lot. In HBM, Samsung has. Almost no share, right? Especially at Nvidia. And so this is like, this is hitting Samsung really hard, right? Despite them being the largest memory maker in the world, has, oh, everyone's always like, if you said memory, it's like, yeah, Samsung's a little bit ahead in tech and their margins are a little bit better and, and they're killing it, right? But now it's not quite not the case because on the low end, they're getting a little bit hit. And on the high end, they can't break in or they keep trying, but they keep failing. On the flip side, you have, you have companies like SK, Heineken, Micron,

• 1:03:44Speaker 00

  who are converting significance amounts, significant amounts of their capacity of, you know, sort of commodity DRAM to HBM. Now, HBM is still fungible, right? In that if someone hits a certain level of technology, they can swap out Micron to Heineken, right? So it's fungible in that sense, right? It's a commodity in that sense, but because reasoning requires so much more memory and the cost of goods sold of an H100 to Blackwell, the percentage of cost to HBM has grown faster than the percentage of cost to leading edge silicon. You've got this big shift or dynamic going on. And this applies not just to Nvidia's GPUs, but it applies to the hyperscalers GPUs as well, right? Or accelerators like the TPU, Amazon Tranium, et cetera.

• 1:04:26Bill Gurley?

  And SK has higher gross margins than memory companies have.

• 1:04:30Brad Gerstner?

  Correct, correct. If you listen to Jensen at least describe it, you know, it's not all memory is created equal, right? And so it's not only that the product is more differentiated today. There's more software associated with the product today, but it's also how it's integrated into the overall system, right? And going back to the supply chain question, it sounds like it's all commodity. It just seems to me that at least there's a question out there. Is it structurally changing? We know the secular curve is up and to the right.

• 1:04:59Bill Gurley?

  I'm hearing you say maybe it may be differentiated enough to not be a commodity.

• 1:05:03Speaker 00

  It may be, and I think another thing to point at is, funnily enough, the gross margins on HBM have not been fantastic. They've been, they've been good, but they haven't been fantastic. Actually, regular memory, high end, like server memory that is not HBM is actually higher gross margin than HBM. And the reason for this is because Nvidia is pushing the memory maker so hard, right? They want the faster, newer generation of memory, faster and faster and faster for H, for HBM. But not necessarily like everyone else for servers. Now, what is this like, what is this like meant? Is that, hey, even though Samsung may achieve level four, right? Or level three or whatever that they had previously, they can't reach what Hynix is at now. What are the competitors doing, right? What is AMD and Amazon saying? AMD explicitly has a better inference GPU because they give you more memory, right?

• 1:05:48Speaker 00

  They give you more memory and more memory bandwidth. That's literally the only reason AMD's GPU is even considered better.

• 1:05:53Bill Gurley?

  On chip?

• 1:05:54Speaker 00

  HBM memory.

• 1:05:55Bill Gurley?

  Okay.

• 1:05:56Speaker 00

  Which is on package, right?

• 1:05:59Speaker 00

  Specifically, yeah. And then when we look at Amazon, their whole thing at Reinvent, if you really talk to them, when they announced Trinium 2, and our whole post about it, our analysis of it is like, supply chain wise, this is, looks, you squint your eyes, this looks like an Amazon Basics TPU, right? It's decent, right? But it's really cheap, A, and B, it gives you the most HBM capacity per dollar and most HBM memory bandwidth per dollar of any chip on the market. And therefore, it actually makes sense for certain applications to use. And so this is like a real, real shift, like, hey, we maybe can't design as well as Nvidia, but we can put more memory on the package, right? Now, this is just only one vector of like, you know, there's a multi-vector problem here. They don't have the networking nearly as good. They don't have the software nearly as good. Their compute elements are not nearly as good.

• 1:06:43Speaker 00

  They've, by golly, they've got more memory bandwidth per dollar.

• 1:06:46Bill Gurley?

  Well, this is where we wanted to go before we run out of time is just to talk about these alternatives, which you just started doing. So despite all the amazing reasons why no one would seemingly want to pick a fight with Nvidia, many are trying, right? And I even hear people talk about trying that haven't tried yet, like OpenAI is constantly talking about their own chip. What, how are these other players doing? Like, how would you handicap? Let's start with AMD, just because they're a standalone company, and then we'll go to some of the internal program.

• 1:07:18Speaker 00

  Yeah, so AMD is competing well because silicon engineering wise, they're amazing, right? They're competitive.

• 1:07:26Speaker 00

  But, yeah, they kicked InTEL's ass, but that's like, you know.

• 1:07:30Bill Gurley?

  They started way down here. Over a 20-year period, it was pretty amazing.

• 1:07:35Speaker 00

  AMD is really good, but they're missing software. AMD has no clue how to do software, I think. They've got very few developers on it. They won't, they won't spend the money to build a GPU cluster for themselves so that they can develop software, right? Which is like insane, right? Like Nvidia, you know, the top 500 supercomputer list is not relevant because most of the super biggest supercomputers like Elons and Microsofts and so on and so forth, they're not on there. But Nvidia has multiple supercomputers on the top 500 supercomputer list, and they use them fully internally to develop software, network software, whether it be network software, compute software, inference software, all these things, you know, test all these changes they make, and then roll out pushes, you know, where if XAI is mad because

• 1:08:18Speaker 00

  of, you know, software's not working, Nvidia will push it the next day or two days later, like clockwork, right? Because there's tons of things that break constantly when you're training models. AMD doesn't do that, right? And I don't know why they won't spend the money on a big cluster. The other thing is, they have no idea how to do system level design. They've always lived in the world of, I'm competing with Intel, so if I make a better chip than Intel, then I'm great. Because software, x86, it's x86.

• 1:08:44Bill Gurley?

  I mean Nvidia doesn't keep it a secret that they're a systems company, so presumably they've read that and they're

• 1:08:50Speaker 00

  Yeah, and so they bought this systems company called ZT Systems. But they're, you know, the whole rack scale architecture, which Google deployed in 2018 with the TPU V3.

• 1:09:00Bill Gurley?

  Are there any hyperscalers that are so interested in AMD being successful that they're co-developing with them?

• 1:09:08Speaker 00

  So, the hyperscalers all have their own custom silicon efforts, but they also are helping AMD a lot in different ways, right? So Meta and Microsoft are helping them with software, right? Not enough that like AMD is like caught up or anything close to it. They're helping AMD a lot with what they should even do, right? So the other thing that people recognize is, if I have the best engineering team in the world, that doesn't tell me what the problem is, right? The problem has this, this, this, this, it's got these trade-offs. AMD doesn't know software development, it doesn't know model development, it doesn't know inference, what inference economics look like. And so, how do they know what trade-offs to make? Do I push this lever on the chip a bit harder, which then makes me have to back off on this? Or what exactly do I do? Right? The hyperscalers are helping, but not enough that AMD is on the same timelines as Nvidia.

• 1:09:52Bill Gurley?

  How successful will AMD be in the next year on AI revenue? And what kind of sockets might they succeed?

• 1:10:00Speaker 00

  Yes, I think they'll have they'll have a lot less success with Microsoft than they did this year. And they'll have less success than they did with Meta than they did this year. This is because like the regulations make it so actually AMD's GPU is like quite good for China, because the way they shaped it. But generally, I think AMD will will do okay. They'll they'll profit from the market. They just won't like go gangbusters like people are hoping. And they won't be a their their share of total revenue will fall next year.

• 1:10:30Bill Gurley?

  Okay.

• 1:10:31Speaker 00

  But they will still do really well, right? Billions of dollars of revenue is not nothing to stop.

• 1:10:35Bill Gurley?

  Let's go with the Google TPU. You you earlier stated that it's got the second most workload. It seems like by a lot like it's it's firmly in second place.

• 1:10:45Speaker 00

  Yeah, so so this is this is where the whole systems and infrastructure thing matters a lot more. Each individual TPU is not that impressive. It's impressive, right? It's got good networking. It's got good, you know, architecture, etc. It's got okay memory, right? Like, it's it's not that impressive on its own. But when you when you say, hey, if I'm spending X amount of money, and then what's my system? Google's TPU looks amazing, right? So Google's engineered it for things that Nvidia maybe has not focused on as much, right? So actually, their interconnects between chips is arguably competitive, if not better in certain aspects, worse in other aspects than Nvidia's, because they've been doing this with Broadcom, you know, the leader, world leader in networking, you know, building a chip with them. And since 2018, they've had this scale up, right? Nvidia is talking

• 1:11:30Speaker 00

  about GB 200, NVL 72. TPUs go to 8,000 today, right? And and while it's not a switch, it's a point to point, you know, it's a little bit. There's some technical nuances there. So it's not just like like that. Those numbers are not all you should look at. But this is this is important. The other aspect is Google's brought in water cooling for years, right? Nvidia only just realized they needed water cooling on this generation. And Google's brought in a level of reliability that Nvidia GPUs don't have. You know, the dirty secret is to go ask people what the reliability rate of GPUs is in the in the cloud or in a deployment. It's like, oh, God, is not they're reliable ish, like, but like, especially initially, you have to pull out like 5% of them.

• 1:12:12Brad Gerstner?

  Why is TPU not been more commercially successful outside of Google?

• 1:12:17Speaker 00

  I think Google keeps a lot of their software internal when they should just have it be open because like who cares? You know, like that's one aspect of it. You know, there's a lot of software that DeepMind uses that just is not available to Google Cloud. Two,

• 1:12:32Bill Gurley?

  Even their Google Cloud offering relative to AWS had that bias.

• 1:12:37Speaker 00

  Yeah, yeah. Number two, the pricing of it is sort of, it's not that it's egregious on list price, like list price of a GPU at Google Cloud is also egregious. But you as a person know, when I go rent a GPU, you know, I tell Google like, hey, like, you know, blah, blah, blah. You're like, okay, you can get around the first round of negotiations, get both down. But then you're like, well, look at this offer from Oracle or from Microsoft or from Amazon or from CoreWeave or one of the 80 Neo clouds that exist. And Google might not match like many of these companies, but like they'll go down because they, you know, and then you're like, oh, well, like, what's the market clearing price for a, if I wanted an H100 for two years or a year? Oh yeah, I could get it for like two bucks.

• 1:13:19Bill Gurley?

  Right.

• 1:13:19Speaker 00

  A little bit over versus like the $4 quoted, right? Whereas a TPU that's here, you don't know that you can get here. And so people see the list price and they're like, eh.

• 1:13:28Bill Gurley?

  Do you think that'll change?

• 1:13:30Speaker 00

  I don't see any reason why it would. And so number three is sort of, Google is better off using all of their TPUs internally. Microsoft rents very few GPUs, by the way, right? They actually get far more profit from using their GPUs for internal workloads or using them for inference, because the gross margin on selling tokens is 50 to 70%, right? The gross margin on selling a GPU server is lower than that, right? So while it is like good gross margin, it's like, you know, it's.

• 1:13:56Brad Gerstner?

  And they've said out of the 10 billion that they've quoted, none of that's coming from external renting of GPUs.

• 1:14:02Bill Gurley?

  If Gemini becomes hyper competitive as an API, then you indirectly will have third parties using the Google TPU. Is that accurate?

• 1:14:13Speaker 00

  Yeah, absolutely. Ads, search, Gemini applications. All of these things use TPUs. So it's not that like, you know, that you're not using. Every YouTube video you upload is going through a TPU, right? Like, you know, it goes through other chips as well that they've made themselves custom chips for YouTube. But like, there's so much that touches a TPU, but. You indirectly, you directly would never rent it, right? And that's therefore like, when you look at the market of renters, there's only one company that counts for over 70% of Google's revenue from TPUs, as far as I understand, and that's Apple, right? And I think there's, there's a whole long story around why Apple hates Nvidia. But you know, that, that may be a story for another time, but.

• 1:14:51Bill Gurley?

  You just did a super deep piece on Tranium. Why don't you do the Amazon version of what you just did with Google?

• 1:14:59Speaker 00

  Yeah, so, so funnily enough, Amazon's chip is the Amazon, I, I call it the Amazon's basics TPU, right? And the reason I call it that is because, yes, it uses more silicon. Yes, it uses more memory. Yes, the network is like somewhat comparable to TPUs, right? It's a six, it's a four by four by four Taurus. They just do it in a less efficient way in terms of, you know, hey, they're spending a lot more on active cables, right? Because they're working with Marvell and Alchip on their own chips versus working with Broadcom, the leader in networking, who then can use passive cables, right? For, for, because their certies are so strong. Like, there's other, there's other things here. Their certi speed is lower. They spend more silicon area.

• 1:15:41Speaker 00

  Like, there's all these things about the, the Tranium that are, you know, you could look at it and be like, wow, this would suck if it was a merchant silicon thing. But it doesn't because it's, it's, it's, Amazon's not paying Broadcom margins, right? They're paying lower margins. They're, they're not paying the margins on the HPM. They're paying, you know, they're paying lower margins in general, right? Paying the margins to Marvell on HPM. You know, there's all these different things they do to crush the price down to where their, their Amazon basics TPU, the Tranium 2, right? Is very, very cost effective to the end customer and themselves in terms of HPM per dollar, memory bandwidth per dollar, and it has this world size of 64. Now, Amazon can't do it in one rack. It actually requires them two racks to do 64.

• 1:16:23Speaker 00

  And the bandwidth between each chip is much slower than Nvidia's rack. And their memory per chip is lower than Nvidia's. And their memory bandwidth per chip is lower than Nvidia. But you're not paying, you know, north of 30,000, you know, $40,000 for this, per chip for the server. You're paying, you know, significantly less, right? $5,000 per chip, right? Like, you know, it's like. Such a gulf, right, for Amazon. And then they pass that on to the customer, right? Because when you buy an Nvidia GPU. So there is legitimate use cases. And because of this, right, Amazon and Anthropic have decided to, you know, make a 400,000 Tranium server, uh, supercomputer, right? 400,000 chips, right? Going back to the whole scaling law is dead? No, they're making a 400,000 chip system because they truly believe in this, right?

• 1:17:06Speaker 00

  And 400,000 chips in one location is not useful for serving inference, right? It's useful for making better models, right? You want your inference to be more distributed than that. So this is, this is a huge, huge investment for them. And while technically it's not that impressive, there are some impressive aspects that I kind of glossed over. It is so cheap and so cost effective that I think it's a decent play for Amazon.

• 1:17:33Brad Gerstner?

  Maybe just wrapping this up, I want to, I want to shift a little bit to kind of what you see happening in 25 and 26, right? For example, over the last 30 days, right, we've seen Broadcom, you know, explode higher. Nvidia trade off a lot. I think there's about a 40% separation over the last 30 days. You know, with Broadcom being this play on custom ASICs, you know, people questioning whether or not Nvidia has got a lot of new competition, pre-training, you know, not, not improving at the rate that it was before. Look into your crystal ball for 25, 26. What are you talking to clients about?

• 1:18:11Brad Gerstner?

  You know, in terms of what you think are kind of the things that are most misunderstood, best ideas, you know, in the spaces that you cover?

• 1:18:20Speaker 00

  So I think a couple of the things are, you know, hey, Broadcom does have multiple custom ASIC wins, right? It's not just Google here. Meta's, Meta's ramping up mostly still for recommendation systems, but their custom chips are going to get better. You know, there's, there's other players like OpenAI who are making a chip, right? You know, there's, there's Apple who are not quite making the whole chip with Broadcom, but a small portion of it will be made with Broadcom, right? You know, there's, there's a lot of wins they have, right? Now, these all won't hit in 25. Some of them will hit in 26. And it's You know, it's a custom ASIC, so like it could, it could be a failure and not be good like, like Microsoft's and therefore never ramp. Or it could be really good and like, like, or at least, you know, good price to performance like Amazon's and it could ramp a lot, right?

• 1:19:04Speaker 00

  So there are risks here, but Broadcom has that custom ASIC business, one. And two, really importantly, the networking side is so, so important, right? Yes, Nvidia is selling a lot of networking equipment.

• 1:19:18Speaker 00

  But when people make their own ASIC, what are they going to do, right? Yes, they could go to Amazon or not, but they could also, they also need to network many of these chips together. Or sorry, to Broadcom or not, they could go to Marvell or many other competitors out there like Alchip or, and GUC. Like you could, you can, you, Broadcom is really well positioned to make the competitor to NVSwitch. Which many would argue is one of Nvidia's biggest competitive advantages on a hardware basis versus everyone else.

• 1:19:46Brad Gerstner?

  Right.

• 1:19:46Speaker 00

  And Broadcom is making a competitor to that, that they will cede to the market, right? Multiple companies will be using that, not just, you know, AMD will be using that competitor to NVSwitch, but they're not making it themselves because they don't have the skills, right? They're going to Broadcom to get it made, right?

• 1:20:01Brad Gerstner?

  So make a, make, make, make a call for us as you think about the semis market today. You know, you've got ARM, Broadcom, you've got Nvidia, you've got AMD, et cetera. Does the whole market continue to elevate as we head into 25 and 26? Who's best positioned from current levels to do well? Who's most, you know, overestimated, who's least, who's most underestimated?

• 1:20:24Speaker 00

  I think, I, I, I, Broadcom long-term, but like in the next six months, there is a bit of a slowdown in Google TPU purchases because they have no data center space. They want more, they just literally have no data center space to put them. So we actually like, you know, can, can see how they're like, there's a bit of a pause, but people may look past that. Beyond that, right, it's, the question is like, who wins what custom ASIC deals, right? Is Marvell going to win future generations? Is Broadcom going to win future generations? How big are these generations going to be? Are the hyperscalers going to be able to internalize more and more of this or no, right? Like, it's no secret, Google's trying to leave Broadcom. They could succeed or they could fail, right? It's not just like-

• 1:21:03Brad Gerstner?

  Broaden out beyond Broadcom. I'm talking Nvidia and everybody else. Like, you know, we've had these two massive years. Right of tailwinds behind this sector. Is 2025 a year of consolidation? Do you think it's another year that the sector does well? Just kind of.

• 1:21:19Speaker 00

  Yeah, I think I think the plans for hyperscalers are pretty firm on they're they're going to spend a crap load more next year. Right. And therefore, the ecosystem of networking players of ASIC vendors of systems vendors is going to do well, whether it be Nvidia or Marvell or Broadcom or AMD or, you know, generally, you know, some some better than others. The real question that people should be looking out to is 2026. Do does the spend continue right? We are not good. The growth rate for Nvidia is going to be stupendous next year. Right. And that's going to drag the entire component supply chain up. It's going to bring so many people with them. But 2026 is like where the reckoning comes, right? But, you know, will will people keep spending like this? And it's it's all points to where will the models continue to get better?

• 1:22:03Speaker 00

  Because if they don't continue to get better, in my opinion, will get better faster, in fact, next year, then there will be a big, you know, sort of clearing event, right? But that's not next year, right? You know, the other aspect I would say is there is consolidation in the Neo cloud market. Right. There are 80 Neo clouds that we're tracking that we talked to, that we see how many GPUs they have. Right. The problem is nowadays, if you look at rental prices for H100s, they're tanking. Right. Not just not just at these Neo clouds, right? Where you can you used to have to pay, you know, do four year deals and prepay 25%. You'd you'd sign a venture round and you'd buy a cluster and that's about it, right? You'd rent one cluster, right? Nowadays, you can get three month, six month deals at way better pricing than even the four month or the four year, three year deals that you used to have for Hopper.

• 1:22:48Speaker 00

  Right. And on top of that, it's not just through the Neo clouds. Amazon's pricing for, you know, on demand GPUs is falling now. It's still over. It's like still really expensive relatively, but like pricing is falling really fast. 80 Neo clouds are not going to survive. Maybe five to 10 will.

• 1:23:03Brad Gerstner?

  Right.

• 1:23:04Speaker 00

  And and that's because five of those are sovereign, right? And then the other five are like actually like market competitive.

• 1:23:09Bill Gurley?

  What percentage of the industry AI revenues have come from those Neo clouds that may not survive?

• 1:23:16Speaker 00

  Yeah, so roughly you can say hyperscalers are 50 ish percent of revenue, 50 to 60%. And the rest of it is NeoCloud slash Sovereign AI. Because enterprises purchases of GPU clusters is still quite low. And it ends up being better for them to just like outsource it to NeoClouds. When when they can like get through the security, which they can for certain companies like like Corrieve.

• 1:23:38Brad Gerstner?

  Is there a scenario where in 2026, where you see industry volumes actually down versus 2025? Or Nvidia volumes actually down meaningfully from 2025?

• 1:23:56Speaker 00

  So when you look at custom ASIC designs that are coming, as well as Nvidia's chips that are coming. The revenue, the content in each chip is exploding. The cost to make Blackwell is north of 2X that of the cost to make Hopper, right? So Nvidia can make the same, you know, and obviously they're cutting margins a little bit. But Nvidia can ship the same volumes and still grow a ton, right?

• 1:24:18Brad Gerstner?

  So rather than unit volumes, is there a scenario where industry revenues are down in 26? Or Nvidia revenues are down in 26?

• 1:24:28Speaker 00

  The reckoning is, the reckoning is, do models continue to get much faster, better? And will hyperscalers, are they okay with taking their free cash flow to zero? I think they are, by the way. I think I think Meta and Microsoft may even take their free cash flows close to zero and just spend. But then that's only if models continue to get better. That's A. And then B, are we going to have this huge influx of capital from people we haven't had it yet from? The Middle East, the sovereign wealth funds in Singapore and Nordics and, you know, Canadian Pension Fund and all these folks. They can throw, they can write really big checks. They haven't, but they could. And if if things continue to get better, I truly do believe that OpenAI and XAI and Anthropic

• 1:25:13Speaker 00

  will continue to raise more and more money and keep this game going of not just, hey, where's the revenue for OpenAI? Well, it's 8 billion and it might, you know, double or whatever, even more next year. And that's their spend. No, no, no, like they have to raise more. Money to spend significantly more and that that keeps the engine rolling because once one of them spends Elon is forcing everyone to spend more actually right with his cluster because and his plans because everybody's like well we can't get out scaled by Elon we have to spend more right and so there's sort of a game of chicken there too like oh they're buying this we have to we have to match them or go bigger because it is a game of scale so so you know and sort of Pascal's wager sense right if I underspend that's just the worst scenario ever and I'm like the worst CEO ever of the most profitable business ever but

• 1:25:56Speaker 00

  if I overspend yeah shareholders will be mad but it's fine right it's you know $20 billion $50 billion

• 1:26:02Bill Gurley?

  You can paint that either way though because if that becomes the reasoning for doing it you you're you're more the probability of overshooting goes up

• 1:26:10Speaker 00

  And every bubble ever we overshoot

• 1:26:12Brad Gerstner?

  And you know to me it you know you said it all hangs on models improving I would take it a step further you know and go back to what Satya said to us last week it all comes down ultimately to the revenues that are generated by the people who are making the purchases of the GPUs right like he said last week I'm gonna buy a certain amount every single year and it's going to be related to the revenues that I'm able to generate in that year or the next couple of years so like they're not gonna spend way ahead of where those revenues are so he's looking at what you know he had 10

• 1:26:53Brad Gerstner?

  billion in revenues this year he knows the growth rate associated with those inference revenues and they're making he and Amy are making some forecast as to what they can afford to spend I think Zuckerberg's doing the same thing I think Sundar's doing the same thing and so if you assume they're acting rationally it's not just the models improving it's also the rate of adoption of the underlying you know enterprises who are using their services it's the rate of adoption of consumers and what consumers are willing to pay to use chat GPT or to use Claude or to use these other services so you know if you think that infrastructure spent expenses are going to grow at 30% a year then I think you have to believe that the underlying inference

• 1:27:35Brad Gerstner?

  revenues right both on the consumer side and the enterprise side are gonna grow somewhere in that range as well

• 1:27:41Speaker 00

  There is definitely an element of spend ahead though, right? For sure. Point in time spend versus, you know, what do I think revenue will be for the next five years for the server, right? So I think there is an element of that for sure, but absolutely, right? Models, the whole point is models getting better is what generates more revenue, right? And it gets deployed. So I think, I think that's, I'm in agreement, but people are definitely spending ahead of what's, what's, what's charted.

• 1:28:05Brad Gerstner?

  Well, that's what makes it spicy. You know, it's fun to have you here. I mean, you know, a fellow analyst, you guys do a lot of digging. Congratulations on the success of your business. You know, I think you add a lot of important information to the entire ecosystem. You know, one of the things I think about the wall of worry, Bill, is the fact that we're all talking about and looking for, right? The bubble, sometimes that's what prevents the bubble from actually happening. But, you know, you know, as both an investor and an analyst, you know, I look at this and I say, there are definitely people out there who are spending, who don't have commensurate revenues, to your point, spending, spending way ahead.

• 1:28:47Brad Gerstner?

  On the other hand, and I, and frankly, you know, we heard that from Satya last week. He said, listen, I've got the revenues. I've said what my revenues are. I haven't heard that from everybody else, right? And so it'll be interesting to, to see in 2025, who shows up with the revenues? I think you already see some of these smaller second, third tier models, changing business model, falling aside, no longer engaged in the arms race, you know, of, of, of investment. You know, I think that's part of the creative destructive process, but it's been fun having you on.

• 1:29:20Bill Gurley?

  Yeah, thank you so much, Dylan. Really appreciate it.

• 1:29:22Brad Gerstner?

  Yeah, fun, fun having you here in person, Bill. And until next year.

• 1:29:27Speaker 00

  Awesome. Thank you.

• 1:29:37Brad Gerstner?

  As a reminder to everybody, just our opinions, not investment advice.