Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Dylan Patel, founder of SemiAnalysis, argues the biggest gains in AI don't come from faster chips, they come from software-hardware co-design. Optimizing the model, the kernels, and the silicon together turns a 2x here and a 2x there into 100x. He explains why DeepSeek's experts were shaped for Nvidia's Hopper (and why TPUs struggle to run it), why OpenAI's sparser models and Anthropic's denser ones pull them toward different hardware, and why the so-called CUDA moat was never really about CUDA. Dylan breaks down InferenceX, his living benchmark that runs the latest models on over $50M of donated hardware daily, tracking a roughly 60x annual drop in cost per unit of quality. He makes the case that inference will be a bigger market than oil, that the compute crunch persists because models expand the value of useful work faster than compute grows, and why Jensen Huang is bankrolling neoclouds to engineer a multipolar world. Hosted by Shaun Maguire and Sonya Huang, Sequoia Capital

• 0:00Speaker 00

  I think it's really fun inside of Semi Analysis because we have 90 people and like a big chunk of them are technologists, engineers across the whole supply chain. And then a big chunk is people who are formerly at hedge funds. And you see these arguments, like people are like, oh, well, that doesn't matter. And it's like, then someone's like, well, but cost. And then someone, the engineer's like, no, no, no, but this technology is the coolest. And you see this, you see this organically like fight it out. And we're pretty informal. And, you know, given the fact that I was a former moderator is you can imagine what the

• 0:28Speaker 01

  Well, you're enjoying it. You're enjoying it.

• 0:30Speaker 00

  You don't wrestle with a pig because a pig enjoys it.

• 0:32Speaker 01

  Exactly.

• 0:50Speaker 00

  We're here in the Semi Analysis office with Dylan Patel. You know, I'm Sean from Sequoia. I have my partner, Sonya Huang. It's pretty insane what you've done. Semis five years ago were not very sexy in the West. They were sexy in the East, but people here in the West had kind of forgotten about them. You did not forget about them, though. You went very long. You created probably the premier research company in the space. It's been educating the world and, you know, the state of the art from very technical details to supply chain, you know, to the bigger picture. There's rumors that Semi Analysis recently passed 100 million of revenue. I don't know how accurate those are. Whatever the numbers are, you guys are crushing.

• 1:30Speaker 01

  It's as accurate as the information is.

• 1:32Speaker 00

  Oh, you know, you never, you never know. There's also rumors that you might start a venture funds. You know, I hear all the time in the ecosystem, people wanting, you know, affiliation with Semi Analysis. You've built this trusted brand and so whatever you do, it's working. This is clearly like just the beginning of the journey for you. Congratulations, all of that. But how did this happen? Like, how did you, first question is like, what is the background? How did you kind of get to where you are now?

• 1:58Speaker 01

  Well, when I was a young boy and, you know, coming out of the womb. So, so, okay, so I grew up in like a small business. My parents had a motel. We lived in the motel. We laid our gas station. So, you know, I was selling, you know, I joke a lot of times, the first neural network I trained was racially and visually profiling people based on when they entered the gas station, which cigarette to pick. Basically, you know, the cigarettes were all extrude across the top. And I was too short to actually like, you know, reach them. And technically it wasn't legal to sell cigarettes at that age, but whatever. I had to move the stepstool over to the right area. I started working my first job before it was legal too. So it's a good experience.

• 2:37Speaker 00

  Well, I didn't get paid, right? It's a family business.

• 2:39Speaker 01

  Same, same. We had our motel and then across the street was our gas station. So, you know, sometimes, you know, someone would walk in. And so like if an old white lady with curly hair walked in, I'd move the ladder or the stepstool over to where the camels are. And if, you know, and, you know, different, different age, demographic, profession, you know, race, et cetera, I would move the stepstool over. And I joke, this is the first neural network I trained. Cause if I waited for them to tell me, I'd have to like move it over and then I'd step up versus like just being ready. So, you know, menthols versus, you know, a hundred slims and all these things. You know, I think I joke, that's the first neural network I trained. But I grew up in family businesses, lived in a motel. And it all really goes back to when I was like, you know, it was my eighth birthday.

• 3:20Speaker 01

  My birthday is in May and it was April when the Xbox three sixty was announced. For my birthday, I didn't ask for the Xbox or I didn't ask for a birthday gift. My parents asked what I wanted. I asked for it for Christmas. We celebrated Christmas, but there was no way, at least at the time, I thought there was no way they would ask, would give me the Xbox three sixty for Christmas. And so I got it for, I asked for my birthday for tab for Christmas. Anyways, Christmas comes around, I get it. You know, fast forward a couple of months, my cousin who lives in Alabama, they also lived in a motel, was going to come over for spring break, for his spring break. And we were going to hang out at my house and he's in between me and my older brother in age. Brother's a bit more jockey, so he didn't really care too much about the Xbox. He played sometimes, but didn't really care. But my cousin, you know, I wanted to think of him to think I was cool, right?

• 4:02Speaker 01

  You know, so I bragged many times on the phone. I was like, yeah, I got an Xbox. And then the Xbox broke. There was something, there's a hardware defect called the red ring of death. But long story short, I had to open it up and, you know, short the temperature sensor and it fixed it. But there was many other tricks I tried first and none of them worked. And so that's sort of how I like got into hardware is like open Pandora's box. By the time I was 12, I was like on these forums a lot, reading, posting a lot. And this is around the time when Reddit ate all other forums. And so I became a moderator of, you know, Android and Apple and Google as well as like hardware. And was watching, you know, looking at Intel, Nvidia and AMD and all these other forums, right? So I was build a PC, all these forums I was watching, reading, posting a lot. But some of them I was moderating a lot.

• 4:45Speaker 01

  And so, you know, smartphones, watching smartphones develop from like very simple to speed racing to being technologically more advanced than PCs. In many ways, architecturally and same with like, you know, all the GPUs, like just tracking and watching at that, reading every comment, always having the economic tinge because I grew up in a small business. So I was always looking at the economics, right? There was a time where all the like, I'd say neckbeards on the internet loved AMD GPUs. And like, I personally had bought an AMD GPU too because price performance. But then when it came down to like, what's technically better? I'd always be like, no, no, no, Nvidia is better because they use a smaller chip to get, you know, better performance and better power efficiencies and their margins better.

• 5:25Speaker 01

  And, and so like, I would always like talk about how Nvidia's margins were better than AMD's in the GPU landscape. And so it's like very fun. And you were 12 at the time? I started moderating when I was 12, but this is all through my teenage, tween age and high school years, right? Do you have any other weird hobbies or was it just semis? I played a ton of StarCraft at one point. I was grand master on the North American ladder. Oh, wow. Okay. So you've gotten just obsessively good at multiple things? Yeah, I mean, it's, it's, it's obsession is good. How were your grades? They were decent. I would say like,

• 5:59Speaker 01

  I had mostly As, but there are classes that I like were thought were really boring, or you know, just didn't enjoy. Um, like Spanish, I got like not the greatest grades. Um, you know, but, but it was like, I speak fluent Spanish, by the way, so it's really dumb. Um, but like, it's just sort of.

• 6:16Speaker 00

  But maybe that's why you didn't get a good grade.

• 6:18Speaker 01

  No, I didn't learn Spanish till later, to be fair, but yeah. So it's sort of my grades are fine, right? Like, I mean, I like, they were fine enough for Asian parents. I was better than most of the school, but you know, it wasn't like, you know, try hard maxing for like, you know, all As.

• 6:33Speaker 02

  Okay. So you're very much a student of the internet then. This is how you, how you develop this expertise. Uh, at what point did you decide to start Semi Analysis? And what's been the biggest surprise since starting the company?

• 6:42Speaker 01

  Yeah. So I went to school. I got a few degrees in stuff that wasn't related to semiconductors. Um, was a quant for two years at a small quant risk firm. Um, and then basically, you know, there was a culmination of events that happened, right? One was that my, um, you know, sort of like I got screwed out of a bonus. I'd, I'd made my company many millions of revenue, of risk-free revenue. Cause I exploited like a risk, you know, thing in the market. Um, you know, I think well over 10 million and then someone else took credit for my work and all this sort of stuff. But eventually I did get right-sized, but you know, I lost the social contract with the company I was working with. Uh, I had some, you know, my grand, my grandparents grew up in my house with us, right? Or in the motel with us, uh, they lived with us. And so, you know, very close to them.

• 7:22Speaker 01

  And my grandmother got dementia and she forgot who I was and she, she fell down some stairs and had like a tragic accident and passed away. So all of that happened in early 2020. Um, additionally, there were some like, you know, girl things. And so, you know, there's a few things that happened that made me like kind of very sad. Um, and, and, and so all of those things sort of culminated, then COVID happened and my brother's like, dude, just, just come stay with me. He lived in Nashville. So I came and stayed with him in Nashville. We were like, oh, lockdowns will be a few weeks. You can stay with me while they happen and then you can go back home and, you know, whatever. Famous last words, lockdowns lasted much longer. But, you know, living with my brother for a few months, you know, it was like, sort of like, okay. Didn't know what I was doing. I was now at my brother's home.

• 8:02Speaker 01

  Um, everything was his rules, you know, sort of like, you know, him and, him and his fiance at the time, now wife, you know, were like there. And so like. I basically had to tiptoe around, but I didn't care about my job. And so I was like posting even more than normal. I'd always been posting a lot on the internet. I'd always been trading stocks a lot. But like I made a lot of money shorting COVID and long in COVID and like all this stuff. Semiconductor shortages happened around then too. And anyways, I was like very much obsessed with posting and things like that. And eventually, um, around that time, someone, I got into an argument with someone on the internet and they doxed me. Right? They publicly revealed my identity for my anonymous account. And at the time I was like, oh no, I was scared. I stopped posting for like three weeks. And I was like, what am I doing? Why do I care? So then I just started posting under, I had like, I had like blogs and stuff as well.

• 8:44Speaker 01

  I made a real blog, Semi Analysis, and on my 24th birthday, I posted, um, you know, two blogs. And, and then from there, it just like, it was not a newsletter, but I got so much traction. Cause now instead of posting on anonymous name, it was a real name. And I put a lot more effort into those two posts than I usually did. Instead of like shitposting on the internet, it was like real effort into the blog. You can actually go back and read those if you want. They're, they're not that great, but you know, they were, they were good for the time. They were the best stuff you could find on the internet about semis. And, and I just kept posting, posting, posting. I started getting a lot of consulting business. You know, 2020, I also sort of, I was again, crashing out, didn't know what I wanted to do. So I packed everything up. Or sort of, I, I took my truck, I bought a tent that fits on the back of the tent truck, bought an air mattress, whatever.

• 9:28Speaker 01

  And would like, and drove around all these national parks all around America. And so like two or three or four days of the week, I'd stay in a random motel where I negotiated the price to be like $30 a night for a room. And I was working on semi analysis stuff. And then the weekends I'd read books and oftentimes read textbooks. While in some random national park or hiking and listen to pod, audio books. About semiconductors, about AI, about all the things that I cared a lot about. And got way more educated over these six months where I'm just like going to every national park. And then the whole time I was posting, I was alone the whole time. I was posting blogs. Everyone was like, Dylan, what the are you doing?

• 10:02Speaker 00

  Pre-Star link or the very early days of Starlink?

• 10:05Speaker 01

  Pre-Star link, pre-Star link. Yeah. So it was like very much like, what are you doing? I'd travel around LatAm again, like for, for a year, initially with my friend and then with my ex, you know, for, you know, about a year. And then I go, then 22, 23, 24, end of 21, 22, 23 and 24. I'm completely, I'm still completely homeless since mid 2020, right?

• 10:26Speaker 01

  But I'm traveling around to every conference in the world. I go to 40 plus conferences a year. No matter where in the supply chain it is, I'm like, oh, that looks interesting. I guess I'll go to that. And I'm like, I went to one conference like, wow, this is amazing. You get to talk to the experts. And they just like, they, they're, they're gonna talk to you because and then you're so excited. And in the case of semiconductors, everyone's a boomer. So it's like, it's great to like, you know, they're like, they don't see young people who are like excited about it. So they're really happy to tell stuff. And so you just kept doing that.

• 10:52Speaker 00

  I have to ask on this. Was there like a part of the supply chain or one of these conferences that, you know, particularly changed your view of the semi world or that you felt then or feel now is particularly underrated?

• 11:06Speaker 01

  I think, I think the trade shows like rain and conferences range really widely. Obviously, some of the, you know, the ones I have the most fun at, you know, include NERIP. So why, why is that? Because it's 20,000 AI researchers and they're generally in my distribution of age range. So it's like a lot of fun, but they're also like leading AI researchers and it's a lot of fun and you learn a lot. There's also a lot of parties and then it ranges all the way to like, you know, there's random chemical conference in Japan where it's 300 Japanese dudes. It's like 20 guys from ASML, 20 guys from TSMC, 20 guys from Intel. And those are the only people who speak English. Everyone else speaks only Japanese and you're like, ah, I guess there's still pretty interesting and fun. I think, I think like one thing that I have like a skill set of is like, I'm able to bond with anyone regardless of their background and like,

• 11:47Speaker 01

  who they are, I'm able to talk to them, find something interesting to talk about. Oftentimes it's the tech stuff, but you know, it's, it's, and so I think like the most interesting conferences are oftentimes like, you know, the really big ones because that's where the biggest stuff is happening. But I think the niches that are really, really exciting is like, you know, SPIE. So there's IEEE, which is International Electrical Engineering, something. And there's SPIE, which is another ecosystem. SPIE conferences are super, super deep in details. Every single one that I went to, especially like SPI Advanced Lithography or SPI Photomask. I went to them the first time, I didn't even understand 90% of what I heard. And then I read, read, read, read. I'd made some context, of course. And the next time I went, I understood like half of what I went to.

• 12:30Speaker 01

  Third time I went, I understood like 75% of what I went to. Even now, I went and I was like, I still don't understand everything that's going on. Whereas like you go to like NERIPS, you know, a couple of times you can understand, okay, what's neuro-symbolic reasoning? Okay, what's this? What's that? Like you can, you can kind of get a mapping of what everything is pretty quickly. But some parts of the supply chain are so arcane and so deep and so technical. It takes a lot of times for you to even understand what's happening in, in, you know, on everything, right? For every research paper. Doesn't necessarily mean you didn't, you know, you go to a conference for a few reasons, right? You understand the research, you understand, but like it's all the research that's being published. But what you really care about is understanding how does that research intersection, intersect with technology also, how does that research differ from what's there today? And none of these research papers tell you what's happening today.

• 13:13Speaker 01

  But then you just ask people and you, you build contacts and you learn. And then you like learn about the supply chain and oh, this company supplies this company, even though it's not publicly stated anywhere or like, you know, you learn that the, the this chemical is like cost about this much and a tool uses about this much.

• 13:27Speaker 00

  You hear the horror stories of like this chemical had a shortage and it totally threw off this part of the supply chain. And then it turns out there's only three companies in the world that make that chemical. And it's like.

• 13:40Speaker 01

  My favorite one is I learned a Japanese guy at that specific Japanese conference that I went to where no, almost no one spoke English in very broken English. He told me about how his father worked in this, in, in, in this industry in the 1980s. The, the only factory in the world that built this chemical burned down. And that caused memory prices to like double or triple. And I was like, wow, not too different from today.

• 14:02Speaker 00

  Not, not at all.

• 14:03Speaker 02

  It's crazy. Inference, going to be the biggest market on earth, biggest market beyond earth. Agree or disagree?

• 14:12Speaker 01

  I mean, obviously use of tokens is going to be the biggest market.

• 14:15Speaker 02

  Yeah.

• 14:16Speaker 01

  And the value that's created from tokens is going to be the biggest market. But I think tokenomics, sort of the use of tokens, adoption of AI sort of is the most important thing that's happening. And inference, whether it's open models or closed models will be like one of the biggest markets in the world, much bigger than oil, I think, much bigger than like, you know, many other parts like inference of AI will be, you know, many percentage points of the GDP.

• 14:34Speaker 02

  Yeah.

• 14:35Speaker 01

  Right?

• 14:35Speaker 02

  What you've done with inference X, I think is, you know, industry standard. Maybe say a word on why you started it, what it does. And, you know, what do people misunderstand about performance benchmarking on inference?

• 14:47Speaker 01

  Yeah, so to zoom back, right, like semi-analysis, we do a lot of stuff that's like, you know, a lot of it is like research for institutional clients and our subscription for those products. But a lot of it is also like, hey, you know, this would just be cool to figure out. Let's figure out how to figure it out and just post it publicly. And that begets, you know, more and more scale. And so we've done this with a lot of GPU benchmarking and testing and training performance and inference performance. But, you know, ultimately, we saw like inference benchmarking was like point in time. You know, you test it and you take some time and you release it and it's like slow and arcane and outdated because models change all the time. Every, I feel like every week there's a new model, whether it's a Chinese model or, you know, today, Methos 5, Fable dropped and new models are coming out all the time.

• 15:30Speaker 01

  On the software layer, PyTorch, VLM, SG Lang, new drivers, new, new something drops, you know. In fact, the update cycle for most of these libraries is twice a week. So you basically have the software updating all the time and therefore performance changing. You know, new inference optimizations are coming out and those get updated. And so I feel like it's a relentless breakthrough after breakthrough after breakthrough that keeps driving efficiency and cost down, which is why we've seen, you know, model costs drop for equivalent quality by like 60x a year. It's incredible. But to stay on top of that, you can't have point in time benchmarking. You need to have benchmarks be living and breathing, i.e., you know, constantly running on the latest hardware, on the latest models. And so we embarked on a project and we got a lot of buy-in from the ecosystem.

• 16:13Speaker 01

  This was only possible because we had, you know, enough aura with some of the ecosystem where we're able to get CoreWeave and Crusoe and Nebius and Oracle and Microsoft and Amazon and Google and OpenAI to contribute to us, compute. And then we were able to work with SG Lang and VLM and now Radix Arc and InforAct, which are the private companies who are sort of leading those efforts, the open source efforts, to collaborate with us. We were able to get Nvidia and AMD and Google and Amazon now because we're adding TPU's and Tranium to collaborate. Now we've got all these people collaborating. We've got over $50 million of hardware donated to us. Once we launched TPU's and Tranium, it'd actually be over $100 million of hardware, you know, maybe about like 15 different chip types, all running these benchmarks every single day on all the latest

• 16:57Speaker 01

  Model, right? The best model for moonshot, the best model from Alibaba, the best model from, there's about five different Chinese models, the best open source models, the best Chinese labs there. We run benchmarks on their models every day and then also the best US open source models, GPT, OSS, NemoTron, etc. So we're running these benchmarks every day in an automated fashion and they run on these these servers that are dedicated to us for inference benchmarking. And we sweep across so many different configurations and optimization types. And then what it creates is, and all the results are public and all the configurations are public. So now we have the Pareto optimal curve because a lot of, you know, times when people are comparing inference performance, they're like taking a suboptimal curve or point for someone else and comparing it to their optimal one.

• 17:38Speaker 01

  It's like, well, yeah, I can make, I can, I can stick, you know, if I drove a Porsche versus like some, some race car driver, obviously I'd drive it slower. The same thing with inference benchmarking. And so what we did is we created open source, basically containers for the optimal points across every point on the interactivity, i.e. how fast is it responding to me versus, you know, batch size, i.e. how many users am I simultaneously serving curve? And so now anyone who wants the optimal point can just go to inference X, download it and run that as the optimal point and they can check every day if they want, or they can even auto download the most optimal point for that model and, and their inference performance will be near peak.

• 18:12Speaker 02

  Is that curve like the most important curve in your opinion? The throughput interactivity curve is the most important?

• 18:18Speaker 01

  Yeah, I think, I think most things in hardware, infrastructure, model, application layer, everything is downstream of that curve, right? Is it, is it something that needs to be super, super fast, super low latency? And I don't really care about the cost, so I make batch size very low and I use techniques like speculative decoding or multi-token prediction heavily. And there's so many, you know, possible techniques there. Or is it something where actually I'm batch processing a ton of documents? And I don't really care about all these things. I don't use these techniques that actually are worse on cost efficiency to help you with speed for an individual user because I just want to pack a bunch of users. I don't care if the document takes all night to process, right? And right now, the way we treat AI infrastructures, it's like one size fits all. But over time, we're going to get to the point where, you know, there's stuff where you have batch workloads

• 19:03Speaker 01

  or, you know, you need instant response and there's, there's the whole curve that's going to matter for users. And so we see this with Anthropic, right? Claude code, fast mode costs way more than regular mode. And same with OpenAI's priority queue thing.

• 19:17Speaker 02

  Sorry, dumb question. How does cost factor into the start?

• 19:20Speaker 01

  So if I, let's say, imaginary example, I have a batch size of 100.

• 19:25Speaker 02

  Okay.

• 19:25Speaker 01

  And I can do 10 tokens per second per user. So in total, I'm doing 1,000 tokens per second off of that one piece of compute. That's one side of the curve, super slow, 10 tokens per second. You know, other side is I have 500 tokens per second, but I can only have one user. And so maybe 250 tokens per second, one user. And then there's points on the middle that are more Pareto optimal, right? The average person actually wants like 50 or 100 tokens a second. And maybe, you know, the number of users I can batch together. So the curve is, okay, 1,000 tokens total per second or 250 tokens total per second, depending on how many users I batch. And there's a curve in the middle. And so ultimately, some workloads will actually want the 4X cost decrease because the same unit of hardware can do 1,000 versus 250.

• 20:10Speaker 01

  And some users, I'll pay 4X more because I don't care about the price. I care about time because the person using the tokens is expensive.

• 20:17Speaker 02

  Yeah.

• 20:17Speaker 01

  Or the feedback loop that I have here is expensive.

• 20:20Speaker 00

  If you had to guess, you choose the timeframe, 10 years or 15 years, what percent of inference compute you think will happen in space? It can be 0%, 50%, 99%.

• 20:33Speaker 01

  This is a tough one.

• 20:34Speaker 00

  You choose the timeframe, like 10, whatever timeframe in your.

• 20:38Speaker 01

  So I think, I think the non-consensus, or at least against SpaceX thing, you know, I love SpaceX, by the way, and I would totally would buy the IPO if I could buy stocks.

• 20:47Speaker 00

  Not investment.

• 20:48Speaker 01

  Not investment advice. Thank you. Thank you. Not investment advice. From Squealer. I don't think that space data centers will really matter in the next, you know, three to five years. With that said, I think in, you know, 20 years, I think the vast majority of compute will be going in space. And so the real factor there is sort of.

• 21:09Speaker 00

  You know, what's the cost?

• 21:10Speaker 01

  It's the timeframe. It's the cost of building power on terrestrial land and how much power are you going to be able to do on terrestrial land? And I think obviously my views of where inference, you know, you know, how many gigawatts or terawatts are devoted to inference is, it's a crazy curve for me personally.

• 21:25Speaker 02

  What's your forecast? How many gigawatts or?

• 21:27Speaker 01

  Yeah, I think, I think by, you know, 2030. Just OpenAI and Anthropic will have over 100 gigawatts combined. And then you'll add Meta and Google and so on and so on and so forth. It's a humongous amount of compute that will be dedicated to inference. And by 2040, it'll be terawatts, right? The curve of productivity that we're going to get and so inference deployments is going to be huge. And so if you look at 2040, I think probably more than half of incremental compute will be going in space. But if you look at 2030, I think it's sub 1%.

• 21:56Speaker 02

  Do you think intelligence per watt has been increasing? And then it seems like there's still a giant gap between where we are, intelligence per watt versus like human biology. And so if we are, do you think we are to close that gap? And if so, where is that game going to come from?

• 22:09Speaker 01

  Yeah, I think it often depends on what you're doing too, right? Like a TI84 is way more intelligence per watt in terms of doing math than us. And I was like 30 years old, right? So obviously, this is like a dumb, dumb, you know, sort of...

• 22:19Speaker 02

  General intelligence.

• 22:20Speaker 01

  Yeah, but general intelligence wise. So one of the things inference X does is we also measure the power and cost of all of this hardware. And so we offer not just throughput versus interactivity. We offer cost versus interactivity. We offer power versus interactivity. And so as far as has intelligence per watt been increasing? I mentioned it's been a 60x cost decrease for same benchmark level. We've also seen the same on intelligence per watt. It's not been exactly 60x, it's been closer to like 40x. Some of the efficiencies are non-powerways. But there's been a humongous improvement in intelligence per watt on an annual basis. At least so far this year, last year, year before, year before. And I expect that to continue.

• 23:02Speaker 01

  As far as where we are from the human brain, we're many orders of magnitude away. Thankfully, it doesn't really matter. We can devote a lot of power to computers, much easier to power computers than human brains. Like we have sickness, disease, and like food preferences.

• 23:15Speaker 00

  Sleep.

• 23:16Speaker 01

  Exactly.

• 23:18Speaker 00

  Let me just ask one more question on the general theme. In my opinion, in terms of like, you know, intelligence per watt or intelligence per dollar, like any of these metrics, I think there's kind of three levels of input. You can get hardware improvements where the hardware is more efficient. You can get low-level systems optimizations, like kernel-level improvements, matrix multiplication libraries, things like that. Or you can get like high-level, like model-level algorithmic improvements at the highest level.

• 23:57Speaker 00

  It's like, to me, it seems like in the last three years, most of the gains have come from hardware level and, you know, and some from the model level. Like, do you think that that is what, do you agree with that? Do you think that's what it'll look like in the future? Do you like, do you think there's a bunch of juice to squeeze in the, say, like kernel level?

• 24:17Speaker 01

  Yeah, Sean, I completely disagree with you, by the way.

• 24:19Speaker 00

  Great, great, that's why I asked the question.

• 24:22Speaker 01

  Okay, so I, I think, you know, one way is to look at it as these three different layers. And in that sense, like, okay, from Hopper to Blackwell, which is all we've had over the last three years, roughly 30x improvement on DeepSeek on the most optimized deployment, which is, you know, you can see on inference X, there's about a 30x improvement. But, you know, over the last three years, we've had way more improvement in intelligence per watt. A lot of that coming from the model layer, right? If you look back three years, it's GPT-4. Now it's like, you know, maybe like Qwen, one of the smaller Qwen models that's like, you know, 27B parameters total and like 2 billion active is like way better. And so you've got this huge improvement on model layer, you've got this pretty sizable improvement on hardware, but it's that co-design layer. And I think that's, that's what's important, right?

• 25:05Speaker 01

  If you look at the architecture of, you know, any of these models, but DeepSeek is the most famous one, at least that's public and people have seen it.

• 25:12Speaker 00

  Yeah, DeepSeek got huge efficiency gains from like co-optimization or kernel level optimizing memories.

• 25:18Speaker 01

  Yes, I think it's, it's like kernels, of course, but it's actually, you build the hardware architecture for the chip. So if you look at the shapes of all the experts in, in DeepSeek, V3, they were all optimized for Hopper. And if you look at for V4, they're optimized for Blackwell and Huawei's chip. And what's interesting is despite the fact that TPUs are objectively an amazing chip, you know, and, and they run all of DeepMind and they do all the training for Anthropic as well, on the pre-training site at least, TPUs suck at running DeepSeek. But they are really, really great at running other kinds of models that don't run well on Nvidia. There is some level of such deep optimization that has been done, whether it be shapes, network I/O, patterns, you know, how you do the collectives, how you do things around,

• 26:02Speaker 01

  you know, the, the arithmetic intensity, the attention mechanism, all these different things are co-optimized. Between the model and the and the and the hardware and the infra software in between. And it's it's hard to say you can disentangle the gains.

• 26:15Speaker 00

  Do you think that like my understanding is that like China has done this a lot better than the West? The last few years like in the deep sea was one of the first models to really like do this.

• 26:27Speaker 01

  I don't necessarily think so. I think it's more so that the West doesn't tell people what they do. Right? Like OpenAI didn't tell people that, you know, GP four O was how sparse it was, what the shape size was, all these things. But GPT four O is roughly the same size, slightly smaller than deep sea V three. And four O came out, you know, a little bit earlier, right? If I recall correctly.

• 26:47Speaker 00

  Is your is your view that like all three of these things have been happening simultaneously at like roughly the same rate? And the most, the biggest gains are when you just co optimize.

• 26:57Speaker 01

  Yeah, I would say I would say there's been more gains on the model layer than on that co op. And then then on the sort of software infrastructure layer and the hardware layer. But there's been innovations on every layer and and and really the biggest gain. And the beauty of the best labs is when they co optimize all three, you know, and and and that's what like, you know, when anthropic is is, you know, even though they use many different kinds of hardware, they don't really inference too much on TPUs. They mostly train on TPUs and and they inference a lot on trainium and GPUs. And GPU is more the jack of all trades, but they've optimized their hardware, they've optimized their model, they've optimized everything so they can do that. Whereas OpenAI, they're, you know, prior models were optimized for Hopper more. Now they're more optimized for Blackwell and, you know, you you step forward through time, these these um, these labs. And and the same with Google, right?

• 27:42Speaker 01

  They've they've they've optimized, you know, Gemini two was really optimized for the TPU uh, V uh, V six E or T P Gemini three was. And then Gemini, uh, you know, the next Gemini that's coming out is really optimized for TP V seven. Um, and so sort of like a lot of these things are being co optimized. And actually, when you pull that model and put it running on the old hardware, it's really not that great. And so I think a lot of this co optimization is is the most important thing. It's called software hardware co design. And that's what's like really exciting about like, you know, sort of what what, you know, I think my day to day is like, you know, great. You get to look at one layer, there's. All these innovations happening here. There's all these innovations happening on every layer. But the real breakthrough innovation is when you leapfrog a few layers, you co-optimize and co-design them.

• 28:25Speaker 01

  And now all of a sudden, you've you've taken what could have been a 2X here, 2X here, 2X here. And instead of being multiplicative to 8X, it's actually 100X because you've optimized it across all three layers. And so that's what's really exciting about sort of like what you see at the labs, which you see at like a company like Nvidia, who's not co-optimizing on the model layer per se, but a little bit from the model layer all the way downstream to, you know, silicon. Or you look at a company like TSMC, they're co-optimizing not just, you know, fabrication, but all the way from the components and the consumables and the tools, all the way upstream to what the designs, their chips are, the customers are telling them, is this co-optimization across many layers of the abstraction stack.

• 28:59Speaker 00

  There will always be bottlenecks somewhere in that optimization that they're like lagging behind and then need to get pulled forward.

• 29:08Speaker 01

  And Band-Aids to

• 29:09Speaker 00

  Exactly. If you had to predict, like what are, at any level of the stack, it can be literally anywhere. What are some of the bottlenecks you're most, like you're kind of tracking most acutely the next year? And not necessarily in the supply chain, not in like scale, but in terms of the actual. And it can, it can be in the supply chain too, but just like, you know, is it memory improvements? Is it, is it that like just like scaling?

• 29:36Speaker 01

  So memory, memory is, memory is an easy one that everyone's talked about, but I'm not going to talk about it from a supply chain angle. I'm talking about from a technology angle, right? Memory capacity and bandwidth have been improving very slowly. The NAND cell was invented like 25 years ago, the DRAM cell was invented like 40 years ago. And there's been no major breakthrough in, in cell, like, you know, what a NAND cell is. Obviously NAND is like a very simple gate or DRAM cell. There, there is stuff that could come down the pipeline that could be hugely innovative. But even over the last, you know, five years, all we've really done is make the HBM, you know, more stacks faster. But actually there's like new innovations coming in the next few years where instead of, you know, stacking the HBM separately from the chip, you stack the memory directly on the chip and that makes your bandwidth explode.

• 30:20Speaker 01

  And so there's interesting companies in that space and interesting POCs that companies are trying to do there. I think like memory bandwidth is one of the biggest. Another one is For the history of silicon, basically for the last two decades at least, you know, how many watts a chip is can be easily predicted just by looking at it for for a data center or desktop chip. It peaks off at one watt per millimeter squared. And so if a chip is 100 millimeter squared, generally the power consumption is around 100 or a little bit less. And if you look at the newest Nvidia silicon, the newest TP silicon, it's still on that range of one watt per millimeter squared. So, you know, chips are now getting to, you know, 1,400 watts. The next generation is 2,000 watts for Nvidia with Ruben and such. And you move forward to Ruben Ultra, it's going to be like 4,000 watts or something

• 31:05Speaker 01

  like that. But really, there's increasing the amount of silicon. What's exciting is we're now finally doing things, and it's in development right now, where you actually can pump the amount of power into the silicon to be way more, more than one watt per millimeter squared. And now that all of a sudden means you need less silicon. Obviously, it's running at higher power and it's less efficient in some cases, but you reduce the amount of silicon and you're able to like overcome certain problems.

• 31:29Speaker 02

  Without running like thermal issues?

• 31:31Speaker 01

  Thermal issues, there's interference of like electrical interference issues. There's all sorts of different issues that crop up, and that's why it's a hard engineering problem. That's why we've stuck at about one. But what's exciting is the world is trying to change these things. I think interesting, like in a different part of the supply chain is sort of like, you know, people, people will talk about like energy is hard and, you know, we have energy bottlenecks and it's like, yeah, but there's actually like very simple solutions, you know, one could think of, right? Take the millions of diesel engines for trucks that the US has the capacity to make. You can very trivially convert them to be using for gas in the assembly line and then stick them up to a electrical motor, like backdriving it so the electrical motor generates

• 32:12Speaker 01

  electricity rather than the electrical motor causing the rotation of the wheel, for example, but doing it the opposite direction. And now you've generated electricity by pumping gas into something that US can make millions of. And then, okay, well, that sounds like a pain in the ass to service, right? Because now you have to have hundreds of these on a data center site. Well, actually, you can just pull people out of car mechanic shops and have them run around and repair truck engines. Actually, it's actually pretty trivial to not, I don't want to say it's trivial, I couldn't do it.

• 32:41Speaker 00

  I think you're making a really good point, which is that. Like because the West wasn't really thinking about semiconductor even hardware more broadly the last 20, 30 years, we didn't have like much innovation. We'd have the best minds like thinking about how do you improve these things.

• 32:57Speaker 01

  Why would you, why would you want to go work in hardware where you can

• 32:59Speaker 00

  Pro, you make ads, sell ads.

• 33:01Speaker 01

  Yeah, exactly.

• 33:03Speaker 02

  Okay, I'm dying to ask Nvidia versus TPU. What are your thoughts?

• 33:07Speaker 01

  I think, I think like everyone wants to pick one or the other for this, but it's really like a function of like, look, you know, you look two years from now, Google's going to make 10 plus million TPUs and through their supply chain and Nvidia is going to make, you know, many more million, tens of millions of GPUs and both are going to be 100 plus billion dollar, you know, well, Google's going to be 100 plus billion dollars, you know, of TPU created a year and, and Nvidia will be, you know, 500 plus or, you know, whatever is, I'm not making a specific estimate.

• 33:34Speaker 00

  It's not a revenue forecast, this is just a thought experiment.

• 33:36Speaker 01

  Yeah, our research has that.

• 33:37Speaker 02

  Sean, you've been media trained so well.

• 33:38Speaker 00

  Absolutely, you know, getting ready for the SpaceX IPO.

• 33:42Speaker 01

  Are you guys big in SpaceX?

• 33:43Speaker 00

  Yeah.

• 33:44Speaker 01

  Okay, so that makes sense. We're very lucky to be very large investors. Awesome, awesome. So I would say the, the case of sort of like Google TPUs versus Nvidia GPUs, they both have like points that are really like in their favor, right? You know, Nvidia will be like, oh, we have switches and we're general purpose and, and TPUs will be like, well, we're more optimized, we're actually more energy efficient and, and our network is actually more optimized for certain types of network architectures. And so you have like these counterpoints that both would really get into and, you know, I could with a straight face argue with you like that GPUs are way better than TPUs or TPUs are way better than GPUs, but it comes down to hardware, software, code design. So actually the way OpenAI's models are headed, it would be a terrible

• 34:26Speaker 01

  decision for them to use TPUs potentially. And the way that Anthropic and Google's models are headed, it's actually a terrible decision potentially for them to train with GPUs. I mean, it'd be fine for them to train.

• 34:35Speaker 02

  Why is that? What's the, what's the fundamental difference there?

• 34:37Speaker 01

  There's various things, right? Like, the size of the matrix multiply unit is different, as a very simple thing. And therefore, the shape of the matrix multiply you do, the attention mechanism you use, the way that attention mechanism is structured, the way the experts are structured.

• 34:52Speaker 02

  So OpenAI and Anthropic are converging to very different model architectures?

• 34:55Speaker 01

  I think they have quite different model architectures, in fact. You know, OpenAI's are much more sparse, and that has benefits. And then Anthropic's are, you know, they're still sparse, but more dense in general, and that has different benefits. And there's many other things, right? The network topology, right? Nvidia, all of their chips are connected to switches, NVLink switches. For Google, they have no switch. But what they've done is they've been able to, you know, Nvidia, the NVLink can only connect 72 GPUs. For Google, their ICI can connect 8,000 chips at super-high bandwidth, but you have to pass through other chips to get there because there's no switch. And so there's like, there's trade-offs there. There's positives and negatives, and that influences the model architecture. It's not necessarily that you should, you know, claim one is better than the other,

• 35:39Speaker 01

  because at the end of the day, how do you say that this is better than that when you can't measure them in isolation because it also extends up to the model layer? Right?

• 35:49Speaker 02

  But I remember for a long time thinking, you know, one, the programmability of Nvidia, and then just CUDA as such a big mote. It seems to me that narrative has kind of changed, at least in my mind for the last three, six months. Like, model companies no longer care about, if we have to write custom kernels for, you know, this other chip, so be it. We'll work with four or five chips if we have to. Claude and Codex are actually quite good at doing a lot of that optimization work. And so it seems like some of the, and then, and then it's, you know, it's not like there's 10,000 model companies that are each, you know, each need programmability. There's on the order of tens, maybe, model companies. And so it seems to me that, like, if you, the fundamental premise of, like, tens of thousands of big customers that need CUDA compatibility, like, it seems that kind of thesis is changing.

• 36:34Speaker 01

  Yeah, I mean, certainly, the CUDA mote and software mote is at least partially disentangled because, you know, models are just great at coding, and all software gets commoditized in that case. I do think there is some level of Like open source and you know what people call the CUDA mode is not actually anything to do with CUDA. But it's like the fact that DeepSeek, Kimi and Zippo AI and Alibaba and Tencent, all these, all these companies. Xiaomi had an awesome model recently. Their models are a co-designed for GPUs. And therefore, if I want to run them on TPUs, actually, in some cases, they don't run really well on TPUs. Now, Google just has to create their own open source model ecosystem or open source models themselves. So they have the Gemma models. And so you end up with like, well, that's not really CUDA as a mode.

• 37:19Speaker 01

  It's that the downstream product is more optimized for Nvidia. And in these cases, these companies are just open sourcing them, or like Nematron is just open sourcing it. And then the users of it, for example, to open, you know, the inference API providers, the RL companies that are trying to take open models and customize them for companies' business use cases. All these different companies are downstream of the fact that like, okay, well, I guess I need to use Nvidia. Because the ecosystem uses Nvidia, even though I don't particularly care about writing CUDA kernels. Because the models are great at that, but it's like the shape of like, well, this expert, the D mod is this. And you know, the hidden dimension, blah blah is this, right? And so therefore, it's better to run on Nvidia GPUs than it is on TPUs and vice versa, right? If Google were to actually open source really good models, you know, this would be the same thing, right?

• 38:04Speaker 01

  People would take their models and they'd be like, oh, wow, these don't run that well on Nvidia GPUs. I should actually just rent TPUs or buy TPUs and do it on there. For small teams, you're going to want to use all the open source software like VLM, SGLA, PyTorch, all that stuff. But the big labs, they don't necessarily need to use all that, right? OpenAI has forked PyTorch long ago and you know, Anthropic and all these other people don't necessarily rely heavily on the open source implementation of, you know, these things. They've forked things or built it on their own already. And so they don't need to rely on the open source. And therefore now it's more like, you know, I'll choose the best hardware and I'll co-design my model and infrastructure software through and through for that hardware that is the best and most cost efficient. And you know, I'll have AI help me write all that software.

• 38:46Speaker 02

  What do you think of Cerebras?

• 38:47Speaker 01

  I think Cerebras is a really innovative company. I think in some spots of the market, they're really, really good. Very fast inference, I think that's a big market. We use fast mode almost exclusively at some analysis.

• 39:02Speaker 02

  By the way, I love how disciplined you've been about accounting for, I don't know if that was one exhibit you did, or if you do it consistently, but accounting for the dollars spent in the ROI on each task. Awesome analysis.

• 39:13Speaker 01

  Yeah, yeah, we do it pretty diligently. So thank you, that was the dark GDP article that we wrote. And so, and also like track everyone's token spend by day. And if someone's spiked up, I'm like, what did you do? It's like, okay, thank you for telling me. That seems worth it. Cool. On with my day. I think fast mode is obviously worth a lot for high-end tasks, right? I could just see so many different use cases where, you know, super fast tokens are worth it. I can also see the flip side where there's a lot of use cases where super fast tokens aren't needed. And therefore, the market won't pay for them and they'll use GPUs and TPUs instead. I think the big risk for Cerebras is, I mostly think the best models are the ones that you want to use fast mode on. And small models you necessarily might not use fast mode on. I could see that being wrong with, you know, financial markets, maybe, or something like that.

• 39:58Speaker 01

  Like a Jane Street high-frequency trading or something like that, or medium-frequency trading. But ultimately, you know, running really large models at really long context is very difficult on SRAM-based chips like Cerebras, like Grok. And so now it all of a sudden is like, you know, what happens then if like the models get too big, right? If OpenAI's model's not, you know, on the order of, you know, hundreds of billions of parameters or, you know, low trillion parameters, but it's actually 10-plus trillion parameters, now all of a sudden, I don't think that that will fit on Cerebras, right? And then if that doesn't, with a long context length, right? If you have a million context length, now that makes it really difficult to justify. You know, and is all, so far, we've seen the bulk of revenue and usage at the labs be on their best model. Even when the model price has gone up, we've seen that.

• 40:43Speaker 01

  There's some data that shows that even though Fable just released today, they've had incredible amounts of people switch to Fable and Methos, sort of that next tier model, even though it's way more expensive. And so. And that's volume by dollars totally, but without volume by tokens. Well, I guess who cares about volume by tokens? It's about the dollars. Fair enough. Right? If I don't care that there's, you know, I don't know, 200,000 Mini Coopers or Toyota Camry sold. If, you know, I don't know, 4.150s are 5x the ASP and they sell only half as much. Okay, fair enough. And therefore, the most lucrative market is pickup trucks in America, right? Mostly being facetious, but like.

• 41:19Speaker 00

  Do you think this is one of the things that you've done so well and differentiates you from almost everyone else? Is that you care so much about the economics in addition to the technology? And I think very few people bridged those two things well.

• 41:32Speaker 01

  Yeah, thank you. I think it's really fun inside of semi analysis because we have 90 people and like a big chunk of them are technologists, engineers across the whole supply chain. And then a big chunk is people who are formerly at hedge funds. And you see these arguments, like people are like, oh, well, that doesn't matter. And it's like, then someone's like, well, but cost. And then someone, the engineer's like, no, no, no, but this technology is the closest. You see this, you see this organically like fight it out. And we're pretty informal and, you know, given the fact that I was a former moderator is you can imagine what the.

• 42:01Speaker 00

  Well, you're enjoying it. You're enjoying it. You don't wrestle with a pig because a pig enjoys it, right?

• 42:07Speaker 01

  Exactly.

• 42:08Speaker 00

  On this topic, before going to the next question, are there like trigger topics in semis for you? You know, like if someone's, which is like such a meme, you think this person must be a moron. Like if, you know, if it's like, oh, you like memory is the bottleneck.

• 42:26Speaker 01

  I mean, it's true, but like, I think, I think moreover. The one that really gets me is people are like, AI has no ROI.

• 42:36Speaker 01

  Infuriates me, right? Like there's like, what's the ROI or like denying model progress, right? There's these people that are like, models aren't getting better. They're not reasoning. They can't think. They're going to dead end and plateau. And it's like, bro, the line has been up and to the right in terms of capabilities this entire time. And they're like, look, this benchmark didn't improve. That's because it said 90%. Look at the new benchmarks. Yeah, you saturated. Now they're skyrocketing, right? It's like, I think that's more so the issue and challenge. Like, I think semis are really complex and I don't fault people for lacking like, understanding of it. Like, I learned stuff every day about the semiconductor supply chain from people. And I've been studying it for, you know, arguably 18 years since I started moderating the forums when I was 12, right? Like, you know,

• 43:21Speaker 01

  arguably been studying it for that long, but even then, like, and it's like live, breathe, and that's all I care about. But there's so many layers of the abstraction stack. It's like, like, I learned about a new chemical that does like $100 billion of sales like yesterday. And I'm like, whoa, didn't know this one existed. And what process it did. It's like, but it's like, you know, you learn about things all the time. It's like, okay, 100 billion dollar sales in a you know, couple hundred billion dollar industries, whatever. But like, you know, it's like,

• 43:44Speaker 00

  But it's essential.

• 43:45Speaker 01

  It's essential. And it's like, actually, every chip requires it. It's like, wow. I guess there are 1,000 process steps. And I, you know, it's like, oh, yeah, you like semiconductors name every process step. It's like, no, come on. What I think is the most funny is when people have all the facts in front of them, and then they get the conclusion completely wrong.

• 44:01Speaker 00

  That lives in our job all the time, too.

• 44:03Speaker 01

  Yeah, yeah. I mean, I can't, I get, I think my attitude is not to be mad that you do that. It's to do it as fast as possible.

• 44:10Speaker 00

  I think the industry, because it's so, it's just like, AI is the most important thing in the world right now. And there's so many near term bottlenecks. We talk a lot about the near term. Are there longer term things that you're really excited about? Like, say on a 10 year time frame. We talked about orbital data centers, but like, like silicon ultralines, you think they're underrated or overrated on a 10 year timeframe? Are there other things that on a 10 year timeframe?

• 44:32Speaker 01

  Yeah, I mean, I think on space, I think space is like super crazy awesome in the 10 year timeframe that I'm, you know, for space data centers and all these sort of mining asteroids and all these things, which is, you know, super excited about the vision of SpaceX, right? Again, not investment advice before you hop in. I think, I think on the semiconductor side, tremendous market movements and tremendous like things can happen just when like things happen one year later or sooner. And so that's all like technology that like, you know, in terms of like co-package optics, like, well, like, everyone knows it's gonna happen by the end of the decade. The debate is like 27, 28, 29, 2030. But some point along there, it's gonna happen. I think the more interesting thing is like, there's companies like, uh, I, did you guys invest in Naveen Rao's company?

• 45:12Speaker 00

  We did.

• 45:12Speaker 01

  Okay, yeah, so I think like he's trying to innovate on like the silicon layer, on the software abstraction layer and the model layer simultaneously. And he fully understands that it's not like, you know, we're gonna do this in a few years.

• 45:23Speaker 00

  It's not a two-year timeframe.

• 45:24Speaker 01

  Yeah, it's not a few-year timeframe. It's a long-term bet. And like, stuff like that is like, okay, we're gonna bring like, potentially like analog compute with energy-based models and like all this crazy shit all at once. It's like, that's exciting. Probably won't work, but you know, that's exciting and I like really look forward to it.

• 45:40Speaker 00

  And it definitely won't work quickly.

• 45:42Speaker 01

  Yeah, definitely won't work quickly is what I should say. I believe in Naveen and like, you know, I, I, I met him very, you know, I think he's one of the first people I met in the industry, fondly enough, like in 2020 or 2021. Actually, 2020.

• 45:56Speaker 00

  It says something about him. I think he's someone, in my experience, he's always trying to.

• 46:00Speaker 01

  I baited him on the internet. I baited him on the internet.

• 46:03Speaker 00

  He's always trying to help the younger generation. He's trying to identify talent and.

• 46:07Speaker 02

  He was also so ahead of his time with Mosaic. I remember getting pitched.

• 46:10Speaker 01

  No, it was 2019. I was still, I was still anonymous then, actually. I baited him on the internet and he started replying and then I just took it to DMs and then took it to a call. And like, that was the first person who was like really important that I talked to in the entire semiconductor industry. But yeah, sorry to interrupt.

• 46:27Speaker 02

  Yeah, that's funny. What do you think is the end state of the ecosystem? Like, do you think every lab, every hyperscaler just has its own chips? Like, Trainium seems like it's now working, right? So do you think we end up with every lab, every hyperscaler has its own chips, at least for inference and then maybe for training, you go to Nvidia or whoever? What do you think is the end state?

• 46:44Speaker 01

  I think everyone will try and they won't stop trying. I think ultimately, you know, supply chains matter, what technology you can bring in matters. And more and more as the industry gets bigger, supply chain diversification happens. You know, right now, everyone's chip more or less looks the same. It's a big logic compute die in the center and there's some HBM on the right and left and on the top and bottom. Top side is networking and then the bottom side is PCIe and other I/O. And that is the exact same structure for Trainium, TPU, Nvidia chips and most of the startups. Not Grok and FreeRis, those are doing weird shit, but that's. Cool. Now, I think like as you step forward, we're going to get more bifurcation of hardware

• 47:27Speaker 01

  architecture and model architecture, and therefore people are going to co-optimize them. And you know, some of them will end up in local minimas, right? You know, as we're, you know, if this is like gradient descent, like people are like trying to go to the most optimized solution, some people will race to a local minima. And then the question is like, how do you leap, how do you scoot back over to like the absolute minima? And some, to some extent, like a general more, Nvidia will always be more general purpose than anyone else's chip in general, at least on a parallel AI compute basis, because they have so many customers who care about different things, who will always give them feedback in the design. You know, the minima will always be better than them, but is that minima a local minima? Like, is, is the TP or Tranium or Grok or Cerebrus or whoever's design optimized

• 48:11Speaker 01

  awesomely for here, but in the end state, actually, you got to go over here. And so they're wrong. And maybe they make a great time, they're great for a little bit of time, but then they end up being wrong. It's like, that's the real question. And so I think, I think there will be a big market for general purpose AI compute, because you talk to people at labs, they don't even know what architecture they're going to be doing in a year. Like, right? Like, they literally don't know what architecture they're going to be doing in a year. They have bets, they have many research bets, and that's this exciting thing, but they don't know where it's going. Generally, they like know what hardware they have, and they're trying to co-optimize. But ultimately, like, if a new breakthrough happens on model architecture, it's like, just replace the tension mechanism with something else, right? Who knows? Or, you know, all of a sudden, you know, something happens, the best hardware

• 48:53Speaker 01

  will change. And therefore, like, are people going to make five-year investments on hardware solely on, you know, an ASIC that is more specialized? Or are they going to do, they're going to have some bucket of more general purpose compute? And so you see this with, like, Google's paying $11 an hour per GPU to XAI for GPUs, right? Like, that's insane, right? It's a very high amount of, obviously, compute is limited, and so on and so forth, but it's like, very like, Insane, but despite the fact that they have TPUs. And so there's like some questions there, like, why do they do that? Google actually has three different design programs for TPUs. They're making a TPU with Broadcom, that's a different architecture than the TPU with MediaTek. That's a different TPU than the architecture that is, I won't disclose by research. But they're making different architectures.

• 49:38Speaker 01

  It's not just like, oh, they're making TPUs with a couple vendors and it's the same architecture. It's different architectures. And the third one is a very different architecture from the first two. And so I think people recognize that the local minima can happen. And therefore, I think everyone will have their own ASIC program. I think everyone will deploy billions of dollars of their own ASICs, tens of billions of dollars. In the case of Google, hundreds of billions of dollars a year of their own ASICs. But ultimately, they're also going to have workloads that don't use TPUs, right? Some of the Google bets that are not Gemini DeepMind actually primarily use GPUs. They don't use TPUs. Some of them also primarily use TPUs, right? It's a bit of a broad thing, but like, you know, maybe for drug discovery or for Waymo, you might not want to use TPUs. I don't say which one it is, but like, you know, there's different architecture bets

• 50:23Speaker 01

  and different paths for AI. AI for science may have different algorithmic patterns than general intelligence AGI models. And so I think we'll see diversity continue to proliferate.

• 50:33Speaker 02

  Yeah.

• 50:34Speaker 01

  And because the market has gotten so big, niches will be carved out. And so that's, makes it possible for companies to have their niche and actually make money, even if the majority of the pie goes to Nvidia and TPU and Traneum.

• 50:46Speaker 02

  Okay, love that. Can we talk about the data center build outs? Like, one, it seems like, I mean, by all accounts, if you look at the charts, like dollars per compute hour, we are in the middle of a crazy compute crunch. And it seems like it's both a demand and supply side crunch, right? Like, demand for long horizon agents skyrocketing, supply, all these data center build outs are delayed. Do you think that we're in a compute crunch for the foreseeable future? Or do you think it alleviates at some point?

• 51:10Speaker 01

  Yes, every quarter, we're deploying vastly more compute than the prior quarter, and there's more data centers built than the prior quarter.

• 51:18Speaker 01

  This year, there's going to be 20 gigawatts, even accounting for the delays. And next year, there's going to be more than 30 gigawatts, accounting for the delays. Of course, delays happen on everything, right? Anything hardware can have a delay, that's just the reality of life. Are we going to have a compute crunch for the rest of our lives? It depends on what happens with models. But like, the TAM for Methos, you know, Methos 5, Fable 5, is not just like 2X that of Opus, right? The model is so much better, and it can do so many more tasks. The TAM for it is way larger than that. And yet, compute in the world did not double in the last, you know, six months, right? From, you know, Opus, or maybe like seven or eight months since Opus 4.5 would launch to now. Huge, you know, 4.6, 4.7, 4.8 were improvements, but Fable and Methos were like a huge step function improvement.

• 52:03Speaker 01

  The world's compute did not double in that, or quadrupled or whatever in that same timeframe. But the demand for useful tasks that can be done by AI, the number of useful tasks and the value of them that can be done by AI has. And so now the question is, what happens? Well, obviously, Anthropic in Q2 is profitable. Their net income profitable, excluding stock-based compensation.

• 52:27Speaker 01

  And I think by Q3, they may even be profitable, including stock-based compensation. That's like how profitable they're getting. And their margins on an Opus token, at least Opus 4.8 token, is like north of 80% for the API price. They've got a lot of deals where their total corporate gross margins gets clawed down a little bit because of like how they do Bedrock deals and Vertex deals and things like that. But ultimately, their per-token margin is so high. Well, then if you don't have the capability, they have the capability to pay. Ultimately, every GPU they buy at above-market rate, you know, they also bought GPUs at above-market rate from SpaceX, which is below the rate of Google, but that's because they signed earlier. You know, it's something that, you know, other companies, maybe a venture-backed company

• 53:10Speaker 01

  or a company that's not really got positive margins, can't necessarily do, right? What is the cost-benefit ratios? Like, every GPU I rent, because I'm out of compute capacity, I can immediately turn around and sell tokens on it, or every TP or every trainium, I can immediately sell tokens on it at a positive margin. And if I'm running 75% gross margin and I double the cost of the compute, it's fine, I'm still running 50% gross margin. And spinning up more compute nodes is not really necessarily a human requiring task for them if they're renting them. And so ultimately, it's like, well, my NOI still goes up, right? And so I'm gonna rent GPUs at whatever price, at some level, whatever price I want to pay, I can pay.

• 53:46Speaker 02

  I have like, almost a reverse question of like, at some point, does this compute build out go bump a night? Earlier today, I think there was a tweet, like Crusoe publicly said, one of their customers had asked to halt construction on one of the data center build outs. Like, it seems like everybody in the ecosystem is so levered right now to like, we gotta build, we gotta go build, we gotta build. High leverage, high growth, to me is like, makes me very, very nervous as investor, like.

• 54:08Speaker 01

  Wait, hold on, high leverage, high growth means small amount of equity has huge upside. You're not a dead investor.

• 54:15Speaker 02

  Credit, you're an equity investor.

• 54:18Speaker 01

  Look, you gotta, you gotta go to the school of private equity. Levered buyouts only.

• 54:23Speaker 02

  I actually come from a school of private equity.

• 54:24Speaker 01

  Oh, awesome.

• 54:26Speaker 00

  She forgot the school of VC for too long.

• 54:28Speaker 02

  No, I just do revenue multiples. Are you, do you see any signs of that? Are you worried about that?

• 54:34Speaker 01

  I see what you mean, right? And that sort of goes back to the model point, right? Obviously, if the models expanding the total economic valuable like work, that's sort of the dark GDP report that we did, and that you mentioned earlier. If the work that these models can do does not expand faster than the compute capacity, then that tide turns, right? And over the last six months, that tide has been, you know, very much levered in this direction of, you know, the models can do more work or is expanding their TAM of work they can do faster than the compute is increasing. And so prices go up. It's very possible that all of a sudden, model progress stops. You talk to anyone at Anthropic or OpenAI, maybe they're drinking the Kool-Aid. But you talk to basically all of them, they're like, no, no, no, no, model progress still go up.

• 55:19Speaker 01

  And so, you know, ultimately, you know, current methods could stall somewhere. I'm not sure where that would be. It seems like we have line of sight to model improvement, rapid model improvement. And in fact, models are improving faster than they were six months ago or a year ago because there's, I would call it recursive self-improvement, but basically the engineer, the models are helping write all the info and. Launch the next model sooner and sooner and sooner. So you've got this like pseudo recursive self improvement loop going. And so the models are getting better and better and better faster. And so but ultimately, you know, capital is a big problem, which is why Google raised capital. You know, they they've got a godly amount of space, right? They own like 5% of the company.

• 55:57Speaker 00

  I think a little more, but yeah.

• 55:58Speaker 01

  Yeah, maybe huge. I think it's one point, they're like 10%.

• 56:00Speaker 00

  Larry Page invested a billion dollars, it did 10 billion dollar valuation, got 10% of the company, it got diluted, like all this, but that was one of the greatest investments of all time. Good job, Larry.

• 56:10Speaker 01

  Yeah, so so. So they know they have like 100 billion dollars in the bank that they can sell in, you know, nine months or whatever from the lockup. And they have all the gross profit they do, and yet they still modeled that. And they were like, we need to raise capital. And so they did an offering and it's like, that's insane. So that tells you how much they think they need to spend. But capital is like really, you know, you know, Meta's do it. Meta did announce that they're going to do a raise, stock tanked, people don't like it, but you know, that's all these companies are going to raise capital, whether it be debt or equity. At some point, money spigots will have to, you know, slow down. But right now, every GPU that Amazon adds, they're making higher revenue or every TP or Tranium, you know, whoever anyone adds is is making is making gross profit.

• 56:51Speaker 00

  I'm gonna do a little bit of a tee up on this to turn it into a question for you, but like, as we talk about this, for me, the thing that's going through my head that's almost that is almost an alternative hypothesis for like the Crusoe example. I'm using analogy in oil, like in oil, Saudi Arabia has way lower cost per barrel to produce oil than a lot of other countries. There's also like the purity of the oil. A lot of, you know, Saudi has generally like very low contaminants in their oil, which makes the refining easier, all of this. The question from me is like, when you look at for every gigawatt that's being put in the ground, if call it the 20 gigawatts coming online today, like how much? Like, how much homogeneity do you see in those gigawatts?

• 57:36Speaker 00

  Is it something like, and you can tell me whatever metric you think is right, but like, are Google's gigawatts two times more valuable than, say, most Neo clouds? Because they have optical switches and they have like, they've been doing it for a long time and like, they know how to do power smoothing. Because I think this could be the alternative hypothesis that some of the people that are, it's like the people that are good at building data centers, they should, they should just do it to the max because there's so much demand and there's so much better than it. But then maybe we're starting to see the early signs, the people that are like not as good at it, kind of getting hit a little. So I don't know the reality, I'm just curious.

• 58:16Speaker 01

  Yeah.

• 58:17Speaker 00

  How you think about it?

• 58:17Speaker 01

  So far, there are metrics for this, right? So, uh, Tranium sells at sub $10 billion per gigawatt rental rate to Anthropic and to OpenAI. GPUs, at least before the craziness of the last six months, usually went around 12 to 13 billion dollars per gigawatt. So the rental rate, and this is from a Neo cloud versus Amazon even, and now when Amazon sells GPUs, they'd also be 13 or so.

• 58:42Speaker 00

  And my understanding of that also is that those numbers, like, Amazon subsidized that a little bit so that it's like, I actually think the numbers were even, like, I think the disparity is even more.

• 58:52Speaker 01

  It's less than 10, it's less than 10, but there was like some weird, basically, how much?

• 58:56Speaker 00

  And like, look, I, my understanding, like Anthropic played a big role in making Tranium useful in terms of, you know, writing all the libraries, et cetera. And so like, I, everything I hear is that Tranium's really freaking good hardware and it's getting way and way, like, way better. And obviously Anthropic's now using it a lot. So hopefully we would see that price go up, you know, like per gigawatt.

• 59:19Speaker 01

  The, the, the deal they did was actually like, there was a floor mechanism and like, if it didn't do well, it would be like cheaper and then to the point where it's cancelable. And, you know, if it, if it did really well, the price is kind of higher. But effectively,

• 59:32Speaker 01

  Less than 10, right, is where Tranium shakes out at, whereas GPUs, I mean, the SpaceX deal again was like 25 or something crazy billion dollars per gigawatt, or 25 million dollars per megawatt, right, a year rental rate with Google. I was like, that's a crazy divergence. Now, obviously, if Amazon was selling Tranium today, it'd probably be more expensive than 10 because of the compute shortages. But you do see this already in the sense of with data centers. Oftentimes, the rental price of a data center, if you're doing co-location, right, not compute in there, but just power, here's the data center. You price it generally on a dollars per kilowatt per month. And so they used to be $60 per kilowatt hour per month, and now you see things transacting at anywhere from like 120 to 160.

• 1:00:17Speaker 01

  But different quality data centers, this actually, I've seen data centers go as high as 200. When the customer is not such a great credit rating, and then the data center is a pretty good one. And I've seen Steph go as low as 100 still, or in like India go like as low as 80 because their grid's not reliable, the internet connection's not great. And it's a pretty mid-data center, but at least it's a data center. And so you see this huge discrepancy there already. In the case of like data center construction, usually the pitfalls, they just fail. There's a lot of people who fail, you'll claim they're like, they're like four guys, they're like, yeah, I bought some turbines, I put the money down for them. I'm going to build a data center, and then they get delayed, delayed, delayed, and fail. So you have to like probability weight, time weight, time lag, the teams that suck versus don't.

• 1:01:02Speaker 01

  And sort of, you know, our data center model does that, we kind of track every data center and try to do this for every single one based on, you know, equipment that they're using and all these things. One of the things you mentioned about Google is, you know, in a gigawatt data center, they actually put like 1.5 gigawatts of hardware. And because they have such understanding all the way from workload to, you know, they're able to slosh the power around. And so instead of, you know, constantly, you know, a gigawatt of compute, which typically runs at like 60 or 70% utilization, in terms of power consumption, not utilization of the hardware, someone's always renting it. They're now running it at like, you know, you know, that 60 to 70% means it's at a gigawatt, and you're using the full gigawatt. You see people doing deals with, including Google, with utilities, where they're like, oh, well, I know this grid can sustainably take a gigawatt.

• 1:01:47Speaker 01

  But you know, except for three days of the year, you can actually do two gigawatts. So give me two gigawatts and then just tell me to turn off. And so they'll do that. And so these sorts of tricks, and then you need to have supreme management of workload, backup power, all these things, generators onsite to figure out how to actually keep it two gigawatts sustainably. When people do this, they're able to charge more, whether it be I'm actually selling two gigawatts despite only having one gigawatt because those three days I'm able to deal with via battery, gas, et cetera. Or I figured out how to build power onsite. Now I have a gigawatt where no one else does, and so I'm able to do it quickly. It's not necessarily transacting for a higher price. It's that I'm selling more gigawatts. And sometimes there are levers where you're selling more gigawatts, where each gigawatt is selling at a different price. I think it's more on the data center and energy layer.

• 1:02:31Speaker 01

  It's more about just having it versus not, and then that being delayed or not. It's more binary. But on the compute side, I do think there's a lot more interesting work there, right? A gigawatt given to Anthropic is objectively worth more revenue than a gigawatt given to OpenAI. And it seems that both of them could sell every gigawatt that they have right now, given rate limit problems and token max limit and all these sorts of things that OpenAI and Anthropic, especially since Codex 5.5 came out, it's much better. And then likewise, if you gave a gigawatt to SpaceX, you know, they'd turn around and sell it.

• 1:03:04Speaker 00

  My guess, like, my suspicion is that they're, they probably make better use of the, you know, hardware than most people. Just like, I think people underestimate how much networking experience they have from Starlink in particular, and also how much just like power management experience they have via from Tesla.

• 1:03:25Speaker 01

  Yeah, people like Brett Mayo are like incredible, like.

• 1:03:27Speaker 00

  Pretty good. Yeah. And so I think, like, for me, that's actually, I think, probably the thing that might, I don't actually know the answer, but I think that might be missing from the analysis a lot of people are doing.

• 1:03:36Speaker 01

  I think, I think it's also the fact that when CoreWeave builds a gigawatt, even though their GPU compute is objectively better than Amazon or Google or Microsoft's in terms of performance, we've tested the performance and reliability. The problem is Google sells it six months before they have it up, and they need to turn around and take that paper that they signed to get debt with that credit backing and then turn around so they can actually pay for the PO that they've already issued, you know, for the order that they've already issued. Whereas SpaceX was like, no, no, no, this is running now. Buy it, right? And it's a big discrepancy when you have a balance sheet to do that versus not. And that also helps your revenue per megawatt like be much higher.

• 1:04:15Speaker 02

  Why does the NeoCloud opportunity even exist? Because if you had asked me five years ago, I would have said hyperscalers are going to own this. And you know, you mentioned just now Core Weights has better performance than than the hyperscalers. Like, why does this opportunity exist maybe at the macro level and then in the execution level?

• 1:04:29Speaker 01

  Yeah, so in 2023, I wrote a report that had, uh, Amazon really hate me. It was called Amazon Cloud Crisis. So I talked about how Amazon was the best cloud because they had their Nitro NICS, which offered like tenant isolation. All the hypervisor ran on the NIC and then you could sell all the cores. And they had, you know, custom SSDs that they made and they'd buy the raw NAND and they'd have lower costs because they'd buy the raw NAND and build their own SSDs. And, you know, they had their custom Graviton CPUs and that drove down costs for per core. And so they had all these things that enabled them to sell more cores, have better security, good networking for it. But this was all for the traditional CPUs, better storage, for the traditional, you know, cloud world. But in the AI cloud, a lot of this stuff hurt performance, right?

• 1:05:11Speaker 01

  These Nitro NICS were bad for performance, still are worse performance, although they've caught up a lot because they've had a couple of iterations to like, you know, improve them, but they're still worse for performance. A lot of the security stuff doesn't matter because it's not like I'm time splicing users or splicing a socket into many users, right? It's like, no one rents a single GPU in an 8 GPU server. No one rents a single GPU in a 72 GPU rack. They rent the whole rack and in fact, they write many of the racks. And so, and then, and then there's no like, oh, I rent for six hours and I give it back. It's everyone has these long-term contracts. And so the mechanics of the GPU rental market meant that a lot of the expertise of the hyperscalers fell away. And a lot of the expertise that they did have were actually, some of them were detrimental, right? Network performance for Google and Amazon.

• 1:05:54Speaker 01

  It was, they had custom networks that were better for traditional CPU and for the stuff that they were doing, but actually worse for AI.

• 1:06:02Speaker 01

  And then in other cases, it's like, well, you know, Microsoft would save money by building their own data centers. But their data center teams are actually, were not actually that great. And so when it came time to run, you know, when it was predictable building, it was like, fine. When it came time to like, actually double your forecast for the year, it's like they fell on their face. And they had to go get a bunch of NeoCloud capacity. I think so, performance, I think, you know, I think time to market is another one, right? Neat, you know, these massive organizations, no one's getting rich from building this data center faster. Right? But you look at Crusoe, for example, Chase and, and, and all the other people at the team. You know, I was gonna name some people at the team, but I'd rather not, you know, these, all these people are getting rich if they fucking deliver these, this compute faster.

• 1:06:44Speaker 01

  They're, they're, you know, they're levered, they're hyper levered equity owners.

• 1:06:48Speaker 00

  Hey, look, they're also all coming from Bitcoin and, you know, you're not supposed to say that.

• 1:06:52Speaker 01

  I mean, a lot of the data, like their main data center guy came from Microsoft.

• 1:06:55Speaker 00

  I don't know, I'm just, I'm just teasing, but it's, you know, it's like, you learn, you learn a lot when you're in a very high fluctuation, you know, market.

• 1:07:05Speaker 02

  How much of this do you think was Jensen playing for the chess?

• 1:07:07Speaker 01

  Jensen absolutely hates a world where all the hyperscalers have all the power. There's a reason he's like, blowing money on like, random AI labs that like, I don't even know if like, it makes sense to, but like, you know, he's blowing money and pumping them up and going to, you know, everyone around the world and saying, you should invest in this company. Because he wants to create a multipolar world. That's why he loves Chinese labs, because he wants to create a multipolar world. A world where OpenAI and Tropic and Google models are the only models is one in which he's screwed.

• 1:07:32Speaker 02

  Yep.

• 1:07:32Speaker 01

  Right? A world in which, you know, the hyperscalers are the only ones building compute is one he's screwed in.

• 1:07:39Speaker 02

  Yep.

• 1:07:39Speaker 01

  And so, you know, of course he needs to point the allocation gun at NeoClouds, help backstop their clusters, do anything and everything. Because while today, a GPU sold to Crusoe, and a GPU sold to Core Wave, and a GPU sold to Google, and Amazon are all the same price for him. Five years from now, Crusoe and CoreWeave existing means Google TPU will be weaker and means Amazon Tranium will be weaker. And more inference being done with, you know, non-close source model labs is better for them. So I think, you know, the NeoCloud ecosystem is, you know, it's these people that are Wild West, these Neo Labs as well, a lot of them have investments from Nvidia. It's the Wild West, some will fail, many will fail, but, you know, some will emerge as really great teams.

• 1:08:21Speaker 01

  Whether it be, you know, oddly Crusoe, who's a bunch of crypto guys, who then started building data centers and doing flare gas stuff, or, you know, CoreWeave, who initially was a bunch of New York hedge fund guys.

• 1:08:30Speaker 00

  They were also, and then they were doing Bitcoin mining.

• 1:08:32Speaker 01

  And crypto guys, but then they, they, they like built, you know, there were a lot of people who didn't bubble up like them, started around the same time and just failed, right? So I think, you know.

• 1:08:40Speaker 00

  I gotta say, both those teams are phenomenal. I just want to give a lot of credit and it's like, that's your point, but.

• 1:08:45Speaker 01

  Yeah, I mean, my point is like, he, he, you know, you throw, it's like, you throw a bunch of like bait into the water and the best fish will figure out and survive, right? And sort of the same way with the NeoClouds and, and he hopes the Neo Labs as well. We'll see if any of the Neo Labs really bubble up, but like, you know, Thinking Machines has a few hundred million dollars of ARR, right? That's pretty impressive, even though they've had, you know, in the media, it's like, oh, they've lost all this talent. It's like, well, but Tinker is doing a few hundred million dollars of ARR. Like, that's pretty impressive for out of the gate, a product that's less than six months old or whatever. And, and you, you know, we hope the same happens to other Neo Labs. And, and so, you know, he wants a multipolar world.

• 1:09:19Speaker 00

  Truvi, congratulations on the success. Thank you, thank you. And just the last thing I'll say is, I've seen a little bit of this, I think the public, they can probably tell from listening to you, how hard you work, but like, it's clear you've just been working your ass off for more than a decade and it, you know, led to the last few years of being in the right place, right time. But I think it's unbelievable what you've accomplished and I know it's just the beginning. Thank you so much. So thank you for doing this.

• 1:09:43Speaker 01

  Awesome.