Ep 53: SemiAnalysis Founder Dylan Patel on New AI Regulations, Future of Chinese AI & xAI’s Scrappy Surge to Hyperscale

In this episode of Unsupervised Learning, we sit down with Dylan Patel, Chief Analyst at SemiAnalysis, to break down what these sweeping changes really mean. From how they consolidate power among Big Tech to China's narrowing options for AI dominance, we unpacked the impact of this regulatory shift.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Dylan Patel has become one of the go-to thinkers on all things hardware and AI with his writing over at Semi Analysis. Today on Unsupervised Learning, I had an awesome time chatting with him on a host of different things. It was particularly fun because we talked right after the AI diffusion rule came out. So I got his live reactions to what it means for hyperscalers, what it means for China, the UAE and others, and what the future of geopolitics looks like given these regulations. We also talked a lot about some of the key questions in AI. What will happen with open source models? What are Nvidia's long-term competitive moats? And we also talked about what's required to build really large clusters. He had some amazing stories from XAI's 100,000 GPU cluster. I think folks will really enjoy this conversation. Without further ado, here's Dylan.

• 0:38Jacob Effron?

  Dylan, thanks so much for coming on.

• 0:39Speaker 01

  Yeah, thank you for having me. It's a very cozy, comfortable place.

• 0:42Jacob Effron?

  I can't believe we got you like during AI diffusion rule week. I feel like this must just be madness right now.

• 0:47Speaker 01

  Yeah, I mean, the fun story was it was supposed to be originally Monday. They dropped on Sunday night slash Monday, right? And so I was like, I was up all night working on it and had to cancel. And we shipped it back a few days, so thanks for being flexible.

• 1:02Jacob Effron?

  You shipped a pretty long piece in the interim.

• 1:03Speaker 01

  Yeah, yeah, it was great.

• 1:05Jacob Effron?

  I'm curious if you just give our listeners some like quick context on what's going on on the regulatory side with the AI diffusion rule? And then maybe, given the, you know, goals of what they're trying to do, what kind of grade would you give it?

• 1:14Speaker 01

  Yeah, so I think originally, right, the October 2022 regulations were primarily on the semiconductor industry. But the wording there was like not minced at all, right? It was, we want to regulate AI. We believe it's going to advance rapidly. It's October 22, which is like

• 1:32Jacob Effron?

  They were scaling pilled early.

• 1:34Speaker 01

  Pre-Chatty PT, but they were scaling pilled, right? Exactly. And like, you know, if you get a chance to talk to some of the people like, you know, Ben Buchanan, who's like the special advisor in the White House, like, he's like, you know, no, no, this is exactly why we did it. We knew it was coming. And it's like, oh, so the government is actually quite competent. And it's quite interesting because, you know,

• 1:55Speaker 01

  They they they hit really hard. They have the goal of like the US must be ahead of China on AI, right? Because the next five years of progress, next three years, next two years of progress are going to shape the next, you know, century of like hegemony for the world, right? And so this is like taking an axe to like, try and stop Chinese progress. Um, as far as the grading, right? October 22 was really well intentioned from if you if you have this worldview, right? That that AI is, you know, in the next five years going to transform the world. Um, you know, if you have the view that it's going to take 20 years, then it's a very different like story. And actually, these regulations are actually pretty bad, um, because they're going to limit US competitiveness long term. Uh, but at least over the next five years, they definitely will like

• 2:39Speaker 01

  keep the US further ahead. So October 22 is really well intentioned, but there was like quite a few loopholes. So they did another round in 23, then they did another round in December. Um, and all of these like have like slowly patched over loopholes. There's still some, um, you know, major ones, right? Namely, Chinese companies can get GPUs from, you know, foreign firms, right? So if you've seen the fuss that Oracle's been throwing, um, they've they've just been like ranting about it. Well, it's like, look at who their largest, one of their largest cloud customers is. It's ByteDance, right? Um, or, you know, hey, you can build a data center in Malaysia. Now all of a sudden it's fine, right? And so Malaysia from 2024 to 2027, uh, not the country itself, but companies operating there, mostly Chinese companies, many of them claiming they're now Singaporean companies, right?

• 3:23Speaker 01

  Like the largest operator in China of data centers in China, GDS, moved to Singapore and says they're Singaporean now. It's like, but now they're, these companies are building three gigawatts of data center capacity. To put that in context, at the beginning of 2024, that was roughly Meta's global footprint.

• 3:38Jacob Effron?

  Yeah.

• 3:38Speaker 01

  Right? So it's like, oh, wow. Like the entire world's like, social media is running off of, you know, what Malaysia is putting up in three years.

• 3:46Jacob Effron?

  So what happens to all these like, you know, Malaysian data center builds, given like, you know, uh, obviously a big part of this regulation is is both during countries and then also applying much more regulatory oversight of like who's actually training models within your data centers. Um, obviously, all this planning has gone into to building these things, but now it seems like the rug's kind of been pulled out in some way.

• 4:03Speaker 01

  Yeah. I mean, this is definitely the most far reaching regulation that I've ever seen, right? Um, probably there were big, there were bigger things during the Cold War or World War II. I'm not sure. Um, but like, This is very far reaching in that like it's regulating clouds overseas, right? Um, it's regulating foreign companies. It's it's limiting what they can buy to a huge extent. And so Malaysia has a huge amount of data center capacity. That's gonna be that's being built. Um, we'll see if they continue building it, right? You know, uh, I think most people still continue building it. Um, companies like Microsoft and Oracle have significant assets uh being built in Malaysia. They probably won't stop. Um, but other companies, you know, what what what happens now? You know, at the end of the day, the US is still very limited on data center footprint as well. We're building as fast as we can. There's a lot of regulatory bottlenecks.

• 4:47Speaker 01

  There's there's a number of different um things in that nature. Um, but you know, what what happens as you deploy, you know, American companies deploy AI globally, right? It's like, is OpenAI gonna serve GPT in in in Japan and Taiwan and South Korea? Or like, do they just run it out of Microsoft data centers in Malaysia maybe, right? Like, this could be a reasonable way to like subsume this. But like, the really important thing here and the little sort of, you know, even if you're like very pro America, anti-China, there's still huge negatives to this regulation in that like, a ton of buyers can't buy, you know, that data center capacity in Malaysia actually deploy, right? Like, American companies can't do that, right? And so, you know, Oracle's a good example, right?

• 5:30Speaker 01

  They're one of the parts of the rule is you can only have seven percent of your data center capacity in a in a country in any country that's not like a US ally, right? These tier one countries. And Oracle was planning to have like twenty percent of their capacity in Malaysia. So now it's like, what do I do with these data centers that I was gonna build? I'm obviously an American company. Fine, I won't rent it to Chinese companies, but like, what do I do now? So there is a bit of like, you know, the companies that have so much capacity in the US um are the only ones that can now take over these data centers in in Malaysia and build them up.

• 6:01Jacob Effron?

  Seven percent rule.

• 6:02Speaker 01

  Right, because of the seven percent rule. So like, it's like Microsoft, Meta, Amazon, Google, right? These four companies have, you know, you know, seventy plus percent of their data center, AI data center capacity in the US. Um, and so they can they can like deal and they're gonna they're building super fast. You know, they can deal with like, hey, I'll take I'll take five hundred megawatts. I'll take a gigawatt in Malaysia and that won't break me out of the ruling, right?

• 6:23Speaker 01

  Everyone else is kind of like SOL. So now you've like reduced competition massively in the market, right? And you've created sort of a monopoly. And so this is sort of like, why these regulations while they are like, yes, if your belief is like this whole AI like acceleration and like, you know, you need to combat China. Uh, this is very important. US must maintain hegemony. That's one side of the fence. The other side is like, well, if you want to maintain hegemony, it turns out like, what did we do during World War II? Well, we like, we're like Henry Ford. Here you go. You're going to be absolutely freaking rich because you're going to make all the tanks, right? Or whatever, right? Trucks. Um, this, this is sort of the same thing they're doing with AI is like, here you go, Sacha Diddella. Like you, you get to

• 7:02Speaker 01

  do this in AI, right? Um, and so there's a bit of like antitrust, like, you know, I like decentralized power. You like decentralized power. You like innovation and does this like stamp out innovation in some ways, uh, you know, probably not for American like startups, but for like infrastructure hardware, it definitely is like pretty, pretty bad.

• 7:21Jacob Effron?

  Obviously there's been, you know, like a proliferation of players like CoreWeave and these folks, like how, how do these regulations like, I mean, ultimately, you know, I've heard you say it obviously favors, you know, the, the giant tech companies, but presumably like they can have footprints in these tier one countries as well. Like what happens to the kind of like, you know, the mini cloud ecosystem?

• 7:37Speaker 01

  So I think, I think CoreWeave is rapidly like become quite close to the hyperscale level, right? Like, and so, and then their, their next set of build outs, like large build outs are, you know, continued in the US of course, but then also Europe. Um, I think they could expand overseas for a while and still be fine. They're going to have to hit certain regulatory like, you know, levels for, you know, being able to monitor their customers' workloads and all these, like, you know, just when you throw up.

• 8:02Jacob Effron?

  The more requirements you add, it.

• 8:04Speaker 01

  The less people who can do it.

• 8:05Jacob Effron?

  Yeah.

• 8:06Speaker 01

  Yeah, exactly. The rest of the cloud ecosystem though, right? Excluding CoreWeave like is very much hit hard, right? There's a lot of clouds that are in foreign countries that are trying to build up. There's Sovereign AI, which is like, you know, a lot of startups are start, in, in the Bay are serving Sovereign AI firms in Malaysia, Singapore, and India, in the Middle East, whatever. That's their biggest customer. And it's like, these guys are heavily, heavily impacted by these regulations. Um, there's a lot of, um, you know, cloud companies that are, you know, I guess like tier one is like the hyperscalers. Tier two is like CoreWeave and Oracle. And then like be below that is like, everyone is just kind of like really hit really badly. Um, and so, you know, I think the startup ecosystem is going to have to learn like, what do I do if potentially, you know, my big Middle East customer or my big Singaporean customer or my big.

• 8:51Speaker 01

  Korean customer, whatever, maybe not Korea, because Korea is like allowed tier one. But, you know, one of these countries, sovereign AI countries, are they, am I now regulated? Like, this is actually like potentially bad for startups in some way.

• 9:02Jacob Effron?

  It also is diversity on the foundation model side, right? I guess like on the sovereign AI point, like maybe contextualize before this regulation, what was happening both in kind of the data center build out world, you know, that's government fueled, as well as like the kind of training core models like for, you know, for a specific country or geography.

• 9:18Speaker 01

  It was basically the Wild West, right? The US had an executive order a while back. It was basically just like if you're building a model that's 2026 flops, which is like very large, that's like twice as large in terms of flops as Lama 405B, right? So I presumably like very few people have done that yet, but it's like a notification requirement. That's it. But it's in the US, right? So if you're in China and you're doing this, or if you're anywhere else, it's fine. Of course, there were not really data center build outs of the scale to hit that quite yet, but they were coming next year and the year after, right? Or 2025, 2026, these build outs were coming. And so, you know, there was not much regulating foundation models, regulating other countries from building foundation models. Like, obviously, like OpenAI and other companies were like told like, hey,

• 10:00Speaker 01

  I know the UAE is telling you they're going to build you a massive cluster. You're not allowed to do that, right? Like, there's obviously things like that tacitly under the table happening, but there were no like, here are the regulations. So, you know, if you were a company in China, you would, you know, do whatever you want. And if you're a company elsewhere, you do whatever you want. And you could rent GPTs from wherever you want, and you could build whatever you want. And this like smacks it down, right? You can't like export model weights, right? Out of outside the US slash outside of like trusted clouds, which is again the hyperscalers, right? Of of foundation models that are really the like biggest big ones, right?

• 10:36Speaker 01

  You know, like think Lama 4 and GPT 5 and that kind of stuff. Or you can't like, there's like various things around like. Protecting against synthetic data generation, which all the Chinese companies are doing a ton of synthetic data generation of G P T four. So their models can be good, right? Like this is part of the like dirty secret about why like, yes, Alibaba and Deep Secret really good, but they're also just generating a bunch of data from G P T four and using that to like post train, right? Partially, um, there's a lot of um, regulations around, um, yeah, accessing clouds. Um, and so a lot of company, a lot of clouds, you know, they didn't have a core, you know, you know, like when you look at like the big tech companies, they've sort of all pared off, right? Um, you know, Metas themselves, Googles themselves, but then Microsoft, OpenAI, Amazon, Anthropic,

• 11:21Speaker 01

  um, Elon's world is is X A I Tesla, you know, there's the lines are blurry, right? X, the lines are very blurry between these companies. Um, but like, these are sort of the hyperscalers in the U S. But in China, um, Alibaba was doing good at A I. Um, and they were building out, but besides that, like, there's a bunch of new players. There's Moonshot, there's Deep Seek, and there's all these other players that are popping up. But they don't have a hyperscale partner, um, uh, uh, ByteDance, of course, right? And ByteDance is, they do run a lot of their infrastructure, but they, a lot of it they rent. Um, you know, what do they do? Well, they decided, well, let me just like, not by, not, uh, Deep Seek, but like ByteDance is just like, oh, I'll just rent from any GPU cloud out there, right? So there's random clouds in like, the, in Europe, there's random clouds all over Asia, right? And it's like, their business case is like, you're gonna make

• 12:06Speaker 01

  a decent profit, right? And you're gonna build GPUs and rent them to me, right? And I'm not, you're not gonna have any observability to my workload. I'm just gonna do whatever I want. Uh, most of it is just serving TikTok, right? Let's be clear. It's not actually like illicit, but you know, the potential is there and they're obviously working on language models and generative video models. And there's always the like fear that they're gonna manipulate our teenagers' minds and, you know, destroy us all.

• 12:26Jacob Effron?

  Not after Sunday, I guess.

• 12:27Speaker 01

  I mean, they're still gonna be able to do the latter thing, right? But but they're heavily, heavily, heavily limited on, you know, the size of clusters they can get now. And and who and what and the notification requirements and like, the companies have to be able to be observe their workload, which is like bad security. So it's, it's a really, really big blow to, of course, ByteDance and many Chinese A I players, but it's also a huge blow to all these random cloud companies whose business case was like. I'm gonna sell to ByteDance, right? Or I'm gonna sell to this Chinese company.

• 12:55Jacob Effron?

  So where does Chinese AI go from here? And like, what, you know, do you see any kind of like, if they're looking around for loopholes? Like, what, you know, what is the path forward?

• 13:02Speaker 01

  There is, there is one obvious loophole, which is, um, well, there is like strict caps, right? Like, each country can only buy 50,000 GPUs for the next four years. And it's like, that's kind of nothing when Nvidia is making, you know, 6 million plus this year. Right? Um, it's like, huh? Um, so, so each country can only buy 50,000 GPUs, but then there's still loopholes, which is, if you buy 1,700 or less, that doesn't count to the 50K. Um, so, so the obvious thing is you just spin up a bajillion, you know, shell companies and you just buy, you buy 17, you buy 1,600 GPUs and, and then you route them to China or whatever, right? So there are still some loopholes. Um, but generally, you know, this is, this is much harder to do.

• 13:42Speaker 01

  I think, I think more, more so is just like, you know, China has a ton of amazing engineers and they will have to innovate, right? Like DeepSeek is like amazing at engineering, right? So it's like, um, you know, they're, they're crushing companies with similar levels of compute, right? Um, because they're just engineering better, but they're not quite like OpenAI Anthropic level. Now they need to take this like same like compute for collabs continues to scale like crazy and theirs cannot scale nearly as fast. So it's like, we not only have to be better at engineering, we have to be like way, way, way, way better at engineering.

• 14:12Jacob Effron?

  With this new push to like test time compute, like, is that feasible, uh, with, with like a huge compute deficiency?

• 14:18Speaker 01

  Yeah, so, so I think there's, um, some interesting aspects of test time compute, right? Um, it is way more compute intensive than people think. Like, people are like, oh, training is dead. And it's like, no, building a test time compute capable model requires tons of, of training.

• 14:34Jacob Effron?

  You've written some really interesting data about this.

• 14:36Speaker 01

  It's not just like pre-training, it's post-training, right? So you have to generate a ton of data, you have to throw most of it away. Um, you have to verify, verify the data to make sure it's like actually accurate, right? Um, the chain of thought or the reasoning chain. Um, and then you have to train the model, right? And you have all these reward models, it's like very complicated push training stuff, so, um, you know, and, and it's very compute intensive, actually. Um, so. You can make reasoning models today, right? Like O1 and like, you know, DeepSeek's R1, I think it is. And Alibaba's a reasoning model. I can't remember the name off the top of my head, but these, these models like are like, the way you should think of it is like, you know, scaling laws are log log, right? So you can scale, you can scale data, you can scale parameters, you can scale.

• 15:19Speaker 01

  And, and, and, and it's like, well, both of those are sort of like petering out, right? Because data's not scaling fast enough and, you know, just infinitely growing parameters doesn't give you anything, right? Or it gives you, doesn't, it gives you diminishing returns. Um, but we've scaled up this rung many times, right? The training runs that these frontier labs are doing are billions of dollars, right? And, uh, next year, probably tens of billions of dollars, right? It's like scaling very fast. You can only get like a logarithmic improvement on that front every so often, right? Whereas test time compute, we're like at the bottom rung of the ladder. Right. And it's like, oh, I can like climb up this ladder really rapidly because, oh, right now we're spending hundreds of thousands, millions, you know, tens of millions, hundreds of millions, billions, right? And, and like the compute deficiency doesn't like matter until you get to the billions plus scale. Yeah.

• 16:03Speaker 01

  Right? Um, and so there are many rungs that they can, you know, out engineer with the limited compute resources. And, and sort of catch up. Uh, but there's also the flip side of like, well, Anthropic, OpenAI, DeepMind, et cetera, right? Uh, XL, these guys are, and, and, and many startups in fact, right? Like Mira startup and so on and so forth, they're all scaling up this ladder really quickly as well, as fast as they can. Uh, and so it's, it's, it's interesting to see what's going to happen. I, I think that, you know, that's on the training side and on the inference side. You know, it's like, uh, if you actually like do a query of like O1 versus GPT-4O. It's like, yeah, it costs like 20 cents for a query and it costs like $6 for a query, right? It's like, it's like a, a dramatic difference.

• 16:47Speaker 01

  Now, obviously the quality is better and so that like, and that $6 is still way cheaper than like paying, you know, someone who makes way more than $6 for that work. Um, but it is like, You know, and it's scalable, right? Whereas like, human resources are less scalable. You know, it's kind of like, this is, this is, this is a huge difference in cost. And that's six dollars is all inference. And it's like, well, you're limited on GPs you get into China. You're limited to 50,000 in all these countries that are friendly to China, or even neutral to China. Right? So it's like, it's gonna be very hard to get that scaling of compute on inference, right? And, you know, you can throw as much money as you want on training, but to actually like, change the world, right? Again, like, if you believe AI is gonna change the world, then you have to spend a ton on inference, right?

• 17:32Speaker 01

  Even if the margins are low, right? You know, even though like Anthropic's and OpenAI's gross margins on inference are like 75%. A lot of these other companies' gross margins on inference are like 10, 20%? Even if it's 10, 20% gross margins, you know, to amortize the cost of like a $5 billion model, right? Like, okay, you need to do like $10 billion of inference revenue. And it's like, okay, well, then that's like $8 billion of hardware cost, which, you know, and that hardware cost of $8 billion is like actually depreciated over multiple years. So in that year, you bought like $30 billion of hardware. Congrats, right? It's like, this, and that's like way too much, right? Like, that's way, that's like, exceeds the caps dramatically. So I think, you know, test time compute, they'll be able to like, deliver impressive results probably. The question is if they'll be able to scale it and sell it.

• 18:18Speaker 01

  And, and so that's like the, the, the issue. And like, on the flip side, it's also like, China, the cost of developing software in China is like, way cheaper than in the US, right? They just have way more talented engineers, uh, software engineers. And, and so, you know, their development practices and all these things are different as well. But like, every company there, you know, there's a lot of like, platform SaaS doesn't work in China, right? This is like, pretty well understood. It's like, like, and like, in the VC world, it's been like amazing for VCs is that platform SaaS type stuff has been awesome. Because you can sell to all these companies, because develop it, you're, you're developing it in one place and you're selling it to everyone, right? Whereas in China, it's like, okay, you're developing in one place, but everyone can just develop it internally. It won't be maybe as good, it won't be best in class, but whatever, devs are cheap, right?

• 19:00Jacob Effron?

  Which kind of seems like the future we're headed to with all this AI coding progress.

• 19:03Speaker 01

  This is what I'd be curious about your take, yeah.

• 19:05Jacob Effron?

  No, I mean, I definitely think it's. It's a huge risk for these SaaS providers, right? You know, the ability, I mean, obviously Klarna and folks have talked about it very publicly, but I think the kind of moat that was spinning up these applications in the past is certainly a lot weaker now. And I think like the ability also, you know, that combined with the ability to more easily migrate data out of whatever system of record you're in. And basically, all the like painful stuff that probably stopped someone from ever really upgrading their software in the past is now just like a relatively simple thing to do with a bunch of like AI agents. You throw it at, go away for like a few days and there it is. Yeah, it's a real threat to SaaS going forward. But a lot of countries have, you know, I think clearly from the beginning have realized the importance of having this, you know, this expertise and models in their own country. But if anything, this, I feel like this regulation makes incredibly clear, you know.

• 19:47Speaker 01

  Yeah, yeah, I mean, like.

• 19:48Jacob Effron?

  You could at any point be cut off.

• 19:49Speaker 01

  I think like if you look at UAE, this is exactly what's happened. And I think it's been pretty well reported in the media, like at one point, G42, which is the UAE sort of champion of sovereign AI funded by, you know, all sorts of like, you know, UAE money and it is based in UAE and they, they, they were at one point like threatened of being cut off and they were like, the CEO was like a Chinese guy from China and it was like CCP links, potentially blah, blah, blah, all this stuff. And then like, you know, goes like all the media reporting on this goes silent for like six months. And then the next, you know, thing to come out from like, whether it was Bloomberg or Reuters or Wall Street Journal, whatever, was like, yeah, G42 and Microsoft are going to do a partnership. And it's like, huh, I wonder like what happened here?

• 20:32Speaker 01

  Like, is it, is it purely organic or was there like some sort of like tacit like, well, if you're just going to run your, all your own infrastructure, you know, maybe there's something bad here, but you know, now that we know Microsoft has partnered with you, you know, it's good. And like, you know, now when you look at it, it's like G42 and the US government are actually like talking all the time and they're like, like G42 will be a big cloud player. And they're deploying GPUs in the US even, not just in, you know, the UAE, but in Europe and the US.

• 20:57Jacob Effron?

  So it's like. Through these hyperscalers, I guess you can enforce the American sphere of influence of like what, you know, both obviously, you know, oversight of what, you know, models are out there and but also like, you know, ultimately like who gets out, you know, I think right now there's what some human rights terms involved in like the tiering of the companies, you could have the countries. And you could imagine like all sorts of ways that administrations use this stuff going forward.

• 21:16Speaker 01

  Yeah, I mean, I mean, like the Biden, the, the regulation itself does mention human rights. Obviously, the new administration is coming in. If you had me guess, I'd say like 95% probability that Trump administration just keeps these rules and maybe like, like obviously tweaks them to some extent. But the human rights thing, they're probably going to throw out and like, you know, and switch for like. You know, American economic sphere of influence, or like industry, or like whatever, whatever the priority of like, you know, purchasing energy and weapons from the US government, right? Like, these are the sort of things that might matter more for the Trump administration versus like, human rights policies might matter more for the Biden administration. But at the end of the day, yeah, it's like a weapon for sort of the arsenal of democracy, you know, this is like a regulation. So I don't know if it's like.

• 21:56Jacob Effron?

  But I mean, take a step back, I feel like you've talked before about, and I think it was maybe in the context of the chip bands, about this like, Goldilocks that you're trying to strike of like, preventing China from training their own cutting edge models, but not making it like, so barren that like, everything gets built up within the country itself. And I wonder like, you know, back to the, I guess, the grade question, like, does it feel like this, you know, hard handed approach, like, is that the right way to further these policy objectives?

• 22:18Speaker 01

  So I think, um, October 22, and then the 2023 regulations, they were, they were more like, you know, we're gonna try and limit you, um, but they didn't like, fully limit like, China building their own domestic supply chain and all these things, right? Um, because the intention is, you can't build your own domestic supply chains, and you can't access it, but you can access it at like, some tier, right? Now, like, with these most recent regulations, it's like, no access at all to this stuff, right? Like, you have, you have order of magnitude less compute than like, any of the US like, labs, right? Um, that's what we're trying to enforce on you. Um, in which case, like, obviously, there's gonna be like, some leakage, and so maybe it's not a whole order of magnitude, maybe it's like, you know, 75% less compute, whatever, right? It's still a humongous amount.

• 22:58Speaker 01

  So, now it's like, China only has two options, which is like, build your own supply chain, or, you know, and then if we limit that effectively, right? Like, so like, Ben Thompson has this take, and he's like, yeah, if you, if the regulatory regulations are so effective on preventing China from building their own semiconductor industry, um, or, you know, sort of like, or they like, build their own semiconductor industry, and it's like, competitive, then like, action on Taiwan is much more likely, right? Whereas like, if they feel that they're still benefiting from Taiwan, right? Um, they feel that they still can build it up and get access through Western sphere, right? This is like, the sort of like, geopolitical stuff I have no idea about, by the way, right? Uh, talking out my butt, but like, there's sort of the like, you know, the Goldilocks zone of like, yeah, you don't want to ban

• 23:40Speaker 01

  them too hard, you don't want to let them just have everything, because then that like, induces risk, right? But if there's like, that Goldilocks zone of like, You just can't, right? Like, you're kind of like, you're behind, but not so far behind. There's hope of catching up, then it's maybe fine.

• 23:53Jacob Effron?

  Well, I guess switching gears to, you know, the cluster buildouts that we're seeing today. Maybe just give some context to our listeners on, you know, obviously, you're the guru of tracking all this. Like, the size of cluster buildouts we have today, those being planned, and maybe just talk a little bit about kind of the biggest blockers that we have to larger clusters today in these tier one countries.

• 24:09Speaker 01

  Yeah, so I mean, when you take a step through sort of like G P D four, twenty twenty, you know, twenty twenty two, sorry. Roughly twenty, you know, few thousand G P U's, A one hundreds, right? And they train this model, change the world, right? Now, what's the next sort of step up, right? You know, you got to look at like from two to three to four, right? There's, you know, a couple orders of magnitude of compute increases. Now, you know, with G P T sort of five, quote unquote, or Orion, or whatever's coming out, they're, you know, working on it. This is a scale of, you know, hey, well, we can't necessarily go for, you know, order magnitude more G P U's because it's just too expensive and there's a lot of difficulties with building that out. So we go up, you know, we go from twenty thousand G P U's to a hundred thousand, right? And so X A I's built a hundred thousand, Meta's built a hundred thousand.

• 24:54Speaker 01

  Both of them have actually been building more than that, right? In one, one spot to do training. Open A I's plans are also in Anthropic, you know, so on and so forth. Anthropic has four hundred thousand Tranium server G P U's coming up this year. Or Tranium chips, not G P U's, but like, there, there, people are in the hundreds of thousands range for this year. But, you know, sort of what's built today and what's training models today is on that order of magnitude of a hundred thousand, right? And the G P U that's being used is the H one hundred. So you got, you have five X the G P U's and you have three X the performance per G P U roughly. So then you're at like fifteen X more compute. And we'll see what comes of this, right? Like, you know, the, the thing is like, you don't build a cluster or run a model and immediately get like, you know, something great out of it, right?

• 25:37Speaker 01

  You have to like do tons of experimentation and, you know, training takes months and post-training takes months and safety takes months, right? It's not like snap of the finger, you get the model right now that you've built the cluster. And so there is a bit of a lag, right? And so, you know, this year we'll see models come out that are from that sort of quote unquote, last generation cluster. But the next generation one is being built now. And what's this next generation one is like hundreds of thousands, right? And for context, right, like. You know, each GPU, right? Once you include networking, building, all this sort of stuff, maybe is, you know, the GPU itself of H100 is like 24,000, but once you add everything else up, it's like 40, 45,000 dollars per GPU all in of everything, right? And so the cost of like 100,000 is like, like a lot, right?

• 26:22Speaker 01

  It's like $5 billion for 100,000 GPUs, right? Um, and so Elon's round was 6 billion and, uh, you know, he built 100,000 GPU cluster, right? Like, you know, obviously there's like other costs, right? They had to buy data, they had to do like, um, you know, employees and they had to do like, they had to convert a factory into a data center, right? And like, set up all this crazy shit that they did, right? Which is like, you know, talk about that in a second, but like, there's a lot of stuff that needs to be done to get, um, you know, a model trained. And now, now that Elon's built the cluster, it doesn't mean a model's gonna come out, you know? I'd be surprised if they have a leading model even this quarter, right? Um, it's gonna take some time. And there, and, and so like, what is, what is, you know, that's on the training front. Um, and, and these next generation clusters, they didn't be built this year, not $5 billion, they're, they're, you know, 15, right?

• 27:07Speaker 01

  You know, and obviously you can't scale the money up, you know, like insane amounts, but 3X, right? And, and obviously-

• 27:12Jacob Effron?

  And the cost of that Anthropic raising is like $2 billion that, you know, in, in this current VC round seems perplexing.

• 27:18Speaker 01

  Well, so what's nice is, you know, Anthropic, OpenAI, Anthropic especially can, you know, and, and OpenAI and, and, and, you know, some of these companies that are the startups, right? Uh, can, can raise a round and that money can only be enough for maybe a year, year and a half GPU rental, right? And that's a rental.

• 27:35Jacob Effron?

  Right.

• 27:35Speaker 01

  They're not the one buying the GPUs, right? So Amazon might be the one spending, you know, 5, $10 billion, but they might for one year rent it for 3, 4, right? Um, and you know, when you look at the revenue, the credits they get from Amazon, plus the round they raise, right? That's enough to pay for something like that, right? Now, the, the, and, and the Tranium server is not $10 billion because it's like a cheaper internal chip for Amazon, but like, you kind of get this like benefit if your cloud partner really loves you, right? And in the case of like OpenAI and Microsoft- Kinda, but they were able to make a deal, strike a deal with Oracle, right? Oracle spending like $10 billion plus for them this year to build out data centers and GPUs, right? Um, for OpenAI, uh, not Microsoft, interestingly enough.

• 28:18Speaker 01

  Uh, Microsoft is doing stuff as well, but like, but they're not, they're not charging $10 billion this year, right? They're charging a few billion dollars this year. So there is, there is that benefit, right? That you don't need to necessarily, you might have signed the dotted line and like Oracle may like have taken the credit risk, right of like, well, or, well, OpenAI and this contract exists in three years. I mean, you and I believe yes, but like, this is something that you have to consider as a possibility, no. Um, and so there is that risk. Um, and like the numbers that like companies are raising are not enough to pay for the whole cluster. But that just means, you know, Anthropic is going to raise even more. Of course. Nine months from now or a year from now, whenever they release their reasoning model, you know? It's a constant fundraising thing, right? That's like Sam's strength is like he's a god tier fundraiser, right? Um, and so that's like one aspect of it.

• 29:03Speaker 01

  The other aspect is like, when you look at the hyperscalers, right? Meta, Google, uh, Amazon, um, XAI, you know, all these guys are building their own data centers. They're the ones delivering the CapEx. It's really hard to get the electrical infrastructure, uh, the substation, build it all out, get it to work, uh, get all the chips to be networked properly, uh, deal with all the failed chips. Because when you're talking about, you know, this number of chips, actually quite a few were failed. A lot of them are silent failures. There's a lot of difficulties in getting a cluster to work. Um, and so like, you know, the big, big data center guys have like, sort of like known this, right? The hyperscalers, XAI kinda came in and was like, Elon's like, well, we're just gonna do it. And it's like, well, we can't find a data center anywhere, cause all of them are already taken. Right. Right, uh, for, for the timeframes we need. So what do we do, right?

• 29:47Speaker 01

  And, and like, what they did is like, just like the most like Elon awesome thing ever, right? Which is like, they found a random appliance factor, closed appliance factory in Memphis, Tennessee, right? Located next to like a giant power plant, a water treat, treatment facility, like a garbage dump, like, like, just like, literally like what most people in SF would be like, this is the worst place on fucking earth, right? Like, not, not really, but like, um, you know, you talk to people at XAI that actually go to the data center, like this place kinda sucks, right? Memphis has really good wings, by the way.

• 30:18Jacob Effron?

  Great food across the board.

• 30:19Speaker 01

  Yeah, great food in Memphis, but like wings, barbecue, etc. But like, you know, it's like, okay, we bought this appliance factory, but why did we buy? Well, one, there's a gigawatt, um, natural gas plant right next door, right? Two, there's a natural gas line, a main that they tapped and they set up their own generation capacity on site. You know, three, they're upgrading the substation, take more power from the grid, right? So they're like doing, they set up their own like fricking mobile generators to like create power, setting up Tesla battery packs. Uh, they're now planning, they've filed permits to build their own fucking power plant, right? Like, uh, on site, not just like mobile generators, but like a massive natural gas combine, you know? So they can do a million GPUs is what he claims, right? Or what he says he's gonna do, which, you know, we'll see.

• 31:04Speaker 01

  Um, I won't bet against Elon, right? There's, he's doing all these like things to like, get the data center now, right? Because there's no power, you know, the substation wasn't ready, right? The power's right there, but the substation was all, was not, was too small and it was being upgraded too slowly because, you know, US power grid upgrades take a long time. It'll take at least six months a year. They had already thought about it, but it was only like six months a year. Like, okay, we'll just generate power on the site, right? So they tapped the natural gas line that's next door and they set up all these mobile generators, right? Oh, the power from these generators is pretty dirty and GPU training is like, you know, the moment I stop training, power just goes to zero. Or, you know, maybe if I'm doing a gradient update, now all my GPUs go idle and like they're exchanging weights. And it's like, oh shit, power went to zero. Then it's like, oh shit, power went to a hundred megawatts. And like this sort of stuff can blow up grids.

• 31:47Speaker 01

  Um, and so, you know, Elon's like, oh, they will just throw a bunch of Tesla battery packs. There's doing all sorts of crazy things. Oh, how do we, you know, how do we cool this, this facility? Because it wasn't ready to be able to cool this much amount of stuff. Oh, well, water cool everything and then we'll rent a bunch of chillers, water chillers, like many of them restaurant grade, like fricking container things and just place them outside and like, cool. Right? It's like, they're doing all this like crazy stuff and it's like way more complicated than this, right? And it works, right? It's like, it's like very impressive, um, how they deal with all the problems. Uh, I guess one more funny story is like this power problem I mentioned. Meta, they accidentally open sourced this code. Um, it's literally called power plant, no blow up. It's a flack. And so when you're doing the gradient update, right? When you're like exchanging weights,

• 32:30Speaker 01

  instead of like having the GPUs go idle, right? Basically, when you're when you're sending all the weights in the network, instead, they just have them do fake matrix multiplication.

• 32:37Jacob Effron?

  Just so it's not similar.

• 32:38Speaker 01

  So the power stays stable, right? Because otherwise, like, the power plant might blow up if you go like this. And it's like, what the hell? It's like, this is like the crazy stuff that's happening. It's like, you know, there's so many complicated things going on in these, you know, $10 billion build outs that people are doing.

• 32:53Jacob Effron?

  Fascinating. If you were running like the government right now, I mean, what like, obviously, it seems like the energy side is like a massive policy and just like bureaucratic blocker to more of these data centers happening. Like, what's the best thing that government can be doing to unblock this stuff? And, you know, is every future data center build out like this going to have to pack together Elon style, you know?

• 33:09Speaker 01

  I mean, I mean, like, to be clear, right? Like, the like number of like normally built, conventionally built data centers is like this. It's just like what we want is like straight line up, right? Like, you know? And so, you know, for example, gas generators are sold out for two, four years from GE, right? Like, it's like, oh, wow. Vernova, I guess, now, right? Substation equipment, same thing, right? It's like sold out for four years. So like, now you have to get like creative, right? And so like, even though it's like growing really rapidly, so like, some of the stuff is like held up by like, I think in one part of like the Midwest, it might be Ohio or Indiana, the power, the cost, like, if you build a new power plant, to transport that power on the grid to a customer costs more than the cost of actually generating the power.

• 33:53Speaker 01

  It's like, what the flip, right? Like, it's like, so like, the grid needs huge investments, right? There needs to be like a removal of a lot of the environmentally, environmental regulations.

• 34:04Speaker 01

  But also, to an extent, there needs to be like more vibe shift of like ESG type stuff, right? It's like, what is like, is, is like, and this is like the attitude of like people at AI Labs. I don't know if it's actually true or like, it's the correct one, but like, maybe you just say, screw it, to build AGI faster, we do it with natural gas. And then, because we're doing it with natural gas, and then AGI will like, create enough economic wealth and prosperity that we can just do carbon sequestration and it will all be good, right? It's like, this is a pretty like wild take, but this is like, this is what some people believe. And to some extent, Meta has kind of a...

• 34:39Speaker 01

  You know, either for either, you know, they laid off their whole like DEI thing, they they uncensored politics. There's been a massive vibe shift. One of the things they're doing is like they're setting up two gigawatts. Right? I told you earlier, four, sorry, three was like their total global capacity beginning of 24. They're setting up two gigawatts in Louisiana alone, basically all powered by natural gas. Um, and it's like, yeah, now they can get in two years, right? And they've like figured out how to get the substations and all that, whatever. They've they've gotten all the supply chain stuff and it's like, yeah, if we just throw ESG out the window can go faster. Right? And and and like, you're doing it in states that don't have as much environmental stuff. Texas and Louisiana and and so on and so forth, right? Many other states, uh, you know, all over the country, but it's like, you know, that's part of it, right?

• 35:21Speaker 01

  Now to this to this moment, right? Like, Google and Amazon are still quite like committed to their green pledges. Microsoft is sort of in the middle and Meta and XAI obviously don't give a fuck, right? So it's like you or or they don't, not that they don't give a fuck, right? Like, obviously, like XAI, like Tesla does way more for green than what they're destroying, right? So it's like all sorts of like, you know, it's complicated, but like, is there a way to like, change the vibe on those two? And therefore they move faster, right? Or is there a way to like, um, make it so you don't have to do crazy stuff? Like grids are upgraded, power generation is is, you know, like onsite power generation, like all these different things. Like, um, help build solar and wind and, uh, batteries faster and cheaper in addition to doing all the gas, right? Um, there's a lot of complicated things to do, um, on the energy side.

• 36:06Speaker 01

  And there's ways to still make it green and fast, right? Like, you could just throw up a lot of solar, you could throw up a lot of wind. And, you know, the the power generation of like renewables are like this, but wind and solar do not, don't actually like correlate that well. So then you're like, most of the times you have power, obviously you have excess power sometimes. Um, but most of the times you have power and then you throw on like a small amount of battery. And now it's like almost all the time you have power, but there's still like times where the sun isn't shining. And the wind isn't blowing for a long enough period that your batteries are drained. So now you have some backup gas, right? And it's like, these are ways to like, make it greener and still do it fast. But yeah, I think I think like grid, um, and industrial policy around that is like the biggest like thing that needs to be done to accelerate this.

• 36:46Jacob Effron?

  Kind of like the domestic version of, uh, now now that we've done the, uh, the international regulations. It's the next next area of focus.

• 36:52Speaker 01

  Yeah, you you would think, right? It's like, uh, like I I myself have like far more of like an attitude of like, we should win harder rather than we should like try to make them lose.

• 37:03Jacob Effron?

  Yeah.

• 37:03Speaker 01

  Right? Like, that's sort of like. I think like most people think that's a better ideal and like what America stands for more, but you know, we kind of like, yeah.

• 37:12Jacob Effron?

  So we have these hundred thousand GPU clusters, obviously now where we're going to see the output of these models at some point in 2025. And I assume that will obviously impact a ton of the, you know, future investments based on what we see there. But obviously people have to invest way ahead of that. And so we already kind of know what we're getting in for, uh, for, for build outs in this next year. As you kind of think, you know, uh, just kind of putting your betting hat on for like two years from now, five years from now, like how big clusters do you think we'll have?

• 37:36Speaker 01

  Two years from now?

• 37:37Jacob Effron?

  Yeah.

• 37:38Speaker 01

  Um, I would, I would imagine it'd be like, you know, like there's a couple of things, right? Like, you know, you could say it by number of chips, but it's not fair, right? A100s were like 400 watts. H100s are 700 watts. Blackwell is 1200 watts, right? You know, sort of the power per chip is scaling rapidly. So saying the number of chips is not fair.

• 37:54Jacob Effron?

  I guess energy.

• 37:55Speaker 01

  Right? So energy is like a good metric because it's like how much useful work you get per energy is like growing rapidly, right? But then like the amount of energy you're devoting to this, right? So, so 150 megawatts is these 100K clusters, right? We're seeing like, you know, 400, 500 megawatts, uh, cluster sort of being built out now, right? Like, you know, and, and then, you know, in 2026, I'd imagine it'll be like gigawatt scale clusters, right? Or that's what it looks like based on, or, you know, one gigawatt in 2026 ish. Uh, Metas, Metas like trying to do like two gigawatts by early to mid 27. So it's like, and others are all like sort of doing similar stuff. Um, in different parts of the nation, it's like these build outs are, are rapidly like sort of escalating to like

• 38:38Speaker 01

  earth shattering amounts of like power in one spot, right? So, you know, and then, and then like, obviously the like dollars per chip and then the chips power, all this, these are variables that change over time. But yeah, the, the, the build outs are, are scaling, right? When you go from like 20,000 GPUs, 400 watts each, you know, that was, you know, that was like on the order of like, you know, 20 megawatts to two gigawatts from 2022 to 2026, 27. It's like, holy crap. You just scaled that much power.

• 39:08Jacob Effron?

  Totally.

• 39:09Speaker 01

  In, you know, two orders of magnitude in, in like five years.

• 39:13Jacob Effron?

  So if we're sitting here at the end of 27, how, how much money is OpenAI raised?

• 39:18Speaker 01

  I I think I think it'd be, you know, really difficult to say, you know, I I really do think it would be like north of a hundred billion, right? Like, that's that's like. Yeah, like, like, uh, it's it's it's pretty clear that they've been trying to convince, like, um, all these like sovereigns to invest and like they've been like passive on getting them, but like. I think like, you know, you're gonna see O three release, you'll probably see another reasoning model release, you might see G P five release. It's like, how are they not gonna raise a ton, ton more money, a ton higher valuation? Revenue skyrockets, right? Like, it's like, pretty, I think I think that's what's gonna happen. Um. And I think the similar stuff, obviously, OpenAI will lead the pack, but like others will also get, you know, similar trajectories happening, like Anthropic, X, A, I.

• 40:02Speaker 01

  You know, I think I think like, yeah, the the amount of capital that will be deployed is gonna be insane.

• 40:06Jacob Effron?

  You know, on the model side, obviously, we've seen this massive scaling of test time compute, you know, it seems to work really well with easily verified data, you know, coding, math. I think there's this big question of like, what other tasks will fall into these test time compute models? What is your thinking on that?

• 40:17Speaker 01

  So I think I think like, I really like, like, when opening, I released the like five levels thing, I thought a lot of people were like puzzled, like, what the fuck is? You know, it's like reasoning and then and then agents, like, because everyone's been, you know, they start up because it's like. Chatbots, agents, right? Um, and I think that it's like, I think it makes a lot of sense because test time compute, like. You're you're like verifying your outputs and you're making sure they're accurate, right? So I think I think like, and these these like test time compute methods, right, of synthetic data generation? And then throwing away a lot of it and verifying it and using it, right? This is where other modalities can come in, right? So like, computer use. Is like kind of predicated on all these synthetic data generation pipelines and test time compute working.

• 41:00Speaker 01

  Agents are all predicated on like the reliability of a model getting high enough to where you can actually like. You know, get the answer reliably and chain multiple together, right? And it's like, these different paradigms, I think, will stack together, right? Like, it's like, you can't do computer use without test time compute. You can't do. Um, agents without test time compute. And, uh, when you combine test time, you know, computer use agents, it's like, oh, wow. You know, it's like this now, it's like autonomously working on the computer. Without even like, you know, having to like manually code in all the APIs that's accessing all this, right? It's like, I think like what tasks it can do will like bran- branch out drastically. Once you get the core of like reasonings, like sort of not solved, but like at a really strong point.

• 41:43Speaker 01

  Um, and the pipelines for doing all this stuff, right? Um, and so I think that's probably now like, what kind of use cases that is is like, I think it's like whatever, you know, SaaS business model, of course, like coding is great, but it's like, what about software engineering, right? Because coding is very different than software engineering. Um, you know, it's like, it's like customer service, like great chatbots do that kind of, but like not great, right? Like, oh, searching through documentation, right? Like rag is okay, but like actually you need to like rag is really bad because there's no way to verify that you properly pulled in the right stuff, right? Your retrieval actually worked. Um, you know, people can be like, oh, our retrieval is really accurate, but it's still like not accurate enough for an agent. Uh, and so like, you know, when you stack these things together, I think like any, and like, it's really any like information task. I, I don't see why it wouldn't work now.

• 42:28Speaker 01

  Obviously what, what gets done in 25, 26 is very different than what might get done in 2030. Um, I, I don't know if I'm like the best to like, say like what specific use case though, right? I think I can just see sort of the writing in the walls like clear.

• 42:41Jacob Effron?

  What about like the open source side? I mean, obviously open source models have been relatively quick to catch up to, you know, the through GPT four O type level models. I think there is a big question on, on, you know, with these test time compute models, even in the regulations themselves, right? I mean, I think the, the way they framed it is like, you know, open source models all fair game. And there's much more, you know, whatever the open source models capabilities are is up till then is fine. And then basically it kind of almost assumes there's going to be some massive gap between, uh, between closed and open models that just increases over the next few years. Is that how you see things too?

• 43:10Speaker 01

  I think, I think so. I agree, right? Like, I think Meta will release Lama four and it'll be better than GPT four, right? Um, you know, obviously like, you know, you can say open source models have closed the gap, but it's like, does anyone have a voice mode? Right? Does anyone have like, um, quite the level of inference cost that Anthropic and OpenAI do? No, not really, because their models are, are really efficient, right? Um, a four O is like much smaller than Lama four or five B in terms of active parameters, right? Like way smaller. So it's like, you know, there is still like open models are, are behind. Uh, maybe not in capabilities entirely, but in some ways, I think what actually like, um, ends up happening is that like.

• 43:49Speaker 01

  Reasoning models are not necessarily going to be fully open sourced. Even like, you know, like you see the DeepSeek model, you see OpenAI's model, right? Meta will release better models as long as they think it makes sense to, but will they only release chat and completion models? Or, you know, internally, they have their own code model that like is trained on their own code database in addition to all the stuff in Llama. And it's like, well, that's not being released and that one's far more capable, right? At least at coding, right? And it's like, um, so there's like things like this that like, I think it might point to like, okay, Meta's not going to just open source everything.

• 44:23Jacob Effron?

  Right.

• 44:23Speaker 01

  Right? I don't think that necessarily like means that like you're putting together the reasoning system, right? Or the agent system, right? It might mean here's the based weights. Congrats, we didn't do any of the reasoning training or like, here's the instruct model. Or maybe we even do some reasoning, right? But it's like, we're, we're. I guarantee you if Meta has the best model in the world, they will not open source it. Like, that's like, I would, I would bet like.

• 44:47Jacob Effron?

  Can they be like, you know, can it feel like, hey, they're six, nine months behind with a reasoning model? Like, you know, when, when, uh, will there be like an open source equivalent to O three in six, nine months?

• 44:55Speaker 01

  Yeah, I mean, that's, that's a great question. I'm, I'm not, I'm not exactly sure. I don't think so, actually.

• 45:01Jacob Effron?

  Yeah.

• 45:01Speaker 01

  Um, I think like O three, like especially depending on like what level of like compute you throw at it, right? Like if you looked at the ARC challenge, like seems unlikely to me.

• 45:10Jacob Effron?

  Yeah.

• 45:10Speaker 01

  Um, but at the same time, Google's already got a reasoning model and Tropic allegedly has one internally. That's like really good, better than O three even, but you know, we'll see, um, when they eventually release it. Like, it's like, what actually ends up happening is going to be very.

• 45:24Jacob Effron?

  It feels like the whole question is like, how much CapEx is Meta willing to spend to, you know, to keep up with this?

• 45:28Speaker 01

  I mean, I think, I think they'll keep up on compute. It's like the science is hard too, right? It's not just like, you know, as, you know.

• 45:33Jacob Effron?

  Well, at some level of compute, I mean, the numbers you're counting of OpenAI by the end of 27 is raised a hundred billion dollars. That's a lot of money to put toward, uh, free open source models.

• 45:41Speaker 01

  Yeah, I mean, like, but like most of, like, to be clear, most of Meta's GPU purchases are for like recommendation system on Instagram reels, right? Like, it's like. You can say like Zuck can be like, oh, yeah, I bought this many GPUs on like some stream or whatever. But then when you dig into it, it's like, well, more than half of those are like for recommendation systems like chill like they're not like, you know, you're not spending $30 billion on LLM infra, infra or transformers or you're doing this on like all your products, right? Um, I think, I think, yeah, I just think metal won't open source, like everything they do. Right? Um, and I think that like Lama four will be open sourced, but like the moment

• 46:22Speaker 01

  that they have something at such a capability level, I don't see why, like they would open source it because they get benefit from their get to attract a lot of talent. They get to have the whole community tell them what they're doing wrong with their models, right? Like you look at Lama two Lama three, people are like, oh yeah, you're doing your, um, your, your RL wrong these ways, right? You're like lobotomizing the model in these ways. And people are like, and Meta is learning from that, right? You're like, uh, you know, I think, and they're attracting all this talent that, you know, if they have the best model, they're not going to do that. If they're like six to nine months behind, um, maybe they're six to nine months behind on capabilities, but then the cost is way higher for inference, or maybe cost is the same on inference, but their capabilities are worse, right? And it's like, um, if you look at any specific benchmark, it's like,

• 47:05Speaker 01

  oh, five percentage points. Great. But it's like those benchmarks that are in the seventies and eighties are irrelevant. It's about like, oh, you're like 10% on this benchmark versus 50%, right? And like, when you look at like, like, like Sui Bench and like things like that, it's like,

• 47:17Jacob Effron?

  Oh, deltas are getting really big.

• 47:18Speaker 01

  Yeah, the deltas are really large, right? Or like R, right? It's like, oh yeah, yeah, there's only like two models that like even score, you know, any reasonable amount, right? It's like the, these are the sort of things that I think matter. It's like, can you do a task or not versus like, oh yeah, it's 5% better at this task.

• 47:33Jacob Effron?

  As you think about like the broader, you know, uh, all the kind of big questions you're thinking about, I'm wondering if there's like, you know, a set of two to three questions that are like most, if you could just like zoom forward two, three years and get the answer to them. Uh, you know, I assume one of them will be the performance of these models on the hundred thousand, you know, GPU clusters, but like, what else is, uh, do you feel like are the big questions that if you could flip, you know, if you had a time machine, go forward to two, three years, uh, you'd most want to know the answer to?

• 47:58Speaker 01

  I think a lot of it would be like how they do stuff right in the model level. Right? Um, cause there are a lot of outstanding questions on like with model development, how that changes hardware development, like how that changes networking development, like how that changes data center build outs. Do you need all the data centers to be right next to each other? Do you need them to be? Can they be geographically spread? What level of bandwidth do you need between them? What model? You know, research stuff comes out, right? Like if a year ago, you explained reasoning and test time compute and how that works. To me, I think like that would have been huge, right? But like, I didn't know that was even the question to ask, right? Like, okay, I knew synthetic data was a thing, but I didn't think it was like that huge of a deal, right? It was a big deal, but not like, that was more for like distilling stuff, right?

• 48:36Speaker 01

  Um, but it's not, right? There's a lot of other use cases. So it's like, I don't even know what question on the model side is like the important one, right? Like what really gets agents to work reliably and well, right? What is the trick that gets computers use models? Cause Anthropic's computer use model today sucks, right? But like, I mean, it at least is something, right? No one else has something. Um, but now that you have something, it's clear to see like how you can like climb up and be like, oh, this is good. So what, what are the tricks that people figure out? And how does that impact, right? Like, what is the average like, when you're doing inference, like what is the average sequence length now, right? Like, is reason, what's like, what's reasoning? Or what's like test time compute in terms of like search look like, right? Like, these are questions that like, I don't even know what the answer would be.

• 49:15Speaker 01

  Um, and I don't know, I'm, I guess I'm, I'm definitely not smart enough to know what question to ask in two years, like besides like, what is the secret? What is the biggest development, right? Like it's like, ah.

• 49:23Jacob Effron?

  Well, with this development, I guess of, of, uh, you know, more compute being used to create synthetic data and a difference like, you know, that obviously has massive implications on the data center hardware side. It feels like we might be able to get to potentially more.

• 49:34Speaker 01

  But like, like for example, right? Like when you're doing synthetic data generation, you're, you're verifying which reasoning chains that you've generated are good. You're throwing away a bunch of them. And then you're training on them and you're grading them with all these reward models and you're, and you're training on them. Like how much of this is online versus offline, right? Can you fully separate the synthetic data generation from the, uh, uh, post-training like RL with all the reward models? No, you can't because you need to update the model that's generating the reasoning data constantly. But then like, how often do you need to update it? Can you, do you, can you, do you need to update it every? Thirty seconds, right? Like if you look at like training of llama, I think I think the way the gradients got updated every fifteen seconds, right? And it took a few seconds and then you're training for like ten seconds or fifteen seconds, whatever, right?

• 50:15Speaker 01

  And it's like, oh, okay, you're spending this much time in network versus compute. Like I can like think about why how you need to design your network now. But what what is it? What is that ratio? What does that mean on reasoning? Like when you're generating that data? Uh, how much? How much time does it like? How much time are you generating data? And then doing post-training and then before you update the way model and send it back to the data generation nodes to do it again and grading their outputs like? This stuff like it's kind of unknown to me. Um, and probably unknown to anyone besides like OpenAI and Anthropic and SSI like. What is like actually the future and what is what matters there? What can you? Like, how would you design a cluster to do to accommodate that? Or how would you design your network?

• 50:56Speaker 01

  Or can you be not co-located, right? Maybe maybe just really, really good fiber layouts between these two data centers is good enough. Or uh, maybe they do need to be co-located with super high bandwidth, right? Like these are sort of questions that like, I don't know, right? Uh, yeah, the paradigm exists, but like there's so many details.

• 51:11Jacob Effron?

  Totally. No, I mean, I think it it obviously has implications. I feel like you must get asked all the time. And I think you've you've talked about very publicly on other podcasts, you know, I'm sure people are like, well, Nvidia like, you know, paint the likeliest picture where they lose or, you know, what are like the actual, uh, you know, long-term modes here? And I know you've talked about obviously the hardware, software, networking side of things. I'm sure you've seen a bunch of hardware startups, um, and folks going after different parts of it. What what of that is like most interesting to you?

• 51:35Speaker 01

  Um, I think I think like there's there's a couple of classes, right? There's the old AI hardware startups, right? Cerebras, Grox and Manova, a tense torrent, right? A quote unquote old, right? They've been around for a while. And then there's a new age of them, right? Like, um, there's, there's like, you know, etched and positron and uh, Maddox and, you know, there's a number of other ones. There's a couple that are like some pretty high profile ones that are, I think, still under stealth, so I won't say anything. Um, but there's quite a few AI hardware startups and their approaches are quite different. The way they think is very different, the what they're focusing on is very different, like each each hardware company has a quote unquote a gimmick, right? Because like. You're not gonna beat Nvidia by engineering better than them on all axes and doing this, doing stuff the same way. So you have to do things like quite differently.

• 52:20Speaker 01

  Um, meanwhile, Nvidia every generation is like making, you know, large enough changes to their architecture. So it's like not necessarily like the same, but you know, they're making large enough changes to their infrastructure. Um, and so, you know, the question is whether or not like, you know, what, what are the like quote unquote gimmicks people are doing? How are they solving them and do those give them like? Does that differentiate them enough? And then at the same time, you have the like big problem of like all the models are being developed on Nvidia hardware. Um, cognizant of what's gonna happen and with the, with the drawbacks and benefits of what Nvidia hardware does. And what's the next generation one gonna do? And so like all the model research ideas like, hey, this research idea would work, but it's like,

• 53:03Speaker 01

  if it runs really inefficiently on GPUs, I'm not gonna pursue it because I don't actually care about like how many operations I did. I care about how much time it took. Right? And if like, even though that is theoretically 10x less operations, but it took 10x longer because it's just a algorithm that doesn't work well on GPUs. And all of a sudden it's like, all of a sudden it's like, this is not worthwhile pursuing.

• 53:24Jacob Effron?

  Yeah. So the entire research landscape is.

• 53:25Speaker 01

  Yeah. So like it, it, it like Nvidia's dominance influences where research goes. Right? And so like you have to be different, but you can't be too different because then you're like, you know, like, well, then models didn't develop the way I, I wanted them to and now I'm so out, right? Like, could you have a model that's like extremely, you know, far fewer parameters, but needs way more bandwidth and you know, could run all entirely on chip potentially, but like, how are you gonna train that model, right? And it's like, well, I'm gonna train it with Nvidia GPUs, which doesn't make sense because I can do something better and how am I gonna deploy it? Well, I'd rather deploy, have it be able to be deployed on all the GPUs I've already bought. Right? Um, so it's like, you know, you've got the chicken and the egg kind of problem too. Um, I think like some people are explicitly targeting like inference and like training and like, that'll be interesting because

• 54:10Speaker 01

  um, when you talk to like, you know, Nvidia, you talk to the chip teams at various hyperscalers like the TPU or Tranium or you know, et cetera, right? Like these various, like one of the labs has an AI chip team, like all these companies. All these people believe that training chips are inference chips, which is very funny. Um, and I think it's because like they, they see the flexibility. They see like the workload changing. It's like, so like, does an inference chip, like what does it materially mean? Cause there are a few of those in works. Right? Um. By the time you get your chip out, maybe you projected a 5X cost difference or 10X cost difference versus Hopper.

• 54:45Jacob Effron?

  They've already captured it.

• 54:46Speaker 01

  But now like Blackwell, Nvidia's pitching 10 to 15X improvement in costs. It's like, well, you know, they're massaging the numbers, marketing. Actually, I think they said 30X in their GTC, but like, you know, massaging the numbers 10X, 15X even is like probably could be like reasonable or not. Um, depends on the workload. Standard chat inference, it's probably not that, but for like, uh, for medium sized and small models, but for like really, really large models or reasoning models, it's actually, I do think it's going to be like 10, 15X. Um, so it's like, oh, wow. Okay. So you've captured, you like narrowed the delta massively, right? Um, so that's like a big, uh, that's a big like question also is like, so, so what is interesting is like, yeah, a lot of these companies are taking approaches that are super cool, right? Like, and, and the theory that I mentioned, the new age ones like Mad X, Positron and

• 55:31Speaker 01

  Etched are taking approaches that are super cool, right? And could work out, uh, or models could develop in a way that isn't good for them. Right? Um, so that's, that's the big question, right?

• 55:42Jacob Effron?

  I think you've called Tranium like the Amazon basics of chips. And so, uh, you're obviously the hardware guru, but like, one thing I've just been curious about is like, to what extent is Anthropic's use of them like a sacrifice versus, hey, it doesn't really matter.

• 55:54Speaker 01

  Um, yes, it's, it's the Amazon basics TPU because like its topology of networking looks exactly like a TPU and like a lot of different design choices they have are like, this is just like TPU stuff, right? Um, obviously they're doing a lot of their own independent stuff. There's a lot of unique things, um, that they've implemented, like stochastic rounding and all these other things, but like, it's like, you know, these are unique things, but um, it is the Amazon basics TPU, right? Uh, from Google, right? Google TPU. Um, and so like, cause it's like worse, but it's cheaper and yeah, like, well, now Anthropic, like what is like, they barely use Tranium today, right? They barely use Tranium in Inferentia. Uh, they've primarily used Hopper, Hopper at Google, Hopper at Amazon. Um, they did use some TPU at Google, of course.

• 56:35Speaker 01

  Um, but now this year, like, it's like very clear, they've bet all of their eggs into the basket of Tranium. Um, how much of this is forced, right? Like, you know, how do I do, did I need to do this to get Amazon to invest in me and like make me their partner? Uh, partially, yeah, absolutely, absolutely. Right. But did I do this out of my own volition because it actually made sense? I think partially, yeah, because like the Amazon basics TPU gives you on some metrics. It gives you the best uh, performance per dollar, right? And those metrics are like memory bandwidth and memory capacity per dollar. It actually has better than any other chip in the market, including TPUs or GPUs, right? Uh, cause it's like some of the like supply chain choices that Amazon chose, right?

• 57:16Jacob Effron?

  Yeah.

• 57:16Speaker 01

  Um, and, and, and things around that, right? So it's like, what?

• 57:22Speaker 01

  You know, I think, I, I don't think it's as simple as like Anthropic was forced to. I think, I think like they had to weigh the benefits, right? They're like, we could partner really closely with Amazon. We could raise more money from them and do this. We're gonna raise a little less money and, and like try and get GPUs from them, right? It's like, they had to partner with someone to do the capex, of course, right? Cause the whole like rental versus like capex of like the cluster. Uh, so they definitely had to deal with that. And then they had to have the distribution channels of Amazon, right? They needed those things.

• 57:49Speaker 01

  But like, I'm sure they could have struck a smaller deal with Amazon and gotten GPUs if they really wanted to. Now, was that better or was it better to like, take more money, go trainium, go all in? Um, you know, do like, try and emphasize your future model architecture on the benefits that trainium offers you versus GPUs? Because there are some, right? Uh, majority, no, they're worse, right? Just straight up worse, but some benefits exist. So that's the sort of question that, uh, um, you know, it's a good question. And I, I, I think mostly like, it's like hand wringing and, and a bit of like, yeah, we can engineer our way out of this.

• 58:21Jacob Effron?

  Does it make sense for OpenAI to like, build its own chip?

• 58:23Speaker 01

  I think it does, right? I mean, like, it doesn't make sense if you think like, you know, their growth is like linear, like, you know, it's gonna go from like, you know, you know, like, what do you think OpenAI's revenue this year is gonna be? If it's less than $10 billion and the year after that, less than $15 billion, should not make a chip, right? But like, if you think their revenue this year is gonna be north of $10 billion, by, by $10 billion, I mean, run rate.

• 58:45Jacob Effron?

  Yeah.

• 58:46Speaker 01

  Cause we're, we're, we're at a VC, right? We don't, we don't think about things in, uh, no, run rate is more important. I think it's just simply more important, right?

• 58:53Jacob Effron?

  Whatever the higher one is, we'll take.

• 58:54Speaker 01

  Yeah, exactly, exactly. Uh, I mean, it's like more indicative of like what the current state of the business is, right? Um, so like, You know, if they go like well, well, north of $10 billion on run rate and the year after that, what are they at like north of 20 like, you know, you know, the costs start to scale rapidly and like, it makes a ton of sense to make their own chip, right? Um, and so like, you know, just because they have the chip team doesn't mean that like they're not gonna, you know, they're not like collaborating super closely with Nvidia and working with them on new architecture, right? Like there's a wall in in OpenAI, like the people working on the chip do not get to work with Nvidia on what their next generation architecture is, right?

• 59:30Jacob Effron?

  Well, I think Nvidia is no stranger to this like cooperation. I mean, all of its major customers.

• 59:34Speaker 01

  Exactly, exactly. Like Meta is doing it. Google's doing it. Amazon's doing it, right? It's like they're used to it and their like attitude is very much like, good luck, we're gonna crush you, right? Like, we're just better, right? And like, so far, that's worked, right? Um, and and like, I think their attitude to OpenAI would be the same. Um, and OpenAI's attitude is always like, well, you know, how much, you know, and it's the same as hyperscalers, right? Like, well, if we spend like, you know, a few hundred million dollars, billion dollars a year on this, um, it gives us a call option. Also, we can always just use it as a negotiating chip, right? Like, if if, you know, I spend a billion dollars on my own chips in 2627, right? Uh, but then, you know, the alternative is, you know, and and and I'm not really deploying them in huge scales. I'm mostly

• 1:00:19Speaker 01

  still buying Nvidia chips, but if I got a 10% discount on Nvidia chips, it paid for itself.

• 1:00:22Jacob Effron?

  Yeah.

• 1:00:22Speaker 01

  And it's like, does, does Tranium help Amazon get a 10% discount or a 5% discount on $30 billion of GPU purchases? Like, okay, great. It's like worth it, right? And it doesn't even like, you could like, just not do anything with it and it'd be worth it, right? Like, obviously, it's a game of chicken. Like, Jensen, like, knows how much you're making and all these things. But yeah, I think I think that's the big, um, question. And I think, I think from my perspective, it absolutely makes sense, even if it's not successful.

• 1:00:48Jacob Effron?

  Yeah.

• 1:00:49Speaker 01

  Um, to do this. And if it's successful, it's humongous. Yeah.

• 1:00:52Jacob Effron?

  There's this whole crop of companies too that that popped up, like, you know, the fireworks and togethers of the world, um, you know, that are, they're obviously just, you know, focused on inference optimization today and places to run a bunch of models. What what happens to that market long term?

• 1:01:04Speaker 01

  I think that, um, enterprise AI, uh, is is actually got like really decent legs in some regards. Um,

• 1:01:15Speaker 01

  Because of, again, like all the unique data that enterprises have and unique use cases and, you know, now that you can, enterprises like to think their data is good, but now like, most of the times it's pretty dirty and garbage, right? Like, just to be clear, but now like with the synthetic data pipelines, reasoning, all this sort of stuff, people can actually figure out how to generate a bunch of data that's specific to a business and verify whether it's good or not. And then like they can actually improve the models and like they can do reasoning on a much smaller scale for enterprises. I think like this works hugely, right? This is massively good, right? Now, obviously like fireworks and together, I think are like, there's like 20 inference API providers out there that are just taking open source models and serving them. You obviously named the best two in class, right?

• 1:01:57Speaker 01

  Two companies that like build their own inference engine, are very efficient, much more efficient than like VLM or TensorRT LLM, like all of these like sort of things have good go to market, all these sort of things. Like, you know, these companies are good. I think they will be good because they're going to continue to be A, you know, Meta is going to open source models, B, Chinese will continue open source models, C, enterprises will be able to work with people to make better, you know, reasoning type models right now, like together helps people train models, right? And so does Databricks and like sort of like all these companies like help train models. And there's many other like sort of services or like people can partner with a startup to help them make a reasoning model.

• 1:02:37Jacob Effron?

  I feel like the whole, yeah, the whole mosaic pitch was like, you know, train your own models. That went out of fashion for a bit because the general purpose models were just getting better at everything way faster than like a Bloomberg GPT was, but now with these reasoning chains that need to kind of verify this, you actually might be back in that world.

• 1:02:50Speaker 01

  Yeah, yeah, I think, I think like Mosaic really lost its kind of, and Databricks kind of lost its footing for a bit. It'll be really interesting to see if they can get RL and reasoning to work because I think that like, you know, if like, and I'm sure, I'm sure Ollie at Dabry kind of knows this is like, you know, this is the thing that like would make training for your customers make sense again, and maybe not pre-training, right? Because that's what Mosaic was doing. Mosaic was doing really, really good pre-training. You know, they had the best model in the world twice, two different instances, open source model for, for short periods of time, but they had the best model in the world for short periods of time, twice, like once with the MPT and then once with DBRX. But. They, you know, like, they've kind of fallen off because it's like, well, I can't really compete and spend the CapEx that Meta is, right? Um, or, or, you know, even like Alibaba, right?

• 1:03:35Speaker 01

  Um, so, but now, again, right, it's like, I take the open source pre-trained model. I generate all these pipelines for reasoning synthetic data. And my customers' data, again, like, you know, it kind of sucks, but like, you know, now I can like verify it in some way. Figure out the recipe and like actually apply it to everyone's business use cases. I think that is, that is like potentially, and, and, and these people need to serve models, right? So it's not just like Databricks, Mosaic ML that's going to do this, but like, everyone's going to do this, there's going to be startups that do this as a service. Um, like, uh, like adaptive ML, like they're going to do this as a service and they've actually been doing it as a service. Um, for like a year now, um, which is like really cool. Like these, these people can partner with enterprises, they can do white glove services, they can do consulting services, like whatever it is.

• 1:04:16Speaker 01

  Help them build models that like are uniquely, uh, you know, unique use cases for this stuff, right? Um, and then, and then like those will need to be deployed, right? Or companies will want to do it on their own, right? There's always going to be, and there's always going to be like security stuff, like, so I think, and I think like Together and, and Fireworks will be like the partner of choice for inference of these models, right? Even if it's not necessarily on their GPUs, it's like, let, let me, you know, do a virtual private cloud on Amazon with your software. Uh, running the model, right? Because you've built all these like variable batching and all these like various things and it's way more efficient, right? Than like the public software.

• 1:04:53Jacob Effron?

  I'm going to ask like a very basic question about Core, we've, uh, just, uh, for my identification. Um, and that's just like, what made them so successful? I mean, from the outside, sometimes it feels like there was this decision from Jensen to allocate GPUs among a bunch of players. They were kind of rewarded in that. Um, and I think it kind of goes to, and you're, you're an expert on this, like the extent to which some of these mini clouds can differentiate beyond like just access to the GPUs, which it felt like in the first bit was, was one of the big things.

• 1:05:19Speaker 01

  So I think, I think like part of it is, um, I think I think it's there's three factors, right? One is the allocation bit, right? Like Jensen is no idiot. His customers are all trying to build chips. Um, they also rent most of the gyps to other people. You want to have more competition, right? So let me spark competition. Let me do a small investment in like four different clouds. I think I've invested in four different clouds so far. Um, and and they haven't been large investments. They've been pretty small. Um, but like it's sort of given other people confidence like, oh yeah, they're gonna get allocation, right? And then those GPUs are allocated there. World's so tight in GPUs. They're able to rent them for a lot of money, make money, and get the engine going on a, on a, on a cloud, right? So CoreWeave now has like 200K plus GPUs, right? Like a lot of GPUs, right?

• 1:05:59Speaker 01

  They're doing like, you know, billions of dollars of revenue, right? Like it's not, not mundane business now, right? Um, and so, and you know, they're worth like whatever, like 20 billion dollars or something like I think on the last round. Whatever, right? And and they're gonna go public this year, probably at like more than that, right? Well, more than that. It's like, this is a real business now, right? Um, how do they differentiate, right? Like one is, yeah, they still continue to have access, newest generation. They're still gonna have access. They're still gonna, you know, maybe Google gets for every, you know, 10 that Microsoft gets, CoreWeave maybe in the beginning can get three. Now, at some point, you know, the production capacity is not 13, the production capacity is 100. And you know, again, I'm just pretending there's only two people. Now, you know, Microsoft gets 90 and CoreWeave only gets, you know,

• 1:06:41Speaker 01

  three, right? So now it's, you know, 93, right? Whatever, right? Like, you know, like the, the, but like at the beginning is of, of a new GPU's useful life. You know, it is worth the most because at that time you can rent it for way more than over time. It narrows as more and more competitors enter the market and there's more and more of them. Um, that's one. Two is dispute of build outs, right? Like, uh, Microsoft's data center organization and RAC and all these things like server organization, same with Amazon. Um, they're custom, they're like 10s of thousands of people. They've always been used to modifying servers, customizing them to squeeze every penny out. Right? Um, and and and in the era of like GPU compute, it's like,

• 1:07:19Speaker 01

  Well, what if I just do what Jensen says, right? Like I build, you know, I might make some tweaks. I might put a little bit less memory. I might change the storage around. I might change the network a little bit slightly, but like in general, what if I just take the reference design and like modify it a little bit? Uh, now I am paying a little bit more, right? Not, you know, that, that, but then I have the like time to market, right? Like the moment the GPU is being produced by TSMC, it's packaged. It can go to the server maker. Reference design is ready to ship to me, right? Whereas like Amazon, you know, especially Amazon, their Blackwell rollouts are much later than many of the other uh players because they have to do all this customization.

• 1:07:53Jacob Effron?

  Yeah.

• 1:07:53Speaker 01

  Right? Um, and, and then like lastly, like sort of there's like, um, I think, I think the other aspect is data center capacity, right? We talked about this ESG stuff. Core Wave gives like, they're very, they've been very aggressive always on like renting GPUs, getting credit at like what, what at the time seemed like crazy terms, right? Like double digit loans, like what the flip to deploy GPUs? It was the right decision, right? Now they're, they're established and so their loans are single digits again, right? Like, you know, higher, higher single digits, but it's like, um, getting data centers is very hard and they've been very, very creative. Initially they were just like getting anything and everything. That capacity all dried out and the hyperscalers woke up and they went to try and get everything, everything. But then Core Wave started going to crypto companies, right? Like, um,

• 1:08:35Speaker 01

  you know, they went to a couple of different crypto companies, um, offered to buy one of them out, offered to, you know, and then eventually like just like struck a deal where they're retrofitting all their data centers. Like, yeah, throw all this crypto crap away. I'm going to convert your, your thing into a data center, right? And one of these crypto data centers has a, like a 200 megawatt uh natural gas plant in the middle of like seven buildings, right? And each of these buildings is a data center. So they're, they simultaneous and it's like a natural gas plant. It's like, like Meta, what they're doing is like, okay, the gas plant is like 20, 30 miles away and I, I, I struck a deal with this solar power company that's putting, you know, tons of solar panels, like, you know, in a similar distance away. But I'm not necessarily consuming those electrons. It's not clean, right? In the case of like Core Wave, it's like, no, no, no, I don't care, right?

• 1:09:20Speaker 01

  Like, the power is here, like, right onsite, I'm converting this to crypto data, these crypto data centers to real data centers. There's a metric called PUE, Power Usage Effectiveness, is basically, how much power are you pulling uh, from like, the grid slash power generation, versus how much are the chips consuming? Right? And so, or like the servers, right? And so, like, the ratio that, like, hyperscalers try to design to is like 1.1, right? Only 10% of the power is lost in transmission and cooling, right? But like, this crypto data center, once they convert it, because it's like a really, like, really dumb design, because the crypto people didn't know how to build the data center properly, right? Uh, even though it's being retrofitted, it's still gonna be like 1.4. So it's like, you go from 1.1 to 1.4, it's like, oh, if it's consuming 200 megawatts,

• 1:10:03Speaker 01

  what was 20 megawatts of power being burned, uh, on cooling and electrical transmission and conversion, now is like 50, right? 60, 70, right? And it's like, okay, this is a lot of inefficiency, but it's like, screw it, we need GPUs now.

• 1:10:18Jacob Effron?

  It was worth it. Like I said, the inefficiency, both on the financing side, on the, uh, on the power conversion side.

• 1:10:23Speaker 01

  And then, and then they do like certain things like really efficiently, like they're just a small organization that's well-run.

• 1:10:27Jacob Effron?

  Yeah.

• 1:10:27Speaker 01

  Right? And their software, their, their cloud software for GPU rental is literally better than Amazon's and Google's.

• 1:10:35Jacob Effron?

  Yeah.

• 1:10:35Speaker 01

  Um, it's, it's actually hilarious. We'll have a piece about-

• 1:10:37Jacob Effron?

  What makes it better?

• 1:10:38Speaker 01

  Um, just the way they do their managed services, the way they manage their network, uh, the way they do storage, like, there's a number of things here that are just like objectively better with Core Wave. Um, and it's like, what? Um, how did you guys build a stack that's better?

• 1:10:53Jacob Effron?

  Loser that game.

• 1:10:54Speaker 01

  But it's like, it's like, well, you have this legacy and, you know, innovator's dilemma, and you don't build a brand new thing that's purpose-built, you like, morph and you have all these like weird requirements for random customers, like, random defense company, you also need to make sure your GPUs are rentable to them, or like, you know, and it's like- Corviva is like, well, that's like, they're starting the problem anew and they have a brand new team, clean slate, really good engineers, and they're building what needs to be built, purpose-built for this business, right?

• 1:11:18Jacob Effron?

  Yeah.

• 1:11:19Speaker 01

  And they don't have like, a team of like, 2,000 people who work on storage architecture, that used, that used to be primarily targeted for these kit, this use case. And accelerators are different, right? Like what you need in AI, not exactly now. I guess it's similar, right? Like, you know, but they're trying to morph this massive codebase to that, rather than like, ground up, build something new, right? And it's like, this is the

• 1:11:39Jacob Effron?

  The case for every startup in the AI world.

• 1:11:41Speaker 01

  Yeah, like, I think, I think like, there is innovator's dilemma with the big companies.

• 1:11:44Jacob Effron?

  Yeah. What are like, I mean, you obviously must get, see all sorts of kind of interesting investment plays on this massive hardware buildup going on. There's the obvious stuff. There's data centers, liquid cooling. Any like, second or third order effects you've seen where you're like, that's a clever way to play the, uh, the massive data center buildup?

• 1:11:59Speaker 01

  Yeah, I think like, I mean, like, some of the stuff is like, uh, yesterday I had like, an hour-long discussion with a guy who's building a data center in Armenia, because there's a nuclear power plant there. It's like, great, this is like, awesome. And he's like, doing a venture fund, he has a venture fund in New York, but he's kind of just like, screw it, I'm gonna work on this, pretty much. It's like, oh, this is awesome, right? Like, it's like, you know, there's silly stuff like that, not silly, like, it could work, right? Uh, although the regulations kind of screw him, I talked about that. Uh, but like, I think like, a lot of like, interesting plays around networking and optics, right? There's a lot of stuff going on in networking and optics, which is like, a big bottleneck. Um, especially as we continue to scale the density of GPUs, um, and the, uh, you know, the,

• 1:12:42Speaker 01

  the amount of data that needs to be communicated between GPUs as models get, A, larger, and B, the context lengths get longer and longer because of reasoning. Um, you know, this is sort of like, and search, like, all these things require like, more interconnected GPUs. Um, I would say that there's a lot of interesting stuff around like, transformers, right? I told you they're sold out for like, four years, but like, there's all sorts of interesting startups working on like, solid-state transformers, and like, partially solid-state transformers, and like, all sorts of like, hardware stuff that's like, sexy, right? Like, there's like, there's like, funny stuff, there's like, cool companies on like, carbon sequestration, because yes, Meta and Microsoft, Meta especially in XAI, like I said, like, screw the environment, right? Like, not screw the environment, but like, we're gonna build AI as fast as we can. You know, like, there are at some point gonna be like, yeah, let's be great again, right?

• 1:13:25Speaker 01

  And it's like, carbon sequestration stuff is like, in, in a wrapped up in a data center play is like, actually pretty interesting, right? It's like, there's all sorts of like, you know, like, interesting stuff around, you know, storage, what companies are doing there. Uh, because video models are gonna require a lot of like, interesting work there, right? Like, and it's like, you know, purpose-built solutions for the plethora and sea of innovation that's happening. Um, and targeting all these things, I think if you look at every layer of the stack, there's more innovation happening today. Um, you know, not just in each stack, but also like bridging multiple layers of the stack, uh, to like, come up with like, a huge improvement. Um, some of that is happening in the big companies, some of that's happening at startups. I think, you know, some of these startups will be get acquired by bigger companies, some will actually like, find a product market fit, right? Like, um, I think, uh, some of them will license technology, like whatever it is.

• 1:14:11Speaker 01

  I think there's a lot of like hardware technologies that are interesting. And then I think the really like, fun area is software infra.

• 1:14:18Jacob Effron?

  Yeah.

• 1:14:19Speaker 01

  Right? So software infrastructure is like, also getting very diverse. Um, because not everyone can just build their own stack. So I think that, where, where people are like, very in tune with the hardware, understand how to make things run fast. Um, also in tune with what's going on with models, um, and like bringing, you know, you bring your data and all your capabilities. And I bring you something that lets you build and serve your actual use case, right? So this is, software infra is not, you know, was not always sexy. It was always like, you know, SaaS was more sexy, or like, you know, like, um, and, or like, you know, some sort of like. I, I mean, I guess software infra is to some extent, uh, SaaS, but not really. Um, I think these are the sort of the businesses where there's a lot of innovation happening as well.

• 1:15:00Jacob Effron?

  The infra layer has just been so hard to invest in because the models change so quickly that any kind of like scaffolding or things you're building for this current generation of models, you're like, is that relevant in, in six, twelve?

• 1:15:09Speaker 01

  This is, this is why like, some companies have like, you know, bet on things that were like, further out, right? Like, like, like I mentioned, adaptive, right? Like, they were, they were working on, they like, it's like, a few French guys who moved to UAE and they built the Falcon model in UAE, right? And then they like, moved to the, back to France and now they're in New York, right? But like, while they were building the Falcon model for UAE, they were doing pre-trading. They did a hundred eighty billion parameter model to chill optimal, it's like fun, right? It was not like, it was a good model when it got released, it was just like, you know, whatever, right? Like Lama just came out too, right? It was like, it was like, really good, but I was like, yep, they're like, immediately, pre-training doesn't matter, right? It's like, we're gonna work on synthetic data and reasoning, it's like, what? Okay, sure. And then like, for a while, I was like, what are you guys doing? Like, I don't know. And then like, all of a sudden, it's like, oh, wait, wait, this is right. Like, so you have to like,

• 1:15:50Jacob Effron?

  You saw something.

• 1:15:51Speaker 01

  Yeah, it's like, it's like, there's a lot of companies that are like, doing something that may not matter today, and you just have to like, doesn't seem to be the focus, the popular thing today, but could be and will be in the future.

• 1:16:03Jacob Effron?

  Hundred percent. What about, what do you think of like, these AI hardware efforts, like folks that are using, you know, AI for chip design? Or, you know, circuit board layout, or some of these other things?

• 1:16:11Speaker 01

  I think it's really interesting that YC all of a sudden, like, decided to invest in like 12 different AI for chip design companies all at once. Uh, in, in the last few batches. Um, a, a, a couple of them are cool. I actually even invested in one of them myself personally. Just to be clear, I'm not hating on the idea. It's just like, very interesting that this happened.

• 1:16:29Speaker 01

  But like, you know, AI, chip design is very expensive, right? Um, it's very difficult. The number of engineers working on chip design has not grown that meaningfully over the last two decades, right? In the, in the US.

• 1:16:41Jacob Effron?

  Like, this is a start of like, every AI app pitch. It's like, whatever, insert your profession here is stagnating in growth.

• 1:16:47Speaker 01

  Right, but, but the output has been dramatically improving, right? And that's because of EDA software, right? Electronic design assistance. And EDA software, as much as you can hate on it, as much as you can say it's like, old shitty software, it's actually really awesome. Because it's like, the, the productivity gains that chip designers are getting year on year are meaningfully large, right? And that's pre-AI. Now, now Cadence and Synopsys and Siemens are all, uh, Mentor Graphics, right? Are all investing in AI for chip design. But there's also a lot of companies doing it, right? Uh, Nvidia has written multiple papers about their internal tooling. Um, and they've hidden a lot of it, too. They haven't said a lot about it. Um, Google, same way, right? This is absolutely going to be huge. Now, is AI going to chip, design chips right away? No, but is it going to be a force multiplier for a very like, in-demand profession? Like, and yeah, like, and, and like, the other thing is like,

• 1:17:33Speaker 01

  today, like, a lot of workloads just have to use like, like CPUs, and then GPUs are the general-purpose parallel computing platform, right? Now, obviously, they're generally making more and more enhancements for certain types of AI models. But like, GPUs are not good at running CNNs.

• 1:17:46Jacob Effron?

  Yeah.

• 1:17:47Speaker 01

  Right? Like, and this is a conscious design decision by Nvidia, right? Like, they're okay, they're good, but they're not like, amazing at running CNNs like they are at transformers, right? And, and this, I believe, is like a conscious design decision. Um, what, you know, how do you go from like CPU, which is the catchall general purpose GPU to like general purpose for parallel processing to like, what is the next step? And like, and Nvidia is like branching their GPU architecture out. They have different like architectures for like automotive versus uh gaming versus um,

• 1:18:15Jacob Effron?

  Yeah.

• 1:18:15Speaker 01

  Um, you know, pure data center AI. So they're not like, just like sitting flat footed, but it's like, how can we proliferate architecture to target use cases better? Um, and, and today for, for it to make sense for you to design a chip for a TAM, that TAM needs to be billions.

• 1:18:30Jacob Effron?

  If you can bring down the cost.

• 1:18:31Speaker 01

  Right, so like Google made a YouTube chip, right? Because that was a billion dollar, multi-billion dollar TAM for them alone internally. That's what ASIC made sense. But when, when chip design costs can drive down and it's like AI for chip design will do this, right? Like, I just think it won't be like, you know, it's not going to go from zero to one. It's going to be like a force multiplier. And I think it's, it's one of the areas I'm super, super excited about, whether it be floor planning or, or RTL generation, or many other things like verification. Verification is like half the spend of a chip design. It's like quite funny, I was like, more than like half the cost is making sure what you designed actually works. I was like, okay, interesting. Um, like there's tons of like stuff there um that I think um could be done. And yeah, it's like, I'm very excited about that space, actually.

• 1:19:13Jacob Effron?

  Although I feel like this has been a fascinating and really wide ranging conversation. We always like to end with a standard set of quickfire questions. Um, and so maybe to start, uh, what's one thing that's overhyped and one thing that's underhyped in the AI world today?

• 1:19:24Speaker 01

  Overhyped rag. Um, underhyped various other forms of like semantic search um and like retrieval. Um, there's some interesting papers that came out recently, yeah.

• 1:19:37Jacob Effron?

  Will model progress uh this year in 25 be more or lesser the same as in 24?

• 1:19:41Speaker 01

  More.

• 1:19:41Jacob Effron?

  What's your weirdest prediction on like the implications for all this AI progress on the future?

• 1:19:45Speaker 01

  I think, you know, there's a lot of like worries that like there's going to be huge inequality. Um, I'm actually like on the flip side, I think that like. Um, blue collar type jobs that were generally hated on were like, are gonna do well until robotics comes, which I think is pretty soon, but uh, you know, um,

• 1:20:03Jacob Effron?

  So for that year or two.

• 1:20:04Speaker 01

  But like, no, but like, I think that like, generally the poorest people in the world will also improve in quality of living massively, right? And it's not gonna be like a have versus have nots, ultra elite versus not. I don't think that's gonna be the future.

• 1:20:17Jacob Effron?

  Yeah, I mean, I guess you better hope the inference costs on these reasoning models comes down.

• 1:20:21Speaker 01

  Yeah, I, I like that the, what is, what is opening eyes thing? Intelligence per uh, dollar, like maximizing intelligence per dollar. And it's like, well, right now it's really freaking expensive, right?

• 1:20:31Jacob Effron?

  Yeah.

• 1:20:31Speaker 01

  And I expect, I expect the models to get even more expensive, right? Like a query from like GPT five, oh, five or whatever, right? Like reasoning version of GPT five is gonna be like dozens of dollars, if not hundreds of dollars, right? But it's good, the work it puts out is gonna be crazy.

• 1:20:46Jacob Effron?

  Do you have a go to query you use when like a new model comes out, tested out?

• 1:20:49Speaker 01

  Um, it's funny, I actually asked them about semiconductor manufacturing because uh, I know the training set quite well, I've read a lot of the papers. Um, and, and then I see if it actually understands it and a lot of times models get it wrong. Um, in today, even to this day, Claude is the only model that you can like, once you create enough, can get the patterning stack for lithography, like somewhat accurate. And that's just because no one publishes on it. So it's like, it has to like, understand the papers and infer what people are actually doing. Uh, which I thought was very interesting.

• 1:21:17Jacob Effron?

  That is a cool way to do it. And we've talked about a bunch of different parts of the AI startup world, but any like spaces within the AI startup world we haven't talked about that you find particularly interesting or exciting?

• 1:21:25Speaker 01

  I think there's a lot of cool stuff happening on the distributed training side, right? Like with Noos and with uh, um, you know, although I'm not, you know, like, you know, there still needs to be a lot more to prove what they did is like good and real, but like very exciting. And then prime intellect, same thing. Um, and yeah, they also like, you know, like are doing really cool stuff. I think distributed uh, training and inference is like gonna be really cool. Um, I think that um,

• 1:21:49Speaker 01

  I think that a lot of these like multi-agent reasoning startups that started a while ago, um, but have actually been focused on reasoning like, like, like Synth Labs and Adaptive and like, people like that, rather than like, you know, agents and pivoting to reasoning because everyone's doing a reasoning startup now, right? Like, uh, I think that's like a really cool area. Um, and I really just like people who in startups that are like, have been singularly focused like forever on the same thing. And like, they're like, and they're right. It's just like, you don't know when they're going to be right, right? And it's sort of like, you know, me be loving chips so much, you know, I had a good business, but like, it did not pop off until like, you know, AI, like, there are a few people working in my company, not 20, right? So it's sort of like, same thing as like, being so internally focused and good at it, but then all of a sudden

• 1:22:33Speaker 01

  the time does come and you were right, right? Like, it's like, this is like, uh, those are the kind of startups I like the most. Not necessarily people who are like outsiders to a field and just want to revolutionize it.

• 1:22:42Jacob Effron?

  Well, we always like to leave the last word to you. Um, I can't imagine there's much of a Venn diagram of folks that listen to this podcast and aren't subscribed to semi-analysis. So I assume some folks are at least a high level familiar with you, but, uh, would love to just, you know, where can folks go to learn more about you, the work you're doing? Any plugs you want to make? Uh, the floor is, uh, is yours.

• 1:22:59Speaker 01

  Yeah, I would say if you want to see the serious side, go to the website, uh, check out the newsletter or check out the institutional sort of products, which is like the main business, like the data sales and consulting. Um, and, you know, we sort of like have industry experts all the way from, you know, former industry experts all the way from like AS, have worked at ASML and like Lam Research and Intel all the way up to like, uh, Microsoft and Nvidia formers and like, you know, everything in the middle. And then also like half the companies like X hedge fund people. So it's a cool mix. And then we also have, and then I would say, if you want to see the like, lighthearted bit, right, where I will tweet jokes like, hey, um, you know, AI regulations came out on Monday, arsenal for democracy. Oh, look, Israel Palestine ceasefire, two days later, wonder how, that's absolute joke, but you know, that's what I

• 1:23:42Speaker 01

  posted on my Twitter, right? Like, so you know, like, I think like, you know, there's a, there's a range and that's Dylan 522P, right? So I think that's sort of the, uh, the shilling I'll do for myself.

• 1:23:51Jacob Effron?

  Amazing. Well, Dylan, thanks so much for coming on. This was awesome.

• 1:23:53Speaker 01

  Yeah, thank you. Thank you.