Pod Insights a16z Podcast Chasing Silicon: The Race for GPUs About this episode With the world constantly generating more data, unlocking the full potential of AI means a constant need for faster and more resilient hardware.
In this episode – the second in our three-part series – we explore the challenges for founders trying to build AI companies. We dive into the delta between supply and demand, whether to own or rent, where moats can be found, and even where open source comes into play.
Look out for the rest of our series, where we dive into terminology and technology that is the backbone of the AI, how much the cost of compute truly costs!
Topics Covered:
00:00 – Supply and demand
03:03 – Competition for AI hardware
04:46 – Who gets access to the supply available
06:28 – How to select which hardware to use
08:59 – Cloud versus bringing infrastructure in house
13:26– What role does open source play?
16:48 – Cheaper and decentralized compute
20:16– Rebuilding the stack
21:31– Upcoming episodes on cost of compute
Resources:
Find Guido on LinkedIn: https://www.linkedin.com/in/appenz/
Find Guido on Twitter: https://twitter.com/appenz
Find a16z on Twitter: https://twitter.com/a16z
Find a16z on LinkedIn: https://www.linkedin.com/company/a16z
Subscribe on your favorite podcast app: https://a16z.simplecast.com/
Follow our host: https://twitter.com/stephsmithio
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 We currently don't have as many AI chips or servers as we'd like to have.
0:06 Steph Smith? How do I get access to the compute that I need? Who decides this?
0:11 Speaker 00 You're looking at some very large investment projects that take some time to adjust. We're rebuilding a stack. You can't look at AI just as a new application, but honestly, I think it's probably a better way to look at it as a different type of compute.
0:24 Steph Smith? With software becoming more important than ever, hardware is following suit. And with the world constantly generating more data, unlocking the full potential of AI means a constant need for faster and more resilient hardware. That is exactly why we've created this mini-series on AI hardware. In part one, we took you through the emerging architecture powering LLMs, from GPU to TPU, including how they work, who's creating them, and also whether we can expect Moore's Law to continue. But part two is for the founders trying to build AI companies. And here we dive into the delta between supply and demand, why we can't just print our way out of a shortage, how founders can get access to inventory,
1:09 Steph Smith? whether they should think about renting or owning, where moats can be found, and even where open source comes into play. You should also look out for part three coming very soon, where we break down exactly how much all of this costs, from training to inference. And today, we're joined again by A16Z Special Advisor Guido Appenzeller, someone who is truly uniquely suited for this deep dive as a storied infrastructure expert with experience like.
1:36 Speaker 00 CTO for Intel's data center group, dealing a lot with hardware, the low-level components. So it's given me, I think, a good insight how large data centers work, what the basic components are that make all of this AI boom possible today.
1:50 Steph Smith? Despite working with infrastructure for quite some time, here's Guido commenting on how the momentum of the recent AI wave is shifting supply and demand dynamics.
1:59 Speaker 00 The biggest thing that is striking there is just the crazy exponential growth of AI at the moment. AI has been booming since mid-last year. I think nobody expected how quickly it would move. And that has just created a demand which, at the moment, the market can't fulfill.
2:14 Steph Smith? As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures.
2:44 Steph Smith? In a recent article, Guido even stated that some reputable sources indicate that demand for AI hardware outstrips supply by a factor of 10. Here's some commenting on how that dynamic is impacting competition.
2:59 Speaker 00 We currently don't have as many AI chips or servers as we'd like to have. So for some of our portfolio companies, finding the compute capacity that they need to run their applications is actually a real challenge, right? There's a whole value chain behind that. It's a combination of many things. We have some bottlenecks on the chip manufacturing side. We have some bottlenecks on building the actual cards. These development cycles take some time. So it's a combination of factors. But probably the biggest thing that is striking that is just the crazy exponential growth of AI at the moment.
3:30 Steph Smith? Maybe this is a silly question, but what really is stopping companies like Intel, like Nvidia, from going and 10Xing their production? Like, is that on the roadmap where we're just going to see a lot more chips? And we won't see this discrepancy between supply and demand? Or is there something more complex at play?
3:46 Speaker 00 It's a bit more complex because if you want to make a chip, right? The way you do it is you make it in a foundry, right? Which are extremely large, extremely complex. Intel makes chips on their own foundries, but most companies manufacture with Taiwan Semiconductor, TSMC, right? And they are capacity constrained, right? You often have to reserve capacity long in advance. There's different processes. So, you know, it might be for a certain process which you don't want to use, there's capacity. But for another one that you do want to use, they don't have the capacity. And you could just say like, well, in that case, let's just build more fabs. But building a fab takes you a couple of years and probably a couple of billion or 10 billion of investment. So you're looking at some very large investment projects that take some time to adjust. And that's sort of what prevents us from reacting more quickly at the moment.
4:29 Steph Smith? While some countries are making major multi-billion dollar investments in new semiconductor production plants, AKA fabs, these will take time to scale. And there are also no promises, given that expertise is concentrated in a few companies. So with demand not subsiding, what does this mean for who gets access to the supply available?
4:52 Steph Smith? It doesn't sound like the demand is going to subside, especially because we see this really, what seems like intrinsic relationship between the power of these models and then the compute that's thrown at them. And so if we do expect demand to continue, I guess the question that arises is, how is this demand allocated? So how does a company, let's say if I'm a founder today, how do I get access to the compute that I need? Who decides this? Is it just who's willing to pay the most? Or how is that supply being distributed?
5:23 Speaker 00 Yeah, there's some of that, right? At the moment, capacity is expensive wherever you go. You know, I tried just to run some personal experiments, tried to reserve an instance while the cloud service providers a few days ago, and they just didn't have any. I was like, no, not available. And what we're seeing is that often in order to get access to the newer cards, the newer chips, right? If you want that at scale, you have to pre-reserve capacity. So often, these are negotiations between a company and the large cloud where you say, okay, I need this many chips for this amount of time. What they'll often ask for is they ask for a certain time commitment. So I'll be like, okay, we can give you this many chips, but we want you to sign basically that you get them exclusively for two years and you paid for that amount. I think OpenAI wasn't in use with that, right? Where you have investment deals, where for example, a cloud provider comes in and invests in a company.
6:07 Speaker 00 And as a result, the company gets capacity. So we're seeing all kinds of deals being struck. As with any scarce resource, right? There's a lot of deal making going on.
6:16 Steph Smith? It's not just a matter of getting access to compute. It's about ensuring you get access to the kind of compute tailored to your needs. And cost is not the only factor here. What would you say in terms of the considerations that they should be keeping in mind? Really, how much should founders know about hardware? And again, selecting which hardware to use?
6:38 Speaker 00 I think the first question honestly I would ask is, do you really need to consume the hardware directly? Or do you really just want to consume something that runs on top of the hardware? Right? Let's take an example. If I want to generate images with stable diffusion, for example, for my mobile phone app or something like that, it might be easier to go to a SaaS company like Replicate, for example, that essentially will host the model for you, where you just pay for for access to the model, and they send you back the generated images. And they will manage all the provision of compute infrastructure and will find the GPUs for you. If you do want to run your own model, I think my number one advice would be to shop around, right? There's a fair number of providers. The large clouds, in my experience, are not always the best option, right?
7:20 Speaker 00 If you price it out, we've seen that the startups typically are more likely to go with specialized clouds like CoWeave or Lambda, right? That are basically specialized in providing AI infrastructure to startups. Shop around, look at the different offers, compare prices.
7:34 Steph Smith? And when you're shopping around, in addition to price, which I feel like is a major motivating factor, what other factors are there in terms of these other companies who maybe aren't the big clouds? How are they differentiating relative to one another? How are they standing out in that market?
7:50 Speaker 00 There's a whole sort of decision tree there. But the first thing is, one thing that often drives the decision is how much memory do I need in my cards, right? If I have a small image model, right? I might be able to work with a more consumer-grade card, which is much cheaper, right, per hour if I reserve it in a cloud. Whereas if I, for example, train a very large language model, I not only need a card with the most memory I can find, but I probably want to have as many cards as possible in one server, because communication between them matters. And I may even care about the networking fabric behind it, right? Some of the very large models, you actually network constraints in terms of how quickly you can train them. So it really becomes a question of what's your objective? Is it inference, is it training? If it's training, how big is your model, right? And based on that, you figure out what the card is, what kind of server you need, what kind of fabric you need between those servers.
8:35 Speaker 00 And then you sort of can decide what the right fit is for your application.
8:39 Steph Smith? Even prior to this AI wave, compute was a major line item for many software companies. And the calculus of leaning on the easily accessible cloud versus bringing infrastructure in-house was becoming an increasingly important consideration. Here is Guido touching further on that very calculus in today's era and where scale comes into play. Compute is expensive. It's a major line item for many companies. And this is even before the AI revolution, you could say.
9:07 Speaker 00 Even more so today, yeah.
9:08 Steph Smith? So how do you think about, again, like how that impacts different companies' bottom lines? And whether they really factor that in to having their own allocated GPUs versus using something more like replicate?
9:19 Speaker 00 You really have to figure out what is the right fit for you. And it probably depends a lot on the scale at which you need them, right? If you need a lot, frankly, you have to pre-reserve them. You have to have your own. There's just no way around that. But if you need a smaller quantity, you may be able to reserve them on a more short-term basis. Or you have various models where you can consume only while your application runs, but at a higher price, right? And so this really comes down to what kind of load do you have? Well, we're typically seeing if somebody is training, they're more likely to do a long-term reservation for a GPU because you want to make sure you have access to it. If somebody has more continuous workloads where availability is important. If I just do inference, but I want to make 100% sure that if a request comes in, I can service it. I can never be down, right? They probably need to reserve capacity as well, right?
10:04 Speaker 00 On the other hand, if I have more batch jobs where it's like, ah, this job runs an hour later. That's not the end of the world. Then you probably can go with variable capacity and just preserve it ad hoc. But it's really a conversation of what is your usage pattern? What's your demand pattern? And from that comes the best pick for the partners that you work with.
10:21 Steph Smith? We've seen that companies, even prior to AI, have benefited from building their own infrastructure by basically bringing that in-house. Because before that, they were renting and they were paying a lot to rent that compute.
10:34 Speaker 00 Yeah.
10:34 Steph Smith? Do you think that will be a differentiator for companies moving forward? Or how should founders be thinking about that relationship between owning the infrastructure and renting it?
10:45 Speaker 00 Owning the infrastructure comes with cost as well, right? Because you need to now hire people that run it, right? You need to get money for the CapEx and so on. So my guess is that most early-stage founders, and probably even most mid-stage and late-stage founders, are better off by renting capacity, renting a cloud or using, consuming a SaaS service, right? There's a couple of exceptions. If you have really, really specialized needs, right? You may just not find anybody who has exactly the kind of hardware that you need, right? There might be some cases where you have geopolitical concerns. Your data is just too sensitive. You need to run your own data center. And there's probably a certain scale where it makes sense for you to run your own data center. But it's a pretty large scale. If you're spending $10 million a year, you're probably still under critical, right? If you're spending $100 million a year on infrastructure,
11:30 Speaker 00 that may be a reason to look into options for your own data center.
11:33 Steph Smith? But if everyone is competing for the same compute, are there other ways to stand out? Where's the moat here? You could say a moat is getting access to different training data. But that actually doesn't necessarily have to do with compute or money being thrown at the problem. It's getting access to differentiated data.
11:53 Speaker 00 If you have access to differentiated data, that could be a moat. I mean, it's a bit more subtle because, look, if you had an area where there's just not much public training data, that's probably right. There might be areas like in finance or so where that's the case. But for example, for a large language model, it turns out that just making a larger model and training on more data has more benefits than just absorbing more knowledge. It also means that it's better in reasoning and understanding abstract context and answering really complex multi-stage questions and so on. So probably, if I have to guess, I think the future will be that we'll still train on all the data we can find, right? And then maybe you fine-tune, meaning you're subject to some additional training on a particular problem domain with your private data, if that makes sense.
12:35 Steph Smith? Right.
12:35 Speaker 00 And so you first go to elementary school to learn reading and writing, and then you go to your vocational training for the specialized job that you have to do in the future.
12:43 Steph Smith? Another important question worth addressing is who can realistically compete? If compute is expensive, will all the largest, most heavily capitalized companies win? Says they can build the largest models with the most data? Or what role does open source play? As one of many emerging examples, Vicunia was created by fine-tuning Meta's Llama-1 model for chat.
13:07 Steph Smith? The cost of fine tuning added only an additional $300, but the result is competitive with much larger models like ChatGPT or BART. So what might this example and a growing number of open source projects tell us about the future of open LLMs?
13:26 Speaker 00 So first of all, in general, larger models, if they're everything else being equal, perform better. And so the really small open source models that we're seeing out there today, they're not yet at a level of a GPT 3.5 or GPT 4. And there's actually a website that runs sort of regular bake offs where they basically ask users to prepare answers. And it's it's used to be pretty clear that the large ones are still a little bit ahead. That said, we're making big advances there and we're figuring out a couple of things. So one thing we've learned is there's something called the Chinchilla scaling laws that basically give us an idea how does data correspond to model size? And if we overtrain, so don't train as sufficiently as we could, you can actually get potentially a smaller and better model, right? So you can can match the performance of a large model with a smaller model if you
14:11 Speaker 00 train it more, right? So that's interesting. That reduces model sizes. And the trend at the moment is to make slightly smaller models and train them more to get equal performance. The second thing is that when we talk about models, there's models for slightly different purposes, right? You have the base large language models, all they're trained in practically speaking is completing text, right? Literally, how you train them is you give them text and say, guess the next letter. And then you tell them, nope, that was wrong. Or yes, that was right. I didn't, and basically back propagate the how they predict. And they're really good at that, completing text, right? That's not quite the same that you want from a chatbot or from a model that you can tell to do something. So there's usually another step afterwards, which is called fine tuning for instruction following or for chat specifically, where basically I tell a model, look, if somebody
14:55 Speaker 00 asks you to come up with a list of to do items, like a list of steps, how to make pizza, right? This is roughly what I expect you to answer, right? These models are very good in learning these things. So whether you first train them, just complete text, and then you train them. How to react to human requests and instructions, right? It's called instruction fine tuning. And so Lama, for example, that was a Facebook model where they published the weights for researchers. And then some people took that and they fine tuned it, meaning they took a bunch of instruction findings to turn it into alpaca or ikunya, which is a much, much nicer model in terms of interacting with it, right? For humans, much, much more useful. And so the biggest challenge at the moment we have on the open source side is there's currently no large open source L L M out there, right? G P T three also one hundred seventy five billion parameters.
15:39 Speaker 00 There's currently nothing in that weight class that's open source and that people could use to fine tune or to play with or to modify.
15:47 Steph Smith? It is worth noting that since this recording, several more open models have been released, including Lama two with seventy billion parameters and an open license, unlike its predecessor, Lama one. Another forty billion parameter open source model Falcon was released as well. Both of these are still dwarfed in parameters compared to closed models like OpenAI's G P T three at a hundred and seventy five billion parameters or G P T four at an estimated one point eight trillion parameters, although the latter is speculated to be a collection of multiple smaller models. However, parameter count is not the only driver of performance.
16:27 Steph Smith? For example, while Lama two has fewer models than G P T three, its performance is actually much better due to being trained on more data. In fact, Lama two is currently comparable to G P T three's successor, G P T three point five, the current default of Chat G P T. And as many of these models continue to get larger, we may see some models compress, becoming more efficient and enabling inference on your device. You already mentioned stable diffusion can run on your computer's G P U. Do we expect to see more of that? Because right now they are all hosted by these companies, right? They're trained by these companies on their dedicated servers. And then even if you interface with Chat G P T.
17:11 Steph Smith? It's running that inference for you. Do we expect to see that change at all as compute becomes cheaper, maybe more decentralized? Or how would you think about that?
17:20 Speaker 00 That's a really good question. And we're speculating a little bit here, but my guess is we will, right? And we're seeing some of these smaller models getting pretty good. They run on your laptop or even your phone. We're starting to see stable diffusion implementations that run well on phones, right, which I would have never thought, right? And they take a couple of 10 seconds to create an image, which is comparatively slow, but there's certain applications where that's acceptable. So my guess is as both the devices get faster and the models get more optimized, right, this will be a trend that we see more and more. And in the future, it might just be part of the operating system to have a large language model, a large language generation model.
17:56 Steph Smith? Maybe I'm off base here, but we've talked about how expensive compute can be and how ultimately that can be a major line item for companies. And I guess probably the model training will remain with those companies and not necessarily on folks' devices. But in terms of the inference, I assume that's still a pretty significant cost. And in a way, if if someone is able to run that locally, doesn't that disjoint the company from having to pay for that compute? Because it's running on, let's say, someone's MacBook GPU?
18:27 Speaker 00 Oh, yeah, totally. I mean, look, if I can generate an image from my phone directly, all it takes is some battery power and it gets a little warm, right? And that's it, right? So that's a huge advantage. At the same time, there's probably going to be a little bit bifurcation there around quality and parameters, right? You can run things locally, but you can probably run them a lot better in the cloud, right, because you have a much bigger server there. So it probably depends a little what you want to do, right? If I just want to have a better spell checker that checks my email or maybe just some simple completion, that's perfectly fine. I can run that on my phone. On the other hand, if I want something that is more write a good speech or summarize a complex text, they might be like, oh, that I'm going to run the cloud because it takes so many more operations.
19:04 Steph Smith? Hopefully, this is getting your wheels spinning in terms of what can be built. And here is Guido speaking to how this presents a fundamentally new stack and what that means in terms of opportunity.
19:17 Steph Smith? It feels like this really is like this massive wave, this renaissance of innovation.
19:21 Speaker 00 It's full of opportunities, right? I mean, we're rebuilding a stack. You can look at AI just as a new application, but honestly, I think it's probably a better way to look at as a different type of compute, right? We traditionally build software by composing algorithms in a way that we understand well. And where the end result was programmed or sort of bottoms up constructed. Now we have a second type compute where we just train a large neural network. And the big advantage is we don't actually need to know how to solve a problem. As long as the network can figure it out, right? The neural network can figure it out, we're fine. And that opens up a bunch of new applications, but it also means you need a completely different stack in terms of all the different pieces, right? You probably want vector DBs to retrieve context. You you want different types of hosting providers that are good in hosting these models and providing them to us as a service.
20:06 Speaker 00 Right? It's it's a whole like Cambrian explosion of creativity, has a whole new ecosystem forming. And I think there's a ton of opportunities to to build right companies.
20:15 Steph Smith? I think that paints a pretty incredible picture of opportunity across the stack. And as many of these trends continue to progress, like supply and demand, the calculus of renting versus owning compute, closed versus open source models, we look to part three of the series to answer a very important question. How much does all of this cost? We'll explore all this in depth, including how much startups are really spending on AI compute and whether that's sustainable, how much it really costs to train a model like GPT three, the difference in cost between training and inference, and how all of this will change with time. We'll see you there.
20:58 Steph Smith? Thank you so much for listening to part two of our AI Harbor series. We spent a lot of time trying to get these episodes right, so if you are enjoying them, go ahead and leave a review or tell a friend. We'll also have an animated video version up on our YouTube channel soon. But for now, you can find some of our recent videos, like my conversation with Waymo's Chief Product Officer, Ina Waymo, or a conversation I recently had at the Aspen Ideas Festival where we discussed the Classroom of 2050. As always, thank you so much for listening.
Previous Wartime vs Peacetime: Ben Horowitz on Leadership Next A True Second Brain Page updated September 29, 2026 · transcript and summary are machine-generated
