Pod Insights a16z Podcast The True Cost of Compute About this episode With software becoming more important than ever, hardware is following suit.
As the world generates more data, unlocking the full potential of AI means a constant need for faster and more resilient hardware.
But how much does this all really cost? In this final segment of our AI hardware series, we tackle that question head on.
Be sure to check part 1 and 2, where we explore the emerging architectures and the momentous competition for AI hardware.
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:01 Speaker 00 There's very few computational problems that complex that mankind has actually undertaken.
0:06 Steph Smith? How do you think about the relationship between compute, capital, and then the technology that we have today?
0:13 Speaker 00 Yeah, that's the million-dollar question or maybe trillion-dollar question. The expectation at the moment is that the cost of training these models may actually sort of top out or even go down a little bit as the chips get faster, but we don't discover new training material as quickly.
0:31 Steph Smith? With software becoming more important than ever, hardware is following suit. And with the world constantly generating more data, unlocking the full potential of AI means a constant need for faster and more resilient hardware. But how much does all of this really cost? In this final segment of our AI hardware series, we tackle that question head-on. But if you're just catching up, be sure to check out Part 1 and Part 2, where we explored the emerging architectures and the momentous competition for AI hardware. And today, we're joined again by A16Z special advisor Guido Appenzeller,
1:11 Steph Smith? someone who is uniquely suited for this deep dive as a storied infrastructure expert with experience like...
1:18 Speaker 00 Intel's data center group dealing a lot with hardware and the low-level components. So it's given me, so I think, a good insight how large data centers work, what the basic components are that make all of this AI boom possible today.
1:31 Steph Smith? Here is Guido touching on the reality of these models and how much they cost today.
1:36 Speaker 00 Training one of these large language models today is not a $100,000 thing. It's probably millions of dollars thing. Practically speaking, what we're seeing in industry is that it's actually more of a tens of millions of dollars thing.
1:50 Steph Smith? As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures.
2:20 Steph Smith? In Guido's recent article, "Navigating the High Cost of AI Compute", Guido even noted that access to compute resources has become a determining factor for the success of AI companies. And this is not just true for the largest companies building the largest models. In fact, many companies are spending more than 80% of their total capital raised on compute resources. So naturally, this begs the question: is this really sustainable?
2:49 Speaker 00 The core technology that you're building in your early days towards more a complete product offering, right? There's just a lot more boxes to check and features to implement and all the administrative parts of your application if you're getting to the enterprise. So probably you'll have more normal software development that's not AI, right, or classic software development happening. You'll probably also have a larger headcount of people that they have to pay. So at the end of the day, I would expect as a percentage that to go down over time, right? As an absolute amount, I think it'll be going up for some time just because this AI boom is still just in its infancy.
3:21 Steph Smith? The AI boom has just begun, and in part two, we discussed how it's unlikely for compute demand to subside anytime soon. There, we also discussed how the decision to own or rent infrastructure can make a non-trivial difference to a company's bottom line. But there are other considerations when it comes to cost. Batch size, learning rate, and the duration of the training process all contribute to the final price tag.
3:50 Speaker 00 How much does it cost to train a model depends on a myriad of factors, right? Now, the good news is we can simplify this a little bit because the vast majority of models that are being used today are transformer models, right? That was the transformer architecture, huge breakthrough in AI. They've proven to be incredibly versatile. They're easier to train because they paralyze a little bit better than previous models. And so in a transformer, you can sort of approximate the inference time as twice the number of parameters, floating point operations, right? And the training time is about six times the number of parameters. So if you take something like GPT-3, right, which is OpenAI's big model, they have 175 billion parameters, so you need twice as much. So 350 billion floating point operations to do one inference.
4:32 Speaker 00 And so based on that, you can sort of figure out how much compute capacity you need, how this is going to scale, how you should price it, you know, how much it will cost you at the end of the day. This also gives you for model training, an idea how long the training is going to take. Right. You know how much your AI accelerator can do in terms of floating point operations per second. Right. You can sort of theoretically calculate how many operations it is to train your model. In practice, the math is more complicated because there are certain ways to accelerate that. So maybe you can train with a reduced precision. But it's also very hard to achieve 100% utilization on these cards. If you naively implement it, you're probably going to be below 10% utilization. But you know, you can probably get into the tens of percent with a little bit of work.
5:15 Speaker 00 This gives you a rough swag, how much capacity you need for training and for inference. But at the end, you probably do want to test it before you make any final decisions on these things. Make sure that your assumptions hold on how much you need.
5:27 Steph Smith? Now, if all those numbers confused you, that's okay. We'll walk through a very specific example. GPT-3. GPT-3 has about 175 billion parameters. And here's Guido on the computation requirements for training the model. And ultimately inference, that's when you're prompting the already trained model to elicit a response.
5:48 Speaker 00 So if you just do very naively the math, right, let's start with training, right? We know how many tokens it was trained on, we know how many parameters the model has. So we can do soft napkin math. And you end up with something like three times 10 to the 23 floating point operations. That's a completely crazy number, right? It's like a number with 23 digits, right? It's like hard to write down. There's very few computational problems that complex that mankind has actually undertaken, right? It is a huge effort. Then you can be like, okay, so let's take, say, an A100, right? The most commonly used card. We know how many floating point operations it can do per second. We can divide that, right? That's give us an order of magnitude estimation, like how much time it would take, right? And then we know how much one of these cards costs, right?
6:29 Speaker 00 Like renting an A100 costs you between, I want to say between $1 and $4 probably, right? Depending on who you rented from. And you end up with something in the order of half a million dollars, right? With this very naive analysis. Now, there's a couple of things there, right? We didn't take into account optimization. We also didn't take into account that you probably cannot run this at full capacity because of memory bandwidth limitations and network limitations. And last but not least, you probably need more than one run to get this right. You probably need a bunch of test runs, they're probably not going to be full runs and so on. But this gives you an idea that sort of training one of these large language models today, it's not a hundred thousand dollar thing, it's probably millions of dollars thing. Practically speaking, what we're seeing in industry is that it's actually more of a tens of millions of dollars thing. And that's because you need the reserved capacity, right?
7:14 Speaker 00 So if I could get all my cards for the next two months would only cost me a million dollars. But the problem is they want a two-year reservation. So really, the cost is 12 times as high. And so that basically adds a zero to my training cost.
7:28 Steph Smith? Right, and how does that compare to inference?
7:31 Speaker 00 So inference is much, much, much cheaper. Basically, my training set for a modern text model, for example, the training set is about a trillion tokens, right? And if I run inference, each word that comes out is one token, right? So it's a factor of a trillion or so faster on the inference part. If you run the numbers like a large language model, you actually add a fraction of a cent, like a tenth of a cent or a hundredth of a cent somewhere in that ballpark for the inference. Again, if we just naively look at this, right? For inference, your problem is usually that you have to provision for peak capacity, right? So if everybody wants to use your model at 9:00 AM on a Monday, right? You still have to pay for Saturday night at midnight when nobody is using it. That increases your costs substantially there. For some of them, specifically image models, what you can do for inference is that you use much, much cheaper cards.
8:15 Speaker 00 Because the model is small enough that you can run it on essentially the server version of a consumer graphics card. And that can save you a lot of cost.
8:23 Steph Smith? And unfortunately, as we discussed in part one, you can't just make up for these inefficiencies by piecing together a bunch of less performant chips, at least for model training.
8:34 Speaker 00 You need some very sophisticated software for that, right? Because the overhead of distributing the data between these cards would probably outweigh any saving you get from cheaper cards.
8:43 Steph Smith? Inference, on the other hand?
8:45 Speaker 00 For inference, you can often do the inference on a single card. So that's not really a problem. If you take something like stable diffusion, right? A very popular model for image generation that runs on a MacBook, for example, that has enough memory and enough compute power so it can generate an image locally. So that'll run on a relatively cheap consumer card. And you don't have to use an A100 for it to do inference.
9:05 Steph Smith? So when we're talking about the training of the models, clearly, the amount of compute is just drastically more than the inference. And something else that we've already talked about is the more compute, often, not always, but often, the better model. And so does this ultimately, these factors all ladder up to the idea that heavily capitalized incumbents win? This race or this competition? Or how do you think about the relationship between compute, capital, and then the technology that we have today?
9:34 Speaker 00 Yeah, that's the million dollar question or maybe trillion dollar question. I don't know. So first of all, training these models is expensive, right? For example, we haven't seen yet really good open source large language model. And I'm sure part of the reason is that training these models is just really, really expensive. Right? I mean, there's a bunch of enthusiasts out there that would love to do this, but you need to find a couple of million or 10 million dollars of compute capacity to do it. And that makes it so much harder, right? It means you sort of need to create a substantial effort before something like that can happen. All that said, the cost for training these models overall seems to be coming down. And in part, I think it is because it seems to me like we're becoming data limited, right? So it turns out there is a correspondence between how big your model is and what the optimal amount of training data is for the model.
10:18 Speaker 00 Having a super large model with very few data doesn't get you anything. Or having a ton of data with a small model also doesn't get you anything, right? Your software, the size of your brain needs to roughly correspond to the length of your university education here, right? Otherwise, it doesn't work. And what this means is that because some of the large models today already leverage a good percentage of all human knowledge in a particular area. I mean, if you look at GPT, it was probably trained on something like 10% of the internet, right, and all of Wikipedia and many books, like a good chunk of all books, right? So going up by a factor of 10, yeah, that's probably possible. Going up by a factor of 100, that's not clear if that's possible. I mean, we as mankind just haven't produced enough knowledge yet that you could absorb all of that into one of these large models.
11:02 Speaker 00 And so I think the expectation at the moment is that the cost of training these models may actually sort of top out or even go down a little bit as the chips get faster, but we don't discover new training material as quickly. I mean, unless somebody comes up with a new idea to generate training material. And so if that assumption is true, I think this means that the mode that's created by these large capital investments is actually not particularly deep, right? It's more of a speed bump than something that prevents new entrants. I mean, today, training a large language model is something that is definitely within reach for a well-funded startup, right? So for that reason, we expect to see more innovation in that area in the future.
11:45 Steph Smith? All right, that is a wrap for our AI hardware series. We genuinely hope you came away with a little more knowledge about this increasingly important space. Because if software is indeed eating the world, well, hardware is coming along for that ride. And as a reminder, if you haven't yet listened to Part One, where we explore the emerging architectures and who's creating them, or Part Two, where we dive into the future AI stack and how founders can participate, well, those are already live and ready for consumption. And as always, thank you so much for listening. We'd actually like to leave you with a fun fact from GPT-4 itself, commenting on the technology that created it.
12:26 Steph Smith? And yes, we did fact-check this, and this is also AI-generated audio from Eleven Labs. We'll see you next time.
12:33 Speaker 02 ChatGPT and its sibling models are trained on diverse internet text. However, the exact amount of data used can be hard to comprehend. If we were to print all of the data used to train these models, it could fill a large library.
12:49 Speaker 03 Consider that one single book may contain around 1 million characters. If we estimate that the training data is hundreds of gigabytes of text data, let's take a conservative estimate and say it's 100 gigabytes.
13:02 Speaker 04 Considering that one character is approximately one byte, this would mean the model was trained on approximately 100 billion characters. If each book has 1 million characters, then the data used to train ChatGPT is equivalent to the text in approximately 100 million books. If we take the size of a large library...
13:21 Speaker 05 If we take the size of a large library, such as the New York Public Library, which has around 53 million items, not just books, the training data is equivalent to the text in almost twice the number of items in that library.
13:32 Steph Smith? Thanks, ChatGPT. A quick note to close out, that many models today are even bigger, with Llama 2, for example, being trained on 2 trillion tokens, or about 8 trillion characters. Now that is a lot of libraries.
13:48 Steph Smith? Thank you so much for listening to our full AI hardware series. We spent a ton of time trying to get these episodes right. So if you did enjoy them, go ahead and leave a review or tell a friend. We would so appreciate that. And you can also look forward to a video animated version of these up on our YouTube channel soon. But for now, you can find some of our recent videos there, like my conversation with Waymo's Chief Product Officer, Ina Waymo, or a conversation I had at the Aspen Ideas Festival, where we discussed Classroom 2050. As always, thank you so much for listening.
Previous Comfort Food, Climate, and the Future of Our Plates Next From Big Bang to James Webb: Exploring Space with Nobel Laureate John Mather Page updated September 30, 2026 · transcript and summary are machine-generated
