Reiner Pope – The math behind how LLMs are trained and served

Did a very different format with Reiner Pope - a blackboard lecture where he walks through how frontier LLMs are trained and served. It’s shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk. It’s a bit technical, but I encourage you to hang in there – it’s really worth it. There are less than a handful of people who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him. Recommend watching this one on YouTube https://youtu.be/xmkSf5IS-zw so you can see the chalkboard. Reiner https://reiner.org/ is CEO of MatX https://matx.com/, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software https://arxiv.org/abs/2211.05102 efficiency https://jax-ml.github.io/scaling-book/, compilers, and TPU architecture. Download markdown of transcript here https://gist.github.com/dwarkeshsp/79100f0fdeed69d76241903bb0604dbe to chat with an LLM. Wrote up some flashcards and practice problems https://reiner-flashcards.vercel.app/ to help myself retain what Reiner taught. Hope it's helpful to you too! Sponsors * Jane Street https://janestreet.com/dwarkesh needs constant access to incredibly low-latency compute. I recently asked one of their engineers, Clark, to talk me through how they meet these demands. Our conversation—which touched on everything from FPGAs to liquid cooling—was extremely helpful as I prepped to interview Reiner. You can watch the full discussion and explore Jane Street’s open roles at janestreet.com/dwarkesh https://janestreet.com/dwarkesh * Google’s Gemma 4 https://goo.gle/Gemma4 is the first open model that’s let me shut off the internet and create a fully disconnected “focus machine”. This is because Gemma is small enough to run on my laptop, but powerful enough to actually be useful. So, to prep for this interview, I downloaded Reiner’s scaling book, disconnected from wifi, and used Gemma to help me break down the material. Check it out atgoo.gle/Gemma4 https://goo.gle/Gemma4 * Cursor https://cursor.com/dwarkesh helped me turn some notes I took on how gradients flow during large-scale pretraining into a great animation. At first, I wasn’t sure the best way to visualize the concept, but Cursor’s Composer 2 Fast model let me iterate on different ideas almost instantaneously. You can check out the animation in my recent blog post https://www.dwarkesh.com/p/what-i-learned-april-15. And if you have something to visualize yourself, go to cursor.com/dwarkesh https://cursor.com/dwarkesh Timestamps (00:00:00) – How batch size affects token cost and speed (00:32:09) – How MoE models are laid out across GPU racks (00:47:12) – How pipeline parallelism spreads model layers across racks (01:03:37) – Why Ilya said, “As we now know, pipelining is not wise.” (01:18:59) – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal (01:33:02) – Deducing long context memory costs from API pricing (02:04:02) – Convergent evolution between neural nets and cryptography This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dwarkesh Patel?

  Today, I'm interviewing Reiner Pope, who is CEO of Maddox, which is a new ship startup. Previously, he was doing TPU architecture and many other things at Google. This is a very different format from my usual interviews. This is going to be a blackboard lecture. We're going to get up in a second. We, in fact, built this whole new studio with specifically this format in mind. And so it's a pleasure to get to inaugurate it with you. We're going to be talking about model architecture, ML inference, and many other things. And the reason I think it's an important topic is because once you actually understand how training and inference actually work in a cluster, as we'll see, a lot of things about why AI is the way it is, why AI architectures are the way they are, why API prices are the way they are, fundamentally, also, how, why AI progress is the way it is, start making sense.

• 0:45Dwarkesh Patel?

  And you need to understand the details to get there, and you need a blackboard to understand the details. So, Reiner, thank you so much for doing this.

• 0:50Speaker 01

  Yeah, very happy to be here.

• 0:52Dwarkesh Patel?

  Just a heads up, this is a lecture with graphs and equations and all that stuff. So if you can, I would really recommend watching it on a video platform like YouTube. Okay, full disclosure, I am an angel investor in Maddox, but that's unrelated to this podcast.

• 1:08Dwarkesh Patel?

  Reiner, maybe to kick us off, I'll ask this question. So, we have a couple of companies like Claude and Codex and Cursor are offering something like Fast Mode, where for 6x the price, they'll give streaming tokens at 2.5x the speed. Mechanically, I'm curious what's going on here. Why, like, why is it the case that you can pay more to get faster latency? And two, could you keep going? Could you pay 100x more and somehow get even faster speeds? Or much, much faster speeds? And three, could you go the other way? Could you have something like Claude Code Slow Mode, where if you are willing to wait for minutes on end, you could get even cheaper prices? So maybe this will help motivate the kind of analysis that you'll be doing through the lecture.

• 1:49Speaker 01

  Great. I mean, to jump to, a little bit to jump to the conclusion, the big effect is batch size. But what we're going to do now is quantify exactly what that looks like and what its implications are on latency and cost. There's going to be another effect, which is, you can call it speculative decoding or multi-token prediction. We can maybe come back to that later, but I think the first thing that we'll talk through is batch size. So what I'd like to introduce is the two principles of analysis. Firstly, we're going to look at a roofline analysis of how we run a transformer model on a cluster of chips. We'll take a, let's say, a Blackwell NVL 72 cluster, so a rack of 72 GPUs. And so the roofline analysis means we look at memory bandwidth and compute performance.

• 2:34Speaker 01

  And then the other side of that is that we're going to look at just two simple factors of the model, which are the time to operate on the weights and then the time to operate on the context, the KB cache. So let's jump in. What we're going to try and do is we're going to try and estimate the time that it takes to run an inference of a certain shape. Now, we're not perfect here. We can't exactly predict the time. And so instead, we're going to approximate. And so we're going to say that the time must be greater than or equal to a certain quantity. And so we're going to consider two different aspects. We're going to look at the time for it takes to do the memory fetches and then the time it takes to do the compute. And it'll turn out that this actually gives us a very strong predictive power, even with

• 3:19Speaker 01

  a simple model. So one by one, what is the time that it takes to do the compute? So there are really two things I need to do in the compute. I need to multiply by all of the active parameters, and then I need to do some work on the attention. So multiplying by all the active parameters. I have a certain batch size that I'm running, and then I've got a number of active parameters in my model. And then I'm just going to divide this by the compute throughput, which is the flops of the chip. So this is hardware concern. So this actually accounts for all of the compute time for all of the weight matrix multiplies.

• 4:04Speaker 01

  There's a little caveat here. We've sort of ignored the time to do any of the attention computation, but that in general will be quite small in comparison to this. So we'll ignore this.

• 4:12Dwarkesh Patel?

  Maybe I'll just interrupt from time to time to ask some very naive questions or to clarify some basic points, but. Just for the audience, you're not serving one user at a time. The batch refers to the fact that you're serving many different users at the same time.

• 4:24Speaker 01

  Yeah.

• 4:25Dwarkesh Patel?

  And that's a whole batch.

• 4:26Speaker 01

  Yes, I can motivate the batch at least a little bit. So, I mean, we will see exactly why batch is such a favorable optimization. But what will turn out to be the case is that if you do not batch together many users, the cost and the economics you get is can be like a thousand times worse than if you do batch many two users together.

• 4:45Dwarkesh Patel?

  Yeah.

• 4:45Speaker 01

  And we'll be able to see that quite explicitly. And then a number of active parameters, this is saying like if I think look at it, for example, a deep seek model, the deep seek B3 model has about 37 billion active parameters and then 700 billion total parameters. So this is we're focusing on just the ones that are active for a single token. Okay, so we're modeled to compute performance. I'm going to keep writing equals, but in all of these cases, you can think of this time as being at least this much. And maybe there will be some terms we ignored.

• 5:16Speaker 01

  On the memory side, what do we need to do with memory? We need to fetch, we need to fetch all of the weights. And so there is some time to fetch all of the the total number of parameters, not just the active parameters.

• 5:30Speaker 01

  So there's wait fetch time. And then in addition, there's a KV cache fetch time. So there is, this actually depends on batch size. So for every element of the batch, we have to fetch an entire context length

• 5:46Speaker 01

  worth of tokens. And then there's a size per token, so like bytes, bytes for one token. And so that is a model parameter.

• 5:56Dwarkesh Patel?

  And maybe just back in, let's just explain what the KV cache is real quick.

• 6:01Speaker 01

  Yeah, so when I do a forward pass, let me draw actually a how the auto aggressive inference works. So this is during decode.

• 6:10Speaker 01

  So if I think I have a bunch of tokens of text, I'm drawing a tensor because ultimately the tokens are represented as some like tensor of in some embedding dimension. And then in this direction, I have the sequence length.

• 6:27Speaker 01

  The work of running a decode is I have to run each token through a through a whole bunch of matrix multipliers over a bunch of different layers. And I have, in general, I'm going to have to do that work over all of these tokens. But then one step of decode is actually to produce just this one additional token. Yep. And so what I'm going to do there is I'm going to run a full forwards pass of multiplying by all of the weight matrices in the entire model. But then I've got this attention mechanism where this token sort of it's like looking at all of the past tokens in this way. And what is it looking at specifically?

• 7:10Speaker 01

  It is looking at some internal representations that the model has produced of the tokens. And we call that the KB cache. So this process of attending this, this single token attending to all of the history of tokens. That's attention. It is mostly dominated by memory fetches rather than the matrix multipliers. So we've got the amount of memory that we're fetching shown over here. And then there's of course just then divided by the memory bandwidth. So, so the memory bytes per second. So in fact, this, these equations here are actually enough for us to now draw some fit lines. And so the things that we'd like to look at are sensitivity to batch.

• 7:54Speaker 01

  And then also, which we'll draw separately to context links. So we said that the big, big effect you can get is like some, some trade off in latency versus versus cost in batch size. So let's draw them out. I think there's just really two graphs we want to draw. We'll first just draw batch size versus time here. So when we look at the shape of this, we've got a maximum of, well, the sum. And then, and then another term. So let's look at these terms one by one and how they scale the time for compute and, and memory and how they show up.

• 8:36Speaker 01

  So let's first look at this compute time. This is just purely linear, linear in batch size with no, no offset. So it is some code like this, this is, this is t compute.

• 8:53Speaker 01

  And then on the memory side, we've got some portion here that is just this constant that is constant in some base offset here, which is the wait fetch. Wait fetch.

• 9:09Speaker 01

  And then finally, we have this term here, which is the KB fetch, which we're going to draw as

• 9:19Speaker 01

  this is the KB fetch.

• 9:23Speaker 01

  Which is which is linear in batch size. So it looks like that. So the sum of this plus this maxed with this. So let's at least first draw the sum.

• 9:38Speaker 01

  So the two memory times in conjunction end up looking on this curved slope like this.

• 9:44Speaker 01

  And then we get a the overall maximum is I'll draw a little thicker here is the maximum of all these two curves.

• 9:52Dwarkesh Patel?

  Makes sense.

• 9:54Speaker 01

  Okay, so what does this mean actually? So this is a latency plot.

• 10:02Speaker 01

  So if I grow my batch size, I get initially some not very strong dependence on batch size. And so there is some lower bound on latency here, latency lower bound.

• 10:15Speaker 01

  Lower bound.

• 10:20Speaker 01

  So this already partially answers the question for a given hardware configuration. And then we can talk about varying hardware configuration. But for a given hardware configuration, there is a lower bound on latency, which is simply the I need to read all of my total parameters from from memory into the into the chips. And that takes us a certain amount of time. If I use all of my memory bandwidth, I can't do any better than that.

• 10:44Dwarkesh Patel?

  It's even the way you've drawn the slopes for Compute time and how the KV grows. And what implication the KV has on memory time. That.

• 10:56Speaker 01

  Yeah, what if this were above or below?

• 10:59Dwarkesh Patel?

  Yeah, or is that necessarily the case? Because if this is always true, then as batch size grows, compute always dominates KV, and which suggests that if you have a big enough batch size, maybe memory is never an issue.

• 11:11Speaker 01

  Yeah, this is really sensitive to the context length. So I think we should come back and explore this. The there will be, as you vary the context length, the KV fetch time will go up and up. And so that will cause a transition from compute limited to memory limited.

• 11:24Dwarkesh Patel?

  And is there something especially significant about the slope being exactly the slope of the compute time?

• 11:34Speaker 01

  Yeah, whenever we have balance points, it kind of says that you're getting it exactly right. And so for the particular context length where the slopes match, that says I am equally memory bound and compute bound, which is a really desirable place to be.

• 11:47Dwarkesh Patel?

  But suppose it's like,

• 11:50Dwarkesh Patel?

  this is a very simple algebra problem. But suppose it's, you know, the optimal is 100K context length. And you go to 200K context length. Does your MFU go down to like 50%? Like, does it have a humongous impact on MFU?

• 12:06Speaker 01

  Yeah.

• 12:06Dwarkesh Patel?

  To be like slightly outside of context length optimal range, Goldilocks zone?

• 12:10Speaker 01

  That's right. So that is true as modeled here. There's a key point here that I'm modeling this context length as, or I'm modeling the memory fetch as linear in context length. That actually depends on model architecture. It is true for many of the, or all of the model architectures with dense attention.

• 12:28Dwarkesh Patel?

  Yeah.

• 12:28Speaker 01

  There's a sparse attention actually scales much better than that.

• 12:31Dwarkesh Patel?

  Got it. And is sparse attention that everybody uses in practice?

• 12:35Speaker 01

  I'm pretty excited about sparse attention. It's hard to know what the labs are using. DeepSeek has published a sparse attention mechanism. I'll just like put a plug in that sparse attention. Some of the DeepSeek papers that have published spust attention end up putting a square root in this term. Okay, so so far we've done, we've looked at the latency. It's kind of hard to read off cost from this. So if I think, what does cost mean?

• 12:58Speaker 01

  I'm going to like, to run this inference, I'm going to use the GPU for a certain number of seconds, like 1 millisecond or 20 milliseconds or something like that.

• 13:06Speaker 01

  And I have to pay the rental time for that for that time. So like it's $2 an hour per GPU or something like that.

• 13:13Speaker 01

  So so that's the cost of this inference. But how much value have, how many tokens have I processed during that inference? That is the batch size. And so what we actually want to plot is going to be the the cost versus batch size, which is like T over B versus batch size.

• 13:33Speaker 01

  This is the cost per token.

• 13:37Speaker 01

  So like we have to imagine dividing each of these three curves by by B. So multiplying by this reciprocal. And so what we end up with there is the compute curve is gonna. It was linear, we divide by B, that makes it a constant here. And this is T compute.

• 14:02Speaker 01

  The the KV fetch was linear, now it becomes a constant as well.

• 14:09Speaker 01

  KV fetch.

• 14:13Speaker 01

  And then the

• 14:17Speaker 01

  the the weight fetch was constant and now we're divided by B. And so it becomes this hyperbola.

• 14:31Speaker 01

  And so again, we're going to compute the the max of the sum.

• 14:37Speaker 01

  So the sum of these two terms shifts the the the parabola up, some of the KV fetch and the weight fetch gives us a sort of a a higher parabola that's like this. And then we're going to take the max with the compute.

• 14:54Speaker 01

  Here, so we end up with this, this being the overall shape that we care about.

• 15:01Speaker 01

  So again, so like we see some limiting behavior. The cost initially starts very high at batch size of one. Actually, like it almost goes to infinity. Like it's because we've got so many weight fetches, which are not amortized over a large batch size. But then, as we increase the batch size, the weight fetches become amortized over so many different batch elements that their cost grows very small. And eventually, the compute time ends up driving the cost. So there is a limiting, like lower bound, lower bound on cost.

• 15:38Speaker 01

  Which is this one here.

• 15:39Dwarkesh Patel?

  Yep.

• 15:41Dwarkesh Patel?

  So Claude code slow or Codex slow or whatever would just live on this line. And it wouldn't help much because you're, you're not able to amortize the KV values over a much bigger batch.

• 15:53Speaker 01

  Yeah, yeah, they're unique per batch. The compute is also unique per batch. And so what is the minimum work you can do per batch after amortizing everything else away?

• 16:01Dwarkesh Patel?

  So this point where you are no longer memory bandwidth bound, what practically, how big a batch do you need to? Like, how big are the batches practically for frontier models?

• 16:17Speaker 01

  You can, you can just solve for that, actually. And it's not even particularly sensitive to model architecture. So let's go ahead and do that. So what we're talking about is we're going to say when the memory time is equal to the compute time. That's, that's what that question is.

• 16:32Speaker 01

  For now, I'm going to discard the, because I focused on what the batch size is. And really, there's a question of what, when the weights are amortized over the, the multipliers. I'm going to focus on comparing the weight fetch time to the weight multiply time. I'm going to disregard the KB fetch term, just to simplify the analysis. So we can get a kind of a clean answer out. So we're going to equate

• 16:59Speaker 01

  this portion with this, with these two terms.

• 17:03Dwarkesh Patel?

  Yep.

• 17:06Speaker 01

  So writing that out, we get N number of total parameters

• 17:13Speaker 01

  over memory, memory bandwidth.

• 17:18Speaker 01

  Is equal to batch size times number of active parameters

• 17:27Speaker 01

  divided by the compute performance.

• 17:31Speaker 01

  So looking over here, everything on the top, these are model parameters. Everything on the bottom, these are hardware parameters. It turns out to be nice to rearrange them such that we have the hardware parameters on one side. So this is equivalent to

• 17:50Speaker 01

  flops over memory bandwidth

• 17:53Speaker 01

  being equal to batch size times number of active parameters

• 18:01Speaker 01

  divided by the number of total parameters.

• 18:05Speaker 01

  So this is a hardware parameter. Actually, this actually ends up being a dimensionless constant. If you look in terms of flops, what are the dimensions of this? This is multiplies per second, this is bytes per second. So that's not quite dimensionless. But what you do is you say like multiplies per second times, let's say I'm doing FP4. So I do like how many FP4 multiplies per second times the fact that each FP4 is half a byte. And so I can actually make this ending up being dimensionless. And this ends up being on most GPUs around 300.

• 18:45Speaker 01

  Somewhere around 300.

• 18:46Dwarkesh Patel?

  Is that ratio changed over time as we've gone from model generation to model generation where the flops keeps increasing?

• 18:51Speaker 01

  So this is a hardware parameter. To what extent has the hardware changed? So from like A100 to H100 to B100, the flops has increased substantially. The memory bandwidth has also increased substantially and it has remained reasonably stable. And we can express this one as well. This is a sparsity parameter. And I might even phrase it slightly different. Let's solve for batch size in total. We end up with, I'm sorry, just moving this back over to the other side. We end up with batch size needs to be bigger than approximately 300 times sparsity.

• 19:25Speaker 01

  So for example, if I have 100, like I activate in DeepSeek, I activate 32 out of 256 experts. So this would be like eight for DeepSeek.

• 19:35Dwarkesh Patel?

  Got it. Okay.

• 19:36Speaker 01

  So this actually gives you a ballpark, which is remarkably accurate to practice. Generally, people will go a little bit larger than this. They don't really want to be exactly at the balance point because real world efficiencies aren't as good as a refined analysis would say. But like, take this and maybe double it or triple it.

• 19:53Dwarkesh Patel?

  Okay, so basically it's like 2,000 to 3,000 tokens per batch. But then if you included the KB cache.

• 20:02Speaker 01

  Yes.

• 20:02Dwarkesh Patel?

  The implication would be that the optimal batch size.

• 20:07Speaker 01

  Should grow larger. So this is like, we solve for the equivalence between when compute time is equal to memory time. If I add in more memory bandwidth, like something that consumes more memory bandwidth, then I have less available for the weight loads. And so I need to grow the memory bandwidth more and therefore the batch size more.

• 20:26Dwarkesh Patel?

  This seems incredibly small. Like a batch, this would be like less than one sequence, right?

• 20:30Speaker 01

  Yeah, okay. So, so I guess this is, keep in mind that I'm talking about the number of tokens that I'm generating one more token for. So, so it's like, it's actually 2,000 unique sequences.

• 20:42Dwarkesh Patel?

  Okay, we're just talking about the, a single forward pass on these sequences. This is like, do you think of it like the batch as the number of sequences rather than like?

• 20:50Speaker 01

  That's right.

• 20:51Dwarkesh Patel?

  Okay, cool. When I'm prepping for interviews, I often talk to experts in the field. So for Reiner, I chatted with two of Jane Street's engineers, Clark and Axel. Clark, who works on low latency trading systems, walk me through why Jane Street uses FPGAs to make sure that they have predictable nanosecond latencies. You can just build these like giant grids of compute very easily that do exactly what you need to touch 100 megabytes of SRAM and then get your response back in tens of nanoseconds very easily. And that's basically impossible on simple CPU. He then went on to explain why CPUs just wouldn't work for this kind of thing.

• 21:22Speaker 01

  And so if you have a clock that's going every three nanoseconds, you actually have several bytes of information at a time to make your decision. That's as opposed to a CPU where you'll just collect up a whole packet. You know, let's say a 1,500 byte packet. And then you say, OK, this packet's ready. Here you go, CPU. You can start thinking about it now.

• 21:38Dwarkesh Patel?

  FPGAs allow you to react to the earliest part of the packet as it arrives, rather than having to wait for the full thing. We also talked about liquid cooling, network design, and many other things. If you're interested in this stuff, Jane Street is hiring. You can check out their open roles at jane street dot com slash thor cash. And if you want to watch the full prep conversation, we posted it there, too. If you've got a frontier model and you are actually doing inference, surely they must have more than 2,000 concurrent users.

• 22:07Speaker 01

  Yeah.

• 22:08Dwarkesh Patel?

  Is there any added latency from the fact that you need to have the whole batch fill up? Or is it, if you have a reasonable amount of users, it's so unlikely that you wouldn't, it would not take you 100 milliseconds to fill up the next 2,000 slots?

• 22:21Speaker 01

  Yeah. The way to think about this, I guess we think of it as like, when does the train depart as a model? So let's say I've picked a batch size that I'm going to run at. Maybe I pick, you know, this batch size. And so like, well, and by the way, this intersection point is the same intersection point here. So I pick this batch size. I know that it's going to take, for example, maybe it's something like 20 milliseconds. It's a common place to send up landing. What I'm going to produce is like, so this is a timeline of what is running on the GPU. It's going to start a new batch every 20 milliseconds regardless. And so so each of this is 20, this is 40, I guess.

• 23:05Speaker 01

  You can think of this as a schedule for the train. A new train departs every 20 milliseconds. Any passengers who are ready, board the train. If the train is full, then they wait to the next train. If the train is not full, the train's going to go anyway. And so in terms of what that means for queuing latency, it means that the worst case is that you like a request arrives just after the train departed. It has to wait for the next train. So that's up to 20 milliseconds. And then it has to wait for that train to to complete. And so the worst case latency is 40. Sure. How is it 20 milliseconds derived? I mean, the rule of thumb, but where it comes from is not fully explained yet, but.

• 23:46Speaker 01

  So far, we've focused on memory bandwidth and compute time. When we look at memory, the other consideration is that we want to use all of the memory capacity we have. And so generally, we're going to use all of that memory capacity to store the weights or the KBs. And so we just want to read, like in the time of doing a forward pass, maybe we want to read all of the memory capacity into the chip. And so that is capacity divided by bandwidth. That tends to be 20 milliseconds on many different generations of HBM.

• 24:16Dwarkesh Patel?

  The units make sense. You would have a byte divided by bytes per second.

• 24:22Speaker 01

  Yeah, so for example, I mean, on, I think, the Ruben generation, it is something like 288 gigabytes divided by 20 terabytes per second.

• 24:34Speaker 01

  And this looks like it comes out to about 15 milliseconds.

• 24:42Dwarkesh Patel?

  Yeah, I need to make sure I understand what this is saying. I mean, I understand why the units can't do that, the sort of unit analysis. But what is it saying is?

• 24:52Dwarkesh Patel?

  We can evacuate and replace the HBM in this amount of time. And so we don't want to be in a situation where the HBM is not big enough that we're not actually able to keep write everything you want to it or take everything out of it. Or we don't want to be in a situation where our ability to write back and forth is so small compared.

• 25:17Speaker 01

  Yeah, there's sort of two scenarios. Why don't we pick a latency that is bigger than 15 milliseconds? And if I think what that means, it means I actually have time to read the HBM like twice.

• 25:27Dwarkesh Patel?

  Yep.

• 25:28Speaker 01

  By the way, most of HBM accesses is reads, not writes. It's like almost all reads because the weight matrices are read only. And then almost all of the KB cache accesses are reads. So in, like, let's say around 30 milliseconds, I can read all of HBM twice. But what's the point of that? Like, I don't want to read the weight matrices twice. I don't want to read the KBs twice.

• 25:48Dwarkesh Patel?

  Yeah, makes sense. Makes a ton of sense. Okay, so a couple of actually quick questions. One, if it is the case that the optimal batch size is something like 2,000 and that actually true, it's totally dependent on sparsity. It's not dependent on the model size or anything.

• 26:02Speaker 01

  I mean, sparsity shows up in model size, but beyond that, it only depends on sparsity, not on scale.

• 26:06Dwarkesh Patel?

  But that's a very interesting result. And that seems to imply that you can

• 26:12Dwarkesh Patel?

  One question is, how much of a push toward centralization is it that you would have these economies of scale from inference, from batching?

• 26:19Speaker 01

  Yeah.

• 26:20Dwarkesh Patel?

  But it seems like it's not that big a deal. Like, I don't know, is 2,000 users at the same time a lot? It doesn't seem like a lot?

• 26:24Speaker 01

  We can do a bit of analysis on this, which would be actually, it's like, you can think of it in terms of number of users, but maybe a more productive way to think of it is in terms of number of tokens per second. So what does this batch size mean in terms of tokens per second of the system? So tokens per second, tokens per second is going to be equal to the batch size. We run a batch many tokens and then we do that every T. So every time in tools, which is, let's say, which is, which is, this thing is equal to the 15 milliseconds, 20 milliseconds number. So this ends up being batch size itself times about 60. So like 64 times B.

• 27:07Speaker 01

  And so this ends up being around 2,000 times 64. So like 128, 128 K tokens per second. So this is sort of in more digestible units. Like, it's hard to reason about concurrent users, but what is the global traffic for a system?

• 27:29Speaker 01

  When you look at some of the announcements, sometimes the API providers will brag about how much traffic they have. The, the numbers that I've remembered from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide.

• 27:43Dwarkesh Patel?

  Yeah.

• 27:44Speaker 01

  So, so about a thousand, like, this is 1,000th of that.

• 27:49Dwarkesh Patel?

  But I mean, Gemini is big. So that's actually 1,000th of Gemini is a lot.

• 27:53Speaker 01

  Yeah.

• 27:54Dwarkesh Patel?

  To actually be like, to be competitive at scale, you need to be able to serve at least 1,000th of Gemini.

• 27:59Speaker 01

  Yeah.

• 28:00Dwarkesh Patel?

  That's interesting.

• 28:02Dwarkesh Patel?

  Cool. Okay, so

• 28:07Dwarkesh Patel?

  the more sparsity you have, the less compute you need.

• 28:13Dwarkesh Patel?

  And it does seem that as batch sizes get bigger, compute ends up being the bottleneck, according to this analysis. So then the question is, how far can you take sparsity? That is to say, as the sparsity ratio increases, as you have fewer and fewer active parameters relative to total parameters, how much is performance of the model degrading? And is it degrading faster than you're saving compute by increasing the sparsity factor?

• 28:40Speaker 01

  Yeah, so equality of the model, rather than speed of the model.

• 28:44Dwarkesh Patel?

  Yeah.

• 28:45Speaker 01

  So unfortunately, we're not able to answer that analytically. That's, that is an empirical question of model quality. Best I can do is pull up a paper and answer that empirically.

• 28:55Dwarkesh Patel?

  All right. Should we pull up the paper now? Or is it?

• 28:58Speaker 01

  Yeah. So this paper, this is Unified Laws for Routed Language Models. It's a somewhat old paper by this stage, but one of the things that they did is looked at, if I keep increasing sparsity, what is the model quality impact? This answer is very sensitive to the actual choice of mixture of experts. Mixture of experts has been around for a really long time. I think it was even back in 2017. But the techniques have changed a lot. Deep-seek mixture of experts was a big change in how it worked. There have been older papers, which are G-charge switch transformer. So the actual empirical results are going to depend on all of that. But on one of the older techniques that is shown here, you can see, if I hold constant the number of active parameters at a certain size, and then I increase the sparsity,

• 29:42Speaker 01

  which they call expert count here, the quality keeps increasing. And then if you imagine like drawing a horizontal line from 1.3B dense across, you end up seeing that, for example, in this case, the 64-expert, 370 million activated parameters model is as good as a dense 1.3 billion model.

• 30:02Dwarkesh Patel?

  So in some sense, it's actually not amazing returns where you need to increase total parameters 100-fold to get the equivalent of 10x as many active parameters.

• 30:13Speaker 01

  Yeah, I mean, actually, even more so. Yeah, it's a huge increase in parameter count for a modest increase in.

• 30:19Dwarkesh Patel?

  Yeah, so in this case, actually, what is it? 4X?

• 30:21Speaker 01

  64X for 4X.

• 30:22Dwarkesh Patel?

  Yeah, so while it is, while it is true, I guess, that the, you get this benefit of.

• 30:32Dwarkesh Patel?

  Being able to economize on your compute time if you increase sparsity.

• 30:38Dwarkesh Patel?

  Naively, it would seem like, oh, that's a trade-off worth making. But if you're decreasing this by 2X and then having this go up by 8X every time you double sparsity.

• 30:51Speaker 01

  So is that good or bad, actually? Even from a memory point of view, keep in mind, you are doubling this portion of the memory fetches, which is amortized by batch. And so just keep running at larger batch size.

• 31:06Speaker 01

  From the point of view of the analysis we've done here, this is pure win. Keep doing it. Keep doing it until you run out of available users, basically.

• 31:16Speaker 01

  So there's actually this equivalence between if I want to go sparse, or if I have a lot of users, I can go to a much sparser model. So from that point of view, it's a reasonable trade-off. The other trade-off that shows up here is that it also consumes memory capacity, which we've only reasoned about memory bound with it.

• 31:34Dwarkesh Patel?

  I see.

• 31:35Speaker 01

  It's a consuming memory capacity.

• 31:36Dwarkesh Patel?

  So let me just make sure I understood. You're saying we want to spend less time computing, therefore we do more sparsity. To make that work, we need bigger batch sizes, which means we need more memory capacity.

• 31:56Speaker 01

  Yeah.

• 31:56Dwarkesh Patel?

  To have more sparsity.

• 31:57Speaker 01

  Yeah, so I mean, maybe this would be a good point to actually talk about how a mixture of experts layer is typically laid out on a rack of GPUs or something.

• 32:07Dwarkesh Patel?

  Yeah, yeah, makes sense. Yeah, where are we?

• 32:11Speaker 01

  Spars mixture of experts.

• 32:12Dwarkesh Patel?

  Yes.

• 32:13Speaker 01

  Maybe how we lay that out on a GPU.

• 32:15Dwarkesh Patel?

  Yep.

• 32:17Speaker 01

  So let's zoom in on the mixture of experts layer first and sort of draw what that looks like. So we typically will have some kind of a router layer, which is making the decision of where we route the tokens to. So we have tokens coming in here. They go through a router layer, and then we have a bunch of different experts.

• 32:43Speaker 01

  I'll draw a few more to line them up. And then the router will make a decision, which experts am I going to route to? And it'll be a small fraction of them, maybe one in 32. So maybe it'll make a decision to route to this one, maybe this one, and maybe this one.

• 33:04Speaker 01

  These experts, so each expert itself is a normal MLP. It has an up projection and then a down projection and a nonlinearity in between. And then finally, we sort of do the inverse operation. So where we were broadcasting things out here, we're going to bring them back in and sum them up. So bringing them in like this.

• 33:28Speaker 01

  And then finally, we have our residual connections. The token is also passed through here and it gets added to the result of the MOE layer. So this is a normal MOE layer. What I want to talk through is how this is mapped to a GPU rack and what this means for communication. Because I think this will start to show some of the limits of how sparse we can go.

• 33:52Dwarkesh Patel?

  Yeah.

• 33:53Speaker 01

  So the standard practice here, and it is the best solution, is to use expert parallelism. So that means different experts go on different GPUs. So if we take something like a DeepSeek model, they have 256 experts.

• 34:10Speaker 01

  Let's say we want to run that on a Blackwell rack. So there are 72 GPUs. We have a divisibility problem. This is not a power of 2. So we'll just like simplify and say we're only going to use 64 of them. Just ignore the other eight, it's not a big deal. And so we have four experts per GPU, very simple.

• 34:33Speaker 01

  For the sake of the diagram, I'll actually just say, let's say we have two experts per GPU. So we end up just putting, these are the GPU boundaries. Every pair of experts is on its own GPU.

• 34:48Speaker 01

  And then we can look at the communication cost. We had some experts stored, some tokens stored centrally here. They get routed to all of these experts. And so there is some communication cost paid here. There's the same communication cost paid on the output. And then the hope is that this does not become communication limited. Now, what is the traffic pattern here? The traffic pattern here is that any GPU in fact will be talking to any other GPU, depending on the decisions made by the model. So this is an all-to-all traffic pattern.

• 35:23Dwarkesh Patel?

  So when you say any GPU in the pretense, the router is more than one GPU?

• 35:28Speaker 01

  Yeah, so I drew this as one router. In reality, you would actually have many copies of the router. And so you would have as many routers as GPUs, in fact.

• 35:39Dwarkesh Patel?

  As the incoming traffic?

• 35:42Speaker 01

  Yeah, so these are 64 GPUs. These are 64 GPUs. It's actually the same GPUs. We just like draw them as a separate because they're serving different purposes. So at this point, any GPU can be sending to any GPU. So this all-to-all pattern of communication that shows up and how the Blackwell racks are configured is a perfect fit for the communication pattern that the MOE actually wants to do. However, if you think, maybe I want to do, like maybe one rack is too slow and I want to do two racks. Then I have this challenge that like, maybe I've got some sort of rack boundary drawn outside here like this.

• 36:27Speaker 01

  And I no longer in fact have all-to-all communication between all the GPUs in two racks. And so the rack-to-rack communication ends up being a substantial bottleneck. So this sort of like, the fundamental thing here is that one rack is actually the bounds, the size of an expert layer you can do. And so this has been part of what's been driving towards larger and larger interconnect domains.

• 36:50Dwarkesh Patel?

  Yeah, before we, it may be worth you explaining what exactly a rack is, the differences in bandwidth between a rack. And within a rack?

• 37:00Speaker 01

  Yeah.

• 37:01Dwarkesh Patel?

  And the all to all versus not all to all nature of communication within versus outside?

• 37:05Speaker 01

  Yeah, and this is a place where it starts to be very different, in fact, between Nvidia, for example, and Google, and then others, including us.

• 37:13Speaker 01

  The, so generally, a rack is a,

• 37:19Speaker 01

  it is a physical structure. It's a few meters tall, a meter or two wide, it depends on configuration. And it stores some number of GPUs or XPs, which is typically about 64.

• 37:33Speaker 01

  The, what constrains it being a certain size is power delivery, weight, and cooling ability. It ends up being about this size in many cases because of these physical constraints.

• 37:46Speaker 01

  So then when I deploy a data center, like I've got, a data center may have thousands of these racks. So I've got one of these tall racks that's got a bunch of GPUs in it and so on. And then I put another rack next to it.

• 37:59Dwarkesh Patel?

  You make it sound so easy.

• 38:01Speaker 01

  Yeah, right, I just drop them in. In Nvidia's case, the, the communication topology is, actually, they put the GPUs on the outside of the rack and then they put these switches on the inside of the rack. So what this ends up being is that there's a set of switches in here. These are the NV switches.

• 38:27Speaker 01

  And then they run a bunch of cables. Every single GPU has cables going, going to the switches in the middle.

• 38:38Speaker 01

  So every GPU goes to the switches in the middle and then the switches have connections to all the GPUs, so all of the GPUs can talk to all the other GPUs in, in just like two hops, going to switch, going to the other GPU. Now, when I want to leave the rack, I end up going via a different path. The GPUs have also a much slower connectivity, which is typically about eight times slower, which is, so, so the green that I drew here in GPU cases is the NVLink. More generally, it's called the Scale Up Network.

• 39:13Speaker 01

  This is the Scale Up Network.

• 39:16Speaker 01

  You will typically also have a Scale Out Network, which allows you to connect to like some data center switch, so data center switch.

• 39:27Speaker 01

  And then all of the GPUs will have some connectivity up to some data sender switch somewhere. But this is about times, like this is the scale out.

• 39:41Speaker 01

  And it tends to be about about eight times slow.

• 39:45Speaker 01

  In badlands. So the the challenge, if you want to, for example, lay out a mixture of expert layer across two racks is that

• 39:56Speaker 01

  half of the GPUs here are going to be wanting to talk to the GPUs here. And so like half of the like, just on average, like when I look at where the tokens on these GPUs want to go, half of the tokens that want to go inside the rack, that's great, they can use the the fast scale up network. But half the tokens are going to want to leave the rack and go to the other rack. And that's not as good, they're going to need to use a much slower network. And so that becomes the bottleneck on on the all to all pattern. A different choice would be, well, why don't I like have a big switch here and sort of like and connect everything to some big switching, like a much bigger switch that actually combines the two racks together. There are many ideas in this direction, but in general, it becomes

• 40:40Speaker 01

  the reason you have this sort of hierarchy of switches rather than one big switch is to manage the cabling congestion. You just need to run a large number of cables.

• 40:48Dwarkesh Patel?

  Sorry, is this is that question you just asked, basically, why isn't it a bigger scale up?

• 40:53Speaker 01

  Yeah, exactly. Why not? Why not just like have like a million chips in scale up?

• 40:57Dwarkesh Patel?

  What has changed that has allowed Nvidia to go from Hopper was eight, then Blackwell is 72, and now Ruben will be is it 500 something?

• 41:09Speaker 01

  500 and something, yeah.

• 41:10Dwarkesh Patel?

  What has allowed that to happen?

• 41:12Speaker 01

  From Hopper to Blackwell is mostly just the decision to switch from

• 41:19Speaker 01

  trays as the form factor, one of these is a tray, just to switching to racks as the form factor. That's a product decision. There wasn't a substantial technical barrier there.

• 41:31Speaker 01

  Switching from from the like 64 to 500 or so. There's a bit of Jenson Math there, but

• 41:41Speaker 01

  there is at least a genuine ForEx increase, which is coming from a much more complicated and difficult rack design. And so that is actually like new new physical design to run more cables.

• 41:51Dwarkesh Patel?

  And the cable complication is just the The the cost of figuring out which cable hops to which or like which signal.

• 41:59Speaker 01

  Yeah, I mean, let's sort of zoom in on this and look at the the wire density.

• 42:06Speaker 01

  I'll draw this diagram just once more, so we have a bit of a cleaner version to work with. I'm in a larger version.

• 42:13Speaker 01

  Let's say I have some switches in the middle.

• 42:14Dwarkesh Patel?

  Yep.

• 42:15Speaker 01

  And let's say I'm gonna have initially, I'm gonna start with just two GPUs on each side or two trays of GPUs on each side. And let's say maybe each tray wants to have two cables coming out of it.

• 42:28Speaker 01

  So I get some kind of I physically run vertical cables that look like this running to the switches. Now, if I want to double the number of GPUs in a rack,

• 42:40Speaker 01

  I need to run like literally twice the density of cables. So I need to run, yeah,

• 42:47Speaker 01

  these as well.

• 42:51Dwarkesh Patel?

  Excuse me, any question? But if you look at a physical data center,

• 42:56Dwarkesh Patel?

  it seems like there's a lot of space within a rack. I don't know, just like the cables are like really big and.

• 43:01Speaker 01

  Yeah, so there is space outside the rack. Inside the rack, like these racks are like, I mean, as they become more optimized, these racks are very tight. So there's connector density going from from from from the tray into the rack and the rack's backplane. And then the backplane itself has a has a really high density. There are other physical constraints, including like bend radius of cables. You don't want to snap them and so on.

• 43:27Dwarkesh Patel?

  Yeah, it's literally the physical space to put a cable.

• 43:30Speaker 01

  Yeah.

• 43:30Dwarkesh Patel?

  That's constraining it.

• 43:31Speaker 01

  Yeah.

• 43:32Dwarkesh Patel?

  I had no idea. Interesting. That seems surprising that like the rack is so big and they're just like, we can't just stuff more cables in there.

• 43:40Speaker 01

  Yeah, so I mean, rack design is not my expertise, but like when I talk to folks and what are the constraints they're up against? It's it's a combination of.

• 43:48Speaker 01

  So what are the big physical things you're optimizing for? Space, weight of the rack, like it's actually really heavy. And so like you need enough metal top to not sag and fall. But then you add more metal and it's heavier. And then power and cooling. And so all of those are competing for like modern racks are pushing all of those to very extreme physical limits.

• 44:09Dwarkesh Patel?

  Deep work is by its nature quite aversive. So even things which seem like work, like Slack and email, can be easy ways to distract yourself. So I often wish that I could just turn the internet off. But if I'm prepping for an interview, even if I have the papers and books on hand, it's still super useful to be able to do a back and forth in an LLM so I can break down concepts and research follow-ups. Google's new Gemma 4 is the first open model that allows me to have this kind of fully disconnected focus machine. It's small enough to run on my laptop, but good enough to actually be useful. So to prep for this episode, I downloaded Ryanair's scaling book and shut off the internet. I was able to have Gemma help me understand the material and answer my questions. If you want an LLM that you can run locally on your laptop or even your phone, you should check out Gemma 4.

• 44:55Dwarkesh Patel?

  When was GPT-4 released again? It was 2022 or 2023?

• 44:58Speaker 01

  Three, three.

• 44:59Dwarkesh Patel?

  And it was rumored to be over 1 trillion parameters. And it seems like only now and within the last six months have models been getting released that are significantly more parameters than a model released three years ago.

• 45:10Speaker 01

  Yeah.

• 45:11Dwarkesh Patel?

  When supposedly there should have been this scaling in the meantime. Is the reason that we were just waiting for racks with enough memory to hold a 5 trillion parameter model along with its KV cache for enough users for a full for a lot of sequences or if you're doing RL, kind of a similar consideration of actually holding the KV cache for all the the batch of problems you're trying to solve. So if you look at like Hopper, you had eight Hoppers and I think that's 640 gigabytes as of 2022.

• 45:47Speaker 01

  Yeah.

• 45:49Dwarkesh Patel?

  With Blackwell finally, which was deployed, what, 2020?

• 45:51Speaker 01

  Very recently, maybe last year.

• 45:53Dwarkesh Patel?

  Last year? You finally have a scale up with on the order of like 10, 20 terabytes, which is enough for like a 5T model plus KV cache.

• 46:02Speaker 01

  Yeah. Deploying in larger scale up domains is a huge unlock. I mean, I've drawn here the sort of Nvidia Blackwell deployment. The Google deployment has actually had very large scale up domains.

• 46:14Dwarkesh Patel?

  And that also explains why Gemini was, seemed to be ahead. Like was Gemini 2.5 was a successful or it just seems like Gemini has that successful pre-trained for longer than some of the other labs.

• 46:24Speaker 01

  Not having been there at the time, I'm not sure how much is coming from like successfully deploying higher sparsity ratios, which could be. It could also be, I mean, there's a whole bunch of actual modeling things of like. Specifically, how do you do the mixture of experts? I've seen the DeepSeek mixture of experts has said actually activate more experts, but finer grained experts was a big innovation. I'm sure that there are many other innovations on the model architecture as well as on the training data. It's kind of hard to disentangle all of them. But what shows up in terms of the limits of what you can do?

• 47:01Speaker 01

  The active parameters, as we saw, is limited by the compute cost. And then the total parameters are limited by the scale up size.

• 47:11Dwarkesh Patel?

  Yep. When you're operating within a single scale up domain, is that a consideration specifically for either forward or backward? Or specifically for pre-fill versus decode? Or is it is it preferred to always be within a scale up?

• 47:32Dwarkesh Patel?

  Whatever kind of workload you have, whether you're doing a pre-training run or whether you're doing RL generation or whether you're doing inference for users?

• 47:41Speaker 01

  Yeah, really interesting. So, okay, so to answer that question, we're going to need to talk about the communication patterns that so we've talked about the mixture of expert communication pattern. That is this all to all.

• 47:56Speaker 01

  There's all to all.

• 48:00Speaker 01

  All to all. All to all very strongly favors full connectivity, which is what we've kind of just shown here, and favors being within one rack.

• 48:12Speaker 01

  There are other kinds of parallelism besides expert parallelism, which we just showed here. In the literature is tensor parallelism. This is with the trend towards smaller experts, this has become much less relevant, so we can ignore that. But the other two things that we have available are data parallelism and pipeline parallelism. And they are actually much, they can be a much better fit for using multiple racks. So let's focus on pipeline parallelism specifically. This is one layer of MOE. I'm going to have like 100 more layers up above.

• 48:48Speaker 01

  I could decide at this point, for example, to move to a different rack, change rack.

• 48:59Speaker 01

  Now, is that going to become a communication bottleneck? So we can actually just solve for when this becomes a communication bottleneck. But before we do that algebraically, like, let's just sort of visualize it out and sketch the path. So we're going to have a bunch, this is another MOE layer, and we're going to have another MOE layer here and so on. So let's say I change rack here, and then some number of layers later, I change rack here as well.

• 49:28Speaker 01

  So our methodology that we're going to use to determine whether we have a communication bottleneck in this, like, in this point where we change rack, is we're going to compare the, this is the scale out, scale out bandwidth requirements to the scale up bandwidth requirements.

• 49:52Speaker 01

  So let's write this, and I mean, the hint is going to be that there's a lot more transcends here. Like, we're sending many things here, whereas we're only sending one thing here. And then we're also maybe doing it many times. So that's going to be the, what makes the difference.

• 50:07Dwarkesh Patel?

  Can I try to guess?

• 50:08Speaker 01

  Yeah.

• 50:08Dwarkesh Patel?

  Just out of curiosity, see if I'm actually understanding. It seems like you're sending like batch size into the rack.

• 50:16Speaker 01

  In here?

• 50:17Dwarkesh Patel?

  Yes.

• 50:19Dwarkesh Patel?

  But the communication within a rack is sort of batch size times number of GPUs.

• 50:26Speaker 01

  Yeah, so number of activated GPUs, right? So like, I don't send to this GPU at all, right? So there's an explosion from one to like, it's three times larger here in this diagram.

• 50:38Dwarkesh Patel?

  Yeah.

• 50:39Speaker 01

  The key thing is that I didn't even need to send to this GPU at all, and so that's a big saving.

• 50:43Dwarkesh Patel?

  I see, yeah.

• 50:44Speaker 01

  Okay, so we're going to talk through sort of how much more, what is the slowdown of, to what extent is scale up a bottleneck over scale out? So we will directly jump to the ratio of the time spent on scale up,

• 51:06Speaker 01

  time on scale up.

• 51:10Speaker 01

  Over the time spent on scale out. This is the quantity we're talking about.

• 51:18Speaker 01

  And the first consideration is that the scale up is like,

• 51:23Speaker 01

  scale up is eight times faster than scale out generally. And so at a baseline, if the bandwidths were the same, we would have this one over eight, which is coming from bandwidth. Bandwidth.

• 51:37Speaker 01

  But then we have some amount of expansion in how much data we're sending. So if one token comes in here, then this one token gets routed to, in the deep sea case, it'll get routed to maybe 32 experts or 16 experts, gets routed to some number of experts. So this is the number of activated experts, number of activated

• 52:04Speaker 01

  experts.

• 52:08Speaker 01

  And then it also,

• 52:12Speaker 01

  the same thing applies on multiple different layers. So maybe I'm going to run two layers. So there's also multiple times number of layers

• 52:25Speaker 01

  per stage.

• 52:29Dwarkesh Patel?

  And there's need to multiply the whole thing by two for the

• 52:31Speaker 01

  For the up and down. Yes, yes. And there's a factor of two. Thank you.

• 52:37Speaker 01

  So what we would like is the, for the scale up time to be greater than the scale out time. Because like the scale up time is the more important and precious resource. And so we just, we want this one, we would like this number to be greater than or equal to one. And this really doesn't seem hard, like we've, there's just a factor of eight that we need to overcome. So we need the product of these three things to be bigger than eight.

• 53:00Speaker 01

  Typically, we have a fairly large number of activated experts. It could be eight by itself. And then we can increase the number of layers per stage a lot until, until we're satisfied with this.

• 53:09Dwarkesh Patel?

  I see.

• 53:10Speaker 01

  So what this ends up looking like is that I can in fact have an entire pipeline of racks where one rack does one layer and then I move on to the next rack and I do another layer. And then I move on to the next rack. I can do another layer.

• 53:20Dwarkesh Patel?

  It's interesting to me that the best parallelism. Strategy and practice ends up being one which physically resembles the actual architecture. It's not some galaxy brain thing. You know, it's like, oh, we have experts. We're going to put them on different GPUs. Or we have different layers. We're going to put them on different racks. Isn't that, I feel like that's interesting that the physical and

• 53:42Speaker 01

  The, the, the model architecture matches, like, the, the, the cutting matches the model architecture.

• 53:46Dwarkesh Patel?

  Yeah, exactly.

• 53:47Speaker 01

  Yeah.

• 53:47Dwarkesh Patel?

  I mean, it could have been something wackier with tensor parallelism and whatever.

• 53:50Speaker 01

  Yeah. So, I mean, I think a way to think of it is, I mean, okay, the galaxy brain way to think of it is, um, like, what are all the different dimensions in which a model is scaled up? Um, and so there is, uh, it is scaled up by layers. It is scaled up by the, like, demodeled, uh, dimension. It's scaled up by the DFF dimension. It is scaled up by the number of experts. Um, every single one of those numbers you can choose to cut along. Um, and if those numbers are big enough, it eventually becomes profitable to cut along there.

• 54:19Dwarkesh Patel?

  Yeah.

• 54:20Speaker 01

  And we have selected two of them. The other two, in the way typical, uh, models are typically sized, are not profitable.

• 54:26Dwarkesh Patel?

  Hmm. So there's a talk by Ilya where he says, today we know not to do pipeline parallelism. And Horace gave my friends and me, I hate that it sounded like Dr. Seuss, quote.

• 54:43Dwarkesh Patel?

  But he gave us a lecture on, uh, these different kinds of parallelisms. And he said, the problem with pipeline parallelism is that it, other than the bubbles, it constrain, it creates these architectural constraints.

• 54:51Speaker 01

  Yes.

• 54:52Dwarkesh Patel?

  On, um, like, Kimi, for example, has these, uh, residuals where attention attends to the

• 55:00Speaker 01

  A fewer back or something.

• 55:00Dwarkesh Patel?

  Yeah, layers a few back. And so that becomes hard to implement in this way.

• 55:04Speaker 01

  Yeah. Um, so, and I guess we didn't really fully articulate even what is the benefit that we're getting from pipelining.

• 55:10Dwarkesh Patel?

  Yeah.

• 55:13Speaker 01

  And so these complexities are real. It's, pipelining is a massive hassle. It's, uh, but it does give you some benefits. Um, the, uh, and then you can then decide whether those benefits are the worth, worth the costs. Um, the, uh, the biggest benefit that shows up, so it can, it has some benefits in inference, maybe bigger benefits in training. Um, in inference, what are we saving on? Are we saving on, um, memory time or compute time? Not really. We're just moving the memory time from one chip to another chip, um, or one rack to a different rack. There's no actual benefit in runtime.

• 55:51Speaker 01

  However, what we are saving on is that the memory capacity is the amount of memory used per rack. If we think that the memory in a rack is a bottleneck, then there's a constraint on how sparse we can go. Pipelining allows us to massively reduce that bottleneck.

• 56:07Dwarkesh Patel?

  I guess, but the opposite connotation to this, which actually, before this was chatting, before this interview, I was chatting with them. Axel, who's a GPU performance engineer at Jane Street, he was explaining, well, to do pipeline, and you had to do micro batches rather than full batches. And if you do micro batches, then you're by definition not able to amortize the way, loading the weights

• 56:33Speaker 01

  That's right.

• 56:34Dwarkesh Patel?

  across all the users or all the sequences. And so the positive connotation of that is you don't have to use such as memory. The negative connotation is that of that is that we can't amortize loading the weights across all those users. Maybe it's worth explaining why you had to do micro batches because you can't.

• 56:50Speaker 01

  So we draw the pipeline bubble. Yeah. Okay, so why do we do? What is this micro batching that shows up in pipeline parallelism? So the, I'll focus on inference first. It's a slightly simpler problem. So, and I'm going to draw. So this is time. And then this is which rack, rack we're on. And so the idea is that maybe I'll have like four racks. So I've got an inference that is going to like step through these four racks in some time like this. So great, this is inference number zero. It runs at a certain batch size.

• 57:34Speaker 01

  And it steps through all the pipeline stages like this. Now, if we were to say, well, we're going to run inference number one here. Like, this is clearly like a massive waste, right? Like, like three quarters of the time, each of the racks is doing nothing. So, so we don't actually run inference one here. We run it as soon as we can, which is immediately after inference zero finishes like this.

• 57:59Speaker 01

  So and then we keep going. So if we hadn't filled this in, we would call this the pipeline bubble. When I've drawn it in this inference context where we're only going in a forwards pass, it's like obvious, like, why would you do the stupid thing? But in a training context, it's maybe less obvious. But in the inference context, it's sort of really natural to make this change.

• 58:18Dwarkesh Patel?

  Oh, interesting. So this sort of virus, but the difference between micro batch and batch doesn't matter at all in inference, because you can just call whatever you want, whatever.

• 58:30Speaker 01

  Yeah.

• 58:31Dwarkesh Patel?

  It only matters in training, because there is an optimal batch size.

• 58:37Speaker 01

  Yes.

• 58:37Dwarkesh Patel?

  And before you do the backward step, you want to have accumulated, before you do a full backward step, you want to have accumulated all the sequences in that batch. And if you want to do pipeline and training, in order to avoid that bubble, you need to...

• 58:52Speaker 01

  Should we draw the training diagram?

• 58:54Dwarkesh Patel?

  Yeah, yeah, let's do that.

• 58:55Speaker 01

  Let's do that. So this is the inference diagram, and I'll call this forward, so we don't have the wrong thing showing up there. So let's do the same thing for training now. We've got a forwards pass, but at some stage, we're going to have to transition to a backwards pass. So we'll do some number of batches in the forwards pass.

• 59:20Speaker 01

  And then we're going to transition to the backwards pass for everyone all at one go.

• 59:33Speaker 01

  So the inference part is the same here, but then we do a hard stop at this point, and then transition everyone to backwards pass. Similar numbering like this.

• 59:42Dwarkesh Patel?

  It may be worth clarifying, the reason there is that hard stop is because you want to do a whole batch at once for the backward step.

• 59:49Speaker 01

  Mm hmm.

• 59:49Dwarkesh Patel?

  And then there is an optimal size for how big that batch should be.

• 59:53Speaker 01

  Yeah, I mean, smaller is always better, actually, is a way to put it. But it's like, from a ML convergence rate perspective, smaller is always better, because basically, you're getting the freshest information from the gradient descent.

• 1:00:06Dwarkesh Patel?

  But total trading time perspective.

• 1:00:08Speaker 01

  Total training time perspective, it's like, smaller is worse from a systems perspective. And so the optimum is the trade off between the two.

• 1:00:13Dwarkesh Patel?

  Yeah.

• 1:00:15Speaker 01

  So you pick a batch size, and you, and then like, for that batch size, you do some amount forwards and then some amount backwards.

• 1:00:22Dwarkesh Patel?

  Yep.

• 1:00:23Speaker 01

  You asked why is there even a hard stop? The pipeline parallelism because of this, the fact that you've got this idle time here, which is the bubble. There are so many techniques in the literature for how to lay this out differently and avoid that. There are more complicated schemes called like zero bubble or one forward, one backward, which sort of interleave the forwards and the backwards in complicated ways.

• 1:00:48Dwarkesh Patel?

  You can mine Bitcoin in that.

• 1:00:50Speaker 01

  Yeah, right, right. More usefully, you can do the weight gradient step, but you can also make it. Yeah. So in inference, actually, the effect of pipelining on anything you care about, like batch size or latency, actually is neutral. It doesn't improve it, it doesn't make it worse. So if you look at the latency of this inference, running it if it were pipelined versus if it were all on one rack, if we're all on one rack, we would just like slide all of the boxes down and still put them in a row and the latency would be the same. So pipelining is neither better nor worse for latency, but it does mean that you just use less memory per rack, like memory capacity, because now instead of needing the whole model, you only need a quarter of the model.

• 1:01:35Dwarkesh Patel?

  Makes 10 cents. So basically, no brainer to use pipelining during inference, but there is this harder trade off during training.

• 1:01:45Speaker 01

  So even in inference, in fact, it is not used a ton. It reduces your memory capacity requirements. There's actually a huge surplus, like I think you're saying that a rack of Blackwell has many, many terabytes, maybe tens of terabytes. That's much bigger than like a trillion parameter model. A trillion parameter model only needs one terabyte. And so it already fits, in fact. And so there's not a huge benefit from pipelining because you're reducing a number that's already pretty small. But it does say that theoretically, maybe you had too much memory and maybe you could have done a different, like built a different hardware that has less memory, in fact. If you were designing your hardware and you said, "I actually didn't need that much memory,

• 1:02:29Speaker 01

  because I don't need the weights to fit in one rack. I can fit the weights in eight racks." Then I could have maybe built a hardware that didn't have so much HPM per GPU.

• 1:02:40Dwarkesh Patel?

  Last week, Horace was kind enough to give me and my friends a great lecture on large-scale pre-training systems. And there were some concepts that I wanted to animate for a write-up on my blog, like how weight shard and gradients flow depending on the parallelism that you're using. So I gave Cursor my lecture notes and a sketch that I'd made during the lecture. And I asked it to visualize a specific hierarchical collective that Horace had explained. The first version was already pretty good, and then I was able to use Design mode to select and tweak any specific components from there. I was able to do all of this without a clear end state in mind. Cursor's composer-two-fast model was quick enough that I was able to iterate almost instantaneously. I could try an idea, test the results in the built-in browser, and immediately make any changes. I went through 10 different versions in under 20 minutes.

• 1:03:25Dwarkesh Patel?

  If you want to check out this animation, I published it along with the lecture notes in a blog post. The link is in the description. And if you want to try out this kind of iterative design flow for yourself, go to cursor.com/loracash to get started. So macro question. Everybody's talking about the memory wall right now. Memory's getting super expensive. There's not enough memory. Smartphone volume will go down 30% because there's not enough memory. Hyperscalers are spending... This is shocking. Dylan said they're spending 50% of their CapEx this year.

• 1:03:56Speaker 01

  On memory?

• 1:03:56Dwarkesh Patel?

  On memory.

• 1:03:57Speaker 01

  But that's believable.

• 1:03:58Dwarkesh Patel?

  Yeah. But what is hyperscaler CapEx? That's like high hundreds of billions, maybe a trillion. And they're spending half of that on memory? Okay, so that is a huge constraint. That's why we're not going to get new laptops and phones this year.

• 1:04:09Speaker 01

  Yeah.

• 1:04:11Dwarkesh Patel?

  But at the same time, we have too much memory? Like, people are willing to put too much memory into these systems?

• 1:04:16Speaker 01

  Right. So this is...

• 1:04:18Dwarkesh Patel?

  Like, why is Jetset shoving all this memory into these racks if you don't need it?

• 1:04:22Speaker 01

  Yeah. So in the equations we had here before we raised them, we were doing memory time, so memory bandwidth and compute bandwidth. Let's now start looking at memory capacity.

• 1:04:32Dwarkesh Patel?

  Yeah.

• 1:04:32Speaker 01

  So we'll start off with just like memory capacity without even thinking about parallelism scheme. And so the capacity of memory or the demand on memory is the number of total parameters.

• 1:04:50Speaker 01

  Plus, so this is what we need to fit the weights in some system that we are using. And then we need to fit the KV's as well. So KV's go as batch size times the length of the context.

• 1:05:06Speaker 01

  Times, turns the bytes, bytes per, bytes per token.

• 1:05:13Speaker 01

  Okay, so.

• 1:05:17Speaker 01

  What I was arguing about in this context and the case I was making for pipelining is that we will actually, there are some techniques that allow us to solve this, are the techniques that allow us to solve this. So let's, let's consider. So we're going to run this on some number of GPUs and we're going to say we're going to have one extent which is.

• 1:05:37Speaker 01

  E is going to be the expert parallelism.

• 1:05:41Speaker 01

  So how many, when we had this charting of expert layer across many GPUs, how much of that, to what extent do we do that, how many GPUs?

• 1:05:52Speaker 01

  So we're going to say that this is fact, for example, 64. And then P is going to be the extent of pipeline. Pipelining.

• 1:06:02Speaker 01

  And so this is the number of racks, which who knows, maybe, maybe we'll, maybe we'll pick four or something.

• 1:06:08Speaker 01

  What we want to calculate, so this is the, this is like the total, total memory requirement across the system.

• 1:06:17Speaker 01

  But now I'm going to calculate a. A memory requirement per GPU, so per GPU memory requirement.

• 1:06:29Speaker 01

  We're going to have, I guess I'll use a lover case, C mem.

• 1:06:35Speaker 01

  And well, obviously we just take all of these numbers and divide it by EMP, really easy. So.

• 1:06:41Speaker 01

  It's this N total. Plus the batch times length of context. And bytes per token. All of this is divided by the.

• 1:06:57Speaker 01

  Okay, so this is like, why is this correct, divided this way? Well, we're saying, we knew that the parameters were perfectly divided amongst all the GPUs in Iraq. The layers are perfectly divided amongst the different racks. So that works here. And somehow we're going to arrange, I'll hand wave exactly how, somehow we can arrange the same perfect sharding of the contexts across GPUs in Iraq and based on layer across racks. And sorry, four is the number of racks. Yeah, for example.

• 1:07:33Speaker 01

  So this is the place where we actually need to go back and analyze this batch size B. And you were making this comment that there's micro patching versus global batching. So let's come back to this pipelining diagram here. We've got one batch going forward here. And then, as I drew it, it kind of just like disappeared. That's not really correct. If you think about how decode is working, I have a bunch of tokens that I have generated already. I do one forwards pass where I generate a new token. And then, and then I push, like, then I write that to my KB cache. And then I do another forwards pass that generates the next token. So I'm actually going to be running this batch zero in a loop.

• 1:08:16Speaker 01

  So in fact, I go forwards. Once I finish, I can start the next iteration of the loop up here. Yeah.

• 1:08:26Speaker 01

  So we'll just fill this in. We'll have the.

• 1:08:35Speaker 01

  Nice.

• 1:08:38Speaker 01

  Yeah, so we've got the two, three. Two, three.

• 1:08:45Speaker 01

  So let's split this batch. This batch will be the global batch size. So B is going to be the number of, number of micro batches.

• 1:08:57Speaker 01

  Times the batch of like the batch size per micro batch. So how many micro batches do we need? So the number of micro batches in this diagram is four, zero, one, two, three. And then the batch size per like the micro batch size. This is still this like 2000 ish number. This is the one that is like, this is the like 2000 times sparsity. Sorry, no, this is the 300 times sparsity, 300 times sparsity.

• 1:09:30Dwarkesh Patel?

  This is the how big the train that takes up every 20 milliseconds is.

• 1:09:34Speaker 01

  Right, yes, this is going to be the 20 milliseconds train. So the global batch size is the number of micro batches times the local batch size. Local batch size is set by this hardware parameter. The number of micro batches. Well, the number of micro batches is as small as possible such that we can like wrap around and not leave any idle time when we wrap around. So if we like, if we had fewer, we would have this idle time when we wrap around. And so you can sort of just visually see that it is equal to the number of pipeline stages. I mean, sort of proof by visual here, like it is four and it's four this way as well. But like, you can sort of look and see that it goes along here and then it wraps around number of pipeline stages.

• 1:10:14Dwarkesh Patel?

  Yeah, and sorry, very basic question. This is what is actually done? Okay, like as in a frontier model today will actually have during inference have pipeline?

• 1:10:24Speaker 01

  For sure, during massive scale training, this is done. It can be done for inference. I'm actually going to make the case for why it is less attractive. It is useful for weights, but not so useful for KVs. Yeah. The big challenge is, so let's fill this in. The micro batch size here ends up being equal to the number of pipeline stages. When we go back and substitute this, all of that into here.

• 1:10:58Speaker 01

  We get a number of pipeline stages times this little b showing up in here. And then when we factor this out, I'm going to split this into like this plus into two terms.

• 1:11:17Speaker 01

  We get the full division by e times p over here. We still have division by e times p over here, but the p's cancel, this p and this p.

• 1:11:29Speaker 01

  They cancel. And so what we find, if you increase the number of pipeline stages, the memory footprint for the number of weights keeps going down and down and down.

• 1:11:39Dwarkesh Patel?

  Of course.

• 1:11:39Speaker 01

  But the memory footprint for the number of activations stays constant. So, so it doesn't actually work. Like, most of your memory ends up like, once you do enough pipelining, and it's really not much, like even two is often enough.

• 1:11:56Speaker 01

  This term becomes very small, this becomes the dominant term.

• 1:11:58Dwarkesh Patel?

  Yeah.

• 1:11:59Speaker 01

  The KB cache becomes the dominant term.

• 1:12:00Dwarkesh Patel?

  Yeah. I know this is wrong, I'm just trying to think why my train of logic here is wrong. If you have many different, you're pipelining through many different stages. The KV values are not shared between layers. So why would it not help to be pipelining across multiple layers? Because then you don't have to store.

• 1:12:18Speaker 01

  Yeah, you only need to store like one layer rather than two layers of KV's, right?

• 1:12:21Dwarkesh Patel?

  Yeah.

• 1:12:21Speaker 01

  So it helps from that perspective, you're right.

• 1:12:25Speaker 01

  What's competing with that, though, is that you need to be keeping all of the racks usefully busy at a time. And so the number of sequences that are in flight simultaneously has gone up.

• 1:12:35Dwarkesh Patel?

  Ah, yeah, yeah, yeah, makes sense, makes sense, makes sense.

• 1:12:37Speaker 01

  So those exactly cancel and you end up not getting a saving per GPU.

• 1:12:40Dwarkesh Patel?

  Right. This is going back fundamentally to the point of you're not able to amortize across KV caches.

• 1:12:46Speaker 01

  Well, so first we did, you can amortize KV caches across batch size. And now we're saying you also can't shard it across pipeline stages.

• 1:12:56Speaker 01

  It sucks from both of those points of view.

• 1:12:58Dwarkesh Patel?

  Yeah, yeah, yeah, interesting. Okay, so then what is done during inference?

• 1:13:02Speaker 01

  So, I mean, a, like, DeepSeek paper reports what they do, which is like, they just do a lot of expert parallelism, you should.

• 1:13:10Speaker 01

  In effect, you should increase your expert parallelism up to your scale up domain size. And then do very little pipelining. Maybe not at all, maybe too, just enough to make the weight storage not too big of an issue. Those are the only two parallelisms that really make sense. In the past, there was tensor parallelism, which was cutting up within an expert. But the experts are so small now that that is not a profitable optimization.

• 1:13:38Dwarkesh Patel?

  So this goes back to the question, does that mean that frontier labs, when they're doing inference, are just basically within a single scale up?

• 1:13:45Speaker 01

  Yes. Yeah, I mean, you can look at how it depends on model size. Like, you could have a very large model, like,

• 1:13:55Speaker 01

  like one that exceeds the memory of a rack. And there you should be doing a bit of pipelining. Maybe it's extremely sparse, for example, and that would be a reason to do it.

• 1:14:05Dwarkesh Patel?

  So I guess this goes back to the question about, or this goes back to the promise at the beginning of the lecture, which was, this will actually tell you about AI progress as well.

• 1:14:13Dwarkesh Patel?

  To the extent it is the case that model size scaling has been slow until recently, because...

• 1:14:20Dwarkesh Patel?

  Let me make sure I understand the claim. The claim would not be, you could have trained across more racks. It was just that it would not have made sense before, like, we didn't have the ability to do inference for a bigger model easily.

• 1:14:33Speaker 01

  Actually, I mean, so pipelining doesn't help with context length. It totally helps with model size. And so because of the ability to do pipelining,

• 1:14:45Speaker 01

  at least a rack should not be a constraint on your ability to fit the model parameters. I guess the other consideration, you're asking like, why hasn't it scaled up more? And why did bigger scale up domains help? So we talked through one aspect of that, which is, we kind of said it's not because of memory capacity. We have a solution to the memory capacity, at least with respect to model size. Not with respect to KV cache size, but at least with respect to model size, we have a solution to memory capacity. The other issue that shows up is latency.

• 1:15:16Dwarkesh Patel?

  I was just about to ask, so what is the going from rack to rack? What is the latency cost per hop?

• 1:15:23Speaker 01

  This is very much dependent on the hardware. It's...

• 1:15:28Speaker 01

  I would, I can't say with a lot of authority. I think it's probably on the order of a few milliseconds, but it could be off by an order of magnitude there.

• 1:15:35Dwarkesh Patel?

  Is four a realistic number of how many pipelining stages you might have?

• 1:15:38Speaker 01

  Yeah, yeah.

• 1:15:39Dwarkesh Patel?

  Okay, so that's not that much.

• 1:15:41Speaker 01

  On a small number of pipelining stages, this is not a huge latency impact.

• 1:15:45Dwarkesh Patel?

  Wait, I guess it's 10 milliseconds per token.

• 1:15:48Speaker 01

  That's right.

• 1:15:49Dwarkesh Patel?

  Two times four-ish. Or I don't know how many you said, but 10 milliseconds per token is actually a lot.

• 1:15:55Speaker 01

  Yeah, if it goes from 20 to 30, right? Or something like that, yeah. So like, just to chart the path that it goes through, here you're going from your GPU or TPU or whatever to a network card,

• 1:16:13Speaker 01

  which then goes to like a top of rack switch

• 1:16:18Speaker 01

  and then hops over to the other rack and does the same thing in reverse. So you sort of have to sum up the latencies of these different things.

• 1:16:24Dwarkesh Patel?

  So this is the same thing as the DCE?

• 1:16:26Speaker 01

  This is the other switch. It may in fact go up to a DCE, so switching back. Depends on deployment configuration.

• 1:16:32Dwarkesh Patel?

  Right, yeah. And because it's decode-insequential, it's also not like they stack up across the stages.

• 1:16:41Speaker 01

  Yes.

• 1:16:42Dwarkesh Patel?

  You can't do them at the same time.

• 1:16:43Speaker 01

  That's right, yeah.

• 1:16:44Dwarkesh Patel?

  Okay, so I guess this brings us back to the question then. Is the size of the scale up at all relevant to why AI model sizes or whatever have been what they have been over the last few years? Whether through training or through inference?

• 1:16:57Speaker 01

  Yeah, so I mean, we talked about latency of the hop, of this hop. There is also just the same T-MEM latency, the memory time latency is actually substantially, like massively improved by larger scale up domains. So I'll recall T-MEM down here. T-MEM for the weights. T-MEM of weights.

• 1:17:28Speaker 01

  This was equal to the number of total parameters.

• 1:17:34Speaker 01

  Divided by the memory bandwidth.

• 1:17:38Speaker 01

  Which memory bandwidth are we talking about here? Is it just one GPU? Or it's in fact? It is the number of GPUs that I can use in parallel to load these weights. So I can't use different pipeline stages in parallel because they're not running at the same time. But I can use all the GPUs in my scale up domain in parallel to load the weights. And so this is actually extremely effective. So basically, I end up with a term here. This memory bandwidth term itself is equal to like scale up size.

• 1:18:12Dwarkesh Patel?

  Times the memory bandwidth per GPU.

• 1:18:14Speaker 01

  Yeah, yeah, times GPU bandwidth.

• 1:18:18Speaker 01

  And so this term doesn't increase a lot. It maybe increases 1.5 or 2x per generation. But this one increased by like a factor of 8 from Hubbard.

• 1:18:26Dwarkesh Patel?

  So the reason the bigger scale up matter is not the memory capacity of the whole scale up, but really the memory bandwidth.

• 1:18:31Speaker 01

  Yeah, yeah. Pipelining totally solves the capacity problem, but but

• 1:18:37Speaker 01

  scale up size helps solve the bandwidth problem.

• 1:18:39Dwarkesh Patel?

  And the bandwidth problem helps you do longer context lengths, which is more and more relevant as these models get more intent.

• 1:18:46Speaker 01

  Yeah, it lets you just run the model at lower latency as a first thing. If I just do a very sparse model and it's on like a little like H100 box,

• 1:18:57Speaker 01

  the latency will be really high.

• 1:18:58Dwarkesh Patel?

  Yeah, okay. A super tangential question.

• 1:19:03Dwarkesh Patel?

  There's Chinchilla scaling, which tells you how big should a model be relative to the amount of data you're going to train it on.

• 1:19:11Dwarkesh Patel?

  But now, obviously, you're not just trying to optimize for the highest quality model you could get with training compute. You want the best results a user can get.

• 1:19:20Speaker 01

  Yes.

• 1:19:20Dwarkesh Patel?

  The mixture of training and inference compute. So then there's a question of how much should you overtrain a model such that that compute amortized overtraining and inference is minimized to get a certain performance. But now with RL inference, there's, or RL, there's another consideration, which is you're going to do some amount of pre-training.

• 1:19:41Speaker 01

  Yeah.

• 1:19:41Dwarkesh Patel?

  That pre-training will be used both for RL generation and then for inference for the final user. And by overtraining here, I mean, while it would have been more efficient just from a training computer perspective to have a bigger model that you train for less time because it can learn faster. Maybe you you get a smaller model, you spend more compute training it than you otherwise would have, but now it's cheaper to give it to users. Like basically, okay, maybe, but let me give a question more concrete. How much more than Chinchilla optimal are models overtrained?

• 1:20:09Speaker 01

  Hmm, yeah.

• 1:20:10Dwarkesh Patel?

  And has that changed as a result of RL generation?

• 1:20:12Speaker 01

  This is a place where we have to do a bit of guesswork because like the the updated scaling laws and the use and the model traffics are not reported. And so we have to guess there.

• 1:20:23Speaker 01

  But one way to look at it.

• 1:20:29Speaker 01

  Let me first just make a sort of a general heuristic claim. If I am, if I have some like cost and I've got a total cost, which is a sum of like cost A and cost B. Like maybe this is the training cost and this is the inference cost.

• 1:20:44Dwarkesh Patel?

  Yep.

• 1:20:45Speaker 01

  And so I want to minimize this sum. For many, for many curves that tend up being the case, the minimum tends to be where these are, where the costs are equalized. That's something of a heuristic claim, but you can, you can, it tends, like, there are many examples where it's true, like where one is one over X and the other one is is X, for example. They tend to be minimized at the point where they equal each other.

• 1:21:13Speaker 01

  It's also true for like E to the X and like E to the minus X and all kinds of other things. Like, so basically I've got some, I've got some curve that's going down, some other curve that's going up, and they tend to be minimized up at this equal point.

• 1:21:30Speaker 01

  Theoretically, I will conjecture that that is true for the setup you described as well.

• 1:21:37Speaker 01

  Like, actually showing that that would be true would require looking at the scaling laws and and like fitting these like weird exponents. But but things that do follow power laws tend to tend to have this property. So I'll just make that claim and move on. So we're going to say that the cost of training

• 1:21:57Speaker 01

  plus the cost of inference, we want to equalize these.

• 1:22:03Speaker 01

  We'll do pre-training only first because it's a little... Well, actually, we can do all of it in general. So so actually, we'll we'll customize. Cost of pre-training, so number of so number of number of active params

• 1:22:17Speaker 01

  times the data on pre-training. So that's the cost of pre-training. There's a factor of 6 out here, which is the number of flops. This is the famous 6ND formula.

• 1:22:29Speaker 01

  And then in in RL, we have approximately the same thing. We've got like same number of active parameters. But now it's the amount of data is the RL data. There is this extra like efficiency multiplier, which is or inefficiency, like the inefficiency.

• 1:22:51Dwarkesh Patel?

  Which is the fact that you're not training on all your rollouts.

• 1:22:55Speaker 01

  Well, yeah, there's that. And then the other, perhaps even bigger inefficiency, is that

• 1:23:02Speaker 01

  this involves a substantial amount of decode, and often decode runs at less MfU than than than training.

• 1:23:08Dwarkesh Patel?

  Okay, so if you're doing a backward pass on every single generation in RL, it would be 6ND.

• 1:23:15Speaker 01

  Yeah, so this could be a smaller number, right? Like, this could be somewhere so.

• 1:23:19Dwarkesh Patel?

  It would at least be 2.

• 1:23:20Speaker 01

  Yeah, somewhere in the range of 2 to 6. So I'll just like, we'll say somewhere in the range of 2 to 6.

• 1:23:28Speaker 01

  And then and then we can add in the inference cost. The inference cost is 2, number of active times the data in inference.

• 1:23:37Dwarkesh Patel?

  That's right, I think I think the way I said it was super garbled, just for the audience, maybe.

• 1:23:43Dwarkesh Patel?

  Forward plus backwards per parameter is 6. Forward alone is 2. That's why RL, where you might, you're definitely going to generate all the trajectories, but you might or might not train all the trajectories is 2 to 6.

• 1:23:55Speaker 01

  Yes, yeah, thank you. And then inference is just 2. So we're going to solve for essentially maybe equality of all three of these terms. That is ballpark where people are going to be. Like labs have more information on what is productive in doing more RL, for example, than versus doing more pre-training. I don't have that information, but I think a good ballpark is 30, 30, like 33% split between each of them.

• 1:24:20Dwarkesh Patel?

  Actually, I'm not sure I understand the intuition for that.

• 1:24:24Dwarkesh Patel?

  Another naive model could have been that RL plus pre-training would be 50%. And inference would be 50%.

• 1:24:29Speaker 01

  Yeah, that's also a valid answer as well. Because this is heuristic, I can't really argue for one versus the other. They don't differ by that much, like 33 versus 25 is on the smart factor.

• 1:24:44Speaker 01

  So let's pick one of them. All equal seems simple enough.

• 1:24:52Speaker 01

  And so we're just going to solve for equality of them. It's pretty straightforward. We can immediately see that the number of activated parameters totally disappears. And so let's factor that out. And we're going to just say that data in pre-training.

• 1:25:04Speaker 01

  I decided to do it your way. It's a little bit nicer, actually. So data in pre-training plus this.

• 1:25:12Speaker 01

  Oh, I didn't have the inefficiency over here either.

• 1:25:16Speaker 01

  Inefficiency, data in pre-training plus some multiple of like alpha times the data in RL. Is just going to be and end up equal to the some beta times the data in inference.

• 1:25:36Speaker 01

  So, and then let's just like roughly size the alpha. This, this, this alpha, it's going to be.

• 1:25:44Speaker 01

  This is like the, it's maybe somewhere in the range of 2 to 6, 2 to 6 over 6 from this term compared to this term. And then we've got an inefficiency term, which I would say is maybe in the range of like 30% or something like that. So, so this alpha is going to be something like 1 on 10, 1 over 10. So then this beta here is actually the same. It's a third, it's one third times 30 to 30%. So it's also equals 1 in 10, something like that.

• 1:26:20Dwarkesh Patel?

  If both of them are 1 in 10, that kind of implies that there's never a backward cost on RL?

• 1:26:24Speaker 01

  Yeah, okay, we can make this like 2 in 10, make it a bit bigger. Yeah. So, yeah, like just write it out once more, like this is 2 over 10. This is 1 over 10. So the number of inference tokens you have, and this is just a function of like, I've got hundreds of millions of tokens per second. Times my model is deployed for, I don't know, 2 months before I shift to the next version. That should determine the number of tokens in RL and pre-training. And then I guess we didn't do the equivalence between pre-training and RL, so we'll do that here. Data, pre-training should be equal to like 2 over 10 times data in RL for them to be cost equivalent.

• 1:27:09Speaker 01

  So, sorry, this 1 over, I got it backwards. Like we pay more cost when it's inefficient, so it's, this needs to be 1 over. So this, tracing this back, back forward. This, this thing ends up actually being, as written here, it's like, oh yeah, so this is like 1.5 and this is 1.

• 1:27:37Dwarkesh Patel?

  Billions of dollars of the compute just flowed the other direction.

• 1:27:40Speaker 01

  Yeah, right. I think like if you do it with a spreadsheet and like actually model it out, you might notice when the money is going down the drain. Yeah. So, uh, yeah, so I think this, yeah, all of these end up being close in as modeled here. This 30% may have been a little bit too generous. So let's say something like 1.5 here and leave this as a 1 here. So I think it like, at this point, you can almost read it off. Like the number of inference tokens should be about the same as the number of pre-training tokens should be about the same as the number of RL tokens within like factors that we're not able to reason it out.

• 1:28:15Dwarkesh Patel?

  But then, so it looks, sorry, I'm making a basic algorithm mistake. It sounds, it seems like there should be less RL tokens than pre-training tokens.

• 1:28:21Speaker 01

  Yeah, that's in general right because RL is less efficient in terms of machine time. And so you, um, if you're trying to equalize the RL and pre-training time, then, then you should have fewer tokens and not have the same wall time.

• 1:28:37Dwarkesh Patel?

  This is all quite interesting that, um, I never thought about it in terms of how much equalizing in terms of data.

• 1:28:45Speaker 01

  I, I, I mean, I think starting with equalizing in cost is right, but depending on how you model the cost, this comes close to equalizing in data.

• 1:28:52Dwarkesh Patel?

  That if every single user who uses, basically if you, for GPT to be trained optimally, every single user who uses GPT-5, the total amount of tokens that they stream should equal the total amount that have gone into pre-training.

• 1:29:04Speaker 01

  Yeah.

• 1:29:05Dwarkesh Patel?

  And the total amount of tokens that have gone into pre-training is the sum of all human knowledge. So like each model should generate the sum of human knowledge on the output that it gets on the input.

• 1:29:16Speaker 01

  Yeah. So I mean, which way are people going to err? Like, uh, if you think that people's power of prediction is not perfect and, and also you run the risk that you're, um, that you make a model that is not a frontier model and then you just throw it away.

• 1:29:29Dwarkesh Patel?

  Yeah.

• 1:29:30Speaker 01

  Then, then like that kind of changes the cost trade-off because there's some like probability that applies to the inference and you should de-rate the inference tokens by some amount.

• 1:29:39Dwarkesh Patel?

  Right. And then can we back out how much more compute, yeah, compute than Chinchilla optimal for a given sized?

• 1:29:48Speaker 01

  Yeah.

• 1:29:49Dwarkesh Patel?

  Model.

• 1:29:49Speaker 01

  So I think we just have to make some real world assumptions here in order to do that. So the inference tokens, we should totally be able to count, right? Like, so let's say a few hundred million. I don't know, maybe it's like 500 million tokens a second now. I don't really know. 500 million tokens a second times a model is deployed for two months before it becomes obsolete. I don't really know.

• 1:30:14Speaker 01

  I can't do this in my head. Can you type it into your computer?

• 1:30:21Dwarkesh Patel?

  2.6 times 10 to the 15.

• 1:30:24Speaker 01

  OK, 2.6 times 10 to the 15. OK.

• 1:30:30Speaker 01

  This number is probably too large. This, because this is going to be multiple models in a family. We, so let's, let's make it like

• 1:30:39Speaker 01

  five times smaller or 10 times smaller or something like that.

• 1:30:44Speaker 01

  OK, so we're estimating maybe 50 million tokens per second per specific model. The model is live for two months. And so this comes out to around 200 trillion tokens.

• 1:31:00Speaker 01

  And then we want to compare that to active parameters on a frontier model. I don't actually know the latest rumors, but some. Do you know?

• 1:31:10Dwarkesh Patel?

  Somebody told me 150 trillion.

• 1:31:12Speaker 01

  Active params?

• 1:31:13Dwarkesh Patel?

  Sorry, I meant that tokens.

• 1:31:15Speaker 01

  Trained on 150 trillion tokens, interesting.

• 1:31:17Dwarkesh Patel?

  Which is similar.

• 1:31:18Speaker 01

  Yeah, yeah, that's actually similar. So data on pre-training.

• 1:31:22Dwarkesh Patel?

  This is not well cited, but.

• 1:31:23Speaker 01

  You want me to not remove this?

• 1:31:25Dwarkesh Patel?

  No, it's fine, it's fine.

• 1:31:27Speaker 01

  And I think often active params, number of active params could be in the range of like 100 billion, something like that.

• 1:31:35Dwarkesh Patel?

  Yeah.

• 1:31:35Speaker 01

  Maybe, maybe a little larger.

• 1:31:38Speaker 01

  So I'm assuming active params of about 100 billion. And so multiply by 20 to get the chinchilla token count. So chinchilla, de-chinchilla would be around 2 trillion.

• 1:31:52Speaker 01

  And yeah, and we see like we're at 100 times larger than.

• 1:31:56Dwarkesh Patel?

  Actually, what does de-Chinchilla actually mean?

• 1:31:58Speaker 01

  Like the token count for pre-training for

• 1:32:03Speaker 01

  that the Chinchilla scaling law would recommend, I guess.

• 1:32:06Dwarkesh Patel?

  Oh, I see. So how much is it overtrained? Got it.

• 1:32:09Speaker 01

  So yeah, like the ratio of this 200 trillion or 100 trillion parameters over over the like the Chinchilla optimal of 2 trillion, that's the amount it's overtrained, which is like a factor of 100 overtrained.

• 1:32:22Dwarkesh Patel?

  That's whatever. Okay, so if you consider this right here, to the extent this is in the right ballpark, just by thinking about, okay, you kind of want everything to be equal in terms of compute. Here's if that OpenAI also realizes that and they're serving a certain amount of tokens per second. That tells you how much data went into the pre-training of GPT-5.

• 1:32:44Dwarkesh Patel?

  Even if it's like 50% off or something, that is that is sort of wild that you can sort of first principles. These kinds of numbers.

• 1:32:50Speaker 01

  This is also, I mean, this is why you should just like approximate everywhere because like there's so big error models on this. But yeah, no, it's kind of like empowering to just like set A equal to B and figure it out.

• 1:32:59Dwarkesh Patel?

  Yeah, yeah, that's super cool. Okay, so in the spirit of trying to deduce things, we can publicly look up the prices of the APIs of these models. And maybe you can learn something from that. So first, with a longer context,

• 1:33:17Dwarkesh Patel?

  Gemini 3.1 is

• 1:33:21Dwarkesh Patel?

  50% more expensive if you go over 200K tokens than if you're below 200K tokens. I mean, at a high level, I understand why that might that be, but why specifically 50%?

• 1:33:34Speaker 01

  Yeah. So I mean, why specifically 50%? Let's let's sort of.

• 1:33:40Speaker 01

  So the high level, even in the first place, is

• 1:33:44Speaker 01

  there is some amount of increasing cost with context length.

• 1:33:48Dwarkesh Patel?

  Yeah.

• 1:33:49Speaker 01

  And and we can bring that back up. That was the

• 1:33:55Speaker 01

  the memory time. This is the compute time. So.

• 1:33:59Speaker 01

  Okay, so we've put up these same equations from before of the the time for memory fetches, which is the weights and the KB cache. And then the time for the compute, which is just the matrix multiplication for the weights. I will also draw the the cost curve.

• 1:34:22Speaker 01

  But this time I'll do it as a function of context length instead of as a function of batch size.

• 1:34:27Speaker 01

  So there's a time over. Yeah, just time. So this is the cost curve as a function of context length.

• 1:34:36Speaker 01

  We'll draw the compute. The cost of the compute is actually constant as a function of context length. There's no dependence here on context length. In reality, there is some dependence, but it is very mild dependence, so we'll ignore it. So this is the time for the compute.

• 1:34:56Speaker 01

  This one. And then we'll also draw the dependence of the memory fetch on context length. And this starts at a large number for the weights and then grows gradually with the context length. So maybe here.

• 1:35:12Speaker 01

  And then grow gradually with context length.

• 1:35:18Speaker 01

  And so you take the maximum and you see there is this inflection point here. So now, so this is the costs that, for example, Gemini might be paying. And then you think, how might you put a pricing structure on top of that? You would like to ensure that no matter what the context length is, you are still profitable. Interesting. And so we've got a two-tier pricing structure. Maybe we've got something that looks like this up to some max context size. Fascinating. So I think it says something about, given that the bump is at 200K, it probably means that this is somewhat aligned with this crossover point. Maybe not exactly aligned with. Fascinating. So we can actually probably even complete that calculation just to see where it lands out.

• 1:36:03Speaker 01

  We can solve for the number of bytes per token. If we sort of make some assumptions about the number of active parameters. So solving for the number of bytes per token. We're going to assume like the point where we equalize the time of memory and the time of compute is at, let's say, 200K tokens. So we equalize these two. We're also going to just assume that the batch size is large enough that the the memory time spent on weights is negligible. So we'll forget about this and we'll focus on the actual memory time spent on KB cache.

• 1:36:39Speaker 01

  That ends up saying copying this term over batch times land context.

• 1:36:44Speaker 01

  Times bytes. A token. Over memory bandwidth.

• 1:36:54Speaker 01

  Is going to be equal to number of activated params.

• 1:36:59Speaker 01

  Over slots.

• 1:37:03Speaker 01

  And then we're going to solve for bytes per token.

• 1:37:27Speaker 01

  Batch size was missing here.

• 1:37:30Speaker 01

  Shows up here and then it cancels out by the time we get to here.

• 1:37:38Speaker 01

  And I dropped the land context.

• 1:37:44Speaker 01

  So we can plug in numbers. This number, this is, this is this, well, is the reciprocal of the number that we saw before. It's, yeah, this is like 1 over 300, which is reasonably stable across many different hardware platforms.

• 1:37:57Speaker 01

  We conjecturally said that maybe number of activated tokens is like 100 billion.

• 1:38:04Speaker 01

  And like the context we said was 200k.

• 1:38:09Speaker 01

  Something is wrong here though. Length of the context should be on the denominator, not the numerator.

• 1:38:28Dwarkesh Patel?

  1, 6, 6, 7. Make about 1, 1 kilo, almost 2 kilobyte.

• 1:38:33Speaker 01

  That's, that is plausible, actually. So you said around 2 kilobytes.

• 1:38:41Speaker 01

  So let's just do a sanity check for this, what this could be. There are two mechanisms that people do attention with a small number of bytes per token. One is dense attention with a lot of reuse across layers. So Character AI has a blog post talking about that, alternating long and short context. And like in the Character AI kind of model, which also showed up in the Gemma models, the global context, which is really what we're talking about here, global context, was shared across all the layers. And so to get this two kilobytes, you could get that, for example, as a de-head of 128 is typical.

• 1:39:24Speaker 01

  And then like the number of bytes is typically number of attention layers times two times de-head times number of queue heads. So this is the number of unique contexts per layer. Do you have, do you share the context across many layers or do you use it only once? So in the Character AI like models, this number is one. We said this is 128.

• 1:40:08Speaker 01

  And this is a choice which typically ranges from one, sorry, this is KV heads, I meant.

• 1:40:16Dwarkesh Patel?

  So there was written a head in the KV head is that?

• 1:40:18Speaker 01

  The KV heads are the heads that are stored in memory, like store the contents of the previous tokens. The queue heads are the retrieval heads there. They're only used temporarily and they're, they're used by the attending token. So in this water aggressive context, I've got KV heads associated with all of the context and then queue heads associated with this new token here.

• 1:40:39Dwarkesh Patel?

  But this had the 128?

• 1:40:41Speaker 01

  Oh, this is, this, this number is actually the same for, oh, sorry. This de-head is the dimension of the vector.

• 1:40:49Dwarkesh Patel?

  Ah, I see.

• 1:40:51Speaker 01

  And number of KV heads is typically in the range of one to eight. So like it is totally plausible to get this by, for example, having eight KV heads and, and a de-head of 128. That gives you exactly this number. Or, or you could have like fewer KV heads, but more layers.

• 1:41:07Dwarkesh Patel?

  Yeah.

• 1:41:09Speaker 01

  So this is one way to get there via dense attention. There's also a way to get there via sparse attention, where you increase all of these numbers, but then you have a lot of sparsity term.

• 1:41:20Speaker 01

  So yeah, I mean, I think this number is plausible, if maybe a little bit small.

• 1:41:24Dwarkesh Patel?

  It's funny that they would leak so much information through their API pricing.

• 1:41:27Speaker 01

  I mean, you are incentivized to price close to your costs, because otherwise, someone could scoop you.

• 1:41:33Dwarkesh Patel?

  Maybe we can learn something about the difference in input versus output prices.

• 1:41:37Speaker 01

  Yeah.

• 1:41:38Dwarkesh Patel?

  And what that tells us about decode versus prefill in these models.

• 1:41:43Dwarkesh Patel?

  And I think last check it's like 50% more expensive or something like that.

• 1:41:47Speaker 01

  I don't remember. What I've seen in the past is like three or five times.

• 1:41:50Dwarkesh Patel?

  Okay, that makes more sense. So let's say it's five more times more expensive. Okay. This is the compute to process the next token in decode. Suppose you're doing prefill, where you're not just processing the most recent token, you're processing all the tokens in parallel. So I want to say that it would be this times len.

• 1:42:15Dwarkesh Patel?

  Len prefill?

• 1:42:16Speaker 01

  Well, length of the pass in general, yeah. If we say like, if we can think of decoder as being a pass with one, and then prefill being a pass with many.

• 1:42:24Dwarkesh Patel?

  Okay, yeah, yeah. So maybe like prefix, whatever.

• 1:42:29Dwarkesh Patel?

  Okay, memory. So you're not storing the KV cache if you're for the tokens that are the prefill tokens.

• 1:42:36Speaker 01

  I think maybe, maybe it's sort of less draw, actually, how prefill shows up here.

• 1:42:42Speaker 01

  If I may clarify. So we do a bit of decode like this.

• 1:42:47Speaker 01

  We may actually come back and do more prefill. Like, like if you think this is a chat session, the user says something, the AI generates response, and then the user says something else, and we prefill this. So like, maybe this is the more common, like this is the general case rather than this.

• 1:43:01Dwarkesh Patel?

  And in fact, this is like you read a file or something.

• 1:43:04Speaker 01

  Read a file or just like the AI is responding to a user input or a tool call or anything that's not generated.

• 1:43:09Dwarkesh Patel?

  Yep, okay. Okay, suppose we're here. So you will need to load. Basically, the, you will have calculated all of this previously. So just the KV of everything that came before.

• 1:43:28Dwarkesh Patel?

  But what is the memory cost of this? Well.

• 1:43:35Dwarkesh Patel?

  Memory bandwidth cost of this. If you're doing flash attention, it would...

• 1:43:41Speaker 01

  Yeah, it's basically temporary. It doesn't even go to main memory. Just ignore it.

• 1:43:45Dwarkesh Patel?

  OK, so then it would just be everything that came before. So is it not just that then?

• 1:43:50Speaker 01

  Yeah, there's actually no adjustment at all to the memory time.

• 1:43:52Dwarkesh Patel?

  OK, great. Oh, it's a very trivial change to accommodate. So this term is making it 5X more expensive. Now, why would that be? Or what does that tell us about? What are we trying to learn here? What does that actually tell us? What variable does it help us clamp? Well, the compute has presumably gotten 5... Like, the only thing that could have changed is the compute is 5X more expensive as a result.

• 1:44:18Speaker 01

  So, yeah, there is the time for one pass, but actually, the amount of tokens is that much larger. So I guess we want the cost per token, in fact, or the time per token.

• 1:44:28Dwarkesh Patel?

  So I'm not sure I understood. This is... This is for processing the next token in prefix?

• 1:44:36Speaker 01

  Well, actually, for processing the entire batch. So in this... Like, at this cost, we have processed this many tokens, like, len, let it prefill.

• 1:44:45Dwarkesh Patel?

  Yeah.

• 1:44:46Speaker 01

  Well, I guess, yeah, like, of the paths. Yeah, not this prefix, but it's this cost.

• 1:44:52Dwarkesh Patel?

  OK, let's just need this in pass. So this is 5X more expensive. Input is 5X more expensive.

• 1:45:04Speaker 01

  No, output is more expensive.

• 1:45:05Dwarkesh Patel?

  Output is 5X more expensive.

• 1:45:08Speaker 01

  So the result we want to work towards is that prefill is compute limited and decode is memory bandwidth limited.

• 1:45:17Dwarkesh Patel?

  Why don't we do this? Why don't we have... Why don't we just start it with, like, len pass on the x-axis?

• 1:45:23Speaker 01

  Yep, yeah.

• 1:45:23Dwarkesh Patel?

  And t on the y-axis.

• 1:45:25Speaker 01

  t... We want the cost per token, so it'll be t over some stuff. t over length of the pass.

• 1:45:33Dwarkesh Patel?

  Mm-hmm. Yeah, that'll be right. OK, so...

• 1:45:56Dwarkesh Patel?

  Okay.

• 1:45:58Dwarkesh Patel?

  It gives me the biggest confusion about this. Length pass is the... It seems like this should be higher when you're doing pre-fill.

• 1:46:05Speaker 01

  Prefill has a bigger length pass, yeah.

• 1:46:07Dwarkesh Patel?

  Right. But then why is it cheaper?

• 1:46:10Speaker 01

  Why is it cost higher? Yeah, yeah. So, I mean, we're gonna... It's this division by length pass that actually makes it all...

• 1:46:20Speaker 01

  So...

• 1:46:20Dwarkesh Patel?

  Okay.

• 1:46:21Speaker 01

  Yeah, this is gonna divide out. This is gonna divide out, but then we're gonna get... All of this is gonna divide by length of pass, and it's gonna make the memory costs cheaper.

• 1:46:29Dwarkesh Patel?

  Okay, yeah, let me think about this then. Okay, so let's do one line for... Basically, we'll have four different lines.

• 1:46:37Dwarkesh Patel?

  Let's do the... Let's do pre-fill first, and so...

• 1:46:44Dwarkesh Patel?

  Actually, let's do decode first.

• 1:46:46Speaker 01

  Oh, so actually, I will... Length of the pass, when it's one, that is decode. When it is bigger, that is pre-fill.

• 1:46:53Dwarkesh Patel?

  All right, okay, I see, I see, I see. That makes sense. Okay, getting back to it. So t compute, if you have... Basically, just this divided by length pass is just this amount. So this actually does not vary based on t, so it'll just be some flat value like this. And this is t compute.

• 1:47:18Dwarkesh Patel?

  And then this is like... This is...

• 1:47:22Speaker 01

  That's decode.

• 1:47:23Dwarkesh Patel?

  Decode, right. Now, t mem, if you have this whole thing divided by length pass, well, it doesn't really matter what's up there. It'll just be something that looks like this.

• 1:47:34Speaker 01

  Right, yeah.

• 1:47:35Dwarkesh Patel?

  Let's say this is t mem.

• 1:47:40Dwarkesh Patel?

  This is decode again.

• 1:47:43Dwarkesh Patel?

  So as the length of the prefix goes up or pass, your memory bandwidth time declines.

• 1:47:58Dwarkesh Patel?

  And that means that to the extent that you were bottlenecked on memory bandwidth before, you can avoid being bottlenecked on memory bandwidth. The fact that they are charging 5x less for pre-fill than decode does suggest that they are bottlenecked on memory bandwidth to quite a degree, such that for them at least, because T is equivalent to cost, right? It's the cost of renting a compute. This is actually like, this would be at 1 and this would be at 5.

• 1:48:27Speaker 01

  That's right. That's right. Yeah.

• 1:48:29Dwarkesh Patel?

  So it is in fact tremendously memory bandwidth bottleneck. The real graph looks something like, the real graph looks something like, like that.

• 1:48:40Speaker 01

  Yeah, I mean, still crosses, but yeah.

• 1:48:41Dwarkesh Patel?

  Yeah, exactly. So yeah, let me do it this way.

• 1:48:45Speaker 01

  Yeah, that's right.

• 1:48:51Dwarkesh Patel?

  And then this is the gap on decode between the memory and the compute time.

• 1:48:59Speaker 01

  Yeah, yeah.

• 1:49:00Dwarkesh Patel?

  Okay, interesting. Another interesting one would be why cache hits are so much cheaper?

• 1:49:05Speaker 01

  Yeah, okay.

• 1:49:06Dwarkesh Patel?

  So I think if I remember correctly, cache hits are like 10x. It's more expensive to write to cache, according to the pricing on all these models. But if you do hit a cache, it's 10x cheaper. So what is going on with?

• 1:49:25Dwarkesh Patel?

  Presumably, this is the cost of keeping something in HBM rather than just evacuating it. But if you do keep it in HBM, then it's cheaper to load again?

• 1:49:34Speaker 01

  Right. So there's two ways you can produce tokens or the KV cache for a token. You can just produce it from scratch by computing it from the underlying like token IDs, which are tiny.

• 1:49:49Speaker 01

  Or you can previously have produced it and stored it in a memory somewhere.

• 1:49:54Speaker 01

  So the cost ratio is really talking about the ratio between those two mechanisms of producing it. A cache miss means you've deleted it from all your memories and you have to recompute it on the tokens directly. In fact, you can maybe even take that a step further and think about which memory tier do you store it in? So you could store it in HPM. There are other slower and cheaper memories than HPM like DDR on your host or flash as well. And so one of the things you can do is a calculation of where it makes sense to be in each memory tier.

• 1:50:29Speaker 01

  And this is related to how long you're going to store for. So we wanted to look at the cost of storage in a few different memory tiers and also the cost of re-materialization. So remat means the cost to rebuild all of the KB cache from scratch having it after you deleted it. So we rematerialize it.

• 1:50:51Speaker 01

  And so basically, it is going to cost the length of the context.

• 1:50:58Speaker 01

  Actually, we'll look at cost per token so that we don't need to carry around this length of context everywhere. So to re-materialize one token of KB cache, I just need to run, I need to run a forward pass on the whole model. And then, so this is going to be the compute time. I have to rerun the compute at whatever speed my GPU does it. And then I multiply it by my GPU dollars per second.

• 1:51:28Dwarkesh Patel?

  I was very extremely naive question. Why is there not a quadratic term?

• 1:51:32Speaker 01

  Yeah, so there is a quadratic term.

• 1:51:37Speaker 01

  It shows up in the compute.

• 1:51:44Speaker 01

  As an approximation, I chose to remove it. The what that I'll just show you sort of quickly what that looks like. It's because so you have the. If you look at the cost per token or the number of flops per token, there is the flops that are coming from doing the weight matrix multiplies as a function of context length. And then there is the number of multipliers that comes from doing the KV cache, which is which goes up linearly with the amount of stuff you attend to. The slope on this is so low that like when you when you draw it like this, it's like it's very well approximated by a flat line. So like it starts to like you start to notice the effect of the quadratic or the linear term up in the in the millions of tokens or so.

• 1:52:29Speaker 01

  So just not super relevant.

• 1:52:31Dwarkesh Patel?

  So what is the reason that there's no company which has over a million token context length? If this is true?

• 1:52:39Speaker 01

  Yeah, so there are two costs of long context. One is the memory bandwidth cost, which we've spent a lot of time analyzing. That's this thing. And then the other one is the compute cost. The compute cost is almost always and sort of actually forced by fundamental principles to be a much smaller slope than than the memory bandwidth cost. And so the primary thing that limits you to have really large contexts are memory balance and memory capacity, which is exactly this effect like.

• 1:53:09Dwarkesh Patel?

  And so there's this idea that Dario said on the podcast and others have said, which is we don't need continual learning for AGI in context learning is enough. And if you believe that, then you have to think that we had to get to 100 million token, 100 million billion context length to have an employee that is the equivalent to working with you for a month. Now, maybe that's no longer true as far as attention or something.

• 1:53:31Speaker 01

  Yeah.

• 1:53:35Dwarkesh Patel?

  Yeah, if you think that, then as a some ML infra thing would have to change to allow for 100 million, like the memory bandwidth to allow for 100 million.

• 1:53:43Speaker 01

  Yes.

• 1:53:44Dwarkesh Patel?

  And context lengths.

• 1:53:45Speaker 01

  I mean, sparse attention gives you a get out for sure because you get this square root like it gives you a big improvement. But I think it's like, if you look at the history of context lengths of models.

• 1:54:04Speaker 01

  From like earlier models like GPT-3, maybe to GPT-4, I don't remember when the transition happened exactly. Like they shot up from like about 8K to 100K, 200K.

• 1:54:14Speaker 01

  And then for the last year or two, they've all been hovering around there. I think that actually indicates that that is sort of the reasonably balanced cost point. And going massively beyond that would be cost prohibitive.

• 1:54:26Dwarkesh Patel?

  Not because of the compute cost, because of the memory.

• 1:54:29Speaker 01

  Because of memory bandwidth cost, yeah.

• 1:54:32Speaker 01

  So I actually don't see a very good path to solving that. Like the HBM is where it is. It's not getting hugely better.

• 1:54:45Dwarkesh Patel?

  And why doesn't sparse attention solve it?

• 1:54:47Speaker 01

  Sparse attention is a big improvement. Maybe that is priced in already, perhaps. It's not an infinite improvement because if you go too sparse, you lose too much quality. But yeah, I mean, the empirical result is that the context lengths haven't been increasing that much. And I think it's because there is no solution to the memory wall. Like, so going too sparse just means like you're attending to a very small subset of the tokens and the quality will get worse.

• 1:55:15Speaker 01

  So what is the cost of these different ways of producing

• 1:55:21Speaker 01

  resynthesizing the KV cache? Computing it from scratch is based on my GPU time. I have to do a certain amount of multiplies in order to

• 1:55:30Speaker 01

  of GPU time that I spend in order to produce it.

• 1:55:34Speaker 01

  Storing HBM.

• 1:55:42Speaker 01

  This really goes as my, I think I had a number here, which was the bytes per token.

• 1:55:49Speaker 01

  So I need to, I need to have some number of bytes per token. And then I need to store this in the HBM. So it's going to use up some of my HBM capacity.

• 1:56:00Speaker 01

  So a way to think of this is that like, if I have too many of these things sitting in my HBM, like if I fill up my HBM with just KV caches that I'm not using, I can't use that GPU. And so how do I price that? Maybe I say that the cost of it is proportional to the fraction of the HBM I'm using. So, so there's also times GPU dollars.

• 1:56:23Speaker 01

  And then let's just do one more memory tier and say something like DDR, store in DDR instead.

• 1:56:32Speaker 01

  The same kind of thing goes up for flash and for DDR. I put these in the wrong columns, actually. I meant to make two columns. The distinction I want to make is that there is the time to, cost to retrieve. And then there's a cost, cost to store, costs to hold, hold on.

• 1:56:56Dwarkesh Patel?

  Yeah.

• 1:56:58Speaker 01

  And so this is like, there's a cost per second, whereas this is like an instantaneous cost. So rematerialization has a cost to retrieve and has zero cost to store it because we've deleted it. This is the one that I put in the wrong location. This is, this is actually the cost to, to hold on, so I will rewrite it.

• 1:57:33Speaker 01

  Okay. So we have, this is the, like, if we're just storing it in HPM, it has this sort of cost profile. And then if we store in DDR, it's actually going to take some time. So it's like, we get the same thing here, bytes. Bytes per token over DDR capacity.

• 1:57:57Speaker 01

  Times DDR cost. A second. But, but now this has a, a cost to retrieve that is, is higher than the HPM because we need to copy it into the HPM. And so this is bytes per token. Over DDR bandwidth. Bandwidth. And then this consumes some amount of the DDR as well.

• 1:58:24Dwarkesh Patel?

  And every scale up has DDR in flash?

• 1:58:27Speaker 01

  There's really a deployment question, and so you can choose that. Nvidia does deploy in this form. It has both.

• 1:58:33Dwarkesh Patel?

  Why isn't the cost to retrieve HBM the memory bandwidth? Or the bytes divided by memory bandwidth?

• 1:58:39Speaker 01

  Yeah, I mean, it depends what you define a retrieve to be. Here, I'm defining retrieve to be

• 1:58:45Speaker 01

  move it into HBM so that you can start actually doing inference on it. And so like, sort of by definition.

• 1:58:49Dwarkesh Patel?

  And because if it's already in HBM, you can be doing compute while you're getting it from HBM.

• 1:58:54Speaker 01

  Yeah, for example. So these are three things, and I guess I ordered them wrong. In general, if you're balancing two costs and you've got different memory, different tiers in the memory hierarchy, you should expect as as this cost goes up, this cost should go down. So you can kind of see where the zeros are and

• 1:59:15Speaker 01

  like, I should have ordered them, this one first, this one second, and this one third. So if you're going to hold on to it for a very short amount of time, then the all of this is like multiplied by the hold time.

• 1:59:30Dwarkesh Patel?

  Yep.

• 1:59:33Speaker 01

  This one is, and so is this one.

• 1:59:38Dwarkesh Patel?

  And interestingly, they have different prices to write for. And as you specify this in the API, for five minutes versus an hour?

• 1:59:47Speaker 01

  Yeah.

• 1:59:47Dwarkesh Patel?

  Which suggests that the five minutes is HBM and the hour is DDR.

• 1:59:50Speaker 01

  I think that's like, I think that's a pretty good assumption. It could, if you look at the numbers, it might also turn out that it's one tier down and it's DDR versus flash.

• 1:59:58Dwarkesh Patel?

  Yeah, okay, interesting. And the price difference, I think, was, I'll look it up. Okay, so the base base input tokens is five per million tokens.

• 2:00:13Speaker 01

  Base, which means remap.

• 2:00:14Dwarkesh Patel?

  Yeah, that's five.

• 2:00:16Speaker 01

  This is five?

• 2:00:18Dwarkesh Patel?

  To like

• 2:00:20Dwarkesh Patel?

  retrieve, quote unquote. And then the to write to

• 2:00:30Dwarkesh Patel?

  Presumably HBM, right for five minutes is 6.25.

• 2:00:35Speaker 01

  So actually, we might actually be able to determine the, um, which memory tier it is by, um, by the durations. Actually, the duration probably tells it to actually.

• 2:00:44Dwarkesh Patel?

  And there was six, five minutes versus one hour.

• 2:00:46Speaker 01

  Yeah, exactly. I think this will probably end up being, um, it's going to be the drain time of the memory tier that you're in. And so what that means is like, uh, like, given that I'm, I know I'm going to be holding something for five minutes. I would like to have, pick a memory that I can read every five minutes. Like I can read the whole memory once per five minutes, ballpark. Um, so that is the drain time of the memory. So if I take the, the, like, all the storage, storage capacity over storage bandwidth, bandwidth.

• 2:01:21Speaker 01

  I would like this to be like equal to five minutes or something like that.

• 2:01:25Speaker 01

  And so actually, we did this calculation for HBM. For HBM, we know that this number is 20 milliseconds. So HBM is much too short, like much too small. DDR could be about an order of magnitude or, or too off from this. And so this is probably in the order of like, actually, I think it might even be in the, in the seconds, like one to 10 seconds.

• 2:01:47Speaker 01

  And then this is really, I don't have these numbers memorized, but generally as you go to slower tiers. Flash is plausibly in the order of one minute. And then like spinning disk, uh, which is massively different, I think is on the order of one hour. So this might actually identify that the tiers are probably flash and spinning disk.

• 2:02:06Dwarkesh Patel?

  Sorry, why, why is this the calculation, this is the storage cap divided by the bandwidth?

• 2:02:10Speaker 01

  So, um, you, you've got a bunch of different memory tiers, like we've listed four of them.

• 2:02:17Speaker 01

  The, your choice, like, your choice of which memory tier is like, you want to minimize the cost.

• 2:02:21Dwarkesh Patel?

  Yeah.

• 2:02:22Speaker 01

  And so you are like, what fraction of the device are you using? You're using some fraction of the device for the holding onto it. And then you're using some fraction of the device to retrieve it.

• 2:02:35Speaker 01

  And so let's say I'm using like 10% of the device. Um, and I want to equalize those two fractions. Uh, that's a sign that I've hit the right, um, the right thing. So let's say I've got some runtime here, like, I'm going to hold on for all of this time.

• 2:02:52Speaker 01

  And then, so this is the time hold.

• 2:02:57Speaker 01

  And then there's going to be some amount of time here, which is time retrieve.

• 2:03:02Speaker 01

  And I want, I mean, basically to equalize the costs, these two costs. I want the retrieval time to be equal to the hold time.

• 2:03:15Speaker 01

  Times the like fraction of capacity.

• 2:03:23Speaker 01

  Because like this is the retrieval time. Yeah, I mean, this is, this is how many other things I can hold simultaneously.

• 2:03:29Dwarkesh Patel?

  Basically, just like, hey, you wanna, you want to store things in there for so long such that the amount of time it's in there is kind of the time to get all your things in there and out.

• 2:03:41Speaker 01

  Yeah, basically, yeah. I think that probably indicates that this is the two tiers of flash and spinning disk. I'm kind of shocked to see spinning disk being used at all because it's such an old technology.

• 2:03:53Dwarkesh Patel?

  I mean, it's also crazy that it's so slow that it takes an hour to load its full capacity into it. And then.

• 2:03:58Speaker 01

  Like, it's a really unattractive technology, but it's useful in some places.

• 2:04:01Dwarkesh Patel?

  So we're sitting down because I want to ask you some questions that I guess don't need to bluff forward. You have this extremely interesting blog post where you talk about how at a high level, the architecture of different cryptographic protocols looks a lot like neural networks. And there's this conversion evolution where they both need to jumble information across all their inputs. For cryptographic protocols, it's to make sure that there's like, each new input into a hash function will totally scramble what happens. For neural networks, of course, they need to consider how this piece of information changes what you should make of this other piece of information. And that is an extremely interesting point. I guess at a high level, the difference in what they're trying to do, in some sense, they're trying to do the inverse thing.

• 2:04:45Speaker 01

  Right.

• 2:04:46Dwarkesh Patel?

  Which is cryptographic protocols are trying to take information which has structure and make it look indistinguishable from randomness.

• 2:04:55Speaker 01

  Yeah.

• 2:04:55Dwarkesh Patel?

  And neural networks are trying to take things which are, look like random, protein sequences, DNA, garbled text, and extract higher level structure from it. So.

• 2:05:08Dwarkesh Patel?

  They have similar high-level mechanisms, but they're actually trying to do the opposite things. I wonder where you make of that?

• 2:05:15Speaker 01

  Yeah. So I mean, like the mixing, like I tried to look for other examples where mixing, like scrambling mixing shows up as well. There's actually almost even like a physical example where like you're stirring something, you're making a cake and you want to stir the batter. And like literally the idea like first stir it this way and then stir it this way is like actually not too bad of an approach. But beyond that, like in back to the digital world,

• 2:05:40Speaker 01

  there are some differences and the one you call out is a pretty strong difference.

• 2:05:46Speaker 01

  The way it shows up, like what makes neural nets? Like if you just randomly initialize a neural network, actually maybe it's a reasonable cryptographic cipher as well because like the random initialization is going to jumble stuff in a complicated way. It may even like do what you want, who knows?

• 2:06:06Speaker 01

  The thing that makes it integral is the gradient descent. So you can differentiate a neural network and get a meaningful derivative.

• 2:06:14Speaker 01

  And we do a lot of work to like not overcomplicate the derivative. So the residual connection keeps it like contained and simple. And the and so does like the layer norm stuff that we do.

• 2:06:28Speaker 01

  One of the biggest attacks against cryptographic ciphers is also to differentiate the cipher.

• 2:06:35Speaker 01

  Ciphers run in a different number field. They run in the field of two elements, so just binary. Whereas neural nets run like in theory in the field of real numbers. And so you have to differentiate with respect to like binary numbers.

• 2:06:51Speaker 01

  But you can absolutely differentiate a cipher. And this is called differential cryptanalysis. And like basically what it says is that if you take a small difference of the input, how like it's quite difficult to make the difference of the output be small. Like, like the whole job of a of a well-designed cipher is to make the difference in output very large. So I guess the distinction is that the optimization goals at that point are about complexifying. They don't have the same residual connections or or like layer norms.

• 2:07:23Dwarkesh Patel?

  Yeah, I mean, I guess a place where the the two merge is backdoors.

• 2:07:31Dwarkesh Patel?

  Okay, so with a backdoor NLLM, you're trying to hide.

• 2:07:36Dwarkesh Patel?

  Would you consider an input? It's not an input into the forward pass, or it's an input into the backward pass. But you're trying to hide an input into the backward pass.

• 2:07:43Speaker 01

  Like this is like an adversarial.

• 2:07:45Dwarkesh Patel?

  Yeah.

• 2:07:47Speaker 01

  So, yeah, I mean, in fact, this is like, this is actually a place where you get exactly the sort of avalanche property that ciphers have as well.

• 2:07:58Speaker 01

  Like adversarial attacks on typically like image classification models, right? Are can I find a perturbation of the image that a very, very small perturbation of the image that totally changes the classification, totally changes the output? That is the common case in ciphers, whereas that's the like undesired case in neural nets for sure.

• 2:08:16Dwarkesh Patel?

  OK, so I was asking you,

• 2:08:19Dwarkesh Patel?

  has have neural networks actually been used for cryptography? And you realize it might be better to just do this on the Blackboard?

• 2:08:26Speaker 01

  Yeah.

• 2:08:27Dwarkesh Patel?

  So I'm curious, are they actually being used for cryptography?

• 2:08:29Speaker 01

  Yeah, so using neural nets for cryptography, well, in general, cryptography, like creating a new cipher is a very, very dangerous proposition. Like almost all of them are broken, like 99% of them are broken. So probably a bad place to start. But the other direction has been very like, in at least one very clear case, quite productive. So there's this construction in, so a construction that exists in ciphers and then was imported into neural nets called a Feistl cipher, Feistl network. So the idea is that you may have some function F, which is not invertible.

• 2:09:09Speaker 01

  But you like the function because it like does interesting things. Like it does it does an MLP, for example, or or it mixes in an interesting way. You'd like to build something out of this that is invertible. So the construction we're going to make is going to actually be a two-input function rather than a one-input function.

• 2:09:28Speaker 01

  And we're going to apply f of x.

• 2:09:34Speaker 01

  We need to actually remember what x was. So we're going to stick x over here so that we can work backwards. And then we also can't drop y. So we're going to remember y and we're going to add them together. And so we form this tuple.

• 2:09:49Speaker 01

  So the way to invert this, like if you think I have this output and I want to recover x and y. Well, I can easily recover x, that's right there, I just read it off. And then to recover y, I like if this thing was called z. I can recover y by z minus f of x. Because I've already recovered x. So that means that this construction is invertable.

• 2:10:15Speaker 01

  This was used in ciphers like a ton, still is used. It's one of the main mechanisms of constructing ciphers. Often you want ciphers to be invertable, especially the layers of ciphers you want to be invertable. Because that has been a cryptographic properties. This has actually been ported over into neural nets. There's a 2017, 18 paper called RevNets, reversible networks.

• 2:10:42Speaker 01

  And what it does is it actually makes the entire, like you can apply it to any network, like a transformer network. You can make, I do a forwards pass, but then I can actually run the entire pass backwards as well. So the whole neural network is invertable.

• 2:10:55Speaker 01

  With exactly this construction. And so this paper reversible networks, like applied to some layer, like a transformer layer, for example. We've got this function f, which is our transformer layer.

• 2:11:07Speaker 01

  Now normally we would have just an input and then a residual connection coming out.

• 2:11:13Speaker 01

  And it gets added like this. Over here. But now the variation of this is going to be we've got two inputs, x and y.

• 2:11:24Speaker 01

  So we've got x and y inputs. X goes through the function, gets added to y.

• 2:11:38Speaker 01

  And then this becomes the new X, the output X.

• 2:11:44Speaker 01

  And then this X

• 2:11:47Speaker 01

  becomes the output Y.

• 2:11:49Speaker 01

  So really what this is doing, this is like, this is actually sort of doing, if you think of two layers back, this is actually, the thing you mentioned before, it's actually doing the residual connection from two layers back. Like this Y came from the previous layer and was the residual connection there. But because of this construction, the whole thing is invertable.

• 2:12:09Speaker 01

  Why do I care? What does invertable matter for? The big thing that it can be interesting for is for training. If I think of a forward pass of training, so I will, let's say I have four layers. I run them in the zero, one, two, three order. I have to write all of the activations to HBM.

• 2:12:28Speaker 01

  And so I get an HBM footprint here that is kind of like linear, linear in a number of layers.

• 2:12:37Dwarkesh Patel?

  Yep.

• 2:12:39Speaker 01

  So this, this actually can be the largest memory footprint during training. And so this is normal training and then, and then I run the backwards pass and I read it kind of in reverse. Like I, I run them sort of forward pass goes forward, backward pass goes backwards. And I have to read them back out.

• 2:12:55Speaker 01

  The idea of this RevNet's paper is that because it's invertable, I don't need to store this at all. I can completely re-materialize it. When I'm running my backwards pass, sorry, I run my forwards pass. And then when I'm running my backwards pass, I'm simultaneously in lockstep, undoing all of the forwards pass steps that I did in order to,

• 2:13:16Speaker 01

  to have the activations that I need here. So this ends up being a memory saving, which is a nice idea.

• 2:13:20Dwarkesh Patel?

  Interesting. And in some sense, you're spending more compute to save memory.

• 2:13:25Speaker 01

  That's right. Yeah.

• 2:13:25Dwarkesh Patel?

  Interesting. Huh. Actually, it's kind of the opposite of what you're doing with the KV cache. The KV cache.

• 2:13:31Speaker 01

  Yeah, yeah.

• 2:13:31Dwarkesh Patel?

  You're spending more memory to save compute.

• 2:13:33Speaker 01

  Yeah. Spending more memory to save compute is generally profitable, given where hardware is at.

• 2:13:39Dwarkesh Patel?

  Yeah, interesting. Cool. That's super fun. Right. Thank you so much for doing it. I feel like it really vindicated the vision behind the studio and the blackboard. Cool. Thanks so much for doing it.

• 2:13:49Speaker 01

  Thanks.