TinyGPU, Massive Learning, with Adam Majmudar

Discover h​ow Adam Majmudar embarked on an exceptional journey to create the TinyGPU from scratch, with no experience in GPU design. This insightful podcast follows Adam's process from learning to implementation, highlighting the progressive contributions of countless engineers and the accelerating role of AI in the learning journey. Experience the unfolding of GPU architecture and gain a deeper appreciation for the technology driving today's AI advancements. RECOMMENDED PODCAST: The AI Daily Brief If you’re looking for an edge to stay up to date on everything AI, subscribe to Nathaniel Whittemore's AI Daily Brief which delivers tight episodes covering all things AI from legislation to new technologies to the philosophical debates around generalized intelligence. Spotify: https://open.spotify.com/show/7gKwwMLFLc6RmjmRpbMtEO Apple: https://podcasts.apple.com/us/podcast/the-ai-daily-brief-formerly-the-ai-breakdown/id1680633614 SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention "Turpentine" to skip the waitlist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ CHAPTERS: (00:00:00) Introduction (00:04:34) Intro (00:07:42) Learning Resources (00:12:11) What is the process of getting your chip back? (00:14:38) What is the scope of the project? (00:17:18) Sponsors: Oracle | Brave (00:19:25) Prioritization (00:23:19) Memory management (00:33:19) What instructions to include? (00:38:03) Sponsors: Squad | Omneky (00:40:42) Registers (00:48:29) Memory Limitations (00:57:51) Compute Pattern (01:01:14) Dispatcher (01:07:50) How does it get translated into hardware? (01:21:07) Compute Core Execution (01:24:57) The Fetcher (01:27:07) Memory controllers (01:37:49) Simulating the design (01:41:09) What did you learn? (01:50:36) Conclusion

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  So these special registers are registers that you can't write to. The GPU itself handles writing to them. And that means that the GPU itself supplies this code execution environment with like, hey, you're this block number. There's this many threads in each block. And you're also this thread number in this block. And so the high level job of the GPU now is like, okay, well, how do we manage? How to divide up these threads into the available compute cores? Wait for them to complete their job. And then give them more jobs when they're done completing their work. The analogy is like if aliens come to Earth and they find a computer, like, how are they going to understand it? Are they going to break it down and look at all the connections, the transistors? And see what's happening? No, they're not going to understand it by doing that. They have to come to some understanding of their own.

• 0:45Speaker 00

  And then test it against the computers that they have. It's a bit of a massively over complex analogy for going through a GitHub repo.

• 0:54Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas. And together, we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Adam Majmudar, creator of The Tiny GPU Project, in which Adam set out to speedrun the creation of a GPU from scratch in just a few weeks, despite having no previous experience designing GPUs. The result is truly remarkable and a great educational resource

• 1:39Nathan Labenz?

  for anyone who wants a better understanding of the foundational technology on which today's frontier AIs are trained and run. In this episode, Adam walks us through the GPU architecture layer by layer, from the tools he used to learn and design, to the GPU programming paradigm of single instruction, multiple data, to the process of hardware implementation and verification in a way that I think will be illuminating for just about anyone who's not already an expert in this critical industry. Along the way, I think this episode has a couple of big-picture lessons to teach us as well. First, it's a powerful reminder of the multi-generational tech stack that all of today's AI breakthroughs are built upon.

• 2:21Nathan Labenz?

  Adam was able to do this work thanks to the compounding progress in chip design and design automation software that thousands, if not millions, of engineers have delivered over decades. Second, it shows just how much AI itself is accelerating the learning process for individuals. Adam relied heavily on ChatGPT and Claude to help guide his learning journey. Whenever he got stuck, he would propose hypotheses to the AI and get feedback, enabling him to make much faster progress than learning solely from static resources. I found it particularly interesting to hear him describe today's leading models as having a sort of intuition for science and engineering, which transcends online resources in important ways. I've experienced this myself, most recently while trying to bring myself up to speed

• 3:05Nathan Labenz?

  on the intersection of AI and biology, and separately when planning a macOS app. And I think it's a truly important capability, which, when combined with the agentic capabilities that we expect from next-generation models, suggests potential for a genuinely transformative impact in the near future.

• 3:23Nathan Labenz?

  Finally, beyond the technical content itself, I came away incredibly impressed with Adam as a thinker, builder, and communicator. The depth of understanding that he demonstrates reflects a level of curiosity, resourcefulness, and relentless drive to figure things out that will surely enable him to make a real impact in whatever domain he decides to tackle next. Personally, I came away from this conversation with a much stronger intuition for how GPUs actually work under the hood, and I'll bet this episode will scratch a lingering itch for many, many people. One note, if you're listening to the audio-only podcast feed, we do use a screen share in this episode and refer to a number of diagrams and code samples from the tiny GPU GitHub repository.

• 4:04Nathan Labenz?

  With that in mind, I would recommend the YouTube version of this episode if that's convenient for you, though we'll also have a link in the show notes in case you want to follow along on your own. As always, if you're finding value in the show, I'd appreciate it if you'd take a moment and share it with friends. And if you do have any feedback, you can reach us via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, without further ado, please enjoy this whirlwind tour through GPU architecture with Tiny GPU creator, Adam Majmadar. Adam Majmadar, creator of the Tiny GPU project. Welcome to the Cognitive Revolution.

• 4:39Speaker 00

  Thank you for having me.

• 4:41Nathan Labenz?

  I'm excited about this. So you've created this project that really caught my eye called Tiny GPU, in which you describe yourself as having no prior experience creating GPUs, but nevertheless set out to speedrun creating one from scratch in just a few weeks. And I imagine you have learned an unbelievable amount from doing that. I've learned quite a bit just from following your progress, and hopefully the audience will be able to learn a lot from following the results of what you did as well. You want to start off by just introducing yourself a little bit and telling us how you got this crazy idea?

• 5:16Speaker 00

  Yeah, sure. I'm Adam. I've been working on a company called ThirdBun for the past three years. Started in college and worked on it full-time during college, and then worked on that for a couple of years in gap year. And I just stopped working on that recently. And then now I've been doing a couple of deep dives, which I've been sharing on Twitter on both different industries that are the overlap between what I think are going to be defining technologies for the next 20 to 30 decades, and areas that I've had demonstrated interest in for my whole life. And my goal has been, how fast can you get to a point of technical competence and industry competence? So it's not just like the technical side of things. It's actually like, understanding the industry and how things are going right now, understanding tailwinds, and also understanding

• 5:58Speaker 00

  the opportunity landscape specifically for young people. Plus, Tiny GPU happens to be the result of the technical part of that, which is I just finished my deep dive on chips and now applying to these other ones. But during that deep dive, my goal was basically learn the entire engineering stack of chips and computation and history of it. And part of that process was basically two two-week sprints. The first two-week sprint was, can I learn how to make a chip all the way from EDA, which is the electronic design side, and then understand how fabrication works, and understand the architectural side, all in two weeks, which is traditionally something that people can spend. Two, two plus years in college to do. And my belief was basically you can cut it down to a hundred X time wise. Of course, you're not going to be learning the same thoroughness as what other people are doing.

• 6:41Speaker 00

  But the question you ask yourself is, is the thing that matters the thoroughness and the quantity of stuff that you learn? Of course, people who spent three years on this know way where we can get out lots of different detail stuff. Or is it that there's some, not really eighty twenty, it's more like a ninety nine one. Where it's like one percent of the stuff is actually most of the important value. And if you can figure out how to extract that quickly, then you you get much more value out of the learning. So that was the context behind the project. And so I did the two weeks, spent that time learning how to make chips. And then by the end of that, I formed this intuition, which is like, okay, now that I know this, what's a really cool project I can build? That will teach me a lot. It will also be fun. It also has potential to be interesting and valuable to people.

• 7:23Speaker 00

  And given the whole wave of stuff around AI, you have GPTUs and tensor crossing units and all kinds of AI accelerators. And also just a whole loop around ASICs. I thought of the H2 to add into the GPU as a foundational way to learn all of that. So that's the context and the concept.

• 7:40Nathan Labenz?

  Love it. Well, let's get into it. One of the challenges that you noted right up front that I hadn't really considered, but definitely makes a lot of sense is that there aren't a ton of great learning resources out there for things like this, because a lot of the technology is proprietary. That includes designs. I've done some of my own prep in realizing that like Nvidia has not published every layer of their technology stack. And so there's questions even to this day about how some of the most advanced GPUs work in some critical ways. And also design tools, the software packages, that those are not always super easy to come by either.

• 8:18Nathan Labenz?

  So maybe for starters, just characterize that landscape a little bit and and tell us what learning resources and and software tools you were able to avail yourself of.

• 8:26Speaker 00

  So I'll contrast this to CPUs, which is a much broader landscape, and there's a lot more learning resources. And the reason is probably because of the function of how old it is. The CPU is basically the oldest invention in terms of actual useful computing. Lots of people will disagree with that statement, but the core concept of that is it. And because of that, there's so many learning resources. If you take an architecture course in college, you'll learn how to build a CPU. There's basically tons of courses around it. And what it means to have it fully documented and taught is not just how it works at a high level. Oh, here's the different parts. There's the data memory and program memory registers. It's actually like getting into the control signals in the low level and understanding how machine language translates into stuff happening in the CPU and stuff like that. So it's like the connection between the software and the hardware level.

• 9:11Speaker 00

  And that's really what it takes to be able to completely replicate it. If you want to build a CPU from scratch, at the end of the day, you need to understand how you can actually program it and how you can turn that into low level signals in your design. And so in contrast with GPUs, there's essentially nothing at that level. So what you'll find is if you go look for Nvidia and AMD GPU designs, you'll see high level architecture and we'll show you all the different pieces of them. And of course, it's a little bit ridiculous to even expect something at the control signal level for designs that large. But actually, there's no resources that dumb these things down into the simple elements that you can at least try to implement something yourself. Now, one person pointed out that Intel actually does have pretty low level diagrams.

• 9:52Speaker 00

  Again, you wouldn't really use it for learning because it's an actual, an actual GPU, so it's not really too easy to understand. And that could be one interesting approach for people.

• 10:00Nathan Labenz?

  Gotcha. Okay. What about on the tool side? I assume you were not able to use all the same software platforms that folks at Nvidia or at AMD. Maybe more at AMD because I believe there's stacks in the more open source. But what were you able to use to actually create these things?

• 10:16Speaker 00

  Yeah, so certainly my stack is completely different from all the real players. So the thing is, in this space, the EDA tooling, like the electronic design automation, which is what you use to actually convert your architectural designs into an actual ship layout. And also do a lot of stuff optimizing your design, which is basically the entire complexity of the process. Those things cost like hundreds of thousands of millions to millions of dollars, like per seat and per company. And so an individual basically never actually use those things. And in terms of shipping and actual production quality layout, I basically need those. Like they have a complete mode on this industry. The nice thing though is there's this project funded by DARPA within the past, I'll say five to 10 years, not sure of the exact number, um, called Open Road. And it's an attempt at making an open source EDA software.

• 11:00Speaker 00

  It's obviously way worse than those real production ones in many ways. Like it's not feature complete. But for simple cases like what any individual would do, or even small businesses would do, it's completely sufficient. And then the last piece that made all this possible is that, again, in the past decade, there's this company called Skywater. And they basically made this open source EDA stack for their own process node, which means that they have their own foundry that produces chips at the one thirty nanometer scale, which is just like the size of the gates and the transistors they make. And they opened it with this thing called eFabulous, where multiple people can basically pay a much smaller amount to get like a hundred chips of this process. And that costs like ten K. And this other group called Tiny Tapeout advertises the cost.

• 11:45Speaker 00

  So they'll be like, okay, one hundred people cannot pay for a tiny slot. I will put like a hundred projects in each of these chips. And so because of this incremental cheapening of everything, it now becomes possible for someone like me or others to use that, basically the Skywater process node to make a chip. And then you use the, something built on the Open Road stack called OpenLane. It's basically just Open Road specifically for that Skywater stack. And now all that's open source. Obviously, there are some synopses and cadence, which are the real production tools, but completely functional for what I would want to do, or like any individual hobbyist.

• 12:20Nathan Labenz?

  Cool. So we'll maybe get back to the sort of hypothetical manufacturing side toward the end once we get through all the design considerations. But do I have it right that at the end of this, you have a micro, a tiny, I should say, GPU that could be included alongside like a hundred other projects on one chip? And then that chip would be what you would actually get if you ordered through the process you described, would be like a chip that has your design plus a hundred other designs all on it? And then everybody just shares in the production cost, but also gets access to one another's designs? Is that how that's working?

• 12:59Speaker 00

  And I'm probably actually gonna do that. So the thing that I mentioned called tiny takeout created by the sky named Adi Gun. He's basically like a goat of making ASICs accessible nowadays. You can find him on Twitter and you can find his course, which is Zero to ASIC. He has basically made this process where they'll buy up one of those E fabulous $10,000 slots. Where you have to get 100 tips, you have to pay 10,000. And as I said, they'll allow 100 people or like a big group of people to submit their projects. And for example, yeah, exactly what you said is I could submit this to that, which I will do. And I'll get the chip. And the cool thing is I'll get the chip and I'll be able to play with my project. But I'll also be able to play with the projects of all other 99 people who submitted. I'm just a cool byproduct of it.

• 13:40Nathan Labenz?

  That's awesome. Other than for learning, what is this good for? Are there projects that people are actually where they're building things where they can't get something that meets their requirements that they can only get this way? Or is this really all for the love of the game and the fun of the DIY?

• 13:57Speaker 00

  And you're now with these stacks opening and the tool becoming better. That there might be like an interesting analog market or a smaller scale ASIC market. Where at least you could do the initial prototyping phase of the stack. I think in practice, there's not going to be huge businesses built on this ever. Like again, it will always make sense for them to go to the real production stacks. So I do think it's mostly hobbyists and I think the most interesting thing on the learning side is getting more college students and younger people. Just getting the intuition to this, since the industry was created and the engineering was much more so popular, several decades ago. Getting a new generation to it is interesting, given that there is like the whole landscape of the chips act and there's some geopolitical incentives there.

• 14:36Nathan Labenz?

  Yeah, cool. Makes sense. Okay, so let's get into the project itself. One thing I wanted to just try to do upfront is figure out the scope. When you say designing a GPU from scratch, there's like a lot of layers to the tech stacks. Obviously, you're not going all the way down to mining your own minerals. So what was like the foundation that you said, okay, like this is deep enough down the tech stack, but I'll kind of as from scratch. But I'll still be able to use this kind of foundation to build a pot.

• 15:04Speaker 00

  Yeah, that's a good question. So there's a couple of things. First of all, obviously I'm not like building the whole EDA pipeline from scratch or anything like that. So the place where it really starts is there along, which is where most design started, which is like. Basically called your register transfer level logic is just designing the the specific memory and the specific wires that connect up the chip um to make all the logic. And so designing something at that layer is the layer that I was targeting. And then also more specifically, because that's very generic. The architecture that I wanted to choose is obviously very nitpickable in the sense that, okay, you have a giant GPU. There's like millions, thousands of features to this GPU. The question is, what are you going to include?

• 15:43Speaker 00

  And because the word GPU is an amorphous term nowadays, like it started out being referred to graphics processors. Now people are still using it for lots of stuff that aren't graphics processors, like processor processors and other NR accelerators. And so there's the blurred line in terms of what GPU counts as. And so because of that, I set my goal of what do I want to focus on with this? Not as like graphics hardware or any of the specific things like that. More if you want to understand this kind of giant blob of areas that spawned from GPUs, what are the core concepts you need to understand? In order to be able to build on top of them and understand little details. And then my goal is, can I create a foundation where using this foundation, you'll be able to learn all of the important stuff?

• 16:26Speaker 00

  And by choosing that, it allows me to strip away a lot of the complexities of some of the nuances of any little engineering decision. If you're choosing to make like graphics hardware or any of the other specific things and just focus on the fundamentals that I was finding out. And so some people would disagree, right? Because it's a blurred line. Some people will be like, oh, that's not a GPU. It doesn't make with graphics hardware or whatever. There's like Netflix you can make everywhere. But at the end of the day, it's just a design decision, the purpose of the code, which is like, how can you teach people how this stuff works most effectively?

• 16:55Nathan Labenz?

  Yeah, makes sense to me. And no doubt, these are super complicated. Um, it's often said these days that chip making is the most complicated industry in the world. And you know, arguably the most complicated thing that humans have ever devised to manufacture at scale. So yeah, not surprising that you'd have to make some simplifying assumptions to pursue a project like this. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:23Nathan Labenz?

  One of the things that you mentioned in your Twitter thread that originally caught my attention was that the beginning of the exercise was a a real challenge in just prioritization, figuring out what really matters most. You came up with three things that you felt mattered the most. We can get into each one in turn, but you want to just give a high level of what those three things are and how you identified them. Maybe a little bit on what you decided to leave out, and then we can go deeper into each of the three.

• 17:53Speaker 00

  Yeah, so the three things I chose are architecture, parallelization, and memory. And granted, that whole Twitter thread, although it was written colloquially, I'm not sure if you'll link it out, but it's actually the result of hindsight. And so like all the architecture design I chose, for example, or the design decisions that I put upfront are actually the result of me having gone through the entire process. I didn't have that, I didn't have that foresight at the beginning. But the reason I chose those things is because, and even maybe I'll rephrase them to two really. Like, architecture is not really saying much. What I meant by focusing on architecture is like, what are the key elements as each piece of the GPU that are important to all GPUs? Like, different GPUs have different little specific elements, um, tailored to their need.

• 18:34Speaker 00

  But one of the things, again, going back to the foundations, that you absolutely need to understand these concepts. In order to understand how this, the whole thing works broadly speaking. And so I wrote that down over time. Now I had the architecture diagram in my, in my tweet, and that was a good breakdown of basically the most simple possible explanation of how GPUs work. And we can get into the specifics of that. Now, the other two terms were actually the more interesting parts. So I'll start with parallelization, because that's the one that is most obvious on the surface level. And that's, yeah, GPUs obviously are able to do parallel computation. That's their whole utility. Basically, in graphics hardware, or just graphics use cases in general, and machine learning. They benefit a lot from doing matrix now.

• 19:13Speaker 00

  And if matrix now happens to be element-wise, in other words, computations on individual elements don't depend on other elements, which means you can do them all at once. And that's a whole reason why GPUs are useful. So the whole point of GPUs is, instead of a CPU where generally things are thought of as being executed sequentially, and practice isn't exactly true, but like generally you execute L1, then 2, then 3, then 4. And now you've done this whole thing in four clock cycles or four cycles of computation. Instead of a GPU, you just distribute it out among a bunch of compute resources. And now you're you're accomplishing this all in a much faster time period. Now, the interesting thing about parallelization, on the surface, it seems like it's just, oh, you need to have a bunch of compute resources or like compute cores. And then you can distribute these workloads across the compute cores.

• 19:58Speaker 00

  Actually, it turns out a lot more trivial than some of the other difficulties. So there's one, the problem of how do you distribute these workloads effectively with your resources? And manage the resource utilization effectively to maximize it. That's like one interesting problem, and that happens on like the hardware level. And then there's also maybe a more interesting thing, which is what is the software pattern of parallelization? As in, how do you actually program these things? It's one thing to make the hardware support this capability. But how do you actually make the software easy to use for developers who aren't usually used to it? And that's arguably one of the core problems that Nvidia solved and what made them win. But the sense of takeout, but their CUDA software designers so good for parallel programming.

• 20:37Speaker 00

  Probably by nature, how similar and integratable it was into the previous stack that these developers who needed to use it were used to, which is C. And so another interesting element to highlight there is since the program side is so important. How does the programming pattern that people are so familiar with actually get implemented in hardware? So that was an interesting element of parallelization. So just to break that down, again, it was the the two interesting things in parallelization were resource management and resource utilization. And then software pattern to hardware, like how does that get implemented? Um, and specifically the software pattern is called same instruction multiple data. And that just means that you're executing the same code on multiple pieces of data, like elements of the matrix. And at hardware, it's called same instruction, multiple thread, which is just like the hardware manifestation of that.

• 21:22Speaker 00

  Uh, and then the last thing, which is the less obvious and more interesting, or maybe the most interesting realization for me. And um, maybe it's more obvious than you're actually doing the program to like GPU programmers, because it's it's actually like a first party thing in in. For example, but memory management is really interesting to me. And the reason is because I had a completely. Wrong notion of memory management GPUs. In fact, I didn't even realize how important it was initially. This is the thing I was saying in my tweet, or George Hoss told me. I like randomly read it to him. I was like, hey, can you give me feedback on my design? He was like, dude, this doesn't like accomplish the whole memory management, which is the most important thing. And what that is, is it would see initially that when you're thinking about parallelization,

• 22:06Speaker 00

  the bottleneck on how much you can parallelize computation is how much compute resources you have. In other words, let's say I want to do a hundred major additions at a time. That means I need a hundred different cores that support addition. And it seems like that's the bottleneck. In practice, it's actually not at all. In practice, it's actually memory. Because in order to do a hundred additions, yes, you need a hundred compute units, but you also need to be able to read from a hundred different memory locations at once from the global memory. And global memory is bottlenecked on how much bandwidth it has. And that bottleneck tends to be the limit because there's a pretty big latency in terms of this to the unit whilst to access memory, then it needs to send the request, and then it needs to wait for it. And if all of these units are trying to access memory at the same time,

• 22:51Speaker 00

  you may actually have more requests for memory than you have bandwidth in global memory, which means that you need to actually manage all these requests in memory. And that was like a huge latency. And to rip a quote directly from what George Hoss sent to me. He said, GPUs is largely the task of trying to work around memory latencies and hide them away at different layers. And that's a lot of what the challenge of GPUs really is about, which is why architecturally, they've had all these things to solve that, like memory inventory is a first standard, and then they also have multiple layers of caches to prevent having to access global memory and shared memory and all kinds of other stuff. So memory pattern is another very important part of GPUs.

• 23:31Nathan Labenz?

  Okay, cool. That's a great. Where do you think it makes the most sense to start in terms of spiking down each of these three? I originally had architecture first, but based on your description there, I might go parallelization first and then come back to architecture. But what do you think makes the most sense?

• 23:50Speaker 00

  It actually might be easiest to go in reverse order because people are probably most familiar with software, right? So you could actually start with the software side of things, the ISA, the software side, and then it'll work backwards into how this is actually implemented in the hardware. Do you think that makes sense?

• 24:04Nathan Labenz?

  Sure, I'll follow your lead. And define terms for us. ISA is Instruction Set Architecture?

• 24:13Speaker 00

  Exactly.

• 24:15Nathan Labenz?

  Yeah, I was confident about instruction set. I didn't know the A actually there.

• 24:19Speaker 00

  So this part might be a bit challenging to do without a video. Like, I think almost all of this start is going to be near impossible to do without video. So that's the caveat. But it's going to be really challenging to talk about different instructions and the actual code and stuff like that, which I think is important to understand it.

• 24:36Nathan Labenz?

  Let's do this. I'll share my screen. I've got these notes up and I can, I'm right on your GitHub page. So, okay, cool. Let's just try to describe it, talk people through it. If you want to get a clearer view of this, we'll have the video on the YouTube version.

• 24:56Speaker 00

  Yeah, so let's scroll down to the kernel system. Yeah, we can do matrix addition. So here's a scroll up a bit. Yeah, let's start. Okay, so the best way to start with understanding this whole system is by understanding the software pattern, just because that's generally most familiar to people, easiest to see the layout of what's happening. And so generally, the way you think about GPU software is it's the Cindy pattern, which is same instruction, multiple data. And so all that means is you're just going to write one instruction. You're not going to do any loops or anything to be like, okay, execute this instruction on all this different data. You're just going to do an instruction that's basically invariant to the data that it receives. And what's that? What that means is your code is going to take any some data, it doesn't know what the data is.

• 25:39Speaker 00

  And based on that data from, let's say, two different matrices, it's going to perform a computation on the data to get the element of the resulting matrix, and it's going to put that back in memory. To take a concrete example of this, we can say matrix addition, which is probably one of the most simple GPU use cases. So let's say we have two 1x8 matrices, which means each matrix has eight elements. And let's call them matrix A and B. And all we want to do is quickly compute the addition of these matrices, which we'll call matrix C. Right, so let's say that matrix A and B are loaded into memory in elements 2 to 7 for matrix A, and then 8 to What is that thirteen eight to fifteen in for matrix B? And we just want to put matrix C, the result of this addition into element sixteen plus of memory.

• 26:23Speaker 00

  And one thing you could do in a CPU is write a program. And that program is going to take like in a for loop, it's going to be like, okay, four I less than eight, as in like for each of the eight elements. Add up the element for matrix A and B, put that into the element in matrix C. And we're going to do that as can take eight cycles, like eight iterations of the for loop. And so that's one way to do that sequentially. Now alternatively, what you could instead do is write this thing called a kernel, which is a GPU program. And this kernel is just going to handle adding up one pair of elements in the M B and putting it into one slot in matrix C. And so we'll say this kernel is going to take the I element of matrix A and B, add them up, and then put it into the I element of matrix C. And so the actual code for that's pretty simple, sure, everyone can imagine what that's like.

• 27:08Speaker 00

  So you're going to basically read some global memory, the base address of A plus the I element, which is just like getting the element from A. You're going to take the base address of B, get the I element, read it out, add it up, and then put it into C there. So actually, the program itself is very straightforward and there's not really any GPU element right there. So the main thing with the GPU is you just want to tell it, how many times do you want to execute this instruction? And on how many different values of I do you want to execute on? So in this case, we want this to be from the zeroth to the seventh element of each matrix is an eight elements. So we want eight different values of I. And that means that, and that's just because we want to add up the eight different elements of the matrix.

• 27:49Speaker 00

  And we're going to call each of those times where we need a separate execution of this code, that's going to be called a thread. And a thread is just executing the same code again on multiple different parts of the matrix. And for this code, all we have to do is specify what each thread is going to do. It's just going to, basically just read the two addresses and then add them up and write it in just. And there we go, say, run eight of them. And the question is, when you run eight of these threads, let's say we now have eight of these exact same pieces of code running in our GPU. Let's just say somehow we we've accomplished that. The question is, how do you actually get each of these threads to do something slightly different? They're all running the same instruction, but you actually need them to be able to perform it on separate data, like the different elements of the matrix.

• 28:30Speaker 00

  And so what you need to do is somehow in the actual hardware of the GPU. You need to make each little area that's executing one of these kernel codes, you need each little area to know something. About which thread is executing. Is it executing the zero thread? Is it executing the second thread or whatever? And if you have a way to do that, then within that hardware, you can, the hardware can know something about its own context. So let's say the hardware knows which thread it's executing. Now the hardware can say, hey, there's this instruction here that's supposed to load this data in from memory. The data that I want to load in is actually based on which thread is being executed here. So if I'm currently executing thread number zero, I want to load in the zeroth element today and zeroth element of B.

• 29:13Speaker 00

  If I'm trying to execute thread number one, I'm bringing the first element of each, blah, blah, blah. And I want to store it in the first element of C. And so that is the most important thing here. So the two important things are, one, we're writing some code that just is about to be executed by a single thread. And then we're going to specify how many threads to execute. The second important thing here is these special values, which are going to be local in the hardware. So each time a thread is getting executed in hardware, they have some broader context about what's going on here. And so if you look at those, for people who can't see, there's these values called block index, block dimension, and thread index. And this will probably be familiar to people who are familiar with CUDA. Obviously, there's more complexity in the actual pattern. But basically, the way this works is that when you have tons of threads that need to be executed for something,

• 29:57Speaker 00

  you're going to group these threads into batches that can basically be executed together on a single compute unit. And those batches are called blocks. And so let's say our block size is four, which is what it is on my current GPU design. That means that batches of four threads are going to be executed together. And so whenever these things are executed, basically what you want to know is which block number is currently being executed? And then within that block number, what is the thread index in that block number? So that means that the threads are each going to have an index from zero to three for each block, because there's four, four threads for block. And then let's say we have 10 blocks being executed. So now if we're on block three, how do we get the current thread that's being executed?

• 30:40Speaker 00

  Very simply, we're just going to multiply the block ID three by the number of threads for block. So now that's like 12. So that means that there's 12 threads that have already been executed in the previous three blocks. And then we're just going to add the thread ID. So now we know we're on the thread number 12, 13, 14, 15. And now those sets can pull in the rest of the data they need to come in. And so that is how the Scien D architecture, like programming pattern is implemented in programming. And then we'll talk about how to implement it in the hardware and it's very simple. And that's the core understanding you need to know. Some kind of what's the kernel programming pattern? And in terms of this as an ask first, like, how did I decide what instructions to include in my ISA?

• 31:23Speaker 00

  Um, it's, as you can see, like, I'm trying to make it as minimal as possible. And so really, my decision function there was basically, what are the minimal set of things I could implement? In order to make some cool useful use cases, and so matrix addition, matrix multiplication were the most obvious things there. And so the instructions I chose are basically just what's actually needed to support those, which is just basically some basic replicate, loading and storing data from memory, because that's how you basically get your inputs and then store the result of your computation and then constants just for some convenience. But, yes.

• 31:55Nathan Labenz?

  Cool. Well, it was all quite well said, so just to try to summarize quickly back. I'll go back to the block code thing, then we'll come back to the instruction set. When you look at this block of code, the key thing to understand is this code only executes the addition of one element from within the arrays that are going to, or the matrices that are going to be added together. And so the key trick is you have to write the code in such a way that it can take in as a variable, which position are you in? And then handle everything else, uh, just based on that one variable that the hardware will actually feed in at runtime.

• 32:39Nathan Labenz?

  So everything is going to be the same, except for what position you're in. And then that will determine the offset of the memory, which will determine what value you load in, which will determine what value you ultimately calculate and therefore what value you're going to put back into which position in memory. All of that is determined by what position you are in the broader array of computation. And you can think of that as, in your case, you've got a one by eight, but obviously you could get, I believe the current GPUs go up to like three dimensions of.

• 33:11Speaker 00

  Yeah.

• 33:12Nathan Labenz?

  There's like, yeah, you want to give us just a little bit of flavor as to how this kind of gets more complicated from here?

• 33:19Speaker 00

  Yeah, so in actual GPUs, first of all, they have more than just block index, block dimension and thread index to identify a thread. There's like a couple more dimensions of just like different groups of batching that basically have to use every single dimension to identify what thread you're in. And then some other interesting things. So I basically just captured the most simple piece of GPU programming here. In reality, there's a couple other very interesting and important things. So first of all, like when you're loading and storing data, that's something that we'll maybe get to later, but by nature of how GPUs have different layers, different levels of memory, and they have shared memory, which is shared between an entire block with threads, which means that four different threads can actually store stuff and read stuff in the same location. It's called shared memory. And then each thread also has its own dedicated memory called a register file. That's important.

• 34:03Speaker 00

  And so one complexity that I didn't implement here, which is there in real GPUs, and it's very important for actual use cases, is that you can actually choose which memory you're putting stuff in. So my pattern, there's only two types of memory you interact with. All of these small add comps, all these instructions, everything that has like an R in its parameter. So it'll be like multiply R zero, R two, R three. Now it's operating on registers for each thread, which means that stuff is all just specific to a single thread, a thread that's currently executing. And it's storing stuff in these local register files that has local data. And it's basically just performing computations on the register values. And then there's the load and store, load and store instructions, which is either loading data from global memory into the register

• 34:48Speaker 00

  or storing data from the register into global memory. There's really only two levels of memory happening here. In a real GPU, you have separate instructions that are going to let you interact with registers, global memory, and also shared memory, and there's also layers of caches. So that's one thing. Then one other interesting thing is in real GPU programming, you have the ability to synchronize different threads with each other with barriers, which means that because of the necessity of shared memory, that means like, let's say four threads, they want to store something in this shared area and then access it from each other. But what happens if one of the threads is ahead of the other? In other words, one thread gets to some step and the other threads are not there. And now, what if this thread is going to read some data from the shared memory?

• 35:30Speaker 00

  Well, if it's going to do that, it needs to make sure that the other threads are at the same point. Otherwise, it could have a corrupt data, or maybe the data is not ready yet to be read by that thread. And so there's these things called barriers where you can basically synchronize threads. So it's like, this thread is just going to wait until all the other threads are going to get to the same point. And that is to do something, maybe where I read some shared memory or something like that. So that's another important thing, which, again, I didn't implement because it's explainable from here. And also, the complexity of implementing it is not worth it.

• 35:58Nathan Labenz?

  Okay, cool. That's good.

• 36:01Speaker 00

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 36:05Nathan Labenz?

  You want to next just talk through a little bit the instruction set? And maybe give a little flavor of what kind of the next in instructions would be if you were to expand from here?

• 36:16Speaker 00

  Yeah, so instruction set, I would say a lot of it is very simple. So you have addition, subtraction, multiplication, and division. Very straightforward. Everything just takes three registers. And so the important thing I'll cover here is just the specifics of the register file. So with addition, subtraction, multiplication, and division. For context, each thread has 16 registers, which means that it has 16 different places that it can store whatever values it wants to store for any arbitrary computations. And importantly, the last three of these register values are restricted. And so we'll call those registers 13, 14, and 15. Those are how, in my GPU, the actual, those custom context values I was talking about. All right, now, the real GPU doesn't necessarily work like this.

• 36:59Speaker 00

  But basically, the important thing is every thread has access to its context. I happen to implement that through these registers. And so these special registers are registers that you can't write to. The GPU itself handles writing to them. And that means that the GPU itself supplies this code execution environment with like, hey, you're this block number. There's this many threads in each block, and you're also this thread upper in this block. And again, that's how in the hardware, the code has access to those values. Now, the last 13 registers are read, write registers, which means the code is free to access these somehow for a while. And so these add, subtract, multiply, and divide, what they do is basically take the values of two registers, which you then specify with four bits,

• 37:42Speaker 00

  which is what you need to specify an address of 16 different possibilities. And it will take the value from these, from two different registers, and it will perform a computation of them, like an addition, subtraction, multiplication, division, and then store that result back into one of the registers, all the registers specified by four bits. Now, with the load and store instructions, similarly, the load instruction will take the address of one register. So it will take the value of one of the registers. That's going to specify a memory address. So for example, if that register has the number four in it, it's going to go load in element number four from global memory, and it's going to put that into another register. And for the store instruction, it's going to take the value of one of the registers

• 38:25Speaker 00

  and store it into a memory address in global memory, specified by another register. So again, those are register computations. That constant is very straightforward. You just load in a specific value that you want to into a constant, to the register, and that might just be because you want to use it for addition or something else. Now, the interesting instructions that I'll spend a bit more time explaining are the BR and ZP, which stands for branch, the branch instruction, and the comparison instruction. And these two work together like the simplest implementation, kind of naive implementation that allows you to do if statements and looping. And this is all CPU stuff, or CNA or DPU's too. But the main thing is that what you want to do is create some condition.

• 39:08Speaker 00

  And then if this condition is true, you can jump somewhere else in your code, which means, like, let's say I want to do something like, if one of my registers is greater than zero, then jump to the end of this, skip this entire segment of code, otherwise go through it. So that's basically like an if statement, right? Or you could do a loop. So you could do some code, you could put like a label at the top of the code. So let's call it like a loop label, which is actually in one of my kernels below. And then you have a whole part of code. And then below that code, you're going to say if some condition is true, jump all the way back to the top of the loop, which is going to keep it looping until that condition is not overturned. The way that you implement this is with the comparison and branch instructions. And it's always basically a comparison then a branch. So what you do is first you do the comparison instruction.

• 39:53Speaker 00

  And that's basically you're going to take two registers. It's going to compare them. And then it's going to check if the result of subtracting the second register from the first is positive, zero, or negative. And it turns out you can actually do a lot with that. So you're just going to compare two registers. And it's going to store that result if they're positive, negative, or zero from the difference between them in something called the NZP register, which is stored in the program counter unit of the GPU, like any GPU unit basically. In our case, it happens to be a thread execution unit in a GPU. And so the program counter now has the current line of code being executed. And now it also has this comparison thing, which is storing whether the result of the subtraction of the two registers is positive or negative.

• 40:38Speaker 00

  Now with that, so you're going to have now some NZP value. It's going to have a one specifying whether it's negative, zero, or positive. Now, the next instruction, BRNZP, which is a branch, is to allow you to specify branch to some specific line, as in jump to some specific line of code if the NZP register holds some specific set of values. You could say if the NZP register is saying that this comparison is negative, then jump back to a loop code, which is exactly what we did in the matrix multiplication. So the matrix multiplication needs to do a loop through some code. So what it does is it compares some counter. Basically, it's just incrementing a counter. And that counter is supposed to do some number of loops through the code. So let's say in our case, the counter needs to go through basically two loops.

• 41:23Speaker 00

  And so it's just storing the value. And that value is going to start at zero and keep incrementing up until it's past the counter. And so basically, the way you would use these instructions is use the comparison instructions. Here you're going to say, compare my counter to the value that the counter is supposed to always be less than. And then if the counter is still less than the the max value, go back and loop. Otherwise, don't loop. And so that's how you'd use those two instructions to create conditional logic, which is exactly what the code does. So those are pretty important. And then the last two instructions are very straightforward. So there's like the no opt, which is basically not even an instruction. It just moves on to the next line. And there's a return instruction, which tells the GPU that it has reached the end of executing a specific code. And that's how the entire ISA and that also.

• 42:07Speaker 00

  Honestly, this part was actually a lot of the hard part. Once you understand this code layer, which is why we started here, then the hardware layer actually becomes a lot more clear what's going on. And that's partly a result of the fact that this code layer is the result, a lot of fine tuning on the hardware layer also in terms of what I actually included here. But yeah.

• 42:25Nathan Labenz?

  Okay, cool. Again, maybe just to try to recap a little bit there. I hadn't understood coming in that you had these three dedicated registers, which is just a position in the local most memory, right? That sort of declare to the kernel, which is the form of parallel programming that says, here's what to do, given what position you are in. That indication of where you are is provided in these dedicated registers. And then separately, you have, I guess, how was that? That's not really related to the size of the instruction set. That's kind of independent, like how many ridges I was conflating those for a second in my head. Because you numbered the instruction set with a similar gap.

• 43:10Nathan Labenz?

  Is there a connection there that I'm inferring correctly or am I hallucinating this?

• 43:14Speaker 00

  So there's a couple of connections in the instruction set to the hardware. There's two important connections. And they're very much demonstrating that this is a dummy project. So first of all, if you look at the instruction set, each instruction is 16 bits, which means that the first four bits of the op code, meaning that basically since there's there can only be. Sixteen different combinations of that. That means that with this design I've created, there's actually only space for sixteen total instructions, which means there's only space for four or five more instructions if I wanted to add them. Now, the the second constraint is that I've made it so that each register is specified by four bits, which means that there can only be up to sixteen registers. If I wanted more registers, I couldn't do that. I would need a fifth bit to specify each register, which means the instruction would have to be longer.

• 43:58Speaker 00

  And then the the second thing is, sorry, the third thing is that the constant and more importantly, the branch instructions, all they both take immediate values, which means they're not taking some value from a register. They're literally taking values that are specified in the code. Those are numbers. And if you look at those, each of the numbers is eight bits, which means that the branch instruction can only specify the branch to eight bits of program memory, because the branch instruction is telling it, if some condition is true, jump to this line of the program. Which means that in order for the branch instruction to cover the entire program, um, my hardware, as in my program memory, can only be specified by eight bits, which means it can only have two hundred fifty-six or by five, six rows to take that memory. Um, so those are the constraints created by the instructions here.

• 44:42Speaker 00

  And again, to, in reality, for this reason, instructions are usually a lot longer than sixteen. Usually they're thirty-two bit, like the instructions. But yeah.

• 44:50Nathan Labenz?

  So if you wanted to have more memory, you would have to have longer addresses for that memory. And if you wanted to have more instructions, you would have to have, obviously, bigger labels for those instructions. And maybe this is where we're headed next, but I'm a little fuzzy on like, basically each one of these instructions is implemented by hardware at this point. This is like the last layer of software when this gets.

• 45:25Nathan Labenz?

  Issued to the GPU or to a compute core within a GPU. It now says, okay, I'm going to actually fire up a particular circuit that's going to do this stuff now, right? There's no further software to.

• 45:41Speaker 00

  Exactly. One caveat on, so I obviously oversimplify. Pretty much everything I said here in this whole thing is simplified in some way. But I feel like one example of an oversimplification here is like in a branch instruction. In reality, the number of bits in the immediate in a branch instruction, if it is immediate, is not limiting the program memory. The reason that I limit it is because your alternative is to make this thing an offset. So instead of it being like literally branch this line, you could be like branch to some offset from the current line, as in branch plus 50 or branch minus 50. In which case you could do it a couple of times and then you're not limited. So you can, you can get infinitely far in memory. There's tons of other ways to approach it. That's just one example of a constraint because I, one, did not support negative numbers here.

• 46:22Speaker 00

  And two, I'm not making an offset. I just wanted to jump to a specific one for simplicity's sake. Now, that's a constraint that in reality, there's tons of ways around this, but yeah, that's an example. And then in terms of the second part, that's exactly right. The important thing to take away from this part, maybe there's still lots of fuzzy details, especially for people listening. The important thing is, one, you understand the, like, the importance of each instruction and why it's there and what it's supposed to do. You understand the general programming pattern. Again, it's hard to draw even just because it's assembly and assembly itself is not necessarily the easiest thing to pick up. It makes sense when you, you look into it, but you just need to understand those parts and the important GPU programming patterns because a lot of the actual implementation will get revealed in the hardware.

• 47:04Nathan Labenz?

  Okay. I think it's just about time probably to turn to hardware, but we'll just scroll through the multiplication one more time because this is where you have basically brought it all together in terms of the complexity that you can support. Where we've still got to be mindful of where we are in the thread. But this time, the calculation, do I have it right that basically this calculation determines one position in the output matrix?

• 47:34Speaker 00

  That's exactly right. And.

• 47:35Nathan Labenz?

  So you do a two by two, you're going to get a two by two. So you're going to have four spots in the output matrix. So that's why this is four threads. Each one of those threads is going to do a different set of.

• 47:51Nathan Labenz?

  It's still going to touch all the numbers, basically, right? But it's going to do them slightly differently to get to the final spot in the output matrix. And to do the sort of rows and columns, it has to do this loop. And this is where you get to the point where you have to have this comparison and branching.

• 48:09Speaker 00

  Yeah, exactly. The reason is just because of the nature, matrix multiplication being a little more complicated than addition. With addition, you just add up to the elements. With matrix multiplication, unfortunately, it's not just like multiplying the elements. It's like a dot product between two vectors in the matrix, which means like you just take two like vectors, which is just a row and a column of elements. And then you multiply their elements by each other and then add up the result. So it's actually a bunch of different computations, not just like a single point wise application or something. Um, which is why there's a loop here and why there's a bunch more multiplications and additions going on, which is what necessitates comparison and direction in this case. And I would say that's the most important thing. For people who want to understand the specifics of the kernel, you can look at it.

• 48:51Speaker 00

  It's relatively simple, but maybe a little bit beyond the complexity of what makes sense to explain on audio.

• 48:57Nathan Labenz?

  Yeah, and I misspoke when I said that it's going to touch all the numbers. It's going to touch one row or column for each of the two input matrices.

• 49:06Speaker 00

  Exactly, yeah. So it's like that's a lot of numbers. In this case, it happens to be three quarters of them for each computation.

• 49:11Nathan Labenz?

  Okay, cool. Well, then I guess it's time to get on to the hardware. So how do you think best to approach that?

• 49:21Speaker 00

  Yeah, so this part is actually going to be a lot to explore, thankfully. And so the best way to start it is to just start with the architecture of the entire GPU. And then we can dive into the thread architecture. And those are the great straightforward. Okay, so we talked about this program, right? So now the nice thing is we can explore, okay, how is this program going to actually get executed in GPU? And that is going to take us through basically the entire architecture and also the motivation for why everything exists here. First of all, let's take the example of our matrix addition kernel. So we have this code. Uh, the question is, are we going to actually load this up into the GPU? And how are we actually going to run it? And so what we'll do here is first we'll look through how this works high level in the architecture. Then we'll look into individual threads and how to see like.

• 50:05Speaker 00

  How this code is executed at an individual thread level? And then in the kind of last piece of this that might be interesting is we can actually look through the test case and briefly explain in the suggesulatory like how it's actually tested and simulated. So it's not just a theoretical. Those kernels that are written in there, there's actually a setup for them to actually get run on this GPU design and see our output and everything that's going on. And so we can touch on that last and maybe like touch on some of the interesting things in the code. So what place to all start is we have this matrix addition kernel. And the question is, how do you actually get this to execute on GPU? On one hand, you need to load up the code into there. But then it's like, okay, how do I actually make this thing run and get the result back? So the way this is going to work is first, we'll think about global memory,

• 50:50Speaker 00

  which is like the place where you're going to load up all of the prerequisite data to this code. First of all, you have the actual kernel code. And that's as we wrote below or before, it's just a bunch of instructions. And those instructions happen to be written in assembly, which is like some nice verbal format, personalize them. In reality, we would compile that assembly, just a preset itself, compile that assembly into our code, which is tested by the ISA. So it's just going to be a bunch of ones and zeros, all 16 bits long and load that into program memory. So now let's assume we've loaded that into program memory. Let's say it's like 15, 12, 15 lines long. So now we have our kernel code loaded up into program memory. And that's just going to specify exactly what a kernel set this in this. And then the second piece of it is in data memory,

• 51:33Speaker 00

  we're going to load up the actual data that we want to perform computations on. And in this case, this would be like the two one by eight matrices. We'll call them A and B. So as I said before, A is going to be in addresses zero to seven. And then B is going to be in addresses eight to 15. And then we'll say that C is going to be in addresses 16 to 23. So that's going to be our result. Now currently our result matrix is going to start out with nothing in it. And so what is the goal of our entire program? The goal is to somehow get the GPU to use that code that we just loaded up in program memory. And somehow at the end, it's going to need to fill out rows 16 to 23 in memory with the correct results of the additions of A and B. And then we're going to read that out at the end. So from the host machine's perspective, which is like the CPU talking to the GPU,

• 52:17Speaker 00

  what we need to do is basically get this some code, load it into program memory, load up our data into data memory. Somehow get the GPU to start this whole computation, it's gonna do its thing. And then, at the end of it, GPU is gonna tell us that it's done. And then, there's gonna be in data memory, the answer. And the host machine can just read out that answer. And, so that's kind of the interface of GPU, basically. So in this picture, you have global memory with program memory and data memory. And that's the one piece of that. And, then, like, another important thing in GPUs is the actual bus where data kind of transfers between the CPU host and GPU. Again, I left that out because of simplicity reasons. But the other important control thing here, besides the memory, is what's called the device control register.

• 53:01Speaker 00

  Now, in real GPUs, it's a lot more complicated, but in this case, if you remember, what we need to specify is basically the number of threads to execute. So we need to tell the GPU that, hey, this thing in program memory, which is just a single kernel code, we need to tell it, you need to run this thing eight times. This is a way that you do that is, in my design, you would store the grand count inside the device control register. And so the device control register is mainly just used to store some, like, high-level data about, like, how you want the GPU to execute. In this case, the only piece of high-level data we really need to specify is, how many threads should you be executing? And so, now we have program memory loaded up, data memory loaded up, device control register loaded up.

• 53:43Speaker 00

  Those are the three things you need to do to get this GPU to be prepared to run something. And then, that's the actual interface. Then we're gonna send a start signal. And that's gonna tell the GPU, hey, everything's loaded and ready to go. You need to start, basically, performing computations on data. And that's like a clean break. So that's like, the interface to interact with the GPU. Now we're gonna get into the computational parts, but I'll just pause there. Does that make sense?

• 54:08Nathan Labenz?

  Yeah, I think so. And, probably just worth reiterating that there's, like, even, in some ways, more complexity on the CPU side, when you say the host machine. We're sort of assuming here that you have a general-purpose CPU that can do the compiling and can handle the, like, outer loop of, here's what I'm trying to do, and get something back, and show it to you on a screen that it's done. And this sort of sits within that. So it's funny, because the programming paradigms in ways is more complicated, but then, in other ways, it's a lot less complicated, because it can take advantage of all of universal computing. Benefits of a CPU host machine. Aside from that, I beg all the sense.

• 54:51Speaker 00

  That's a good point, because GPUs are not really meant to run alone. They're always in Tachus either hooked up to a host that you'll see them next to the computer in like a rack. Or you have you connect to one on the cloud or something like that. That's not really very practical, but depending on what you're doing. But yeah, they usually have a host connected to them that can send them data and interface with them. But yeah, that's a good point. So now we can get into the computation pattern, right? And high level is actually pretty simple. So the main things to focus on is you have this dispatcher. And the dispatcher is just responsible for, as we said, these threads are executed in blocks. And a block is just like a batch of threads that gets executed together at the same compute resources. And so basically, the way to think about this is your GPU has some large number of compute resources that are organized inside cores.

• 55:35Speaker 00

  And so this GPU design happens at four cores. And each core has the capacity to execute a bunch of different threads in parallel. And from a high level, before diving into what's in cores, let's just assume the core is some black box. And we just know that this black box, we can just give it some group of threads, which is going to be a block size of threads. So in this case, four threads at a time, we can give each compute core four threads at a time. Going to do some applications and finish tossing the thread somehow. And that's going to tell us when it's done. And so the high level job of a GPU now is like, okay, how do we manage? How to divide up these threads into the available compute cores, wait for them to complete their job? And then give them more jobs when they're done completing their like complete work.

• 56:17Speaker 00

  And so basically, all it is is just managing all the compute cores, figuring out when they're available for new work. And then dispatching new blocks to these to these cores. Now, in Tachus, this is where one, this is one place where there's a lot of complexity in GPU. Because you can imagine when you have a ton of different cores performing stuff at once, and different cores can perform different blocks. Like they actually have different blocks of threads at once. And they're, they have a complex resource management. You can do a lot of different optimizations to make sure that you're getting maximum resource utilization. And so this is some place where there's like a ton of space for optimization. It shows them using things more efficiently, um, feeding in. Like you might have tried in different threads at once to make sure that everything's getting used.

• 57:02Speaker 00

  And then, of course, for simplicity's sake, in this example, we use a very single version of that, which is just that. The dispatcher looks out for when a core is done executing its threads. And just as it worked, and just like a round robin style. So it just goes, looks at all of the cores sequentially. And then keeps feeding them more threads if they're ready. That's like a simple way to think about it. And so. The dispatcher is one of the more complex elements of the GPU. Now, if we get into the individual compute cores.

• 57:27Nathan Labenz?

  Dispatcher layer for a second?

• 57:29Speaker 00

  Fair, yeah.

• 57:31Nathan Labenz?

  One thing, what level of design did you have to go to in this project to create? A dispatcher? Is that something that you had to? Design? And if so, at what level? Or was it something that you could drag off a menu of? Available components?

• 57:47Speaker 00

  Yeah, so pretty much nothing here was an available component, to be honest. So the nice thing is I had two or three repositories to reference. So there's this thing called Meow GPU, which is like. An open source Verilog GPU made a while ago by some school. I forget which school it is. And there's this thing called Veri GPU, so two open source GPUs. The thing is, these things have like very little documentation. And then they're not really written to be simple. They're written to do like. Attempts at production GPUs. One of them is like completely unfinished. Meow is actually finished. And first of all, architecturally, they're somewhat different in some ways because again, they're optimizing for production usability. And the second thing is because of how big they are, they're just not very easy to understand. And so my process for creating things was.

• 58:29Speaker 00

  Not really, you can't really go through them and try to understand. The analogy is like if aliens come to Earth and find a computer. Like, how are they going to understand it? Are they going to break it down and look at all the connections, the transistors and see what's happening? No, they're not going to understand it by doing that. They have to come, come to some understanding of their own. And then test it against the computers that they have. It's a bit of a massively over. Complex analogy for going through a GitHub repo. That's actually the analogy for mapping out the brand. Which is not a pretty good one there. But yeah, so my approach to going through these repos was not like going through them and trying to understand. It was actually like. Talking to Claude and ChatGPT about. I see like names of things like I'll see a folder called, I don't know, just that or whatever.

• 59:11Speaker 00

  And I'll try to figure out with Claude through first principles, like where does this have to fall into place in the GPU? And then that part's not too hard. So then it's like, okay, I get to something. Then I'll start to propose how I think it works. The reason is because there's not really anything that teaches you how these things work, like anywhere online, because it's all proprietary and the other own algorithms and stuff. And so even to make a simple version, you'd have to be like, so for the dispatcher, for example, I come up with the hypothesis like, oh, I think a simple version of this would be X, Y, Z. Like, I think a simple version of this would be like round robin scheduling while looking at what compute cores are available, which is like super simple. And then I would have to get confirmation. So Claude would be like, actually, that'll be that's the last thing. I think you should think about it like this.

• 59:53Speaker 00

  Or something like that, which is interesting that there's nothing publicly available. But Claude and ChatGPT, in some cases, it's things that are purely intuitive and it just has like the benefit of engineering intuition on stuff. That like I haven't gone into. In some cases, it says stuff that's definitely implemented in proprietary GPUs. And maybe it's still out there through intuition, but it's a little more stuff that's a little more complicated than what I expected it to know, which is pretty cool. But yeah, as along with an answer to your question, there is no like off the shelf stuff. I basically had to design everything myself from how I thought it would work and then get confirmation from it. Maybe look into the repo to see, oh, this repo actually happens to do something a little similar to what I suspected. And now I can actually understand it because I like came up with it first and I had to do that for all the elements.

• 1:00:37Speaker 00

  Granted, some of them are a lot simpler to understand than others, but yeah.

• 1:00:41Nathan Labenz?

  Okay, so this thing implements a round robin where it just says, I'm going to just keep looping through. As long as I'm live, I gotta just keep looping through all of the compute cores, check their status. So that implies then that the compute core is like maintaining a status that can be checked. And if it's available, then I'll send it another thread to process. And if it's not, then I'll come back through on my next loop. And all of that is implemented in hardware? You're beyond the point of, we're beyond the point of programming this at this point, right?

• 1:01:24Speaker 00

  Yeah, this is implemented in hardware. Yeah.

• 1:01:26Nathan Labenz?

  So do you have a diagram of that? Can we look at that?

• 1:01:31Speaker 00

  We can look at the code in the repo. There's no diagram. The diagram is for the red execution, but there's no like dispatcher diagram.

• 1:01:37Nathan Labenz?

  Gotcha. Okay, let's look at the code to just see what that looks like.

• 1:01:41Speaker 00

  So if you go into source and then dispatcher dot S E. Yeah, this is it. You can see all the code is documented with like at the top, you see the important points. And you can see there's some of the things that we're saying mostly there are there. First of all, the important thing with the dispatch is that has a start signal. And what that means is that thing we were saying at the top level, we're going to tell the GPU to start. The place where that actually goes is into the dispatch. So the dispatch says, oh, I've gotten full of stamp. Now I'm going to start sending off these blocks to the actual cores and after the start's execution. That's the start signal you see here. You see the thread count coming in from the device control register, which is again important for the dispatch because it needs to know how many threads does it need to execute in total? How many turns has it already executed?

• 1:02:24Speaker 00

  And that's what's going to tell it like, oh, I have this many threads. Lord execute. Let me go send them off to the compute units. Then you can see the core stage. So that's the thing where you're saying, huh, it looks like the cores must have their own knowledge or like status of of if they're ready or not. And so that's exactly what you see there. You see core done, which is like, oh, this core is done processing. And then so you see core done, core reset and core start. Those are important. This is basically the way the dispatch gets to do stuff, right? It's waiting for the core to be done. It's going to say, oh, here's a core that's done. Let me reset this core. So that's going to set all the state back to empty. So it's going to clear everything out. And then it's going to say, oh, this core just got reset by me. All right, I'm going to go start it up with the next block of threads if there's still more blocks. And if you slow down a bit, you can see there are some intermediate variables which store like,

• 1:03:10Speaker 00

  how many more blocks do I have left? So that's blocks dispatched and blocks done. And that's just saying, okay, I need to process this many more blocks of threads before this is completely done executing. And then the last thing that the dispatcher does, I'm down in a loop, but you can see at the top, there's a done signal exploded from the dispatcher. That done signal is sent out to the GPU itself. And so not that one, but in the actual model definition, a little bit further up, there's just something called the straight up done. You can see it in a couple bases output registered done. And that's going to tell the whole GPU, hey, all, everything is done. And then it's going to stop executing everything. And now it's just going to wait for the host machine to basically pull out the results from memory. And the host machine can just reset the GPU.

• 1:03:54Speaker 00

  And that's like one full execution loop. You can see that the dispatcher kind of high level is responsible for managing that whole high level execution flow. So, yeah.

• 1:04:03Nathan Labenz?

  Cool. This is code. How does this thing get translated into hardware? This is what the package that you're working in does?

• 1:04:15Speaker 00

  Uh huh. So the way that this whole thing works is just in EDA in general, you design your architecture based on what it's supposed to do. It's a functional logic in Verilog, or in this case, system Verilog, which is just like a modern version of Verilog. And then there's these entire software stack, which is what I was talking about before, which basically translates that Verilog into an actual design. And in practice, that's really hard because there's so many different steps that go into it. Like you need to synthesize the Verilog down into just like a logic. There's like a chart of the logic that's specified in there. And then because of the way that people program stuff, that logic is inherently not very perfect. It's going to be way more complicated than it needs to be. You can do some processing to basically get that logic down into its simplest form. That's still equivalent.

• 1:05:00Speaker 00

  It's that process of turning the Verilog into a logic flow is called synthesis. And then, once you do that, you can perform a bunch of analysis on like whether this design is valid or not. Then you convert that that logic that you've created into gates. And then those gates are based on a specific process. So basically you have these things called standard cells, which are specific gate designs, designs for a specific foundries process of reducing days. So you convert them into gates. And then you lay out all these gates on a huge chip design. So the the gates are just like laid out everywhere all over the chip, based on a bunch of different optimizations. Then you hook up all the gates with wire. Like the gates that need to be connected together. And then there's like several cycles of optimization on this whole process to make sure that.

• 1:05:44Speaker 00

  Everything is valid, all the timings are valid because like electrical signals need to flow through this chip. And if any of the signals flow through the chip slower than the actual clock time of the chip, that's going to cause errors. So there's like. Tons of different errors there, even just at a logic level. And then on a hardware level, because this is actually like electronics, it's not just like code. It's nice to write in code, like the complexity is way higher. There's these things called parasitics, which is like. Oh, you're actually dealing with wires and metals. And if there's wires near each other, they actually have capacitance between each other. And there's like tons of other electrical effects that can fuck with the chip. And so you need to do computations to prevent that stuff too. And so there's so many complicated things that at the very end, your whole design is there, it gets outputted into a.

• 1:06:27Speaker 00

  The layout and that's in the form of a G D S two file, which is just happens to be the file format specifying all the metal layers to send to a foundry. And then you do a final comparison called a layout versus thematic, which is make sure that this final layout you got after tons of optimization. It's actually logically equivalent to your original design. And then you can submit that layout to like a foundry. And now in practice, you're actually going to do a ton of formal verification also on this layout to make sure like. All of these bad states are impossible to get into. And again, a hardware design in general, the verification process is so important because. Unlike software, you can't ship and iterate, you're just fucked if you you mess it up. There's all that stuff, but that's what happens. Again, if anyone had to program all of that, that's that's impossible to say to get the cast, but thankfully. All these E D A softwares do it for you for a couple of million dollars.

• 1:07:12Speaker 00

  And thankfully for me, there's open land, which does it for you for free. Enough super level, but yeah. Even that has a complexity going through it, because once you do that, you run through a bunch of errors and you have to fix that in your code. Yeah.

• 1:07:26Nathan Labenz?

  Yeah, you can see where the decades have gone in terms of building up the stack. Talk about on the shoulders, giants. What does this feel like on the open version that you were working on when you go to, I'm not even sure what the right word is, but I want to say compile, but it's almost, it's more like translate from a

• 1:07:45Speaker 00

  Hardening your designs. That's what it's called.

• 1:07:47Nathan Labenz?

  Hardening. So how long does that take for something relatively simple like this? And you're running it on your local?

• 1:07:55Speaker 00

  Yeah. So local laptop, I would say it's pretty powerful for this stuff. It's an M2. So that's probably on the better side of like a hobbyist doing this stuff. And it can take 20, 25 minutes to go through the whole flow. It does like 50 different steps and that takes like a bunch of time. Does like tons of optimizations, really cool things. Like you can see all the different stuff going on and try to understand them. And then you do it as errors on any step. Often you do. And the one thing is like, obviously the stuff isn't like perfectly documented or there's like millions of errors you can run into. Obviously I think that creation, but so many different errors you can run into and it's not really well documented. Almost none of them are well documented and they're also like really random error messages. And so one of the challenges of it is it's a lot less easy to use. Like you just gotta figure stuff out and it's so much of your stuff. You have no idea what's going on.

• 1:08:40Speaker 00

  And you just gotta try your best to figure out what's going on and debug it. So I think very low transparency on the debugging side, which is one of the biggest. It does the biggest thing because that's the whole flow. Like that's basically the output of the flow. It's like what bugs do I need to fix before I can get a finalized GDS? But it's still, and you just push through it, then it works. So we got Eric Wyler, it's magical. And then you can start visualizing it and it's pretty sick.

• 1:09:03Nathan Labenz?

  Yeah, it's fascinating. How many cycles would you say you went through in this project of that 20, 25 minute hardening process? Before you got one that actually didn't error out that year?

• 1:09:15Speaker 00

  Oh, a lot. I have all of them saved in here. I can say it's like 30, 35, something like that. A lot of different cycles. It's not in the repo. They're just in my local.

• 1:09:27Nathan Labenz?

  Yes. Yeah. Okay, cool. So let's come back to the Boval GPU architecture. So we've got these modules, components that are defined in code that are hardened into an actual, ultimately, I guess the classical word would be like etching, but now with this, it's like a, it's a light process, right? At least at the most advanced nodes. So it's, I forget. Yeah, well, thank you. So the, that translation is happening with this like decades in the making, shoulders of giants, software stack.

• 1:10:02Nathan Labenz?

  And that gives you the ability to define each of these things and then it go to come back in 20 minutes and see if it worked or not.

• 1:10:10Speaker 00

  Yep, exactly.

• 1:10:11Nathan Labenz?

  And you have to then also define the interface, right? Like the compute cores have to maintain. So Max, I'm interested to see a little bit of the interfaces is always, I think about programming a lot in terms of interfaces, like where is it that it's declaring whether it's done and not done so that the dispatcher can reassign or not reassign as appropriate. But take us through the rest of the architecture, however you think best and we could.

• 1:10:34Speaker 00

  Yeah, let's look at the, let's look at the GPU dot S E file first. That's the high level interface. That's going to be interesting. And then we could look into the compute cores in detail and put into their execution stuff. Yeah, this is the high level file. There's some parameters at the top. Those are mostly unimportant. The interesting ones are the number of cores and the number of threads per block. So here, yeah, I guess that's like the two. That thing you can just totally change around. You could switch the number of cores, three or four or whatever. And you can switch the number of threads in each block, which is just going to change how many threads can get executed on each core at once, which means that there's going to be more compute resources and register files and everything on each core. And that will make more sense later. But you can see the high level interface here.

• 1:11:12Speaker 00

  You have the clock and reset. Those are mostly unimportant. Those are just like the, you have that for every design and that's just going to allow you to have clock cycles on your GPU and reset stuff. The interesting part.

• 1:11:22Nathan Labenz?

  Good question on clock also. It seems like if everything is built perfectly, everything would take the same amount of time. I'm a little bit confused as to why, if I distribute four threads across this thing, like, why don't they all finish at the same time?

• 1:11:42Speaker 00

  Good question. So the reason, the sole reason for this, and this is the reason behind my memory intuition and why George Hops said the whole problem is memory. We'll see that when I dive into that execution is like, if there was no reading and writing from memory, it would all finish at the exact same time because everything is deterministic there. It just takes the same amount of cycles for every instruction. The thing is, when you have memory, like loading from global memory, that memory is DRAM. So first of all, it's using capacitors to store memory, which is it has some latency to it. You don't just get it back in one clock cycle, basically, which has some non-optimism. The bigger thing is the bandwidth issue. So like, if I have a thousand compute cores requesting some values from memory, and memory can only read, support a hundred reads at once, that means that this is going to start to get queued up.

• 1:12:26Speaker 00

  So there's going to be like a thousand compute cores request data, then a hundred of them are going to get back data, then next hundred. Which means that certain cores, even if they're in the same, if even if they're being processed together, they might make requests to memory at the same time. They may not get the data back at the same time, because memory has an unknown latency, it's asynchronous. And that means that if one thread gets memory back earlier or later than the other ones, then it may have to wait for the other ones, or there's a bunch of stuff that happens around that. So my design, it waits for the other ones, which means it can't just keep going. And then similarly, across the cores, even if they all start executing threads at once, one core might finish late before the others, because it just stopped data from memory back earlier.

• 1:13:05Speaker 00

  It's about, it's really the bottleneck to see things, which is why Eric Baut said it's all about matching memory latencies. But that really is the bottleneck.

• 1:13:14Nathan Labenz?

  And the clock, just to set a foundation on that as well, is essentially an electrical impulse. It's a change in voltage that is getting applied through the whole chip, is that right?

• 1:13:27Speaker 00

  Yep. So the clock is just basically a, a cycle, it's like a sine wave almost, of some signal going from high to low over and over at some like periodic interval, usually a couple of milliseconds, twenty seconds, or something like that. And basically, every time the clock goes from zero to one, it's just like a sine wave, but every time the clock goes from zero to one, registers in the chip, which like deep with flops, they will take in a new value. And so that creates the way for state, that type of state transfers on the clock. So like, whenever the clock happens, which means that accepts the pulse of the sense happening in the D P. So it's like. Generally, in each clock cycle, you're going to have the start of the clock cycles and new values are set. Then you have all these wires propagating signals and the next clock cycle, the values are going to set again.

• 1:14:11Speaker 00

  And so it sets this time interval where state is set on the clock cycle. And then stuff that happens in between that time of where state is set. And that's going to help you influence what's the next state. And the other thing that's important about the clock is you can't just set the clock insanely fast because the chip actually has physical constraints, right? Electricity actually needs, if you need something on all the way on the left side of the chip to communicate, something all the way on the right side of the chip in one clock cycle. There's actually constraints there because the electricity takes time to propagate across the chip, which means that your clock cycle needs to be longer than the maximum propagation time of signals that need to communicate in your chip. And that's actually something that EDA stacked up. So that's wanting some static timing analysis, make sure that all the timings in your chip are valid. Based on the clock cycle time.

• 1:14:55Nathan Labenz?

  Gotcha. Okay, cool. So if something is waiting, again, it sounds like you could implement this in probably a lot of different ways, but you could have your cores sort of saying, I'm in waiting mode, so this cycle, I basically just do nothing because I'm still waiting for the memory to return. And then next cycle, if I got it back, then I can actually do it. And that's because there's different latencies, they can just diverge in terms of where they are in their particular execution flow. In an actual, like, production environment with the over many

• 1:15:35Nathan Labenz?

  gobs and gobs of gates ultimately that are being included in a single chip now. Presumably, there's also some difficulty there, like some just like inconsistency or whatever that could create divergence. I don't know if that's the kind of thing that would get disclosed, but I imagine there must be some sort of engineering just to try to not allow some local defect to ruin the whole thing, right?

• 1:16:04Speaker 00

  Yeah, I would say that's more on the fabrication layer. There's tons of. Like local defects is actually one of the biggest problems in the fabrication industry. That's why contamination control is such a big thing because as you start scaling down the size of individual transistors. Now, smaller and smaller defects that didn't matter before. Captures break everything because they can get in the way of gauge or. Getting, basically, there's tons of different places in the chip where a contaminant can just ruin the whole thing. And that's why there's just like all the technology on actual fabrication has been advancing to smaller, smaller scales. The contamination control technology has had to get better and better. But not being said, that's more so a concern of the actual fabrication process. The reason is that they actually have a kind of testing and the testing is built around. Like, one, they have a lot of contamination control.

• 1:16:46Speaker 00

  Two, they have a testing to make sure that even if there is some contamination breaking the chip. That they're going to be able to test that and discard the bad ones. That process is imperfect, but generally speaking, that's sort of the biggest thing that determines chip yields in practice. And generally speaking, that's a concern of the the the foundries. Not really someone like Nvidia is going to design, they'll just ship off their designs to TSMC. And the TSMC will have to be concerned with that. But generally speaking, Nvidia can probably assume that most of the chips are working once they get.

• 1:17:14Nathan Labenz?

  H, cool. All righty, should we go back to the diagram?

• 1:17:17Speaker 00

  Yeah, you can do that.

• 1:17:21Speaker 00

  So now we can go into what's going on inside each compute core. So we already saw the interface of the dispatcher. So each compute core is basically going to get told by the dispatcher. You're executing this block ID and you're going to get this many threads in the block ID. What's a, when dispatchers would say, hey, execute four threads in block number two. And tell me when you're done, basically. So what's going to happen is the dispatcher is first going to reset the compute core. Then it's going to tell it, here's your block number, here's how many threads to execute. And it's going to tell it start, it's going to send it a start signal, similar to how the GPU got a start signal. And then the question is, what's going on inside the compute core? To process those threads up until the point where it's reporting back to the dispatcher, then it's done. That's like the high level interface, interface of compute core. And so. There's a couple pieces of the core.

• 1:18:06Speaker 00

  And first of all, there's at the high level control logic that's shared between all the threads. And then there's like the actual thread execution where we split out into a number of different threads within the core. First, we can talk about the high level logic that's shared between all threads. And then we could add to the thread. So at high level, there's basically three units that are shared between everything. So there's the scheduler. And again, this is where there's a ton of complexity. It's now managing the resources of how many, however many different groups of resources there are in a compute core. Now, in practice, if a PCore isn't actually thought of as having a certain number of threads. It just has some finite number of resources. And then however many threads it needs to execute, they just stay executed on those resources. And typically, a scheduler is responsible for handling the difference between like.

• 1:18:49Speaker 00

  If this doctor says, hey, execute eight threads. And you may only have four, like four different groups of resources on your compute core. So now the compute core needs to manage the execution of these threads in parallel or in sequence, or how her determines is most efficient. So again, that's why there's a lot of complexity in scheduler. In my case, what I basically said is each compute core is just going to process one block at a time. So it's not going to be able to say like, hey, you can give me eight threads and I'll do like four at a time. It's just going to be like, just give me however many threads I support. So each compute core supports four threads at a time, which means that each resource that the thread needs. It's going to have one, one resource. So like each thread needs a register file. So a PCore has four register files. Each thread needs its own ALU. It's going to have four ALUs. So you get the point there.

• 1:19:32Speaker 00

  And basically, what happens is the scheduler handles the execution of the threads in terms of. There's let's say fifteen instructions that each of these threads needs to execute. So now the scheduler is going to constantly execute the control flow of how these instructions get handled. And so the way that happens is, let's say we're on instruction number zero. So we've just been issued such a group of like four threads for the dispatcher. And we need to execute these threads. So we're on instruction number zero. So what we're going to do is, first of all, we only know the star with instruction number zero. We don't even know what that instruction is. So first, we need to somehow get that instruction from our store, a stored in program memory. Because that's where we put it before at the start.

• 1:20:16Speaker 00

  And so the fetcher is the first thing that's going to execute. So the scheduler is going to tell the fetcher, hey, go retrieve this instruction from program memory. So that's the whole job in the fetcher, basically, is to get instructions from program memory. And then in practice, it actually doesn't catch it, because instructing are very repetitive, like all the threads are going to be executing the same instructions. So what optimization we can do that's nice is that the fetcher, once it gets an instruction, wants it to just store it locally, and then all the threads can just use that again, so it's going global memory. So that's the fetcher. And then what the decoder is going to do, which is going to basically get us into the thread, the whole is we now have this instruction, which is a bunch of ones and zeros. It has its opcode, which is going to tell you what the instruction is supposed to do, and it has all the registers and specific values.

• 1:21:00Speaker 00

  Now we need to somehow translate that instruction into something that all these different compute resources can actually use. And so that's the job of the decoder. And as we get into this, as a thread, it will become a lot more clear what the decoder is actually doing. So let's look at the actual compute resources that are in thread. This is the last piece of this diagram, and then we'll go off to the next diagram. Within a thread, in my design, there's basically four key pieces of memory and resources. But each thread has its own register file that we've already talked about, and that's each thread has its own ability to perform computations on some data. And then importantly, it has its own program counter, which means that each thread can be on its own line of the program. And that's because

• 1:21:44Speaker 00

  based on the data, different threads might have to jump around different lines of the code. And in practice, that's challenging to implement, so that's called branch divergence, which means different threads thrash to different lines of the code. Now, in our design, even though each thread has its own program counter, for simplicity, all the threads are assumed to be basically continuing on the same instruction. And it just so happens that I wrote programs where that is the case, like all the threads stay on the same instruction. So this design is sufficient for implementing that. And then last two things, which are important. So each thread has its own ALU, arithmetic logic unit, which is the actual compute that's like performing multiplication for addition. So that's through all the arithmetic instructions in there, and it's the formally computations of the registers. And the last thing is the load store unit, or the LSU.

• 1:22:27Speaker 00

  And that's the thing that's responsible for fetching data from memory. And so that's the thing that does the load and the store instructions. So as you can see, each thread has its own unit that is separately able to Load data from memory and store data into memory. And importantly, before we dive into the specifics of how all these pieces interact. So I know it's like completely unclear at this point. It's just like, oh, I guess it's like these high level things. I wonder how they actually work in practice. The important thing is that the fetcher and the LSUs. There, there's like a bunch of them across the GPU, right? Because each core has a fetcher. And that means that if we have four cores, there's going to be four fetchers. And they might all be requesting instructions at once. And similarly, there's going to be like, there's four LSUs per core and four cores. There's going to be 16 LSUs in the whole GPU. And those would all be requesting memory at once.

• 1:23:12Speaker 00

  So the question is, how do we manage the constraint of? Memory has a fixed bandwidth. Like, let's say it can only take two requests at once or something. And all these resources are requesting memory at the same time. So you need something to actually like throttle all the memory requests. And on the GPU side, hold them. So it's like, here are all of our requests. And then send them slowly to memory to respect the bandwidth that every app she supports. And then slowly send those responses back into the individual compute units in the core. So that's what the memory controllers perform. So there's one for program memory, one for data memory. And those controllers are basically responsible for respecting the bandwidth that the global memory actually accepts. And then taking all of the requests from compute resources and throttling them to the bandwidth of what the memory accepts. That's what memory controllers do.

• 1:23:58Speaker 00

  And yeah, with that, I'll pause there. And then after this, we can dive into the app for threat execution.

• 1:24:04Nathan Labenz?

  Yeah, I think I get it. And again, all of these things are, at this point, purely physical, right? There's, we've left the realm of software some time ago. So the software really stops at the point where the kernel is compiled, is that the right word in that case? And put into the program memory. And then everything after that is a purely a function of the hardware. Hardware is routing everything through just an insane tangle of ultimately logic gates.

• 1:24:40Speaker 00

  Exactly.

• 1:24:43Speaker 00

  Yeah.

• 1:24:43Nathan Labenz?

  Does anyone have an intuition for that at this point? It seems like we've got so high up this tech stack over the decades that I wonder, is there anyone that can really go to the sort of gate level and understand like how something as complicated as a data memory controller works? Does anybody have an intuition for that at the sort of logical gate level?

• 1:25:07Speaker 00

  Yeah, definitely. Yeah. So I think most people doing architecture and design stuff, they'll probably understand most of these elements at a gate level. Obviously, more complicated logic, it's futile and there's no point really to try to understand exactly what's happening at a gate level. I think the main intuition you need is understand the core of how memory works at a gate level, usually like static RAM. And that's pretty interesting. You understand how like a latch works and then a flip block works. And that's like the core of understanding memory. In practice nowadays, every like most memories dynamic and only cache is static. But that's something people understand at a low level. And then like ALUs and again, we're going to go into the diagram and that's going to show you basically as opposed to the low level as you get. It's going to show you multiplexers and stuff, which again are pieces that people understand at the hardware level. But yeah, I would say people do understand it at that level.

• 1:25:51Speaker 00

  Now, in practice, you don't really need to, as long as you know, as long as you trust the process, you're saying, okay, I know that this is going to get translated into gates. So this general A, you can study the gates in computer architecture class and you don't really need to go past that. And then I would say really crack architecture people, there's a ton of stuff. The thing that makes them really good is that they're managing a lot of stuff in their head, which most people are not. And usually that's not really a gate level logic, although I'm sure they have that in their head too. It's more like understanding the implications of stuff besides just a high level. Like design is like really nice, high level and practice, there's actually a lot of complexity. For example, writing code in Verilog, like knowing how many gates a piece of code actually translates into is important. Like for me, it's like, oh, I might just write the divide sign as my division instruction.

• 1:26:34Nathan Labenz?

  This is busy.

• 1:26:35Speaker 00

  In reality, nobody would ever do that. Like, the divide sign actually turns into a gigantic piece of hardware, which you would never actually want to use. You wouldn't actually want to implement the division thing as an instruction. So it's things like that. And then also being able to manage all the parasitics and knowing the actual electronics of it, and how the electronics work if you place things in different places. There's like tons of things like that. Honestly, the way I would describe it high level is people have a surprising level of understanding of the EDA flow, and the things that influence that flow in their head, and they can get it by intuition. Obviously, not the whole thing, but pieces that most people want. And somewhat I've noticed from my very limited perspective, that's what a lot of the really good architecture people have. So I would say certainly there's people with intuitions on all these things.

• 1:27:17Nathan Labenz?

  Fascinating.

• 1:27:17Speaker 00

  Okay.

• 1:27:18Nathan Labenz?

  Cool.

• 1:27:19Speaker 00

  Yeah.

• 1:27:19Nathan Labenz?

  So should we go to the thread diagram?

• 1:27:22Speaker 00

  Yeah, we can do that. And this is gonna be nice, especially for people who are familiar with CPUs, because this is gonna get into very familiar territory. So you can see here, something that looks very similar to a CPU in many ways. It almost is exactly like a CPU, except for the LSU and my little details in the register file. But what you see here is, first of all, most importantly, everything that has red text and blue text, that is stuff that's coming from the decoder. So now you can actually see exactly what the decoder is doing, which is the decoder is responsible for converting the instruction into these red and blue signals. And these red and blue signals are gonna be used as control signals to control the execution of an individual threat. What you see here is a single thread. Obviously, there's multiple of these per core,

• 1:28:07Speaker 00

  and all of them are gonna get the same control signals once, since the decoder is just shared between the core. And now you can see how the execution of a thread is happening with the control signals. And you can see the familiar elements from the diagram above. So you have the register file, you have the ALU, you have the LSU, you have a PC. And this is some standard CPU stuff, but the general flow of what we're seeing is you have the register file, you have the ability to access to registers from the register file, which are called the source and transfer registers. That's why they're called RS and RT. And those are also like there in our ISA. And using those registers, there's a ton of different stuff you can do with it. And so the job of a decoder is to tell the compute resource that's dedicated to the thread at any given point in time,

• 1:28:50Speaker 00

  what stuff am I supposed to do right now? And so it's gonna basically do that by turning off and on different parts of the compute unit. For example, if you look in the ALU. Let's just focus on that one unit. So you see the arithmetic unit. And that can do four different things. So we know from our instructions that the arithmetic unit and the ALU needs to also do addition, multiplication, subtraction, and division. And those are going to be like literally four separate hardware circuits that do each of those things. And so now the question is, the arithmetic unit is taking in these two values. It needs to perform one of those computations on it, right? So what it's going to do is it's going to take in this control signal, the ALU arithmetic multiplexer, which is much as a shorter multiplexer.

• 1:29:34Speaker 00

  And that's going to tell it, hey, so actually what it's going to do is it's going to do all four at the same time. It's going to both multiply, add, subtract, and divide all of them. And it's going to divide those four outputs into this thing called a multiplexer, which is just like a signal chooser. And so now you have all these possible values it could choose. Now this multiplexer value you set by the decoder is going to say, hey, which one should I actually take? So let's say we did like an addition instruction. The decoder is going to set this max value to zero, which is, we'll say, the thing associated with addition. And now that multiplexer is going to take these four inputs, just going to output the one specified by division. And that's going to come out into the ALU output box. And then here's another multiplexer, just to be exact with the design. So now you see like, okay, we have the output from the arithmetic unit.

• 1:30:17Speaker 00

  Then again, there's also going to be an output in the comparison unit on every single cycle. So it's not just like we're only performing the computation done by the things specified by the instruction. Actually, all the computations are happening because there's just some wires. So they're all just connected up. All the wires are always connected. The question is, which computation are we actually taking this time? So the ALU output multiplexer is going to choose, hey, on this instruction, this should be set to zero because we're going to take the arithmetic unit's output this time. And that's going to get outputted. And then that value you can see in the rest of the design, most importantly, that's going into the register input multiplexer. And so just to finish this part of the flow, the register input multiplexer is going to be zero this time. It's going to say, hey, we got some value from the ALU.

• 1:31:00Speaker 00

  And we actually want to use the ALU value this time and store it back in the register. So you see, okay, we're choosing the ALU output. And then that's going into the destination register, which is what RD is. And we're going to save that back into the register file in the registry specified by the instruction. And so now you see, okay, well, that's actually a full loop. For example, with the add instruction, like it's basically. Competing the addition, choosing to output that addition from the ALU, and then choosing to store that output back in the register file, and test how you get the addition value. And so similarly with the LSU, pretty simple. So you have control signals to enable read and write. And when you set those signals, and everything else is going to be off those times as specified by the decoder. It's going to make a request out to the memory controller.

• 1:31:43Speaker 00

  The memory controller is going to make a request out to memory, and it's going to type it back. And this LSU is going to be in a waiting state. And the scheduler, which we talked about before, is going to see that this thread's LSU is still waiting. And it's just going to keep waiting until the LSU gets response from memory. And the last thing, which I'll cover, is the PC. So if you go over to the program counter unit in the top right. So this is the stuff I was talking about before with the whole branch and compare instructions. So generally, the next program counter is just going to be the next line of code, which means we just want to add one to the current program counter. Pretty straightforward. However, if we're on a comparison instruction, it's still up a tiny bit to the MCP register. So if I'm on a comparison instruction, we are going to actually set the MCP register.

• 1:32:28Speaker 00

  And that's going to tell us, hey, the comparison we just did, it has these MCP values. It was either negative zero or positive. Then on the branch instruction, it's going to use that MCP tester. You can actually see the MCP signal coming in from the instruction that we said before. And what that's going to do is basically check, hey, did the comparison instruction match this MCP, like the set of MCP values? If it does, then we're going to jump to that immediate value specified by the instruction. Again, you don't really need to understand exactly what every single control signal is doing. If you want to, you can look at the diagram and it will become much more clear, more clear than my explanation. But the important thing is that you understand generally that, oh, this is the layer where actually the instructions are getting literally translated into hardware signals.

• 1:33:10Speaker 00

  And it generally the flow of how the thread gets executed, how the program counter gets updated, and how register values and global memory data values get updated. And that's basically how the entire thread gets executed, like in the kernel. And if you remember, in the register file over there. There's those three special values which give us context on where we are in for execution. That's getting set by the core and basically that's how it turns the executed. This is like the low level of the individual compute of the GPU. And again, it's very similar to CPU because at this level, we have basically entirely some similarities to CPUs. Basically, there's like a mini CPU here.

• 1:33:47Nathan Labenz?

  Cool. That's really very well done. And if you're listening in audio only mode at this point and a little confused, definitely check out the YouTube or jump straight to the GitHub project and check out the diagram because it definitely will help. If anything is unclear from the verbal description. Okay, you've designed all this. You've got all the code. We've taken the spike from sort of what you would do at the programming layer, how that gets propagated through the hardware. Is it time to go on to how you actually get to visualize and verify that this is working?

• 1:34:28Speaker 00

  Yep, we can go to that quickly. So if you go into the test folder, I'm going to show you quickly how I simulated everything. And then on the tweet, there's also a video of what's actually happening in GPU. But yeah, so in the test folder, we can still get, for example, the test that had kernel. So literally all we have here is I manually compiled all of those, all those kernels I wrote into the like valid machine language for my GPU. So here you see the matrix condition kernel. And then in Python, I wrote this external interface since the memory is external to the GPU design. It's not in the hardware, so I made like a simulator right there that basically simulates exactly the behavior of memory. On every cycle, it just checks that there's like a request to the memory from the memory controllers. And if there is, it responds with the data.

• 1:35:12Speaker 00

  The reason is because memory is not part of the GPU. It's usually like an external DRAM or a compatible with memory or something like that. So you wouldn't really include that as part of the design. So that's what we're simulating here. In the actual, if I actually take this out of the party table, there'll be like an external DRAM similarly like this. And so we load up the program memory here, load up the data memory, as I said, exactly like I was talking about there. And both these are simulated memory. And then also right there, I've set the the value a little below for the device people registered, telling it to execute eight threads. And then basically all you do is, so I have this setup program and all that does is it's actually going to take those memory values and load them in. It's actually going to take the threads and load them in. And then it's going to call start. And so it's going to set the start cycle to high and it's going to trigger everything.

• 1:35:57Speaker 00

  And then we have this like little piece of code displaying the data memory just for visualization purposes, you can see what's starting out as. And then now you're seeing the exact way to interface with it. So you're seeing like, just wait until the GPU's done signal is set to one. In other words, the GPU says it's done. So it starts out at zero, whereas I keep running cycles until the GPU says it's done. And for each cycle, the GPU is going to be processing and I have this um nice piece of code here, which is written in one of the helper files, which is just going to display out state in the GPU. And so what that actually does is it's going to show you every single thread in every single core. What are the registers being read? What are the current register values? What is the ALU output? What is the LSU output? What are like the states of the core? Is it like waiting or is it executing or is it decoding or fetching or whatever?

• 1:36:41Speaker 00

  Uh, those are really nice to see, just like the control flow and debugging. And I encourage anyone who's really curious to see that, that's like a sick learning process. Just go run the Map Matrix Edition kernel. There's instructions in the repo and then go look at the log file that gets generated. You're going to see 500 cycles. Like, oh, every single thing that's happening inside the GPU to execute this code, which is pretty fun. But yeah, it's pretty simple. And then I did the same thing for matrix multiplication. So you can see the whole execution trace. Then you could see at the end, it prints out the final data memory. And you can actually see the final values that have been correctly computed. And for anyone curious to see if it doesn't want to run it. There's a video of it, which is in the tweet thread that will probably be linked here. Cool.

• 1:37:18Nathan Labenz?

  This is awesome. Anything else we want to talk about at the real low level? Otherwise, maybe I have just a couple of high level questions to ask to conclude with.

• 1:37:26Speaker 00

  Yeah, this is great on the low level. People have questions and DM me or put an issue on GitHub or whatever. Some of the people have been contributing to the GitHub more than I expected, but yeah.

• 1:37:36Nathan Labenz?

  Yeah, there's definitely a lot of curiosity out there about this sort of thing right now, but it's not easy to go as far as you have. So I'm not surprised at all that people want to follow in your footsteps. Certainly, I'm one of them. Coming out of this project, is there anything that still feels mysterious to you? Or questions you weren't really able to get a good grasp on, things you wish you understood better? Do you feel like you have it all reasonably well understood?

• 1:38:02Speaker 00

  Yeah, so I think I got basically my goals on it and my goals with anything learning wise in tech, like in a technical capacity is usually not necessarily to be like super niche deep in every single area. Because honestly, it's an infinite time going deep in any engineering discipline. It's more so to get to a level where I understand the entire landscape from an engineering standpoint. I have all the key intuitions and I'm actually deeper than most of the people who are just going in for like an entrepreneurial engineering goal. So I don't get able to spar with the engineers in anything and like fully understand everything they're talking about. But then I'm mostly like entrepreneurially motivated in terms of, obviously in this case, there's not really much entrepreneurial opportunity. It's like a super windy industry. It's like, I'm not going to go compete in GPUs or something. But it's just how I approach technical things.

• 1:38:45Speaker 00

  And so I feel like I got to the level of intuition where I can now understand anything I need to. And I put at the bottom of the repo read me like some of the more interesting, more advanced GPU concepts that I looked into that I didn't implement. So that was fun. But yeah, generally, I think I accomplished my goals. I don't think I would want to spend the time to go super deep into any one of these things unless I find to be unless happen to be interesting to some frontier problem where it's actually useful to go into those. But that's just how I think about it.

• 1:39:13Nathan Labenz?

  What do you think this tells us, if anything, about the future of the chip industry? I feel like there's a lot of debate, obviously, around is Nvidia going to take over the world? Is AMD going to make a comeback? Why is like TSMC stock not up nearly as much as the chip designers? And I feel like somebody might say, geez, you made it pretty far in two weeks. If you were actually to try to go manufacture something, you wouldn't make it nearly as far in two weeks. That would seem to suggest that there's something a little out of whack where the like, why is all the value accruing to the design side as opposed to the actual manufacturing side?

• 1:39:56Nathan Labenz?

  But what's your take on that, if any, coming out of this experience?

• 1:40:00Speaker 00

  Yeah, I have my own opinions, but I think my opinion is so far from being credible and experienced enough to have a useful take or saying. The one thing I'll say is that I don't think there's anything wrong with the value accrual to like people with massive moats. It makes sense, especially like this late on. It's such a windy industry at the same. I think for people curious about that stuff, all I would say is just read Seven Powers. And Seven Powers will just explain why moats make sense and why, even though the guy did this in two weeks, realistically speaking, you you can't do anything to challenge these people in most cases. And again, disruptive innovation is the place where you get to challenge incumbents. But that's not really probably going to happen much here, and the incumbents are also very competent. Like Nvidia and TSMC, it's because it's such a well dissected wise industry that like

• 1:40:43Speaker 00

  incumbents can't possibly become incompetent because of how dissected it is across all this. They're busy driving the world to semiconductor industry. So it's definitely challenging and there's also like a lot of takes. Again, like I can't claim to know about if you can challenge TSS here in video. But a lot of employees in Nvidia, for example, there's like the one viral tweet the other day where the guy who just left in video was like, now that I've left a video, I can confirm that you're not got you out to them in the next 10 years. And I would think that those could affect us are probably pretty accurate, given the nature of the industry and the economics of it. It sounds like it definitely is too pill to climb. I think one thing that's really interesting, I'll shout out one company here. They don't need my shout out at all, but like it's called Atomic Semi. And I think that is probably the coolest attempt at doing anything in this industry in the past 20 years.

• 1:41:28Speaker 00

  It's this kid Sam to move and still a semiconductor fab in his garage. And now he's trying to build cheap affordable semiconductor fabs for initially the mid-market, like the FPGA market. And there's not really an incumbent there. Like basically, nobody has ever done it before. The question is not like, can he beat the incumbents? It's just, is it possible to create it? If he succeeds there, he's going to create a gigantic mid-market. It's kind of like what we were talking about here, where I was like, oh yeah, hobbyists knew this stuff. But realistically, like most people, if you want to hit rock and scale, you just have to hit the incumbency. You just have to use the incumbents. I think this is something where because he's tackling down market at first, he's actually legitimately creating a new market opportunity if he succeeds. So I think that's the coolest attempt in terms of this market, trying to do something that will make an impact on the young person.

• 1:42:14Nathan Labenz?

  And the value there is custom designs for relatively like comparatively low end use cases?

• 1:42:21Speaker 00

  It's not even that. So it'll actually scale probably pretty far, but it's more like a current factory costs like $10 billion. And if you want to use them, you have to like chip off your stuff and wait a long time to get back test samples. And then there's also the FPG market, which is a lot faster, but you don't actually get like a real design. It's like PDA. And basically what we're doing is creating fabs that are like way cheaper, like on the order of hundreds of thousands instead of billions of dollars. And you can get your designs immediately because you just buy a fab and you just have it with you. Which is first of all, very obviously going to completely change it for a market. But eventually it may actually replace down and mid-market opportunities. So it's actually pretty big in terms of you don't even need to go to the huge foundries. You can just use these things. It's way more price efficient. So that could be huge.

• 1:43:04Nathan Labenz?

  Yeah, fascinating. So the, the idea there would be like enterprises would run their own small fab in the same way that today they might run their own cluster. Or they could go to a cloud provider in the future. They might run their own fab internally because it's been modularized.

• 1:43:20Speaker 00

  To the point.

• 1:43:20Nathan Labenz?

  Exactly. Yeah. Fascinating.

• 1:43:22Speaker 00

  Yeah.

• 1:43:22Nathan Labenz?

  Do you have any thoughts? One of the paper that has really caught my eye recently was the Microsoft 1.58 bit paper where basically they showed that. You could implement neural networks without, there's obviously been a ton of work in like quantization of the the values within neural networks, right? We go from full long floating points down to smaller and smaller bit resolution. The smallest that seems to have been demonstrated to work so well so far is this 1.58 bit, which is just minus one, zero and one are the only available values in a network and they still show that you can actually get them to work with that.

• 1:44:04Nathan Labenz?

  One of the things they said in that paper was that this opens up potentially totally different hardware future because obviously you don't need nearly as much complication to just handle one, zero and minus one as compared to all the math if you're doing longer precision numbers. Does this give you any intuition for how that might play out in the future?

• 1:44:26Speaker 00

  Yeah, so huge of two. One of the things that people criticize about the big measurements of compute power nowadays is flops, which is floating point operations per second. And that's like kind of the benchmark for compute towers, especially in GPUs nowadays. And one of the reasons they criticize it is because if you have 32 bit floating point values, if you just switch those to 16 bit floating point values, like drastically reduce the flops because it just takes less time to perform computations on that size. And sorry, drastically increase the flops. And that's actually what Nvidia has been doing. So like they just released their black well. One of the things they've been doing, I obviously they've been boosting their flops in many other ways, but their black well now has smaller floating point like numbers because ML doesn't really need gigantic floating point numbers on like graphics.

• 1:45:09Speaker 00

  And so by doing that, just by nature of doing that, it massively boosts the flops. And it is actually meaningfully faster for the use cases of less than two. And so it's exactly an implication because if you use two bit, 1.5 bit values to stuff, the flops is going to go up like crazy. And the important hardware thing is, yeah, I heard it's not to happen. Like what that actually means is that those register values, like in this case, again, this is what I meant by it's a bit simplification. You can explain important things with it. So the register values in this case, they're 15 bit register values, right? They store 16 bits in them. And in real GPUs, they have a whole host of registers. They have registers that can store floating point 16 and 32 bit numbers. They have scalar registers and they also have vector registers. So what this means is like in the hardware.

• 1:45:52Speaker 00

  You'd actually have registers that store two bit values or something like that, one bit, two bit values. And that's going to be way faster. And that's how it gets implemented in the hardware. But yeah, that's pretty interesting. Yeah.

• 1:46:02Nathan Labenz?

  It seems to be a big simplification in terms of just a lot of the the components too, right? Like you imagine the sort of arithmetic layer that you had where you're saying it's doing addition, subtraction, multiplication, division all the time. If you're down to just a much more limited set of logical operations that you need to perform at each given step, that can presumably just get a lot smaller too, right?

• 1:46:29Speaker 00

  Yeah, definitely. Although I'm sure that's what determines the the flops increasing, but I'm sure we're grossly oversimplifying a lot of the things because I bet you wouldn't actually just have just two bit floating point registers in your whole GPU. So there's probably a lot of complexity to it, but as a high level intuition, I'm sure this is generally valid.

• 1:46:46Nathan Labenz?

  Okay, cool. That's all I've got. Anything else you want to talk about? Otherwise, I maybe just ask you what you're on to next.

• 1:46:53Speaker 00

  Yeah, honestly, this was it was cool. It was good talking about it too. I didn't expect it to be this much technical detail. I guess another interesting thing is the place that AI played in this. I guess there's not too much to say there, except that it's very clear that AI has a huge place in learning now. And I think part of like part of my learning philosophy. And in many ways, people may see me like people have seen my stuff now may think of me as like a chipped person right now or something like that. But really, I'm just trying to learn a lot of stuff really fast based on a completely different high level framework I've been using to decide on the stuff which I'm going to talk about. But I think what AI has made possible is my learning philosophy, which is extremely aggressive. Like I believe because of this, a number of different things that I can learn stuff and people can learn stuff like a hundred X, two hundred X faster than they think. Obviously, as I said, it can be added with less complete depth, but with the

• 1:47:39Speaker 00

  same or more practical knowledge. I think AI has enabled that in a pretty crazy way. And this has helped me to really feel the truth of that. I was like, in many ways, this could not have been possible without AI. So that's one interesting takeaway I have.

• 1:47:51Nathan Labenz?

  Cool. Love it. This has been a fabulous walkthrough of Tiny GPU. Congrats on a successful speed run. I'm looking forward to what your next one will be and certainly will be following you for updates. For now, I will say Adam Majmudar, thank you for being part of the kind of revolution.

• 1:48:10Speaker 00

  Thank you for having me.

• 1:48:12Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.