Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]

Today, my guest is 21-year-old Gavin Uberti, who dropped out of Harvard to build Etched, which is one of the most fascinating companies I’ve seen. The topic of our conversation is the ongoing revolution in artificial intelligence, and more specifically the chips and technology that powers these incredible models. To date, general-purpose AI chips like Nvidia GPUs have powered the revolution, but Gavin’s bet is that purpose-built chips, hard-coded for the underlying model architecture, will dramatically reduce the latency and cost of running models like GPT4. We’re about to embark on what Gavin calls the “largest infrastructure buildout since the industrial revolution”, and I won’t spoil what he thinks this will unlock for all of us. It is so uplifting to me that someone so young can be working on something so big. Please enjoy this great conversation with Gavin Uberti. Check out Etched. AI Listen to Founders Podcast For the full show notes, transcript, and links to mentioned content, check out the episode page here. ----- This episode is brought to you by Tegus. Tegus is the modern research platform for leading investors, and provider of Canalyst. Tired of calculating fully diluted shares outstanding? Access every publicly-reported data point and industry-specific KPI through their database of over 4,000 drivable global models hand-built by a team of sector-focused analysts, 35+ industry comp sheets, and Excel add-ins that let you use their industry-leading data in your own spreadsheets. Tegus’ models automatically update each quarter, including hard-to-calculate KPIs like stock-based compensation and organic growth rates, empowering investors to bypass the friction of sourcing, building, and updating models. Make efficiency your competitive advantage and take back your time today. As a listener, you can trial Canalyst by Tegus for free by visiting tegus.co/patrick. ----- Invest Like the Best is a property of Colossus, LLC. For more episodes of Invest Like the Best, visit joincolossus.com/episodes. Past guests include Tobi Lutke, Kevin Systrom, Mike Krieger, John Collison, Kat Cole, Marc Andreessen, Matthew Ball, Bill Gurley, Anu Hariharan, Ben Thompson, and many more. Stay up to date on all our podcasts by signing up to Colossus Weekly, our quick dive every Sunday highlighting the top business and investing concepts from our podcasts and the best of what we read that week. Sign up here. Follow us on Twitter: @patrick_oshag | @JoinColossus Show Notes: (00:03:41) - (first question) - Born too late to explore the world, too early to explore the stars (00:05:59) - Interpreting and defining superintelligence (00:07:20) - Excitement we can have for an AI driven future (00:09:25) - Overview and basic terminology of the transformers that power AI (00:15:53) - What Q* is and the rumors around it (00:20:41) - Robotics, machinery, and what’s interesting about them (00:23:18) - The problem of latency and computing power (00:8:55) - Needing to build physical infrastructure that doesn’t exist (00:32:18) - Inference and training AI models (00:36:00) - Major stages of chip design and the upper limits of speed (00:45:56) - Customers for billion dollar generative AI models (00:48:56) - A sci-fi-esque reality and the politicization of AI (00:50:38) - The Bitter Lesson and the implications of powerful AI models (00:56:27) - The most important companies in the AI space today (00:61:52) - Strategically building a defensible AI product (01:04:07) - Software development and why other AI companies fail (01:06:51) - Specialization and chip performance improvement (01:15:34) - Why the transformer remains the leading architecture (01:17:26) - A proliferation of models beyond the major players and data access (01:21:19) - The kindest thing anyone has ever done for Gavin

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Patrick O'Shaughnessy?

  I know firsthand how complex the tech stack is for asset management firms. And seemingly every new tool and data source makes the problem even worse, adding more complexity, more headcount and more risk. RidgeLine offers a better way forward. One unified platform that automates away the complexity across portfolio accounting, reconciliation, reporting, trading, compliance and more, all at scale. RidgeLine is revolutionizing investment management, helping ambitious firms scale faster, operate smarter and stay ahead of the curve. See what RidgeLine can unlock for your firm. Schedule a demo at ridgeline.ai.

• 0:35Patrick O'Shaughnessy?

  Hello and welcome, everyone. I'm Patrick O'Neal and this is Invest Like the Best. This show is an open-ended exploration of markets, ideas, stories and strategies that will help you better invest both your time and your money. Invest Like the Best is part of the Colossus family of podcasts and you can access all our podcasts, including edited transcripts, show notes and other resources to keep learning at joincolossus.com.

• 1:00Speaker 01

  Patrick O'Neal is the CEO of Positive Sum. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of Positive Sum. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of Positive Sum may maintain positions in the securities discussed in this podcast. To learn more, visit psum.vc.

• 1:29Patrick O'Shaughnessy?

  Last week, we heard from 99-year-old Charlie Munger on the show. Today, my guest is 21-year-old Gavin Uberti, who dropped out of Harvard to build Etched, which is one of the most fascinating companies I've seen recently. The topic of our conversation is the ongoing revolution in artificial intelligence and more specifically, the chips and technology that powers these incredible models. To date, general-purpose AI chips like Nvidia GPUs have powered the revolution. But Gavin's bet is that purpose-built chips, hardcoded for the underlying model architecture, will dramatically reduce the latency and cost of running models like GPT-4. Gavin thinks that we're about to embark on what he calls the largest infrastructure build-out since the Industrial Revolution, and I won't spoil what he thinks this will unlock for all of us.

• 2:14Patrick O'Shaughnessy?

  It is so uplifting to me that someone so young can be working on something so big. Please enjoy this great conversation with Gavin Uberti.

• 2:23Patrick O'Shaughnessy?

  Gavin, it's really hard to know how to tackle this enormous topic, but you've become one of my favorite people to learn about the future of artificial intelligence, models, hardware, software, data centers, a million things that affect tons of public equities, tons of soon-to-be applications. Maybe an appropriate place to begin would just be one of your early investors told me, you said this interesting line to him, which was something like, "You felt like you were born too late to explore the world, but too early to explore the stars." And you were sort of depressed by this. But then, maybe this whole world of superintelligence and AI allowed something in between or even more interesting than the stars. I would love to hear the origin of that reflection of yours and what you think about it now.

• 3:06Speaker 02

  I strongly agree with this. One can't help but think that 100 years ago, 200 years ago, there were so many mysteries left to be solved, that low-hanging fruit, so to speak. Somebody smart could go into their, whatever they had instead of a garage, think for a couple of years and come up with optics or physics. You could go and explore continents. Then, as technology developed, that became no longer feasible. That stuff had already been done. And yes, you could still do interesting and novel work, but it often required going to school for 10 years before you could learn all the mathematics you needed to go do new mathematics. Say, chemistry. You had to go study all that had come before. And someday, mankind will go to the stars. They'll be able to explore other planets.

• 3:51Speaker 02

  And that fruit will be low-hanging, too. But that stuff is very expensive today. But I think AI is very unique. Because unlike mathematics and unlike chemistry, the field is so young that we haven't had 100 years to develop it. We haven't had 100 years to solve these low-hanging problems. A single person can make a major, major impact. And it's got even better with the transformer. All this previous technology of convolutional networks and support vector machines was thrown away in 2020 when GPT-3 came out, showing that transformers really are the best way to do things. I am really emboldened by this transformer revolution. I think it is the most interesting problem of our time.

• 4:34Speaker 02

  And I think that it is very accessible because there's no hundred years of history built up around it.

• 4:41Patrick O'Shaughnessy?

  Can you give me your interpretation of the idea of superintelligence?

• 4:47Speaker 02

  I don't think superintelligence is some black and white thing. There are some people who say that one day the machine will be as smart as us and the next they'll be a hundred times smarter. I don't think that's true. And the reason I don't is because to go from GPT-2 to GPT-3 to GPT-4 requires spending exponentially more compute. So I don't think you get to this crazy godlike superintelligence just overnight. I think you have to go build an enormous data center to support a thing like that, way, way bigger than anything on earth today. I don't believe in this concept of fast takeoff. I think that we're going to spend two years building GPT-5. And then we're going to build new data centers for GPT-6 and that'll take three, five years. And then we'll keep iterating on that, slowly building better and better and better ones.

• 5:33Speaker 02

  And eventually they'll begin to seem human level. They'll begin to show human level agenticness. And then they'll become better than humans in some avenues. GPT-4 already is a better lawyer than I am. So I think superintelligence is much more a spectrum than it is a binary thing. And eventually we'll look and we'll say, well, GPT-12 is obviously super intelligent. This thing is way smarter than any human. It'll be clear in retrospect. But when you're in the middle of that curve, it looks flat.

• 6:02Patrick O'Shaughnessy?

  What role do you think humans will play at that stage? I mean, obviously like safety and alignment and all these things are euphemisms for managing fear of something that is a hundred times smarter than us, even if that takes much longer than we think there is no fast takeoff.

• 6:18Speaker 02

  I'm super excited.

• 6:20Patrick O'Shaughnessy?

  Yeah. What's the balance of excitement and fear that you have?

• 6:22Speaker 02

  It is very hard on the excitement side. I cannot wait to see what kind of books, what kind of movies, what kinds of video games a machine that's 10 times smarter than me can build. You look at the films, somebody like Spielberg, a great director versus some Joe Blow like myself, you say, wow, that is incredible. Imagine what a director ten times better than Spielberg could put together. Wouldn't you love to see that? So I think there's so much to be excited about. AI is making breakthroughs in science and technology, providing some sort of universal basic income. And yes, that stuff's probably more impactful.

• 6:59Speaker 02

  But like viscerally, I am most excited to see what kinds of art and games and movies and TV shows these things put together.

• 7:08Patrick O'Shaughnessy?

  In that world, is there a role for people?

• 7:11Speaker 02

  Of course.

• 7:12Patrick O'Shaughnessy?

  Are we going to do anything but be the ones that are entertained?

• 7:15Speaker 02

  Well, I think chess is a great analogy here. The machines have been superhuman at chess for twenty five years. And back when Deep Blue first beat Garry Kasparov, people said chess is gonna die. Whether through lack of interest or through rampant cheating. And the opposite has happened. Chess has become more popular today than it was back then. Go as well. Machines became super intelligent at Go in twenty sixteen. And yet there was no death of all the human Go players. People just said these things are better at it than us, but there's still a lot of value to be extracted by humans playing other humans. And to even go a step back, machines have been faster than humans for a long time. I can outrun a car. But yet I still go running. People still run marathons.

• 8:00Speaker 02

  There is so much value, so much fulfillment in doing a task against other humans, even if the machines are in some sense better.

• 8:08Patrick O'Shaughnessy?

  Can you explain a lot of the basic terminology to us? I want to step back and think about this as an opportunity to really learn the basics of AI. It's the component technologies that make this possible. So that we have those building blocks to talk about what they might do for us in the world. And you've got lots of really interesting ideas about what might happen. But let's start with the transformer. So what is a transformer? What does architecture mean in this context? If you're baking one of these things onto the silicon? Start with the basic terminology of why the transformer is such a key piece of technology or a key idea that's driven so much of what's exploded in the last year.

• 8:48Speaker 02

  Well, at a high level, transformer is a sequence to sequence model. You put in a sequence, you get out a sequence. And the way that you train it determines what conversion is done. But that's not a very satisfying answer. So let's go one level deeper. One of the other real breakthroughs for the transformer was realizing that.

• 9:07Speaker 02

  It takes a lot of data to teach a machine how to be helpful, how to be honest, how to give these useful responses. But we don't have to train it on just helpful, honest and useful responses. Trick they found is imagine we take a machine and we give it the first hundred words of a document. And we say, given these first hundred words, guess what the next word is? And then given those hundred and one words, guess what the next word is? When we play this game again and again and again, trillions of times. Now, at the end of that, you're going to get a machine that is bad at guessing the next word. Maybe it gets it right eighteen percent of the time.

• 9:48Speaker 02

  But to even get it right eighteen percent of the time, you have to have a huge number of concepts stored within that machine. So they take these transformer models, they feed them a huge amount of text, playing this guess the next word game. And at the end, you get a model that is able to predict the next word in science papers by understanding some of science. The next word of internet stuff, having read a lot of webpages and so on and so forth. And the second step on top of that is what they call RLHF. Where you take the model that has been pre-trained with predicting the next word and has all these concepts stored somewhere inside of it. And then you ask it to, hey, mimic this text that is helpful and honest. You give it some examples of debugging code and being helpful.

• 10:33Speaker 02

  And this second piece requires a lot more extensive data, takes much less computational time than that first step. So after you stack these two things, you get your GPT model that takes in a word. I'll put the next word that it thinks this helpful, honest assistant would say. And I think a good analogy here is children going to grade school.

• 10:56Speaker 02

  We want to go teach kids how to be good investment bankers, how to be good computer programmers. But you can't really just throw a toddler into a computer programming course and expect them to get it right. That's too hard. You need to get them to have some baseline stuff. And this baseline stuff has to be easy to grade. It doesn't necessarily have to be super relevant. So you put the kid in grade school for 13 years, having him do tasks that are not that related, but easy to get feedback on, much like the transformer goes to the pre-training step where it takes in a bunch of words, predicts the next one. It's not super relevant, but it is easy to grade. And after grade school or after pre-training, you have a

• 11:38Speaker 02

  model to understand some core concepts. And then, and only then, he is sent to college or RLHF, where you can put these other concepts on top of this and make it into a helpful honest assistant or a good computer programmer.

• 11:53Patrick O'Shaughnessy?

  It's pretty amazing how cogent and incredible some of the answers you get when you consider that what is going on is that it is predicting the next token or the next word over and over again, that it's not thinking ahead of that token. It's sort of constantly guessing just the next one. This seems like an extraordinary limitation. What other kinds of models? You've talked to me about tree search before. I'd love to hear your thoughts on that. Like, I can't believe how powerful this thing is just by predicting a little bit ahead. Do you think we'll be able to get to a point where we predict a lot ahead and have yet more impressive answers?

• 12:27Speaker 02

  I do want to clarify a misconception a little bit. Just because the models output words word by word, doesn't mean that they're not planning ahead. They have all these internal layers and there is very strong evidence that there is a little bit of planning going on. Much like when you and I talk, I give one word at a time, but I'm able to make internal plans that you can't see about what I'm going to say after that. And as you would expect, if you have some internal plan about what words come next, it makes you better at guessing the next word. So this pre-training helps to develop this skill. Now that said, I do think there is a lot of work to be done to make these models not as autoregressive as they are right now. Here's a good analogy. Imagine a chess engine, like a stockfish.

• 13:12Speaker 02

  The way these work, the neural ones that are the most advanced today, is not by just looking at a board and guessing the next position. Instead, you're gonna try a huge number of possible sets of moves. You'll play hundreds, thousands, maybe millions of games against yourself and your simulated opponents. And at the end of all of those, you'll say, okay, this position I found myself in, how much do I like that? Then you can kind of backtrack, saying, okay, based on where I want to be, this was the best tree to get there. And I think that language modeling might go a similar direction. Where the model makes a couple hundred plans that, hey, here are the next hundred things I could say. I like this path the best, so I'm gonna go pick that. However, this hasn't really caught on quite yet.

• 13:58Speaker 02

  There's a concept called a beam search in the literature that does some of this. Beam search, I think, is much more primitive. It usually has at most four beams, but they don't look very far ahead. But first of all, models can be kind of helpful. And beam search also requires four times more compute, which is a very large overhead to pay. So this beam search, tree search stuff, has largely not caught on yet. Although, with the rumors of Q-Star learning from OpenAI, if you believe them, that does suggest that they are doing similar tree search stuff in there. That's where the star comes from.

• 14:31Patrick O'Shaughnessy?

  Maybe describe, just for the uninitiated, what Q-Star is? A rumor I should say around Q-Star.

• 14:37Speaker 02

  Well, allegedly, it is the combination of Q-Learning and A-Star tree search. This Q-Learning being a way to do stuff on top of RLHF. And the A-Star being, hey, we're gonna pick which of these paths is the best. That said, I don't want to put too much credence in this. It really is just a rumor, and there are so many ways I think like this could work.

• 14:57Patrick O'Shaughnessy?

  Talk to me a little bit about how you see the future unfolding from where we sit with a handful of very impressive models. We'll just use GPT-4 as the main one that I think people know about and have used before. I'm using this 50, 100 times a day, like, pretty much constantly. Lots of really great use cases, but for the most part, I'm in a chat window in their OpenAI's window. I'm putting something in, I'm taking something out, and that's kind of the extent of my use. It's super valuable, I'm happy to pay whatever it is, 20 bucks a month for it. But I think you see a world where these models get used in a lot more places. So I'd love you to just sort of open our minds to how you think the future might or should unfold, and what the constellation of technologies will be that are required to get us there.

• 15:41Speaker 02

  Well, I think the right way to think about where this tech will go is to think about how this tech got here. And there are a number of innovations that have brought us to this point. But the biggest by far in my mind is scale. We have gone from a one and a half billion parameter GPT two model to a hundred seventy five billion parameter GPT three model to a multi trillion parameter GPT four model. And companies seem set to spend ten times more cash training their next generation models as well. Making these models bigger makes them better. And now I want to call this out because this is not how AI used to work. Previously, if you took a stats class in like twenty twelve, you'd hear about overfitting. If you put too many parameters in your model, it will fit the data too closely.

• 16:27Speaker 02

  And it will get worse results when you actually test it and use it in real world use cases. Imagine that you're sort of teaching the test. You give it the answers to the test and that makes it do very well on the test, but it's not going to do very well in real life. But for whatever reason, transformers seem to not fall into this trap quite so much. They are, to quote one of my teachers in college, the first capitalist AIs. Adding more money makes them better. So based on that, I think we are going to see GPT five, next gen models from Google and Anthropic be smarter than GPT four is today, and that's going to unlock some interesting use cases. Agents are a thing people have talked about for a long time.

• 17:09Speaker 02

  Imagine a GPT four thinking to itself and taking actions autonomously. And to date, people haven't had very much success here. In large part because GPT four is not quite smart enough to get out of loops and to go do the kind of planning needed to operating on its own. But I think we're almost there. I think when we scale up with GPT five and models beyond that, we will see agent use cases becoming more mainstream. And I also believe that this chatbot interface that I love as well is definitely not the end use case for these models. Any scenario where you are reading chat, the language model writes token by token by token is not a use case that is expensive.

• 17:54Speaker 02

  The model is probably going to be cheaper than human eyeballs. Human time is very expensive and the first thing you'd think to try. But there are real technological limitations that stop other use cases from becoming the standard. Take, for example, speech, this podcast we're doing right now. I'd love to be able to talk to GPT-4 much like I talk to a human. It makes such a difference at talking versus descending text. It's so much easier to be misinterpreted then. And while people have tried to build transformer speech to speech models, it's challenging because of latency. You talk to it and there's this pause before you get your response back. And that really breaks the flow of a conversation.

• 18:38Speaker 02

  So I think we need tech that will enable stuff like tree search by bringing the cost of running these models down by a factor of 100. I think we need tech that will bring the latency down by an order of magnitude so we can go have conversations with these models. I think we need tech to allow them to generate 1,000 tokens per second and make that code writing take two seconds instead of 20. So I think there are so many use cases beyond this chatbot interface that are waiting to be built. The tech's not quite there yet.

• 19:10Patrick O'Shaughnessy?

  Can you talk a bit about robotics? And by robotics, I mean like physical machinery, technology. I know you did some stuff with the robotics in high school. What's interesting to you about robotics vis a vis these models and performance and latency specifically?

• 19:26Speaker 02

  I think robotics is a very interesting use case that's currently kind of stuck in the before times.

• 19:33Speaker 02

  Robotics is a field that works very heavily on feedback loops, on hard coding algorithms, on having some while loop that says, okay, move the arm until it goes and welds and then do this again and old tech kind of stuff. People have tried to go and put some AI models into these robots, but it's very hard because of this lack of training data. For images, you can go feed it a million things from ImageNet, a million frames. For robotics, you can't have it break there about a million times, port figures out how to walk. That's just not affordable. And while some people have tried to build simulations where the robot can go interact with the world, these don't work super well because of the difference between a simulation and the real world.

• 20:18Speaker 02

  You need the robot to understand the real world without actually having to break itself 100,000 times. But transformers, I think, are an interesting way to solve this problem. There's a great paper out at Google, two years ago, I think, RP1, that proposed a robotics multimodal transformer. One of the other beauties of a transformer is that it can take in inputs in almost any modality. And the modality is a thing like text or images or video, or in this case, a robotic sensor input. And you can train a model on many different inputs. In this case, they pre-trained the model with a huge amount of text, but then it built some understanding of the world from this guess the next word game.

• 21:02Speaker 02

  And then on top of that, add in this robotics encoder stuff. So I think that robotics is a super interesting use case for these models. I think that when transformers are put onto robots, they won't be trained just with the robotics inputs. It'll be pre-trained like today's and you'll add those other sensors later. It's tough, though. You also need some very good response times for a robotics model. That for text, it's okay if it takes a second to go off your machine, be processed and then come back. A second is too long if you're about to follow. So latency is so, so critical for any kind of application here. It's not quite there yet. There's been some limited adoption. And the Tesla self-driving cars today, they use vision transformers.

• 21:46Speaker 02

  So much like a language model, but it takes images instead. The internals are almost the same. But they haven't hit the mainstream for robotics, largely because of this latency problem.

• 21:57Patrick O'Shaughnessy?

  It seems like latency is like the thing to focus on. I mean, there's two roadblocks, I guess, to more huge applications, companies, use cases being built on top of this technology. One is just how smart the thing is, and that seems to be on a one-way trajectory. I'd love to hear what you think the limits to that are. Then the second seems to be just latency, like so many of the things you've described, I would say we need real-time AI or something. Whether that's conversation, translation, robotics, so many of the things you've described seem like real-time is the thing that's really missing here. So maybe describe how we can get over that hurdle.

• 22:33Speaker 02

  I strongly agree that real-time AI is the bottleneck here. And to go a step deeper, what do we mean when we say real-time AI? What do we mean when we say we need better latency? We measure it in two ways. First, the number of milliseconds between when you submit your response and when you get your first token back. So you hitting enter when you're seeing the first answer from chat GPT. And then the time between subsequent tokens. Now, why does the first token take so much longer? In theory, it's only running the same model. Well, that's because it's running the model on the entire prompt. If you give it, say, 200-token prompts, and then it has to run all those 200 tokens before it can give you your answer back.

• 23:16Speaker 02

  And it's actually a little bit worse than this, because imagine that you have some conversation, six messages long. And so very expensive to go store all that in RAM, all waiting for you to write your response, is usually cheaper to just recompute it. So if you hit enter and your chat's already six messages deep, you and chat GPT talking to each other. It's going to have to feed that entire message history back into the model before it gives you your next tokens back out. So instead of being 200 tokens, even if that was your last message, it has to go compute this model on 1,200 tokens. So if we want to bring this initial delay down, we're going to need chips that have way more compute and are able to use that really efficiently.

• 24:00Speaker 02

  And I think the best approach here is model-specific chips. Imagine a chip where you take the transformer model, this family, and burn it into the silicon. Because there's no flexible ways to read memory, you can fit an order of magnitude more compute and use it more than 90% utilization. This lets you solve this problem and get responses back in milliseconds instead of seconds.

• 24:25Patrick O'Shaughnessy?

  Talk a little bit more about the actual machinery of that way of doing things. This makes me think of ASICs from the Bitcoin era when those got so popular. Maybe describe what an ASIC is, but I remember these chips that became ever more specialized, that literally just Bitcoin mining, just guessing hashes basically over and over again as fast as humanly possible. And anyone that had an ASIC outperformed someone that had a GPU, outperformed somebody that had a CPU. So maybe just talk us through, everyone's heard those three terms probably, but may not know like what the differences are and what the trade-offs are. What are ASICs and how does this work?

• 25:00Speaker 02

  Well, let me give it to you in the context of Bitcoin. In the very old days, back when the Bitcoin algorithm was first devised, people ran it on CPUs. And to go mine a Bitcoin, like you're saying, you have to solve a very hard math problem. And the only way to solve this math problem is by guessing and checking answers very fast. A CPU can do many things. It can do this too, but it's not very fast about it. CPU mining was originally the standard. People then said, hey, let's run this thing on GPUs. A GPU has several hundred mini CPUs scattered across the silicon. And because of this, it's able to go do many things in parallel. Each one of these cores in a GPU can do all kinds of different tasks.

• 25:43Speaker 02

  Once people wrote GPU miners, CPU mining became so bad by comparison that you'd lose money running it. But then people said, well, what if we took a step further? On a GPU, only a tiny fraction of that GPU is used for mining Bitcoin. We don't need the caches, we don't need the IOs, you don't need the crazy instruction processing and L0 caches. What if we specialize a chip? Put every transistor on that chip so it would just mine Bitcoin. And when Bitmain and MicroBT did this in the 2012, 2013 era, we saw an explosion in how fast we were able to solve these problems. The Bitcoin algorithm made them much harder to compensate, and GPU mining became unprofitable overnight.

• 26:28Speaker 02

  Because of this, GPUs have not really been seen in Bitcoin mining for almost 10 years. That's not to go say that GPUs were scrapped overnight. They found other use cases. They're flexible. They went to mine Ethereum, or went to go do AI, or went to play games. But not Bitcoin. Once you have an ASIC for something, it is just not commercially viable to run on GPUs anymore. I think the same exact thing is going to happen for transformers. Previously, in like the 2010 era of Bitcoin, there wasn't really enough volume to justify spending $100 million to build a custom chip. But once Bitcoin became popular, there totally was. Then the chips came out, and everything else becomes no longer feasible.

• 27:14Speaker 02

  For transformers, even 18 months ago, there wasn't enough demand to justify spending $100 million to burn that algorithm into the sewer. But now, with ChatGPT, GitHub Copilot, with Character, with all these models from Google and Anthropic, it does make sense. The first transformer ASICs are coming.

• 27:34Patrick O'Shaughnessy?

  Can we talk about this progression? So I'm just going to use, I'll stop saying ChatGPT just to abstract away. But let's say we're at model level four right now, and we're going to go to five and six and seven. You referred to the need to probably build things that don't yet exist, not just chips, which is obviously what you're working on, not just training the models, not just gathering data, but actually the infrastructure, the physical infrastructure to do this. What do you think happens to data centers and power sources and things like bandwidth? It just seems so interesting that we're going to have to build new stuff to enable each successive progression of new model. So describe how you think that iterative thing might play out.

• 28:13Speaker 02

  I think there's a great analogy here, actually. And that is semiconductor fabrication, the buildings that make microchips. These incredibly advanced facilities have incredibly complex supply chains. Many have heard of a TSMC and the machines they buy from ASML for $300 million. And the mirrors ASML buys from Carvel's ice. Well, that's just the photolithography machines. A huge array of complexity goes into this stuff. And that means that building a new fab costs 10, 20 billion dollars. And I think we're going to see a similar thing for a models of level five and six. This whole chain will have to be redone.

• 28:53Speaker 02

  If you want to go and scale from four to five, have the same multiple of compute over a three to four or two to three. You have to have 10, 100 times more chips. And that means 10, 100 times more power. This is not going to be a 20 megawatt data center. That's a two gigawatt data center. It's a data center that consumes the entire energy output of a big power plant. And I think this is also going to be very centralized. If you compare, say, the bandwidth of an entire undersea submarine cable. The bandwidth between a couple of a GPU racks next to each other. The GPU racks have double the bandwidth of a literal undersea cable. So there is so much incentive to centralized data centers.

• 29:37Speaker 02

  So I think we're going to see a few very, very large facilities that do all this. Now, the same is true of semiconductors for reference. TSMC's fabs are all very large, very self-contained buildings. But there's so much else has to happen, too. You need way more power. You also need way more redundancy. If you have 10 GPUs, then sure, none of them are going to fail. If you have 10,000 GPUs, then okay, maybe one fails sometimes. And then you reset to a previous checkpoint and you try again. If you have 10 million GPUs or 10 million transformer ASICs, then you're going to have one failing every couple of hours.

• 30:17Speaker 02

  You're going to have to figure out, how do I deal with one of these things crapping out without having to solve the whole thing? No one's solved that problem yet. Or heat. We've been able to go cool data centers in the 200, 300 megawatt range. What happens when the two gigawatts that go into that building are all turned into heat? How does that get pumped out? Is that just dumped into the atmosphere? What happens next? I really think a whole set of industries going to evolve around building these massive, massive AI models. We've only seen the beginning of it.

• 30:50Patrick O'Shaughnessy?

  All that, which sounds so remarkable, that heat, that heat dissipation, the power, the bandwidth, number of chips, all of that, I think, if I understand correctly, is especially needed in training. Each marginal inference is less of a centralized, crazy, power-intensive thing, although collectively, obviously inference will get quite big, too. We haven't really talked about training. And I know you're working on chips for inference to allow you to do orders of magnitude faster and cheaper inference, which will power the apps. But what about training? What is actually going on in training of one of these models for those that don't understand that process?

• 31:29Speaker 02

  Well, so first, I want to go talk about inference for a second, then we'll build into training. Because I believe that inference is going to be run in similarly large, power-hungry data centers, too. And here's the reason why. When we run a transformer model, we have this huge number of parameters. And each one of these parameters is a number. And to use that number, we take in a number from our input, we multiply them together, and we add them to a running total. So every one of those parameters, in the case of GPT-3, 175 billion, is loaded from memory once, and then used in a math operation once. It turns out that loading a thing from memory is way more expensive than doing the math. So how do we solve this problem? Well, we say that these weights are the same across

• 32:14Speaker 02

  one user, or two users, or four users, or eight users. So we batch together a huge number of queries, then we load in that weight once, and we use it 16, 32, 64 times. And that's one of the really interesting things that a transformer ASIC can do. You can have a much, much larger batch, not 64, but 2,500. So we're able to go load that weight in once, pay the expensive price, and then amortize that expensive price over a huge number of users, making inference much, much cheaper. And now, while this sounds good in theory, this does mean that you have to run that model in a place where you can have a huge number of users all kind of grouped together. So I think that means inference will be centralized in most of the same way as training.

• 32:58Speaker 02

  To go to training specifically, why are these machines so power hungry? Well, in inference, you have to run the model forwards. But for training, you have to compute how much each one of those little weights contributed to the final error. Do that, you have to run the model backwards, which takes about double the compute. Additionally, running forwards and backwards requires a different kinds of network primitives, different kinds of connectivity than just running forwards does. So training is a more challenging problem. But the economics that make a transformer-specific inference chip make sense, I think will also apply to training. If you're spending, say, $100 million on an AI model, and it costs $100 million to

• 33:40Speaker 02

  build a new custom ASIC, then no matter how good the custom ASIC is, you're not going to make your money back. But if it costs a billion dollars to train your model, a custom ASIC costs $100 million, and makes you 20% better, then yes, it does make economic sense to build that custom chip. If you're spending not $1 billion, but $10 billion, it is a no-brainer. You have to do it. Now, building that custom training chip is not an easy thing to do. Right now, going from conception to a mass production of a microchip can take four or five years, and we can't wait that long for training our new models.

• 34:21Speaker 02

  So I believe that to support this demand, support these custom ASICs for training that we will eventually need, there has to be a whole redesign of the way semiconductors are designed.

• 34:32Patrick O'Shaughnessy?

  I'd love to understand a little bit about how the cycle time for new chip ideation, design, and then delivery has happened in the past, because from everything you're saying, it sounds like the iteration speed of chips themselves will need to go a lot faster than it has in the past in order for us to have all this great stuff that we've been talking about, and that that is a vector of innovation that's as critical, but certainly like extremely critical alongside everything else we've talked about. So how's that worked in the past? If I'm Nvidia or something, I'm whatever company, designing some new chip. Talk us through like the major stages of chip design, maybe even starting with the thing trying to be solved,

• 35:16Patrick O'Shaughnessy?

  how you arrive at the problem to be solved in the first place. So that's step one and step X is the chips are in a data center somewhere being used. And how long does each of those steps take? Like, I'm sort of a novice on semis. And so I'd love to learn how it's done in the past. Then we could talk about how we might innovate to change that in the future.

• 35:33Speaker 02

  Well, there are a few steps involved in bringing a new chip to market. No matter what year you're building in, first have to say, okay, what's the problem we're trying to solve? And then from there, you'll divide an architecture and a microarchitecture to sort of a more detailed variant of the architecture of how your chip works at a high level to solve this problem. And as part of this architecture work, you're going to get a number of blocks. You'll say, okay, in order for me to do this giant matrix multiplication, I'll need a matrix multiplication block. And that matrix multiplication block will be made up of many sub-blocks that each do a multiply add operation. It'll be made up of a control block that sends a signal to the full matrix block and so on. Then you'll take these blocks,

• 36:17Speaker 02

  number of blocks a chip has varies widely, but it can be as high as hundreds. And you'll give it to an RTL engineer who will write code in a hardware description language, kind of like a programming language, that tells the block how to behave. Then this block is verified. Because it costs so much to manufacture semiconductors, you have to make sure there is not a mistake in that RTL block before you go pull the trigger and tell a TSMC or whoever it is to begin cranking out wafers off their line. So a verification engineer will very carefully write a test bench to check that that block behaves the way that it should. And they'll also check that their test bench actually runs every single line in that verilog file. They call that line coverage.

• 37:02Speaker 02

  And then they'll go a step further. That verilog, that code, will be compiled to a netlist. This is how each group of transistors talks to each other. And then you'll run toggle coverage on this. You make sure every one of those transistors will flip to either state. So that's what they call the front end of the chip. Now you have a netlist that comes out of that, a set of all the transistors and how they connect to each other. But now you had to put on actual silicon wafer and lay them out. That's a hard problem. So you'll give it to a backend engineer who will make sure that while both beyond the logic being correct, there's no issue where, hey, this block takes too long and it's not going to be done by the time this other block has to take its results.

• 37:42Speaker 02

  They call that time enclosure and they'll make sure that on a chip, especially a large chip, process issues, manufacturing problems in the TSMC line can make it so that one part of the chip behaves differently than the other part of the chip. Even things moving at different speeds. So they'll make sure that it works no matter what kind of wafer is made, whether it's fast or slow, what temperature it is. They call this getting all the corners closed and a number of other checks too. There's not going to be some weird radio frequency interference thing and there's not going to be some pool, some vacuum in the chip where other photo resist will flow in. They check a huge number of problems. And eventually after this backend process is all finished,

• 38:24Speaker 02

  they'll give it to a masked house, masked shop. It will take these designs of where your transistor goes and they'll say, all right, they make these transistors, we need to go and have silicon in these parts of the chip, but not these parts of the chip. If you ever heard of screen printing, the process kind of works like that. From this file, this GDS two file, which says where there should be and where there shouldn't be a transistor. They make a photo mask that will shine light on the parts where there are transistor and not on the parts where there aren't. And then those photo masks are sent to a fab that actually makes the chip, who then uses them to project an image onto those silicon wafers after coating them with a chemical called photo resist

• 39:09Speaker 02

  that hardens when exposed to light. So that means that after all this is said and done, you have parts of the wafer that are hardened in just the right pattern for you to put your transistors down on them. So this is a very long process. Real production samples coming off an assembly line for modern semiconductors. And if we want to build the next generation of transformer specific ASICs for training and for inference, this has to shrink. Now, there's been a huge amount of progress to make this faster, especially on older nodes. A Google in particular has driven a lot of this. They have a series of open source efforts to automate parts of this backend process.

• 39:53Speaker 02

  Because really, it's not that hard. It's just checking rules and it should be done by a computer. And if you're building on a node like those from it, if you're building for old technology, old semiconductor tech from like 2005, you can have the backend done automatically in a day instead of nine months by using one of these automated tools. And we haven't gotten to this point with enormous modern scale chips yet. It's possible in theory, and I'm very hopeful that more smart folks will begin working on this problem. And same thing for actually writing the chips, doing this RTL coding. But today, it's all done by hand. But for example, C and Python, we have seen language models themselves begin to help out with it.

• 40:38Speaker 02

  GPT-4 is very good at both those languages, as is GitHub Copilot, because there's so much code in C and in Python available online. So having access to these AI tools can make a developer much faster. For a Verilog in these hardware languages, that's not true yet. While software loves open source, hardware really doesn't. So it's very, very hard to get all this code into your model as training data, because it's all proprietary. And I'm very optimistic that somebody will be able to go to Synopsys and go to all the old vendors and say, hey, your code for CPUs from like 2005, that's probably not very useful to you anymore. Can I use that to train my AI model?

• 41:23Speaker 02

  And if enough companies are convinced to say yes, clearly the open source is, but probably not, maybe just to go train that model, then you can get a language model that was good at Verilog, as good at Verilog as it is at, say, C and Python, and use that to speed up writing some of this RTL development.

• 41:41Patrick O'Shaughnessy?

  What do you think the outside limits are on how fast this could happen? Where you said five, four, five, six years might be the start to finish time now. Where do you hope it is 10 years from now? What do you think the upper limit is of how fast this iteration time could be?

• 41:59Speaker 02

  I really want to go call out a lot of the uh Google led silicon ecosystem. I want to call out Oba Foundry's, the Skywater Foundry for open sourcing their PDKs to help make this process fast. Now, if you're a hobbyist who wants to build a chip on one of these older technologies, you can do it from conception to putting it together, a few transistors, taping it out, weeks, months, even if you're starting from absolute scratch. That is so cool. They're able to do it for pretty cheap because they take all the hobbyist projects, they put them together on one mask set. You make a one photo mask that has all the chips on it. And I'm really optimistic that this can be scaled up. Now, it's definitely not easy.

• 42:44Speaker 02

  These are hobbyist chips on old nodes, have you know, hundreds of thousands of transistors on them. And the amount of effort it takes a computer to decide where to put these things is super linear. And the number of transistors going from a hundred thousand to eighty billion, like you have on the Nvidia GPUs, is more than eighty billion, over a hundred thousand times harder. So instead of being able to do it on one beefy server, you suddenly a huge, huge data center working on this optimization problem to get it working. So I am optimistic there's a future where, just like for a hobbyist building a chip on an old node today,

• 43:24Speaker 02

  we'll be able to get companies building chips on new nodes by spending up huge data centers to do this backend process and training these language models to help them write the frontend. I think the upper limit really is a couple of months instead of the four or five years of today. A lot of work is going to have to happen to bring that down. And it's also a problem where the first chips that are made with some faster process probably aren't going to work. People don't like rocking the boat because of how expensive it is to build a chip only for it to come back dead. And once a few people do it, it'll be the obvious solution, but nobody wants to be first. Why do you want to try? Because it has to be done. Someone's got to do it. Someone's got to do it.

• 44:10Speaker 02

  That's the only way that we get to our ten billion dollar training runs. That's the only way that we get to GPT six. That's the only way that we get these AI-generated books and movies and TV shows that would make the artists of today be impressed. Someone has to do it and might as well be me.

• 44:29Patrick O'Shaughnessy?

  It's interesting to think about in this future world now, the existing landscape of companies and stakeholders, and how they'll start to interact with this technology as it becomes available. So when I went and looked up the top buyers of H100s or something from Nvidia, you see kind of exact ones you would expect. You see, I think, Meta and Microsoft are the two top purchasers. Amazon's a big purchaser. So these hyperscaling, hyper-large technology companies that are either providing cloud services, in the case of Azure or AWS, or they're Meta, training their own models, open-source models, buying all these H100s. Who do you think are the customers in this different world, where data centers are different, ASICs are the predominant chip,

• 45:11Patrick O'Shaughnessy?

  Fortune 500 companies are all trying to find their version of applied generative AI? Do you think that the market map and market structure looks a lot different in terms of like, is it the same buyers buying these chips as buying the H100s?

• 45:26Speaker 02

  Again, there's a great analogy in the world of chip making. Well, the world of chip making used to be pretty cheap. Anyone designing their chips probably had their own little chip fab because it cost a couple million dollars. And as these things became more and more expensive, all but the biggest players became priced out. Now, only two or three firms, TSMC, Samsung, and maybe Intel, are able to make the world's state-of-the-art chips. And I think the same thing will happen for a generative AI and these mega-scale data centers. Today, many companies are trying to explore probing the waters because it is so cheap to do so. But as having a state-of-the-art model becomes a 100-million, billion, $10-billion endeavor,

• 46:11Speaker 02

  it'll be much more common to outsource that to one of the few firms who will have one of these mega-scale data centers. And don't get me wrong, you'll still be able to fine-tune on top of these models these companies build. But they'll be much more like TSMC, the pure-play AI-generating house, than they will be some full-stack integrator. I just think the world cannot support 100 different $10-billion models.

• 46:37Speaker 02

  But I can support three. So I see a world where you have these few enormous leading edge state of the art development houses, possibly at the current hyperscalers, possibly at some new company that specializes on just this kind of thing. And they'll be a tale, much like there are for semiconductors. Not every application needs leading edge chips. For example, the US government and US military care much more about chips being produced in a reliable American factory than the chips being state of the art. So fabs like Global Foundries have accepted they're not going to be as good as TSMC, but there's still a market for them to produce by being American. And I think we'll see a similar thing for these AIs.

• 47:18Speaker 02

  You'll have a few smaller data centers that don't try to compete on being as purely smart as the big megaliths, but they'll find some niche.

• 47:29Patrick O'Shaughnessy?

  Obviously, TSMC is one of the most interesting and intriguing geopolitical assets that we talk a lot about in the news. It's even a political issue. Do you think that these data centers and the control over the specific models becomes like the central political issue? Like there's a presidential campaign where the candidates on the stage are mostly just talking about who has the best models and where are they located and what's the security around them and nationalistic type issues? It's all seems almost like the start of a sci fi novel or something.

• 48:01Speaker 02

  If I said that we were going to have threats of war over a chip fab 10 years ago, that sounded like a sci fi novel too. This has to be the conclusion here. Some countries are going to have these hyperscale data centers. And if you don't have one, it's going to be really hard to catch up. China is famously trying to catch up on semiconductor development. They realize how critical that is for AI, but they did not get on the EUV train when it was leaving the station. They're now on this older immersion technology and it's really holding them back. And while they have produced some great quote unquote seven nanometer chips, I do think they'll continue getting better technology. China is not state of the art for semiconductors today and does not seem like it's on a path to be either.

• 48:47Speaker 02

  And I think for data centers as well, the United States is very lucky to have all the big hyperscalers, very lucky to have the big AI companies too. But we cannot be passive about this. We're in a good state now, but that could change. So I think that export bans for state of the art AI chips are really in the national interest here.

• 49:11Patrick O'Shaughnessy?

  Can you explain the essay, The Bitter Lesson? And then talk about the implications of the upper limit of how good these things could get based just on the available data in the world. These things have been trained on insane amounts of data, but there is a finite amount of data in the world. There's more being created every day. But talk about the bitter lesson and the importance of data and the sort of upper limits of how good these things could get.

• 49:35Speaker 02

  So The Bitter Lesson is an essay put out by Rich Sutton, great pioneer AI in 2019, before the whole transformer creates. Rich looked at the field and saw that what would happen all the time is somebody would take a model that worked and they'd hard code some knowledge into it. For example, somebody would say, hey, here's a great image recognition model, but it gets this one weird case wrong on occasion. I'm going to hard code a hacky fix on it. And inevitably, doing this lack of more fixed strategy works in the short term. You get a little bit better performance, you can go publish your paper, all is well, but it never scales.

• 50:15Speaker 02

  The only techniques that have routinely worked are those that leverage the fact that compute is getting cheaper. For example, I mentioned chess engines before. In the very old days, people would write chess engines by saying, well, a rook is worth five points. A bishop is worth three. I'm going to add a rule to my machine that says, hey, if you can go trade a bishop for a rook, that's a good trade to make. So you should go make that move. And again, that works a little bit, but it doesn't really generalize. So the strategy that eventually worked for chess that enabled Deep Blue to become the world chess champion was not encoding human knowledge, not hard coding rules, but by just leveraging a massive amount of computation.

• 51:00Speaker 02

  Deep Blue did search, much like this tree search we were talking about earlier. It would test hundreds of thousands of positions, find the one that it liked best, and then play moves to get there. And the same thing is happening for artificial intelligence. In the very old days, the image recognition, people would say, hey, there's a block of white pixels in this kind of shape. It might be a bird, it might be a lake ball. They had these big trees of fixes for it. And that didn't really work, didn't scale up. But the thing that did scale up is making neural networks very big and feeding them huge amounts of data. So the lesson of the bitter lesson is that the only techniques that will really advance the field of artificial intelligence

• 51:43Speaker 02

  are those that are able to leverage the cost of compute getting cheaper, that are able to leverage these enormous data centers that must be built. And you mentioned data as well. A key part of the learning algorithms is that data. And there's only so much data available in the world. Some people have raised the alarm that we're about to run out. The models aren't going to be able to go and get the next trillion tokens. It doesn't exist. And yes, I do think that'll be a problem eventually. But we haven't even begun to tap into the most valuable source of data, the video. I don't know about you, Patrick, but when I learned to walk, I learned to pick up objects. I didn't do it from a book.

• 52:23Speaker 02

  I did it by looking with my own eyes and seeing, hey, does this behave the way that I think it does? And we've begun to kind of dip our toes to it. People have built vision transformer models. And usually what they do is they train a model on text. And then at the very end, they add in the image part, just as a little side feature. And I think that's wrong. I think transformers tomorrow will be trained from day one with enormous amounts of video data. So that solves the problem in the medium term. But eventually, these models will get hundred times bigger, that require a hundred times more data, they will exhaust all that is available right now. But I think that self-play

• 53:04Speaker 02

  and generating your own training data is the final frontier here. One good analogy is the machines that play Go. They tried training AIs on human games to make them better than humans at the game of Go, and it worked. We ran out of those games very fast. So what Google did with the AlphaGo is they made the model play against itself and learn from those games. And the same trick has begun to show some promise in natural language processing, where you have an AI that writes out some output, then you ask it to analyze that output and say, was this good, was this bad? As a human, you can do that, you can get better. That's why I revise the things that I write. And I think that technique of the models generating

• 53:48Speaker 02

  their own training data will allow us to scale even once we run out of all the images and video available on the web.

• 53:56Patrick O'Shaughnessy?

  When we get to that stage, though, what's the objective function? In Go and chess, we know what winning is, we know what good and bad is, we can define it. I suppose we could define the rules of physics or something in a open-ended play yourself game that these crazy models might do it themselves. These are goal-seeking things. How do we define good objective functions in that case?

• 54:17Speaker 02

  Well, one of the beauties of language models, I think, is that they learn these concepts themselves. You can test this. You can go ask GPT-4, here's a scenario, is the human behaving ethically in this scenario? So the AI knows what is good, what is ethical, and it got the understanding by reading all the text we've generated as a species. So when we ask it to go say, hey, pick the response that is more ethical, pick the response that's better, it is using that concept that developed from reading all there is to read.

• 54:49Patrick O'Shaughnessy?

  Can you walk me through the most important big companies of today? And just riff on each a little bit, like there's the obvious ones, there's the hyperscalers, there's OpenAI, there's Anthropic, there's Nvidia, there's all of these different key players. I'd love you to pick the ones that you think are most important, really just through the lens of your own strategic thinking, like you're building a business. There's been enormous scale benefits in this last cycle of technology companies. It's staggering how big the big technology companies are. How do you think strategically about the role that they'll play? How, as a young upstart business, you might position yourself strategically or counter position yourself against these incumbents? Why won't those same five companies just win everything?

• 55:32Patrick O'Shaughnessy?

  Like this whole strategy game is really, really interesting to me. And I'm curious how you think about it, or if you have certain lenses to approach it.

• 55:40Speaker 02

  I know that it looks from afar, like Microsoft and Google and Amazon are dominating everything. And then you dominate, but at one layer of this enormously complex supply chain. Let's think about it from the top. At the very top here, we have the state of the AI models being trained by folks like OpenAI and Anthropic. These folks have enormously good talent. I think that's the big moat, but they don't really run the compute themselves. They get it from one of the hyperscalers. So we have the models at the top. Below this, we have the hyperscale tech giants. Those are the ones people are scared of. Google, Microsoft and Amazon are the big three in this domain.

• 56:24Speaker 02

  Buying those chips, racking and stacking them, supplying data center with power, and then selling that to these companies on top. Where do those companies get their chips from? Right now, it is almost entirely Nvidia, but I think that's going to change. AMD is coming out with their new MI300X next year. I think it's going to be a viable competitor to the H100. And companies like us are trying to replace this idea of general purpose AI chips entirely. And they'll specialize ones just for the transformer. So I think that us, along with some of the general purpose guys, become this layer underneath it. But us, Nvidia and AMD cannot make the chips ourselves. We have to go to a fat house.

• 57:08Speaker 02

  TSMC is the one of choice. But the chip is just one piece here. On a chip like the MS one hundred or on the H one hundred, the memory actually costs more than the silicon itself by more than a factor or two. Now, these memory modules are made by only two or three companies as well. Samsung, SK Hynix and maybe Micron. Two of those are in South Korea and Micron's American. So these people are other really, really critical folks in that supply chain. And folks talk all the time about Nvidia, TSMC, ASML relationship. They don't talk enough about Nvidia to Samsung, Nvidia to SK Hynix relationship.

• 57:51Speaker 02

  Because these memory modules are so critical for building the state of the art GPUs. But then these companies themselves, where do they get their machines from? And now we get to a little bit of the diminishing returns. ASML is a much smaller company than TSMC. And there are so many others that help make these chips as well. But I think that the supply chain of multiple provider on top, then paper school data center below that. Then chip designer below that and chip fab below this. This is the fundamental stack of the future. And now what I did not mention here is the end application. What does the user see? I don't think this will be built by OpenAI and by Anthropic and by these model companies.

• 58:36Speaker 02

  It'll be folks building on top here, this fifth level of a stack. And some of those folks will be incumbents. Big established firms that want to build some new AI product, a lot of it will be startups who say, hey, chatbots are boring. I have this new interface, speech to speech. I have this new interface, enormous amounts of document injection, query. I have this new interface, AI-powered search of the whole internet. So I think there's the most opportunity. Well, I think there's opportunity at all these levels. Building new AI products, innovating AI models, building new kinds of data centers that are designed to operate these crazy high scales. Building specialized chips and building them fast for running these models

• 59:20Speaker 02

  and fabricating those chips in a way that is much faster than today's processes.

• 59:25Patrick O'Shaughnessy?

  Given that the application developers at the top of the stack have kind of an open green field, which is really cool. I'm sure we'll see amazing things. The way they built competitive advantage and succeed seems kind of clear, just like it has been in the past. You're operating at a much deeper part of the stack. How do you think about building a better product, but also doing so in a way that three years from now Nvidia doesn't just get one of them and look at it and say, okay, well, we're the scale player here. We sort of have the right to win in ASICs, just like we did in GPUs. How do you think about that strategically? And I know the world needs all these things and it's a big world and there's a big and growing pie. I don't mean to suggest anything different, but how do you think about competing at a deeper part of the stack where there's more supply chain, more interdependency, huge giants that move fast and so on?

• 1:00:11Speaker 02

  Let's be clear, Nvidia is a fantastic company. And I believe they will eventually build specialized transformer chips, much like we're doing. It just makes sense. But right now they're stuck in the innovator's dilemma. The H100 is selling like hotcakes and has an enormous margin on it. And if they were to build, not even a transformer specific ASIC, but say H100 for inference or H100 with a little bit of the graphics stuff ripped out. That would reduce the margin they have on the H100 and would all be bad for them entirely. They can't innovate or the advantage goes away. And this is especially true for specialized chips. One of the biggest modes for Nvidia is CUDA, their software stack that is very, very flexible.

• 1:00:56Speaker 02

  It's able to run graphics, able to run AI, crypto. That is a great mode in the world where general purpose chips, general purpose AI chips are dominant. But specialized chips like ours are much easier to program.

• 1:01:11Speaker 02

  In a world where you burn the transformer architecture into the silicon, you don't need CUDA in the same way. If the world goes to specialized chips, Nvidia kind of loses that moat. So while I do believe they will compete, they're not going to be first. The only people who can really innovate are startups like ourselves. So I see a world where we get to market first and Nvidia gets to market a year later or so. Now, what happens in that year? I think, again, going to the crypto analogy makes a lot of sense. A lot of companies, thirty, I think, tried to build Bitcoin mining ASICs. They're much simpler than GPUs. You can tape them out much faster. But the companies that got to the space first ended up winning. There it's now this duopoly between MicroBT and Bitmain.

• 1:01:57Speaker 02

  Bitmain, when they were testing the waters for IPO a couple of years back, was valued at more than fifty billion dollars. So because when specialized chip move into a space, general purpose devices become no longer competitive. The person who gets there first is almost guaranteed to win. I think that to go back to us, this means we have to move as fast as possible. Back when we began this company last year, before transformers were big, before this was obvious, then yes, we could afford to be a little more relaxed. But today, when people are saying, why is there no transformer ASIC? We must move as fast as possible, ship on time and ship silicon that works for us to win.

• 1:02:39Patrick O'Shaughnessy?

  You mentioned the importance of CUDA for Nvidia. We haven't talked about the software that you have to build at the same time as you're building hardware. What role does that software play for a company like yours? And the way that the chip itself can be programmed or tapped or accessed by other developers? Say a bit about those two parts of this story.

• 1:02:59Speaker 02

  Let's get to the question of why did other AI companies fail? People have tried to build general purpose AI chips in the past and have generally lost out to GPUs. Graphcore, Sabanova, SeriBris and Grok have had some modest success, have not taken over the world the way that folks thought they would. And I think this is two reasons. First, the software. And second, the chips generally weren't better. We'll do software first. It is extremely hard to do what Nvidia does. Gouda has been developed over, I think it came out in 2007, more than 15 years. And because it is so old, it has become stable, reliable, the workhorse people can depend upon. A startup trying to get into the space won't have that.

• 1:03:45Speaker 02

  They'll be putting out something a little bit more raw, that's much harder to use. And if you look at, say, the ease of compiling code for Nvidia versus even say AMD, it's no contest. Gouda is great because it is production grade. And the only way to solve this problem, besides being old, I think, is to be less ambitious. If you're doing a simpler thing, you can get to real production grade quality much faster. And luckily for us and for specialized chips, because there is so much less programmability on the thing, because you're only taking transformers to transformer compiled code, the software stack is much, much easier. Now, that's not to say I want to be complacent about it.

• 1:04:29Speaker 02

  We already have a prototype of this running internally, running with our chip and simulation. And because of this, we can begin iterating on that software, getting it stable, reliable, production grade, so that when the chips come back, the software works the way that you expect it to. But the second piece of why these companies didn't succeed is that they weren't better. If you look at, say, the amount of memory bandwidth that they have versus the GPUs they compute it against, it's the same, if not worse. If you look at the amount of flops compute they have, again, it's the same, if not worse. And that, I think, is a much bigger problem than just the software itself.

• 1:05:10Patrick O'Shaughnessy?

  If you think about the performance improvement that you want to drive, you just said, look, these things didn't work because they weren't better. How much better do you want to be? Is it five times, 10 times?

• 1:05:21Speaker 02

  It's gotta be more than 10 times. I don't want to share exact numbers right now. We gotta have some secret sauce. But because we're able to specialize, you can get a huge amount more compute on these chips. And let me just give a concrete example here. It takes about 10,000 transistors to build a fuse multiply add unit. That's the building block of any matrix multiplier. And Nvidia has not that many on their chip. Only about 4% of the chip is the matrix multipliers. Because to feed them requires so much more circuitry. That's the cost of being so flexible. Whereas us, we will have more than an order of magnitude more of these multipliers.

• 1:06:04Speaker 02

  An order of magnitude more raw flops. And of course, turning that into a useful throughput. That is still not trivial. You need a very good memory bandwidth there as well. But the reason I can confidently say that we will not fall into this trap and that we will be better. Is that I already know the metrics on our chip. I already have PPA figures. We will just have so much more raw compute. Substantially more than an order of magnitude. That's not just for compute, but also for latency. People talk a big game about, hey, if the cost of compute goes down, you can do so much more stuff. And I do think that's true. But it's not what gets me excited. With chips that have 20 times the better latencies than the state of the art GPUs.

• 1:06:48Speaker 02

  Because they're able to run a huge prompt in total parallel. That lets you build products you could not otherwise build. Let's you build this real speech to speech machine. Let's you build a machine that ingests the whole internet in a second instead of hours. And that, that is what gets me excited.

• 1:07:08Patrick O'Shaughnessy?

  One of the things that's so clear to me is the role of strategic leadership in making things like this happen. And by that, I mean usually in a person, a single person. Obviously, like on the software side, Sam Altman seems to be a or the person that's really driven a lot of this forward. He clearly early recognized an opportunity and galvanized talent and a team around it. And has managed them to build something really spectacular. I sort of think of you like the hardware person potentially that could do this. How do you think about strategic leadership? How do you lead a technical staff? I don't know how technical Sam is, but my sense is that he is more of a traditional leader, not in the deep, deep weeds. How would you characterize yourself?

• 1:07:52Patrick O'Shaughnessy?

  And what do you think are the key components of a good strategic leader? In the hardware side of this equation? Given that we're already familiar with some of the leaders in the software side?

• 1:08:04Speaker 02

  Let me tell you a bit of the story of how we got here. And then I'll bring the into what I think a good leader has to do. When I was on my gap year working for Octomel, living as a digital nomad versus off in his dorm room. We had this idea for, hey, let's build a transformer specialized ASIC. I bet we get a huge amount of performance improvement. We went to a big industry vet who told us that, nope, not gonna work. But also, he said that despite being a veteran, an industry vet, he wasn't the guy to know for sure. But he knew just the guy to tell us why our chip would not work. Mark Ross, his resident doubter. Mark Ross is a seriously legit guy, kind of person who makes you a little bit scared.

• 1:08:48Speaker 02

  He was a CTO of Cypress Semiconductor for a time, which sold for $9 billion in 2019. And Mark has shipped, I think, five chips that have done more than a billion in revenue. So we go to Mark, we show him our tech. And he says, well, it looks like this isn't gonna work, but it's too early to tell. You guys should go build a functional simulation. You guys should go write a white paper and then come back to me. And we can talk about, hey, here's the problem that you're gonna hit. Here's why this isn't gonna work. And you guys are gonna learn a lot by doing it. And then maybe from that, we can figure out how to pivot and build a chip that is viable. So over a lot of long nights, Chris and I do just that. We write our technical white paper, we do the original architecture for the chip.

• 1:09:32Speaker 02

  We build our functional simulation and we go back to Mark. And Mark says, shit, this works. But you guys are college students, he says, this is semiconductors. Just to get started, you're gonna need $2 to $3 million and a hell of a lot more after that. This isn't data cats, you guys have never built a chip before. And then we raised five and a half million, went back to Mark and asked if he knew anybody to be the chip architect. And after a series of conversations, he used to be a chip architect and we talked him out of retirement. So Mark Ross is now our chief hardware architect. And from there, kind of the rest of the dominoes fell. We got Ajat, our VP of Engineering. Fantastic chip guy, Intel VP before this, who was known for building these

• 1:10:17Speaker 02

  very large chips on advanced nodes with teams, just like a dozen people. That is not how that normally happens. One of the few good men there who can still build the chip on a budget. And we got a Saptadeep, the co-founder, Oradine. We got Reynolds, the first chip guy at Cruise, and dozens of other vets from Google, Microsoft, Amazon, you name it. What does a good leader do? Kind of tie this all back. It looks from afar like Sam Altman himself is pulling all the strings at OpenAI and masterminding the whole thing. But I think it's the folks under him who are doing the magic, the folk under them who are making that into a real product. The role of a good leader, to set the vision, get the right people in those chairs, and not running out of money. That's it.

• 1:11:01Patrick O'Shaughnessy?

  I've heard stories, though, about you in meetings where you go to the one inch or one centimeter level on some very technical part of what's being designed. And I'm curious how much you'll miss that if you give it up and you just do those other things. And/or whether or not there might be a world where you being in the technical details, I think about like an Elon or something that apparently can walk the SpaceX floor, the factory floor, and know every frickin' detail somehow. What do you think is the prospect of being a more technical, hands-on leader for you?

• 1:11:34Speaker 02

  I think there's a lot of value in a leader being technical. Understanding your team's problems, understanding why they're doing the things that they're doing is a requirement for bringing in the right people. There aren't that many levers you can push at the top. It's very coarse things. To figure out the right guy to bring in, you have to understand that problem. And that's what being technical gets you. This is why Greg Brockman at OpenAI, despite being the CTO, you still see that guy coding. He can't understand how far along the model is unless he himself has seen the internals. Even though I love the folks who work here and I trust them completely, I can't understand how far along is design verification? How far along is this block on our chip?

• 1:12:19Speaker 02

  Because I'm talking to that guy and I understand why he's doing what he's doing. And also, to be quite frank, it's fun.

• 1:12:27Patrick O'Shaughnessy?

  Any closing thoughts about things in this world that would be most surprising to investors, to end customers out there, to CEOs of other businesses that aren't technology or aren't AI-related? You've gotten so deep in the weeds on what's going on and trying to drive a certain future. Is there anything that would surprise people most that we haven't talked about yet?

• 1:12:51Speaker 02

  Let's take it back to this idea of transformer is becoming the way neural networks are, where it's not going to be replaced, it's going to be a building block. You're going to build, say, retrieval augmented generation on top of that to make the model smarter. If you read Microsoft paper where they have agents living in a simulation, you're going to build memories on top of these transformers to let them remember things. So a real key storage cache. I think that folks have talked and talked and talked about how the transformer is going away. Much more productive line of reasoning is how do I build on top of this? And when I look out at what's happening in this landscape, it is the folks who are building on top of it, who are building the most successful, coolest products,

• 1:13:34Speaker 02

  AI games based on having this memory, real conversational things, speech to speech, although with a higher latency than I would like today, or a primitive AI lawyers of the future. So again, the transformer is not just the next fad, this is a building block.

• 1:13:53Patrick O'Shaughnessy?

  What makes you most confident in that? When I asked around, I heard lots of people that agreed, but why that architecture, which obviously has led to all the stuff we're talking about. So obviously it's incredibly powerful, but why doesn't it tap out in two years, three years, four years, five years? Why might there not be a different architecture that just totally takes over?

• 1:14:14Speaker 02

  If the world was fair, there could be, but the world is not fair. The transformer, by virtue of being the dominant architecture, is suddenly the thing that has way, way more humans thinking about trying to improve, trying to optimize, not just people, hardware too. Nvidia is building a little bit of hardware support for the transformers to optimize certain kinds of memory loads into their next GPUs. That is going to preferentially favor the transformer over any replacement that comes out. For a software, Nvidia has three very highly optimized transformer libraries. That mean if you want to run a transformer model, it'll be super easy. If you want to run, say, some other kind of thing, state space diffusion,

• 1:14:57Speaker 02

  you have to make the model so much better that it outweighs the factor of two, three performance improvement you get by optimizing it so heavily. So for a new architecture to go and beat the transformer, it has to be so incredibly good that it will outweigh the hardware disadvantage it will have because transformers are favored. It will outweigh software advantage it has because transformer libraries are so common. It will outweigh the end user application because end users are used to working with these transformer models. That is just such a heavy lift. And again, to go back to this neural networks analogy, random forests or a S V Ns could have been the dominant AI paradigm, I think.

• 1:15:37Speaker 02

  But now to go cash that up to the state of neural networks is just totally infeasible.

• 1:15:43Patrick O'Shaughnessy?

  One thing we haven't talked about is in the world where there's a G P T X and maybe a llama Y and an anthropic Z. What about building models that are specialists that rely on access to a very proprietary data store? So there's companies out there or people out there that have closed access to really valuable data that could make these models better. What happens there? Do they train their own model? Do they just somehow add their data to one of the big existing models? What do you think about a proliferation of models beyond the top three, four of them?

• 1:16:23Speaker 02

  I think this data is very, very valuable, but it can be learned later. And the reason I'm so confident that's okay to learn this stuff later because humans do it all the time. If you go to work for, say, Bloomberg or a company that has valuable financial data, you come in there having a good understanding of the world around you by not having the Bloomberg secrets and you're able to get caught up pretty fast. I think the same thing will happen for these AI models. OpenAI will train G P T X. It'll cost $10 billion. And then on top of that, Bloomberg will say, hey, I'm going to fine tune that with my Bloomberg proprietary data. I'm going to pay OpenAI 10 million for this, and I'm going to use that fine tuned version.

• 1:17:07Speaker 02

  Other companies say semiconductor company that wants to build a semiconductor based AI for coding chips or some investment bank, you name it, they will fine tune on top. And the second piece of why I believe this is going to be the future is that there's not a whole lot of advantage training your own thing from scratch. Remember how we talked earlier about the pre-training phase? Where you do things that are tangentially related, but easy to grade. Then you do the RLHF on top of that to really make those capabilities come out. Pre-training phase is way more expensive than the RLHF stuff on top. The pre-training is going to look the same regardless of whether you're an AI for finance, an AI for law, AI for therapy.

• 1:17:49Speaker 02

  There's no sense in a large company to make their own thing from scratch. Doing the expensive thing again, when it's going to look the same, you just fine tune.

• 1:17:58Patrick O'Shaughnessy?

  Is there anything else to talk about in hardware, edge computing, or other things in the hardware world that we haven't talked about that you think are going to be important beyond just the chips and training and inference?

• 1:18:09Speaker 02

  Is there a market for smaller models? I think absolutely. I don't think there's a market for medium sized models, though. I think there's a lot of demand for a thing that's super, super low latency. Going to the server and then going back is so slow that it causes problems. Much like, for example, speech recognition today. It used to be the case that when you said hello, Google, and then asked it a question, it would send the audio to the data center, it would transcribe it, and it would come back. And the latency there was annoying. So Google built a speech recognition model on the device for better latency. Now, that model is very small. And I do think we're going to see other very small transformers for tasks like setting the next autocomplete word, doing a little bit of drafting to help with your texts.

• 1:18:54Speaker 02

  These things are not going to be the crazy smart agents we're going to have in data centers. However, I don't think there's a future for medium sized models, things in 100 billion parameter range. That's just too big to run on a device. And if you're going to run into data center, you might as well use a smarter model. And you might say, well, 100 billion is going to be cheaper. But I think there are enormous economies of scale here because the cost of loading in those weights is so much higher than the cost of doing marginal extra computation. It is really easy to kind of get added to that batch. So I think we're going to see a promising future for Edge and a promising future for the biggest of the big and these hyperscale data centers and nothing in between.

• 1:19:36Patrick O'Shaughnessy?

  Gavin, I look forward to, I don't know, hopefully decades of learning from you about all the stuff that's happening in this world. It does seem to just be the most, certainly for me, the most interesting, entertaining, impactful, incredible explosion of technology that I've come across. And it's so cool that you're going to be driving the hardware side forward as folks like Sam and others drive the software side forward. Always ask the same traditional closing question at the end of my interviews. What is the kindest thing that anyone's ever done for you?

• 1:20:02Speaker 02

  The kindest thing anyone's ever done for me? This guy Gunar Mine, when I was a kid in middle and high school, I was very interested in engineering. But yeah, I'm young, it's hard to go turn that into impactful results. Gunar dropped out of college in Germany, went to work for Microsoft for a couple of decades. And then decided that instead of working for Microsoft, he wanted to go teach high school, which is quite a pivot. He went and taught programming classes at my high school, which I token they were good. But really, he stayed very, very late to run the robotics program to run Xanier stuff.

• 1:20:40Speaker 02

  Gunar thought that a bunch of high schoolers could build a nuclear fuser to go turn a deuterium into a helium, real nuclear fusion. It would generate power. These things haven't been built before, but the thought that a bunch of high schoolers could do it was a little bit crazy. Gunar Mine decided that my school needed an electron microscope. And we didn't have the money to buy one, but he saw a $1,000 broken Jill 87 machine that was a user for parks being sold in Everett. He, some of my friends and myself, drove over in a U-Haul, picked up the machine, which weighed several thousand pounds, drove it back to the school. I spent the next year and a half fixing it. We got it working, actually. It was a very old microscope.

• 1:21:25Speaker 02

  It had a Polaroid camera, if you can believe that. It would flash pixels one at a time as it scanned across the image. Because each pixel would be exposed, it slowly would be turned into the full image. And we ripped that out, placed it with an Arduino. We tried to go into, rip out the high voltage box, replace it with a newer one, because it kind of broke sometimes. We got that working. We never figured out how to keep it from leaking. But that's what we had. The whole microscope would take about 30 minutes to get running. You'd have to go to the room, turn the machine on, wait for the vacuum to be drawn, turn the fusion pump on, wait for that to be pumped out. It had to be done totally manually. Because the automated box had been broken. Gunar would go sit there and supervise us while we messed around with that thing

• 1:22:08Speaker 02

  until like 9:00 PM at night, almost every night. And that got me interested in engineering, got me to Harvard in some ways, and got me to where I am right now.

• 1:22:20Patrick O'Shaughnessy?

  What an awesome story. So incredibly unique. What a cool guy. So many stories like this of someone taking an interest in teaching and sharing the world with someone so young. Amazing closing story. Gavin, thank you so much for your time.

• 1:22:34Speaker 02

  My pleasure.

• 1:22:36Patrick O'Shaughnessy?

  If you enjoyed this episode, check out joincolossus.com. There you'll find every episode of this podcast complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus Weekly, where we condense episodes to the big ideas, quotations, and more, as well as share the best content we find on the internet every week.