E29: The AI Chip Revolution with Andrew Feldman of Cerebras

Nathan Labenz sits down with Andrew Feldman, CEO and Co-Founder of Cerebras Systems, a company building a new class of computer system for accelerating AI and changing the future of work. Cerebras Systems is the creator of the world’s largest chip, at 2.6 trillion transistors. In this episode, they discuss the founding story of Cerebras, the experience of creating the world’s largest chip, and the process that goes into chip design and manufacturing for an AI-focused chip. This episode is the first part of our hardware exploration series focused on the people building at the forefront of hardware applications in AI. LINKS: Cerebras: https://www.cerebras.net/ Book: The Chip War by Chris Miller We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. PODCAST RECOMMENDATION: The AI Breakdown: https://pod.link/1680633614 As anyone in AI knows, the pace of progress of new releases is relentless. The AI Breakdown is a daily podcast (10-20min long) that helps us ensure we don't miss anything important by curating news and analysis. TIMESTAMPS (00:00) Preview (04:27) Andrew’s story of creating the world’s largest chip and Cerebras (07:19) What is a chip? (08:14) The diversity of chips and what they can accomplish (09:47) What is it like to design a 2.5 trillion transistor chip? (12:41) The founding story of Cerebras and building the team (14:20) Sponsor: Omneky (23:00) What was the hardest part about building the company? (26:11) What happens after designing the chip’s blueprint? (27:29) The tradeoffs needed in chipmaking (34:08) The comparison between chips and neural networks (38:31) The generalization vs specialization of a chip (40:11) Sparse compute vs dense compute (43:55) Ghost in the machine (46:54) Supply chain challenges of the Cerebras chip (54:59) The future for chips (58:19) Building chip clusters (58:57) The Cerebras business model (01:00:41) Building a chip cluster vs using a Cerebras chip (01:02:57) Giant chips on the edge (01:05:32) What is the edge? (01:08:04) Andrew’s favorite AI products (01:10:08) Would Andrew get a Neuralink implant? (01:14:16) Consciousness and chips (01:17:50) AI hopes and fears TWITTER: @CogRev_Podcast @andrewdfeldman (Andrew) @labenz (Nathan) @eriktorenberg (Erik) SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected]. Music Credit: MusicLM More show notes and reading material released in our Substack: https://cognitiverevolution.substack.com

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This wasn't about making money. This wasn't about moving the ball forward a little bit. We wanted to move an industry forward. And we wanted to put our shoulders to it and see if we could transform an industry. We tried to see if we could come up with a solution that was creative. And that would be substantially better, not a little bit, not 10 times better, but 50 or 100 times better.

• 0:20Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas. And together, we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. In the era of deep learning and scaling laws, it's often said that the three fundamental ingredients to AI are algorithms, data and compute. We've spent a lot of time on this show exploring different aspects of algorithms. And the centrality of data has run through just about every conversation that we've had.

• 1:05Nathan Labenz?

  But today, we take our first step into the world of compute. My guest, Andrew Feldman, CEO of Cerebras Systems, makers of the world's largest ever chip and inductee into the Computer History Museum, is the perfect guide to help understand the landscape at both a metaphorical and also a very practical level. For the last few years and still today, AI computing has been performed on GPUs, or graphics processing units, simply because GPUs have been the best way to parallelize computation at scale.

• 1:41Nathan Labenz?

  Nvidia, the dominant maker of GPUs, is now a $750 billion company. That's up more than double so far, just in 2023. Yet, GPUs were not specifically designed for AI workloads. And they require an extremely complicated software stack to wire them together and to shuffle all the data around. For many of the large-scale AI training projects conducted over the last few years, making the GPU clusters work was a significant part of the overall effort.

• 2:13Nathan Labenz?

  GPT-4 training is rumored to have cost some hundred million dollars or more, and next-gen systems could stretch into the billions. This makes computing infrastructure a focus of geopolitical strategy and industrial policy as nations position themselves for the AI era, and also a possible supply chain choke point that could lend itself to regulatory control. Thus, to have a sense of where AI is going, it's critical to understand the fundamentals of compute. Andrew and his colleagues at Cerebras Systems, all veterans of the chip industry, recognized the opportunity presented by the rise of deep learning and the "compute-intensive workloads" that AI systems would require well before most others.

• 2:58Nathan Labenz?

  And they set out to build a chip from the ground up with AI systems in mind. What they came up with and what they are selling today is not just the world's biggest chip, but a full multimillion-dollar computer that eliminates much of the complexity associated with AI workloads simply by scaling the hardware itself. Of course, as you'll hear, building the world's largest-ever chip, while conceptually simple at a high level, is extremely complicated in theory and in engineering. Of course, Cerebras is not alone in designing chips specifically for AI. Nvidia is getting into the game along with Google, Meta, and Microsoft. But in a market that's growing as fast as the AI market currently is,

• 3:41Nathan Labenz?

  there will be plenty of space for many companies to flourish, and I certainly expect that Cerebras Systems will. Now, I hope you enjoy the introduction to compute for AI that is this fascinating conversation with Andrew Feldman. Andrew Feldman, welcome to The Cognitive Revolution.

• 3:59Speaker 00

  Well, thank you, Nathan. Thank you for having me.

• 4:02Nathan Labenz?

  I am super excited about this conversation because you have built, along with your co-founders, and I know you're always very gracious about giving credit to your teammates, but really a remarkable company that I didn't know a lot about before we connected, but which has built an awful lot of stuff that kind of goes from a very foundational layer increasingly all the way up toward a very productized layer. And you're in hardware. So I think our audience doesn't know probably as much about hardware as we do about AI and especially all the applications that we're seeing right now. So I'm excited to get into it with you. I've been trying to study up myself to prepare,

• 4:44Nathan Labenz?

  but probably going to ask some rookie questions and hoping that you can help guide us through kind of an understanding of this hardware world, but also the very unique path and approach that you have taken on it.

• 4:57Speaker 00

  Happy to do it.

• 4:59Nathan Labenz?

  The thing that you guys are most known for and that you're in the Computing Hall of Fame for is the world's biggest ever chip, 2.5 trillion transistors, if I understand correctly. So let's just start off with, what inspired you to go try to set a world record with the biggest chip ever? And what do you have now that is so special?

• 5:22Speaker 00

  Well, thanks, Nathan. That's a really good question. I think chips are a little bit like cars. Some are good for taking kids to soccer practice in the grocery store. Others are good for moving lumber and bricks. Still others are really fun to drive on Saturday. We wanted to build a chip that was optimized for one thing, and that was for AI work. And AI work presents a very specific set of challenges. It has a huge amount of relatively simple compute and a tremendous amount of communication of moving information around. By building a very big chip, we can keep that information on chip. And communication

• 6:07Speaker 00

  on chip is fast and it's power efficient, order thousands of times faster and thousands of times more energy efficient than if you have to leave the chip boundary, travel across a motherboard, maybe out a switch to another chip. And so by building a very large chip, we were able to do AI compute extraordinarily efficiently and blisteringly fast.

• 6:33Nathan Labenz?

  Cool. Okay, I want to get into all of this, you know, layer by layer. And just for your reference too, I think our audience probably has a pretty good sense that obviously, you know, people are training larger and larger models with, you know, bigger and bigger data. And that just requires, you know, a ton of compute, right? Everybody kind of knows that. I think we also probably have a decent sense that like, we can't use CPUs for that because we need some element of parallelization. And then I think a lot of people are still kind of at the moment where they're like, so that's where GPUs come in and it gets complicated from there. So again, just so you can kind of help calibrate yourself and help fill in the gaps for us.

• 7:15Nathan Labenz?

  I just kind of want to start, again, layer up the chain. What is a chip? You know, this is something that's so basic, but tell us what a chip is in the first place.

• 7:26Speaker 00

  A chip is a, an electric circuit that is put on a piece of silicon. It's just that simple. It's a transistor that is embedded into a bit of silicon. And as they get more complicated, there are more and more of these circuits, these transistors, that are in chunks of silicon. And these are the, the things that, that power computers, that power the displays, power sort of most of our digital world is, is one form or another of these circuits that are housed on, on a piece of silicon.

• 8:07Nathan Labenz?

  And so there's like tons of different kinds, right? I mean, the kind of canonical ones would be your CPUs, your GPUs. Now you've got the, you know, world record biggest ever.

• 8:19Nathan Labenz?

  Just give us some other sense too of like the diversity of chips and sort of the things that can be accomplished with a little thing carved on a piece of silicon.

• 8:27Speaker 00

  Sure. I, I think they're very tiny ones that, that are sensors. They're ones that draw sort of milliwatts of power, absurdly little amounts of power. There are ones that control dishwashers. Your, your fancy coffee maker is controlled by a little chip.

• 8:49Speaker 00

  Much of your car now is controlled by one form or another of a chip. Much of what we do every day, our phones, our laptops, our computers, the cloud. All of these are powered by one form or another of a different size, different shape compute. And chips are where compute lives.

• 9:17Nathan Labenz?

  So the first thing you have to do when you're going to set out to build something like this is design it. And this, I think it would be really interesting to contrast kind of how you guys have built your company. Because I, I don't know if vertical integration is quite the right term, but it seems like you do operate at a lot more layers of the value creation stack than most. So first, there's firms that specialize entirely in designing the chip. And you guys do that, but then you do more up the stack as well. So let's start with the design process. What is it like to design a 2.5 trillion transistor chip?

• 9:53Speaker 00

  First, it's really hard. And it takes years and it takes hundreds of millions of capital. It takes a passion for really hard engineering projects. Right? You have to love hard technical challenges. You're absolutely right. We love those challenges and we love being a system company. So we build the chips, we build the motherboards that they live on. We build the enclosures, right? The whole computer, the server. We integrate the power supplies, the cooling. And so what we deliver is a whole computer optimized for AI.

• 10:37Speaker 00

  And sometimes we deliver clusters of these, including in some of the largest supercomputers optimized for AI ever built.

• 10:47Speaker 00

  So we do work all the way up the stack from the compute silicon to the compiler, to the management software, all the way up the stack. And that's a really fun problem. And you asked about the chip you begin usually having done simpler ones in the past. That's sort of the way it works, Nathan, is 20 or 30 years of increasingly complicated chips. One has a collection of friends and colleagues who are leaders in chip design.

• 11:28Speaker 00

  And you get together and you begin thinking about which problem you want to solve. And this is my fifth startup. And each time we've built dedicated chips and print circuit boards and put them in a system and built the system software and solved a hard problem. And so in late 2015, the founders and I got together and we said, which problem do we want to solve? And we got interested in the problems of AI. And we asked ourselves, what does the AI work really want from the underlying machine? What's hard about AI work? And what is the right type of computer to build for it? We came up with the idea

• 12:13Speaker 00

  that we could build a dedicated machine, not repurpose a graphics processing unit, but a dedicated machine where every ounce of its energy, its focus was aimed at this particular problem, not lots of other problems, but this problem. And then every decision we made, how big it was, the shape of the core, how many cores, every decision was made towards the end of optimizing for AI work.

• 12:41Nathan Labenz?

  So you set that vision from the founding in 2015. The models at that time were obviously a lot smaller, right? So you had clearly some foresight there to be skating where, you know, the puck is going to be, so to speak.

• 12:57Speaker 00

  I have no foresight, Nathan, nor any technical vision. All of that's my co-founders.

• 13:03Speaker 00

  I have good ability to pick founding, other founders and co-founders and world-class employees, team members. No, we, we were interested in the problem of AI. It was clearly something on the horizon and sort of, it presented a new challenge to compute. In the same way that cellphone compute presented a new challenge to the compute world. It was different. Right? And when there's a new compute challenge, historically, new players emerged, right? None of the leaders in x86 compute, not Intel, not AMD, were able to capture this new work in the cellphone world.

• 13:49Speaker 00

  Right? All of it went to ARM, 100%. We saw that sort of sea change happening with the, with the rise of AI. We were probably wrong. We probably underestimated that the size and scope of, of what AI would be. But we knew that it was interesting. It presented a new problem to computers and that we could build a, a, a better machine optimized for this particular problem. And that, that was the, the founding impetus for the company.

• 14:20Nathan Labenz?

  So maybe we could kind of build up the story a little bit also in terms of the team and kind of the roles and how you layer out the company over time because I have to imagine that this is your vision, the first thing you're going to do is design, right? And there's, so there's a certain skill set that comes there, but then next you're going to have to go get this thing manufactured and that presumably, you know, brings out a new set of problems and skill sets. So like, who is the first group that you hired to design this thing and like, how long did that take?

• 14:51Speaker 00

  I think, Nathan, most of us who are serial entrepreneurs, we, we work with many of the same people time and time again. All of the founders worked with me at, at, at my last company. Our, our CTO, Gary Lauterbach, was my co-founder at the last company. Michael James, JP Fraker and Sean Lee were all technical leaders at our last company. And, and so you, you generally begin with the best people you've ever worked with. And I'm sure that's the same in your business and in many businesses is that you, you begin with people who you really enjoy working with and who inspire you every day.

• 15:32Speaker 00

  And you, one of the things you do as a CEO is you have lists in your mind of great engineers and smart investors. And some of your time, you're talking to the really smart investors and some of the time you're engaged with extraordinary engineers, some of whom have worked with you in the past, others maybe worked with partners or you had a chance to engage with them and they really impressed you. And so at the beginning, usually the founders get together, they settle on a vision. And often it's a very high level. We wrote two things on the whiteboard. We wanted to work together again. We wanted to do something important. This wasn't about making money. This wasn't about moving the ball forward a little bit.

• 16:16Speaker 00

  We wanted to move an industry forward and we wanted to put our shoulders to it and see if we could transform an industry. And so, once we'd identified, we were meeting regularly and once we'd identified a problem that interested us, we tried to see if we'd come up with a solution that was creative and that would be substantially better, not a little bit, not 10 times better, but 50 or 100 times better. And once we had that idea, we began to socialize it with some of the best people we'd worked with over the years and see if they were excited by it. And of the first 20 or 30 employees, almost everyone had worked with us previously. And that's one of the really fun things about Silicon Valley.

• 17:00Speaker 00

  And, you know, we have people here who I worked with in 96 at our very first company. We have people who we'd worked with 2000, 2003, from 2003 to 2007. Each company has people you note and say, that's an extraordinary person. If we ever need that set of skills again, I'd love to work with them again. They make those around the better. They bring interesting ideas to the table. And so you, once you have an idea and a direction, you begin systematically reaching out to, to, to the best people you know. And while you're doing that, you begin to engage

• 17:41Speaker 00

  with investors. We were fortunate, we made eight pitches and got eight term sheets. And so we were able to fund the company very quickly with world-class investors, the best in the business. We were able to, to bring some super angels to the table, some really wise people to, to act as sort of advisors and, and sort of rabbis to, to, to keep us from making mistakes that were too egregious. And the, the, usually, the first problem is one of architecture. And the problem of architecture is, you know, you, you're at point A and you want to get to Z. How, how do you break this up? What, what, what are

• 18:22Speaker 00

  the pieces? How, how do you attack this large unknown problem? And which parts of that problem are going to be extraordinarily different, difficult? And which parts are going to be sort of solid engineering problems, but don't require vast invention? And once you've broken up the problem into pieces, once you've identified which piece is going to be hardest, we like to begin with the hardest piece. And for us, that was, how, how do you build a chip that is orders of magnitude faster than a graphics processing unit? And how do you do things that graphics processing units are historically bad at?

• 19:06Speaker 00

  When, when all your, when your audience works with GPUs, they often encounter problems with memory and particularly with memory bandwidth. How, how do you resolve those problems? How do you make sure you're not a little better, but you're hundreds of thousands or tens of thousands of times better on these key dimensions that, that, that underpin the AI workload? And so we began working on the problem and you work on the problem every day. You know, the, the technical team is engaged. Uh, I'm out talking to, uh, to other engineers, seeing if they want to talk to us further. Step by step, you build up your team. Uh, each time you bring in a guy you've worked with, you, you ask him or her, who are the best people you've worked with? You know, who, we haven't worked together in three or four years.

• 19:50Speaker 00

  Who are the best engineers you encountered? And then you go talk to them. You say, look, we heard great things about you. Uh, got the people we really trust are, are really impressed with the work you do. Uh, would like to talk to us. And you build a cadre, a nucleus of, of extraordinary people. And engineers like to work with other extraordinary engineers. They're motivating, they're, um, they, they, they get your juices going. And so at this point, you've got a team, maybe 20, 25 people. You've identified the hard parts of the problem and you're systematically attacking it. That's how you get going.

• 20:27Nathan Labenz?

  For context, one thing I did in preparing for this was just read the recent book, The Chip War.

• 20:32Speaker 00

  I was just trading emails with the author.

• 20:35Nathan Labenz?

  Definitely a really good kind of general history primer on chips, you know, why they're important. And one of my big takeaways was just the extreme specialization and the extreme kind of coordination of the global value chain that all kind of comes together to create things. And there's like seemingly multiple, almost single points of failure in terms of like one company, you know, makes the machining without which, you know, nothing else can.

• 21:02Speaker 00

  Only one, right? Just one Dutch company makes the manufacturing tools that all chips use. Yeah.

• 21:09Nathan Labenz?

  What that has been thinking for you is like, when you're going so far outside the bounds of what has kind of been done. And you're like, not doing something that's, you know, on maybe anybody else's, you know, Tik Tok sort of roadmap. Uh, you know, what ended up being the hardest thing? And, you know, to some degree, I mentioned that had to be that like, nobody can help us with some of these things, right? Even the wafer itself, I believe there probably was no wafer that big for you to even start with.

• 21:34Speaker 00

  I don't know if you've ever seen those old maps from the 14 and 1500s. And that, that they have the land they knew. And then they have these areas where they had no idea what was there. And they'd draw a picture of a dragon and under it, they'd write, here be dragons. And we were only interested in sailing there. Right? We have no interest in charting territory that other people have seen and done. We want to be out there at the leading edge. We want to be out there with the dragons. And that's hard, right? There are bad days out there. There are challenges that nobody's ever encountered before.

• 22:19Speaker 00

  People had tried and failed to build big ships. And there was a big literature on why you would fail. And we solved all those problems in about six months for $20 million. But nobody ever got further than that. Nobody failed further up the mountain, right? And I tell people, imagine you're at the base of Everest, having beers before it had been summited. And some climbers just came down and they said, you know, about halfway up, it's brutal. We couldn't do it. You're gonna have a real problem there. For the first time, you guys go up, you get all the way to the top, you come back down and you're having beers with these guys again. And you say, guys, that's not the hard part, right? The hard part was way further up.

• 23:03Speaker 00

  And at every stage, we had to invent things, right? There weren't tools. There weren't manufacturing equipment. There weren't ways to cool. There weren't components. At each stage, when you do pioneering innovation, there's not a catalog where you can look up and say, oh, here are these other components that will fit right in with this. And so you have to be ready for that. And we had some experience doing extraordinary innovation, but it's not easy. But what we found is we were able to inspire our vendors. And they also got excited. I remember one of them said, you know, what you guys are doing is why I became an engineer.

• 23:45Speaker 00

  Right? And that wasn't one of our engineers. That was a vendor's engineer who had work to do to modify their parts. It was inspiring when people do amazing things. So Nathan, at every stage, it was a challenge. There was no, there was no easy bit.

• 24:03Nathan Labenz?

  I want to get into the weeds a little bit more of the hard bits with you. So what I understand is you've reassembled the, got the gang back together for, you know, one more heist. And it's all the best people you've ever worked with. And you, it sounds like first spent six months architecting the chip itself to kind of satisfy yourselves that you could design something that could meet the performance criteria that you set out. So now you've got a software, you know, described essentially, right? Essentially a blueprint for a circuit that is beyond comprehension in terms of its complexity.

• 24:48Nathan Labenz?

  And now, where do you go from there?

• 24:51Speaker 00

  Usually you have a software simulator. So you've, you've built software that you believe will behave the way your, your chip will. You have different types of simulators that have been built. And you begin experimenting to understand what trade offs you can make. And while that's going on, usually some guys begin the early stages of chip design. And they begin thinking about how they would construct the chip, how they would organize these integrated circuits into delivering what you want. And, you know, they begin thinking about the hard problems of if we need a third row of seats, what does that

• 25:35Speaker 00

  mean for the length of the car? Right? If you want a truck bed, what does that mean for the backseat? It means it's really difficult.

• 25:47Nathan Labenz?

  Let me just go a little bit down more to root level again, though. Like if, so what are these trade offs? Like you, you're kind of talking about cores versus memory, you know, how much of like relative inputs?

• 25:58Speaker 00

  What you have as a chip? Architect is an amount of real estate. Right? You, you have a fixed bit of, of real estate. That's the size of the chip you want to make and you need to allocate some of it to compute and some of it to memory and some of it to communication, to the moving of information around the chip. And you have to make decisions. Do you want a big core? Do you want a little, lots of little cores? Do you want memory centralized? Or do you want each, each core, each little compute engine to have its own dedicated memory? And there are pros and cons of, of each and different types of workloads.

• 26:39Speaker 00

  Say database or graphics would lead you to very different decisions. And so while you're doing this, you're, you're, you're examining what, what does AI want? What, what is it? It asks of the underlying compute, what's hard about it? Uh, is it one big compute problem or lots of little compute problems? Do you do the work once and then you're finished? Or do you do lots of little problems, learn a little bit, lots of little problems, move the data, lots of little problem? Each one of those would produce a very different architecture. And that's exactly the way you, you think about this problem.

• 27:20Speaker 00

  And AI has some, some very particular characteristics. The cores don't all work on the same problem. Right? When the reason CPUs aren't, aren't optimized for this work is the CPU has a, a big pile of memory in the middle called a cache. The cores all have access to this. And if you want to put one chunk of data in and have lots of compute engines work on it, that's a great way to do it. That's not the way AI works. AI has lots of little calculations which then move the work. Right? We think about the layers, right? You're doing work in one layer and then you move the work to the next layer. You do the work in the next layer.

• 28:02Speaker 00

  There's a sequentialness about AI. And so we, we thought, what, what is the right mapping of compute, of memory, of communication bandwidth to this particular problem?

• 28:20Speaker 00

  Right? Does AI present big, complex problems? Do you need 64-bit double precision? Right? That's historically been the data format of supercomputers. Turns out you don't need it. So we don't have it. The GPU is also designed to solve all these other problems. So they, they, they have to waste some of their real estate with circuitry for a problem that's not AI. We take that out. And, and so you, you think about which problems you're going to be good at, which problems you're not even going to try to be good at. Right? Focus. And you think about for this class of problems, what is the best way to achieve your goal?

• 29:03Speaker 00

  Small number of big cores, large number of little cores, distributed memories, centralized memory, lots of little I/O, a small number of big pipes or lots of little pipes? Uh, how are you going to get data onto and off of the chip? All, all of these are, are sort of the, the, the questions that, that, that architects ask themselves at the, at the beginning of a problem.

• 29:25Nathan Labenz?

  And so to, you said earlier that the problems that people encounter with GPUs is not enough memory or slowness in, you know, moving things around in and out of memory. You know, you can elaborate on this if you want to, but I think the, you know, the kind of core reason that that happens, especially in like the training process, right? Is you have to kind of keep track of all of these parameters and, you know, the back propagation, uh, you know, gradients at all of these different points. And you can't kind of lose that stuff until you've done, you know, a certain batch of computation, made all the necessary updates and then can kind of save that and, you know, free up some memory. That's kind of my, you know, very poor man's interpretation of the situation.

• 30:09Speaker 00

  It, it's a good poor man interpretation. There, there are two problems that, that haunt GPUs. And the first is we now are working with models that, that have very large parameter counts. And too many parameters to store in their memory.

• 30:27Speaker 00

  So right away, they need to shard your parameters across many GPUs. That's a complicated problem. The second problem is the the actual calculation of a big layer is too big for a GPU. And so they have to take that giant matrix multiply and break it up into little pieces and spread that over many little GPUs. All right. Now, when you do that, to get the answer, you have to wait until the last one has given its part back. Right? That's Amal's law.

• 31:05Speaker 00

  And so you care desperately about how far away in time each of those parts of the main calculation are. So right away, you see the problem. You've you've got these giant networks. You've got lots of little GPUs, maybe hundreds or thousands. You're trying to think about how to place work on each individual one, how to break up your problem, how to break up the number of parameters you have. Those are painful problems. And that's the distributed compute edge of AI. Right? And if you look at these papers, whether it's the Lama paper or the GPT-4 paper, at the back, they usually give credit to dozens of people who are involved in just this part,

• 31:50Speaker 00

  not the AI, but the distributed compute, how to break up the problem and spread it out. You have a big chip, takes minutes. You don't have to worry about that at all. We have in our architecture enough memory for trillions of parameters. We have enough enough cores, 850,000 cores, so that even the largest layer of the largest neural network can fit, so we never have to break things up. And so it vastly simplifies how you place big neural networks onto our computer.

• 32:31Nathan Labenz?

  It kind of sounds like with this giant chip.

• 32:36Nathan Labenz?

  The function is probably a lot more like what people visualize for themselves as going on in a neural network architecture when they sort of envision like there are layers and you know, things are kind of moving from layer to layer. I'm gathering kind of that the parameters kind of can stay in place on the big chip and then the data can kind of move, but the parameters, you know, just get kind of loaded and then you don't have to fuss with that. Whereas on GPUs, like it looks nothing like what you think and instead it's like a total, you know, uh, you know, incomprehensible distributed mess.

• 33:17Speaker 00

  That's right. I think, let's think about sort of the history for a sec. In 2014, 2015, we saw the rise of AI start. Um, we saw GPUs begin to get used. 2015, 2016, we saw that that communication was going to be the hard part of this problem. Our solution was to build a bigger chip so you had to communicate less. Nvidia's solution was to buy Mellanox, to buy a company that specialized in Ethernet and InfiniBand's ability to tie together lots of little chips. Tying together lots of little chips is very complicated and messy. Uh, performance

• 33:57Speaker 00

  is almost always sublinear. And that's the same in almost all our lives is that if you say your job takes you X amount of time, if you add a second person, it doesn't take half X ever, right? If you're lucky, it takes less time, but not half. And if you add 10 or 50 people, it never takes a 50th, right? In fact, frequently it takes more time because you've got to coordinate, you've got to organize, you've got to divide up work, you've got to distribute work. It's the exact same thing in building clusters of computers. That as these problems got so big, we couldn't do them on one GPU. We tried to break them up, spread them over hundreds.

• 34:41Speaker 00

  Now the actual problems of coordination, of breaking up the problem, these are the same problems that management exists for, right? That's the overhead. We're able to sidestep those with a very big chip. You almost had it exactly right. We hold the activations on the chip and we don't move those. The data streams in, the parameters stream in, and the activations stay and we stream the gradients out. And so what we're able to do is move information less frequently. We're able to keep data closer to compute.

• 35:23Speaker 00

  So it takes less time to access it. Every core has its own dedicated memory. Never any contention, right? These are things that we thought about would help the AI work. Part of the architecture.

• 35:44Nathan Labenz?

  And contention, just to contrast, this is like when multiple processes need to access data that may be in the same data access bottleneck?

• 35:53Speaker 00

  Exactly, that's exactly right. Contention is when lots of people are trying to check out a Safeway at the same time, right? There's just a queue. And where's the queue form in a GPU? The queue forms when lots of cores are trying to get to off-chip memory. They have this memory that lives off-chip. And all these workers need data. They're all trying to get off-chip at the same time. And that's memory bandwidth contention right there.

• 36:29Speaker 00

  Right? And that's a fundamental challenge in the design of GPUs.

• 36:38Nathan Labenz?

  So how general is this? I'm struck by the fact that you started the company before Transformers.

• 36:45Speaker 00

  And are still the fastest on earth at Transformers. That's pretty good architecture that the team did. We're really proud of that.

• 36:52Nathan Labenz?

  Yeah, it's awesome. Like, how narrowly tailored is the chip to, if all of a sudden there was a recurrent network breakthrough, would that cause you problems?

• 37:04Speaker 00

  Nathan, you put your finger on, I think, what is one of the fundamental questions of computer architecture. And that is which part of your design to be general and which part to be specialized?

• 37:22Speaker 00

  Right? Now Nvidia, in earlier GPUs, in the V100 and the A100, they put customized circuitry in for 3x3 convolutions.

• 37:34Speaker 00

  And if you were doing computer vision, those were really helpful. But if you were doing language and LP work, they were extremely inefficient.

• 37:47Speaker 00

  Right? That's the price of specialization. You get good at it if you work on that problem. But if another problem comes along, you're less good at it. Where we focused our energy in our design was in the underpinning of AI. And the underpinning of AI is sparse linear algebra. All of the AI problems, whether they're graph neural networks, whether they're computer vision, whether they're big language models, all of them decompose down.

• 38:21Speaker 00

  To linear algebra, and in particular to sparse linear algebra. And so what we optimized our engine for was to be good at that problem. And that turned out to be a really good architectural choice.

• 38:34Nathan Labenz?

  Use this phrase AI compute versus dense compute. I guess AI compute sounds synonymous with sparse if it's contrasted against dense. So help us understand that a little bit better. Like, when am I running something that's dense versus sparse? I know that like a lot of activations, like, I've been seeing papers recently where like, a lot of activations can just be rounded down to zero and you'll be fine. I, I don't know, is that going to hold? First of all, as like more intensive training comes online, those things seem like they'll become less sparse, but.

• 39:06Speaker 00

  First, what sparse means is a sparse matrix has lots of zeros in it. Right? And so you have two matrices, you're going to multiply them together. And one has, or both have a lot of zeros. Now, the GPU goes ahead and multiplies everything together. Now, that's not very smart. Because multiplying by zero takes time, takes power, and produces no new information. You didn't have to do the multiplication to know that anything multiplied by zero is going to be zero. Right? And so doing that is like walking in place in a race to the finish line.

• 39:50Speaker 00

  You make no forward progress, you burn calories, you waste everybody's time. Our machine, because it was designed not for graphics, which is only a dense problem, but our machine was designed only for AI, which is frequently a sparse problem. When it encounters a zero, it doesn't do the multiplication, it skips it. You don't need to do the multiplication because you know the answer. You know the answer before you do it without doing any work. And so as these problems demonstrate sparsity in any number of ways, in their weights, in their activations, in their structure, you can avoid doing useless work.

• 40:32Speaker 00

  Right? And that's a very powerful notion. Right? Because what you want is every ounce of your power and every ounce of your compute pointed at problems that move you towards the goal line of completing your training run or completing your inference and doing your your autoregression inference or your classification if it's something else. And so that's the difference between a dense engine and a sparse engine. Is whether when you encounter a zero, you mindlessly multiply it because that's what you do to everything. Or whether your core is smart enough to say, whoa,

• 41:15Speaker 00

  no need to multiply that. Let's skip that. Move on, move on, move on. And what a whole collection of recent papers have showed is exactly what you said is that sometimes you can round these weights down to zero and lose nothing.

• 41:32Speaker 00

  And when you do that, and you have a sparse engine like ours, you can train much, much faster for fewer compute and using a fraction of the energy. And that's what a whole series of papers we've published have shown. Others have published similar papers. The guys at Neuromagic published a very interesting paper on this and how it relates to inference. The guys at Mnemonic, there are a whole group of people doing extremely interesting research on what we can do to take this absurd amount of compute necessary to train these models and shrink it down without losing accuracy. And that's what you get with sparse.

• 42:17Nathan Labenz?

  So this might be

• 42:20Nathan Labenz?

  too hard or it may just be too gnarly to give us a an intuition for. But what I do find kind of fascinating about this and the parallel between like biology and AI, you know, I don't like to over analogize, but I do see that in kind of both cases, there's this sort of like, where does the intelligence or the, you know, the sort of ghost in the machine? Like, where does that start to emerge? Because at the foundation here, you have this ultimately instantiated in a physical circuit form, right? Where it's like,

• 42:51Nathan Labenz?

  And somehow, there is a fully deterministic process that is engraved, represented in an engraving on a piece of silicon, right? That does that logical switch on when it's a zero, I can somehow skip this step. Can you give us any intuition for how that process is ultimately physically instantiated?

• 43:15Speaker 00

  Let's think about it this way. Say you, your system for moving freight can only move pallets. Okay, so you put a bunch of boxes on the pallet. You wrap the pallet up with plastic. You move it with a forklift. You put it in a truck. You move it across country. Now, in that world, you may not have the ability to determine if one of those boxes is empty.

• 43:45Speaker 00

  Because all your system is doing is putting it on pallets, wrapping it up and moving it. That's a dense world.

• 43:53Speaker 00

  Doesn't matter what's in the package. All right, you're going to treat it the same way. You're going to multiply by it.

• 44:02Speaker 00

  If you had the capability to know what's in every package, you would never ship an empty package across the country.

• 44:11Speaker 00

  Right? That would be brain dead. If you had, in this case, what it is, is the memory bandwidth.

• 44:19Speaker 00

  You could know what's in a package and throw it away before you wrapped it up, put it on a pallet and moved across the country.

• 44:31Speaker 00

  This is sort of what happens when, in the mathematics, when you have huge amounts of memory bandwidth. You don't have to move pallets worth of data over. You can move vectors of data over or even scalar worth of data over. And when you have that sort of fine-grained control, just like when you had control of each box, right? If you could do that, you never have to move empty boxes. And this is achieved by having an extraordinary amount of memory bandwidth.

• 45:07Speaker 00

  Because that allows you to move data to and from your compute engine with tremendous control. And that's one of the advantages of this architecture.

• 45:16Nathan Labenz?

  Okay, you got the chip. Now you gotta go source a wafer. You have to find somebody to actually do the engraving on the wafer. You have to package. Like, these are all the layers where there's this insane supply chain specialization. And I imagine this was all hard because you were probably out of kind of standard spec at every step of the way, right?

• 45:38Speaker 00

  Every step of the way. The process is not quite engraving. It's photolithographic. So you're gonna etch. And you're gonna do that with a laser. And if you want to be at an aggressive geometry, and what that means is if you want to be able to pack an extraordinary number of transistors in a given square millimeter of silicon, you only have two choices. You either go to TSMC in Taiwan, or you go to Samsung in Seoul. And starting at about seven nanometer technology, that is the only two choices.

• 46:22Speaker 00

  In 2016, when we went out, we were at the 16 nanometer node. That means there's 16 nanometers gap between each of these transistors. So you only have two choices. And one of the very few advantages of not being in your 30s is that if you've been in the chip industry for a while, you've built relationships with chip manufacturers. And we had a relationship with Taiwan Semiconductor Manufacturing. We flew out to Taipei, and we met with their leadership. And we told them we had an idea to build a chip that was 56 times larger than anybody had ever done.

• 47:05Speaker 00

  Moreover, our idea allowed them not to change their manufacturing process.

• 47:11Speaker 00

  To use the steps they already use.

• 47:16Speaker 00

  And that we'd patented these ideas. And that together, we believed we could do something that had never been achieved before. And to their great credit, you know, they're a company with tens of billions of revenue. And we were 30 dudes in a PowerPoint. And they listened, they thought about it. And in the meeting, they greenlit the project. Tremendous credit to a company that puts decision makers in a room with innovative startups. They greenlit the project.

• 47:48Speaker 00

  We built the part and first time around it worked.

• 47:52Nathan Labenz?

  There's a couple of things there that are super interesting. I mean, for one thing, when you're raising your initial capital, you know, a risk presumably would be, are you going to be able to convince one of the two companies that could possibly actually manufacture this to take your business, given that they've got a lot of stuff going on? Then because that's such a risk, your answer to that risk was, okay, we're going to figure out how to do a giant chip with the same machinery that currently makes a bunch of smaller chips. And the stepping that you referred to is a sort of etching one

• 48:28Nathan Labenz?

  chip at a time on a wafer. And then those get kind of sliced up. And then you've got like a bunch of little chips, right? You figured out how to do your whole thing with the same core architecture for them.

• 48:38Speaker 00

  All chips first. The way chips are made is there's a 300 square millimeter circle of silicon. Right? That's a blank wafer. And that enters the fab. And just like your mother knocked out cookies, the stepper flashes.

• 49:04Speaker 00

  Little cookie cuts across this 300 millimeter square. And then what happens usually is the wafer goes out to singulation and it's cut along all those cookie lines, just like your mother when she pushed down and the cookie cutter ended up with a cookie, right? And so all chips begin as part of a full wafer. They get etched as part of a full wafer and then they get chopped into little pieces. That's why they're called chips because they're chips off the, uh, off, off the full wafer. We thought, how silly is that?

• 49:47Speaker 00

  To break Humpty Dumpty up into little pieces and then pay for expensive switching to try and get it to behave like it was together again. And that what if we didn't cut it up into pieces?

• 50:02Speaker 00

  And our, our innovations allowed the stepping process to remain largely unchanged and allowed us to communicate across what's called the scribe line, the line that they were usually going to cut, right? If your mom knocked out all those cookies, remember she lifted up the dough, the extra dough, they were the cookies left and then she rolled that into dough and made a few more cookies. That extra dough is what we call the scribe lines and usually it's cut away. We said, how do we invent a technique using their existing methodology to communicate across that?

• 50:46Speaker 00

  So that you don't need to cut it all. And the first block of our IP and dozens of so patents were around how we might do that. Once you have this giant part, and I, I don't know if you have a video version of this podcast, but yeah, this is what our chip is. This is an Nvidia A100, this poor sad little small thing here, right? This is our chip, right? So, um.

• 51:21Speaker 00

  Once you understand that you're going to end up with a chip that's 56 times bigger, has trillions more transistors, you have a new class of problems. How are you going to power it? How are you going to cool it? How are you going to feed it with data? And that was the next step in our system design. We invented techniques to deliver power to it. We invented techniques to cool it. We were among the first to deliver water cooling to AI dedicated processors. When you cool with water, you have all sorts of advantages. Water is an excellent coolant. Air is a really weak coolant. When you cool with water, you can run your chips at lower temperatures.

• 52:06Speaker 00

  When you run at lower temperatures, your reliability goes up. All sorts of advantages. So we had the opportunity with all this compute, right? Hundreds of GPUs worth of compute in one little area. We could cool it with water. We could keep the temperature low. We could feed it with huge amounts of data. All of that to get this world-leading performance.

• 52:30Nathan Labenz?

  Fascinating. So I wonder, it seems like there's kind of a story of convergence that has driven the AI moment that we're in, you know, over the last few years. And I wonder if you're maybe even potentially going to unlock another layer of convergence? If the, you know, the triad, right? Of algorithms, compute and data and those kind of all, you know, have to have them all, right? You've got to have a smart algorithm. You've got to have the compute. Most people, you know, today default to thinking of GPUs. That, you know, unlocks this parallelization. But as we've talked about, it comes with this incredible complexity. And, you know, while you've been building your giant chips for the last seven, eight

• 53:13Nathan Labenz?

  years, there's been a whole other giant effort that has gone toward building up a software stack to manage that complexity.

• 53:22Nathan Labenz?

  If I start to imagine the future, I'm thinking, you know, is there any bottleneck to like how many chips you can produce? And do we go to a world where we get kind of both of those converging, like the complexity of the software as, you know, maybe models get even bigger? And now of like, the cluster is not the Nvidia chips, but your chips? Is that where we're headed?

• 53:41Speaker 00

  We are. A couple of things. First, the software landscape is both more simple and more complicated than it was in 2015. 2015, if you remember, there were half a dozen frameworks. Right? There was Caffe, there was Theanos, TensorFlow was rudimentary. I don't think there was PyTorch yet. There were a lot of different choices. And right now, the world has sort of settled on PyTorch. Right? So, the number of languages that the ML practitioner writes in has really narrowed. Also, because of the work of Chris Latner and others, there

• 54:24Speaker 00

  has emerged an intermediate representation, in particular MLIR, so that this PyTorch can be compiled to something everybody agrees on. None of that was in place, and that's a vast simplification. Now, with that, our models have gotten absurdly large. Right? And not only have they gotten large, but we've learned that using a ton of data is really helpful. And so, we're running very big models with huge data sets. And that results in

• 55:02Speaker 00

  a need for obscene amounts of compute. Where, really, where, you know, dozens of GPUs don't get it done, you need thousands or tens of thousands of GPUs. We replace hundreds of GPUs. And so, what we saw right away was an opportunity to build clusters of our machines. And in November of last year, we announced a supercomputer called Andromeda, built up of 16 of our machines. That's 13.6 million cores. Right? That's big. The largest supercomputer

• 55:40Speaker 00

  on Earth has 8 million cores. They're bigger cores, but just as an idea of how much compute is here.

• 55:51Speaker 00

  And with this, we had the opportunity to immediately train and open source some GPT models. And so, what, six weeks ago, five weeks ago, we put into the open source community seven GPT models, trained to the chinchilla point. And we're one of the initiators in this recent push to not close models, to not do what OpenAI did and keep GPT-4 closed, to not do what Meta did with Llama and have it with a really restrictive license. We were able to cluster these machines so that we could move extremely quickly through a collection of GPT models and that we could put them in the open source community for everybody to use. And use any way they want.

• 56:36Speaker 00

  We used an Apache 2 license. So if you want to make money on them, please go make money.

• 56:42Nathan Labenz?

  It's amazing that you can create a supercomputer out of 16 chips. Is there a, you know, 256-chip version of that? Or is there, is there any sort of, you know, scaling problem that you will encounter as you try to build clusters?

• 56:57Speaker 00

  We've worked on problems past 192.

• 57:00Nathan Labenz?

  And for today, there's just not even workloads that need that much? Are you ahead of like what the max workloads are?

• 57:07Speaker 00

  Well, I, I think there are projects that we'll announce over the next six or eight months that use an enormous number of these machines in, in some of the largest clusters on Earth.

• 57:19Nathan Labenz?

  How many chips are you making on an annual basis? How much do they cost? Can I buy one? This doesn't seem like something people would normally buy one of. It seems more like a shared access model would make more sense.

• 57:30Speaker 00

  Nathan, if, if, if you would like a chip, I might be able to find you a chip, one that somebody dropped or something.

• 57:38Nathan Labenz?

  Yeah, I'll take what I can get.

• 57:40Speaker 00

  Look, we, we, we have customers around the world. We have customers in, in, in, in Japan, across North America, in Europe. We have customers in, in the military, at the government, in large enterprise, in, among tech startups. We have customers who deploy whole systems on premise. We don't sell the chips, we sell the whole system, the whole computer on premise. We have customers that, that use our cloud.

• 58:10Speaker 00

  And so we've tried to think about how to meet customers where they are. You know, younger AI practitioners want everything in the cloud.

• 58:21Speaker 00

  Big Pharma often has, or companies in the energy industry often have extraordinarily proprietary data that they like to keep on premise. We have customers in both, GlaxoSmithKline, Total, all customers of ours.

• 58:36Nathan Labenz?

  I wonder how abstracted away folks are from, like if you're a developer, if you're a machine learning practitioner like a GlaxoSmithKline, and you insist that, you know, we're not going to send anything off premise, so that's just a nonstarter. And then you have an option of like, okay, we can go assemble our own GPU cluster, or we can buy like one of the biggest chip ever in a system. How is the experience then of actually doing machine learning at this company change depending on which choice you make?

• 59:08Speaker 00

  In both cases, you'll likely write it by Torch. In our case, you will spend very little time thinking about distributed compute. With the GPU, you will spend a lot of time thinking about distributed compute.

• 59:23Speaker 00

  In the GPU's defense, if you want to do rendering or some other graphics work, you have that ability with those GPUs. With us, you don't. We're going to be a dedicated AI machine. We're likely to be tens or hundreds of times faster, depending on how many GPUs you replace. So your iteration time for your research will be vastly accelerated.

• 59:47Nathan Labenz?

  Is there a sort of peril, like contention, I guess, question as well? Like, let's say we have one system and we have, you know, 10 machine learning practitioners. If they have a GPU cluster, they can sort of split up the GPUs. How do they divide access to your system?

• 1:00:05Speaker 00

  In a time division. I mean, I think if you have 10 and you're using a cluster, everybody's got a tiny little bit of a cluster and everybody's drinking coffee waiting for their training runs to complete. I think on our machine, each training run will be very, very fast. And the results will be, for everybody, will be completed sequentially, but in a tiny fraction of the time. This is an extraordinary time in AI. And I, I think the last six months have been a veritable revolution. And I, I think we are still very much at the beginning still.

• 1:00:46Speaker 00

  Um, mostly what we're doing today is replacing things we already do a little bit more efficiently. Right? We already write copy. We, we, we already generate stories. We already do many, many things. Um, we haven't got to the point yet where we do new things. Things that we've never done before, where we reorganize around AI. And historically, that's when you get this unbelievable burst of a productivity gain.

• 1:01:19Nathan Labenz?

  Do you see giant chips like this on the edge at some point? You know, how, how far does this go? Like, is my, you know, future Tesla gonna have kind of, you know, one big world record holding chip in it?

• 1:01:34Speaker 00

  No. Um, I, I, I think, uh, you know, e- e- each problem ha- has, uh, characteristics that, that the solution needs to, to meet. Uh, the problems of the edge are, are one of distribution, right? You have lots of cars. They're all gonna send data back somewhere. That's where you're gonna have the big chips. Right? Uh, they're not gonna be in your cell phones, but the, the training that, that, that the, the cell phones inference engines use are gonna be done somewhere with an extraordinary amount of compute and extraordinary amount of data. That the problem with the edge with regard to training is the edge doesn't hold a lot of data.

• 1:02:17Speaker 00

  The edge almost by definition is storage light. And so the right thing for the edge is something very different than the right thing for the core. And the, the, the, the problem of training is a profoundly data intensive problem. Right? That's not a good problem for the edge. Um, the problem of inference is an extremely data light problem. Right? In-image outclassification. In-prompt.

• 1:02:53Speaker 00

  Out of 1,000, 10,000 tokens. I mean, that's tiny, tiny, tiny. That's a good edge task. Where are you going to hold a trillion tokens on the edge for training? Right? Where are you going to hold the compute power equivalent to 16, 32 of our systems or tens of thousands of GPUs? That's what you want for training. And so I think, just like today in the data center for x86, you have big honking servers that are doing the work in the data center and you have small efficient compute in your phone, it's ARM, and they divide up the work.

• 1:03:36Speaker 00

  Right? Some of the work is done on your phone and sometimes it goes back into the cloud, works on a big server, brings down the answer. That model is exactly what's going to obtain in the AI world as well, my guess.

• 1:03:52Nathan Labenz?

  So maybe I may have a misconception and maybe you can correct me on it, but what I was kind of thinking in terms of the edge is, my understanding is like the memory requirements for training are usually like either 2 or 4X or something, the memory requirements for inference, because you have to like keep track of the back propagation gradients, which you don't have to do at inference time. But it still seems like for my like edge device today, you know, or even if I want to go do like a collab notebook on collab pro, I can only load so big of a model. And I guess what I'm kind of wondering is like, you know, is there something I'm missing that, because I'm envisioning a big chip that kind of solves this memory problem, being able to do like

• 1:04:35Nathan Labenz?

  big model, fast, you know, in a loop inference on the edge, which is kind of, you know, you have to have that loop if you're going to have like a robot or a car, right?

• 1:04:44Speaker 00

  There are two different classes of inference problem. One is the auto-regressive inference problem and the other is a classification problem. Auto-regressive inference is much, much harder.

• 1:04:58Speaker 00

  And you have to, it requires vastly more compute. The problems of self-driving cars are one of classification. Child, dog, pile of leaves.

• 1:05:12Speaker 00

  Right? Your car's got to behave differently with that classification. That problem is, can be done extremely quickly in a very small chip.

• 1:05:22Speaker 00

  After a great deal of training. And that's why Elon keeps bragging about how much compute he has in the data center. The problem of auto-regressive inference, which is ChatGPT and others.

• 1:05:35Speaker 00

  Is so hard and challenging that, you know, Microsoft and OpenAI are capping your ability to use it after they set it out for free. Right? Because that actually takes a fair bit of compute.

• 1:05:47Speaker 00

  So, most of what robots do.

• 1:05:52Speaker 00

  Is image. And is a classification problem which can be handled very quickly in a very small chip. Most of auto-regressive inference is still being done in the data center. And is a big opportunity for big chips like ours and requires big clusters of GPUs as well.

• 1:06:13Nathan Labenz?

  So do you guys offer managed inference type service today?

• 1:06:17Speaker 00

  Not yet, but it's around the corner.

• 1:06:20Nathan Labenz?

  I have a couple kind of usual fun closing questions if I could throw a couple of quick hitters at you. First, just your favorite AI products and experiences that you'd recommend others check out.

• 1:06:30Speaker 00

  I'm awed still by the power of ChatGPT. I like Replit for code generation. And the work I'm just doing is really interesting. Dave over Jasper has built a nice company and using Jasper for brand voice is, we use it here. It's really, really interesting application. Those are all consumer applications. We have non-consumer applications that I'm extremely proud of. We, our work was part of the work done by Argonne National Labs that won the top paper at the last Super Compute Show.

• 1:07:16Speaker 00

  In which we were predicting, based on COVID virus DNA, we were predicting mutations.

• 1:07:25Speaker 00

  And that was really cool work. Um, and is may well be the foundation for uh vaccines uh of uh that cover more potential mutations. There's work being done at Galaxia Smith Klein in epigenomics. There's this collection of work that that people don't see because it's not consumer focused uh that I think is extremely powerful and will will have a profound influence on on on how we live.

• 1:07:58Nathan Labenz?

  Yeah, the uh intersection of AI and biology is a very difficult dynamic to predict, but it definitely is going to have a huge impact. Speaking of which, our our second quick hit closer is let's imagine a hypothetical situation where Elon's got Neuralink's in one million human heads. You know, general safety profile looks pretty good. Uh, there may be some noise on the internet about it, but like, you know, best received wisdom is it's like clinically approved. Would you be interested in getting one at that point so you could control your computers with your brain?

• 1:08:36Speaker 00

  I think the the problem that E- E- Elon's identified and I I know Elon, um, he's right. The the problem is one of input and output. That one of the really interesting things about our brain is the amount of I O we can deliver to it through our eyes. And it is vastly more than we can figure out how to deliver even to big chips like like ours. And so what he's trying to solve, the problem he's trying to solve for there is an I O problem. And I I think that's enormously interesting. I I don't know if if I would leap in it. Uh, I'm sort of happy with my human limitations.

• 1:09:21Speaker 00

  Um, but uh, the notion that one of the things that makes the brain so interesting is its ability to get data to it at a vastly higher rate. Orders of magnitude, hundreds of orders of magnitude, maybe thousands of orders of magnitude, faster than we can deliver I information to a chip.

• 1:09:44Speaker 00

  And deep down, that's the insight I think he's chasing. And that's what Neuralink is about. And I think that formulation of the problem, that we have all this compute on chips, and the binding constraint is I/O. And when we look at our brains, we can get so much data in through our eyes, through tactile, through our sense, smell and taste. Unbelievable how much info we can deliver to the brain. And if you look at babies, that's what they're doing. They're touching things, they're tasting things, they're looking at the world.

• 1:10:29Speaker 00

  And that's just, their brains are being inundated with data. And I think the recent work in large language model shows just how important that data is. That it's not about parameter count. It's really about data and the quality of data.

• 1:10:46Nathan Labenz?

  I also think of it in terms of sort of potentially unlocking communication direct from the brain in a sort of, or to the brain in a higher dimensional space than the, you know, language bottleneck that we can kind of speak with.

• 1:11:01Speaker 00

  The language bottleneck is a problem. Right? It takes a lot of words to describe a picture. Right? And what you're thinking about there is, right? You're converting experience into a relatively simple bit stream, words. Whereas images, smells, taste, sentencations are vastly richer. And so I think your formulation is a very reasonable and absolutely accurate formulation of this I/O problem.

• 1:11:47Speaker 00

  How we get away from the relatively simple stream of vector characters.

• 1:11:53Nathan Labenz?

  Okay, here's a far out one. I don't know where consciousness comes from in a person or why I have any subjective experience. And, you know, I just Googled quickly 100 trillion synaptic connections in the human brain. I compare that to your 2.5 trillion transistors. And I really wonder, why is everybody so quick to say that we are so confident, you know, that there's nothing that it feels like to be these new, you know, extremely complicated substrate that somehow, you know, percolates up this sort of, you know, pretty interesting and seemingly

• 1:12:37Nathan Labenz?

  intelligent behavior. Do you have an intuition for either like why we should be confident in feeling like there's no subjective, and I don't mean a human like subjective experience. If there is a subjective experience, I expect it to be very alien and weird, but people are so confident that there's none. And I'm wondering, do you have confidence that there's none? If so, can you tell us why? Or are you open to the possibility that your chips may, or chips plus algorithm may feel something?

• 1:13:05Speaker 00

  I'm confident I have no idea. Really, and I, I, I think what's actually happening underneath all this is a lot of addition and multiplication. Right? That's what sparse linear algebra is. Right? Can extraordinarily complicated things be built up on extremely simple foundations? I think we know they can. I think we know they can. Would we know what self-awareness for a machine would be? It's not clear to me. We would know it. You know, economists used to say,

• 1:13:45Speaker 00

  humans behave in a self-interested manner. And then, and they're rational. And then others came along and proved they're not. But the truth is, as a rule of thumb, most of the time, people are pretty self-interested. It's an okay way to understand behavior. I, I don't think we're going to know or understand it, but I think we'll have some rules of thumb. You know, it, it will behave as if.

• 1:14:14Speaker 00

  It was thinking like a human. It will behave as if it had feelings, whether it actually had feelings or not, I have no idea. And I don't even know the right framework for attacking that problem intellectually. But I think there's a big difference between saying, humans do rational calculations about self-interest constantly, which we know they don't. They're thinking is biased. They don't. And saying that, look, as a rule of thumb, you get a pretty good result if you estimate people's behaviors about self-interested. I think it'll sort of be

• 1:14:56Speaker 00

  the same, that we will see behaviors we recognize, we will probably misattribute it, as the economist did, to a feeling, to self-interest. It will be a reasonable rule of thumb. And that's sort of the way I think about it. But Nathan, there are certainly many people wiser than me on this topic for you to have on your show, people who've thought more deeply about it, who weren't encumbered by trying to build a chip and a system and a software stack and make it easy to use and build big clusters and run big models, who really think about these problems holistically.

• 1:15:37Nathan Labenz?

  Well, it's a tough one. I don't know that there's actually all that many people with a read on this out there. So I'll just ask you one, you know, final question, which again is kind of pushing a little bit beyond, you know, your wheelhouse of the chip business that you've built, which this has been a phenomenal angle on that whole industry and also just, you know, really deep dive into, at least from our standpoint, deep dive into all the problems. I know you guys go a lot deeper still yet, of course. But just zooming out and trying to take stock of where we are in this seemingly kind of phase change moment, what are your biggest hopes for and fears for society at large as AI really comes online?

• 1:16:21Speaker 00

  There is tremendous opportunity to change the way we work, live, and play. And with every powerful technology, there are ways for it to be used for grand evil. And facial recognition can simultaneously be used to persecute minorities, and it can be used to identify terrorists. I think we have to remember to compare our technologies against not idealized versions, but actual reality, right? Humans are terrible drivers. We kill other humans all the time. The goal of self-driving cars is not to avoid killing people, right?

• 1:17:04Speaker 00

  The goal of self-driving cars is to kill a lot less people via driving. And I think that's really important. You know, we are, despite 50 or 70 years of sort of thoughtful effort to reduce biases in our decision-making, people are still extremely biased in their decision-making. It's not surprising that our first models are biased in their decision-making. But for me, I think it's easier to correct algorithms than it is to correct people. And once you correct the algorithm, they stay corrected, whereas people tend to regress. And so I think there is opportunity for good and evil.

• 1:17:46Speaker 00

  I think, like the web, which sort of began dashing into pornography and into the dark web, and the rest of the benefits sort of caught up some years later. I think the opportunity for AI to be used for thoughtful phishing, for sort of painful scamming, for oppression is real and large and has to be thought through carefully. But I think on balance, I think this technology will fundamentally improve the way we live, work, and play. And I think that that is the case for for everything important. There is no important technology that can't be used for evil.

• 1:18:31Speaker 00

  I want to thank you for having us, Nathan. I think it was really fun conversation. Clearly, you know, I do a few of these, and your questions were really thoughtful and fun to ponder.

• 1:18:45Nathan Labenz?

  Andrew Feldman, thank you for being part of the Cognitive Revolution.