The Computing Platform Underlying AI, with Jensen Huang, Founder & CEO of NVIDIA

So much of the AI conversation today revolves around models and new applications. But this AI revolution would not be possible without one thing – GPUs, Nvidia GPUs. The Nvidia A100 is the workhorse of today’s AI ecosystem. This week on No Priors, Sarah Guo and Elad Gil sit down with Jensen Huang, the founder and CEO of NVIDIA, at their Santa Clara headquarters. Jensen co-founded the company in 1993 with a goal to create chips that accelerated graphics. Over the past thirty years, NVIDIA has gone far behind gaming and become a $674B behemoth. Jensen talks about the meaning of this broader platform shift for developers, making very long term bets in areas such as climate and biopharma, their next-gen Hopper chip, why and how NVIDIA chooses problems that are unsolvable today, and the source of his iconic leather jackets. No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: Jensen Huang | NVIDIA Nvidia's A100 is the $10,000 chip powering the race for A. I. | CNBC Nvidia CEO Jensen Huang: A. I. is at ‘inflection point’ | Fortune Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Nvidia Show Notes: [1:26] - The early days when Jensen Co-founded NVIDIA [4:58] - Why NVIDIA started to expand its aperture to artificial intelligence use cases [10:42] - The moment in 2012 Jensen realized AI was going to be huge [13:52] - How we’re in a broader platform shift in computer science [17:48] - His vision for NVIDIA’s future lines of business [18:09] - How NVIDIA has two motions: Shipping reliable chips and solving new use cases [25:41] - Why no one should assume they’re right for the job of CEO and why not every company needs to be architected as the US military [31:39] - What’s next for NVIDIA’s Hopper [32:57] - Durability of Transformers [35:08] - What Jensen is excited about in the future of AI & his advice for founders

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  Computer programming has now been completely disrupted. That for the very first time in the history of computing, the language of programming a computer is human. Any human language. It doesn't even have to be grammatically correct. And it's fairly incredible that anyone can program a computer now. That's a big deal.

• 0:30Sarah Guo?

  This is the No Priors podcast. I'm Sarah Guo.

• 0:33Speaker 02

  I'm Alad Gell.

• 0:34Sarah Guo?

  We invest, advise, and help start technology companies.

• 0:37Speaker 02

  In this podcast, we're talking with the leading founders and researchers in AI about the biggest questions.

• 0:49Sarah Guo?

  So much of the conversation in AI today is about new models and new applications. But it's all built on the back of Nvidia and specifically Nvidia GPUs. The Nvidia A100 has been called the workhorse of today's AI ecosystem. And this week on No Priors, we're so excited to have Jensen Huang, the founder, president, and CEO of Nvidia. So over the last 30 years, he's created a company that began with a goal to accelerate graphics and has transformed the way we compute to become a $674 billion behemoth, as far as I could tell this morning, but that the CEO describes as a company that he's always trying to save. Thank you for doing this with us, Jensen.

• 1:25Speaker 00

  Glad you do it, Sarah.

• 1:26Sarah Guo?

  Why don't we start at the beginning? You worked at LS-I and AMD before starting a company. How did that happen?

• 1:32Speaker 00

  They gave me a job. Let's see, I was at Oregon State University, and it was a campus company day. And I interviewed at a lot of companies. And two companies really, really connected with me. I love designing chips and designing computers. And at the time, in our lab, in the computer science lab, there was a poster of a 29,032-bit CPU from AMD. And I always thought it'd be kind of cool to build that. On the other hand, there was another company that was a startup at the time, built by one of the legends of Silicon Valley, Wolf Corrigan. They started a company to design chips using software,

• 2:17Speaker 00

  to design chips not by hand, but by using programmable logic. And you would describe it in language and it would synthesize it to chips. And of course, I chose to go to AMD. It turned out that I went there to design microprocessors. And my lab partner, not my lab, but my office mate, ended up going to LSI. And she insisted I go there after I was there. And after she went there and the LSI team said, "Hey, we were recruiting this kid from Oregon State and we really wanted him to come work at LSI Logic." And it turned out to have been her office mate. And so they all reached out to me. And I decided to go there because it was at the beginning of the EDA industry. It was at the beginning of designing chips using computers.

• 3:00Speaker 00

  And it was probably one of the best things that ever happened to me. And it was in the beginning of the ability for every company to build their own chips. And it's the reason why I met some really great computer architects. Andy Bechtosheim was the founder of Sun. I got to work with a bunch of great architects at Silicon Graphics. And John Rubenstein, who was at a company called Dana Computer, who became the Vice President of Engineering for Apple. And then, of course, the two founders of Nvidia, Chris Malkowski and Curtis Prein, myself. And so I got a chance to work with some really amazing computer architects. And I learned a lot about building computers with chips. And so that's my early days.

• 3:37Sarah Guo?

  And you were a star at LSI with your co-founders. At what point did you know, "I have to start a company"?

• 3:43Speaker 00

  It wasn't my idea, it was theirs. Chris and Curtis wanted to leave Sun. They had their own reasons. And I was doing really well at LSI Logic and I enjoyed my job. And we had two kids, Lori and I. And just like you, they wouldn't stop hounding me. And they said, "Hey, we want to start this company and we really need you to come along." And I told them that I really needed to have a job and they needed to figure out what to do. At the time, the valley was, the way of designing computers was rather split between general-purpose computing versus using accelerators. And about 99% of the valley was believed in general-purpose computing and about 1% believed in acceleration. For 25 years, 99% was right.

• 4:26Speaker 00

  We decided to start a company on accelerated computing. And at the time, the only thing you could really do with accelerated computing is find applications or find problems that were barely solvable or unsolvable by general purpose computing. And that's kind of what we dedicated our company to do, to solve problems that normal computers can't. And if you follow that mission to its limit, it led us to self-driving cars, it led us to robotics, it led us to climate science problems, digital biology, and of course, one of the most famous ones is artificial intelligence.

• 4:57Sarah Guo?

  You were working on this huge set of applications before the current wave of artificial intelligence. What was the original technical advantage of Nvidia and artificial intelligence? And when did you begin to realize that this was going to be an important use case for you guys?

• 5:12Speaker 00

  So we had expanded the flexibility of our accelerators to be more general purpose. And we invented a new computing model called CUDA. And we're doing this podcast like at four o'clock or something like that in the afternoon. It was like at the lowest point of energy. Isn't that right?

• 5:32Speaker 02

  Yeah.

• 5:33Speaker 00

  So this, we need some.

• 5:34Speaker 02

  That's why we need.

• 5:35Speaker 00

  We need some nerds.

• 5:36Speaker 02

  Nerds.

• 5:37Speaker 00

  That's right, we need some nerds, guys.

• 5:38Sarah Guo?

  Thank you, nerds.

• 5:39Speaker 02

  Now with gummy clusters as well. So that's very exciting new technology.

• 5:42Speaker 00

  We need some energy. We need some accelerated computing right now. So we wanted to make our graphics processors more and more general. And the reason for that in the beginning was because some of the effects that we had to do related to general purpose image processing, post effects, you render the image and you do post image effects. Other applications, of course, we wanted to bring the scene to life. And so we had to do physics processing. And you have to do physics, you have to do particle physics, fluid dynamics, so on and so forth. And so we expanded the aperture of our accelerated computing platform to be more and more and more general purpose. The problem with general purpose-ness is that the more general purpose you are, the less acceleration you get in any particular domain. And so you've got to find that line really, really carefully.

• 6:24Speaker 00

  And that's one of the gifts of our company, to find that line between, on the one hand, every single generation bringing enormous amounts of acceleration well beyond what CPU could do to the application. And so if you become too general purpose, you're like, just like a CPU. How can you accelerate a CPU with a CPU? And so you have to find a way to walk that line. On the other hand, if you don't expand the aperture of the applications that you serve, the R and D dollars that you're able to generate. Wouldn't be enough to stay ahead of the CPU, which had the largest R and D budget of any chip on the planet. So if you think about this problem, it's actually really nearly impossible. Because you have a small application, let's call it, you know, a billion dollar market at the time.

• 7:06Speaker 00

  And out of that billion dollar market, you're investing $150 million a year. Out of $150 million a year, how do you keep up with a few hundred billion dollar industry? It's not even sensible. And so you have to find that niche very, very carefully, where $150 million would accelerate this particular application abnormally and insanely. And then over time, you could expand your application space so that it goes from a billion dollars to $5 billion, to $10 billion, so on and so forth, without falling off that cliff. That is the fine line that we walked. And so we kept expanding the general purposeness and it led us to molecular dynamic simulation, which is what this image seems to look like. And seismic processing was another industry.

• 7:51Speaker 00

  And slowly by surely, we expanded our aperture. But one of the things that we did well was to make sure that irrespective of whether somebody used our platform for general purpose computing, accelerated computing, we always maintained the architecture compatibility. And the reason for that is because we wanted a platform that would attract developers. If every single Nvidia chip in the world was incompatible, then how would a developer be able to pick one up? Even if they learned that CUDA was going to be incredible for them? How would they pick that up and say, I'm going to develop an application that's going to run on that? Which chip would they have to go figure out? And nobody could figure that out. And so we said, if we believe in an architecture and want this to be a new computing platform,

• 8:34Speaker 00

  then let's make sure that every one of our chips perform exactly the same way. Just like an x86, just like ARM, just like any computing platform. And so for the first five, 10 years, you know, we had very few customers for CUDA, but we made every chip CUDA compatible. And you can go back in history and looked at our gross margins, it started out poor and it got worse. So we were in a really competitive industry and we were still trying to figure out how to do our job and build cost effective things. So it was already challenging as it is. And then we layered on top of this, this architecture that was called CUDA that had no applications that nobody paid for.

• 9:08Speaker 02

  Yeah, it's kind of amazing because now when I talk to people in the AI world in terms of, one of the reasons that they really love using Nvidia GPUs is because of CUDA and then because of the ability to scale interconnect. And so you can really like highly parallelize these things as well, which you can't necessarily do with other approaches or architectures that are in the market today.

• 9:24Speaker 00

  Yeah. And so this computing platform, like it's strange in the sense that it performs these miraculous things. And we carried it to the world on the backs of GeForce, which is a gaming card. The first GPU that Jeff Hinton got for his lab, ELO would tell you that Jeff came in and said, here's a couple of GPUs. It's called GeForce and you guys should try to use that for a DNN. And so it was a gaming card that he bought.

• 9:47Speaker 02

  What applications did you have in mind? Because to your point, you started with gaming, or at least you were very popular with gaming, starting in the 90s when you started the company. And then, you know, I started hearing about Nvidia GPUs more and more, both in the context of cryptocurrencies and sort of mining, and then in the context of AI. And it seemed like those were the two markets where a bunch of people were organically just adopting you. Were you marketing to those communities? Was it just people started realizing that they needed linear algebra?

• 10:09Speaker 00

  That's the beauty of a computing platform, right? In the beginning, you have to target the applications. And in the beginning, we did. One of the first applications was NAMD, Seismic Processing. Both of them are, one of those kind of particle physics, the other one is image processing, if you will, and so inverse physics, if you will. And one particular domain, you know, we just went out to hire it to research. We went to scientific computing centers and we said, what kind of problems are just beyond your reach? And the list of applications are include quantum chemistry and quantum physics and you know so on so.

• 10:41Speaker 02

  What was the moment when you said, wow, this AI thing is really important for us?

• 10:44Speaker 00

  It happened around 2012, I guess, and it was because simultaneously Andrew Ng reached out to Bill Dali, our chief scientist, to work on a way to get the neural network model that they were working on onto GPU so that they could, instead of using thousands of CPU servers, they could use a few GPUs to do training. So that was one. Simultaneously, Jeff Hinton reached out to us and we started hearing about that and the same thing was happening with Yan Le Cun in his lab. And so simultaneously in several different labs, we're starting to feel that there's this neural network emergence and that attracted our attention.

• 11:23Speaker 02

  Yeah, I guess 2012 was also the year when AlexNet came out. So I felt like that was a year of transition for deep learning in general in terms of really, that was the moment in time at least that I remember thinking, wow, this really exciting wave of AI coming.

• 11:33Speaker 00

  Yeah.

• 11:34Speaker 02

  And then I feel like for 10 years, nothing really happened for startups, but a lot of incumbents started adopting this technology.

• 11:39Speaker 00

  Yeah, we started feeling it. We started hearing about it before that and then ImageNet kind of was the big bang, if you will, got all of our attention.

• 11:45Sarah Guo?

  You talk about early AI labs as pulling this from Nvidia using gaming cards because you were solving a problem nobody else could solve and like efficiency and scale. Is there a point at which Nvidia begins to like invest in an application because they think it's a growing application? Or is it more it's a platform and the market will take it from us?

• 12:04Speaker 00

  No, in every single case, when an application finds use, we ask ourselves, how can we make it even better? And this time with deep learning, the good insight that we made, it was piecing together observations in a lot of different ways, but realizing that this isn't just going to be a new algorithm for computer vision, which is really most of the applications in the beginning, but which was going to be very helpful. I mean, just if it was just computer vision, we could use it for all kinds of interesting applications like self-driving cars and robotics, and we did. But we observe that this might be a new way of writing software altogether.

• 12:46Speaker 00

  And asking ourselves, what's the implication to chip design, system design, interconnect, the algorithm, the system software, to really reason about not just why is this exciting? Why was it so effective? Which that alone was plenty miraculous. That ImageNet without specifically any human engineered algorithm would reach the level of effectiveness compared to 30 years of computer vision algorithms overnight. It wasn't by a small amount. And so the first question, of course, is why is it so effective? And was this going to be scalable? And if it was going to be scalable, what's the implication to the rest of computer science?

• 13:26Speaker 00

  What problems can't this universal function approximator, if you will, that can solve problems of dimensionality extraordinarily high? And yet you could learn the function using enough data, which at the time we were starting to believe we can get plenty of, and to systematically train this model into existence? Because you train them one layer at a time.

• 13:51Speaker 02

  Could you talk a little bit more about, I've heard you be very articulate in terms of how you view this as a broader platform shift? Just even in terms of how pages are served versus generated or other aspects of that? Could you talk a little bit more about what's really happening right now more broadly in computer science with this shift to AI?

• 14:05Speaker 00

  Yeah, so you fast forward now a decade. The first five years was about reasoning the impact to computer science altogether. At the same time, we're developing new models of all kinds, right? And so CNNs to ResNets to RNNs to LSTMs to all kinds of new models and scaling them larger and larger, making great strides in perception models particularly. And of course, the transformer was a big deal, BERT was a big deal, all of you know that story well.

• 14:33Sarah Guo?

  Did you guys see like a step change in volume growth with transformers and BERT and such? Because it feels like having an architecture and an attention mechanism that allowed for scaling of these models really was also a kickstart in the industry.

• 14:47Speaker 00

  Well, the ability for you to learn patterns and relationships from spatial as well as sequential. Data must be an architecture that's very effective. So I think on its first principles, you kind of think transformers is going to be a big, big deal. Not only that, you could train it in parallel and you can really scale this model up. And so that's very, very exciting. I think that when transformers first came out, we realized that there's a model now that overcame the limitations of RNNs and LSTMs and we can now learn sequential data in a very large way. So that was very exciting. BERT was very exciting. We trained some of the early language models ourselves and we saw very good results.

• 15:29Speaker 00

  But it wasn't until the combination of reinforcement, learning, human feedback, and of course, some of the breakthrough work that was done with retrieval models, dialogue managers that does the guard railing. It wasn't until some of all of those kind of pieces start to come together that, of course, that we all enjoy ChatGPT. And Elad, the point that you're trying to make is the observation that computer programming has now been completely disrupted, that for the very first time in the history of computing, the language of programming a computer is human. Any human language. It doesn't even have to be grammatically correct. And it's fairly incredible that anyone can program a computer now. And so that's a big deal.

• 16:11Speaker 00

  The fact that you program it differently, it writes different applications. What is the reach of this new computing model? Apparently quite large. And it's the reason why ChatGPT is the fastest growing application in history.

• 16:22Sarah Guo?

  We had Alex Gravely, who was the chief architect for Copilot on the show as well. And his favorite, like obviously it's, you know, very powerful to have sequential code prediction, but his favorite use cases of Copilot have been like people telling him that they don't code, but now they do.

• 16:37Speaker 00

  Yeah, right.

• 16:38Sarah Guo?

  Which I think is very democratizing, as you said.

• 16:40Speaker 00

  It's quite amazing that you could give ChatGPT a problem to solve and it reasons through it step by step, but yet it arrives at the wrong answer. On the one hand, on the other hand, you could tell it to write a program to solve the same problem and it writes a program that solves the problem perfectly. The fact that there's an application that on the one hand reasons and tries to solve a problem and does a fairly good job at it, it's almost there. On the other hand, it can write a program altogether to solve that same problem. You've got to really wrap your head around the implication of this. So do you view it as like the future world is some form of machine sentience? First of all, I don't even know what that word means in a technical way. Yeah. I'm fairly sure that I'm sentient, less so today.

• 17:23Speaker 00

  So we have nerds fart.

• 17:24Sarah Guo?

  Keep eating.

• 17:25Speaker 00

  That's why I need nerds to crank me up here.

• 17:27Sarah Guo?

  I'm trying to.

• 17:28Speaker 00

  I know, I know, today was a tough day. But I don't know. Do I believe that we now have a software that can reason through a problem for many, many types of problems, reason through a problem and solve and provide a solution? Or a program to systematically provide a solution on an ongoing basis? The answer is yeah.

• 17:47Speaker 02

  Yeah. And then as you look forward to that world, how do you think about where you want to take Nvidia's lines of business? But also, you mentioned in the past that Nvidia has done things like train models and you've done some really interesting things there. Is that going to be an increasing part of what you do in the future? Or are you mainly focused on the chip side? Or how do you think about that mix of helping to push forward some research as well as being the underlying platform for the industry?

• 18:09Speaker 00

  Well, we're a computing platform company. And we have to go up the stack as far as we need to so that developers can use it. And so the question is, what is a developer? And in the beginning, of course, a developer is somebody who controls their own operating system. And so in those days, we might only have to go as far up as device drivers or the layers slightly underneath that somehow to enable developers. But for scientific computing and all these different domains, the developer is actually using maybe a solver. And they need the algorithms of that domain to be somehow expressed in a way that could be accelerated. Which is the reason why when we moved into these multi-domain physics problems, we

• 18:54Speaker 00

  realized that we have to develop the algorithms themselves. Because the algorithms of solving a problem relates to the computer architecture that's underneath. And if the architecture is CPUs, connected through MPI and Ethernet or whatever it is, that algorithm is surely very different than thousands of processors that's connected by a fabric inside one GPU and thousands of GPUs inside a data center. So obviously, the algorithm has to be reframed and refactored. And so our company got very good at designing computer algorithms. It could be for particle physics or fluid dynamics. And then, of course, one day, it was related to deep learning and neural networks.

• 19:34Speaker 00

  And CuDNN is essentially a domain-specific language for accelerated deep learning. And so we've done that for deep neural nets. We've done that for computer graphics, ray tracing, that's called RTX. All of these different domain libraries really is about understanding the domain of science and then redesigning algorithms that make them go incredibly fast. Now, in the future, what's a developer? Well, I think in the future, a developer is likely going to be somebody who engages large language models and foundation models. Now, if somebody could use ChatGPT or OpenAI's model, I'd really encourage that. And the reason for that is because they do such an incredible job. If somebody could use it through Microsoft, I'd really encourage that.

• 20:19Speaker 00

  If somebody could use it through Google, I would really encourage that. But if somebody needs to build a proprietary model for a domain, maybe create a new foundation model, and let's say the domain was proteins, or let's say the domain was chemicals, or let's say the domain was climate science, multi-physics. That foundation model is pretty nichey, and it's not a small market, obviously, because the field of drug discovery is large, the field of climate science is large, climate tech is large. However, it's not likely to be horizontally useful for every human. And so we might decide to go do something like a foundation model for 3D graphics, virtual worlds, because they're super important to us.

• 21:01Speaker 00

  We might decide to build a foundation model for robotics, because it's at an intersection of the things that we do very well. And even then, we'll probably take it as far up as necessary, but no further than that. We're not trying to be an AI model company. We're trying to help industries create AI models. Mostly, we're trying to help developers.

• 21:21Speaker 02

  Yeah, that makes a lot of sense. You're basically following your customers up to whatever level they need you to.

• 21:25Speaker 00

  That's right.

• 21:26Speaker 02

  And then you hand it off to them at that proper point.

• 21:28Speaker 00

  We're trying to be as lazy as we can.

• 21:30Speaker 02

  Yeah.

• 21:30Speaker 00

  You know, do as little as possible, as much as necessary. The first principles of computer science, we all know well, right? To reject work as quickly as you can. To defer whatever work you are left for as long as you can until you could be rejected. And then whatever remains, you have to do. We try to do as little as we can, as much as possible. That's kind of the principles of the company. Laziness, the principles of the company, that's the takeaway. It's up on the wall outside, I think, too. Be as lazy as you can.

• 22:01Sarah Guo?

  I'm trying to square that with some of these very long-term commitments that the company makes. Right? Like, Kuda is a very long-term bet. And we met a decade ago when Nvidia was valued at one one-hundredth of its value today and was facing activist investors and such. And it was probably like, let's say, a little harder to make long-term bets. How do you balance the pressures of being a large public company and the opportunities of today with architectural commitments or long-term bets? And sort of think about that prioritization?

• 22:32Speaker 00

  Investing in the future and being sustainable now are not in conflict with each other. And so the challenge for all startup CEOs and for all CEOs is to find a way to be able to do what you believe in, the fundamental core belief of the institution, and to be able to afford doing it. That is the purpose of the company. And it's part conviction, it's part skill. Making money is not a matter of conviction. Making money is a matter of skill. And it's a learnable skill. And it took me a long time to learn it, I'll admit that. I've been at this for 30 years and for the first, well, apparently for the first 20 years,

• 23:17Speaker 00

  since you went back 10 years, for the first 20 years, I was still trying to figure it out. But it's a skill, learning how to make money, learning how to run a company efficiently. Those are all skills and the company has to develop the skills. And the way that we ultimately do it is we ask ourselves, do we really believe it or not? And if we really believe in doing something, then it is the purpose of the enterprise, is the singular purpose of the institution to go pursue its beliefs. And the rest of it is up to all of the cleverness of the company and, you know, try to do our jobs well and build things that people want to buy and try to make it as cost effective as possible and make the company as efficient. Those are all skills.

• 23:59Speaker 00

  The hard part, as it turns out, is not the skill part. It took me a long time, but a lot of companies know how to make money, obviously. So the fact that there are more than one company that makes money suggests it's not that hard. If somebody else can do it, hard hard can be. And so singularly advancing a new computing model, we call accelerated computing, and we believe that someday, that on the one hand, accelerated computing can help us solve problems and tackle problems that normal computers can't, and it exposed us to all of these amazing applications like digital biology that I'm excited about today, like climate change that we're excited about, like robotics and self-driving cars. If not for the fact that we're pursuing applications that were

• 24:41Speaker 00

  impossible with normal computers, why would we have discovered all of those things? Why would we have discovered artificial intelligence? Why would we be the workhorse of large language models? Because large language models are barely possible. And if you are doing something that's barely possible, you call us. We're the horse you call to solve those problems. And so I love that aspect. I love the fact that we get to discover those future. On the other hand, we deeply believe that someday, everything will be accelerated. And the reason for that is very clearly that the CPU will run its course. And there's a limit to how far you could scale general purpose computing. And you'll always need

• 25:26Speaker 00

  it. You'll always need CPUs. But the type of applications that we're all going to run, acceleration is really the best way forward. And at our core, we believe that from day one. Thirty years ago, that's the reason why we started the company. And so it's the true conviction.

• 25:41Sarah Guo?

  You have been enormously vindicated on this thirty-year belief. You must have felt that conviction challenged at some point in thirty years of running the company and learning the skills to run the company. What was the nearest death experience or the most concerned where you're like, maybe I'm not right? Or has that ever happened?

• 25:57Speaker 00

  I'm not right for the job?

• 25:59Sarah Guo?

  No, you're not right about accelerated computing and how important it will be.

• 26:02Speaker 00

  The second one's yes.

• 26:07Speaker 00

  First of all, I don't think anybody should assume that they're right for the job. And so you should be gut checking on that almost every day.

• 26:14Sarah Guo?

  To be clear, that wasn't the question.

• 26:16Speaker 00

  But I'll more than have to answer that.

• 26:18Sarah Guo?

  Yeah.

• 26:18Speaker 00

  Answer that question. Did I ever believe that it was wrong? No. I believe that accelerated computing is the absolutely the only way to solve problems that are impossible by definition.

• 26:30Sarah Guo?

  Okay, axiomatically, yeah.

• 26:31Speaker 00

  And on the other hand, if you can solve problems that are impossible today, and someday you need that application to be broad-based, would accelerated computing be the best approach? The answer is yes.

• 26:44Speaker 02

  When do you think the CPU hits its limits? You mentioned that, you know, eventually you think everything will move over, or at least big chunks of the future will move over. Is that five years away, 10 years away?

• 26:53Speaker 00

  For certain applications, it happened 12 years ago.

• 26:56Speaker 02

  Yeah, yeah.

• 26:56Speaker 00

  Right? Jeff Hinton and Jan Le Cun and Andrew, right? Andrew Ng, they discovered that 12 years ago. It was the only way forward. And computer graphics, it's the only way forward.

• 27:08Speaker 02

  Yeah. Has the way that you organize and run the company changed as AI has gotten more and more prominent? Like, have you realigned aspects of the business around it? Or how do you think about management in general in this environment, where things are changing so rapidly and there's so many exciting things happening in this area?

• 27:24Speaker 00

  You're asking a really good question. And maybe if I just take a step backwards, the company's architecture should not be generic. Every company in the world should not be built like the US military. And in fact, if you look at every company's org chart in the world, they kind of look like the US military. There's somebody on top, and then it comes down. And yet, the number of direct reports of CEOs are very few. And the direct reports of the people who are just learning how to manage, first-level managers, are very large. It's exactly the opposite of how it should probably be architected. You would think that the people that report to the CEO requires no management at all, and in fact, it's generally true. My direct reports are sophisticated, they're really talented, they're incredibly good

• 28:07Speaker 00

  at their job, they're excellent leaders, they have great business acumen, they have excellent vision, they're incredible, every single one of them.

• 28:14Sarah Guo?

  I guess that means you have more than the management book accepted six or seven or whatever.

• 28:19Speaker 00

  Yeah, I have forty somewhat direct reports and no one on ones, no career coaching, you know. So, what would you like to do with your life? Those are conversations you have with new college grads and early career, and we love those conversations, of course, and helping them shape their career and mentor them, give them access to new experiences. But at the executive staff level, we're organized so that we can pursue a whole lot of different things at the same time. However, one of the most important things about a software company is you have to understand computer architecture, and one of the most important thing about computer architecture is you can only afford one. Just as some of the largest companies in the world only have two operating systems,

• 29:03Speaker 00

  the single largest company on the planet only has two. How is it possible that so many companies have so many different computer architectures and they have seven or eight or nine instruction sets that they're keeping around? We have one instruction set, we have one computer architecture, and we're super disciplined about that. And so where we need to be focused, we are. Where we allow for innovation and discovery at the senior level, we allow that. So I think the company is tapered and organized in a way that is consistent with the nature of our work, you know, so that's the most important thing, and that's probably the takeaway for what I've learned building our company is there is no one generic architecture for every company.

• 29:44Speaker 00

  It should fit the function of the company, its purpose, and of course, the leadership style of the leaders.

• 29:51Speaker 02

  Yeah, I think that's a really important note that most people don't really realize is that a company should almost be a bespoke structure supporting the CEO and their staff and what the company is delivering to customers versus it's always the same thing.

• 30:02Speaker 00

  Exactly.

• 30:03Speaker 02

  And I think that gets lost a lot.

• 30:04Speaker 00

  Exactly. Yeah, there's some particular chief that you need and a chief that that you need and a chief this, you know, there's some chiefs that you do need.

• 30:11Speaker 02

  Yeah.

• 30:12Speaker 00

  But aside from that, you should start from first principles and architect something that makes sense for the leader and as well as the function.

• 30:20Speaker 02

  Yeah. When I was at Google, they had the famous 80 20 10 where it was like 80% is core, 20% is like core adjacent slash new stuff and then 10% was hyper experimental. Do you have any frameworks or ways to think about that stuff or it's just kind of like, let's see what organically is used in terms of this generic platform that we've built with CUDA and other things that are built in to help support a lot of use cases and as they emerge, we say, okay, let's go support that new thing.

• 30:41Speaker 00

  I don't have any wise framework like that. There are a couple of things that our company is shaped and structured to do. There's one part, a very large part of our company is designed to build very, very complicated computers perfectly. And so that is one of its missions. That kind of architecture, that kind of organization is a invention and refinement organization. We have a whole bunch of skunkworks, if you will. And the reason for that is because we're trying to invent things 10 years out that we're not exactly sure whether it's going to work or not. And there's a lot of adaptation, a lot of pivoting. And so, you know, our company actually has two different ways of working. One of them is rather organic, shape shifting all the time.

• 31:25Speaker 00

  If a particular investment is not working out, we give up on it, move the resources somewhere else. And so that's the agile part of the company. And then there's a part of the company that's not rigid, but it's really refined. And so these two systems have to work side by side.

• 31:39Sarah Guo?

  Can you talk a little bit about the H100, next workhorse? And what the most important innovations are and like what the design and ship process for that looks like?

• 31:47Speaker 00

  I would say the big breakthrough for Hopper is recognizing that quantization, the numerical quantization, the numerical formats has a fair amount of innovation and ability to reduce because it's statistical in the first place. And now the question is what kind of models could be created and trained? And we believe that 8-bit floating point. Rather than, if you look at scientific computing today, sixty-four bit floating point. And so, just by breaking up sixty-four into eight, you could increase the performance of an AI supercomputer just by a factor of eight, by not doing sixty-four bit.

• 32:29Speaker 00

  So that's almost a factor of, if you will, a factor of ten almost, in just a couple of generations, just by recognizing that sixty-four bit floating point wasn't necessary. And so, one of the big things is that. The second thing is Transformer. The Transformer engine is so universal and so useful that it's possible for us to design a pipeline that is shaped for learning and inferencing transformers. And so, those are probably the two biggest things. Otherwise, it's the largest chip the world's ever made. It's, you know, the fastest chip the world's ever made, and super energy efficient, and uses the fast memories of the world's ever made. And then, we connect a whole bunch of these things together so that it's fast and energy efficient. But those are all, you know, kind of brute forcey things.

• 33:10Speaker 00

  But the big architecture idea is FPA and Transformer engine.

• 33:15Sarah Guo?

  And when you think about then, so that's the big project refinement part of the company. We think about the more agile piece. What's the impossible application you're working on today that's ten years out you think is likely to be important? I'm sure there are a ton of them.

• 33:27Speaker 00

  There's a whole bunch we're working on that don't work at the moment, but I've got a lot of confidence it will work. Okay, so for example, autonomous driving is still making progress. But I have every confidence that it will work. I have every confidence that a robotic foundation model will be discovered. And that through expressing yourself using human language, you could cause a mechatronic system of almost different types of limbs and agility to be able to figure out how to bend itself, articulate itself, to do a particular task.

• 34:04Sarah Guo?

  What do you think the blockers are to that today?

• 34:06Speaker 00

  I have no idea, but I can't tell you. I'm just, I don't know, I don't know. Yeah, because we have to discover our way there. But one of the things that we do know... Is that we do know how to learn structure from unstructured information, language, images, and of course, the next big thing is video. And if we could just watch video and learn the structure from the video we're watching, we might be able to learn how we articulate, and we might be able to generalize that and be a de-articulation system for robots. And so I think the road signs, if you will, would suggest that the pieces are coming together. But when we get there, I have no idea. But I think it's probably less than,

• 34:51Speaker 00

  my guess is going to be less than 10 years, probably about five years. And I think you're going to see some pretty amazing robots.

• 34:58Sarah Guo?

  And so exciting.

• 34:59Speaker 02

  Yeah, there's some things along those lines, too. I guess like Paul Me from Google came out recently. That's sort of a step in that direction. And I guess that's still in the transformer architecture. And you mentioned the sort of transformer pipeline and sort of baking that into what you all are doing. Are there other new architectures on the AI side that you're watching or especially you think will develop into something especially interesting?

• 35:19Speaker 00

  Well, there's a whole bunch of derivatives of transformers. And they're all just kind of generally called transformers. But that basic architecture is being refined and delta'd. And on the one hand, on the other hand, some of the stuff that we're really excited about, that we did a lot of work in, we started with Ian Goodfellow's work on GANs. And we did some really great work on style transfer and high resolution generation of images, and which led to a whole bunch of work in variational auto-encoders, which then became, if you will, a bit of a cousin of the diffusion models that came out. And so that entire path, we played a very large role in. And there's a whole bunch of derivative work that's going to come out of that. Between the ability to learn structure from a giant amount of data, whether it's video

• 36:04Speaker 00

  or multimodality, learning is going to be a very big thing, of course. And then the next part of it is generating content. And if you can generate images and you can generate 2D and 3D images, why can't you generate proteins and chemicals? And you can generate all kinds of stuff.

• 36:19Sarah Guo?

  There are almost no other entrepreneurs that have gone from three founders to CEO, 30 years and 700 billion of market cap. What advice do you have for entrepreneurs that listen to the show?

• 36:31Speaker 00

  It's a really hard job. I don't mean the CEO job. Just building a company is hard. The two of you are associated with a lot of companies being formed from the very, very beginning. There's nothing easy about building a startup. And I don't even understand it that anyone would build a startup twice. It is such a ordeal.

• 36:49Speaker 02

  Yeah, I try to talk people out of it, like second time founders, because I started two companies. And the second time, like, are you sure you want to do this?

• 36:54Speaker 00

  Oh, there's no question you shouldn't do it. Yeah, there's no question you shouldn't do it.

• 36:58Sarah Guo?

  It has to be some sort of like forgetting mechanism, like with having kids. You're like, oh, it wasn't that bad the first time.

• 37:03Speaker 00

  You got it. You got it. It's exactly right. For example, you have to forget how hard it was. And I don't know how I do it, but I just do. I forget, I forget the pain and suffering that goes along with doing something. Once you achieve something, you just move on to the next thing. And once you achieve that, you move on to the next thing. And it's just like life. You put one step in front of the other. What advice? I'm reluctant to give them any advice. And the reason for that is this, almost any advice will probably discourage you from wanting to do it. I think ignorance is one of the superpowers of an entrepreneur. And you'll never get it again. You'll never have it again. And so, the thing I really love about our company is we're reinventing ourselves constantly. We're kind

• 37:45Speaker 00

  of entrepreneurs inside this company. And all the meetings that I go to are really startup meetings. And they're all painful. They're all painful, because you're starting something from the ground up again. You have no momentum. You're basically at zero. And it reminds me every single time how painful it is. But it's also so rewarding when you, when you build something and the people you built it for appreciate it. And somehow it made a difference. And then you combine that skill with some other skills, and some other capability, and all of a sudden, you can do something even greater than that. On the one hand, I would tell them that building a company is extraordinarily rewarding, and all the people you get to work with. That's genuinely true. On the other hand, the pain and suffering

• 38:27Speaker 00

  of doing it is unlike anything you could imagine. And so, you know, you're vulnerable, you're a superhero one day. You're a jerk the next day. You go through these cycles and somehow you have to look beyond all of that and focus on what you're trying to do. So I don't know if I gave him any wisdom aside from if you're determined that you want to do it, don't wait too long, just go do it.

• 38:51Sarah Guo?

  Before you lose your ignorance?

• 38:52Speaker 00

  Yeah, before you lose your ignorance. If therefore, there's one attribute, I would say, you have to be determined enough to stay with your conviction on the one hand. On the other hand, you can't be stubborn so that you can have agility, so that you can continue to learn. And so somewhere in that balance of I believe in what I'm doing on the one hand, and I simultaneously believe that I could be wrong on the other hand, that is weird. And you have to believe both equally hard.

• 39:24Sarah Guo?

  My firm's name conviction, you can have agility.

• 39:27Speaker 02

  Okay, I'll start that as a candy brand.

• 39:29Speaker 00

  Yeah, yeah. And we've seen startup CEOs that are incredibly talented and they're almost right. But they were so determined to be right, they forgot to be agile, to learn along the way and pivot and adapt. And so I think that's on the one hand. And so that's one thing to remember. And then the other is resilience, which comes along with forgetting. You have to forget the pain, move on. And it's a little bit like coaches saying, don't worry about the last point. You just got your face kicked in, you know? And you missed a quarter, you know, like when I missed a quarter, when you mentioned crypto, my hand started to sweat. I know, my heart started to beat, you know, faster. Because I remember missing the quarter and you, when we missed a quarter during crypto, we missed

• 40:12Speaker 00

  it hard. Crypto was hard to predict and we went from having no supply to too much. You know, who misses a quarter by, you know, $2 billion? I mean, that's, that's a big number. Most of the time you hear CEOs miss it by $15 million.

• 40:27Speaker 02

  Yeah.

• 40:28Speaker 00

  Not $2 billion.

• 40:30Speaker 02

  I think Sarah had a great point in terms of you've built now really one of the marquee companies in the tech world. And you're really pushing forward, what is potentially one of the most important waves of all time in technology, which is AI. 10 years from now, 20 years from now, looking back, are there any specific things that you want to accomplish, either through the context of the company or more broadly, or other things that you, looking back 20 years from now, you really hope happen?

• 40:50Speaker 00

  That's, that's a good question. And in fact, that's a good way to think. The best way to think about what to do today is to go out into the distance, stand in the future and look back. You guys probably do the same. And so I'll go out 10 years and look back at what did I wish I had done then? Then do it now. That's the answer. And so there are a couple of industries we really believe we can make a contribution to. One of them is healthcare and drug discovery. This is a problem that is computationally, numerically insanely complex. The number of combinations is beyond the number of atoms in the universe. It's a very large problem space. And we finally have the necessary tools to maybe chip away at that.

• 41:34Speaker 00

  And at the very minimum, we now have the ability to understand the language of, and now potentially the meaning of amino acids and sequences and proteins and chemicals and such. And so if you can understand the structure, you can understand the language, you can understand the meaning of the problem space, you might have a chance of solving it. And so I think, one, we're very excited about that. I'm really, really hoping that we go create a foundation model for multi-physics, for climate science, and so that we can ask it questions. If these human factors and these human drivers and we make these impact, what would happen to the Earth 10 years, 20, 30 years from now? It's an insanely complicated problem.

• 42:19Speaker 00

  Computationally, people have estimated it's anywhere from a billion to 10 billion to 100 billion times more computation than the fastest supercomputer on the planet today. That basically says we'll never get there. On the other hand, with artificial intelligence, we might have a real chance of reducing that computation by a billion times, 10 billion times. So I'm hoping that we have the opportunity in our generation to make a huge contribution to these two areas. So we're doing it. Earth Two is our climate science system. And Clara is our medical and healthcare system to understand better how to contribute in that space. Very exciting.

• 42:58Sarah Guo?

  I have one last question that's as important as attacking the most computationally intensive, largest search spaces in the world to save humanity and the Earth. From our audience, where do the leather jackets come from?

• 43:11Speaker 00

  My wife.

• 43:12Sarah Guo?

  Okay. So you don't know? We have to ask your wife.

• 43:14Speaker 00

  I have no idea.

• 43:15Sarah Guo?

  Amazing. It remains a mystery.

• 43:17Speaker 00

  My wife, my daughter, they're always hunting for jackets for me. Most of the jackets, I have to admit, that I have hanging up are too fashion-forward for me to carry out. And so these are the more modest ones. But some of them are just, you have to actually be cool to wear them. And so I just don't want to look out of place. Synthel Foundation models are about. Whoever eats this should not wear those jackets.

• 43:45Sarah Guo?

  Well, thank you so much for doing this, Jensen. It's been an inspiring conversation.

• 43:48Speaker 00

  Thank you. Really enjoy it. Keep up the good work.

• 43:52Sarah Guo?

  Thank you for listening to this week's episode of No Priors.

• 43:56Speaker 02

  Follow No Priors for new guests each week and let us know online what you think and who in AI you want to hear from.

• 44:01Sarah Guo?

  You can keep in touch with me and Conviction by following at Serenormous.

• 44:05Speaker 02

  You can follow me on Twitter at AlodGill. Thanks for listening.

• 44:10Sarah Guo?

  No Priors is produced in partnership with Podpeople. Special thanks to our team, Synthel Galdea and Pranav Ruddy, and the production team at Podpeople. Alex McManus, Matt Saab, Amy Machado, Ashton Carter, Danielle Roth, Carter Wogan, and Billy Libby. Also, our parents, our children, the Academy, and tyranny.ml, just your average friendly AGI world government.