Turing Award Winner: TPU vs GPU vs CPU, Computer Architecture, RISC vs CISC | David Patterson

David Patterson is a Turing Award winner famous for his contributions to computer architecture. I interviewed him about his past work, thoughts on GPU/TPUs and career advice from half a century of experience. • My ergonomic keyboard project I mentioned, you can follow along here: https://read.compose.llc/ • The Kickstarter page for it: https://www.kickstarter.com/projects/ryanlpeterman/compose-simple-ergonomics-beautifully-done Podcast links: • YouTube: https://youtu.be/Pn4ZwlEh5nw • Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript: https://www.developing.dev/p/turing-award-winner-tpu-vs-gpu-vs?r=n49ky Timestamps: (00:00) Intro (00:42) RISC vs CISC (12:51) Compilers (17:38) GPUs (23:07) GPU vs TPU vs CPU (32:12) Is Moores law dead? (38:04) GPU benchmarks (41:40) How to have a bad career (49:59) Courage and optimism (55:56) Advice for his younger self (58:15) Outro Where to find David: • Wikipedia: https://en.wikipedia.org/wiki/David_Patterson_(computer_scientist) Where to find Ryan: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman Referenced in this episode: • AlexNet paper: https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf • David Patterson's “How to Have a Bad Career” talk: https://www.youtube.com/watch?v=Rn1w4MRHIhc • “Life Lessons from the First Half-Century of My Career”: https://cacm.acm.org/opinion/life-lessons-from-the-first-half-century-of-my-career/ • The 7 Habits of Highly Effective People (book): https://en.wikipedia.org/wiki/The_7_Habits_of_Highly_Effective_People • Working (book): https://en.wikipedia.org/wiki/Working_(Terkel_book)

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Well, if Moore's law puts a lot more transistors in a chip, how come they're not getting hotter and hotter?

• 0:05Ryan Peterman?

  This is David Patterson, Turing Award winner, famous for his contributions to computer architecture. And we discuss the historic RISC versus CISC debate.

• 0:14Speaker 00

  And he said, and CISC won. Well, that's a pretty myopic view. Everything is energy bound now. Registry allocation by the compilers was so poor.

• 0:24Ryan Peterman?

  And his thoughts on modern GPU and TPU architecture. What's the high level differences between a CPU, a GPU and a TPU?

• 0:33Speaker 00

  What they realized for machine learning and AI.

• 0:36Ryan Peterman?

  Here's the full episode.

• 0:42Ryan Peterman?

  Maybe you could explain RISC versus CISC? Kind of like what that debate was and why was it controversial?

• 0:49Speaker 00

  What happened is in kind of in the, you know, the microprocessor was invented in the 1970s. But in the beginning, it was basically a toy. It'd be something that would go in microwaves and things like that. Those of us who believed in Moore's law, believed eventually the microprocessor would be the way we did all our computing with the W of transistors every year or two. Eventually, there'd be enough transistors that on one of these microprocessors would be a serious computer.

• 1:18Speaker 00

  So the people who are designing microprocessors like Intel and Texas Instruments weren't really computer architects. So they just imitated what the big companies did. So companies, leading companies like IBM, or at the time, Digital Equipment Corporation, IBM for mainframes, Digital Equipment for what were called mini computers. They kind of drove the architecture, the instruction set design. So what they were doing with Moore's law was building more and more sophisticated instructions. They had, you know, mini computers would be the size of a refrigerator or a couple of refrigerators. And mainframes would be the size of many of those. So that's what they were doing with more hundred sources.

• 2:01Speaker 00

  And the philosophy at the time was that by having a more sophisticated instruction set, you kind of raised the level of abstraction closer to the software. So that had some inherent benefits over having something at a lower level. Now, from some of us who were in that area, like my friend John Hennessy at Stanford and I, we thought that wasn't necessarily the right thing to do. Compilers would go from programming languages down to the instruction sets. So why couldn't compilers hold that? And then the question is, what's the right instruction set for this emerging microprocessor? And the prevailing wisdom of these more sophisticated instruction sets,

• 2:43Speaker 00

  complex instruction sets, and if we think of it like a vocabulary, it was like having lots of polysyllabic words in your vocabulary. And the alternative was going to be, what we call the reduced instruction set computer, was having lots of reduced instructions, like monosyllabic words. So you might guess that for a program to execute these instructions, if they're simpler, they'd take more of them. And if they're sophisticated, they take fewer. But the sophisticated ones might take longer to run. So kind of the question came down to what was that ratio? And in the beginning, in the 1980s, when these debates were happening, there was this kind of very vociferous debates, partly of like what those ratios are going to do.

• 3:28Speaker 00

  But the big part was kind of philosophical. Weren't you doing damage to the software industry by lowering the instruction set level? So the gap between the programming languages and the instruction set was larger. That was kind of the ferocity of the debates. Well, after the dust settled a few years later and we started getting numbers, it turned out you needed like maybe 30%, 40% more simple instructions to execute in a program. But you could run them four or five times faster. So the net was, or five times faster. So three or four X speed up was the potential for risk over CISC.

• 4:05Ryan Peterman?

  You mentioned that, I guess, philosophical difference, like that gap. Why would that be controversial?

• 4:12Speaker 00

  I mean, unfortunately, I'd say computer architecture in the 1970s and even 1980s, a lot of it was kind of, people designed this by using their intuition or using their gut, their feelings was this is the right things to do. And even the textbooks of the time were like catalogs. They, here is a computer and just list all its features. And here's another computer listing all its features. So it was pretty. Dissatisfying what was going on because these debates, it would seem like we ought to be able to decide this scientifically with with numbers that are on there. But absence of that, it was more like a philosophical debate, like how many angels

• 4:53Speaker 00

  on the head of a pin, right? You could you could have arguments qualitatively, but you couldn't settle the arguments. And because there wasn't ways to settle the arguments quantitatively, you know, people would just argue about it.

• 5:06Ryan Peterman?

  Would you say that, you know, RISC versus CISC, did one side win the war?

• 5:13Speaker 00

  Yeah, I was just reading there's some guy online who revisited it and he said, and CISC won. Well, that's a pretty myopic view is in the PC era, because of the importance of distributing software in binary. So once the x86 was established and people in the PCs would ship software in binaries, that was very hard to overcome. That was a huge impediment to changing the instruction set. So PCs are designed defined by the x86 architecture largely.

• 5:46Speaker 00

  But about also in the 1980s, there's this company in England that wanted to do a personal computer. They called it was the Acorn Personal Computer and they decided they wanted to have the they needed their own instruction set of architecture to do that, their own chip, they were unsatisfied. The chips that he had available at the time they were doing it weren't fast enough. And they were influenced by the papers that we did at Berkeley. And so they built what they called the Acorn RISC machine. And then one of the benefits of this kind of reduced instruction set is that it could be simpler, it would take less resources and take less energy to execute. And then several years later, when Apple was looking for a microprocessor that could power, you know, one of their personal devices that this one

• 6:31Speaker 00

  was called the early forerunner, the iPhone called the Newton, they came to this company and said, boy, I really like it. Let's get rid of the Acorn name, so they renamed it the Acorn RISC machine ARM, and they rechristened it the Advanced RISC machine to get rid of the Acorn. And then Apple used it in the Newton. Now, the Newton wasn't a commercial success, but it demonstrated the benefits for RISC architecture for mobile devices. So Nokia came along just a few years later with their GDM, Sell Your Phone, which is one of the first popular ones, and they embraced ARM. And so ever since, ARM has dominated all the mobile devices.

• 7:14Speaker 00

  So I think I just checked, there's been 350 billion ARM processors, microprocessors with ARM technology in it today. So it's like today, 99% of all processors in computers are RISC. Even in personal computers, Apple switched over to ARM from the x86 architecture. So even PCs, their RISC architecture is significant. And it's starting to get in the cloud. The cloud's largely been defined by the x86 server architectures. But Amazon, and I think Amazon, Microsoft, and Google all have their own development of ARM processors.

• 7:58Speaker 00

  So ARM is becoming, or RISC processors are becoming more popular in the cloud. So right now, I'd say the x86 architecture market is shrinking while the RISC architecture market is growing leaps and bounds.

• 8:12Ryan Peterman?

  Well, how could someone say CISC 1 when you mentioned like 99%?

• 8:17Speaker 00

  Yeah, well, that's just if they're doing something in history and they define computers as personal computers and maybe servers, and they go up through around 2000. If their story ended then, well, it looks like a CISC 1. But, you know, once we get into this post-PC era, I don't understand how somebody would reach that conclusion.

• 8:43Ryan Peterman?

  You mentioned the energy expenditure. So like, you know, RISC makes sense in places or maybe on mobile devices, things like that.

• 8:52Speaker 00

  Well, even in the cloud, I mean, everybody cares about, everything is energy bound now. So if, you know, these days, of course, you're not dealing with millions of transistors, but billions of transistors. So it matters somewhat less today. There's so many things going on that you can hide that. I mean, even what the x86 architecture did is to compete is that it translated the x86 instructions in hardware into RISC instructions. And so you had to pay that extra overhead of that translation step to get RISC instructions. And then you could any good ideas that the RISC people had that the x86 could do.

• 9:33Speaker 00

  But it was worth it financially for that extra overhead because of the value of the PC software base. So it made a lot of sense for Intel to do that. And they did that in the early 2000s. You know, it was a great commercial idea.

• 9:49Ryan Peterman?

  Is there any niche use case where CISC makes sense? Is there any, you know, is this an engineering trade off or CISC is objectively worse?

• 9:57Speaker 00

  Yeah, so if you want to, if we want to go kind of one level deeper into all of this, the, what was actually going on in, you know, in designing computers, the hard part is the control. And so what happened is in the beginning, it was kind of, control was kind of ad hoc. You would figure out, you put the gates together to make it to work. A, one of the computing pioneers, Maurice Welch, figured out a more elegant way to design control. And he said, well, we could just list all the control signals as the output of a memory and we could have something would keep track of where we were in the memory and to issue those control signals.

• 10:40Speaker 00

  And he called this, he called the, this effort, the instruction, the, the control signals, you could think of instructions. So he called that a micro instruction. And he called the programming of those instructions, micro programming. So where technology was in the 1960s, that made a fair amount of sense. And so IBM built the so-called micro program computers. So it was basically an interpreter with very simple instructions that would interpret this much more sophisticated instruction set above it. But you would play this interpretation overhead. And classically, you know, in computer science, interpreting versus compiling is something like a factor of five or 10.

• 11:23Speaker 00

  But given the latencies of the memory technologies and the possibility of doing this out of read-only memory, it made sense up into the 60s and 70s. But then the question came up, this came around 1980, is like, is this still a good idea? Should we have this microcode interpreter inside there? And so the alternative is to think of, well, rather than we've got this microcode interpreter in there, why don't we just compile directly into those instructions? And that's pretty close to the RISC ideas. The microinstructions themselves used to be like 100 bits wide and really complicated. So if you make them not quite so long, you make them kind of more natural, still we could

• 12:08Speaker 00

  skip the interpretation step. So with going this level deeper, kind of the question would be today, would people invent an instruction set that was so sophisticated it needed a microcode interpreter? And probably they wouldn't do that. I mean, you could do it. Nothing prevents you from doing it, but you wouldn't want to design an instruction set that was forced to use a microcode interpreter. It might make sense in some very tiny applications, possibly where you have tens of thousands of transistors. Maybe this would work, but I don't think anybody's invented an instruction set in the last 20 years that has anything that would need a microcoded interpreter.

• 12:51Ryan Peterman?

  You mentioned the compiler multiple times here, and it seems like that's a critical piece that kind of makes RISC work so well. Can you explain the role of the compiler and how it manages the relationship between software and hardware?

• 13:06Speaker 00

  Well, you're writing in a programming language like C or C++ or maybe Python. But the quality of the code that gets generated is up to the compiler. If you had a specific example is that it's useful when you construct computers to have registers, and those registers are actually kind of visible in the assembly language or the machine language programming. There's, it can be 8, 16, or 32 of these registers. For the people programming at a low level to use. Well, it used to be very difficult for compilers to allocate registers efficiently. The computers just weren't fast enough. We didn't have the algorithms that we could look at a section of code or a subroutine

• 13:51Speaker 00

  or something and say, how can we most efficiently do registers? In fact, the C programming language, which was invented to do systems programming in C, before that, to write an operating system, people wrote them in assembly language, believe it or not. But Unix people showed, Ken Thompson, Dennis Ritchie showed that if we had a low-level language, pretty low-level language, we'd get the benefits of writing in something that's much easier for humans to understand and debug. Because register allocation by the compilers was so poor, they had to put in the ability for the programmer to give hints of what variables should go on registers.

• 14:32Speaker 00

  It was just easier for them to have the programmer step in and say, if the machine had eight registers, I want these six variables to be in these registers. Don't leave them in memory because it runs so much slower. Registers are so much faster. So, a big part of the RISC argument was the compiler algorithms were getting better. They could handle these low-level instructions. They could allocate registers efficiently. And that was another reason in these debates about why it made sense to have a simpler architecture. What we did in the RISC architectures, well, if registers are really important, register allocation is really important, one way to make it easier for the compiler was just to have a lot more of them.

• 15:15Speaker 00

  So typically, the RISC architectures at times would have eight or maybe 16 registers. So we put in 32, and that was one of the arguments, right? Well, if it's hard to efficiently use a small number, let's give them plenty. So... Even if it wasn't that good, there'll be enough registers so most of the time. And registers weren't that much more expensive to include in machines because of Moore's Law.

• 15:38Ryan Peterman?

  I saw somewhere in one of the talks that you'd given that somehow the compiler, it's more easy for it to optimize the code for RISC, but in CISC there were these complex, bigger ones, and it almost never used them. Is that a shortcoming of the compiler?

• 15:56Speaker 00

  So what happened, if we go back in the compiler days, there's this argument is that by having more sophisticated instructions, this would raise the level of abstraction. There'll be a smaller gap and they'll make it easier to compiler. But that was a philosophical argument. It wasn't something that necessarily compiler people could work. Compiler people weren't making that argument. It was the architects who were making that argument. And as it turned out, when we looked at the programs when we were doing the early research on CISC versus RISC, is the compilers didn't really use those instructions. Often the compiler writers, the architects would come up with a sophisticated instruction, and the compiler writers would say, well, we don't need that. In fact, we found examples

• 16:39Speaker 00

  where like for a procedure entry, there'd be a special instruction built in to do all this work that what the architects thought the compilers run it. And the compiler designed and said, well, we don't need that. It's faster. It's actually faster for you to use it with separate instructions than use your sophisticated instruction. And so we found a bunch of examples like that. So you pay the extra overhead of the microcode interpreter to have these sophisticated instructions, and then the compiler doesn't even use them. Right? So this is kind of a nonsensical situation that we're in. And, you know, this is kind of like, you know, why are there startups or why are there scientific, you know, breaking points like that? As we looked at all the technologies with Moore's Law,

• 17:23Speaker 00

  you know, ideas like caches, where compilers were in using sophisticated instructions, the ability to allocate registers more efficiently, it made sense to change the directions away from these microcoded instruction set architectures to these simpler architectures.

• 17:38Ryan Peterman?

  When we talk about instruction sets, I mean, we're kind of assuming that we're talking about these general-purpose computer, these CPUs. And I know GPUs are kind of talked about a lot, and maybe there's other forms of computing. Are there instruction sets for those types of machines as well?

• 17:55Speaker 00

  So what happened is around 2000 is when GPUs came along, and these were what we called domain-specific architectures. So a GPU is a graphics processing unit. It had one job. It didn't need to do everything the general-purpose processor needs to. Didn't have to support virtual memory. It didn't have to support a compiler, which is, you know, a pretty radical idea for architectures. It was just for graphics. And so around 2000 is when Nvidia and the other GPUs out there, because they were trying to do, you know, give you the graphics for games, to give you the graphics for movies. So it was a niche product. So what happened kind of in the computer industry is it was driven by Moore's Law,

• 18:40Speaker 00

  which I've mentioned many times, and also this lesser-known law called Denard scaling, which is kind of an interesting question. Well, if Moore's Law puts a lot more transistors in a chip, how come they're not getting hotter and hotter as we double the number of transistors every year? And the reason was this observation made by Bob Denard, that what, as you added more transistors, people would also lower the threshold voltage, which is the distinction between a zero and one, and that had like a squared effect. So you would end up doubling them in transistors, but you'd lower the threshold voltage. So microprocessors stayed at like 20 or 30 watts.

• 19:21Speaker 00

  In fact, we did a, John and I eventually did a textbook, and I was just checking, it came out in 1990. We didn't even talk about power as an issue for the first three editions. You know, the third one came out in 2000. Power wasn't even a topic because micro, Denard scaling was going on, and so microprocessors just stayed in the tens of watts even as they got faster and faster. What happened is about 2005 or so, Denard scaling stopped working. And that was a shock. All right, so Intel actually had a microprocessor that failed, one of their generations, because they just couldn't get the power down enough. It was too hot to be able to do that.

• 20:02Speaker 00

  And then, so then what that forced us to go to multi-core is that before that, it was easiest for the programmer for everybody if there was one very sophisticated processor that did everything, but we couldn't do that anymore. So it went from one sophisticated processor to two and then four and then eight simpler processors. Then it was up to the programmer to deliver on the potential of Moore's Law by paralyzing their code. So we stayed that way for about another 10 years and then Moore's Law started slowing down. So the general purpose microprocessor was barely improving. It got a little better, but it wasn't getting dramatically better.

• 20:44Speaker 00

  In the, you know, in the 1980s, 1990s, 2000s, you have a laptop and your friend's laptop would be like four times as fast as yours and, you know, you were jealous. So you would throw away perfectly good hardware because your friend's thing was so much faster because of the rapid change in performance. Well, that all ended in the 2010s where you wouldn't throw away a laptop because the new ones were hardly that much faster than the old ones. You'd throw them away when they break and slow down. Nevertheless, programmers were used to this dramatic improvement in performance every few years because they could add more features to their software and stuff like that. So what were architects going to do?

• 21:27Speaker 00

  They'd already done the multi-core trick in 2005 or so. So in around 2015, the idea was we would do domain-specific architectures like the GPUs. So if you tell me I only have to run a narrow class of programs and I don't have to necessarily run all the operating systems, everything else, well, yes, I could shuffle those resources and do something much more efficient for something well. So you could do some things well and there's other things either you do poorly or not even at all. So then the question is, okay, what domain? So just coincidentally, where we were technologically, right around 2012, 2015 is when machine learning AI burst on the scene.

• 22:12Speaker 00

  And so that is evident, what domain should we do? And the domain was machine learning AI. So just coincidentally, where we were technologically, this new domain came along. And then I would say, you know, I work for Google, but I think it's fair to say Google was the one who really saw, it was certainly the first big company who understood the potential of machine learning AI and bet that they, or feared that they were going to be swamped with demand and needed to do custom hardware. So the tensor processing unit, GPU, that Google debuted in 2016 really kind of shocked

• 22:56Speaker 00

  the world and got people to realize that we should be designing hardware for machine learning. And we could continue to improve performance dramatically for that one domain.

• 23:07Ryan Peterman?

  What's the high level differences between a CPU, a GPU, and a TPU?

• 23:13Speaker 00

  The CPU has got to be this general purpose thing. And basically, even today, they have these general purpose cores that each core is very similar to what the instruction sets are, used to be like 20 years before that. It's not a surprising design and it's just got lots of cores. And the modern ones today could have 50 or 100 cores in there. So that's, that's the feature. For graphics, what they decided to do, which to do the graphics job is they need to get a lot of performance of their memory system. So they went to multi-threading. So they have hardware threads that you would send a memory request out and the hardware would switch over to do something else while the memory is coming out.

• 23:57Speaker 00

  So it's this highly threaded architecture and that's what they developed and could run well for graphics.

• 24:05Speaker 00

  And graphics, it turns out, doesn't need various, doesn't need a powerful floating point, it needs, doesn't need wide floating point. So 32-bit floating point is plenty for graphics, even 16-bit. So they were doing, pushing graphics with 16- and 32-bit floating point in this multi-threaded kind of architecture. And because it was kind of its own unique thing, it has its own set of terminology all to itself. It wasn't kind of out of the main branch of computer architecture. So when you look at our textbooks, we have kind of like a Rosetta Stone, but says, here's the terms that Nvidia uses to describe GPUs. This is what it means in kind of normal, well, normal, in the mainstream processor

• 24:50Speaker 00

  design. So they were this niche product that happened to do pretty fast, single precision floating point and even half precision floating point. And they were pretty cheap. They were like hundreds of dollars. So when people, so kind of, but some people were thinking, boy, for some applications, if I could turn my program into like, pretend that it was doing, creating an image, if I could turn my problem into image generation, I could use these pretty cheap GPUs, which had very good floating point performance per dollar compared to anything else. And

• 25:31Speaker 00

  so people started playing around with that. And the founder, CEO, Jensen Huang, really liked that idea. So in 2006, he funded an effort to create a programming language that, that would kind of handle this multi-threaded hardware architecture that's for graphics to make it easier to program. And so that led to the invention of CUDA, which I can't remember its acronym. But it's really a proprietary programming language for the multi-threaded GPU architecture.

• 26:05Speaker 00

  To make it kind of easier to program. It was a C-like language, but you can't just compile C programs and run it. But it was C-like. But people actually liked it. It was certainly much better than trying to turn your program into an image generation. But people liked it. But Jensen Huang, his vision was try to get these teenagers in the basement who are playing the game to learn how to program these things. And he had a couple of markets that he was interested in, like, you know, some of the Department of Energy labs, like Fluid Flow and some of the things like, you know, some... And he would build special-purpose libraries that could handle that domain, and then use his GPUs to do that kind of special-purpose computing, but breaking out of graphics.

• 26:49Speaker 00

  But that's kind of the heritage there. Now, for the TPU program, and so what's happened, people started, right from the very beginning, people started using GPUs because they had much better single-p precision floating-point performance than the CPUs. They had a lot more processors on them, so they had the potentially much more performance. So they're kind of one of the breakthrough movements, moments in, it was in 2012, when within the machine learning community, this neural networking piece, which had a few advocates, but a lot of people didn't believe in it, they went into a competition to see who could do the best image recognition. And in 2012, the so-called AlexNet beat all the competitions.

• 27:33Speaker 00

  This was this historic moment in the, in the, in machine learning and neural networking. And once they did it, and the guy who did that had taken a coup d'oeuvres at the University of Toronto, learned how to use it. And so he says, "Well, as long as I'm doing this, I'll do it on a GPU." And so he was able to explore a lot more space on this cheap, fast GPU. So he entered it in the competition, 2012, and within, he was the only one using neural networks, and he crushed the competition. And within a few years, everybody switched over, and not only they were doing neural networks, but they're using GPUs. So GPUs have this heritage of the graphics, in graphics engine, but more programmable,

• 28:17Speaker 00

  and started getting used for machine learning.

• 28:21Speaker 00

  When Google came along, they decided, you know, it was a clean slate. They didn't, they didn't care about graphics. So the, at the heart of neural networking is a matrix multiply. That's the, that's the thing. So they designed a processor for at the time, microprocessor at the time, had a giant matrix multiplying unit. That was the main thing. And then they threw a bunch of stuff out that they didn't need. So a lot of general purpose computing is, what's on the chip is maybe three levels of caches to try and have, so you don't spend all your time going to the relatively slow memory. Well, for machine learning, they knew when their memory accesses were so they could schedule that. So they didn't, a hardware cache didn't make any sense. They would just have a memory, the software understood and would transfer in time.

• 29:06Speaker 00

  So those were some of the innovations. Also, they innovated on the floating point format. You didn't need, you know, hot scientific computing, cares a lot about precision. They have, you know, most of it's done in 64 bit floating point where the exponent is, you know, less than 10 bits and most of it is, is the precision, you know, the fraction can be 50 some bits. Well, they, what they realized for machine learning and AI, they don't need all that precision. They needed the range. So Google did the first floating point format where the exponent was bigger than the fraction. That's, that was a radical idea, that so-called BrainFloat 16, the part of Google that was doing this was the brain research group.

• 29:48Speaker 00

  So they brought out this architecture, they had this narrow floating point format, they had big matrix multiplying unit. It only had one, one processor in it, unlike the other ones. It was just dedicated for machine learning and it just kind of blew the doors off of everybody in the field. It was, you know, like 30 times better at inference than the contemporary GPU and 80 times better than CPU by having this dedicated one. So when Google made this announcement at their annual retreat a year or so after.

• 30:22Speaker 00

  After they had deployed it inside, it just shook everybody up. Intel started buying companies. You know, Nvidia started modifying the design for to be much more enhanced machine learning. And then a bunch of other competitors started hyperscalers started doing their own effort. So I think that was the watershed. I think the TPU announcement was the watershed moment there.

• 30:48Ryan Peterman?

  So it's almost like levels of specialization, like the CPU is the most general.

• 30:53Speaker 00

  Yeah, if we could still, you know, if we still had the Nard scaling, if you know, Moore's law and the Nard scaling, where would we be today with general purpose processors? We should have 100 terahertz, you know, microprocessors. If we could build 100 terahertz microprocessors, that's what we'd do. We would, GPUs would still be a niche, you know, if we could do that, it raises all boats. That would be fantastic if we could do that. But that that's long in the history. We haven't been able to do that for 20 years. And so, you know, there were probably two or three gigahertz microprocessors in 2005. And that's kind of where they barely improve today. So we can't do that. So there's still areas where it's important to have CPUs.

• 31:38Speaker 00

  They're the right, you know, you need operating systems, compilers and things. They're the right solution, but things have been specialized. And then now in terms of industry terms, huge emphasis is being put into the AI accelerators or just AI in general is where the money is being invested. And so the question for any processor designers was how does my processor design fit into this AI universe? And what should I do? How should I change it for this important application?

• 32:12Ryan Peterman?

  So you mentioned that kind of Moore's law is slowing down. And I watched this other talk from Jim Keller saying that

• 32:20Speaker 00

  Oh, yeah.

• 32:20Ryan Peterman?

  Moore's law isn't dead, but is it controversial to say that it's

• 32:24Speaker 00

  No, I mean, not not to real engineers. The Moore's law is very simple. It says in a in a chip, the number of transistors will double originally said every year. And then he meant it. To every two years. Just, just look, do the chips, are the number of transistors doubled? And no. Now, I think what people assume that means if we are no longer on Moore's Law, that technology is not improving. That's not the same thing. It's not improving at the rate that Moore projected. And what was amazing about Moore's Law, which lasted 50 years, is that it guided the investment of semiconductor manufacturing. We need to deliver on doubling

• 33:07Speaker 00

  transitions every year or two. How are we going to do that? How are we going to build the equipment? So it was a guideline for the whole industry, which was kind of remarkable. So what we are doing now is there's pieces of the technology to get better and there's pieces that don't improve at all. So one of the big pieces, important pieces on a chip is the static RAM, SRAM. So that's hardly improving at all. Logic gates still continue to improve. They are, the gates are getting better. So if you're doing like adders or multipliers, those are getting there. So pieces, it's not uniform improvement anymore. And we're also going

• 33:48Speaker 00

  to more exotic packaging to be able to deliver it. So people not, you know, forever it was, a single chip was the best way to package. Everything fits on one chip and that's what we're going to build. And now there's these ideas of chiplets or packaging multiple chips together, the latest GPUs, and I think the latest TPUs has actually two, what are called full reticle design dies, packaged, put into a package. And so a full reticle design is the maximum you can build on a semiconductor. They have a step and repeat motor and there's, you know, in the old days, you'd have many chips inside that. Now there's just one. So

• 34:32Speaker 00

  two maximum reticle designs form a node. So they're using packaging. So if you look from outside, you can say, well, there's, look, there's, look how many transistors are, it's, quote, Moore's Law is continuing. But if you see what's inside the chip, you know, that's, that has tapered off. And I think part of it is for a fair amount of the industry. If you were in a semiconductor manufacturer and people ask you what you do is I make Moore's Law. I sustain Moore's Law, what I do. And if you've been doing that for decades and somebody says Moore's Law is over, it's like it, my career is over. So I think there's an emotional side of it. But, you know, just look at the data. The data doesn't back up what Keller says. But I also didn't say that the technology is not improving.

• 35:18Speaker 00

  The technology is continuing to improve and specifically domain-specific architectures. But if you look at the general-purpose architectures or even other memory technologies, you know, it used to be the DRAMs would improve by a factor of four in density every three years like clockwork. And now it's maybe 10 years between factors for it. So you can see plenty of evidence that Moore's Law no longer applies.

• 35:41Ryan Peterman?

  Is there some new version or some analog to Moore's Law that is kind of guiding, you know, microprocessor design and architecture these days?

• 35:54Speaker 00

  I guess a question is, both Nvidia and Google are continuing to deliver much faster processors for machine learning, tremendously better. What are they doing? Well, part of it is what I said about packaging, you know, to getting, to be able to get more transistors and you keep them closer together because distance matters. Part of it is innovating on the floating-point formats. So, you know, unlike the supercomputers of 64-bit, it's not even 64-bit, not even 32-bit, not even 16-bit, but 8-bit and 4-bit floating-point is going on. So narrowing of the data types.

• 36:36Speaker 00

  The, you know, having kind of what's called the matrix multiply instructions. So these powerful units that are put in there that can, that can do special-purpose applications that are very important, getting, making those bigger and faster. So those are the things that are going on. But we don't have this simplifying guideline kind of underlying all this. You have to be aware of each piece of the technology, gauge how fast it's improving, whether it's they can deliver on what's going on. And then you assemble that together and make your bets. You know, there's a paper that we've just got approved that I think we're going to put in archives and.

• 37:21Speaker 00

  Which is talking about the Google TPU line. And pretty remarkably, the Google TPU line, particularly for training, has stayed pretty constant in the basic architecture design. Things have gotten bigger and faster. But if you look at the the design of it, going back to the first training TPU, it's that block diagram still works all these, you know, a decade later. So, you know, the people who designed that in whatever 2015 or two did a really great job. But the main components of a big matrix multiply unit, the so-called high bandwidth memory,

• 38:01Speaker 00

  a vector unit to go with the matrix unit, and the basic building blocks are still there.

• 38:07Ryan Peterman?

  I think in your career with the CPUs benchmarks were huge. They played a huge role in in measuring which computer architectures were performant and not. In this this new space of floating point operations for AI, is there a benchmark that people use for GPUs? And can you also apply it for TPUs?

• 38:27Speaker 00

  I was, you know, I and some friends were helped involved in called the ML perf effort. So it was inspired by the the spec CPU effort was the spec benchmarks that there are these competing companies and they'd all make claims about theirs was better than the others. And they realized that wasn't good for the industry. So they agreed on a set of benchmarks. So the ML perf effort is this run now, but what's called the ML comms is an attempt to do that. Is if we're going to do this comparisons, let's let's let's not argue about what the benchmarks are. So that's a serious effort in that. Kind of interestingly, what's happened, I'd say, is because

• 39:10Speaker 00

  there's two pieces to the design. There's the the machine learning libraries that are to implement a lot of features that you need for for these applications. And the libraries will even be rewritten for specific applications rather than just having general libraries. This has turned out to be A big advantage for Nvidia because they have a large corporation with lots of people that are available, lots of engineers who could build these libraries for them. So it's turned out

• 39:43Speaker 00

  not quite a, it's not a, it's not a neutral evaluation, right? It's, it's the architecture plus the libraries that go with it and the compiler too. But specifically the libraries that you tailor each time. So Nvidia brings out, when they announced the new architecture, they create a new set of, they modify the libraries to run that really well or to run applications really well. So that's a powerful combination. Kind of in business terms, people refer to Nvidia as having this CUDA mote and part of it is CUDA, the programming language. But a big part of it is the libraries that Nvidia, Nvidia makes.

• 40:25Speaker 00

  So this has made it difficult for startups to be able to compete with Nvidia. Partly because, you know, they didn't put enough emphasis in the software. And partly because, you know, Nvidia just has many more engineers than they do to be able to tailor the libraries. So it's the libraries plus the architecture. There's this powerful advantage why most people do things on GPUs. Now, Google has been able to develop their own libraries. They don't have as many engineers. They use compilers more than I think Nvidia does. So they, they have a set of libraries that they can do. But up until recently, Google has, it's all been internal.

• 41:08Speaker 00

  It's just for Google to use or you can use it being the cloud.

• 41:13Speaker 00

  But these startups have, you know, have to have a difficulty. As a result, that's why the ML perf hasn't been as popular. Not everybody runs them because Nvidia runs them really well, really better than everybody else. And the startups have a hard time showing off what they can do. Given they don't have the engineering effort going into libraries that you need to do well in ML perf.

• 41:40Ryan Peterman?

  You gave this popular talk about how to have a bad career and it's kind of the negation of advice to kind of have a good career. And I was just wondering if you could kind of summarize maybe the the top three things that you kind of think people should take away.

• 41:58Speaker 00

  Yeah, so, yeah, well, there's been a few things. So the story is, early in my career, my friend, a good friend and I said, how can we teach graduate students how to give a good talk? And we thought it'd be funny to explain how to give a bad talk. And then if you didn't want to give a bad talk, this is, so this is how to do it badly. And if you don't, here's the thing to do not to do it badly. And so then I later did a how to have a bad career. That's pretty much focused towards academia, you know, so for researchers to be able to do that. I later did a how to have a bad, how to build a bad research center, how to build a bad research lab. And then recently I've been giving how to give AI a bad carbon footprint. That's my latest one. But I think the thing that might be more relevant is, at the end of my talks, I would,

• 42:43Speaker 00

  I would kind of reflect on my career and talk about lessons learned. So I've written a paper called, I think it's lessons, life lessons from the first half century of my career. I wrote that recently. And so those, those are divided up into kind of career advice and personal advice there. On the, the personal side, I'd say, if you have a family, you know, make sure your family's first, keep your family first. The technology we've invented makes it really easy for your, you know, to take your work home with you and not pay attention to your family. I had a, when I was first here at Berkeley, I gave a senior faculty member a ride home, dropped him off late at night coming up Sillon Valley.

• 43:26Speaker 00

  And he said, well, Dave, if I had to do it all over again. I wish I'd spent more time with the family and I never wanted to say that. And nobody on the deathbed says, you know, I wish I'd spent more time in the office. So, so you gotta, you know, whenever, it's easy to kinda, you know, let the, let your career overcome the needs of your family, but you just, you gotta just remember that. I would say in my life, kind of growing up in the 50s, you know, the idea was to be happy, you had to be wealthy. But those are actually two different goals, wealthy and happiness. So I always made decisions towards happiness versus wealth. And I felt very good about that.

• 44:09Speaker 00

  And even now, like, why would I pick, why would I pick something that made me wealthy and unhappy? Why would you do that? And so, and in the field that we're in, it turned out there was a lot of wealth to go with it. So, you know, optimizing happiness didn't mean you had to suffer beyond that. I think it's important to have fun personally. I think when you're a kid, you don't have to tell kids to play. But as you get dealt, you get so busy, you, you don't think you have time to have fun. But, you know, you only get to do this once. So, so I, right now, I play soccer. I run my bicycle to the interview. I lift weights, I body surf, I, you know, do things with my wife and family, my sons.

• 44:53Speaker 00

  So it's important to have fun. I think on the career side is one of the pieces of advice is by Akante. He wrote this book, The Habits of Very Effective People, I think. And he has a little quadrant and he divides it of urgent and not urgent and important and not important. And kind of, there's so many things in our technology like email and texting for you to focus on the urgent things, but you really shouldn't be spending a lot of time on the unimportant urgent things. And it takes self-discipline to set aside time for the important, non-urgent things.

• 45:37Speaker 00

  But if you don't block that out, you can just not have time to do anything. I get to see other people's calendars at Google and there's managers that every half hour from eight to six, five days a week are scheduled. And I don't see how you have time to think and reflect on things like that.

• 45:56Speaker 00

  Other career advice things is I remember I kind of woke up one morning and it was like God spoke to me. I was like thunderstruck and it says, it's not how many things you start, it's how many things you finish. It seems relatively obvious, but you know, that's not the way I was acting. I had, they had many things going on, but after that, it's like there's one main thing I'm doing at a time. So when Hennessy and I wrote our textbook, that was the main thing. When I was department here, head here, that was the main thing I did. I would do some other little things there. And John Hennessy, he wrote a book to a kind of a, a career advice book based on his presidency. And one of the things he said in there, you know, you're only going to be remembered for the five or six things

• 46:41Speaker 00

  you've done in your life, not for the hundreds of little things. So to give yourself a chance to have some things you're really proud of, it's better to concentrate on a few of them, hoping that some of them will turn out to be a big deal, rather than scatter yourself to, to many things. But if you're interested, you can, yeah, if you look for life lessons, David Patterson, first half century, you can see the whole list of, I think there's 16 lessons altogether.

• 47:11Ryan Peterman?

  You mentioned that you didn't want to reflect back on your life and feel like you didn't have enough family time. And yeah, I've, I don't think I've ever heard anyone say the opposite. Why do you think that is? Why is it that everyone looks back on their life and they never regret, you know, a lot of family time? I imagine there's got to be at least one person that says, I spent too much time with my family.

• 47:37Speaker 00

  My career suffered?

• 47:39Ryan Peterman?

  Yeah.

• 47:41Speaker 00

  Well, you know, what's, you know, what's, it's, it's pretty philosophical. I mean, what's, what's, what's life all about? What's success, right? You have to, you have to figure out what that means for you. I mean, if you, if you have, I don't know, if you have financial goals of being a, you know, a millionaire or billionaire or something like that, and that's how you judge your life.

• 48:03Speaker 00

  Okay, good luck. You know, it's pretty hard to make it. It's pretty hard to do. But I think it's the, you know, when I was finishing my PhD, I read this book by Studs Terkel called Working, where he interviewed all these people in his careers and they'd look back and what they liked and what they didn't like, what they felt about their careers. And what I got of it, the people who worked with people, like ministers or teachers or doctors, felt really good about what they did with their careers. And the people who did more ephemeral stuff, you know, like, you know, technology things or airplanes or something like that are long gone, didn't feel as good about it. It was the people that they worked with that they really cared about. And I went out with a retired engineering dean here and he

• 48:47Speaker 00

  into a meeting who I knew and he said, you know, Dave, as I flink my care, it wasn't the projects, it was the people that I worked with that mattered. And I thought, I knew that from a long time ago. So that's, I mean, this is kind of senior persons offering advice. My, I think you're going to care more about the people you've worked with. The people you've helped will be a bigger deal. That's part of, you know, one of the other things about personal happiness is they studied happiness. Psychologists used to just study crazy people, but they started like, why are people happy? And they know what the reasons are, you know. Have a job that you like, you know, have friends and family. Helping other people, helping other people makes you happy. They know this. Having something kind of either a religious

• 49:32Speaker 00

  side of it, it doesn't have to be a formal religion, but like contact with nature, the grandeur of nature, but the kind of the list of things you need to do to be happy is well understood. And, and, you know, there are these hat, and there's, our world is filled with unhappy billionaires, right? If money was the thing that made you happy, why are these people, very wealthy people? So mad about things. So yeah, this is me passing on advice.

• 50:00Ryan Peterman?

  In one of your talks, you had, it said, what worked well for me? And it was kind of some reflections. And one of the things in there I thought was unique and interesting. You mentioned that courage was a big part of your career. And I don't hear that too often. I was curious why you say courage is so important in a career.

• 50:19Speaker 00

  Yeah, I think that's, I mean, that might be partially my personal makeup. But, you know, I was, I was, I was kind of the youngest kid in my class and kind of small. It took me a while to grow. It's by age, so I was always small, but I, my parents encouraged me to go out for wrestling. And wrestling gives you self, physical self-confidence because you, you know, spend years doing that. I did it in high school and college. And so I think partly, you know, technically, you know, having courage to do things, it kind of goes along with the advice is fortune favors the bold.

• 50:59Speaker 00

  That's this, that goes, that's advice is 2,000 years old. I mean, it's hard to figure this out. Helen Keller wrote, you know, even, even trying to play it safe, you still get caught. And so it turns out you might as well, you might fail no matter what. And if you take a big chance, that's you can succeed. If you don't take the chance, you probably won't succeed if you play it safe. So fortune favors the bold. And I think it takes courage to do that. I think also it just for me intellectually, I feel like if there's something not right, I need to stand up and confront it. And I think that kind of ironically comes from the wrestling side of my personality, where if I see something, somebody getting.

• 51:45Speaker 00

  Picked on or something like that, I'm gonna stand and try and stop it. And I feel that same responsibility intellectually if people are making bad arguments or doing something that we need to stand up and do it. And I feel good about that. The cautionary part about that is, because I guess one of my senior faculty members saw this nature of me, he said, "One of the sayings is friends come and go, but enemies accumulate." This is an old saying. So if you think about it, you kind of, people you went to high school with, friends, you kind of forget, but somebody who you really does dislike, you never forget that you dislike that person. So, standing up when it's important, but be careful when you make enemies because, you know, they're gonna stick around for a long time.

• 52:29Ryan Peterman?

  When you say something's gone bad technically, do you mean someone was incorrect?

• 52:35Speaker 00

  Yeah, when they're weak, you know, either politically or, you know, or technically, when it's, you know, it's a weak argument, right? I think of us, I think, I really like there to be a marketplace of ideas and we hone the ideas by arguing them. And so if it's a specious argument, even if a person is, you know, a leader of a company and it just doesn't make sense, I feel it's kind of the responsibility, it's better for the company if somebody stands up and points that out than to just let them get away with it. And then, and then, and, you know, it's a little bit confrontational, but, you know, as long as people all agree that, you know, this is for the greater good, we're gonna,

• 53:20Speaker 00

  we need to get the right ideas out there, and so let's argue about the ideas to see, you know, polish them to make them stronger. I think that's important in science and engineering and kind of in life, too. There's a lot of stuff going on right now in the country that is worrisome. And I've certainly stood up and wrote op-eds about things that I think are wrong and need to be corrected. And, you know, if people are afraid to do that... It's hard to be optimistic about the future if people are afraid to stand up when when there's wrongs and try and stop them.

• 54:02Ryan Peterman?

  You also mentioned optimism in the talk and you had this story, I wonder if you'd like to share.

• 54:08Speaker 00

  So I would say in engineering, you know, it's hard to know, right? But I think you need to be kind of optimistic or positive, have a positive outlook because so many things could go wrong. And then some of my story, personal story that illustrates is going back to high school when I'm 16, I'm dating this very attractive girl and I screw up my courage and ask her if we would be exclusive. At the time, the phrase we used was going steady. And she looked at me and said, and you know, she was 16, she had dated other guys and thought we were pretty young. And she's, well, Dave, you're such a nice guy, I don't know how to say no. For me, as a logical person, not a no, sounded like a yes. And so I hugged her and said, great.

• 54:52Speaker 00

  And so she, she, in her mind, she thought, well, I'll let him down gently later, but we've been married 59 years now and she, she hasn't let me down yet. So that was a case where optimism paid off.

• 55:07Ryan Peterman?

  I think everyone that hears a healthy relationship for that long, they might wonder how you did it.

• 55:13Speaker 00

  I used to tell people, you know, if you go to, if you go to weddings, the marriage vows are really great, right? But nobody can remember their wedding vows. I used to say, remember your wedding vows, but nobody remembered that. So we boiled it down to nine magic words and it's just three sentences and they start I, you, I, and you gotta say all three. And it's, I was wrong, you were right, I love you. Okay, those are the, those nine words. And this applies to both, both partners in a relationship, not just one partner. But yeah, if you can say them all and no substitutions, I was wrong, you're right, you're a jerk, you know, you can't do that. It's, if you can remember those nine words, that can help you have a long relationship like my wife and I have.

• 55:56Ryan Peterman?

  And then last question for you, like knowing everything you know now from your career, if you could go back to yourself when you had just entered the industry and give yourself advice, what would you say?

• 56:09Speaker 00

  I mean, I think when I got here, because, you know, the imposter syndrome, I was a UCLA graduate student and suddenly I'm a Berkeley professor. So that just doesn't seem like, that was very intimidating. But after a while, I just thought, well, I'm probably not going to get tenure, so I should just have a good time. So I think I already, I already had a right attitude about it. I think that first year, I think it was very stressful for my, while I was trying to handle the imposter syndrome and be a Berkeley professor. But I think after that, I handled it pretty well. I, you know, I did all the things with the kids and stuff. So there's a version of that question is like, is there anything I would

• 56:49Speaker 00

  do over again? There's one thing I would have. I was chair of the, of the architecture community, the SIG Arch, as it's called, and they have an annual conference this year. And I was, you know, this was in the 1990s, I think I was the chair. And what I wasn't aware is that at these conferences, there were men who were harassing young women at this conference. I just didn't think, you know, people like, you know, young people, people like me, nobody would do that. Only an idiot would do that. That can't possibly be happening. But it was happening. And I wish I, somebody had said something to me

• 57:29Speaker 00

  about it. And, because I would have, I would have straightened out any man doing, I would have threatened his life if he were to do that today. The only comforting thing is Sarita Abbe, who's a famous computer architect, she said she also was not aware that that was going on. It became clear later, you know, I mean, five or 10 years later, it became more clear that this was going on and there are mechanisms. But that's the one thing I wish, you know, if I could go back in time, I would have. Figured that out and I would have straightened men out who were doing that. And they wouldn't, that would have stopped, I believe that would have stopped them.

• 58:08Ryan Peterman?

  Yeah.

• 58:10Ryan Peterman?

  Thank you so much for your time today. I really appreciate it.

• 58:12Speaker 00

  All right. Thanks for the interview.

• 58:15Ryan Peterman?

  Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Liskoff, Mike Stonebreaker, Mark Booker. These were all people that I brought on because someone left a comment. On another note, aside from the podcast, I'm working on building the ergonomic keyboard that I wish existed. Here's a glance at the prototype. It's a split keyboard, so there's two sides. This is in the case. But yeah, we launched on Kickstarter and we hit our goal within eight hours of launching. I really appreciate it if you were one of the people who grabbed one of the early units. We're now working on the long journey of building the tooling now. And so if you still want to pick one up,

• 59:00Ryan Peterman?

  I've left the Late Pledges open on Kickstarter, so you can grab one there. I'll put a link in the description. Thank you again for watching the podcast and I'll see you in the next episode.