The World’s Largest AI Processor with Cerebras CEO Andrew Feldman

The GPU supply crunch is causing desperation amongst AI teams large and small. Cerebras Systems has an answer, and it’s a chip the size of a dinner plate. Andrew Feldman, CEO and Co-founder of Cerebras and previously SeaMicro, joins Sarah Guo and Elad Gil this week on No Priors. They discuss why there might be an alternative to Nvidia, localized models and predictions for the accelerator market. Show Links: Andrew Feldman - Cerebras CEO & Co-founder | LinkedIn Cerebras Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @andrewdfeldman Show Notes: (0:00:00) - Cerebra Systems CEO Discusses AI Supercomputers (0:07:03) - AI Advancement in Architecture and Training (0:16:58) - Future of AI Accelerators and Chip Specialization (0:26:38) - Scaling Open Source Models and Fine-Tuning

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  The shortage of compute for AI, also known as a GPU crunch, is increasingly impacting AI companies, big and small. It is delaying training runs and launches for multiple players in the generative AI world. One company coming to the rescue is Cerebras Systems, which is developing the largest computer chip and one of the fastest purpose-built AI processors ever. This week, Sarah and I are joined by Andrew Feldman, CEO of Cerebras Systems. Andrew is one of the few entrepreneurial veterans in the semiconductor world. He previously started C-Micro, a pioneer of energy-efficient, high-bandwidth microservers. C-Micro was acquired by AMD in 2012. Andrew, thank you so much for joining us today.

• 0:44Speaker 01

  Well, and Sarah, thank you so much for having me.

• 0:47Speaker 00

  I think you all recently announced that Cerebras has closed $100 million deal with G42 to develop one of the largest AI supercomputers in the world.

• 0:54Speaker 01

  We did announce a strategic partnership with a group called G42, and we announced that we were building nine supercomputers. Each supercomputer would be four exaflops of AI compute, so in total 36 exaflops of AI compute. That was extraordinarily exciting. When you encounter a partner that shares your vision and wants to build with you, and you get to start building the biggest compute on Earth. It doesn't get better than that.

• 1:27Speaker 00

  And I think in general, you all were very forward-thinking and early to identifying AI as a really important market for custom semiconductors. Could you tell us a little bit more about your thinking early on as you started Cerebras and why you focused on AI many years ago?

• 1:41Speaker 01

  In late 2015, five of us started meeting regularly. I think we were meeting in Sarah's offices, actually, at that point. All of us had worked together at our previous company, and we began working on ideas.

• 1:54Speaker 01

  And one day, our CTO Gary leaned back and said, why would a machine built for pushing pixels to a monitor be ideal for AI? Then he said, wouldn't it be serendipitous if 20 years of optimizing a part for one job left it really well suited for another? That got us excited. We began looking at GPUs, looking at AI work. And by early 2016, we decided that we could build a better part for this work. And our strategy was not to build a little bit better, but it was to try and do something vastly better. And we went to a technology that had never worked before called wafer scale.

• 2:37Speaker 01

  And we build a chip that's sort of the size of a dinner plate, whereas most chips are the size of a postage stamp. And we did that because we knew that this workload would go big and that the problems of memory bandwidth, problems of breaking up work and spreading it over lots of little machines would be daunting. This is my fifth startup and this is the first time I was wrong on the market size on the underside. I had no idea it was going to be this big. And I think very few people saw, even those of us who were in it, how big this was going to be. And so, early 2016, we went out, we did eight pitches, we got eight term sheets, we raised

• 3:21Speaker 01

  money and we're building multi-exa-flap AI supercomputers for customers around the world.

• 3:27Sarah Guo?

  It's really incredible thinking about the foresight. I actually went back and looked at my notes from like our end of 2015 meeting. And if, I'm sure you remember very well, but for our listeners, Andrew had a slide in there of how the top seven problems for deep learning were all long training times. And like the whole industry was bottlenecked on computing. This is even before the scale up of transformers had happened. It's kind of wild how correct you were, at least on, you know, the depth of this problem.

• 3:58Speaker 01

  I think what you need to do is save that snippet and send it to my wife. We got that right. I think also we got the fundamental architecture right in a large sense. You know, we laid down the architecture before transformers exist and we're the fastest at transformers by a lot. So when you break up a problem and it encounters things it has never seen before, and it's still really good at them, that's a sign in hardware architecture that you really got the architecture right.

• 4:27Speaker 00

  What are some of the benchmarks that you use in order to assess the performance of your chip versus others? And how have you all performed relative to GPUs and others sort of more standard?

• 4:36Speaker 01

  We are not big believers in canned benchmarks. When I was at AMD, we had a team of 30 whose job it was to game benchmarks. When our CTO was at Sun, then a team of 70 whose job it was to game benchmarks. How long it takes your customer to train a model is the answer. And usually a collection of models. If you're trying to do a 13 billion parameter model, medium size, right? Almost always, you begin at 100 million. And you do 100 million, you do a series of sweeps, right? You're trying to do your hyperparameter tuning. So you do 100 to 300, 600, maybe a billion. And you're trying to find the right optimization points. And I think how you do on all of those is the answer.

• 5:18Speaker 01

  How long it takes you to move from customer model to distributed compute to finished run? And if you have to spend months doing distributed compute, doing tensor model, parallel distributed compute, I mean, if you look at the back of some of these papers, they're crediting 20 or 30 people, sometimes more, who helped on the distributed compute. And if you don't need to do that, like you don't with our equipment, because we run strictly data parallel, that's all time that you get back as a customer, right? That's all time that you get to train models and do interesting work rather than try and

• 5:59Speaker 01

  think about how to break up a large matrix, multiply and distribute it over a collection of GPUs.

• 6:06Speaker 00

  Can you actually talk a little bit about the architecture of your chip relative to others?

• 6:11Speaker 01

  All right. It's a data flow architecture. It's comprised of about 850,000 identical tiles. Each tile is a processor and memory. And so it's a fully distributed memory machine. And so you have huge amounts of memory bandwidth because the memory is speaking to a processor that's one clock cycle away. So by data flow, it means that the cores wait until they get a token or a flit that arrives, it tells them what to do, do work on this and where to send it. And then it sends it forward. And this is a particularly good

• 6:55Speaker 01

  architecture for the type of work AI is. AI is a, is a flow, right? It's a learning system. It is interesting. It was developed in the eighties at MIT by Guy Armin and they didn't have a good workload for it. So it just sat there. Nobody, and then we got a good workload for it. It's phenomenal for this. And that's a really interesting part of computer architecture where a really cool architecture can have no interesting workloads for it. And then you get a good workload and now it's ready. Now it's good at this particular problem. We keep a huge amount of SRAM on the wafer. All right. And so there are no memory bandwidth problems ever.

• 7:37Speaker 01

  That also allows us to harvest sparsity, which is something that others really struggle with. The cluster we build keeps the parameters off chip in a parameter store and it streams them in. And the result of this architecture is that we run strictly data parallel, which means even in a 64 node cluster, you run the exact same configuration on each machine. Each machine sees the exact same way, but they see a different portion of the dataset.

• 8:11Speaker 01

  And it scales linearly, which is unheard of in compute.

• 8:15Sarah Guo?

  Can you, for a broader audience, describe like how this addresses some of the challenges of using traditional GPUs for large scale machine learning?

• 8:24Speaker 01

  Sure. One of the real challenges of a traditional GPU is they fix the amount of memory on the interposer, right? So you can buy a GPU with either 40 or 80 gig. What if you want 120? What if you want 20? What if you want a bigger parameter store, right? You gotta buy more GPUs even though you don't want those. And so what this strategy does is it allows you to support extraordinarily large models by disaggregating the memory from the compute. And in all GPUs, they're tightly coupled. And that means if you want more compute without memory, you still have to pay for the memory. If you want more memory without compute, you still have to pay for the compute.

• 9:06Speaker 01

  And that was an idea that came from supercomputing that we knew really well, that we could organize this so you could run an arbitrarily large, now a trillion parameter network on a single system. It would be slow with a very big network, but you could run it on one system, you can identify layers, you can debug, you can do all this, and you can never do that on the GPU, right? Because there's nowhere to store the parameters. And so by separating parameter storage from compute, we allowed you to mix the various ratios at your will, rather than at the GPU vendor's will.

• 9:47Speaker 00

  So I guess in terms of the relationship that you have with G42, there's a variety of different applications that you've looked at across healthcare, across a variety of different verticals. What do you view as some of the first use cases or best applications for your architecture?

• 10:00Speaker 01

  So the first thing that they allowed is a model called BTLM, which is the number one three billion parameter model on Hugging Face right now with more than a billion downloads. We have Fortune 100 companies running on it right now. We have them doing their own internal work on it. On Wednesday, we're announcing that we're open sourcing with G42's group called Inception and MBZ UAI University, the largest Arabic LLM. So we're putting that into the open source community. That was a collaborative effort with our strategic partners there. When you have a lot of compute, the mind boggles and all the cool stuff you can do.

• 10:45Sarah Guo?

  Why did you guys get into the training and open source model game?

• 10:51Speaker 01

  We had so much compute and that it was a way we could prove to the world how easy it was to train on us. We felt it was evidence that we could build and our systems could train the biggest and fastest models, the most accurate models in the space. In March, we put seven GPT models in the open source community. Everybody else was putting one. Why? Because it's really hard to redistribute work across a GPU cluster. For us, it's one keystroke. So we put seven. People are coming to us with extraordinary ideas. We had a customer who came and they said, look, we'd like you to design for us a model

• 11:36Speaker 01

  at three billion that we could prune and quantize such that it could be served off a cellphone. Really cool.

• 11:44Speaker 00

  What are the biggest milestones that you're thinking about going forward?

• 11:47Speaker 01

  There's been a lot of back and forth. Everybody thought we'd keep getting bigger. You know, north of one hundred and seventy-five billion is a giant network. It's expensive to train. And maybe even more important, it's really expensive to serve. Right? Seventeen and a half times more expensive to serve than a ten billion parameter model. And so I think a lot of the models in production are thirteen billion and smaller as an effort to keep the cost of production down. We are training right now a whole collection in the thirty to one seventy-five category. I think there's going to be a spike in multilingual models, right? Stability just put out one in Japanese. There was one in Spanish that

• 12:32Speaker 01

  came out. Inception, MBZ UI and Cerebras put out one in Arabic. I think these are underrepresented languages. And I think we have a challenge there in that most of the big data sets are giant internet scrapes and most of the internet's in English. And so it's really hard to build big data sets in languages that aren't English. We don't have enough tokens. And so techniques to carefully think about how you can do that, I think, will be very much in demand. I think every nation wants to feel like its language and culture is captured fairly in an LLM.

• 13:12Speaker 00

  That's interesting because a lot of the training also seems to capture social norms versus data. And one can imagine as well as you think about different regional models or different language-specific models, people want their own set of cultural norms to be part of the model in some sense versus something that's imposed.

• 13:27Speaker 01

  That's exactly right. Language captures experience. I mean, there's words that are different and have very unique, I mean, it's just like the New York experience includes Yiddish.

• 13:41Speaker 01

  Right? It will be important that we find ways to capture these cultural experiences and we do that in language. And if you overwhelm the minority language with English, all that gets sort of sifted out.

• 13:56Speaker 00

  I think the Yiddish LLM would just complain all the time.

• 13:59Speaker 01

  That would be hilarious. That would be hilarious. Yiddish LLM, it's like, well, that's your brothers are doctors.

• 14:08Speaker 00

  Every response is, why aren't you a doctor yet?

• 14:11Speaker 01

  Are you gonna finish your PhD, Andrew?

• 14:15Sarah Guo?

  Oh, my God, no, don't worry. The Chinese LLM says that, too.

• 14:17Speaker 01

  Oh, it will. That's right. Those exact same, same mothers.

• 14:21Sarah Guo?

  I think it's like, actually, like, kind of a deeper insight in the joke, which is there's cultural experience baked into both the actual tokens and pre-training curation and like what you're weighted toward, as well as in the RHF layer, right? Because whatever, you know, human annotators are saying, like, this is, this is what I want, somebody wants to kvetch with their chatbot, too.

• 14:43Speaker 01

  That's exactly right. One of the things we learned was that it's really important taking into account the language that has the fewer tokens, especially, you know, Arabic, Hebrew, Hindi, their characters sometimes require two bits rather than one bit. And so the tokenization scheme has to be really thoughtfully done, or you further bias the model in favor of the English. I think, you know, you put your finger on exactly why our strategic partners are so excited by this is they want their culture and their language properly represented. They want it reflected. They want the nuances in their culture reflected in an LLM and not sort of sanitized or washed

• 15:27Speaker 01

  with, you know, 99% English tokens.

• 15:30Sarah Guo?

  Can we zoom out and just like, talk a little bit about what it's like to start a chip company? It's a very uncommon experience, especially, you know, in as far deep as you guys are now with the money you've raised. Like, what's the hardest part of bringing a new chip to market?

• 15:45Speaker 01

  I think the structure of the chip market is challenging. I think one of the things that VCs love about SaaS and about software companies is for not very much money, you can get a prototype and you can start seeing if customers like it. Chips are the opposite. We're going to spend two, two and a half, sometimes three years and fifty or sixty million dollars building our first part. And then we will learn if people like it or hate it. It has a huge gaping chasm between when you start and when you begin learning from customers. And that's one of the things that the software doesn't have. Now, it turns out that if you're building a SaaS company, you end up often using as much money.

• 16:30Speaker 01

  You get insight into what your customers like much, much earlier than in a chip company. And I think that's perhaps the biggest structural difference. The structure of engineering is different when you've got a two or three year project. You can't do a hundred and fifty one week friends. The way you manage QA versus development is different. Often in chip companies, for each developer, you have three or four QA people, DV people. And that's because a bug can cost you twenty or thirty million dollars. It's not like the outfit set in four dot one dot one. You're going to have to re-spin the chip if you have a big bug. And so the commitment to tooling, to QA, to simulation is profoundly different in

• 17:15Speaker 01

  chip, in the chip world than when you're building software. Now, we still have to build a huge amount of software and we're about seventy-five percent software. When you build a chip, the chip is the muscle and the software is the brain. You still need a huge amount of software. You need low-level software, software that runs right against the hardware. You need a compiled stack that allows the user to write in PyTorch and just run as if it were GPU. And you end up with a lot of different types of engineers in a system company.

• 17:49Sarah Guo?

  Andrew, you have such a unique expertise and depth and a question that everybody is thinking about right now. Can you give our listeners an overview of the AI accelerator market right now? Like, who are the big buyers? And then how might they decide to do anything but Nvidia?

• 18:04Speaker 01

  Nvidia's made hay when the sun shines. And they've done an extraordinary job and you have to take your hat off to them, right? I think they're now in a situation where they're extorting customers, they're extremely expensive, they're unable to ship.

• 18:18Speaker 01

  And that has, among other things, opened the door for many of us who have alternatives. Nobody likes being dependent on their vendor. You can ask the guys at Google and Facebook who are dependent on Intel for years how much they hated that. They dislike that intensely. And so I think there's a battle between people's sort of dislike of being dependent and the need just to keep running forward. I think large enterprises continue to be a good part of our business. I think often where you can provide a little consulting to help them accelerate their model deployments is something that provides an opportunity for smaller companies to get in the door.

• 19:03Speaker 01

  And when you show them how much easier it is, we had a situation where they were trying to train on a GPU cluster and they were at 60 days and it wasn't converging and we stood it up and three and a half days later their model converged and they're like, holy cow. We have an internal cloud so customers can just jump on your cloud, they can begin training right away. If they want to try before they buy, they can begin with a little training run and go from there. We have customers in large enterprises and generative AI startups, we have customers in the military, in the intelligence communities, we have customers in the super compute market world. We sort of up and down organizations of different sizes.

• 19:44Speaker 00

  How much specialization do you think there will be in the semiconductor layer going forward? In other words, do you think there'll be increasingly different chipsets for inference versus training, for transformer-based architectures versus diffusion models, for multimodal versus not? I'm just sort of curious how you view all that.

• 20:00Speaker 01

  The fundamental question in computer architecture is what do you make special and what do you make general? And the second fundamental question is what are you good at and what are you not good at? What do you choose not to be good at, right? And those things are really important, you know? If you put in circuitry for transformers, that has a cost, there's a penalty every time you do a convolutional network. And so these trade-offs aren't free. You put in circuitry for one thing, if you're not doing that thing, that circuitry just sits idle and you've wasted power and space. And so I think...

• 20:43Speaker 01

  It is a TBD question on whether we will continue to see more specialization at the chip level in training. We do not do it. We crack the problem in a different way. All of the problems are linear algebra and are predominantly sparse linear algebra. So we focused on underneath the transformer. Now, whether you'll have different silicon for training and for inference, I think you will.

• 21:11Sarah Guo?

  So your view on this, Andrew, is less a strong point of view that there will be something beyond transformers that is very important in the near term and more you've already solved in a more general way?

• 21:23Speaker 01

  That's right. The trick in architecture is to solve hard problems in a general way so you don't have to rely on product management to sort of guess what's the next cool layer type, right? If we go to a type of model that doesn't require very much data, if we go to single-shot learning, right, Nvidia's totally out of luck, right? Us too, everybody. These models are designed for extremely data-intensive transformer-based. But whether you're doing convolutional neural networks or transformers or diffusion models, or certainly new things will be invented. I mean, we haven't run out of ideas. We haven't even run through the

• 22:05Speaker 01

  ideas that the guys put together in the 80s, right? We still have plenty of work to do to improve our model making. And as the industry matures, and we're still in the early innings, you might see more customization for a particular model type, but that's not my preferred approach.

• 22:26Sarah Guo?

  Are you making bets internally on separating training and inference?

• 22:31Speaker 01

  We make all sorts of bets, Sarah. We lose frequently and we win sometimes. And that's the real fun of building companies in a dynamic environment, right? I think inference, especially generative inference, right, is a very different problem than all other forms of inference.

• 22:53Speaker 01

  Right. Basic classification is the exact same as the first pass in a training run. But when you're doing inference on generative AI, it's not, and you have to hold a ton more information at the inference point. And so I think there will be, we will discover, we will invent new and different ways to attack that problem. It's extremely expensive right now. I mean, people using eight H one hundreds to do inference on a big model. I mean, that's half a billion dollars.

• 23:27Speaker 00

  You know, there is this ongoing crunch to supply for GPU and other sort of AI centered compute. And that's causing different people to delay launching their companies or training their models. And those are often the same thing for different types of AI companies. What do you view as the main cause of this supply crunch? And when do you think it loosens up? Or when do you think things loosen in the market?

• 23:49Speaker 01

  The chip market has a profoundly inflexible supply chain. And so TSMC, their fab is a pyramid in modern society. It is the absolute peak of manufacturing capability. It's a twenty billion dollar building in which nobody works in it. And ideas come in and out come chips that cost a hundred dollars. I mean, these are unbelievably unbelievably amazing things.

• 24:18Speaker 01

  But they can't turn on a dime. And so when somebody misses their forecast by a lot, and they come back to the fab and say, I need twice as much or three times as much. Often that takes six or eight months to sift through. And that's the problem that's coming in right now. I mean, it's not just that Wall Street missed what Nvidia would sell. Nvidia missed it. They missed the forecast. And you can go to TSMC and you can ask them for wafers, but many of their wafers are already allocated to AMD, they allocate to Qualcomm, they allocate to other giant companies, right? And so you struggle forward. And so in addition to taking a long time to get to

• 25:01Speaker 01

  first customer learning, there's a huge premium placed on the ability to forecast accurately chip business.

• 25:10Speaker 00

  Given that we're going through this big discontinuity from a technology perspective, adoption perspective, et cetera, do you think we'll be accurate in forecasts going forward anytime soon? Or do you think there's likely to be a crunch for the next couple of years?

• 25:21Speaker 01

  No, we're going to continue to get it wrong. We're going to continue to get it wrong for a while.

• 25:25Speaker 00

  Do you think we're largely going to underestimate it?

• 25:28Speaker 01

  Oh, yeah.

• 25:29Speaker 00

  So basically, the market is accelerating rapidly. Everybody's underestimating it. And, you know, fundamentally, we all think that this is going to be many times bigger than anybody thinks two, three years from now.

• 25:39Speaker 01

  Yeah, and the dynamics of our supply chain are when you get it wrong and you order too much, you still have to take it, right? You don't get sale, guess what, guys? I really don't want those wafers anymore. And they're like, see this $10 million worth of wafers? Yours.

• 25:55Speaker 00

  Yeah, that happened during crypto, right? So I think Nvidia had a really big miss during crypto.

• 25:59Speaker 01

  It happened during crypto. It happened. It happened to Broadcom. I mean, Arista today is at 52 week lead times on switches.

• 26:09Speaker 00

  Mm hmm. Given that you've been involved with training some incredibly performant open source models, and you mentioned that as you scale up a model, inference cost starts to really kick in as you get to bigger and bigger models from a parameter perspective. What do you view as the limits to scaling these models? Like, at what point do they get too big? Or do you think we just keep scaling them until we have these sort of hypermodels?

• 26:30Speaker 01

  It's a really interesting optimization problem. They get big, they get harder to work with, it's harder to retrain, right? What even GPT-4 came out, it didn't have any insight beyond 2021, they had to race. And that was because it was so big. Right? And so I think there are trade offs. There are trade offs between accuracy and size. There are trade offs between size and cost to do inference. I mean, maybe I want a larger model for radiology files. Right? Or for my doctors, maybe I'll take a smaller model for my chat or my customer service bot, right? We're going to have to think about these in terms of what they're delivering, what the cost of being wrong is, what the cost of serving is,

• 27:14Speaker 01

  and think about this as a business decision. And at first, everyone's just running, trying to say, I can make a bigger model. I can make a bigger model. And then the guys who are trying to run businesses are like, well, I can't afford to give that away. And so we're going to be down at three and at six billion and at 13 billion, because that's, I can get pretty good, pretty darn good, and not break the bank with free inference. And so I think that there are these trade offs that are really challenging, and we're just beginning to grapple with now.

• 27:45Speaker 00

  Now, I guess OpenAI, related to that, recently announced a partnership with Scale around fine tuning. How important do you think fine tuning will be going forward?

• 27:52Speaker 01

  I think fine tuning will be really important. We're still just learning about the capacity of these models to hold information, to hold insight. And fine tuning is an extremely exciting approach. I think we've seen again and again that human feedback makes these models vastly better. And that's not surprising, right? I mean, that's how you learn to dance. That's how you learn to do everything, soccer, gymnastics, ballet. And somebody saying, no, that's wrong. Nope, do it again. That correction is a mechanism for improvement somehow doesn't seem very surprising to me. I think we don't yet know how far fine tuning is going to take us

• 28:36Speaker 01

  and who will benefit from a model trained from scratch versus continuous training versus taking a model off the shelf and adding to it their unique data.

• 28:46Sarah Guo?

  You know, there is this point of view from some of the large labs that very few people will do training, and it will get quite concentrated. But I've heard you speak about how you think that there are really interesting data sets in different large enterprises. Tell us about that.

• 29:03Speaker 01

  You know, at first, everybody scraped the internet, then everybody had that. And now the companies that have extraordinary data sets, Reuters, Bloomberg, Galaxia Smith-Klein, Abbe, right, these companies that have years of research and exceptional data are going to, I think, step in the floor. The AI has made progress to the point where people are looking around for exceptional data sets, unique data sets, the rise of data as an asset. I think it's the new gold. And I think that's something that isn't top of mind enough and will be very much top of mind going forward.

• 29:44Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.