#228 Rodrigo Liang: How SambaNova Systems Is Disrupting AI Inference

This episode is sponsored by RapidSOS. Close the safety gap and transform your emergency response with RapidSOS. Visit https://rapidsos.com/eyeonai/ today to learn how AI-powered safety can protect your people and boost your bottom line. In this episode of the Eye on AI podcast, we explore the world of AI inference technology with Rodrigo Liang, co-founder and CEO of SambaNova Systems. Rodrigo shares his journey from high-performance chip design to building SambaNova, a company revolutionizing how enterprises leverage AI through scalable, power-efficient solutions. We dive into SambaNova's groundbreaking achievements, including their record-breaking inference models, the Lama 405B and 70B, which deliver unparalleled speed and accuracy—all on a single rack consuming less than 10 kilowatts of power. Throughout the conversation, Rodrigo highlights the seismic shift from AI training to inference, explaining why production AI is now about speed, efficiency, and real-time applications. He details SambaNova's approach to open-source models, modular deployment, and multi-tenancy, enabling enterprises to scale AI without costly infrastructure overhauls. We also discuss the competitive landscape of AI hardware, the challenges of NVIDIA's dominance, and how SambaNova is paving the way for a new era of AI innovation. Rodrigo explains the critical importance of power efficiency and how SambaNova's technology is unlocking opportunities for enterprises to deploy private, secure AI systems on-premises and in the cloud. Discover how SambaNova is redefining AI for enterprise adoption, enabling real-time AI, and setting new standards in efficiency and scalability. Don't forget to like, subscribe, and hit the notification bell to stay updated on the latest breakthroughs in AI, technology, and enterprise innovation! Stay Updated: Craig Smith Twitter: https://twitter.com/craigss Eye on A. I. Twitter: https://twitter.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  If you look at the strong grip that Nvidia had on training, it's not like that. It's coming to an end with inference. Right. Not to say there won't be a player, but when the power and the performance and the cost are so far off from what others are able to provide, now you're starting to see large, large number of developers coming into other offerings and get their APIs from there. You can come into Google, Google Cloud, you can come into AWS, you can come in from wherever you want because it's just an API, it's just an API. And so we're able to host those APIs wherever the customer wants. In fact, here, we're in Saudi Arabia today and we're powered Saudi Aramco.

• 0:38Craig S. Smith?

  Hi, this episode is sponsored by Rapid SOS. Terabytes and petabytes data is exploding in our daily lives. How many connected devices do you own? Do you ever wonder how all this data could be used in emergencies? By 2030, we'll have over 32 billion IoT devices worldwide, double what we have now. Despite this data abundance, there's a critical safety gap. Vital information isn't reaching emergency responders in time. This gap results in reactive rather than proactive emergency response,

• 1:20Craig S. Smith?

  putting lives and property at unnecessary risk. Many of us have had emergencies in our homes where we depend on EMS to respond quickly. Rapid SOS is closing this safety gap with their AI-powered intelligent safety platform. They connect lifesaving data from over 540 million devices to more than 21,000 public safety agencies across six countries. Despite 200 billion in annual safety system investments, enterprises often lack timely, actionable information during emergencies.

• 2:04Craig S. Smith?

  Rapid SOS is changing that. It's not just about avoiding problems. Safety investments are linked to increased customer satisfaction, employee retention, and long-term firm value.

• 2:19Craig S. Smith?

  Close the safety gap and transform your emergency response with Rapid SOS. Visit rapidSOS.com/eyeonAI. That's eye on AI, E Y E O N A I, all run together. Visit rapidSOS.com/eyeonAI today to learn how AI-powered safety can protect your people and boost your bottom line. That's rapidSOS, R A P I D S O S, dot com slash eye on AI. Visit them today.

• 3:02Craig S. Smith?

  Can you just introduce yourself and what your background is, your educational background, and how you started?

• 3:10Speaker 00

  Yeah, Rodrigo Leon, co-founder, CEO of Summonova. I did my undergraduate and graduate work at Stanford. Been in the high-performance chip design business for 30 years now, starting with PA Rescat HP and then ran some microsystems, Spark processors for 15 years. And started this in 2017 with two Stanford professors who are still there, really driving a grounds-up way of thinking about artificial intelligence of semiconductors and driving for power efficiency, for performance, and ultimately for scale.

• 3:43Craig S. Smith?

  So I wanted to talk to you about the new inference, the system on a chip that beats Grok and Cerebrus's new inference service. Just, we don't have much time, so why don't you tell us what that is and why inference is becoming the new field of competition?

• 4:09Speaker 00

  Yeah, really proud of it. Today, we're now SambaNova Cloud, and it's an inference service, API service, for developers to be able to engage and use the best open source models as a consumption model, so token services. And so we've seen other players do this now, and you're providing tokens for some of these models. And so in that game, it's all about accuracy and performance. Right, and so we, today we announced that we're able to take the best model from Meta four or five B, the Llama three point one four or five B. We actually inference in full precision at a hundred thirty two tokens per second. In comparison, Nvidia is taking um uh for for the uh four or five B.

• 4:53Speaker 00

  They're having to get down to eight bit.

• 4:56Craig S. Smith?

  Hmm, right.

• 4:57Speaker 00

  But precision, which you lose accuracy in that, you know, when you quantify it down. And still running at something close to about thirty to forty tokens per second. And so, so you look at the comparison, not in terms of just accuracy, but performance. It's just number one, it's actually only one in the world because the Reber Sangrock cannot do it today. They don't, they don't offer it today.

• 5:16Craig S. Smith?

  Right.

• 5:17Speaker 00

  Um, oh, and and then we also set a new work in the seventy B model. Full preci- full precision, um sixteen bit uh running again. Um, and uh on Llama seventy B, we're doing five hundred seventy tokens per second. And again, number one in the world, and so really excited about that. But probably what's most uh exciting to me, Greg, is we're doing that in a single rack.

• 5:42Craig S. Smith?

  Hmm.

• 5:42Speaker 00

  A single rack that's running less than ten kilowatts.

• 5:46Craig S. Smith?

  Wow, yeah.

• 5:47Speaker 00

  Right, I've I've said all along that AI is going to production. The cost of inferencing will be ten times more than what we spend on training.

• 5:57Craig S. Smith?

  Yeah.

• 5:58Speaker 00

  I'll say ten times more, right? Production AI is going to be all about inferencing and fine tuning your private data into those models. And so you need to find a way to inference those efficiently. Efficient means speed and performance. It means accuracy, but it also means power.

• 6:16Craig S. Smith?

  Yeah.

• 6:16Speaker 00

  And we think that today, most of the other services, whether it's the startups or the big companies, they try to not talk about power.

• 6:23Craig S. Smith?

  Right, right.

• 6:24Speaker 00

  Right?

• 6:25Craig S. Smith?

  Yeah, uh let me ask you, uh the uh.

• 6:30Craig S. Smith?

  Cerebrus has had trouble getting traction in the market. Grok has had some trouble getting traction in the market. Because Nvidia has a lock on the cloud.

• 6:42Speaker 00

  Yeah.

• 6:44Craig S. Smith?

  And and they have a lock on developers with CUDA.

• 6:49Craig S. Smith?

  This API inference service sidesteps that. Because people can build, this is a fast inference, much faster. So they can they can build on top of it. But how do you get the clouds to adopt Samba Nova? And and switch from Nvidia?

• 7:15Speaker 00

  Yeah, that's a that's a beautiful thing today. If you look at the strong grip that Nvidia had on training.

• 7:22Craig S. Smith?

  Yeah.

• 7:22Speaker 00

  It's not like that. It's coming to an end with inference.

• 7:25Craig S. Smith?

  Right.

• 7:25Speaker 00

  Not to say there won't be a player. But when the power and the performance and the cost are so far off from what others are able to provide. Now you're starting to see large, large number of developers coming into other offerings. And run them, you know, and and get their APIs from there. Now, you can summon over, you can come into Google, Google Cloud, you can come into AWS. You can come in from wherever you want because it's just an API.

• 7:53Craig S. Smith?

  Yeah.

• 7:53Speaker 00

  Right, it looks like OpenAI is just an API. And so we're able to host those APIs wherever the customer wants. In fact, here, we're in Saudi Arabia today and we we power Saudi Remco.

• 8:05Craig S. Smith?

  Oh, is that right?

• 8:05Speaker 00

  We powered their internal meta brain, right? Which is, we take ninety years of data, train it into a private and secure model. And we inference it completely privately, deployed in their own data centers here in Saudi Arabia. Completely secure. And they're influencing, you know, their data in these models for their employees. And so this is the use model that we think that is going to be extremely fast for people to adopt. Because you no longer need to worry about CUDA and what you trained it with. You can take any open source model, dump it in.

• 8:36Craig S. Smith?

  Right. What about production and? Capacity, because that's, you know, I wrote an article on cerebras. One of the pushbacks I got is, yeah, but does cerebras have the production to to meet demand?

• 8:53Speaker 00

  Yeah.

• 8:53Craig S. Smith?

  Uh, and and are the, you know, how much capacity do they have to serve APIs through the cloud?

• 9:00Speaker 00

  Yeah, that's a really valid question. And one of the things I tell people, take a Lama seventy B. Right? The most important question people should ask is, how many sockets does it take to do that? Right? Or how many wafers does it take to do? How many racks does it take to do that? And why is that important? You're running one model. If that one model takes hundreds of chips. Or many, many very expensive wafers to run. Or in Nvidia's case, still saying a lot of power to run, that doesn't scale. Right? And so for someone who, we're running these models in a single rack at less than ten kilowatts. And so what that allows us to do in a very modular way is just deploy more racks very efficiently.

• 9:43Speaker 00

  And going into existing data centers, you don't need to go build a liquid cool, brand new data center to power these systems. Because it's too expensive.

• 9:52Craig S. Smith?

  Yeah, which is, which is what Nvidia is doing, right? They're starting to build their own data centers for inference, is what I've read anyway.

• 9:59Speaker 00

  Exactly. You have to, because the chips are so power hungry. And they need all this new liquid cooling, and that cooling has to be done a special way. Someone will be decided, we don't want to do any of that. All we want to do is delivery performance, delivery availability, all in the existing infrastructure that there is. Data centers that have ten kilowatts are able to actually push this rack in and achieve one hundred thirty two tokens per second on four or five B. Single rack.

• 10:26Craig S. Smith?

  Where is your production? Where are you making the chips?

• 10:29Speaker 00

  The chips have been in our TSMC in Taiwan.

• 10:31Craig S. Smith?

  Yeah. Are you having trouble getting, I mean, everyone?

• 10:36Speaker 00

  Yeah, I mean, this is one of the really great things about uh uh the RDU, the reconfigurable data flow chip that's ours. We're able, because it's data flow, we're able to collapse the total number of chips by an order of magnitude.

• 10:50Craig S. Smith?

  Yeah.

• 10:51Speaker 00

  So what you would have thought would take, say, a hundred chips of Nvidia to do, we can do it with ten-ish chips. And so, so anyway, that's kind of allowed us to significantly reduce the demand of chips, which then has all these other supply chain benefits.

• 11:05Craig S. Smith?

  Yeah.

• 11:05Speaker 00

  But uh but as a well-funded startup, we raised one point one billion in venture capital in the first three years of the company.

• 11:11Craig S. Smith?

  Yeah, how long have you been around?

• 11:13Speaker 00

  We've been six, almost seven years.

• 11:15Craig S. Smith?

  Yeah.

• 11:15Speaker 00

  The first three years, we're able to raise a lot of capital from some of the top investors out, you know, this is Google and Intel and you know, BlackRock and SoftBank and Temasek, you know, some of the, you know, uh great investors.

• 11:27Craig S. Smith?

  The Saudis, I'm guessing?

• 11:29Speaker 00

  Well, you know, so, so the, yeah, well, yeah, we're, the ones that we've announced are are the ones that are public, but uh uh but these are the the the investors that have allowed us to actually be able to get these chips way in advance.

• 11:42Craig S. Smith?

  Yeah.

• 11:42Speaker 00

  Get the chips early so that we have them, we can supply them, just supply them quickly. But, right, as you're seeing this, this drive to, you know, for demand growing at this rate, there's always going to start having a lead time. We're starting, you know, over the last couple of months, we've started kind of really actively managing allocation because it's starting to get to the point where we we we have to do that despite the fact that we pre-bought so much material.

• 12:06Craig S. Smith?

  Yeah, yeah. So you you have a s- a good pipeline of chips coming through. You're negotiating presumably with the cloud. I mean, one thing, you know, I'm a journalist.

• 12:18Speaker 00

  Yeah.

• 12:18Craig S. Smith?

  Uh, I wasn't aware of SambaNova.

• 12:21Speaker 00

  Yeah.

• 12:22Craig S. Smith?

  Is uh, wh- why is that?

• 12:24Speaker 00

  I know, I know.

• 12:25Craig S. Smith?

  Because you, and are you focusing on a particular kind of inference? Or is this wide open?

• 12:33Speaker 00

  Yeah, our history has been that we go into enterprises, we go on-prem. Today, we're the most deployed AI chip startup in the US government. People don't know that.

• 12:41Craig S. Smith?

  Yeah.

• 12:41Speaker 00

  Right? We go, you know, we go into all the major labs, we go into a bunch of different places, we're in banks, we're, you know, in a, you know, three continents today. We're going to enterprises where people have a lot of private and secure data, they don't want to disclose it.

• 12:53Craig S. Smith?

  Right.

• 12:54Speaker 00

  To somebody else. Eighty-three percent of enterprise data sits on-prem today.

• 12:58Craig S. Smith?

  Yeah.

• 12:59Speaker 00

  Right?

• 12:59Craig S. Smith?

  Yeah.

• 13:00Speaker 00

  They have all that data, they don't know what it says.

• 13:02Craig S. Smith?

  Right.

• 13:03Speaker 00

  So Sanmanova, for the last several years, we were able to bring our racks in because we're so efficient. Instead of having to create an entire data center to read your models, I can just roll in a couple racks. You've got a model, I can read the data, and now you have your own private GPT.

• 13:17Craig S. Smith?

  Right, right.

• 13:18Speaker 00

  And so, um, so, so I think that's kind of for, for us is just something that we're able to do very efficiently and get into engagements with enterprises very quickly.

• 13:28Craig S. Smith?

  Yeah.

• 13:28Speaker 00

  Now, as we've moved since the world's going to production, we see this tremendous energy around inference. And frankly, our technology is really good for inference, as we've now shown with four or five B being world record, seventy B being world record, all sixteen bit. And so we're able to drive these results in a way that others can't and do it in a, a, a fraction of the powers that everybody's doing. So that's what drove us to do this cloud now, because now we can take what we're doing at the customer's premise. We're just putting them into cloud environments. And these aren't our data centers, these are our partners' data centers, and we run the service through them.

• 14:04Craig S. Smith?

  Right.

• 14:04Speaker 00

  Right, and so putting it through our partners' data centers so that then we can offer these services to a broad range of people that perhaps they don't want to do it on-prem by, by themselves.

• 14:12Craig S. Smith?

  Yeah, and this move toward, well, first of all, the, the higher speed gives you a much larger context window. Is that right?

• 14:21Speaker 00

  Well, there, well, there are three things that, that you get from Sanmanova. One is speed, right? The speed that you're getting is, you can use it for a couple of different things. One, the world wants to go real time.

• 14:34Craig S. Smith?

  Right.

• 14:34Speaker 00

  But a lot of AI wants to be real time. And if you haven't, if you don't have the speed, it's too slow, we're in impatient world.

• 14:40Craig S. Smith?

  Right.

• 14:40Speaker 00

  Right, and so after a couple of seconds, we think the machine, right? And so, so real time, you need the speed, most of the slower models just won't work. So that's one thing that we do with Sanmanova, that we're able to give you that speed. Second, because we're driving the four or five, four hundred billion, all the way to a trillion models, we're giving you the accuracy, the bigger the models, the more accurate. Unlike what others says, the bigger the chip is better. No, no, the bigger the model is better. The bigger the chip just burns more power. If the big chip can't run the big model, then you can't, you're not getting the benefit. And so we want the bigger models. Four, five B from Meta is the best open source.

• 15:15Craig S. Smith?

  Yeah, and how is that architected? Are you putting the weights directly on the chip? Is it in memory just?

• 15:25Speaker 00

  Yeah, so SambaNova, yeah, so SambaNova has a very sophisticated memory hierarchy with a lot of SRAMs on chip. But we have HBM and we have DDR. So in a single eight socket system, we have 12 terabytes of DDR directly attached to SambaNova's boxes, to SambaNova's chips, which allows us to do all sorts of different things, right? And so we're able to, in a very small footprint, host a 400 billion parameter model and run it really efficiently. Efficiently, but the other thing that we can do now is we can also concurrently host hundreds of your AP model.

• 16:04Craig S. Smith?

  Yep.

• 16:05Speaker 00

  In a virtual manner.

• 16:06Craig S. Smith?

  Sure.

• 16:07Speaker 00

  Right, so now I just brought virtualization into the world of AI, which Nvidia and all the other startups don't do.

• 16:12Craig S. Smith?

  Right.

• 16:13Speaker 00

  Right, and you say, why is that important? Because one day, you will have your own Llama checkpoint, he will have his checkpoint, everybody wants their own. And today, in the, in the existing legacy world, your model, one rack of hardware, his model, another rack of hardware. We can multi-tenant hundreds of these checkpoints in one system and dynamically switch virtually in the way that VM, we made an entire business for two, three decades, right? And so, so that's what that memory has allowed us to do. I mean, it's a lot more than just a memory. The software is very intelligent to do that. But we give you that efficiency because we give you multi-tenancy, we give you speed, and we give you the actual raw power of the rack.

• 16:53Craig S. Smith?

  How fast is the API? Because the inference can be fast, but you still need the bandwidth out to, out to whoever's using the API.

• 17:06Speaker 00

  And SambaNova, what we do here is we, you know, I mean, the model, our model, the AP model, 0.09 second time to first token, that's number one in the world.

• 17:13Craig S. Smith?

  Yeah.

• 17:13Speaker 00

  That's number one in the world. But like you said, it depends on where you are. So why are we able to actually get our service in this way? Is we're dropping these racks across the world. Because I don't need to build gigawatt data centers.

• 17:27Craig S. Smith?

  Right.

• 17:28Speaker 00

  We can go into existing data centers and turn that into a micro pod for service.

• 17:33Craig S. Smith?

  Yeah.

• 17:33Speaker 00

  Right, we could put eight racks, eighty racks, eight hundred racks.

• 17:37Craig S. Smith?

  Yeah.

• 17:38Speaker 00

  Right, because it's very modular and it's all a single rack and a single rack runs the full model.

• 17:42Craig S. Smith?

  Right.

• 17:43Speaker 00

  Right, and so it gives us this flexibility of bringing AI locally, and we've already done it because we do it for people on prem.

• 17:49Craig S. Smith?

  And and ju- just on the on prem point, right now, you're doing the open source, not as open source, presumably other open source models. Uh, command R or or.

• 18:02Speaker 00

  We have a broad range of models that we support, all open source from Hugging Face. Our top model is called Samba One. Samba One is a composition of experts, and what that is, is basically a virtual platform that allows you to bring many checkpoints of open source models. And then the model will route to it based on your prompt.

• 18:22Craig S. Smith?

  Right.

• 18:22Speaker 00

  Right, and that and that's a that's the architecture of how we achieve multi-tenancy. There are ninety one different models today offered on Samba One.

• 18:31Craig S. Smith?

  OpenAI is still the king, they're proprietary, uh, I mean, there are other proprietary models that are that are powerful. Uh, what what's it gonna take to get OpenAI to use Samba Nova instead of Nvidia GPUs?

• 18:48Speaker 00

  Well, we've got to run it better than, you know, kind of what they have with Nvidia, for sure, right? And as these models are getting bigger, Samba Nova's technology advantage on the big models start getting bigger and bigger and bigger, right? And so this is what we play, and we play really, really well. We can run these large models better than anybody else. And even the modest models are already showing number one, but the bigger the model, the more our architecture shines.

• 19:12Craig S. Smith?

  Why is OpenAI so married to Nvidia, do you think?

• 19:15Speaker 00

  I don't know. I don't know. I think it's time. I think I think a lot of the the decisions there were probably tied to legacy decisions that they've made, investments you you you've already made on GPUs. Because GPUs were there before us.

• 19:28Craig S. Smith?

  Sure.

• 19:28Speaker 00

  Right, but we're at this point in time now where the scale, the next phase, the scale of hardware investment you have to make is an order magnitude larger.

• 19:38Craig S. Smith?

  Where is it going with inference? Because there was so much focus on training, foundation models. Uh, but now the focus has shifted to inference.

• 19:48Speaker 00

  Yeah.

• 19:48Craig S. Smith?

  You mentioned someday you're each gonna have your own MAMA. Check point. Yeah. Where do you see that going? What kinds of products do you think this is? This is going to?

• 19:59Speaker 00

  Well, three things. I think the world's gonna be open source. So you go open source and that's gonna continue that way. I think it's gonna be more and more. The models are really good, good now. Two, I think it's going to be bigger and bigger over time. Why? Because people want accuracy. They want multimodality, right? They want a bunch of different things that pushes the model to be bigger. And so that you have to handle that. And three, people want customizations. People want to train their private data into it. They want to understand me. I want to create my own checkpoint. I want to create my own value. And so those three things are driving us to do the things that we do to run that future outcome much, much better, which is it's got to be able to handle fast, big,

• 20:42Speaker 00

  and it's got to be able to handle lots and lots and lots of concurrent checkpoints.

• 20:46Craig S. Smith?

  And you think that that there are going to be a few big players? You know, it's been Nvidia for a long time. Cerebrus is making noise. You guys have not popped up. You've been around, but but you're you have this new service. Are we gonna end up with ten different options?

• 21:04Speaker 00

  I don't think you're gonna have ten. It's a very capital intensive venture. So I think you have to be able to do and do it consistently. We're in our fourth generation hardware already, right? You know, in the market. And so you have to do consistently. But the world wants choice. There wasn't one choice and you you had the best products going to attract people because power is a real problem in this world. And if you don't have power, you don't have capacity.

• 21:29Craig S. Smith?

  Hi. This episode is sponsored by Rapid SOS. Terabytes and petabytes data is exploding in our daily lives. How many connected devices do you own? Do you ever wonder how all this data could be used in emergencies? By 2030, we'll have over 32 billion IoT devices worldwide, double what we have now. Despite this data abundance, there's a critical safety gap. Vital information. Isn't reaching emergency responders in time. This gap results in reactive rather than proactive emergency response,

• 22:12Craig S. Smith?

  putting lives and property at unnecessary risk. Many of us have had emergencies in our homes where we depend on EMS to respond quickly. Rapid SOS is closing this safety gap with their AI-powered intelligent safety platform. They connect lifesaving data from over 540 million devices to more than 21,000 public safety agencies across six countries. Despite 200 billion in annual safety system investments, enterprises often lack timely, actionable information during emergencies.

• 22:55Craig S. Smith?

  Rapid SOS is changing that. It's not just about avoiding problems. Safety investments are linked to increased customer satisfaction, employee retention, and long-term firm value. Close the safety gap and transform your emergency response with Rapid SOS. Visit rapidSOS.com/eyeonAI. That's eye on AI, E-Y-E-O-N-A-I, all run together. Visit rapidSOS.com/eyeonAI today

• 23:35Craig S. Smith?

  to learn how AI-powered safety can protect your people and boost your bottom line. That's rapidSOS, R-A-P-I-D-S-O-S, dot com slash eye on AI. Visit them today.