#251 Sid Sheth: How d-Matrix is Disrupting AI Inference in 2025

This episode is sponsored by the DFINITY Foundation. DFINITY Foundation's mission is to develop and contribute technology that enables the Internet Computer (ICP) blockchain and its ecosystem, aiming to shift cloud computing into a fully decentralized state. Find out more at https://internetcomputer.org/ In this episode of Eye on AI, we sit down with Sid Sheth, CEO and Co-Founder of d-Matrix, to explore how his company is revolutionizing AI inference hardware and taking on industry giants like NVIDIA. Sid shares his journey from building multi-billion-dollar businesses in semiconductors to founding d-Matrix—a startup focused on generative AI inference, chiplet-based architecture, and ultra-low latency AI acceleration. We break down: Why the future of AI lies in inference, not training How d-Matrix's Corsair PCIe accelerator outperforms NVIDIA's H200 The role of in-memory compute and high bandwidth memory in next-gen AI chips How d-Matrix integrates seamlessly into hyperscaler and enterprise cloud environments Why AI infrastructure is becoming heterogeneous and what that means for developers The global outlook on inference chips—from the US to APAC and beyond How Sid plans to build the next NVIDIA-level company from the ground up. Whether you're building in AI infrastructure, investing in semiconductors, or just curious about the future of generative AI at scale, this episode is packed with value. Stay Updated: Craig Smith on X:https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Intro (02:46) Introducing Sid Sheth (05:27) Why He Started d-Matrix (07:28) Lessons from Building a $2.5B Chip Business (11:52) How d-Matrix Prototypes New Chips (15:06) Working with Hyperscalers Like Google & Amazon (17:27) What's Inside the Corsair AI Accelerator (21:12) How d-Matrix Beats NVIDIA on Chip Efficiency (24:10) The Memory Bandwidth Advantage Explained (26:27) Running Massive AI Models at High Speed (30:20) Why Inference Isn't One-Size-Fits-All (32:40) The Future of AI Hardware (36:28) Supporting Llama 3 and Other Open Models (40:16) Is the Inference Market Big Enough? (43:21) Why the US Is Still the Key Market (46:39) Can India Compete in the AI Chip Race? (49:09) Will China Catch Up on AI Hardware?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We we work with TSMC today, that is our primary fab partner and a lot of the prototyping work we do is with TSMC. And you know, they have a lot of programs where you can try out your ideas on a smaller scale, where you can build smaller chips and test out your technology ideas and we have done a lot of that. We actually, for every generation of product, we've we've typically done a few test chips before we productized it. And they've been very supportive and it is very possible to do that. So that's that's kind of been our way of getting our products to market and testing out our technology before before we actually productize it.

• 0:34Craig S. Smith?

  This episode of Eye on AI is sponsored by the Definity Foundation. Definity Foundation is a Swiss not-for-profit that is home to some of the world's leading cryptographers, computer scientists and experts in distributed computing. Their mission is to shift cloud computing toward a fully decentralized state by supporting the Internet Computer, also known as ICP. If you don't understand anything about the Internet Computer, you can see my episode with Dominic Williams.

• 1:12Craig S. Smith?

  ICP's vision is that most of the world's software will be replaced by network resident software that's an evolution of traditional smart contracts. To achieve this vision, ICP is designed to make smart contracts as powerful as traditional software, all while remaining tamper-proof, unstoppable, transparent and verifiable. Since Definity's launch in 2016, they stand as Switzerland's most extensive blockchain research and development initiative and have been awarded more than 500 research grants worldwide.

• 1:53Craig S. Smith?

  Definity remains steadfast in its mission to drive the advancement of the decentralized internet. Network resident software can now be used to run AI models and RAG infrastructure, preventing them from becoming quote-unquote Hot wallets from where data can be stolen and increasing resilience. Furthermore, the technology has been designed to allow AI models to spin up and modify running web applications and internet services solo by addressing several key challenges. If you're interested in reading more about the internet computer, visit internetcomputer.org.

• 2:38Craig S. Smith?

  I also encourage you to listen to my episode with Dominic Williams in which he explains the internet computer.

• 2:46Speaker 00

  So thanks for having me on the podcast. You know, I've been a systems and semiconductor guy for many, many, many years. Started my career at Intel in the mid nineties.

• 3:03Speaker 00

  Actually, I spent the first 10 years of my career being a chip designer. And I, you know, spent a lot of time doing different, different portions of the design. Worked at Intel in the first five years that I was there.

• 3:23Speaker 00

  Worked on circuits, you know, bleeding out circuits for the Pentium processors, worked on, you know, system issues related to bringing up processors. After that, I went to a startup called Alaros, which was focused on building interconnect chips for Ethernet applications. So that was the next five years of my career. So really, the first 10 years of my career were spent in heavy R and D and really working on both processor chips and chips built for interconnects and networking. After that, I want to say, you know, I transitioned. I was, I was still at the startup and I was looking to transition into something that was more

• 4:09Speaker 00

  customer facing and moved into a business role, spent

• 4:14Speaker 00

  another five years working on different aspects of, you know, what it takes to take a product to market. Right? Really focused on go to market and. And I had an advantage because I was essentially taking the chips that I had helped build and working with customers to figure out how to deploy them and making them useful, so to speak, for those customers.

• 4:41Speaker 00

  After which I transitioned to a bigger role at a company called Infai. And Infai was in the high-speed networking and high-speed interconnect space. They were looking to break into the data center market. They had a really good, you know, footprint in the telco market, but were looking to break into the data center market. So I went there and incubated a business for them, which was only focused on data centers. And that was a huge success for the team. Spent about nine years building that business that eventually grew to two and a half billion in revenue from zero. And was acquired by Marvell in 2021, you know, ten billion dollar acquisition.

• 5:24Speaker 00

  I stepped out in, you know, 2019 to start D-Matrix because I was getting very excited by the inference computing opportunity. And, you know, I know it sounds very obvious today, but it wasn't very obvious back then, right? A lot of the narrative was about training and training bigger models and doing inference with, you know, image, image classification and image models were really the, you know, the, all the rage back then. And we quickly realized that it didn't make much sense for us to go do something with training because that was a market that was well capitalized, you know, a lot of investment.

• 6:05Speaker 00

  Didn't make much sense for us to go after the inference market in the edge that was focused on images and image classification.

• 6:15Speaker 00

  So we, we kind of were looking for a new entry point and that is how we started focusing a lot on language and transformers. And I'll talk more about that as we talk about the D-Matrix journey. But that's been my journey. I grew up in India in a family of entrepreneurs. Both my grandfathers were entrepreneurs back in the nineteen forty's, left home to start their businesses. My father was an entrepreneur too, left his home, came to the US, worked here, went back and started a business. So entrepreneurship is probably in my DNA somewhere. It was a matter of time before I, you know, stepped out and did something of my own.

• 6:57Speaker 00

  And here I am, taking my shot at this. We'll see where it goes.

• 7:03Craig S. Smith?

  Yeah, well, let's talk about Demetrius's products and how you

• 7:12Craig S. Smith?

  establish market share in a market that is so heavily dominated by Nvidia and the other big players.

• 7:25Speaker 00

  Yeah.

• 7:27Speaker 00

  So I think it's, you know, I've lived through the process of building a semiconductor business before.

• 7:34Speaker 00

  As I was, you know, stating earlier in my introduction,

• 7:39Speaker 00

  you know, I was, when I was at N5, we essentially incubated a business to enter the data center market, you know, building interconnects for data centers. And what we saw was, and I'll come to your question in a bit, but this this perspective is important because that answers the question that you just asked, which is in the semiconductor business, especially when you're building chips, you know, the very capital intensive, obviously, and take a long time, you know, multiple generations of product to get to success. And the only way to really be successful is to really intercept some kind of a discontinuity in the market.

• 8:20Speaker 00

  There has to be some kind of transition that is underway and you need to intercept that transition at the right time. So at Inphi, the transition that we intercepted, there's really two major transitions that happened at the same time. Was that data centers were going optical. You know, most data centers used to be based on copper interconnect. But they were transitioning to optical interconnect because obviously the speeds were getting to a point bandwidth was becoming very, very critical. And the higher the bandwidth, you know, the harder it gets to run signals over copper. So it was transitioning to optical and the other thing that was also happening at the same time was

• 9:01Speaker 00

  the signaling standard, which was used to be NRZ, which is the one zero standard was transitioning over to PAM based signaling. So four level signaling. So we were going from a two level signaling standard to a four level signaling standard. And so that required new electronics, new optics. And so you had this tech transition and a market transition happening at the same time. And that is what we intercepted very successfully. And you know, the incumbent back then were companies like Broadcom and Marvell and they were giants, right? But Inphi was successfully able to break into the market and dislodge some of the incumbents and build a very successful business doing that, right?

• 9:44Speaker 00

  So same thing applies for where we are today. We see, you know, major transition happening in the market again. Obviously, there is the whole transition from general purpose computing to accelerated computing, which is which is a mega, mega change that is happening across data centers. But within that accelerated computing, we are also seeing the market transitioning away from training to inference and inference becoming a very relevant workload. And that is also a tech transition because you need a fundamentally different architecture as you migrate away from training, which is predominantly a performance oriented workload.

• 10:25Speaker 00

  To inference, which is an efficiency oriented workload. So you can't really build a common platform that does both training and inference.

• 10:33Speaker 00

  You can, it's not like you can't, people have done it, but it's not gonna be, it's like building a Swiss Army knife, right? You're trying to do too much with the same thing. Building a dedicated inferencing solution, given that inference is going to be a much larger portion of the opportunity for AI compute, which means we think a dedicated solution for inference is is gonna win, right? And that's what we are focused on at D-Matrix. And that's why we think we can find enough share in the market to build a very valuable company.

• 11:06Craig S. Smith?

  Yeah, I have to ask,

• 11:09Craig S. Smith?

  I was, I live not far from IBM Research and

• 11:14Craig S. Smith?

  I was over there the other day talking to their head of

• 11:21Craig S. Smith?

  chip design and he was saying they have a

• 11:27Craig S. Smith?

  a fab, I think in Poughkeepsie where they do

• 11:32Craig S. Smith?

  pilots or or proof of concepts or whatever.

• 11:38Craig S. Smith?

  So, once you've designed a chip which which is a feat in itself, where do you go to to build a prototype and and see if it's working?

• 11:52Speaker 00

  So we do all our work, you know, we first of all, we are building a product that is pretty, you know, is built on standard CMOS process technology. So we are, we are not doing anything funky here with the process technology that requires us to go to boutique fabs and and, you know, build partnerships with with fabs that work on very exotic processes and process technology and whatnot, right? So standard CMOS logic process. So, you know, we can work with TSMC, we can work with Samsung, right? We we work with TSMC today. That is our primary fab partner and a lot of the prototyping work we do is with TSMC. And

• 12:38Speaker 00

  You know, they have a lot of programs where you can try out your ideas on a smaller scale. And what they call the the MPW program or a shuttle program where you can build smaller chips and test out your technology ideas. And we have done a lot of that. We actually, for every generation of product, we've we've typically done a few test chips before we productized it.

• 13:04Speaker 00

  And they've been very supportive, and it is very possible to do that. So that's that's kind of been our way of getting our products to market and testing out our technology before before we actually productize it.

• 13:17Craig S. Smith?

  Yeah. And and as far as the inference market goes, as I was saying before we started recording, I've spoken to Cerebras and San Benova.

• 13:33Craig S. Smith?

  You know, with with training, Nvidia has a lot because developers are used to using CUDA. And they they don't want to have to learn a whole nother programming

• 13:51Craig S. Smith?

  language, I guess, or

• 13:56Craig S. Smith?

  instruction set and and all of that. But with inference, developers are primarily using

• 14:10Craig S. Smith?

  hardware in the cloud. And, you know, through API, so they don't have to worry about building on on top of the chip. Is is that the opportunity as well that you see? And how does D-Matrix? I mean, I I had a conversation yesterday with Google Cloud asking about how these new accelerators get a footprint in the cloud. And I know San Benova

• 14:49Craig S. Smith?

  and Cerebras are all both building their own clouds until there's enough of an uptake that that the hyperscalers will will offer clusters of their chips. So how how do you how do you do that?

• 15:06Speaker 00

  Yeah.

• 15:09Speaker 00

  So, you know, we've, first of all, the team at D-Matrix has, you know, successfully sold into the hyperscalers before, right? We've been through, you know, call it ten, fifteen hyperscaler ramps with products, not, not at D-Matrix, but, you know, through our prior experience, right? So we know what it takes to ramp a product at a hyperscaler. We've been doing it for twenty years, right? When, you know, when Google first had a cloud offering,

• 15:42Speaker 00

  and not necessarily a cloud offering, but they were setting up infrastructure to power their own applications. They were, you know, setting up, you know, pretty enormous data center infrastructure back in two thousand seven, two thousand eight. We were building chips that would help them power that infrastructure, right? This was back, you know, many, many moons ago. So my, my, my startup from, you know, the two thousands when we were building chips to connect networking,

• 16:12Speaker 00

  you know, equipment. We were selling products into Google to, to help them power their infrastructure. So, and, you know, all the way from back then to the Inphi journey, where we had our chips sitting in pretty much every hyperscaler on the planet, all, all four big hyperscalers here in the US, three in China. So we have a lot of experience when it comes to deploying product at hyperscalers. We know the anatomy of what a typical hyperscaler ramp looks like. We know what the hyperscalers are typically looking for and how they engage, right? So at D-Matrix, we are kind of modeling our business along similar lines. We are, we are building, we're building product that would embed into the equipment

• 16:57Speaker 00

  and infrastructure that the hyperscalers are deploying. So we are not going to them saying, look, you need to rip and replace what you have to accommodate us. Right, we are looking at their equipment and we try to understand what they deploy and we try to fit inside what they already have. Without them having to rip out stuff to accommodate us. So really what we are trying to do is a cloud augmentation play, right? So we we can go and we build accelerator cards and maybe. I can show you the card while we are on the topic. So yeah, this this is. This is our six hundred watt. Um Corsair accelerator card. It's a you know, it's a PCI card. And you know, it's got two chips here that you can see.

• 17:42Speaker 00

  Uh each chip has, you know, four chiplets. So there's a collection of eight chiplets sitting on a single card that are, you know, connected to. Each other in an any to any configuration. And then if you turn this card around, there is a lot of DDR memory that is sitting at the back. Uh this is a pretty dense card. It's like a twenty four layer card. A lot of engineering that has gone into making this because we have a lot of. You know, interesting chiplet technology that goes on to on to this card that needs to be routed very efficiently. Um so that's that's, you know, the the guts of the card and then. You know, that gets packaged into into this accelerator unit. You see two cards there.

• 18:24Speaker 00

  Are put together um and then they are connected with a bridge. Uh on top, which allows every. Every card, every chip on every card to talk to a chip on the other card directly, right? So from a software perspective. This is treated like a single unit from a software perspective. So just as Nvidia has NVLink. Uh we have the ability to connect. All these chips sitting across these two accelerators uh into a into a single config from a software perspective. Right? So that allows us to unlock a lot of efficiencies. When we are running inference.

• 19:07Speaker 00

  So coming back to your question, I mean, that's that's what we ship. That's the card we ship. That's the unit we ship and that plugs into any AI server that is built by, uh, you know, folks like Dell or HP or Super Micro or, you know, Lenovo or call it, even if it's a custom server, sitting at one of the hyperscalers, uh, as long as it can house a PCIe card, we can plug into that socket and augment that server for AI inference. So that is our go-to-market model where we go to the hyperscalers and we tell them, hey, you've got a fleet of servers that currently don't have our hardware in them, but what if you were to put our hardware into the fleet of servers?

• 19:49Speaker 00

  We could augment that entire fleet with a solution that will completely change the economics uh, of doing generative AI inference. Now, wouldn't that be attractive to you? And they pretty much across the board love that, right? Because it doesn't require them to pull out and rip out stuff, but it allows us to integrate our solution into theirs and and then they can go augment that fleet with more capability.

• 20:13Craig S. Smith?

  And uh, talk a little bit about the uh, the technologies uh, involved in this chip. Like you're, you're putting the weights directly on uh, on SRAM, is that right? Uh, which as I recall is, is what uh, at least what Cerebras is doing. I think Samba Nova stacks

• 20:39Craig S. Smith?

  memory uh, on top of the chips. I don't remember, but uh, but the static random access memory has the, the model weights uh, directly uh, in them. Uh, can you talk about that? How that um, speeds things up? Uh, how it's more energy efficient? And then talk about um, the uh, chiplet architecture.

• 21:12Speaker 00

  Yeah, sure. So a few, few questions in that one question. So let me unpack some of that for the audience, right? So first of all, the chiplets themselves, right? So you, you see, this is, you know, a chip here. You know, there's obviously a lid that if we kind of pull that lid off, you'd see four chiplets inside a chip here and four here. So there's a total of eight chiplets sitting on this 600 watt PCI card. If I remember correctly, the only other 600 watt PCI card on the market is the Hopper H200. Blackwells are still not available in a PCIe form factor, still.

• 21:54Speaker 00

  I'm sure they will make those, but they're currently not available. So the only PCI accelerator from Nvidia, that's the latest and greatest, is the H200. Now, the H200 has about 800 millimeter square of silicon on a 600 watt card, right? We have across these two chips, 3200 millimeter square of silicon. So that's four times the amount of silicon that is sitting on this card at the same power envelope, in the same power envelope, 600 watts, compared to the latest and greatest Nvidia offering. And, and the reason we are able to do that is because of the chiplet technology and the chiplet approach.

• 22:34Speaker 00

  And obviously, a lot of design techniques that we have built into the chiplets that allow us to run the chiplets in a highly energy efficient manner. So we have, you know, almost 4X the amount of silicon on the same 600 watt power envelope, right? And so what's in that, what's in those chiplets, right? So obviously, we've got a lot of compute.

• 23:02Speaker 00

  But again, you know, the bottleneck is not compute, right? If you look at generative AI inference, the bottleneck is really memory bandwidth and then eventually memory capacity. And it's about finding the right balance between compute, memory bandwidth and memory capacity. And that's what we have strived for on this solution, right? So we have, um, obviously 10 petaflops of compute in on that card that I just showed you and 10 petaflops of, you know, four bit compute sitting on on that one card, which is, you know, plenty for now. And but the key thing is to optimize around memory bandwidth. Now, you talked about putting, you know, the weights in SRAM.

• 23:42Speaker 00

  Uh, we have a combination of our in-memory compute and SRAM uh on those chiplets. Uh, so there's a combination of, you know, call it uh across that one card. We have almost uh 2,000 in-memory computing cores. And we have a lot of SRAM also that is built into those chiplets. So we have almost two gigabytes of what we call high-performance memory. So this is memory that is sitting on the chiplets directly, but running at a very, very high bandwidth, you know, call it uh 150 terabytes per second of bandwidth. You compare that to HBM memory that is sitting at, you know, roughly about, you know, 4.8 terabytes or, you know, five terabytes.

• 24:26Speaker 00

  The black walls are closer to eight or 10, but the hopper that I, you know, was comparing our product to, because that's the only other PCI product, is sitting at about, you know, five terabytes, right? So there's almost like a 20x, uh 20 to 20, 30x improvement on memory bandwidth. Now that is where we kind of punch through the memory wall, right? Uh, the, you know, it's not about just having a lot of compute, but it is having a lot of compute and a lot of memory bandwidth to go along with it. So you can feed that compute very efficiently from memory and the memory is, you know, is is local. Now, Uh. There is about two gigabytes of that, which is, you know, typically not enough for a lot of the AI models.

• 25:05Speaker 00

  So we we scale this up further. So we'll have, you know, eight PCI cards sitting in a single server. And then we have eight servers sitting in a single rack. So we have a collection of sixty four cards sitting in a single rack. So sixty four times two gigabytes is about one hundred and twenty eight gigabytes of that high performance memory that is sitting in a single rack. With that one hundred and twenty eight gigabytes, you know, we can easily serve, comfortably serve seventy billion parameter models or hundred billion parameter models out of a single rack. Which is, for the most part, plenty for anyone who, you know, most enterprises, uh, most, you know, very high end

• 25:46Speaker 00

  enthusiasts, uh, many sovereign opportunities that we are going after, including some of the hyperscaler opportunities. A hundred billion parameter model sizes are pretty attractive. They can cover a lot of ground with that type of model. Now, there are frontier models that are much larger and we can certainly run the frontier models across multiple racks or even out of the capacity memory that I showed you. There is, you know, DDR memory sitting at the back, you know, we can run trillions of you know, trillion param- multiple trillion parameter models running out of that capacity memory. They would, they would run a lot slower, but you can run them out of that capacity memory. So we have the ability to do this, you know, you know, we have this flexible memory architecture where we can run a hundred billion parameter models in a single rack.

• 26:31Speaker 00

  Uh, or we can do more across multiple racks, but it's running at these extremely high speeds uh, because of the memory bandwidth problem that we addressed. And all this is possible while embedding our hardware in a customer's environment without having them having to purchase the whole solution from us, right? We can go augment their infrastructure with our solutions and give them the benefits of running what we call, you know, ultra-low latency batched inference. Right? So extremely fast while allowing the user to batch multiple, or the customer to batch multiple users.

• 27:14Speaker 00

  So really, that's, that's what we bring to the table. It's, it's a unique. Combination of this chiplet in-memory compute, you know, technology, highly energy efficient design, you know, how we connect the chiplets to each other, the numerics, you know, we have four bit numerics and eight bit numerics that are block floating point, which is now becoming a standard for inference. All of this goodness built into this one solution, one PCI card form factor that can go augment our customers fleets with, you know, and supercharge those fleets for generative AI inference, low latency generative AI inference.

• 27:55Craig S. Smith?

  And are most of your implementations on premise?

• 28:02Craig S. Smith?

  Or are they, is it this primarily a cloud solution? It seems with inference, that market growing and as large enterprises start using generative AI, that a lot of their, they're putting it in on premise because they don't want their data to leave.

• 28:29Speaker 00

  Yeah.

• 28:31Craig S. Smith?

  Is, is, yeah.

• 28:34Speaker 00

  Yeah. I mean, it's both. I think we are, you know, we are really quite agnostic, right? As, as I was stating earlier, we, our go-to-market is built on taking our solutions and augmenting our customer solutions, right? So we don't, we don't quite care whether the solutions are in the cloud or in the enterprise. But the journey, journey for most AI deployments is starting in the cloud, right? So I think our journey is starting in the cloud today. Whether it's sovereign clouds or, you know, what we call GPU clouds or Neo clouds, which is, you know, the smaller dedicated AI clouds, and then there is the hyperscaler clouds. So these are really the three focus areas for us today. And then, over time, this will migrate

• 29:15Speaker 00

  into on-prem enterprise clouds. And we'll migrate as the workloads migrate there. But right now, you know, the high-velocity deployments are happening in, in the clouds that are public clouds, and then eventually they will go to the private clouds over time.

• 29:34Craig S. Smith?

  Yeah. The, where do you see this going? I mean, as I said, I mean, Nvidia has such a...

• 29:45Craig S. Smith?

  A dominant position is the market opening up. I mean, as I was saying too, if you're in the cloud and this is for inference.

• 29:57Craig S. Smith?

  What matters is speed and cost.

• 30:05Craig S. Smith?

  Do developers care what chip is giving them inference? I mean, do they make that decision? Or is that a if it's in the cloud, is that a decision that's being made by the cloud provider?

• 30:21Speaker 00

  Yes, yes. So typically, I think you need a person who really understands the application that they are trying to accelerate. And what is happening is there's plenty of data that suggests that applications which are highly interactive with the user tend to be more successful, meaning the users are more likely to stay with an application that really interacts with them. Right? Anything that is more offline in the interaction, you know, doesn't tend to keep users around for too long, right? So anything that draws eyeballs is stuff that is highly interactive. So latency matters, right? Clearly. So, you know, you need to look at the application. I mean, there is obviously, you know, other constraints like cost and energy efficiency and whatnot, size of the model, how big is the model?

• 31:06Speaker 00

  And you know, you've got to kind of look at the whole picture to figure out, you know, what kind of underlying accelerator hardware could really allow for a better user experience and a, you know, better cost profile, right?

• 31:23Speaker 00

  So it's a combination of these things. So yes, the hyperscalers have the people, have the right expertise and the right, you know, folks with the right set of skills to make those calls. They can, they can match the right hardware to the right application. Right, and and then they can go deploy the and they also have the ability to go deploy new accelerators in their fleet to enable that that user experience. So, so you know, if if a new accelerator solution can demonstrate that we can improve the user experience which is latency, or it can we can improve cost, or we can improve things like energy efficiency by a factor of at least two to four X, that is a big win typically for a customer, right?

• 32:08Speaker 00

  So if you have the latest and greatest Nvidia hardware and you can show a two to four X improvement over that in that generation, that's a big win. That's a very big win, right? So, so again, for inference, I like to say it is, it has been hard to beat Nvidia on training. And you know, they've done a terrific job of keep, you know, you know, anybody who was a GTC earlier this week and saw the keynote, I mean, they have done a terrific job of advancing their roadmap. And you know, they have a roadmap that goes out, you know, four years and five years. And you know, it's almost impossible to kind of play catch up there, you know, with all the technology that they bring. But when it comes to inference, I think there is no one size fits all. You can't really take a big rack of black walls and just throw it at every influencing problem. It's not going to work. It's not, it's not like every

• 32:51Speaker 00

  single inference request looks the same. Inference request can be latency, you know, may need to be latency optimized, may need to be cost optimized, may need to be throughput optimized, right? So there is, there is no one size fits all. You can't say like here is one piece of hardware that's going to serve all of inference, right? It's, it's difficult to do that. So influencing fleets are going to be heterogeneous. There's going to be GPUs, there's going to be CPUs, there's going to be accelerators. And and that is, that is what we are focused on. We are not, we are not, you know, saying that we're going to be the best at everything, but we are really good at low latency batched influencing. So anytime you

• 33:33Speaker 00

  need multiple users where each user needs a really interactive experience, that's where you need to think of D matrix.

• 33:41Craig S. Smith?

  Yeah, you know, I spoke a couple of years ago, and I was just looking to see if they're, if they're still independent. But there's, there was a company rescale that has kind of a orchestration.

• 33:59Craig S. Smith?

  Technology that plugs into all the clouds and then you give them your your problem, your inference problem and and they automatically route you to whichever cloud has the correct chip or the correct

• 34:20Craig S. Smith?

  resources. Is that something that that did you work with people like that?

• 34:28Speaker 00

  Yes, yes, we work. So by the way, we work, we worked with rescale, you know, we know them well and you know, they but you're absolutely right. I think that kind of skill set, by the way, already exists at the hyperscalers. So you know, they're they're not they they either partner or they have the skills inside the company to build that orchestration layer. In their case, you know, they they are doing query profiling. They have a query coming from a user. They're trying to batch that query based on the profile of that query, right? Whether it's a query that is latency optimized query or is a throughput optimized queries or a cost optimized query. So you know, depending on the region it comes from, right? You may have to, you know, route the query to the right, right?

• 35:12Speaker 00

  You know, right data center and so on and so forth, right? So I think there is just a lot of things you need to kind of take into consideration before you decide where you want to send that query and

• 35:23Speaker 00

  so yes, that orchestration layer, I mean, Nvidia announced Dynamo at GTC this year, that is, that is again, an orchestration layer for inference queries, right? So yes, that is going to be very important because again, going back to the fact that inference is not a one size fits all, not every query is the same. So you need this orchestration layer that is going to be able to efficiently route queries to the right hardware. Right? And we are seeing a lot more of that and that is going to be where influencing hardware is more heterogeneous. It is not just all GPUs. Right? And that is, that is something that gradually, I think, will become clear to a lot of people, but it's pretty clear to us.

• 36:03Craig S. Smith?

  Does it matter which model you're using? And when you put the weights on the chip or on the PCI card, or however you say it,

• 36:21Craig S. Smith?

  that's a decision that you're making? Or that's a decision that the customer is making?

• 36:28Speaker 00

  It's typically done in conjunction with the customer, right? So the customer is trying to solve for a certain problem. And they work with us to figure out what's the best way to get there, right? So there's again, different vectors, right? There's things like, you know, the model size, there is a model quantization, there is a software stack, which is, okay, you know, my model is highly customized. I need to write, you know, my own custom kernels to, to, for that particular model. How do I bring those kernels to your hardware? Right? Oh, I want to focus a lot on latency. Oh, I want to focus a lot on cost, right? I mean, so there are all these different things that going to consider is cannot be done in isolation. It is a highly, highly,

• 37:15Speaker 00

  you know, it's typically a very deep engagement with, with a customer to kind of get these models deployed. Now, sometimes, you know, you have model like Lama 3, you know, call it 70 billion or 8 billion. And, you know, you want to deploy those models. It's a lot easier. And that's why you're seeing, you know, companies getting into that business, which is I'm just, I'm just hosting Lama models, right? Mistral, you know, DeepSeek, you know, these are all open source models and available and readily available. And they all look a lot like each other. I mean, they're all, you know, variants of Lama to a certain extent, variants of transformer models to a certain extent. So once you've done work on a few of them, you can actually incrementally take that work to others. Right? So, you know, it becomes a lot easier to host those.

• 37:57Speaker 00

  So you're seeing more of that, that becoming very popular, right? You know, we'll be doing that too. Of course, we are not going to get into the hosting business. That's not something we do. But we'll be enabling a lot of folks to host models of that, of that, you know, family, that variant with our hardware.

• 38:12Craig S. Smith?

  Yeah. As more of these models come out, how easy is it to, to load a model onto the chip? I mean, if you've got. You know, chips with Llama 3 on them and then suddenly DeepSeek comes out with something that blows Llama 3 away, which I guess it already has.

• 38:39Speaker 00

  Yeah.

• 38:40Craig S. Smith?

  How do you then switch? Uh, can you, how do you do that? Yeah.

• 38:45Speaker 00

  Yeah, so it depends again, you know, what that new model looks like, like DeepSeek R1 Distill, which is a distill version of DeepSeek, the reasoning model, right? It, uh, it is essentially, it looks a lot like Llama, right? It is, uh, so if you've done work on Llama 370 billion, uh, the incremental work needed to do, you know, bring DeepSeek R1 Distill onto your hardware is, is not, it's not like it's gonna take you, you know, you know, six months, right? It's gonna take you probably a few weeks, actually, doing that solution. Because the kernels look, you know, very similar. The model architecture, you know, yes, there are changes, but it's, it's, you know, for the most part, you can leverage a lot of the work you've done, right? Uh, so, so it is, it is a few weeks worth of work and you can onboard a new model, right?

• 39:27Speaker 00

  Um, so that's, that's really what it comes down to is, is what does the model architecture look like? You know, what, uh, you know, things like how many layers in the model, uh, uh, you know, are there any new mathematical operators that the model introduced so that you have to write new kernels? Right? Uh, what is, uh, how big is the model? Uh, what is the scale out strategy, right? What is the scale up strategy? How do you partition that model? Any kind of, you know, parallelism techniques that you need to apply? A lot of this work across the models we just talked about, uh, you know, can carry forward, right? So that means your incremental work is, is a lot less. Um, so, so yes, it, it really depends on, on, um, from model to model.

• 40:10Speaker 00

  Uh, I don't think there's a generic rule that you can apply across every single model.

• 40:14Craig S. Smith?

  Yeah. And, uh, uh, where do you see this going, uh, as, as models proliferate? Uh,

• 40:26Craig S. Smith?

  are, are chips gonna proliferate as well? Uh, and, uh, is the inference market growing so fast that there's plenty of room for, for everybody? Or is there gonna be a shakeout at some point, uh, where, I mean, the hardware business is expensive and?

• 40:47Speaker 00

  Yep.

• 40:47Craig S. Smith?

  Yeah.

• 40:51Speaker 00

  You know, I think the market is big, right? So the opportunity is massive. I don't think there's been a creation of an opportunity like this one in our lifetimes, right? So the the inference computing opportunity, I mean, if you believe what Jensen said at the GTC, I mean, it's in almost ten times larger, there's hundred X more compute needed, the opportunity itself is ten times larger, right? So if the training opportunity was a hundred billion, then the influencing opportunity could be one trillion, right? I mean, let's, you know, let's scale that down a little bit. I mean, you know, even if it is a two to three hundred billion dollar opportunity, right? That's massive. You know, that's massive. I mean, you're looking at the creation of a

• 41:31Speaker 00

  the entire semiconductor market is, you know, five hundred billion. So you're looking at the creation of another three hundred billion dollar opportunity just in the next ten years, which which is going to support a lot of, support a lot of different company. And we talked about inference not being a one size fits all. So it's not like there's going to be just one solution that runs away with the price, right? Um. So yes, there will be multiple players. Uh. The question is, it's not going to be as many as we have today. Right? Uh. So there'll be, there'll be uh some kind of consolidation that'll happen over the next, you know, call it five years, maybe. Um. But there's going to be multiple winners, uh, because first of all, the market, just the inherent nature of inference is such that you can't really build a one single chip that covers all your needs.

• 42:14Speaker 00

  Um. And that is going to allow other companies to break out. And um and that would be kind of the healthy outcome anyway. I think, you know, in the semiconductor industry, we've not had the creation of a yet another large semiconductor company in the last twenty plus years, right? Uh. Because the industry went through a lot of consolidation and uh, you know, this, the return on semiconductor investments was not very attractive, so not many companies got funded. Um. So we've not had a, we have had a very long gap where we've not had any company really break out. So I think this This platform shift, this AI platform shift is going to drive that and this is going to lead to the creation of the next Nvidia, the next Broadcom, the next Marvel, call it.

• 42:58Speaker 00

  And I would expect that to happen, right? And it would be good for the ecosystem.

• 43:03Craig S. Smith?

  Yeah. Are you, you know, TSMC is building this fab in Arizona. I think they're going to start with four nanometer. And eventually get to two is, are you going to be

• 43:17Craig S. Smith?

  manufacturing in Arizona when they're up and running?

• 43:21Speaker 00

  Yeah, probably. I mean, again, you know, the way TSMC is also going to have this, you know, similar fabs in Taiwan. Um, so they're going to have a four nanometer in Taiwan and

• 43:33Speaker 00

  and the pricing structure could be different, just given the cost of labor between Taiwan and the US. Uh, so we'll, we'll take a look at it and we'll, right now we run everything out of Taiwan, which is, uh, we are at six nanometers at TSMC and that's, you know, kind of exclusively run out of Taiwan. Uh, product roadmap goes to four and then eventually to two and we'll have the option of running out of Arizona. We don't have to take that up. We can always still keep running out of Taiwan, but we'll have to just see. You know what makes most sense from a business perspective at that point in time and then make a decision.

• 44:05Craig S. Smith?

  Yeah. And your market is, uh, I would guess for the time being primarily in the United States, but I ran into Andrew Feldman and, and, uh, Rodrigo um, Yang in, um, Riyadh, Saudi Arabia, um, last fall. And they were both there, obviously, uh, we were at a conference, but they were

• 44:34Craig S. Smith?

  talking to Aramco and, and, uh, some of the, the big, uh, inference companies or companies looking for inference. Uh, how much of your market is outside the United States and where would it be concentrated?

• 44:50Speaker 00

  Yeah.

• 44:52Speaker 00

  So yes, we we are working with customers outside the US.

• 44:57Speaker 00

  You know, we have we have customers in the Middle East, we have customers in the APAC region.

• 45:04Speaker 00

  We have customers in Japan, right? So we are working with customers outside of the US. We, you know, we do believe that our primary market will be in the US. Because that is really where, you know, most of the high-volume deployments are happening. And we talked about our go-to-market model, it's a little different from the companies you mentioned, right? We are focused more on embedded, embedding our hardware into a customer's environment, right? We, and so it's a volume-driven business for us. And for that, you know, by that token, we are looking for, you know, very large fleets of server, AI server hardware that can be augmented with our solution.

• 45:47Speaker 00

  And that typically today, at least, given that China is off-limits with all the trade, you know, the trade tension and trade wars, for now, we can't ship our product into into China.

• 45:58Speaker 00

  So that means the US market is going to be a huge focus. The markets I talked about, APAC and the Middle East, still don't have the ability to house our product. They're looking for total solutions. So our our dependence on those markets is going to be limited for some time.

• 46:18Craig S. Smith?

  Why, or or maybe I'm just unaware of it, but I mean, not including China, why are there not

• 46:28Craig S. Smith?

  chip companies sprouting up in in India, for example, or are there that are competing with you guys?

• 46:38Speaker 00

  They are, they are, they're they've begun to spring up. I mean, there's a there's a couple of reasons. One is, I think, just the perception. Is that the software ecosystem in India is a lot stronger than the hardware ecosystem. And that is true, right? There's just a lot more software talent in India than there is hardware talent. But there is good hardware talent, right? And but the startup ecosystem around hardware in India is not that robust yet, right? So there is not enough capital to feed these companies. There is, the markets are not in India.

• 47:16Speaker 00

  The markets are predominantly in the US and, you know, China or Japan or elsewhere, right? So, so it's hard for a company to establish itself and say, I'm going to build product for international markets and be successful, right? Because there is always a company that is already doing that in the US for the US markets, or there is a company in China doing it for the China market, right? So then, building a company in India to serve those markets when the startup ecosystem for hardware is already not that robust is, is making, it makes it very, very challenging.

• 47:56Craig S. Smith?

  Yeah, you know, China has done a lot since the chip, I mean Huawei has done a lot. And it was kind of the leading edge of the chip embargo. And China has a lot of talent, it has a lot of hardware talent, and they're working very hard at developing their own ultra, what is it? The UH, I'll cut this out, but the lithography machines needed to etch, you know, six, sub seven nanometer chips.

• 48:41Craig S. Smith?

  Are they going to be a player in this market eventually, or are they already?

• 48:48Speaker 00

  Are you talking about the EUV machines?

• 48:52Craig S. Smith?

  Ultra high.

• 48:55Speaker 00

  Extreme, extreme ultraviolet, extreme ultraviolet, yeah, yeah, yeah. So, and your question is, are they going to be a player building in the, building those machines?

• 49:05Craig S. Smith?

  In, in, in providing inference chips to the global market.

• 49:09Speaker 00

  Oh, inference chips to the global market, okay, got it. Um, to be seen, to be seen, I think, um, I mean, there's just a lot of, um,

• 49:20Speaker 00

  you know, embargoes right now on China with, you know, getting access to the latest and greatest technology. So it's hard for them to really keep pace with the US on, on a certain class of technology and influencing chips. Um, so, yeah, it's difficult to say whether they will, uh, we, you know, they will be able to keep pace with innovation in the US. Um, I think to me, you know, the US is currently on the forefront of, of, uh, of, you know, just the latest and greatest techniques when it comes to chip design, right? Chip manufacturing, chip, uh, not necessarily manufacturing, but chip design specifically and architecture. And we just have a very robust ecosystem of engineers and customers and partners who put it all together, right?

• 50:05Speaker 00

  Um, and by the way, but even, even before these, you know, embargoes came and the trade tension escalated, right? Uh, I think the US always had a lead when it came to building cutting edge chips. Um, so it is independent, by the way, of, of, uh, what's happened in the last, call it, you know, eight years or 10 years, right? Um, so, um, and it's only, it's only, the gap is only widening now because, because of the trade tension. So, so I would, I would argue that the US, you know, will, will be the one to produce, uh, bleeding edge, cutting edge, influencing chips, you know, at least for the next five to 10 years, right, for the global market. And it'll be difficult for other markets to really, really keep up.

• 50:49Craig S. Smith?

  Yeah, yeah. Okay, well, we're almost up to an hour. Is there anything I haven't asked that that you think listeners should know?

• 50:59Speaker 00

  Yeah, I mean, you know, D-Matrix is a great place to come work.

• 51:07Speaker 00

  So if anyone in your audience is looking to work on really bleeding-edge, you know, inferencing chips where we kind of are solving some really hard, you know, technical problems over both software and hardware, right? You know, you know, compilers, infrastructure software, you know, runtime systems, software scale out, scale up, chiplet technology, chiplet interconnects, compute technology, packaging, a board design. I mean, come talk to us. We'd love to. We'd love to talk to you and we are growing the team. And, you know, we are the only only company that has a purpose built,

• 51:53Speaker 00

  generative AI inferencing solution built from the grounds of foreinference. It's not like we started the company doing something else and then pivoted into this opportunity. We built this company for for generative AI inference, right? Just given the timing of when we started, it worked out that way. So, you know, we we are super excited about the opportunity that we are growing into. And this is, you know, your chance to come work on something super exciting.

• 52:19Craig S. Smith?

  This episode of Eye on AI is sponsored by the Definity Foundation. Definity Foundation is a Swiss not-for-profit that is home to some of the world's leading cryptographers, computer scientists, and experts in distributed computing. Their mission is to shift cloud computing toward a fully decentralized state by supporting the Internet Computer, also known as ICP. If you don't understand anything about the Internet Computer, you can see my episode with Dominic Williams, ICP's vision.

• 52:58Craig S. Smith?

  Vision is that most of the world's software will be replaced by network resident software. That's an evolution of traditional smart contracts. To achieve this vision, ICP is designed to make smart contracts as powerful as traditional software, all while remaining tamper-proof, unstoppable, transparent, and verifiable. Since Definity's launch in 2016, they stand as Switzerland's most extensive blockchain research and development initiative and have been awarded more than 500 research grants worldwide.

• 53:38Craig S. Smith?

  Definity remains steadfast in its mission to drive the advancement of the decentralized internet. Network resident software can now be used to run AI models and RAG infrastructure, preventing them from becoming "hot wallets" from where data can be stolen, and increasing resilience. Furthermore, the technology has been designed to allow AI models to spin up and modify running web applications and internet services solo by addressing several key challenges. If you're interested in reading more about the internet computer, visit internetcomputer.org.

• 54:23Craig S. Smith?

  I also encourage you to listen to my episode with Dominic Williams in which he explains the internet computer.