Why People Are Paying 10x More for AI - and What That Means for the Chip Market | Sid Sheth, d-Matrix

The AI chip market looks monolithic from the outside - NVIDIA dominates, and everyone else is fighting for scraps. But d-Matrix's CEO Sid Sheth argues that the market is quietly splitting into two distinct tiers, and the one that's exploding right now is the one NVIDIA's architecture isn't built for. In this episode, Sid joins Craig Smith to explain the "premium token economy": a new class of AI inference where interactivity is the product, users pay ten times more per million tokens for instant responses, and the memory bandwidth limits of GPU-based systems create a structural ceiling that purpose-built architectures don't have. The conversation is unusually candid about what AI actually looks like at the executive level: Sid describes using Claude as a sounding board for M&A strategy, producing full integration plans in 15 minutes that used to require entire banking advisory teams, and watching AI shift from a tool that echoed his ideas back at him to one that genuinely disagrees, flags what he missed, and pushes back with enough confidence to be useful. He also makes the case that we're at the beginning of a shift from individual agents to what he calls "organizational AI" - teams of agents running entire company functions at a high level of abstraction - and that the infrastructure bet d-Matrix is making positions them directly in the path of that wave. Subscribe to Eye on A. I. for weekly conversations with the people building and deploying the future of AI.

• 0:00Speaker 00

  People talk about organizations being entirely run by agents, right? How far are we from that? Eventually, AI could run large portions of most companies.

• 0:07Speaker 01

  Do you see emerging companies, startups that are agent-first? The New York Times had a piece recently about a guy and his brother running a $1.6 billion sales company that's basically an agent-first company. Do you see that segmented the economy and growing?

• 0:25Speaker 00

  It's really an opportunity for everyone to go into something new. I think at the D-Matrix, we are being very proactive about the use of AI across the whole organization for various different use cases. We have an AI team in the company that is entirely focused on going AI across the whole country, right? So it is not an option, it is mandated.

• 0:43Speaker 01

  Tell us what you're talking about at HumanX. I mean, I read that you've had some acquisitions since the last time we spoke. You're moving into a larger system, not just providing chips. And I'm interested in that. You know, my interest is generally in algorithmic research, but I do try and follow hardware. You guys are a player in that space. I think I told you last time, I talk periodically to Andrew Feldman at Cerebrus, to Rodrigo Yang at Cenbonova, I haven't spoken to Brock.

• 1:25Speaker 01

  If I'm not mistaken, those you and those three are the primary players in this new inference ship space, or maybe not. I mean, if you could give me an overview on what you guys are doing and how you're differentiating.

• 1:43Speaker 00

  The influencing space specifically, right, is kind of bifurcating, right? Because I think everyone looked as inference as one, you know, kind of, you know, singular entity, right? But I think there is lots of nuance and when it comes to inference, it's not a one-size-fits-all.

• 2:03Speaker 01

  Yep.

• 2:03Speaker 00

  Something we've been saying for a long time.

• 2:08Speaker 00

  So, you know, depending on where you are doing the influencing, what your target markets are, what applications you're going after, you can't really build a single chip that serves all of influencing's needs, right?

• 2:24Speaker 00

  And so you've got to really kind of break it down. And not every company is playing in every pocket of the market, you can't, right? GPUs, you know, just because they are so general, the ecosystem is so broad, clearly GPUs can go into many different applications for inference, right? But they're not going to be very efficient because of that, right? And so, yes, they're very broad in their applicability, their general purpose, but

• 2:54Speaker 00

  when it comes to certain breakaway applications where applications need a very specific metric fully optimized, then GPUs will not do well in that market. And I think, so I think the way to look at it is, okay, you have the influencing market, it's going to be the largest part of AI compute.

• 3:13Speaker 00

  It's expected to be over a trillion-dollar market, you know, call it in the next five years.

• 3:19Speaker 00

  But it's, you know, then GPUs will play in that market clearly, but there's going to be portions or segments of that trillion dollars that is, we will need, you know, lots of optimization, a highly optimized silicon. And the one that is truly breaking out right now is the segment that is called low-latency compute. And low-latency compute, low-latency inference is specifically around applications that need high levels of interactivity. So, you know, a great example would be Claude Code. This happened, you know, a few months ago and you have people who are not programmers who are now using Claude Code and are great programmers, right?

• 4:02Speaker 00

  I mean, you know, the joke is, right, the new programming language is English, right? And everyone can program and you can be an ace programmer and know how to prompt the model and how to use it.

• 4:15Speaker 00

  So, but that has led to people wanting to interact. And the faster the model is when it responds back to the user, the more likely the user is to stay with the, to with the application, right? So I think interactivity has become very important and companies are beginning to charge more for those high levels of interactivity. So you can always say, look, I don't want that level of interactivity, in which case, you know, I'll pay, call it $2 for a million tokens. But if I want that high level of interactivity, I'll pay $20 for a million tokens and people are willing to pay that $20, they want that interactivity. So I think, given that this new tier of influencing is emerging, this new tier of tokens is emerging, right? Where we call it a premium token economy.

• 5:00Speaker 00

  So we'll look at the entire token economy, which is inference, this is the premium token economy.

• 5:07Speaker 00

  Not many companies play in that segment because you need a specific compute architecture that is well-suited for the fast token generation, right? So it's about generating tokens very quickly and that typically means you have an architecture that puts compute and memory together in a way where you have order of magnitude more memory bandwidth available to your computer than a GPU would have, right? Because GPUs are based on HBM and they, you know, have a certain cap on the amount of memory bandwidth. Solutions like D-Matrix, and you mentioned, you know, Grok and Cerebras are the other, have architectures that allow, you know, compute to get access to very fast memory.

• 5:52Speaker 00

  And it's an order of magnitude faster than HBM. So that is the breakaway category right now and D-Matrix falls into that breakaway category. And because that category has exploded on us literally in the last few months, we're just trying to keep up. And I think we need to scale up into that opportunity rapidly.

• 6:15Speaker 01

  Scale up meaning manufacturing?

• 6:17Speaker 00

  Manufacturing, just delivering on product, deploying product faster, supporting product faster, you know, making more of our product, all of these need to happen, bringing on more people to help, you know, support more customers. All of this needs to happen simultaneously and that is why we did this acquisition of a company called GigaIO, which allows us to scale into, you know, faster deployment of servers, faster deployment of racks.

• 6:45Speaker 00

  We actually had started a rack scale journey last year, right? We announced a product called Squad Rack in October of last year at OCP.

• 6:54Speaker 01

  You can explain what rack scale means?

• 6:56Speaker 00

  Yeah, sure. So rack scale is, you know, you know, you have, you have, if you go into a data center,

• 7:05Speaker 01

  Yep.

• 7:05Speaker 00

  You know, you have these aisles or rows of hardware and the unit of compute deployment in that data center is typically a rack, right? And racks are then put together into what they call a cluster. Then the clusters are kind of put together into, you know, so, you know, into, you know, eventually multiple clusters will form a data center.

• 7:31Speaker 00

  They have, you know, sub, you can actually break it down into pods. I mean, different people use different terms, right? So there's racks and pods and clusters and eventually you kind of scale up to the data center, right? So the unit, but the unit of deployment in a data center is a rack.

• 7:46Speaker 01

  Yep.

• 7:46Speaker 00

  And these days, I think, just given how quickly AI computing is, you know, the demand is just shooting, you know, through the roof. So I think you need lots of compute and the unit, everyone is looking of compute deployment in terms of racks, right? So I think you need to be

• 8:05Speaker 01

  In their own racks.

• 8:06Speaker 00

  In their own racks or even in data centers, right? I mean, like you go talk to a customer, any customer, hyperscale customer or a NeoCloud or a Frontier Lab and they say, okay, how many racks of compute can you deploy, how quickly? That will be the first question they ask you. They're like, okay, D-Matrix, how many racks of compute can you deploy for me and how quickly? Because then they can do the math in their head, you know, one rack is about 100 kilowatts, you know, depending on, you know, everybody has different ratings.

• 8:33Speaker 01

  Sure.

• 8:33Speaker 00

  Somebody has 50 kilowatt racks, somebody has 100 kilowatt racks, somebody has 150 kilowatt racks. So depending on the customer, they can make the math, they can run the math very quickly because they're like, yeah, you know what, okay. So D-Matrix, you're telling me you can do like 100 racks in three months or five months or whatever the number is. 100 racks times, you know, 100 kilowatts, this is the amount of power and this is the amount of compute capacity I need, which is measured and typically measured in terms of power. So it's so many megawatts of compute or so many kilowatts of compute or typically it's always megawatts. Now you're talking gigawatts, right? So they're kind of working backwards from there. They're saying, okay, I need X gigawatts or X megawatts of compute that translates into X number of racks and so the unit of deployment has become a rack.

• 9:14Speaker 00

  And so I think rack scale is just, you know, is kind of very prevalent unit in terms of, you know, how quickly you can deploy compute.

• 9:26Speaker 01

  Yeah.

• 9:26Speaker 00

  And so that's, that's really why we

• 9:29Speaker 01

  Yeah, let me ask, because a hyperscaler, they have their own racks.

• 9:33Speaker 00

  Yeah.

• 9:33Speaker 01

  And you were selling previously packages or

• 9:39Speaker 00

  Cards, yeah.

• 9:40Speaker 01

  Cards, cards packaged into, as I recall, two cards in a, you know, what do you call the unit?

• 9:48Speaker 00

  Yeah. We had trays and cards, right?

• 9:51Speaker 01

  Trays and cards, okay. And then the hyperscaler, if they were going to have D-Matrix, I'm sure they'd buy a bunch of those.

• 9:59Speaker 00

  Yeah.

• 10:02Speaker 01

  Cards or and and they would slot them into their racks. What's the advantage of you having your own racks? Because the hyperscaler, I mean, how does that work?

• 10:13Speaker 00

  Yeah. So I think two things, right? One is that model has not changed from our perspective, from the point of view of the hyperscaler. Keep in mind, even when we work with the hyperscalers or the big neo clouds, who also tend to buy trays and cards and will deploy, deploy their own racks. You still need to understand rack scale. As a company, we need to understand because you're talking to people at the other end who understand rack scale. Or so you can't be like, look, we don't understand any rack scale, but you know, we leave it to you. No, they they expect you to understand. We, at the end of the day, have to help them deploy our solutions into their racks. If anything goes wrong, we need to have rack scale people at our end who can help them.

• 10:53Speaker 01

  I see.

• 10:54Speaker 00

  So I think there's an expertise and a skill set gap that we are addressing with this in our company.

• 10:59Speaker 01

  I see.

• 10:59Speaker 00

  So that's one piece. But then there is another type of customer that we're going after, which is more the sovereign customers. Longer term, it'll be the enterprise customers who really want racks. They want whole racks to be deployed. Even there, we are partnering with Super Micro today to deploy those racks, right? So it's not like we are going to build our own rack. We're going to have our partners build the racks. But somebody needs to spec out the rack. Somebody needs to design the rack and we would have to do that for our hardware, right? And then Super Micro can go build it to spec. But you still need to have the rack scale engineers. Now, that whole process has been accelerated is all I'm saying is like the deployment process has just got accelerated because of what we talked about earlier in the conversation, which is the low latency computing opportunity has exploded on us and we have a solution that is well suited.

• 11:44Speaker 00

  So that whole process of taking our solution from where it is today to deploying it in a data center, whether it's a sovereign data center or a hyperscale, it doesn't matter. We need to have the rack scale people at our end, the system scale people at our end, who can accelerate the process of deploying the product. And that is what the.

• 12:00Speaker 01

  I mean, because when I read about it, it sounded to me, or I misunderstood that you're, that it's a hardware that you're moving from simply producing cards to producing racks and then, I don't know, having your own clouds with your own racks or providing the racks directly, but you're not building racks. It's it's the expertise that that you require. Yeah. Is everybody doing this?

• 12:32Speaker 00

  Uh, not everyone, not everyone, because I would say,

• 12:39Speaker 00

  depending on your business model, right? I think in our case, our business model was, and I'm assuming your question is, is every other chip company looking to acquire rack scale expertise? The answer to that is yes. I would say pretty much every company is looking to acquire, but not every company is looking to build or partner the way we build or partner, right? I think a lot of the other companies that you mentioned earlier, they're building their own racks. I mean, in many cases, they sell the whole, they sell a rack, they sell.

• 13:06Speaker 01

  I see.

• 13:06Speaker 00

  Their unit of sale is a rack. In our case, the unit of sale is still a card or a tray. We partner with, you know, folks like Super Micro who would then sell the rack, right? And we would revenue share with them or there would be another business model. Or in the case of a hyperscaler, we would not even worry about selling a rack because they build the rack, right? So.

• 13:26Speaker 01

  Yep. It's fascinating, this breakout premium token market. I haven't heard it described that way. Are there other examples beyond coding that that you can give that that's driving that market?

• 13:44Speaker 00

  I mean, OpenClaw? I mean, what happened with OpenClaw end of last year? The Agentic, we haven't even seen the true fallout of that. But I mean, you've seen a lot of Agentic tools now being unleashed, right?

• 14:00Speaker 00

  Claude is, you know, kind of incorporating a lot of Agentic capabilities, right? Pretty much a lot of, pretty much every enterprise class software tool is trying to incorporate Agentic capabilities. But I think to me, two things, right? One is Claude code and the significant improvement in capabilities there. And then the OpenClaw. So this is Agentic communication, machine to machine communication, where machines are kind of, you know, you can spawn multiple agents across multiple virtual machines and a task is, you know, broken down into multiple agents. And these agents are talking to each other, you know, essentially orchestrating all all the different work or the workflows that you have running on your computer.

• 14:40Speaker 00

  And now that is, that is very latency sensitive communication because, you know, machines are just waiting for other machines to finish tasks, right? And you don't want that to happen for too long, so latency begins to matter.

• 14:53Speaker 01

  Yeah.

• 14:54Speaker 00

  Right?

• 14:54Speaker 01

  And and then something like OpenClaw, it's also writing code on the on the.

• 14:58Speaker 00

  Right, yeah, yeah. It could spawn an agent to write code. It can spawn Claude code to go write some code for you on the side, right? So.

• 15:04Speaker 01

  Wow, that's amazing.

• 15:06Speaker 00

  Yeah.

• 15:07Speaker 01

  What about edge compute? It seems to me that that this would apply the latency question. Are you moving at all into into the edge where latency is so?

• 15:22Speaker 00

  We could, I mean, you know, the big, absolutely, we could. We are not currently. The platform, the way we have built it is a platform that is built with, you know, like a Lego block approach, right? We have these chiplets that are the Lego block and then we can, we can scale the solution up. We can, you know, put many chiplets together into a very large solution. So for the cloud market and the data center market, we, you know, essentially take our chiplet approach and we package, you know, close to, you know, eight of them on a single card. And we have sixty four cards in a rack.

• 16:03Speaker 00

  So, you know, we call rack scale, you have sixty four cards each with eight chiplets, right? So that gives you an idea of how many chiplets we sprinkle through the rack, right? But you don't have to, I mean, if you went into an edge market application where you're just doing, say, for example, if it's a robot or it's a, you know, physical AI application or autonomous, you know, right, driving, right, where you need AI capabilities. You don't need that many chips. You might need, you know, like a few chips and we can actually do that because of the Lego block approach. We just scale it down, sprinkle fewer Lego blocks into a physical AI application or an autonomous application. And the software would not change. The software was essentially built to scale with more chiplets or less chiplets, right?

• 16:48Speaker 00

  So the platform has been built today to go into the, you know, edge market.

• 16:54Speaker 00

  However, the edge market is very fragmented. So everyone has a slightly different software stack and, you know, each vertical, I mean, like if you go to the physical AI vertical, the software stack looks different. Autonomous vertical software stack looks different. Or industrial, the software stack looks different. So we just don't have the scale today as a company to go address all these markets while we are trying to address the data center market. So our goal is to, you know, stay focused on the data center market. Then at the right time, when the company has scale, we will branch out into other markets.

• 17:21Speaker 01

  Yeah. Did you foresee this this emerging market with coding and agent to agent communication?

• 17:31Speaker 01

  And if so, what do you foresee coming?

• 17:35Speaker 00

  I wish I could sit here and say, you know, I I saw it all, I envisioned it all. No, I think that some of it we did, some of it was luck, obviously. And the luck is really about, you know, how quickly a lot of this stuff happened and how, how useful.

• 17:50Speaker 01

  Yeah.

• 17:50Speaker 00

  A lot of this stuff became so quickly, right? So I think that that has certainly caught even even us by surprise, right? I mean, how quickly this is maturing. And this is because AI is learning fast, right? So AI is learning fast and it is, AI is actually learning to help itself, right? I mean, so there's an iterative recursive loop, whereas AI gets smart, it learns to build tools that it can use in more efficient ways and more capable ways, right? So I think you have this kind of very quick, you know, positive feedback loop that is, is, you know, a virtual cycle that is underway. And that's why you're seeing these impact, you know, the impact is just so rapid, right?

• 18:32Speaker 00

  But, you know, we had a broad idea, right? We said, okay, the bet was on inference first, you know, when we started the company. And then, and then we said, you know, inference is going to be the big application. But then, you know, what, you know, it's not just going to be about inference. It's going to be about, you know, interactive inference. And then it's not just about interactive inference, but it's about spawning machines that can take all of that interaction and do something with it, right? So I think we kind of had a broader vision of how this could evolve. And then, but, you know, it's even surprised us and pleasantly surprised us at how it's all come together so quickly.

• 19:07Speaker 01

  Yeah.

• 19:07Speaker 00

  Right? And I think what we would be most excited by next is.

• 19:13Speaker 00

  I think we have seen agentic capabilities unleashed. I think we haven't yet truly seen organizational capabilities unleashed, right? So meaning you have human spawning agents and augmenting humans, but can you create teams of agents? And so the abstraction of the task will keep going higher and higher. Right now, the tasks that are getting abstracted are like, okay. I have, you know, I want to make, I want to extract this data from this database, prepare a PowerPoint presentation based on the data for a board meeting that I have in a few, in a few hours, call it. And that can, you know, the agents can go spawn up and do a task like that. Now, imagine you start abstracting to a higher and higher level.

• 19:57Speaker 00

  Everybody say, wait, it's not just about, you know, mundane tasks that I want to get done on a day to day basis. Imagine if you could describe a task at an extremely high level, like, look, I have a customer that I want to win. Right? And you go figure out who the decision makers are, what product, what my product needs to be changed to. And so this thing comes with the whole strategy, like what an entire sales team and a marketing team would essentially do collectively. These agents could go spawn off and create a whole strategy plan. So I think the tasks will get more and more abstracted as you go higher. And very soon, I think, you know, people talk about organizations being entirely run by agents, right? And so I think where are we and how far are we from that? We don't know yet, but I think we are going to levels of abstraction here that,

• 20:42Speaker 00

  you know, eventually AI could, you know, run large potions of most companies, right? Quite reliably.

• 20:50Speaker 01

  Let me ask first how our D-matrix is allowing agents.

• 20:55Speaker 00

  So we are, we just rolled up, we have an AI team in the company that is entirely focused on rolling AI across the whole company, right? We just launched Claude Code to the whole, whole organization.

• 21:10Speaker 00

  We have everyone who has access, we are mandating that everyone uses AI, finds ways of using AI and shares any new tricks or tips that they find with the rest of the organization, right? So it is not an option. It is mandated. This is across the hardware team, the software team, the operations team, the CTO teams, the marketing teams, the corp com teams, everyone, everyone is going to be using AI. Right? And not everybody really knows what they will do with it, but they have to start using it. And the moment you start using it, you know, the use cases will emerge, right?

• 21:43Speaker 01

  Right.

• 21:44Speaker 00

  So yeah, no, I think at D-matrix, we are being very proactive about the use of AI across, across the whole organization for various, various different use cases.

• 21:53Speaker 01

  And Claude Code is one thing, Open Claw is another. I'm, you know, not a coder, not a technologist, I'm a journalist, but I, I started using Open Claw through, you know, My Claw is the interface I use. And it's incredible. And I, I sort of imagine if I'm doing it, you know, I'm seventy years old, I'm a retired journalist. It, it must be happening all over the place. Do you have an agentic framework that, that, I mean, beyond Claude Code that, that you have the organization

• 22:37Speaker 01

  using to, to spawn agents?

• 22:41Speaker 00

  Yeah. So Open Claw, I mean, we don't use Open Claw because it's an open source framework. So there's obviously security concerns around the, there could be, I mean, we don't know what the agents are doing behind the scenes yet, right? There is a certain level of transparency that's missing, but

• 22:59Speaker 00

  we use enough Claude cowork. So, you know, now Anthropic is incorporating all the same agentic capabilities that Open Claw has into, into Claude cowork. So we have Claude cowork enterprise, Claude code enterprise, right? So yeah, absolutely. We have this agentic layer that comes along with the Claude tools that are available to the whole organization. So yes, they can, they can unleash agents across multiple different applications. And the good thing is that there is a zero data retention policy with all the enterprise tools. So anything that the employees enter into, into Claude is not retained by Claude, right? It's only used for that session. And

• 23:43Speaker 00

  yeah, so I think, I think, yes, we are, we are, we are already got an agentic overlay framework on top of all the Claude tools that we're using.

• 23:52Speaker 01

  I've interviewed companies, there is one, Era Technologies, which builds essentially a C-suite co-pilot. And the idea is that a CEO will have this co-pilot in his office and will be talking to it throughout the day or using it in sessions to brainstorm strategy, business strategy, or to change business processes. Do you use anything like that when you're looking at the market? Looking at your organization, trying to decide, you know,

• 24:37Speaker 01

  should I buy a, a rack?

• 24:44Speaker 00

  Rack scale company.

• 24:45Speaker 01

  Scale company.

• 24:46Speaker 00

  Oh yeah, absolutely. I mean, again, a lot of this is, you know, happened in the last three to four months, keep in mind. I mean, but even before that, I always used ChatGPT at the time.

• 24:57Speaker 01

  Really? For that kind of thing?

• 24:59Speaker 00

  Absolutely. And surprisingly, now of course we use Claude, we use Claude Code. So Claude Code is, you know, phenomenal. So for both, you know, I'm doing, by the way, we did one acquisition, we are looking at more, right? And so now what M and A strategy, we actually use Claude as, as a sounding board for our M and A strategy, right? And it's amazing what it can do. It comes up with, you know, beautiful reports on what, how the companies could be integrated. So this used to be entire teams, remember? You know, banking teams and advisory teams that used to come in and help companies integrate and come up with a whole strategy on how to put teams together. Guess what? Claude does it in fifteen minutes, right? So and it presents a beautiful report on how the companies can be integrated and put together.

• 25:41Speaker 01

  Yeah.

• 25:42Speaker 00

  Right? Now, you know, it's not perfect because sometimes the data is, you know, not stale, but I mean, once you have a data room from the other company, I mean, you can throw Claude at it and it'll help you create a beautiful plan on how to put, what an integration plan looks like, right? Now, till this was not available, I mean, till about six months ago, I didn't, you didn't have access to this. So no, I used to use ChatGPT just to run strategy questions. Yeah. Again, we're thinking about a product like this. I mean, what do you think? Right?

• 26:09Speaker 00

  I would say sometimes you know, ChatGPT tended to be very polite, right? I mean, it never disagreed with anything that I asked. I mean, so I think what you're seeing more of now is the models are just more capable and more well reasoned and well thought out. And they do tend to disagree. I mean, they will tell you stuff that you're missing.

• 26:31Speaker 00

  Right? Of course, you know, the personality that each of these chatbots has is, is, you know, I think they've been curated to be very likable and amenable. But I think I've seen more debate in the conversation with these chatbots than in the past, right? And that's probably because, because I think as they acquired this step function improvement in reasoning capability, I think they were able to kind of reason through situations. You could say, you could argue that they became more confident.

• 27:00Speaker 01

  Yeah.

• 27:01Speaker 00

  Right? They have, they have, they have, the, the models have now understood that they have better capability. It's almost like a human being, right? You, you, once you realize that you have acquired a new capability, you become more confident. The same thing with these models, they tend to become more confident with disagreeing in a polite way, but highlighting, you know, things that you might have not seen, right? So I see more of that in the conversation.

• 27:26Speaker 01

  There's a level of trust. I mean, with the sycophancy that, that's addressing. Were you ever concerned that the feedback or the ideation that, that it's helping you with is, it's, it's just reinforcing your own biases?

• 27:48Speaker 00

  A little bit of that. Initially I did. So I, I wasn't sure how much to use it. I mean, now I'm feeling a lot more comfortable because it feels more like a human conversation.

• 27:57Speaker 01

  Yeah.

• 27:57Speaker 00

  I think six to eight months ago, it felt more like a echo chamber conversation, right? There's somebody out there just echoing my sentiments and maybe adding a little more color to, you know, what, you know, I was already saying and presenting it in a slightly different way. And so if you are looking where, where those tools were more useful for me personally was if I wanted to make a point or if I had an idea that I already wanted to convey presentation I wanted to do, like it kind of augmented the way I did it.

• 28:26Speaker 01

  Yeah.

• 28:26Speaker 00

  It didn't change and it was augmented. Like, look, you know, you can present this, you know, maybe talk about this this way. This is another way you can say the same thing. So it really was an augmentation tool. Now it has become more of a. It's a copilot. I mean, I don't know if copilot is the right word, but it's almost like, uh, it's like, um, you know, another version of me that is willing to debate with me, right? It's not just an echo chamber, right?

• 28:48Speaker 01

  Yeah, I mean, then on the, on the circuit from CEO, uh, that's also a problem with employee.

• 28:54Speaker 00

  Right.

• 28:55Speaker 01

  CEOs tend to surround themselves with

• 28:59Speaker 00

  Right.

• 28:59Speaker 01

  people who are eager to

• 29:01Speaker 00

  Right.

• 29:01Speaker 01

  to agree with the CEO.

• 29:03Speaker 00

  Correct, exactly.

• 29:04Speaker 01

  Yeah, that's fascinating. Is Anthropic a, a customer?

• 29:09Speaker 00

  Could be someday. Yeah, yeah, absolutely.

• 29:12Speaker 01

  But this kind of inference that you're talking about is, uh, uh, who are the customers that are, are buying the chips for that low latency?

• 29:22Speaker 00

  Oh, it's all the frontier labs. So Anthropic would be one of them. OpenAI would be one of them. XAI would be one of them. Meta Su- Meta Superintelligence would be one of them. Google DeepMind will, I mean, all of them need this. Everybody needs it.

• 29:35Speaker 01

  Yeah.

• 29:36Speaker 00

  Um, there's no, no other way, right? I mean, they're gonna have to find a way to use uh, different type of low latency computing to augment uh, the throughput based computing that they've been, you know, kind of doing with, uh, HBM based solutions. So the GPUs or other accelerators that all use HBM uh, just don't have the same levels of interactivity that, you know, the solutions we build are having, right?

• 30:02Speaker 01

  I, I was asking about, uh, you mentioned, uh, you know, sovereign AI as opposed to models like, uh, from the foundation labs that you're uh, hitting with an API. Can you talk about that market? Is that because there was this huge transition? You gotta feel for the, uh, the IT, IT guys. There was this huge transition. Get out of your data center into the cloud. Now, they, a lot of people have gone through that and, and now it seems like, uh, things are flowing back into, uh, private data centers.

• 30:42Speaker 00

  Yeah.

• 30:42Speaker 01

  Is, can you talk about that? How, how you see that evolving?

• 30:47Speaker 00

  You know, I, I, I know there was, there was a big outflow, uh, we saw over the last, call it fifteen years where, you know, cloud became more and more popular and, uh,

• 30:58Speaker 00

  but I think it was, it was, you know, a lot of the outflow that happened to the cloud was smaller companies. You know, companies looking to get started. I mean, I'll talk about D-matrix. When we got started, we didn't wanna deal with an on-prem, on-prem cloud, right? I mean, we wanted to deal with, uh, you know, we, we, we could just offload our entire IT infrastructure to somebody like an EWS or a Microsoft Azure and we were often running in like days, you know, you know, right? As opposed to setting up our own data center, setting up our own hardware. Even if it's in a colo, right? I mean, doing all, I mean, that was a traditional model even for small companies, once upon a time. Where you set up your own, you know, small data center or, right? And you, you, you, you kind of ran your own hardware, right?

• 31:40Speaker 00

  And we didn't have to do any of that. Didn't have to worry about security. Didn't have to worry about scaling that hardware, deploying the tools. None of that, right? I mean, uh, um, so you see, I mean, that, that trend has been very strong and will stay strong even to this day. All the AI native startups, right? Companies that are building these new age applications for AI, they're, they're all running in the cloud. They're all still running in the cloud, right? Now, what you're talking about is maybe con- companies that were, you know, kind of further along in their journey, much larger, mid-size to large size software companies, Fortune five hundred companies that carry a lot of data. See, they never fully migrated to the cloud, right? There was, they, they, there were applications that they migrated to the cloud, but there were certain applications that they didn't migrate into the cloud. There's certain things that were data sensitive. Their data never migrated into the cloud.

• 32:26Speaker 00

  Now, the only thing that is different now is you still have the same structure. It's a hybrid approach for some of the larger organizations where uh, some of their applications run in the cloud, but some of their applications that tap into their native or, uh, you know, uh, domain specific data, they, they don't wanna, they don't wanna have those applications sitting in the cloud. So they're still running on. So they're maintaining both. Right? They're maintaining an on-prem data center. They're maintaining, you know, they have a cloud data center or, uh, partner also, right? Now, AI doesn't change any of that, right? AI just is an overlay on top of that. So now you use AI to access the applications running in the cloud, depending on the type of applications you're running. Those applications will get infused with AI.

• 33:06Speaker 00

  And you have an AI overlay on top of those applications. The same thing goes for what they're doing inside their enterprises, right? The only thing is that the cloud, AI overlay that is happening in the cloud is happening much faster. Right? Because the hyperscalers, Google Cloud, uh, Microsoft Azure, Amazon, they are very AI savvy. So they are able to bring AI capabilities into all their cloud offerings a lot faster than a enterprise that has its own data and its own tools internally can do. So bringing AI capabilities natively to a Fortune five hundred company, for instance, right, will take much longer than their applications running in the cloud at, say, Microsoft Azure will become AI friendly lot faster.

• 33:50Speaker 00

  Right? So I think that's the only thing that you're seeing. Uh, it's not like they're looking to pull back stuff that was running in the cloud, uh, back into their organizations, uh, because they left the organization in the first place because they were not concerned about the data that they were accessing.

• 34:08Speaker 01

  Yeah. Um, although in regulated industries, uh, there is concern about sending data to the cloud. Do, do you deal with a lot of, uh?

• 34:19Speaker 00

  That concern has always been there. It's not like a new concern with AI. It's concern means has been there like ten years ago. So I, I don't think anything changed. I think it's all, it's all about where the data is resident. Right? And, um, that concern never changed, right? Because this data has been around for decades in many companies and they were never quite comfortable sending that data into the cloud, right? Uh, I think the one thing that you might be referring to is, um, more, most of these companies have access to a OpenAI API or or Anthropic API, right? And, uh, now they are, there is new data. I mean, and, and what people are doing is they are kind of conducting searches. And that tool has to get in and access databases and has access to that data, right? And, um,

• 35:03Speaker 00

  you know, is that something you wanna do, right? I mean, I think so. There is a, you know, we have the same issue. We have all our proprietary data sitting in a data silo somewhere. And now Anthropic can access all of that data, right, uh, to, to do things for us. Right? And that's why they have the zero data retention, right? None of that data ever goes to Anthropic, right? And, uh, at least the enterprise class too. So that's a guarantee that they're giving you, that all the data that they touch within your organization is not, is not going to them, right? Um, so it's only, and I think maybe that was your question originally. It's like, hey, like, there is an API that you guys are accessing through Anthropic or OpenAI and that API gets access to some of your internal data because

• 35:44Speaker 00

  you are, you are taking some of your internal databases and asking Anthropic or OpenAI to do things for you. Um, yes. So that, that, that, you know, is a concern, I think, uh, has been addressed by these companies, right? And, uh, um, and in some ways is probably not, not very different from, uh, uh, you running, you know, for example, um, you know, Microsoft tools, Microsoft open, you know, you know, Office three sixty-five or, you know, M three sixty-five, uh, has a lot of, you know, your information, a lot of emails, you know, but it's sitting, it is, you know, Microsoft tools have access to it, right? So in many ways, it's not very different from that.

• 36:23Speaker 01

  This market, as, as you said, it's expanding, um, maybe faster than even you anticipated.

• 36:31Speaker 00

  Yeah.

• 36:32Speaker 01

  Are you primarily focused on the US market?

• 36:35Speaker 00

  Today, yes, yeah, yeah.

• 36:36Speaker 01

  Yeah, I've, I travel a lot and, and, uh, you know, the, every country I ran into, wherever you go, Liang in, uh, Saudi Arabia, a year or so ago. Are you looking at these other, uh, centers that are trying to build out compute and, and AI infrastructure for, uh, either for their own country or for, you know, there's a lot of talk about who's gonna be the leader of the global south of the AI movement?

• 37:08Speaker 00

  Right, right, right. Um, uh, we are spending just the right amount of time, not too much. I think, um, we are focused a lot on the US market because that is where the opportunity will emerge and then eventually it will diffuse into

• 37:22Speaker 01

  Yeah.

• 37:23Speaker 00

  other countries in the global south, right? So I think, uh, we wanna start at the source of the problem, not, not go to the destination directly. Um, so I think we're starting at the source where a lot of the innovation is happening. People are trying to figure out how to deploy these solutions, what makes the most sense, uh, which applications really needed, which applications don't, where does it work well, where does it not work well? Once those questions have been answered, then it'll diffuse into, into a lot of the sovereign applications, into a lot of the enterprise applications. We'll ride that diffusion process, but it has to start from the source, you know?

• 37:55Speaker 01

  Yeah. What, do you pay attention to what's happening in China?

• 37:59Speaker 00

  Not so much. We of course pay attention to all the work that's happening there and, you know, uh,

• 38:04Speaker 01

  Yeah, I just wonder, are they moving in the same direction on, on, uh, you know, this premium token market and all of that?

• 38:15Speaker 00

  Uh, they have to be. They have to be. I mean, I just don't see why. I mean, that's a very intuitive thing, right? It's like, I wanna create, uh, I want people to access applications. I mean, everybody wants to access an application uh, in a highly interactive and a real time way. Why would that be any different for a segment of the population?

• 38:32Speaker 01

  Yeah.

• 38:33Speaker 00

  Right? I mean, if you need it in the US, you need it in China, right?

• 38:36Speaker 01

  Yeah. Are you allowed to sell to China under the

• 38:39Speaker 00

  Under export controls, we would be allowed to sell to China, yeah.

• 38:41Speaker 01

  I recall most of your chips, uh, if not all, are manufactured in Taiwan at TSMC, uh, or I, yeah, silicone is etched to Taiwan.

• 38:53Speaker 00

  That's right, that's right.

• 38:54Speaker 01

  You know, TSMC is building, uh, a fab here. I don't know if it's online yet, but

• 39:01Speaker 00

  Yeah, it is not online. It is online. Uh, they're, they're building multiple fabs in, in Arizona and, uh, uh, at least one of them is online since twenty twenty-four.

• 39:11Speaker 01

  I had, have you seen, uh, any of your, uh, manu- manufacturing or fabrication, uh, moving back from Asia?

• 39:22Speaker 00

  Not yet, not yet. Still in Taiwan, yeah.

• 39:24Speaker 01

  Yeah, I would imagine, and I asked you this last time, and I get the same answer from everybody. Uh, oh, we have relationships, uh, but, uh, is there any sort of a bottleneck? I mean, uh, you know, can TSMC serve the world as the demand for inference explodes?

• 39:45Speaker 00

  Uh, well, uh, you know, the, um, that's, the chips are gonna be in short supply for the next, you know, call it three to four years at least, if not more. We have memory, you know, chip shortage, we have compute shortage, right? Um, there's a reason why Elon wants to build his own terra fab, right? Because he feels TSMC cannot meet all the demand for compute.

• 40:13Speaker 01

  Yeah.

• 40:13Speaker 00

  And memory, um, so we'll see how that goes, uh, and Intel is helping him out with that. So we'll have to see how it all, all works out, but, um, but I think for the demand that we have, Dematrix, if I would look at the Dematrix demand picture for the next, call it five years, I think TSMC has got plenty of supply to meet our demand. I'm, you know, we are a small company, we're looking to grow into this opportunity. Um, this opportunity is also very nascent, it's just exploded on the scene literally in the last few months. It's got a long, long way to go. So I think we are growing into an opportunity that's pretty new. And, uh, the choices we have made on how we build our product, like we don't use the bleeding edge process technology at TSMC.

• 40:58Speaker 00

  We use kind of N minus one, N minus two process nodes. We don't use any HBM technology, we don't use any COVAS technology. Uh, so all these decisions that we made early on help us. Uh, so yeah, if you look at a supply chain that uses those technologies, like the ones I just outlined.

• 41:16Speaker 01

  Yeah.

• 41:16Speaker 00

  I think TSMC should be able to make, you know, will it be tough? Yes, for everyone, tougher, it's tough for everyone else, but I think, uh, I think we have a better shot at making supply because of the choices we made.

• 41:28Speaker 01

  Yeah. Is that a constraint on your growth at all?

• 41:31Speaker 00

  Not yet, not yet, but it'll be a good problem to have, right? I mean, uh, you know, if I get to the point where my growth is constrained by how much TSMC can supply to me, I think I'm in a good place, right? I think then I really have to just kind of work through the problem, uh, which, uh, we will work through the problem with them, right? Uh, and to me, I think it comes down to the relevance of the problem, right? If your problem that you're solving is very relevant and there's lots of big customers who care about, uh, solving their problem. Yes, you have relationships and that's all great, but at the end of the day, you know, uh, follow the money, right? I think if, if there's a very relevant, large problem that a very relevant, large customer is trying to solve, and we think we are helping, you know, solve that type of problem, then, uh, the supply chain will make, you know, a place for you, right?

• 42:16Speaker 00

  Because the world needs this, right? So I think they'll find a way to make place for you.

• 42:20Speaker 01

  Yeah. Uh, you, you talked about, uh, the agentic explosion, uh, and, and about the, the, the auto, whatever you call it, uh, AI coding explosion. Uh, which of those is the larger market for you? And who, what kinds of companies are you selling to? Uh, if, I mean, you were saying you're selling to the foundation labs, not necessarily Anthropic, uh, but whose, uh, coding co-pilots are you providing chips for? And whose, uh, agentic, uh, systems are you providing chips for?

• 43:05Speaker 00

  So agentic, I think is not a separate, agentic is a capability that I think pretty much every, every company that is building a coding tool is, is embracing or is, is introducing into their, into their product offering.

• 43:20Speaker 01

  Yeah.

• 43:21Speaker 00

  Right? So I, I don't think you should look upon it as a, okay, there is a segment of, you know, section of companies that's only working on agentic. It's agentic is like, you know, kind of a capability that is now being, Open Cloud demonstrated what's possible. But now you take that capability and introduce it into your products, right? So everyone is doing it, OpenAI is doing it, XAI is doing this. You know, obviously Anthropic, because they sell into the enterprise, they do it, they're doing it faster than anyone else. Um, but it's a capability, right? And that capability adds more influencing needs on top of the coding generation tools that, uh, are already, already, uh, need, need, you know, need inference, right? So you have, you have, it's kind of putting it together. Uh, we, we said, we, we are working with everyone, right?

• 44:04Speaker 00

  I mean, uh, we're working with all the frontier labs that build coding tools, uh, agentic coding tools. We're working with, uh, frontier lab that are building video agents, voice agents. We are working with AI native startup building voice agents. Um, we just had to be careful about not going too broad because I think the company is still not at the stage where we can support all these customers simultaneously. Um, so we have to be, you know, kind of gradually grow into this opportunity with, uh, uh, you know, at the right time with the right company. But we're doing, we're doing it with everyone because it's relevant. Again, going back, you know, I mean, what we are building is relevant for everyone. Everybody wants it. It's not like there's somebody who says, like, and I don't care about, you know, high levels of interactivity. I don't care that my users are able to interact with my application in real time.

• 44:49Speaker 00

  And nobody's telling us that, right? Uh, everyone wants fast tokens.

• 44:52Speaker 01

  Yeah. Or, or is there a segment of the market where you think, uh, you talked about, uh, an agentic enterprise with where uh, it's, you know, agents are, are doing much of the work. Uh, do you, do you see, uh, emerging, uh, companies, startups that are agent first? Uh, and do you think that segment of the economy is going to grow? I mean, everyone's sort of watching, uh, you know, there was a New York Times had a piece recently about a guy and his brother running a 1.6 billion dollar in sales company

• 45:34Speaker 01

  that's basically in a agentic, uh, first company. Uh, do you see that segment of the economy and growing? I mean, that, that's the future.

• 45:46Speaker 00

  That's the future.

• 45:47Speaker 01

  Is that going to, uh, challenge legacy companies in, uh, in, in different?

• 45:55Speaker 00

  I think it depends, I think it depends on what kind of product offering or products or what kind of service, uh, you're building. I think there is a place, uh, I mean, you have those companies today. I mean, even pre, pre AI, right? I mean, there was, there was companies, you know, you had people running a consulting practice, right? I mean, uh, you are a one man, are you one man? Uh, there you go. So you are right, you're already, and now you're gonna start using AI, right? I mean, so you, you could be one of those people like, hey, look, I, I've been far ahead of everyone.

• 46:25Speaker 01

  Yeah.

• 46:25Speaker 00

  I've been a one person company for a long time, right? And now I just use AI as, as an augmentation tool. So it's really depends on the product offering and service, right? I mean, uh, there is, uh, I think there is a lot of companies that, um,

• 46:40Speaker 00

  had a product offering or a service that can be taken over by AI completely, right? And you don't, you don't really need, uh, you don't really need humans, uh, for, for the kind of work that they were doing. Uh, so there will be kind of the low hanging fruit, I would say, that will immediately get consumed. Uh, either they embrace it or then you have, what you just said is a new breed of companies just comes, wait a minute, why are those companies, you know, why do they have humans in those companies? You know, we can, we can just do it, you know, with AI and then they will just become a lot more efficient. Or those companies embrace it themselves and, and, um, you know, rescale the humans into something else, right? Or they find that as an, that's an opportunity for those companies to grow into something different because humans are good at something which AI is not good at.

• 47:23Speaker 00

  So why don't you rescale the humans to do stuff and expand your, expand the opportunities you can go after as opposed to, as opposed to, um, sticking with what you have. It's really an opportunity for everyone to grow into something new. Um, but absolutely, there'll be many more companies that will be too lonely with AI to address those products and those services, right? But then there are certain products and services you really just cannot, uh, you know, build, uh, you know, with, with AI only. I mean, you need manufacturing lines and you need, you know, good product design. And there's emotion that goes into how the product is sold and, uh, you know, how the messaging is done and how you appeal to consumers. I mean, all this is, is something that you need humans. Because at the end of the day, you're making a sale to a human.

• 48:07Speaker 00

  At the other end is a human who's buying the product and you want to appeal to, appeal to their sensibility and their emotions. So I think, uh, not every section of the economy is going to get, uh, consumed by AI only companies, but, uh, there will be a lot of them, right? A sizable portion.

• 48:24Speaker 01

  Yeah. Healthcare is, uh, an industry that's, that's going under, uh, through tremendous change. Uh, do you, do you see much demand for, uh, agentic AI in the healthcare space? Are you serving that space or are you focused on, uh, on a few verticals?

• 48:46Speaker 00

  Yeah, we are not very active in the healthcare space. So I can't meaningfully comment on, um, what, uh, what the latest and greatest is there. Because we, we serve, we are, we are serving the tools. The tools get used by

• 49:00Speaker 01

  Yeah.

• 49:01Speaker 00

  various verticals, right? Healthcare is certainly one of them. Uh, but I think to me, the most exciting thing in healthcare is, uh, really drug discovery, you know, you know, you know, disease, uh, you know, curing, you know, really complex diseases, uh, you know, producing new forms of treatments that could accelerate the development of drugs or, or find cures, right? I think, so I think, uh, uh, to me, to me, that, that segment is, is, is a lot more exciting. Um, uh, and, uh, you know, that is one of the goals of AI, should be one of, should be one of the goals of AI is, yes, you know, you know, more equitable wealth distribution, sure.

• 49:44Speaker 00

  But really, uh, can we, can we, uh, make, you know, health benefits, you know, broadly available? Can be and easily accessible to parts of the population that don't even have access to it, right? And so, so can we get humanity to a point where, okay, everybody has enough? They can live long lives, long, healthy lives, right? And they can live peacefully. I mean, that's maybe the third quest, right? Uh, we can, now moving, maybe into a bit of a utopian vision here, but, but you know, that would be the vision, right? It's like, uh, you know, there's enough, enough, you know, wealth for everyone, enough, you know, health for everyone, enough, you know, peace for everyone. And then, then, then AI has, has really helped humanity, right?

• 50:29Speaker 01

  Yeah, that's right. Well, we're, we're hoping, uh, it's also requires political leaders.

• 50:36Speaker 00

  That's right, that's right.

• 50:37Speaker 01

  Exactly. Right. Okay.