#314 Nick Pandher: How Inference-First Infrastructure Is Powering the Next Wave of AI

Inference is now the biggest challenge in enterprise AI. In this episode of Eye on AI, Craig Smith speaks with Nick Pandher, VP of Product at Cirrascale, about why AI is shifting from model training to inference at scale. As AI moves into production, enterprises are prioritizing performance, latency, reliability, and cost efficiency over raw compute. The conversation covers the rise of inference-first infrastructure, the limits of hyperscalers, the emergence of neoclouds, and how agentic AI is driving always-on inference workloads. Nick also explains how inference-optimized hardware and serverless AI platforms are shaping the future of enterprise AI deployment. If you are deploying AI in production, this episode explains why inference is the real frontier. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Preview (00:50) Introduction to Cirrascale and AI inference (03:04) What makes Cirrascale a neocloud (04:42) Why AI shifted from training to inference (06:58) Private inference and enterprise security needs (08:13) Hyperscalers vs neoclouds for AI workloads (10:22) Performance metrics that matter in inference (13:29) Hardware choices and inference accelerators (20:04) Real enterprise AI use cases and automation (23:59) Hybrid AI, regulated industries, and compliance (26:43) Proof of value before AI pilots (31:18) White-glove AI infrastructure vs self-serve cloud (33:32) Qualcomm partnership and inference-first AI (41:52) Edge-to-cloud inference and agentic workflows (49:20) Why AI pilots fail and how enterprises succeed

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Cerescale is a NeoCloud specializing in lots of different areas around AI. But most specifically today, we're going to focus on inference. I'm VP of Product and as part of my role as VP of Product is working with our development teams to build out unique capabilities for AI services around inference and other areas as well. My background comes out of the GPU space. I was at Nvidia very early on and AMD and several startups that have touched AI models and other type of models in their use cases.

• 0:32Craig S. Smith?

  Although I don't think of that as the cloud provider's job. Is that unusual that you're getting involved in optimizing inference as opposed to just providing the compute?

• 0:48Craig S. Smith?

  Okay, yeah, so Nick, why don't you start by introducing yourself to listeners and then we'll talk about inference.

• 0:57Speaker 00

  Absolutely. So hello, everybody. I'm Nick Pandher with Cerescale. Cerescale is a NeoCloud specializing in lots of different areas around AI. But most specifically today, we're going to focus on inference. I'm VP of Product and as part of my role as VP of Product is working with our development teams to build out unique capabilities for AI services around inference and other areas as well. My background comes out of the GPU space. I was at Nvidia very early on and AMD and several startups that have touched AI models and other type of models in their use cases as well.

• 1:36Craig S. Smith?

  Yeah, tell us first what Cerescale does.

• 1:39Speaker 00

  Absolutely. So we're a more white glove type of cloud services provider. What we're focusing on is trying to understand customer specific issues and when they're looking for outside cloud services. So typically we're taking customers who are coming from an on-premise environment or a hyperscaler and are just looking for services and capabilities that may not be part of what they're getting today. So if we look at it from an AI point of view, it's being able to deliver the correct AI fleets. For given use cases, it's covering any sort of compute needs that might be required for pre or post processing data. It's delivering services that, you know, help bring high performance, reliable type of solutions.

• 2:24Speaker 00

  Specifically, you know, if we look at inference being a differentiated area, we really wanted to, you know, increase those services and expand from our AI accelerated fleet where customers typically bring their own stacks and bring their own solutions into more ready to go solutions that a customer can take and utilize for, you know, training, fine tuning, and also for for inference capabilities as well.

• 2:50Craig S. Smith?

  Yeah. And is that primarily your differentiator primarily in the software? Or is it that you have, you know, specialized GPUs tuned for inference?

• 3:02Speaker 00

  Yes. So that's a great question. So we, a few different facets there. One is we're all about operating a cloud service that's taking the highest performance with the capabilities a customer is looking for with the highest reliability balanced in as well. So if we look at, you know, training type workloads, we want to provide what we call the highest job completion rate. If we're looking at inference, we want to provide the most robust, you know, five nines type of uptime and then unique software capabilities to deliver serverless type of solutions.

• 3:36Craig S. Smith?

  Yeah. And we're talking about inference. There was a shift maybe six months, two year ago, when people stopped talking about training and started talking about inference. I've had Andrew Feldman from Cerebras on and Rodrigo Leong from Samba Nova and some of the other inference focused chipmakers. And those guys have had trouble getting developers to adopt their hardware.

• 4:06Craig S. Smith?

  Because developers are so married to the CUDA programming language. And they've shifted to building their own clouds or trying to get clouds to adopt their chips. And providing inference service or inference as a service is. Can you talk first of all about why there's been this shift to inference? And how you see the explosion of inference workloads? Developing as AI takes gold in the economy?

• 4:40Speaker 00

  Yeah, absolutely. So if we take a look at the shift from training to inference, there's a little fact there that's really important is there's. There's a large group of organizations out there building models, either building models for their own proprietary services. Like look at OpenAI with their ChatGPT type of models. They're building something that works really well for a lot of different use cases. There's a wide range of different uses uses that the model can support. Similar enough, we've seen the same with other models that are out in the space today. If we look at the, you know, probably the number one model everyone's using today from an inference point of view is part of their Office 365 package, which is. You know, Microsoft's provided Copilot. So those are good at specific tasks.

• 5:24Speaker 00

  When you start to look at, you know, more enterprise type workflows and look at agentic use cases and agentic workflows where you're. Taking the model output and utilizing outside workflows and existing applications that might be in the organization. There's a more select group of models that are out there that can actually be used today without no need to. You know, pre-train a model, which is a very expensive, long operation there, or. No real reason to have to post-train a model. What we're really looking at is taking an existing foundational model. There's a whole series of them out there, you know, Meta, Llama, AI2's, Open Models and others.

• 6:04Speaker 00

  And being able to take that model and do some level of RAG where the model can be more applicable to your use cases. Or for the models that allow it doing some level of fine tuning. So there's a lot of options out. There to take such those models, skip that entire training path, leverage one of these existing models and start utilizing, you know, the models and your specific workflows and connect it to existing agentic type of use cases. So there's a lot more that can be done with the off the shelf models than, you know, was perceived, you know, a year ago or even six months ago.

• 6:40Craig S. Smith?

  Yeah, although I don't think of that as the cloud provider's job. Is that unusual that you're getting involved in, you know, optimizing inference? As opposed to just providing the compute?

• 6:56Speaker 00

  Yeah, absolutely. So there's a fundamental change happening in that people are taking those off the shelf type of solutions and wanting to understand that they may have specific requirements unique to their organization and how those models are being served. Let's take an example. We can take OpenAI's ChatGPT, great model. But there may be some organizations who are nervous about their data being shared to a model developer and, and, you know, you're gonna, you're gonna have different organizations with different security type of concerns depending on, is it regulatory or is it just more protecting their proprietary information? And so as a good example is you can take OpenAI's Open weights model, their OSS one twenty, deploy that in a private environment through a cloud provider.

• 7:37Speaker 00

  And now you have a model that is nearly as capable as ChatGPT five, but in a private environment where your data, your prompts and your outputs from the prompts are all protected and it's running in a private environment. So from that assurance point of view, you're getting something that is very unique and you're leveraging a cloud provider for that.

• 7:56Craig S. Smith?

  Yeah, although don't most of the cloud providers offer private clouds, even OpenAI offers private clouds environments, don't they, for those kinds of security reasons?

• 8:11Speaker 00

  Yeah, so the differentiation that we like to see ourselves doing is understanding what the customer has as far as their uniqueness. Are they connecting parts of the workflow to a hyperscaler or connecting parts of it to on-premise infrastructure as part of their end-to-end workflow? And are we able to provide a solution that could be more technically enriched for their specific needs that they may have? If they have specific workflows or specific software stacks they've already utilized in their on-prem, we want to be able to extend those. So if they've already built an inference services layer on-prem and are happy with it and are saying, how do I go extend that to the cloud? That's an area that we can help and we can bring some uniqueness on the serious scale side by providing that connectivity to their on-prem,

• 8:55Speaker 00

  providing higher performance fleet that could be cost-effective for what they're trying to do and kind of giving them that predictable performance. At the other tone, if we're looking at people who are already sitting in a hyperscaler, one of the biggest challenges people are looking at when they're taking these AI workloads and these private AI deployments inside of a hyperscaler is the costs can add up really quick. So it's delivering that same level of performance you expect from a hyperscaler. Where a hyperscaler is really good is you getting on and off and not utilizing that node or those nodes 24/7. Where we're going to be better at, cloud providers like ourselves, the Neo Clouds, is if a customer comes in and needs certain level of compute performance

• 9:35Speaker 00

  for generating tokens in a model, as an example, 24/7, and then needs a higher performance level of that example, weekdays, 9:00 a.m. to 9:00 p.m., where people may be utilizing the model more, where they need five times more capacity. It's sometimes is a challenge in hyperscalers today in some of these model deployments to get a reliable performance. The biggest indicator you see you're having a performance issue is time to first token dramatically increases. So you're paying for the usage of the model, but you're also starting to suffer with some of the performance issues that come with this great uptake people are seeing with AI becoming more prevalent in a lot of different organizations.

• 10:15Craig S. Smith?

  Yeah, so throughput is one of your key metrics, I would imagine.

• 10:21Speaker 00

  Yeah, so I'd almost say it's providing that right compute solution for the customer, which is the right throughput, the performance and that cost efficiency is really the key driver there. So the throughput to really serve those continuous requests. If we look at it from an AI or LLM model, it's those millions of token requests, being able to serve those reliably, providing that predictable performance across those different model architectures. Every model behaves differently, so making sure the fleet of compute that's operating that model, you know, provides the best performance, the lowest time to first token, if it's a real time use case. And then having that cost efficiency so that you can scale, scale up.

• 11:04Speaker 00

  And as your user counts go up or your complexity and the model usage goes up, you're able to cover those really well. And so what we really see, it's that perfect balance of understanding all of those elements and helping a customer really balance the performance and capability needs with the, you know, the economic costs type of areas and deploying a model and getting it to be used in a specific workflow.

• 11:29Craig S. Smith?

  Yeah, and I didn't realize until a year or so ago that even the hyperscalers farm out a lot of their workloads to private data centers. Is that right? And do you guys serve the hyperscalers?

• 11:45Speaker 00

  Yes, so our focus is really supporting the, those who are building and delivering services on top of generative AI or AI models. So we're more talking about specific customers who are building service offerings to end customers or to organizations. You know, we want to work closely with ISVs and SaaS providers who are looking to extend their existing workflows to add AI services, but may not want to lean that in their existing hyperscaler, just from a cost and performance point of view. And then also, what the biggest growth area that we see, you know, coming into the tail end of this year and into next year is there's more enterprise customers who kind of understand what they want to do.

• 12:25Speaker 00

  Look at the solutions from the hyperscalers where they can get a, you know, a model garden of different models, an API endpoint and get out there. And they're looking for a wider variety of models, but somebody who can deliver it as a service offering. So more of a serverless offering, but it's around the models they care about. And in some cases, we're going to see in some organizations, those are going to be proprietary models that might be coming from a SaaS provider or a proprietary model that might be a fine tuned version of a model that's unique to that organization. And so when you start to bring those custom or the semi-custom models into a hyperscaler, you start to break the bounds of some of the services that they offer today.

• 13:03Craig S. Smith?

  Yeah. And what chips are you using them? Are you using Cerebrus or Samenova, which goes sort of blindingly fast inference? Or are you, that was somebody else I spoke to recently, who's speeding up inference simply through a software optimization strategies?

• 13:27Speaker 00

  Yeah. So we're unique in that we're agnostic in the acceleration technologies for AI models. So we've worked with a significant number of incumbent partners. We've started working with a series of new technologies. You just mentioned two of them where we've had relationships with them and we work closely with their teams on how we can work together on customer opportunities. At the same rate, there's a new generation of technologies that have been coming out. You know, and if we look at the existing two largest guys out there, Nvidia and AMD, they're doing extremely well. We do very well with the solutions that we offer in conjunction with them. But there's been a group of newer technologies in the last year or so that are more focusing on differentiated areas and looking at

• 14:09Speaker 00

  providing, you know, energy efficiency, performance, differentiations, and just looking at taking it from a different angle. In that regard, we've been working with Qualcomm and we introduced some offerings specific around Qualcomm's AI 100 Ultra. And it fits really nicely with a software stack that they've delivered, which provides a suite of services that are built around inference to be able to make access to models. Much easier in the same light, you know, one thing we do like on the Qualcomm solution is they've also just recently announced some newer technologies, their AI 200 and AI 250. These are the kind of solutions that we want to see coming into the market. We want to have diversity.

• 14:50Speaker 00

  We want to figure out where each of these solutions fits a specific use case or a specific implementation of a model and how it's used in a specific agentic workflow and understand which solutions provide the best performance. So when a customer comes to us and says, I'm looking for inference with this technology, we can say, okay, let's go actually pilot the model in an environment with different accelerators and find the best for you. And so the solutions like Qualcomm is we want to position those solutions where they fit the best and are able to deliver the performance that is unique when you blend it with the costs and power efficiency that you're going to get as well.

• 15:25Craig S. Smith?

  Yeah. And on the cost efficiency, where does that come from? Simply by increasing throughput, you're getting more inference per token or not per token, per minute. Where does the cost efficiency come from?

• 15:42Speaker 00

  Yeah. So the cost efficiency can come from a few different areas. Number one is the accelerator optimized for specific type of needs. So is an accelerator more optimized for inference versus training? If we look at a lot of the accelerators out there today, they're dual purpose. They're really built to go do extremely complex cluster deployments for training and they're going to be excellent for inference, but they're going to be costly. So there is a demand out there for solutions that may not be looking at training first, but maybe looking at inference first. And for some workflows, they're just going to be the more optimal solution out there.

• 16:17Craig S. Smith?

  Yeah. But in Qualcomm, certainly I've spoken to them, is designed for inference first workloads. But again, is what is it that makes it cheaper? Is it's not the cost of the chip, obviously. It's not the, you know, the bandwidth. It's, there's a reason why it's cheaper than a hyperscaler. Why is that?

• 16:41Speaker 00

  Yeah. So very simply, you're getting a solution that, just as we said, was focused on inference first workloads. So you're looking at it being built for delivering that throughput that's required to deliver the models, having the right capabilities to support different model types, transformers, supporting different types of networks that are used inside of models, being able to support diffusion models, for example. So covering that throughput use cases that are specifically needed, but at the same time, covering a more effective power profile as well. So if we look at the power profile for the Qualcomm solutions, it's still a data center capable device. It's just gonna, when we look at a rack full of compute with a Qualcomm solution versus other solutions out there,

• 17:24Speaker 00

  you're looking at extreme, you know, energy efficiency. So that, that really comes down to is the power savings. And for a lot of these models, the difference between a higher performance accelerator for certain use cases to a, to an accelerator that's balanced for inference first, that's balanced for, you know, power efficiency, you still get excellent performance in the serving up those models for those agentic workflows. So, so it really depends what the, what the customer is looking for. And in a lot of cases, what you tend to see is with the higher performance accelerators, you're barely using out of a, you know, a server that has eight GPUs. You're barely using one GPU. And so you're going to run a model that's barely utilizing one GPU. Wouldn't you want to have something that's more tuned performance where you're utilizing the full GPU?

• 18:09Speaker 00

  And you're able to then scale it into additional GPUs or run additional models that are part of your workflow as well. So better utilize the hardware where you have hardware that's dedicated to a customer, but it's still servicing cost effective. One key differentiation for Cerescale is we don't do, in our bare metal type of solutions, we don't do multi-tenancy. So we really want to lean on partner technologies like we do with Qualcomm's AI inference software stack to be able to handle running multiple models for different use cases on a platform. So there's some differentiation there of the software stack as well, being able to better utilize the hardware and make sure you're getting as much performance out of it, what was still within that lower power curve.

• 18:49Craig S. Smith?

  Yeah, yeah. I actually wrote a piece for Forbes a little bit ago. There were these companies that were appearing that, that aggregate idle GPU and CPU resources and then offer them as a cloud solution because. Most data centers, certainly private data centers, the chips are not being used most of the time. So yeah, I can see why that's important. The, and how does, well, two questions, what kinds of enterprises are you seeing come to you? I mean, it's taken a while, you alluded

• 19:29Craig S. Smith?

  to this, for enterprise to figure out what really works in the enterprise with AI. There has been a lot of pilots and we're just now moving into production workflows in the enterprise. And is it the agentic shift or the shift to AI agents? Or is it other forms of AI that are are finally taking hold in the enterprise?

• 20:01Speaker 00

  Yeah. So I think we could look at two different aspects here. Number one is agentic, which we'll touch on for a second, but let's first talk about automation. So there's a lot of tasks that can automate to improve how a person works. You know, a very good example is if we look at one vertical segment, we can look at, you know, banking, finance and insurance markets. Imagine a mortgage application packet. And I think all of us at some point done a mortgage or a loan where you're submitting lots of documents and, you know, the mortgage provider will always say, oh, we'll turn this around in seven days. And, you know, you know, 21 days later, they're still asking for documents. Wouldn't it be wonderful to be able to submit all those documents, have an automated workflow, parse through the documents and understand are there red flags the underwriter needs to

• 20:43Speaker 00

  go look at right now to be able to go back to that customer and say, hey, you're missing these or they need more information here. Right now, that process is by the time they look through all that material, it's, you know, a week or two. But so providing a some level of automation, a, an LLM prefaced in with a multimodal type of model or even an OCR type of model. There's some modern OCR models that we've been working with AI two on as an example, being able to take, you know, computer generated, you know, printed text or handwritten text, being able to convert that to outputs. Understanding those outputs within a specific model and understanding where certain things filled out correctly, where certain things provided that need to be provided so that our turnaround of like 21 days for a mortgage could turn into,

• 21:26Speaker 00

  okay, within three days, we're telling you everything that you need to provide more information on it and you move along there. Now, that's beneficial to an organization because you're utilizing your people. You still have humans in the loop and there's a lot of this AI is going to replace what humans do. I truly don't believe that's 100% of the case. Can you make humans more efficient so that the people working on specific parts of workflow, like in banking, finance, or even in the medical billing industry as an example, you're having people focusing on the more important tasks and you're having automation deal with the tasks that, you know, typically an organization may outsource out of the country or whatever else to a third party. You can keep a lot of that information internal to your organization and have your internal people actually focus on looking at areas

• 22:10Speaker 00

  that the model identifies as needing additional understanding.

• 22:15Craig S. Smith?

  Yeah, and you guys say you're the first NeoCloud to work with Qualcomm's inference, the new inference chips and software stack. What exactly is a NeoCloud? That's not a term I've heard a lot of, but I'm not deep in the cloud space.

• 22:34Speaker 00

  Yeah, so a NeoCloud is a cloud provider that's more focused on delivering a specialized fleet of hardware. We're not a hyperscaler, but we're, we're delivering compute that's more focused for what a customer wants, more tuned to specific type of compute needs and built around either a training, a fine tuning or an inference workflow. So something a little bit more in tune with what an enterprise organization would want for end-to-end workflows, looking for services that include not just the hardware, but also storage, any specialized networking or specialized connectivity they have to the rest of their workflows.

• 23:13Craig S. Smith?

  Yeah, and is it fair to say, because these enterprises certainly still use the hyperscalers, is it fair to say that there's a class of, of problem of inference or training, but specifically inference in regulated industries where they need a ring-fenced private cloud? And the big hyperscalers don't necessarily provide the.

• 23:42Craig S. Smith?

  The bespoke or tailored solutions that these enterprises need. So as a portion of their inference workload, they send to a NeoCloud. Or am I extrapolating too much?

• 23:57Speaker 00

  No, that's you're hitting the nail on the head. So there's things that a hyperscaler are going to be very good at running SaaS type applications, you know, as workloads are running for specific SaaS applications for a given customer. They're really good at that. They're really good at providing those standardized data lakes for an organization to have all of their proprietary information. Where the hyperscalers are going to struggle is on GPU availability and the compute availability and tying it in to be cost effective for a specific workflow and delivering the performance requirements that you want. So either, you know, you're looking at a customer having everything in a hyperscaler, including their applications where they're going to need an additional lift of

• 24:40Speaker 00

  compute in a NeoCloud to be able to cover use cases that might be more cost prohibitive on top of a hyperscaler. Or you're going to have a customer that has a workflow that truly is hybrid, not just hyperscaler to a NeoCloud like ourselves, but also on prem where they may decide that, well, our primary data lake is actually on prem from a security point of view. You know, or certain applications from a regulatory point of view have to run within a more constrained or controlled environment. So having that ability to mix in the different types of compute, but keeping the data where it needs to be. So the idea is your data stays in the data lake. When you're running inference, you're bringing in the prompts into the model and then the output

• 25:21Speaker 00

  back to wherever it's needed. So uniqueness is we can present those endpoints to the model to be part of their private fabric network that goes to the hyperscaler or to their on prem. So we can do things that are very security constrained for those customers who need it. We can also cover air gapped, truly air gap type of environments. So if they have, you know, security type of oversight that is pretty stringent, we can cover those special use cases as well.

• 25:45Craig S. Smith?

  Yeah, yeah, that's it's really interesting that.

• 25:50Craig S. Smith?

  You know, there was first, everything was on prem. Then things were in local data centers and then, you know, the clouds, the hyperscalers or the early cloud AWS and GCP and those. Everyone went there. And that was a big, it's not complete yet, that that migration to the cloud. But now it seems there is a realization that you need more specific services from the cloud. And so there are these, I guess, what you're calling neo clouds, appearing to provide those more specific services. And is the, well, first of all, is that right?

• 26:32Craig S. Smith?

  And second of all, are the enterprises that you're serving primarily in regulated industries or is it across the board?

• 26:41Speaker 00

  It's across the board. We are trying to focus on areas that are seeing the most uptake for the, where there is a will to spend, you know, banking, finance, insurance, medical, especially on the billing side, medical on the research side. If we look at retail, retail could be a whole series of different customer interaction type areas. It could be, you know, computer vision models mixing in with L L M's for, you know, looking at assumptions of how people are spending, how people are using self checkout, those kinds of areas like that. So it's really taking and arming these teams with the use cases or looking at modernizing from legacy type of models into utilizing, you know,

• 27:21Speaker 00

  different levels of AI to improve those workflows, improve conclusions and improve analytics. One thing that I will touch on is what we're seeing more of is organizations learning in their first initial deployments. And they may have done it on a hyperscaler, you know, serverless type of solution with one or two models. Is they're learning that they need to do what's called a proof of value before they do a P O C or a pilot. Look at all those use cases. So if an organization says, hey, here's a hundred different uses within our organization that could benefit from AI. Proof of value to understand which are the most beneficial.

• 28:03Speaker 00

  And as part of that proof of value, do you have estimates of where you might see some of those savings? And you know, do you have the justification as part of those savings or as part of the readiness that you have to actually achieve some level of, you know, applying in AI, then do you have enough data to go start a P O C? And then you do that P O C to understand, okay, can I now take this from a truly a proof of concept to a lighter weight pilot? Not maybe in the entire organization, but can I prove out that my assumptions from the proof of value to the P O C to the pilot? You know, still hold true so that I can take that to production and see the benefits that I get. And I think organizations really have to get in that mantra to go reinforce doing that proof of value.

• 28:43Speaker 00

  The ones who've shortcut it are the generally the ones who've, you know, had challenges on taking a model to production and, you know, having doubts after they've deployed it.

• 28:53Craig S. Smith?

  Yeah. And the, do you do any, I mean, I'm just curious, the government needs, you know, can't really afford to lean too heavily on, on the big public clouds. Do you get much government work?

• 29:08Speaker 00

  So, so the, what's interesting is the hyperscalers all have done significant work to go after government. So from the federal level down to the, you know, state, local level. And they're going to hit the same dilemmas anybody else does in enterprise where there's going to be solutions that are optimal on top of their existing hyperscaler environments, their solutions that might be optimal to run on prem in their own private environments. And there may be solutions that are more optimal to run with a Neo cloud. Where, you know, we're always growing into new markets and you can expect that government is an area that we will grow on. And a lot of cases, we're going to lean on partners. You know, one of those areas that we just spoke about Qualcomm earlier, one of those areas is we do work closely

• 29:50Speaker 00

  with Qualcomm to understand their downstream partners and how we can work with, with their partners who are utilizing their technologies and then place that in with, with partners that we're bringing in so that there's a holistic solution. So if a customer resonates on certain technologies from a security or other point of view, we want to be. Able to provide a solution that that covers the bases that they need. And we might start to see a new form of data centers deploying for inference as well as part of this. So more inference focused data centers utilized by cloud providers like ourselves. So our inference workloads can run in environments that are deeply connected to other connectivity. So example, it can easily be able to stand up connections to your hyperscaler, but to your on prem.

• 30:35Speaker 00

  And in the government space, that's going to be very important to have that class of data centers tuned for inference. So they can connect to private connectivity that might be more secure or have certain, you know, security requirements built around it as well.

• 30:48Craig S. Smith?

  Yeah, the big hyperscalers, one of the attractions of at least when they first came on the scene was they were self-serve. How do you guys work directly with your customers on specific solutions? Or are they logging on to a web app and configuring virtual machines and things on their own?

• 31:16Speaker 00

  Yeah, so we're a very much a different white glove approach. So it's more about understanding what the customer is looking for. If the customer has the technical acumen and is looking for specific capabilities, we can stand those up. If a customer is saying, hey, I just want something that looks like, you know, open AI enterprise, but with my model or my specific models, we can provide a serverless solution that gives them a, an endpoint and API keys that are binded to their accounts and they can have at it. But if, you know, if an organization comes to us and says, hey, we have a use case, we have a model, we're thinking, how do we go take it? We're happy to advise and we do this a lot today. Now we'll steer into a partner, like that partner may want to help them, you know,

• 32:01Speaker 00

  build out a rag as part of that model or help them fine tune a specific model or just help them go through a P O C process. So we're really all about leaning on partners who the enterprise org is already, orgs already have. So either, you know, the, for the large organizations, those, you know, global system integrators are trying to get into this AI. Services and capabilities, we're totally supportive of that because we want to know there's a services partner within that org. But a lot of organizations don't have that level of capability to lean on. There may be larger enterprise orgs that are less tech savvy. And so, so we want to be able to provide serverless, ready to go type of capabilities for certain types of use cases where if they come to us as an example to say, okay,

• 32:42Speaker 00

  I need a model that fits in for this specific use case. So my employees can ask questions for HR benefits. And here's all my HR data. How do I go throw that together and do a rag and go do that? So we want to provide solutions that help them build that out easier. But at the same rate, when they go one level beyond, we want to be able to say, okay, here's a partner that we've vetted who can help you take this one level beyond. It's something we absolutely can do, but we want to make sure we're scalable and that we're working with partners who are bringing us opportunities in. But at the same rate, we're also bringing them opportunities to grow the customers they're covering and also grow unique capabilities they may be building as part of agentic workflows as well.

• 33:20Craig S. Smith?

  Can you talk about the Qualcomm partnership and why that's important for inference, but agentic workflows in particular?

• 33:30Speaker 00

  Yeah, absolutely. So, you know, there's existing solutions out there that are wonderful in this market. But the key thing for any technology is having diversity in availability of the solutions that are out there. That always moves the needle and moves the market forward. So as serious scale, we always want to be that trusted partner to be able to deliver a wide variety of solutions that are right for the right use cases. And so our relationship with Qualcomm is, well, first of all, we're both San Diego based companies. So there is a close tie and, you know, rooting for our fellow sister company from San Diego. So even though, you know, we show lots of love to the other solutions out there, we also want to start to, you know, build out different technologies and,

• 34:11Speaker 00

  you know, Qualcomm has a unique position because a significant chunk of end user devices in their pockets are tied to some level of Qualcomm technology. So you're looking at, you know, if you look at a good majority of the Android phones out there, there's a Qualcomm, you know, device running in there and there's AI services starting to show up on those devices, you know, device level AI. Services. And so there's a train of thought here of how do you take those technologies that are in a wide variety of different devices at the user's edge and start to leverage those same frameworks and those same software stacks and allow models to run not only on the device, but also lean into the cloud a little bit more. So it makes a lot of sense to have the ability to grow into data center.

• 34:53Speaker 00

  Doing it from a very different approach is, you know, growing from being the all encompassing, you know, edge level computing guy to wanting to go into data center, you know, it's a, it's an area that's interesting to us because it brings a few different shifts. Number one, it brings a different class of accelerator into the market. And it brings a, an accelerator that's built on from a completely different power curve. So power efficiency is going to be a part of it. So, and you need that, you need that raw burstable power for training. You don't necessarily need it for inference workflows. So it's having that balance of, you know, what's truly needed to run those inference type of workloads. And then it's, you know, providing the, you know, all of the software

• 35:35Speaker 00

  that's delivered to be able to make that easier for a cloud provider like us to give that to a customer. Just as you said earlier, there's an incumbent guy out there who's been doing very well in this space. That's built some certain software technologies that everyone in the industry has gotten used to. In inference, it's very different. In inference, there is no tie in to, you know, incumbent languages being part of the workflow. It's servicing the, the AI model. It's providing an endpoint. It's utilizing common technologies that are out there to serve the model and deliver that to the user. And so there's a chance here to deliver something that's a little bit more turnkey for certain types of customers. So as we look at those customers, those enterprise organizations today that are starting to do the proof of value and looking at use cases,

• 36:20Speaker 00

  they may very quickly decide, oh, I can utilize this specific model. As long as I can get it from somewhere, I can utilize this specific model as part of my workflow. And what we really wanted to see was more solutions offering these turn, turnkey serverless type of solutions. So for the less technically, the less, you know, technically advanced Organizations but that still want to deliver great AI services. We're giving them a simpler lift of you can take a solution like Qualcomm's inference offering that we have jointly with them. And be able to make an account, get on there, look at the models that are there, test the models. Look at the endpoint for that specific model, integrate into your application with an API key and have at it.

• 37:03Speaker 00

  And you know, within a, you know, minutes, we're talking from setting up an account to utilizing and finding out the endpoint to getting your API key, you can start integrating that into your specific applications into your workflow. So it's giving the accessibility of ready to go foundational models. But also we're starting to see new technologies appearing in that Qualcomm stack. Number one is dealing with fine tuning models. So deploying a fine tuned model and providing those fine tuned capabilities into the platform. So you can take a model and do some level of fine tuning. So the model is more, the responses on the model are more specific to your organization. And your specific use cases and making that a lot simpler, I think is you know, is a key positive that we're seeing from Qualcomm and building that

• 37:48Speaker 00

  that turnkey ready to go stack and us servicing it as providing it as a service offering, I think is a big, is a big leap forward. To giving that diversity of different acceleration devices in the market.

• 38:01Craig S. Smith?

  Yeah, and I can see that I can see that as inference, I mean, everybody is going to want inference. And we're just at the beginning. They don't necessarily want to have a team of developers going in and configuring GPUs with Coda and CUDA and things like that. They just want the inference. And if all of that work is done on your end or on the chip provider's end, then they're just paying for inference. And at that point, it's speed and costs that become most important. And on why is Qualcomm architecture particularly good for agentic workflows?

• 38:47Speaker 00

  So you're looking into, as part of agentic workflows, you want to have a solution that's optimized for inference, but is able to delete, deal with the correct sized models and provide a cost effective way to go run those. So as part of any agentic workflow, you're going to have a, some level of a foundational, a fine tuned or a rag integrated model. And you want to know, number one, that you're getting the right performance out of that model. You know, time to first token, the token throughputs are good to be able to service that workflow, but that you're also have a, have an environment where you're not having to go pull the specifics of tuning for that specific workflow.

• 39:30Speaker 00

  It should be a, you deploy the model, you deploy the endpoint with your API key, you integrate it into your agentic workflow and it should just work. That's the key thing here is you shouldn't have to go deal, just as you said, with the idiosyncrasies of the underlying accelerator. It should just be a ready to go turnkey solution. So you're focusing on the appropriate parts of building out the right models that are utilized in your workflow. You're spending more time on that because there are going to be certain reasoning and instruct models are going to have better performance for certain use cases. You want to focus your time on understanding your workflow, not configuring the environment and having your team spend more time configuring and keeping the environment up.

• 40:14Speaker 00

  You want to, you want to spend, your developers want to spend the time on understanding which models they benefit from, giving them an easy way to experiment. That experimentation is very important. So letting a diverse group of developers in an organization get specific API keys to an endpoint and be able to have at it to just play around. Because in some use cases, that's part of that initial conversion from proof of value to P O C is you want to play around to understand are, are some of the models just performing better for some of these use cases? And did some of the assumptions you made when you went into the proof of value to the P O C, were they just wrong? Or did you underestimate some of those assumptions? So having something that you can just get on the platform quickly and your team can just have at it, that's very important.

• 40:57Speaker 00

  And especially for these enterprise organizations that might have, you know, developers that have been working on. Workflows into their SaaS applications, you know, are being told to, you know, build a chatbot that somebody can build prompts to go do specific actions within their SaaS applications for a user, as an example. You want to focus on getting the workflow right, getting the right model, building the agentic parts of the workflow. You don't want to have to wonder about what's behind the scenes and is it doing it right? You just want to know it runs, it's cost effective, and it's giving you the best performance.

• 41:33Craig S. Smith?

  Yeah. And working with Qualcomm, I imagine, I mean, they do everything from the edge to mobile to the cloud. So is that one of the advantages of the Qualcomm partnership is you're working with the same hardware provider all the way to the edge?

• 41:50Speaker 00

  Yeah. So I think there's a good, unique differentiation here for end customers. If we look at certain customers who have workflows that touch the edge, they're able to leverage a lot of the same understandings of how models are going to perform in a data center, higher performance environment, but what can be potentially supported on an edge as well. Just understanding the underlying capabilities of the software stacks that they're providing. Again, a lot of that is abstracted from a user, from a customer in these serverless type of use cases. But one unique aspect that Cerescale could do as well is if a customer comes in and says, I'm going to go deploy the Qualcomm stack edge to edge, right? Edge to data center as an example. So end to end, we can, we can provide them the Qualcomm environments as

• 42:32Speaker 00

  bare metal with the Qualcomm stacks, exactly how they would be needed. And then they can go take the Qualcomm development kits for edge and be able to leverage those same libraries, those same frameworks for AI to be able to have some commonality there, too. So for those customers who go beyond the serverless type of environments and want to do something more end to end for their unique workflows, you know, these are areas that we can help those customers as well.

• 42:56Craig S. Smith?

  Yeah. Is this, are we, how, where are we on the curve of adoption for both just general inference and for agentic in particular? It seems to me from people I talk to that we're still very early, that people are still just moving out of the pilot phase. How do you see that? And how is your growth going? I mean, I would imagine.

• 43:26Craig S. Smith?

  You know, it's a it's kind of a slow burn in the beginning, but once people get their P O Cs and move into real production, the volume will explode. And where on that trajectory do you see Cirruscale and the enterprise in general?

• 43:45Speaker 00

  Yes. So if we look at what we're doing today for AI overall, our, you know, historically, our biggest areas in AI, as we saw the transition from HPC to AI, and we were, you know, we've had extremely deep legacy in the HPC space prior to AI coming in. We saw a lot of people in the startup space, building models for specific customer consumer use cases, starting with Cirruscale. And as part of that startup with Cirruscale, they they grow and, you know, we keep a good chunk of them, we keep as customers as part of that growth. And so as they built their models, they're now switching to inferencing. So we have a significant customer base that is utilizing our AI acceleration, you know, across a variety of different vendors for inference today.

• 44:29Speaker 00

  So we do serve a significant inference offering today. For enterprise, though, it is a little bit different. And in enterprise, I think there's two factors here that need to be taken into account. Number one, there's enterprise organizations that really have dug deep into putting the budgets aside for spending on AI, looking at those automation type use cases, you know, empowering employees, potentially, you know, taking outside services and bringing them back in where some of those are being done initially with a model, but then leaning to their own employees. And just the mortgage example that I made was a good example. And so there's some organizations that absolutely see where they need to be. The sad fact is, if you look at some of those organizations and the industries they're in, they may have, you know, a handful of peers who are thinking that same way.

• 45:14Speaker 00

  And they may be, the other 80% of them are like, we need to go do AI. We don't know what that means. So there is that that that dilemma there of every CEO is going to say, we need to do AI, but what does that really translate into? There's going to be a shift. So but the second point is, we're going to see more middleware, more tools that can sit on top of agentic workflows today and provide, you know, chatbot or automation services, but that touch existing applications. So we'll see that in certain vertical customer segments. We'll see it in, you know, federal, state, local government where workflows are going to be able to be accelerated utilizing AI,

• 45:56Speaker 00

  where somebody has decided that they've made a great, you know, piece of glue or a widget that talks to legacy applications, but they can provide a chatbot in front of that, but that's customized, you know, either they've done fine tuning of the model, or they've done a RAG or other customization of that model so that it's more unique to that organization. So I think there's external services partners who really are understanding what they're going to be doing. And I think in some cases, some of these people who really get the gist of building companies around that have come out of the large teams that have been building models. So they understood a lot of these cases. And instead of working for a large organization, building a huge model, they're like, I want to take my skills and maybe spin around where we either go into an existing organization and help them

• 46:41Speaker 00

  build out an AI services team, or we just go start a new team that can provide these services to customers. So I think we're going to see a lift of, of value add, you know, what do you want to call them, you know, value add type providers or integrators, or even the GSIs where they are going to provide stacks of ready to go solutions that can be integrated into customer workflows easier, where an enterprise organization can say, I have this problem, I want to do this, and here's the result I want to get. I think there's going to be organizations that can deliver on some of those capabilities. Now, now for the ones who are totally technically savvy, they're going to say, no, we want to do that on our own. But for the, just as I said, the other 80% who are CEOs are saying, we want to do something in AI, they're going to be really attractive because they'll

• 47:25Speaker 00

  be able to take their legacy SaaS applications, you know, bring agentic AI capabilities, you know, and pair with those and deliver them really quickly. So I think enterprise is absolutely going to go from Chatbot type of, you know, hey, can I provide this information for my marketing team on this copy? And can you SEO optimize this when we put it on a webpage? You know, that kind of simple stuff people are doing today to, you know, more agentic type of workflows, where you're actually taking prompts and having it actually do, you know, services into legacy type of SaaS applications. So that's going to be the future of it. And the players who can deliver those type of capabilities easier for enterprise orgs that may not be as savvy to go deliver, they're going to do well.

• 48:10Speaker 00

  And they're going to pick up the savvy organizations too, because the savvy orgs are going to go, wait a minute, these guys have built out these workflows with some of the applications that I have in the SaaS space. And I don't need to go have my team, I might have my team focus on the more complicated ones, right? If somebody's already built some of these, you know, out of my 150, you know, proof of value type of use cases. If these guys can crack 10, that's huge. I'll go take advantage of them doing the 10 and then I'll go look at the next 90, which are the, you know, two or three I'm going to go do next with my own team, right?

• 48:40Craig S. Smith?

  Yeah, yeah. And so there is going to be the swell presumably. I mean, the reason I ask this is I get reports a couple of times a week from some research company saying how agentic AI is failing, how, you know, 60% or whatever the percentage, some high percentage of pilots don't never go into production. And, you know, you begin to wonder, well, is it that it's just early days? Or is the promise outstripping the reality? Do you have any thoughts on that?

• 49:19Speaker 00

  Yeah, I'll come back to proof of value. So you really need to go build that matrix of all those use cases and then start to score out and look at the, which are the ones that are probably, you know, higher scoring for taking to a

• 49:37Speaker 00

  P O C into a pilot. I think some organizations look at their most difficult problems and say, we want to take this one through AI automation. But when you look at it realistically, objectively, somebody, you know, who's a data scientist may look at it and go, okay, this is actually a hard one to go solve. But if you can solve it, you're going to make a, so I think sometimes organizations try to pick their hardest problems and sometimes they need to just go baby steps and pick the first one or two, deliver some success, start to build up your complexity in what you want to go do. And I think this is where those outside services partners can help. So the inside teams are having these same problems as well. And some of these large organizations who've built up significant teams

• 50:19Speaker 00

  do pivot and learn from their mistakes. There's nothing wrong with making a mistake because you learn from it. And so nothing's going to be perfect, but can we as a cloud services provider, working with our partners who are helping people, you know, deliver on those turnkey type of services, you know, for helping somebody fine tune a model, build out their workflows, you know, are us just working as a partner with like a SaaS provider or middleware provider that's working with SaaS providers? Can we provide that that underlying hardware framework and the cloud services that benefit? So those organizations don't have to be heavily embedded into hardware or costs on operating compute where they can focus on

• 51:01Speaker 00

  helping customers, you know, build out new capabilities. And so, so can we as an, as a, as a NeoCloud, can we help bring on those SaaS providers, those, those middleware or those services partners and be able to say, hey, we give you an easy way to consume the compute. So you can focus on building your services that get more customers onto AI in the enterprise space easier.

• 51:26Craig S. Smith?

  How does someone get started with Sierra Scale and Qualcomm in, in delivering this?

• 51:33Craig S. Smith?

  Real world impact that you're talking about.

• 51:36Speaker 00

  Absolutely. So for those more technically advanced organizations that, you know, have what they need and they're just looking for a better place to run, you know, we can provide those, those compute resources on top of the stacks they've already built on. But however, as we're talking about, you know, a significant chunk of the enterprise space is still probably learning and want to experiment. There's a few different ways that you could start in that. So, you know, Sierra Scale's built out some, a group of different inference technologies as platforms. We've built out our inference platform. We've built out a capabilities called endpoints. But the first place to start is the, the Qualcomm inference offering that we have, because it's a great place to just get on, set up an account. You know, you'll get some level of free usage of

• 52:18Speaker 00

  the platform to be able to do some initial experimentation. You can pick, you know, some of the more leading popular type of models and just start inferencing, integrate them into your frameworks, get a, a endpoint and an API key and be able to utilize those into your type of applications. And because this is all AI based, you know, those organizations who have guys who are writing internal applications are like, I don't know where to start. You can take an AI model and say, how do I connect an endpoint into my application? I'm writing my application in this language. How do I connect an endpoint and throw this prompt into there and get the output? So you can actually utilize the AI models if you're a developer and are nervous about that initial starting point. You can actually take some of these AI models and

• 52:59Speaker 00

  just ask, prompt the model. There's some great models that are more focused on developing code that you can do that to. So you can lean on some of those models. And some of those models are in the Qualcomm platform. So you can not only utilize the platform, you can also utilize the model to help it, help you understand how do you go utilize an endpoint to integrate into your application workflow and, you know, have it help you there. So that's a key part of that as well. And that's starting on that initial inference journey for a customer. But if you're an organization that's already started to build out some initial use cases, you're using a hyperscaler today and you want that model to be a bit more improved, there are some additional capabilities that Qualcomm's added in the platform to help fine tune specific models.

• 53:41Speaker 00

  So that's another area that could be a good experimentation to just come on the platform that's on the Sierra Scale side with Qualcomm and go and take that and just run with it and just start playing around with it.

• 53:53Craig S. Smith?

  Okay, so where does somebody go if they want to give this a try? Is it your website?

• 53:59Speaker 00

  Yeah, absolutely. So go to sarascale.com and there is a way to get to the Qualcomm offering. Basically, just come to our website, go into our menu navigation on the top under products and services, and you'll see under products and services, an inference cloud, which is our offering with Qualcomm. And when you click on the link there, it'll tell you more about the product and it gives you a direct link to sign up and get started and start utilizing it right away. So all you need is to create an account and you're up and running, you know, in like a minute.

• 54:32Craig S. Smith?

  Okay.

• 54:33Speaker 00

  Last thing is because we're a very hands-on, white glove type of organization, we love to understand for teams that are doing their AI journey and feel there may be some level of a fit between Qualcomm and us, we can absolutely do those consultative discussions with a customer, you know, a predictability briefing of where you might need to see your specific workflows or your needs going and how they can be serviced with the inference solutions that we have jointly with Qualcomm. And if things fit outside of that Qualcomm space as well, you know, we're very open to provide advisement on organizations that are just have the technical acumen, but just have never been able to corner an organization, just ask the questions of how they should optimally be deploying these type of technologies,

• 55:18Speaker 00

  how they should be utilizing inference and how they should be covering, you know, for how their, how their IT infrastructure is today. So we're more than happy to help those organizations have a discussion. And again, none of that is obligation. It's more around just understanding where the challenges lie and other areas that we can help.

• 55:35Craig S. Smith?

  Okay, and how does somebody get a predictability briefing?

• 55:39Speaker 00

  Yeah, absolutely. Just go to our website, click to contact us, put your information in, our team will reach out to you. Again, there's no hard sell from us. It's more to understanding what your challenges are and how we can help. And we're always more than happy if there's a different approach to bring that up with a customer and lead them in a different path if that's more applicable as well. We really want to be that trusted advisory partner that helps a customer begin their journey.

• 56:01Craig S. Smith?

  Yeah.