#304 Matt Zeiler: Why Government And Enterprises Choose Clarifai For AI Ops

Try OCI for free at http://oracle.com/eyeonai This episode is sponsored by Oracle. OCI is the next-generation cloud designed for every workload – where you can run any application, including any AI projects, faster and more securely for less. On average, OCI costs 50% less for compute, 70% less for storage, and 80% less for networking. Join Modal, Skydance Animation, and today's innovative AI tech companies who upgraded to OCI…and saved. Why is AI inference becoming the new battleground for speed, cost, and real world scalability, and how are companies like Clarifai reshaping the AI stack by optimizing every token and every deployment? In this episode of Eye on AI, host Craig Smith sits down with Clarifai founder and CEO Matt Zeiler to explore why inference is now more important than training and how a unified compute orchestration layer is changing the way teams run LLMs and agentic systems. We look at what makes high performance inference possible across cloud, on prem, and edge environments, how to get faster responses from large language models, and how to cut GPU spend without sacrificing intelligence or accuracy. Learn how organizations operate AI systems in regulated industries, how government teams and enterprises use Clarifai to deploy models securely, and which bottlenecks matter most when running long context, multimodal, or high throughput applications. You will also hear how to optimize your own AI workloads with better token throughput, how to choose the right hardware strategy for scale, and how inference first architecture can turn models into real products. This conversation breaks down the tools, techniques, and design patterns that can help your AI agents run faster, cheaper, and more reliably in production. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  It started with just inference back in 2014. The reason we started there and having an API platform was inspirations like Stripe and Twilio, where they built a great developer product and people just built amazing things on top of that. And they became kind of the engine for that. And we saw that there needs to be this engine for AI. And so that's the same product on the commercial side as public sector side. We intentionally built a whole suite of UIs that make the system easy enough for anybody to use. And because of that, we've had even intelligence analysts train models themselves, doing all the labeling, picking a training template, looking at evaluation metrics of how they perform without our teams being involved.

• 0:43Speaker 00

  That's key to making the APIs successful, which is ultimately what you're going to use. Once you have a good model, you're going to use the APIs to fire through lots and lots of data for inference. But to get to that model, the UIs are really important.

• 0:56Craig S. Smith?

  In business, they say you can have better, cheaper, faster. But you only get to pick two. What if you could have all three at the same time? That's exactly what Cohere, Thomson Reuters, and Specialized Bikes have, since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment and spend less than you would with other clouds.

• 1:40Craig S. Smith?

  How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50% less for compute, 70% less for storage, and 80% less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is a cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free.

• 2:21Craig S. Smith?

  Head to oracle.com/eyeonai. Eye on AI, all run together, E Y E O N A I. That's oracle.com/eyeonai.

• 2:36Speaker 00

  I'm Matt Seeler and founder and CEO of Clarify. And my history in AI goes way back even into undergrad. That's kind of where it started. A little bit by luck. I was at University of Toronto and in a program where you have to decide if you want to take, you know, many different options of engineering. And I was deciding between the computer option and nanotechnology. And that's when I happened to run into one of Jeff Hinton's PhD students, a guy named Graham Taylor. And he happened to be my resident advisor on the floor I was living in. So that was the luck of it all. And he showed me some of his research. And back then, he was generating videos that looked realistic of a flame flickering. And he said it was all done by AI.

• 3:20Speaker 00

  And so fast forward to today, everybody calls that generative AI. This was 2007. So we've been doing this stuff for a long, long time, well before ChatGPT, where everybody kind of realized its value. And so that got me hooked back then because I learned how to program before that. And there was no way I could program in a traditional sense to generate a video like that. So I decided to dive into the computer option, took Jeff Hinton's course. And then I ended up being Jeff Hinton's first undergrad thesis student ever. That was quite the honor to work with him. And I did a thesis on actually generating motion capture data for, for biological studies of pigeons walking around and how they behave.

• 4:05Speaker 00

  So that was a cool undergrad. And by that point, I knew that I had to, you know, focus on AI. It was going to change a lot of things in the world. And I didn't know enough coming out of undergrad. So I decided to do a PhD. And I looked at all the different schools and chose NYU to study with people like Rob Fergus, who was my PhD advisor and others like Yan Le Cun in his lab. So got to work with both of them and focused on computer vision for my PhD, applying neural networks to understanding images and videos. And that was between 2009, 2013.

• 4:44Speaker 00

  During that time, I also spent a couple of summers at Google Brain and worked under Jeff Dean on the Brain Project. At that time, it was like twenty or thirty people, so it was really fun to work there. And that's where I learned how to scale up AI at the large scale that Google operates at. But I left my second internship a couple of weeks early to come back to New York and start working on building a company called Clarify. Because what I realized is the work I was doing in my PhD was actually working better than what Google had internally for understanding images. And I saw that as an opportunity to start a company around this. And it's been a childhood dream of mine to start a company. It's kind of since undergrad, I knew it would be in the AI space.

• 5:29Speaker 00

  And so this was kind of my shot at that, having really good research. And I incorporated in November twenty thirteen. And about three weeks after incorporating, ended up winning ImageNet with the Clarify name. And that was the biggest computer vision competition at the time. And this was the year immediately after AlexNet, one ImageNet, which kind of put deep learning on the map as actually being an effective method in the first place. So it was great to be the first in the world to reproduce these results and then surpass them. And it was a great way to kick off the company because that got inbound interest from day one, from investors and customers. So that's kind of the origin story for Clarify, and I can go into more about what we do, but I'll pause there.

• 6:14Craig S. Smith?

  Yeah, and on that ImageNet competition, was it you and a team that you had pulled together for the startup? Or were you still working through academia?

• 6:26Speaker 00

  So, good question. I did actually two entries. One was with my PhD advisor, Rob Fergus. We called that ZFNet, or I think other people named us that for Zeiler Fergus. And I got third place in the competition. Somebody else got second, and then Clarify, which was only me at the time, got first place. There was a few different techniques that I was doing as part of the company, separate from my work at NYU.

• 6:53Craig S. Smith?

  Yeah, that's amazing. And and then from there, you very quickly clarified, you very quickly grew into

• 7:05Craig S. Smith?

  one of the first and one of the most influential image recognition companies and you began working with law enforcement and and

• 7:19Craig S. Smith?

  the Pentagon on Project Maven. Can you talk about that? Because I remember at that time

• 7:29Craig S. Smith?

  I or or around 2017, 2018, when I really started focusing on AI, I was, I wanted to talk to you guys and you were like, you know, not available to talk. And I don't remember what it was. I think that there had been some sensitivity around the law enforcement's applications or something that and you guys just didn't want to be out in the media. Can you talk about that period?

• 8:01Speaker 00

  Sure. Yeah. So kind of between starting the company at the end of 2013 to that period, we built a lot of products around computer vision, really a horizontal platform. We had our first APIs up in 2014, serving inference traffic. We started by building what people call foundation models today for computer vision, trained to recognize tens of thousands of things in the world. So you don't have to train them yourselves. And we built a few of those for different use cases like wedding photos, food. Our general one is the most popular still to this date. General purpose like social media, consumer photos, all that kind of stuff. Content moderation, filtering drugs, nudity, weapons, another big popular use case.

• 8:46Speaker 00

  And then in 2016, we kind of allowed the platform to let people train their own models, label data and train their own models. And so people could fine tune it for their particular use cases. And shortly after that is when the government actually approached us to be part of Project Maven because they wanted to see can they take a platform like that and have it fine tune on intelligence data. And they got a demo of our product. I still remember this. It was one of maybe the only customer meeting I had on a weekend. The leaders of Project Maven came up to New York to our offices on a Sunday, carrying a military bag and walked in and had me demo for an hour.

• 9:29Speaker 00

  And after that, they're like, yeah, you definitely need to be part of this program. And we were under contract within a month of that. Wow. Still our fastest government contract since, actually. So it was a great experience with the original leaders of Maven because they took the approach that, you know, an enterprise takes on the commercial world. They evaluate different alternatives. They do their research. They find the companies that are doing cutting edge stuff. They talk to them, evaluate them, and then pick and get them under contract quickly so that they can see the value from it. And so that was our experience with Maven. And, you know, working with the military, it did raise a few questions internally.

• 10:12Speaker 00

  And there was other companies, famously Google, that was part of Project Maven, that had similar conversations happen internally. And I still remember, it's probably like 2018, when I had an all hands, that I just told the company that what we're doing in projects with the government like Project Maven is ultimately going to save lives using artificial intelligence to make decisions faster and in a more intelligent way than humans can do it. And because of that, we're going to be focused on this as a company. And some people didn't like that, but and one or two quit after that all hands. But what it did was stop all of the gossip, stop all the debating internally,

• 10:55Speaker 00

  and got us all aligned. And a lot of people after that all hands, many more than the number that quit, said, oh, thanks for doing that. Finally, we're like past this. I'm like really excited to be working on these kind of programs. And I totally agree. It's going to have a big impact. So that was a big, uh, big catalyst to us continuing.

• 11:14Craig S. Smith?

  And in Project Maven, a lot of people think of it just as uh, vision-enabled drones, but it was much broader than that, right? Can you describe the Project Maven and what happened to it? I've lost track of it.

• 11:31Speaker 00

  It's uh, it's still going. I was literally just drove over from a meeting uh, with a customer right before this and um, I can't go into all the different things that it does, but it broadly became the first AI program within the government and definitely the first successful one. And because of that success, it gathered other lines of effort. Uh, all different types of sensor data you can imagine the military having. They want to apply AI to, to again accelerate those decisions and, and make better ones. And so it started with drones, uh, but it's gone to many different directions from there.

• 12:06Craig S. Smith?

  Yeah, and it's, it's visual data, or is it anything? Signals, intelligence, or?

• 12:14Speaker 00

  Yeah, I mean, it started with visual, but now uh, there's discussions about every, everything you can imagine uh, at this point. And that includes, you know, with, with Gen AI in the last couple of years, becoming very popular and useful. Large language models, uh, what can you do with them in the intelligence scenario? Uh, that's part of the discussions with, with Maven, but also with uh, many other programs across the, the DOT and intelligence community.

• 12:40Craig S. Smith?

  Yeah, and, and the output from uh, from the Maven platforms is for

• 12:50Craig S. Smith?

  uh, what, what's the primary use? I remember early on, there was a lot of talk about targeting. Uh, and, but, but then there was a lot of uh, sort of, uh,

• 13:06Craig S. Smith?

  Pullback from that by the government saying that it wasn't targeting it's it's really for intelligence. Uh, analysis, but where has that gone? I mean, where does it lie now? Is it still with the army? Is it uh, or with the the uh, the joint uh? What is J A I C? I forget what the A was, but uh? Yeah.

• 13:34Speaker 00

  Joint A I.

• 13:36Craig S. Smith?

  Yeah, yeah, yeah, I was joined AI Center. Is it but is it sort of a joint program across the services? Uh, and and yeah.

• 13:45Speaker 00

  No, it's actually transitioned to N G A. Uh, so it's part of the intelligence community now and that uh, maybe hints at kind of the purpose of it. That's so. You know, automating target recognition, but providing just better intelligence.

• 14:00Craig S. Smith?

  Yeah. Um, and so that's ongoing, clarify. And then you also had uh, products for law enforcement. Are there discreet products or were these? Was it a platform to clarify have a platform or APIs that different agencies? Uh, would employ and and cobble together a system on their side or or did you have? You know, uh, uh, uh, or do you have a product that? Uh, that agencies, whether it's law enforcement or not, uh, can use to scan?

• 14:42Craig S. Smith?

  Faces, for example, or you know, whatever the inputs are.

• 14:48Speaker 00

  Yeah, so everything is one platform. And as I kind of uh, mentioned, it started with just inference back in 2014. The the reason we started there uh, and having an API platform was inspirations like Stripe and Twilio, where they built a great developer product. And people just built amazing things on top of that and they became kind of the engine for that. And we saw that there needs to be this engine for AI. And so that's the same product on the commercial side as public sector side. We intentionally didn't want to build two different products as a small company.

• 15:28Speaker 00

  And what that's allowed us to do is we, in order to do that, we had to focus on making the overall platform really flexible in its deployment options. So when you deploy with a lot of these government agencies, they typically want to go on premise, even into air gap environments, on classified networks that aren't connected to the internet. So all these different parameters go into our builds now so that they're flexible enough to go in those environments all the way up to, you know, AWS, Google, Azure, et cetera. And that really helped us build a much better, better product at the end of the day in handling all those things.

• 16:08Speaker 00

  But we, it also focused us on having just that one product had to take it to solve a given problem. There's usually a collaboration with the customer and our professional services team, who can customize things in the product, customize the AI models with the customer. And the platform's easy enough for the customer to do this as well if they choose to. So that the AI is learning how the customer cares about, you know, recognizing or generating what they need. So flexible enough to deploy everywhere and flexible enough to kind of cater to the problem at hand.

• 16:45Craig S. Smith?

  Yeah, but it's a developer's product, an API into your system that they build into solutions.

• 16:52Speaker 00

  Yep. Yeah, exactly. And on top of the API, we have really simple to use user interfaces. So for example, data labeling is a core component to building AI systems. And that requires a lot of user interfaces to, you know, draw bounding boxes or draw per pixel masks of objects. And then there's UIs for the task management, like what do you want to label? Who is the workforce that you want to label it with? Is it complete? What's the progress? All those kind of experiences, they're not really good as an API only. So we intentionally built a whole suite of UIs that make the system easy enough for anybody to use. And because of that, we've had even intelligence

• 17:34Speaker 00

  analysts train models themselves, doing all the labeling, picking a training template, looking at evaluation metrics of how they perform without our teams being involved. So that's key to making the APIs successful, which is ultimately what you're going to use. Once you have a good model, you're going to use the APIs to fire through lots and lots of data for inference. But to get to that model, the UIs are really important.

• 17:59Craig S. Smith?

  Yeah. And then you you you sort of shifted into inference as a as a service. Is that right?

• 18:09Speaker 00

  So I would argue we started there and now we're, yeah, doubling down on it again.

• 18:15Craig S. Smith?

  Yeah. Yeah, but but in, yeah, can you talk about that? Why is, I mean, I've had Rodrigo Liang and Andrew Feldman, you know, that's Samanova and Cerebras. I haven't had the guy from Grok on. We missed appointments a few times, but that's on the hardware side, speeding inference through hardware. And you're doing it through optimization in the code. Is that right?

• 18:49Speaker 00

  That's right.

• 18:49Craig S. Smith?

  Is that right? And so that you can beat these inference chips or these specialized accelerators without any specialized hardware, just simply by optimizing the code. First of all, yeah, why is inference such a bottleneck? Because it feels fast to to consumers. And then

• 19:16Craig S. Smith?

  Yeah. How how did you go about doing this without and and and does this? Would would your inference strategies just accelerate inference on a platform like Cerebras? That much faster, that much more? Uh, or or is is are they is a fork in the road? Either you go down? Code optimization, algorithm optimization, or you go down the hardware route, or you can't have both?

• 19:48Speaker 00

  Yeah, great question. So inference is a good segue from the last conversation. It's kind of the the production use case. Training the model is like a precursor to the inference. Really, what you want is an intelligent system that you're going to run over and over, get accurate results, get get them quickly, or get them at large scale of batches of data, and do that in a repeatable, reliable way. And that's really where the the company started back in 2014 with our first models. We didn't let anybody train in the platform, didn't let them label or anything. It was just inference. And back then, uh, even going into my PhD, uh, we were one of the first people writing CUDA kernels for AI.

• 20:33Speaker 00

  I started getting into GPU stuff, um, definitely in 2012, maybe even 2011. Um, but at that time, it really started out at Stanford. There was some work from Andrew Ng's lab. Uh, then Toronto picked it up with Jeff Hinton's lab and people like Alex Krzyzewski, writing some really good kernels. And then, uh, all there was only four labs doing AI back then. There was those two, NYU, where I was at in my PhD, and Montreal with Joshua Vengio's lab. And so all four of us were collaborating on these these the use of GPUs. And once Alex had some really good kernels, we all shared them. And so

• 21:12Speaker 00

  I remember distinctly in my PhD, I first adopted them for my experiments and overnight they were thirty times faster. And that just changed the game. Instead of waiting for results for a day, I could go for lunch and come back and have my experiment done. And so, thinking back, I was probably one of the first twenty people in the world actually writing CUDA kernels for AI, which was pretty cool. And that kind of deep expertise was the genesis for Clarify. We had to build toolkits before PyTorch and TensorFlow existed. We had to build GPU operators in Kubernetes before Nvidia was building operators to support GPUs within Kubernetes.

• 21:53Speaker 00

  So kind of all the layers up, we have expertise within Clarify on how to build these systems. And so that realization in the last year and a half was that everybody's still struggling to run these big, large language models that have become very popular since ChatGPT. So why don't we take a lot of this learning we have and a lot of the internal tools that we've built over the years to run models efficiently and at very large scales and open them up as a new product line. And so that's what we launched recently is AI compute orchestration. And we call it compute orchestration, not just inference, because inference is just the first workload. You'll see more coming for training, for agentic tasks that run in the background for a long

• 22:38Speaker 00

  period of time, all coming soon. But inference, we view as the most important one because of the scale it operates at and because of a lot of the optimizations we can do, it benefits inference really, really well. And your second question was, do those optimizations work only on GPUs or can they work on other things? They can work on other things depending on the optimization. So we have a mix of low-level things like CUDA kernels that we optimize. We have things like moving Python code to C++. There's a variety of different tweaks we do, but there's also tweaks that will work across accelerators. Like, we're predicting usually many tokens in advance using good speculative techniques.

• 23:23Speaker 00

  And that should work on any kind of accelerator, you know, a Google TPU, a Cerebrus, a wafer, whatever you may throw at it. So it would be a mix. And we found that it depends a lot on the on the use case that you're using these models for, how well these some of these accelerations can work. So there's this year, especially, there's been a lot of excitement about reasoning models and agentic tasks. That's really what we built the the optimizations within the Clarify reasoning engine for. Because if you think of the how these models reason, they're actually repeating themselves many times. They're referencing a lot of stuff in your large context a lot of times.

• 24:07Speaker 00

  And so there's certain optimizations you can do for those kind of repeatable agentic tasks that that perform really, really well. But they won't work as well on on every general task.

• 24:19Craig S. Smith?

  Yeah, and to unpack some of that, just, you know, I get lost in these conversations and I have so many of them. I'm familiar with with the jargon and and I don't want to slow guests down and asking them to explain things that that the the the more advanced listeners already know, but let's explain to people what a CUDA kernel is and why it's important.

• 24:49Speaker 00

  Yeah, go ahead. Yeah, so so CUDA is Nvidia's programming language for their GPUs. I don't actually remember what it stands for off the top of my head, but but it's been around for like fifteen, twenty years almost at this point. And it lets you, you know, there there used to be languages for gamers to program GPUs to make games, but they're really low level like shader languages. And so CUDA became a general purpose language, just like you can program any software on a CPU. You've been doing that for many decades. This was like the equivalent for a GPU. And GPUs are just unlock much more parallelism than a CPU offers because they have thousands of cores instead of tens of cores.

• 25:32Speaker 00

  And that gives a lot of advantages for neural networks in particular because when you're doing the the matrix multiplications that are common in a neural network, it's highly parallel, so GPUs are great fit.

• 25:46Craig S. Smith?

  Yeah, and but then kernel is a function.

• 25:50Speaker 00

  Yeah, and so, yeah, and so the kernels are an example is a matrix multiplication kernel. So multiplying one matrix by another, that would be executed as a single kernel on a GPU. And so when we're talking about CUDA, we're talking about Nvidia. AMD has an equivalent now of CUDA called Rockum. So there's Rockum kernels, there's CUDA kernels. And that's kind of the lowest level, and it builds up from there. A lot of people today don't have to operate at that level because Nvidia and AMD and Cerebras, Grok, etc., whoever's making the hardware is usually making really optimized kernels for their hardware so that people can just use the software on top, like PyTorch,

• 26:35Speaker 00

  for example, which implements the best kernels for Nvidia GPUs, for AMD GPUs, and other hardware providers. But when you're trying to push the limits of optimization, sometimes you can find techniques at the kernel programming level that are even better than what the hardware providers writing themselves.

• 26:58Craig S. Smith?

  Yeah, and I mean, this is essentially what DeepSeek did, right?

• 27:04Speaker 00

  Yeah.

• 27:05Craig S. Smith?

  Yep.

• 27:06Speaker 00

  Yeah, DeepSeek did a lot of really clever engineering, down to the kernels, all the way throughout the systems engineering, everything.

• 27:14Craig S. Smith?

  And how much of, of what, so the hardware producers like Grok, Samenova, Cerebras, who are focusing on inference, their point is that you don't need to go in and tinker with the kernels in whatever the programming language of that particular hardware is. We've done that, and we provide the super fast inference that you can access through API into our cloud or on premise if you want to do that. Uh, in, in the DeepSeek model,

• 27:57Craig S. Smith?

  it's not the hardware. You could use any hardware presumably, but they're optimizing inference on the software side. Um, it, how influenced were you by DeepSeek? Was DeepSeek just uh, the first to make a splash in uh, you know, the zeitgeist was already working in that direction? Uh, or, or did people see what DeepSeek did and say, "Yeah, that's right. We can do that. Why aren't we doing that?"

• 28:34Speaker 00

  Well, we were working on computer orchestration about a year before DeepSeek came out. Um, just because the general trend was that open source is uh, catching up to the proprietary models. And I think DeepSeek was just a huge catalyst to open everybody's eyes in that same direction. And they did a lot of great work uh, on the DeepSeek team. And they published a lot of it too, open source GitHub repos, uh, good technical reports. So kudos to them. Um, and they're not the only ones. Like, there's been a lot of great work out of Alibaba teams for the Qwen models. Um, there's like the Kimi K2 models, another good one out there. A lot of great open source models are coming out of China.

• 29:16Speaker 00

  Uh, the one we've focused the most on is OpenAI's GPT OSS. And the reason for that is, one, uh, it's built by an American company and we do a lot of work with public sector. So we can deploy that model on premise without uh, any question marks. Uh, but even more important, it's a really efficient model from a compute standpoint. You can run it on a single GPU and get really high throughput. Uh, it, and because of that, you can offer it for really competitive pricing and it's very intelligent. Um, so kind of that combo of, of cheap, fast and intelligent is like the, the

• 29:55Speaker 00

  trivecta, I guess, uh, that everybody looks for. And I, I don't think anything's even close um, from those dimensions. Uh, some may be more intelligent, but then they need eight GPUs, uh, and that costs a lot. Uh, so it's a, it's a great model and, and we're the fastest in the world at, at running it because of all the focus we've put onto it.

• 30:14Craig S. Smith?

  Yeah, and so you're offering it as a service? Or, I mean, yeah, it's not open source. The code is not open source.

• 30:26Speaker 00

  Correct. Yeah. So the way our computer orchestration works is we have some of these popular models like GPT OSS offered as a serverless option, so you can pay per token. But if you, and this is what most customers end up doing, if you like a model or maybe you have your own fine tuned version of it or your own custom foundation model and you want to run with some of the optimizations we have, you can deploy into dedicated node pools, we call them. And these are pools of the same type of instance that can scale up and down dynamically based on your traffic needs and they're dedicated to you. So you don't have other customers making requests in there. You're at full control to deploy whatever you want into them and control how things scale up and down.

• 31:11Speaker 00

  And so we have both the serverless token based pricing and the dedicated, which is priced by GPU hour essentially, options. So lots of flexibility in how we offer up these kinds of models.

• 31:24Craig S. Smith?

  And that dedicated GPU pool or whatever, is that on your cloud? You offer a cloud or is that in any cloud? And you have ways to bring fence the GPUs or is it?

• 31:40Speaker 00

  Yeah.

• 31:41Craig S. Smith?

  Yeah.

• 31:42Speaker 00

  So we actually offer both now. So in our VPCs, you can provision clarify compute. And they're in our, you know, overall cloud VPCs that sit across multiple clouds. You can also deploy into your own VPC or even your own bare metal Kubernetes clusters. So wherever your compute may live, you can run. And the big advantages of that are you get that kind of flexibility. So maybe you start on premise because it's going to be the cheapest if you have the upfront capital to buy machines. But you're kind of by definition at a fixed size when you're on premise. And so you need a way to spill over to maybe a Neo cloud as the next option. And then even Neo clouds have a limited capacity.

• 32:27Speaker 00

  If you're really taking off, you may have to spill over to a hyperscaler or multiple hyperscaler. And so you can do that all within the Clarifile platform without having to retool or re-engineer things for every cloud or on-premise differently. You get the same kind of performance on on chips wherever they may be. And that includes not just Nvidia chips, but AMD. And we're working towards support for Google TPUs kind of as we speak. So there'll be other accelerators offered in the near future as well.

• 32:57Craig S. Smith?

  Yeah, and you were saying that that this can run on any hardware. So it could run on Samba Nova or Cerebras. But does it accelerate their inference speed? Does it accelerate inference beyond the speed that they offer? I mean, is there a cumulative effect? Or is it, you know, it's it's going to be what you guys can manage? Or it's going to be what Cerebras can manage on its own? It's not going to be some multiple of that.

• 33:33Speaker 00

  So those platforms, we haven't focused on yet. I'd love to explore that with each of them. But in our experience with new chips, there's a long lead time for the software that they provide to catch up and for especially for other third parties to implement optimized software. And so I think Google TPUs are at that point now, but they've been in the works for like a decade. And and other platforms, I think, are really early on how well the software stack will allow other third parties to to write good stuff with. Cool has been at it for a long time. AMD's caught up with Rockem.

• 34:15Speaker 00

  But the others, it would be hard, I think, for for somebody like us to squeeze performance out. And I think that's actually why, you know, you see people like Cerebras. They used to sell the wafers a lot. That was their business. Now they're focused on like serverless. We're just going to host the models and be really fast at it. And they are, they're great. But I think they found it difficult for people to write the software for the hardware that they were selling.

• 34:40Craig S. Smith?

  Yeah, exactly. And so where does this go? Does how does this change your business? Does it just extend your business?

• 34:51Speaker 00

  Yeah, I I think it accelerates it because I think the what we found with computer vision is the and we're still named leaders in computer vision to this day, like Forrester named us leaders last year in their computer vision platform report. We even surpassed everybody like Google, AWS, so we're we still kind of wear that crown, which is awesome to see. But with computer vision, we find that the use case is always very specific. And you need some of those professional services to train for a given customer. So a good example is we've just signed a deal last quarter with a NFL football analytics company. And they want to use our platform to do all the sports analytics that they're doing.

• 35:36Speaker 00

  But they need our PhDs to help them train the best possible models to do that analytics very accurately. Whereas with large language models, it's kind of the opposite end of the spectrum where the one size fits all model is so general purpose that it can be prompted to do so much that the game there, the focus there is if you can build really performant, really reliable systems that like our clarify reasoning engine is doing now. It's how quickly can you and cost effectively can you offer it to customers? And so we, once we launched our benchmark results on artificial analysis, showing us as leaders in the GPT OSS model,

• 36:19Speaker 00

  we closed the inference deal in five days. I don't think we've ever closed a computer vision deal in five days. So if that signs to come in a in a repeatable way, we're very excited about that that new offering.

• 36:32Craig S. Smith?

  Yeah, the you mentioned token throughput, is that the the the key to inference speed?

• 36:42Speaker 00

  There's a couple important dimensions and the throughput is one of them, the number of tokens per second. The other is usually time to first token. And there's even measures, different measures of that one, which is time to first token period. Or there's time to first answer token. And that means with these reasoning models, they do a lot of thinking before they respond with a final answer. And so the time to first answer token is kind of combining both time to first token and throughput. Because if you have to go through all the reasoning quickly, you have to have high throughput. But then time to first token is kind of when reasoning starts. And we're doing really good at both.

• 37:26Speaker 00

  And that ultimately gives you the time to first answer, the final token of the answer, which is really fast. And we've had some customers, after they saw these benchmarks, were like, let me try this ourselves. And they compared us against other popular inference providers out there. Who really focus on similar things, kernel optimizations, they tout themselves as being fastest at inference. And the customers came back and said we're 65% lower time to first token, 11% faster output token per second. And that resulted in 40% faster time to answer. And they signed up within five days.

• 38:06Speaker 00

  So we're really excited to see the real world performance because it's going to depend again on your prompt, your data, the model you choose to run. There's a lot of variables in in overall performance. And so we urge every customer to to test it out and and work with us to see if we can push it even further.

• 38:24Craig S. Smith?

  Yeah, and again, for the lay listener, a token is is a chunk of data. That's roughly part of a word. And the throughput is how quickly the model can can process those tokens. And that's a matter of computation speed, right?

• 38:51Speaker 00

  That's right. Yeah. So when you're talking about text, a token is usually like three to four characters of text, uh, sometimes bigger, sometimes smaller, depending on the token. And, uh, some of these models have a vocabulary of like 200,000 tokens. So that's kind of the, the options that it gets to choose from every time it generates, uh, takes one generative step. And in terms of our like optimizations with GPT OSS, we're talking about doing like anywhere from 500 to a thousand of those steps per second. Uh, that number of tokens are being generated per second. So it gives you the sense of, you know, every, every little microsecond, actually, when we look at our profiles,

• 39:34Speaker 00

  matters in being able to generate that quickly. Cause if you have to do something a thousand times every second, uh, a millisecond is, is all you have.

• 39:42Craig S. Smith?

  Yeah. And does that affect how well? Uh, because context windows, you know, the, the number of tokens that, that you can provide a model in as input has been growing dramatically. But, but models get kind of lost, uh, when the context window becomes too large. Uh, i- is that, uh, the, does the speed of token throughput, does that improve the model's, um, ability to handle large context windows? Or is it's your view, it's, it's the same problem. It's just

• 40:25Craig S. Smith?

  seeing it faster?

• 40:27Speaker 00

  Right. It, it would still have the same problem. That's more of an architecture choice, training choice on how to get that, uh, they call it the forgetting kind of problem. Um, uh, but with some of the optimizations we've done, it actually helped a lot on the long contact speed. Um, like we improved long context processing, like a hundred thousand tokens or more by 5X from where we started. Um, so, uh, the default implementations you get from some of the open source models and open source toolkits that run them is pretty inefficient in those long context scenarios. But if the model is still gonna forget, it's we we're kind of optimizing performance, not that would be an accuracy problem.

• 41:12Craig S. Smith?

  Yeah, yeah. Well, I've been talking to a lot of people about sort of post transformer architectures to deal with that problem. In particular, state space models like Mamba. I don't know if you know the manifest people in New York, they're coming out with some interesting models. Anyway, those state space models, you work with those?

• 41:40Craig S. Smith?

  And these, you know, sort of post transformer iterations that attempt to deal with the with the context window problem and forgetting and all of that.

• 41:53Speaker 00

  Yeah, I mean, GPT OSS even interleaves different types of attention mechanisms. So attention is usually the operation that has to look. It can become really slow with a long context. So a lot of the focus is there on things like Mamba and others, but GPT OSS interleaves different types. Some of the Qwen next models that came out recently are are doing some interleaved hybrid attention, they call it. So yeah, we're we're familiar with a lot of those. They help a lot. I think that's why GPT OSS in particular is is so fast.

• 42:27Speaker 00

  But I haven't personally dove into the state space models, but I'm optimistic that transformers is not the final architecture.

• 42:36Craig S. Smith?

  And the reason I'm I'm interested in this, if if you can handle massive context without the model getting lost or forgetting or missing information, then you you you you don't

• 42:56Craig S. Smith?

  really need like a vector database or or these other external knowledge bases. You can put it all in the context and have the model do inferences on the information in that context. Do you see that as is where this is going? Or or do you think

• 43:19Craig S. Smith?

  there are other architectures that that'll solve those problems?

• 43:24Speaker 00

  I do think it's where it's going, but it will probably create need for for different architectures along the way. Yeah, I I think the there's a growing need for like, think of use cases like I want a personal assistant who knows everything about me, like every conversation, every meeting, every tool I've used. Uh, that's a lot of data and and and it never stopped growing. So uh, that's kind of the direction we're we're moving towards and it's gonna necessitate a lot of both engineering and novel AI architectures.

• 44:01Craig S. Smith?

  Yeah, and and what are some of the use cases that you've seen for the super fast inference where where inference speed really makes a difference? I mean, of course, everybody wants things to go faster, but what applications do you see this being critical for?

• 44:19Speaker 00

  So time to first token and really the time to answer come in, especially with these chat kind of real time experiences. And we've all experienced it, whether you're using chat GPT, Gemini, Claude, whatever your favorite is. The longer it takes, the the less you want to use it. And I've actually like moved to to Gemini Flash because it's very fast and and very consistently fast. Google does a really good good job at serving these models. And and so the the response speed for like every day I'm in the middle of something, I'm like stuck in in code. I want to ask, is this possible or whatever? Jump over Gemini, get an immediate answer and I'm back at my task.

• 45:04Speaker 00

  If you have to wait even a minute, you're you're distracted, your context switching in your brain and it's gonna slow you down. So the the latency is important for those kind of use cases for.

• 45:17Speaker 00

  A lot of the agentic workloads. So I remember having this conversation with our investors a couple of years ago that about six months after ChatGPT came out, that I said, all the future stuff is not going to be in a chat window. It's going to be agents working in the background. And this year, I'm happy to see the shift, especially on coding tools, to becoming agents that work in the background. And we're using a lot of GitHub Copilot internally to do that. And I think their experience is great. I never took the plunge into things like Cursor where I have to actually change my editor in order to use their product.

• 45:58Speaker 00

  I just want the AI to do the work and I can stay in the editor that I'm familiar with. And in those scenarios where you're chewing through tokens, like they do a lot of thinking, a lot of iteration, but they're asynchronous, the throughput is the important measure there. I don't care about time to first token because I'm asynchronously queuing that off and it's going to send me a notification in 20 minutes when it's done. So I think those are good use cases for, for a lot of the token throughput optimizations we've done.

• 46:31Craig S. Smith?

  So Clarify has launched this as a service.

• 46:38Craig S. Smith?

  And, and I presume one of the reasons we're talking is it's growing fast or, or not. I mean, what, what's, how, what's the response been? And how crowded is the inference market now? Since everything seems to have shifted the inference from draining.

• 46:56Speaker 00

  Yeah, the, the response has been awesome. And, and especially that validation that customers, as they test it out, are, are giving back to us. And we're going to be available soon in, in many different locations. So on the coding agent side, we're integrating into tools like Openhand, which is like an open source agent for coding. We're in, we have a pull request out for, for Cell's AI SDK. We're already in Langchain, Lamindex, popular ones like that you're used to, Crew AI. And we're going to be on some of these open routers as well, like Open Router, where they kind of go across all these inference providers and rank by performance and price.

• 47:38Speaker 00

  So we're expecting a lot of increase in volume because of all these initiatives to basically be available where the developers are. So we're excited for that. It is a crowded space, I would say. Totally agree on that. But I think when people look at vendors, they look at, you know, do they have a track record of doing this? And we have the longest track record out of anybody, having done this since 2014. We were even out two or three years ahead of any of the hyperscalers with APIs for AI. And so they look at that and they build a lot of trust based on that. And then look at the performance we're getting in these kind of public benchmarks.

• 48:21Speaker 00

  And that excites them as well because they can do it in a cost-effective way. And then finally, they look at kind of the flexibility and being able to run in our cloud, in your cloud or your on-premise, and you get to choose, you're in full control. None of the other inference providers are offering that. So we like our position. We're really excited about how this latest product has come together. And now that it's out there, getting that customer feedback to validate it is great.

• 48:50Craig S. Smith?

  Increasingly, inference is moving to the edge. As a lot of this happens on device and now we've got,

• 49:01Craig S. Smith?

  you know, self-driving vehicles hitting the roads where you can't be relying on the cloud for inference. Do you guys work at the edge?

• 49:13Speaker 00

  Yeah, and even some of these government contracts, we've done work on those. Nvidia keeps changing the name. It used to be Jetson and Tegra, then Orin, now there's something else, I think. But yeah, those low-power Nvidia GPUs are great. And, uh, and yeah, we can do all the processing local, uh, within them. And edge also means a few different things we found as we talked to different types of customers. So one can classify it as that, those kind of low-power devices. Another, uh, like telecoms, for example, we're working with a few right now where they have a huge number of data centers, uh, to power internet and your cell phones.

• 49:55Speaker 00

  And realizing that that could be a big advantage in kind of the serving up AI traffic. If you put good compute in those data centers, and we're talking about like thousands of them versus AWS maybe has like five or so regions in the US. Uh, so it's kind of a couple orders of magnitude difference. And so they can be very local to where you are. So really low latency, um, kind of streaming applications like maybe you want, uh, video analysis of a operating room, uh, in a hospital and getting alerts on that, or even driving a robotic arm, uh, automatically using AI. Like those types of applications, you really need lots of processing and you need it to be really low latency.

• 50:39Speaker 00

  So, uh, that's another definition we hear of edge. Uh, it's not a low-power 10-watt device, but it's really close to you, uh, where it's needed.

• 50:48Craig S. Smith?

  Do you think that, uh, right now so little of all the inference in the world is taking place at the edge? Do you think that that will shift as robotics become more versatile?

• 51:04Speaker 00

  Absolutely. Yeah. I, I think the next big wave, well, a couple of thoughts there. So.

• 51:10Speaker 00

  It would be interesting to know how much inference is being done already on Android and iOS devices. I know iOS does a lot. I have an iPhone and I use, for example, dictation functionality all the time. And it works pretty good. It still doesn't get Kubernetes, which is like the most frustrating thing, but one day AI will catch up.

• 51:34Craig S. Smith?

  The word Kubernetes?

• 51:36Speaker 00

  Yeah, it always spells Cooper Netties. It's like a person's name or something. But at least it's happening on the device. So there's lots of inference already happening on the edge on mobile devices and that'll continue. And every time Apple does an announcement and Google does an announcement, they're pushing the limits of how much capability is in their processors on these devices. So Qualcomm's doing a lot in the AI space now, too. So I think we'll continue to see a lot more in mobile. And then to your point, I think the next big wave in AI is going to be the physical world with robotics. I think it's like five

• 52:15Speaker 00

  to 10 years out before we'll be amazed by robots. Even the most advanced ones we see out of Tesla are not that amazing right now. I'm sure they're going to get great, but I don't think it's the AI. Actually, I think it's the the physical capabilities, the actuators and all that kind of stuff. I have a long way to go to replicate what a human is capable of. So I'm excited for that, but that'll drive a lot of the compute towards the edge as well.

• 52:46Craig S. Smith?

  Is there anything I haven't talked about that you'd like people to know about what Clarifile is doing or what you think is going to happen in the future? Whether we're all going to regret this AI moment?

• 53:03Speaker 00

  Yeah, I mean, uh, no, I think we covered a lot. We're, we're very excited about the new offering. A lot of people remember us as computer vision because we started there and continue to lead there. But we do so much more in the gen AI space today. So eager to have more people try it out and give us good feedback and we'll continue to make things even better.

• 53:23Craig S. Smith?

  In business, they say you can have better, cheaper or faster. But you only get to pick two. What if you could have all three at the same time? That's exactly what Cohere, Thomson Reuters and Specialized Bikes have. Since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development and AI needs. Where you can run any workload in a high availability, consistently high performance environment and spend less than you would with other clouds. How is it faster?

• 54:09Craig S. Smith?

  OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50% less for compute, 70% less for storage and 80% less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is a cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com/eyeonai.

• 54:52Craig S. Smith?

  Eye on AI, all run together, E Y E O N A I. That's oracle.com/eyeonai.