Building Durable AI Agents

What does it take to move AI agents from demos to reliable production systems? In this episode, Hamza Tahir explores how MLOps principles are shaping the future of generative AI, covering workflows, agent harnesses, fleets, and the infrastructure needed to build durable, scalable systems. The conversation dives into open source tools, production challenges, and how ZenML's new project, Kitaru, helps developers build resilient, replayable, and observable agent systems. Featuring: Hamza Tahir – LinkedIn https://www.linkedin.com/in/hamzatahirofficial/Daniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: ZenML https://www.zenml.io/Kitaru https://www.zenml.io/product/kitaruMachine Learning Tools Landscape v2 (+84 new tools) https://huyenchip.com/2020/12/30/mlops-v2.html?utm_source=chatgpt.comSponsors: Framer: The enterprise-grade website builder that lets your team ship faster. Get 30% off at framer.com/practicalai http://framer.com/practicalaiUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars! Midwest AI Summit 2026 https://midwestaisummit.com/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:41Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whiteknecht. I am CEO at Prediction Guard and really excited for today's episode because it fits right in the theme of our show, which is practical AI, focusing on some things that are actually useful and practical. Have with us today Hamza Tahir, who is co-founder at ZenML. And they have a new product, a new project out, Kitteroo, which is focused on agents and making agents durable, which is super interesting. And Hamza is joining us today. I think you were out at the AI Engineers World Fair, right?

• 1:24Speaker 02

  Yeah, I am. It's like 7,000 people. All of our crowd gathered in one small, like, big hallway. So it's just fantastic to be in San Francisco when the energy is so high.

• 1:36Daniel Whitenack?

  Yeah, that's awesome. Always inspiring and really cool to see also growth in that from Swix and others, who've really built up an amazing community over time, friends of the show. So if you haven't checked it out, go ahead and check out what they're doing over there. But yeah, excited to dig in today. Hamza, maybe just to set the stage. I know you're a co-founder of ZenML. Is kind of some background with that project? And product around ML Ops. Now you're getting into agent, agentic things.

• 2:16Daniel Whitenack?

  I love your perspective on maybe first off, kind of the world that you have been inhabiting around ML and ML pipelines as now we're all thinking about agents and generative AI and all of these things. Like what, from your perspective, before we get into agents specifically, like what role does the more traditional ML models, training pipelines, et cetera, play in our world moving, moving forward from your perspective?

• 2:48Speaker 02

  Awesome. That's, I think, a great one to start with. And thank you for like inviting me on the show. Appreciate the opportunity. I, so I co-founded ZenML about five years ago. So this was really almost at a point where ML Ops was really reaching fever pitch on, you know, but there was all sorts of chatter about how to productionalize AI and machine learning workloads. And I had done four or five years of that in my previous job where I was co-founding another company, trying to deploy ML models in disparate, you know, compute backends and all over, especially out of Germany where I'm based. So

• 3:29Speaker 02

  the, like, that led me to, you know, like, having a framework internally that we used that you could write workflows and DAX and you could deploy them on these different backends. And that turned out to be ZenML, we open sourced it, we got a bit of traction at the beginning and a few projects and and revenue and we raised and that's been the story so far. And smack dab in the middle of this, from then, from then to today, we have, we had the agent renaissance and it felt like, it felt a bit, a bit funny because in ML Ops, it was like DevOps reinventing itself. And with agents, it's like ML Ops reinventing itself. And at the end of the day, it comes down to these very basic principles of how to write

• 4:11Speaker 02

  good software engineering code that runs non-deterministic code in a way that's safe and reliable and retriable. So I think if any- Thing, even if you throw away every other tool that we ever used in ML Ops, the principles and the learnings that we took from productionizing these applications at scale still translate and are being rediscovered even at the AIE world sphere. I sometimes hear talks, I'm like, I seem to remember I've heard this talk before in the ML Ops conferences.

• 4:39Speaker 02

  So yeah, I'm happy to chat deeper if you're, if you're interested in a particular.

• 4:43Daniel Whitenack?

  It's interesting, like you're talking about the workflows, DAG pipelines. There was very much this phase, at least this is how, how it occurred to me. I don't know if everyone had this perspective, but there was this phase with generative AI around like workflow automation. And there was this very much workflow focus for some time with things like N8N or whatever. And those tools still very useful, of course, but there is like this, this DAG focus. And now it seems like people have thought, well, I remember having Jeffrey from News Research on the show. And he's like, well, with like their Hermes agent or, or whatever.

• 5:26Daniel Whitenack?

  It's like, well, I don't want to impose my workflow into this, but there's still a, there's still a workflow under the hood. Like there's decisions made, there's a workflow executed. It's just like you're not, not defining it. And so from the human perspective, you actually don't see that, that workflow in like a visualized DAG. But it, it sort of exists under there. Is that partially why you think like some of these principles carry, carry over or reinvented in a new way? Because like at the end of the day, there is some flow of things being executed, right?

• 6:01Speaker 02

  Yeah, I mean, a graph, like all software is essentially a graph, right? When you write code, you have if this, else that, execute this, execute that. That's like a sequence of steps you're doing in a graph. I think the, the way we thought about. Workflows before were more deterministic, obviously. For example, I come from the world of machine learning pipelines, so it's like load your data, reprocess your data, train your model, evaluate your model. So, you know, we sort of knew the steps ahead of time. And that lended itself to a graph structure. And, you know, graphs introduced order to the chaos of just willy-nilly scripting things. And then, obviously, DAGs are directed acyclic graphs, but they are graphs, so they're directed in one direction.

• 6:46Speaker 02

  They're acyclic, they don't have cycles, they don't loop back. So in the world of agents, the acyclic part gets very tricky because it cycles all the way down, right? It's loops. So it's, it's, it's where we had to reinvent ourselves as well. So back in like 2023, four users started hacking our pipeline engine to run agents like dynamic steps, conditional branching, state through artifacts, to workarounds. And then, you know, we were like, okay, they're starting to fight our abstraction. And we, so we introduced a new dynamic mode. And I think the main, the key difference here is really, as you said, everything is a workflow.

• 7:28Speaker 02

  Everything is a, an agent, in my opinion, is just an unrolled graph. So it's like LLM call, tool call, LLM call, tool call. Sometimes you do the two calls together. So you're just, it's like a tree structure. So the trace, I guess, is a graph. And, and we just needed to make our system more capable of having graphs that are defined in real time versus statically compiled at the beginning. And, and we did that quite early. And, and since then, the same abstractions have worked.

• 8:00Speaker 02

  Except, yeah, and we can get into this. There's these whole new workload that we need to think about, different things about, like durability, a state management, retries, and how, how those things work and replays. And those are the things I've been working on in the last two years.

• 8:14Daniel Whitenack?

  Yeah, I, I would be curious to know, maybe, as we get into, to those things, maybe just some of the...

• 8:23Daniel Whitenack?

  I guess some of the the stories or types of failures that you that you see, there's like one one piece, which is like how how we handle those, how we instrument things, how we, to your point, make our agents more durable. But that's assuming like they need to be made more durable. They are currently not not durable. They're fragile, right? So so what are the, I guess, help the audience understand some of those main categories of why agents in in our world today are not durable or reliable or however you define that?

• 8:59Speaker 02

  Yeah, so I think in order to in order to to follow that thread, we need to see where the world is going, right? So if you look at how agents, most people, when they think of agents, honestly, still outside of our little bubble of these 7,000 people in the Moscone Center, they they think that agents are these local Claude code instances or, you know, the, you know, or like Hermes or something.

• 9:25Daniel Whitenack?

  Yeah.

• 9:26Speaker 02

  And I think that because we come from this revolution of local local processes that run on your computer and your token maxing, you know, at in your local machine, I think that it's very hard to then understand durability, because that durability, I guess, and recovering from failure in your local machine is is turns out it's a simpler problem than when these things migrate out of your little machine and go into a sandbox or somewhere in your computer, in computers running in the cloud, defined by controlled by your company that are executing arbitrary code and doing all sorts of things like MCP calls and and and, you know, like loading skills.

• 10:05Daniel Whitenack?

  And would this be like to to kind of go off of your point? This would be certainly people are running Claude code or whatever on their local machine like like you're talking about. But ultimately, at least the way I've heard companies express this is to really lean into this element of how the future is going to play out. There there will be this. Digital workforce, however you want to think about it. Agents agents that are operating and actually taking taking action within your enterprise infrastructure, right, that aren't tied to someone's laptop.

• 10:44Daniel Whitenack?

  And they maybe then eventually are not just acting, you know, in a single agent type of way, but eventually there is there are many of these agents that are operating, which I'm sure adds another level of complexity.

• 10:59Speaker 02

  Yeah, these fleets or swarms or, you know, the multi-agent architectures that that that keep coming up, I think they're getting very real now. So I mean, the companies certainly that we work with are deploying these things. And what ends up happening, once you do that, is there is no limit to the scale that you can reach technically, right? Just regarding token spend.

• 11:19Speaker 02

  And you, I mean, why would you not arbitrarily scale that to 100,000 agent executions per, I don't know, hour? If you could afford it, once that you loosen the yoke of the laptop, right? So I I have the I have the feeling that, you know, as things as infrastructure gets more and more mature and architectural practices get more mature, we we I we can't really predict how much volume of agents would be running, once they're not running locally. And that migration is well and truly underway, right, this year? And people who get there tend to just, it's like this sort of slow adoption. And then suddenly, boom, there's like a crazy increase.

• 12:03Speaker 02

  And then at that scale, of course, you have different sorts of problems with agents like, and I guess that's what we we can spend a bit of time talking about.

• 12:12Daniel Whitenack?

  And and these agents that would run in your in your cloud environment, you know, disconnected from your local environment, you know, people might have some ideas of some of the local ones, like a Claude code or open code or open claw or whatever we're talking about. Could you give us some examples of kind of the the the other? Type of agent, like what are they built on top of? Like, what are these? Do you envision these as mostly kind of software vendors that are building vertical focused agents? And they're deploying, you know,

• 12:50Speaker 02

  Yeah.

• 12:50Daniel Whitenack?

  into companies or companies building them themselves? I'm just trying to get like a bit of a vision for people of like what these things might be.

• 13:00Speaker 02

  So I think this is a big battle in the industry right now. Who owns that part of the stack? So you know, you, let's start with the most easiest part of the stack. It's the harness, right? So the harness and the model providers have been for the last year tightly coupled. And I like, maybe we should spend a little bit of time talking about what the harness is.

• 13:22Daniel Whitenack?

  Yeah, yeah, go ahead, please.

• 13:23Speaker 02

  If that's interesting.

• 13:24Daniel Whitenack?

  We've mentioned it on the show, but I think always it's like, this is one of those concepts that has popped up. And it is often very subtle for people like what we're talking about, right?

• 13:35Speaker 02

  It's very ephemeral anyway, because what used to be an agent is now a harness.

• 13:38Daniel Whitenack?

  Yeah.

• 13:38Speaker 02

  So the definition I cling on to in July 2026 is, is basically, you know, we, we, we had this notion of an LLM model, right? The LLM model is simply a token generator. So in itself, it doesn't do anything. It doesn't do action. So the way we did action was we started three years ago with structured outputs and tool calling. And suddenly, what is structured outputs and tool calling? That's simply imposing policies on what type of tokens can the model generate? To predict the next action inside the environment in which it's playing. So it, you can, ahead of time, you give it like five tools and you say, hey,

• 14:23Speaker 02

  you have get weather or you have like ping Hamza or like ping Dan. And like, you have all these bunch of tools to send emails or do different things. And here's the definition of the parameters of these tools. And then whenever you think that you need more information, given that description, you can return me a bunch of tokens, which I can parse and actually map that to my code and execute it. And That while loop, like that's the agent, right? And suddenly we started having a program that runs on your machine at the beginning, which mapped those tokens back to the tool calls. And turns out that's harder than it looks. It's, I mean, I still remember when I did it the first time, like three years ago. It was like, you literally have the, literally do a, exec like an eval in Python,

• 15:08Speaker 02

  which converts the string into code and runs it. I guess that's still what's happening under the hood. But basically, three years of software development later, we had things like Cloud Code, which started doing more things while having that while loop. So, you know, things like compaction when the context window gets too big, or things like ensuring that the tool goals are, you know, have the right parameters.

• 15:34Daniel Whitenack?

  Indexing memory. Yeah.

• 15:36Speaker 02

  Indexing and memory. So that, that software program is called the harness. So it's basically the thing that, that gives your brain a hands, like a body. So your brain is just spewing out tokens and you're converting those tokens into actions. And the, the outcome that comes out of combining the harness and the model is the agent, right? So, so that's how I think about it. I will obviously be very embarrassed in a year when I'm listening back to this.

• 16:03Daniel Whitenack?

  For, for what it's worth, I, I mentioned Jeffrey and, and News. Like he gave the same, a very similar metaphor of like brain and body. So at least we're in, yeah, we're, we're in safe, safe.

• 16:16Speaker 02

  We're in safe, safe territory for now, I mean, until the industry decides to flip this.

• 16:21Daniel Whitenack?

  Yeah.

• 16:22Speaker 02

  So, so, so, so obviously the most famous harness is being like some, some interesting phenomena started to happen. Because what we ended up figuring out is that the model started to asymptote in performance. And what the model providers quickly found out, given their evaluations, is that if you just make this program better and you manage context better and you manage memory and all those things, general purpose tasks get easier to solve. And therefore, you could just make a better program, right? The better program will win, given the model stays static. And then they started a weird RL loop. So what they did was they, like Claude Code, for instance, has a very, you know, Claude Code is the harness and underlying it is Opus 4.8.

• 17:07Speaker 02

  And Opus 4.8 by now, unlike Opus 3.5, understands what Claude Code itself is. So it's self-aware in the way it's that it's running inside Claude Code. So when it calls a tool like Edit File, it actually uses certain parameters and the tool calling is very accurate. If you drop GPT 5.5 into Claude, a code, the harness, that would not be as accurate. It wouldn't perform as good because simply the two things have coupled together. So the harness with the reinforcement learning loop that has gone on for the last year and a half has coupled deeply with the models. Now, on the same time, there has been this renaissance of open harnesses, right?

• 17:50Speaker 02

  So we had Py, we have frameworks like Langraph or PyDantikai. And those harnesses are of the opposite opinion that we should have an open standard that shouldn't be tied to the model because we don't want to be tied to the model. I mean, if the US government decides to ban and unban models, that shouldn't affect our business outcomes.

• 18:09Speaker 02

  And that has been an underlying tension in the industry for the last year and hasn't resolved yet. So I wouldn't know whether, my intuition says that an open harness will win, which standardizes those things. But on the other side, I mean, you know better than most people that Claude Code works, right? So why would I use anything else? So that's another thing.

• 18:32Daniel Whitenack?

  Yeah, I would guess that there's going to be a mix for some time. You see this also with the, you know, existing software vendors who are trying to figure out how they show up within agents, like a NetSuite or a Salesforce or whatever. You see on the one side them officially supporting MCP interaction with their platform because they see maybe, well, the value of our platform is how we manage this data, how we create functions on top of it, the ability to do actions within this environment. And the interface through which people do that, like an agent, really like,

• 19:16Daniel Whitenack?

  are the value is in the platform and that data, not so much the web interface through which people interact with the product. But then on the other side, you see the same individuals, the same companies creating proprietary agentic things in their web, you know, web app, which you can't swap out, to your point, you can't swap out the, you know, in many cases, you can't swap out the underlying endpoint to change out the brain to your point.

• 19:45Speaker 02

  Yeah, because the brain ultimately is commoditized. It's like electricity. So what else are you going to make the next, whatever, a trillion dollars on? And that's the whole stack, right? And we haven't even gotten to the point where it's, so this is all local, right? So this revolution is happening, December 2025, we come into this era of suddenly this works, right? Excuse me for the language. And then it's like, okay, so what has started working? Ah, so we have Pi and we have OpenClaw. And the innovation is that it's running all the time and it has a heartbeat. And it's like, it's managing memory somehow better. And okay, what did we actually change? Or we changed the RNS? So somehow like Peter Steinberger and Mario and Armin,

• 20:28Speaker 02

  like those guys, they had better ideas how to manage that program, which just caught fire. And yeah, and you know, we haven't even gotten to the point where, let's get these things out of the computer and into a Kubernetes cluster or something.

• 20:42Daniel Whitenack?

  Yeah, yeah. Super, super interesting. Sometimes the gap between AI-generated ideas and production-ready work can be really frustrating. Agents can produce things that aren't editable and don't live in your core workflows. That's why I really love what our partner Framer is doing with the way they're integrating agents into their website platform. Agents work in the same place where the real site is designed, managed, reviewed and published. It lands on a canvas, stays editable, and can be published when the team is ready. Framer is a complete website platform, not just Just a builder, so teams can launch and keep improving their sites in one place.

• 21:25Daniel Whitenack?

  Learn how you can get more out of your website from a Framer specialist, or get started building for free today at framer.com/practicalai for 30% off a Framer Pro annual plan. That's framer.com/practicalai for 30% off. Framer.com/practicalai. Rules and restrictions may apply. Well, Hamza, it was really good intro to, I guess, just how to think about or categorize some of these things and the shifts that we're seeing in our mind. Let's go under the assumption that, you know, maybe there's even people listening to this podcast

• 22:07Daniel Whitenack?

  that have envisioned products that have agents in them. They're maybe building agents that they intend to sell into, sell into the enterprise, or maybe it's part of people in a larger company that are building their own agents for operational efficiencies or internal tools. And they are picturing that that future that you described, which is, hey, I actually want these things to run off my laptop. I want to be able to take my laptop, go into a meeting and not, you know, have to keep it up all the time for my agent. So in that sense, then, assume we have some of those agents running in that type of environment. What are some of these, I guess, fragilities or points about agents

• 22:51Daniel Whitenack?

  in terms of how people are architecting them now? How they're building the agent harness piece that make them, that make them fragile, not durable?

• 23:01Speaker 02

  Yeah. So, so I think first, we have to understand the different types of like agent workloads that can run in the background. So if you're talking about a chatbot interface, which is, or a voice agent, that's largely different from a personal assistant that's running and, or a deep research or an auto research that's just doing a single loop

• 23:24Speaker 02

  and, and, you know, always on and, you know, achieving, achieving some outcome. So I think that for all of those different types of use cases, there's different infrastructural pieces and different ways to think about it. So I would say that if it's as simple. Agent that's running, you know, online somewhere. I, I think that I would personally default to something which is very easy, whether that's using the model providers like Anthropic has Anthropic managed agents. They have inbuilt like routines and all those things inside the harness now that you can, that you can deploy and they take care of all of that infrastructure for you. Now, when it starts getting more complex, where your workflows start looking a little bit more business process-y,

• 24:08Speaker 02

  meaning you have a workflow, which is doing things like, okay, fetch an order here and do some processing with an L L M, which is an agentic loop here, and then do some post-processing there. So then it starts looking a bit more complicated. And if you do that at scale, then you might want to own that infrastructure. So the first thing you do is sort of have an agent platform internally as an enterprise. And this, I highly encourage people who are scaling beyond single teams to really invest in, like similar to M L Ops, where the people who build the best M L Ops platforms like Uber, they won their markets. I'm not saying the M L platform was the reason Uber won the taxi game,

• 24:53Speaker 02

  but it was a big reason that they made, you know, things like search pricing and, and, you know, they were the best product out there in the market for a while. And I think the best companies will be the ones that invest internally to build platforms that can run these things at scale. Because simply the act of doing that informs you so much internally of what works and doesn't work for your particular business context.

• 25:15Daniel Whitenack?

  Yeah, and in that, I guess the, the set of things that you need to have in place to support that, let's say I did want to go down that path, I want to own it.

• 25:27Speaker 02

  Yeah.

• 25:28Daniel Whitenack?

  Like, what are some of the main sets of things that like need, I need to be thinking about, I guess?

• 25:34Speaker 02

  Yeah, yeah. So, so the first thing is you need to pick your harness or you need to give your teams the ability to create, you know, using an agentic framework, whatever, let's say you're using Langgraph or Pedant K I or something like that. Then you need to deploy it onto some compute target. Let's say you picked something, probably you have other applications running. So probably you'll use the same thing, whether that's ECS on AWS or whether that's a Kubernetes cluster running somewhere. And now you have an API, so you have a REST API. So I'm more of a Python guy, so I'm going to give Python analogies. So let's, let's say you have a fast API application. You have a POST REST call that kicks off an agent. And obviously, the, the, the first failure mode that will happen is that, you know, you can,

• 26:18Speaker 02

  these, these things are not quick requests, stressful, that just, you know, execute in milliseconds. These are very stateful processes that run for a long time. So they can't run in process, right? It's, this is, this is where you start to farm, farm out things architecturally to things like workers and task queues. And when you, when you start thinking infrastructurally like that, then you need to think about, okay, what is a task queue? How do I, is it a pull-based system? Is it a push-based system? So this is just, we're in the realm of how do we, how do we actually execute this? So that at a certain scale, it keeps working.

• 26:54Speaker 02

  Then obviously you have this, you know, a very simple task queue would be, uh, so, so imagine you have, you have an API and you just, directly from the API, you spin up a worker that, you know, like a salary worker. Well, you don't want to be doing that because what if the worker goes down? Because at scale, the workers can go down for a number of reasons, like network failures or, or maybe the pod didn't exist at that point because somebody was using your underlying compute for, for something else. So a very normal one-on-one architecture is just putting a message queue, a message broker in the middle of that. So you have a bus and the fast API server, which is the entry point, puts events on a bus and then these events are durably persisted.

• 27:38Speaker 02

  And then you have, you have workers that can spin up and down. And you know, you can do an intensive task like a agent to loop inside that worker. From there, you have this problem that, oh, this system is very hard to update. It's, it's, you know, you have to ensure things like eat potency.

• 27:59Speaker 02

  And, you know, you, you need to make that system more observable. So for, for example, if you're processing a file upload and suddenly you have a multi workflow system, like your task consists of two steps rather than just one thing, like maybe you're calling something at the beginning and then you're doing something else. So suddenly you have dependencies between the workers. So suddenly this complexity starts to explode from a very simple, oh, I just need to slap a queue in front of my fast API to, okay, I need a DAG, like a workflow execution thing, right? And then, and then you're like looking at it like, okay, how do I make this DAG orchestration thing work with my harness? And how do I, how do I make things like in-flight updates? Like what if I, my agent up, like what if there's a long-running task for 30 days, right?

• 28:44Speaker 02

  And, you know, that's running for 30 days. I mean, most tasks are not running for 30 days, but they can very soon, right? And what if then the, the next person kicks off another 30-day task? But that's a different, you've updated the code. So what happens in the 31st day? Should I use the latest code or should I use the old code? That's version to that agent system. And, you know, what happens is within the 30 days, the LLM model gets banned by the US government. And I'm going to keep saying that because I'm in the US now.

• 29:12Daniel Whitenack?

  Yeah, yeah.

• 29:13Speaker 02

  So, yeah, I mean, there's a whole plethora of problems that start happening from the infrastructure perspective to make things really reliable and that you don't just write defensive code all the time. Like you don't, you don't want your developers to be writing defensive code. You want them on the offense, right? In use cases.

• 29:28Daniel Whitenack?

  Yeah. And what about, I guess, the, and I, I really liked how you framed this when I was looking through the Kiteru pages and docs is like, there, there's a lot of questions that you could, could be asking, like, what if the tool calls timeout? What if I don't use a rerank or can I use a cheaper? Model for there's a lot of, I guess, developer side questions that that come up. Certainly related to some of those things like the supply chain thing you mentioned about, you know, a model being all of a sudden identified as a supply chain risk, which I think is a very, is a very real one, but there's all these other things that it there's so many possibilities of how

• 30:12Daniel Whitenack?

  you could update your agent harness. It it's very much, um, I I mean, I remember just when I would teach workshops and just talking about ML models or L single L M calls, I would get pushed back saying like, well, how do we test these things like rigorously, right? They're non deterministic and they like I they could return anything. And, you know, I uh, having a background in physics, I I was like, it it's, uh, hey, we wouldn't know a lot about the universe if we weren't able to test things that were were non deterministic, right? So.

• 30:49Speaker 02

  I look, look, this is, I mean, you're so right, because I was lingering too much on the infrastructure side, probably like the developer side, the things that you're talking about is post factum that once those things are running.

• 31:00Daniel Whitenack?

  Yeah.

• 31:01Speaker 02

  So you have some notion of state.

• 31:02Daniel Whitenack?

  Yeah.

• 31:03Speaker 02

  Then you have.

• 31:03Daniel Whitenack?

  It's a multi layered problem. Yeah, yeah, there there is the infrastructure and there it's like both interact in interesting ways, like in in the way how long something takes or how it times out or whatever, certainly related to network and infrastructure. But then there's all of these choices that you can make within the agent, right?

• 31:23Speaker 02

  Yeah, yeah, I'll give you a good example. I mean, imagine you have a coding factory, like this is the buzzword for A I E right now, right? Coding factories. So we're going to automate software engineering and imagine that uh, an agent is running in such a system that I described in workers and suddenly um, it obviously has to execute code, right? So it needs a sandbox. And this is why the sandbox providers like E to B, Daytona Model are so popular nowadays, because you need a sandbox to execute arbitrary code. So that's another problem. Suddenly that appeared on the infrastructure side. And now, what what if the file system that you're editing the code, you fail before you can commit the code? Like, how do you? How do you get back to that state? Um, when, you know, twenty thousand tool calls in, Cloud Code is almost about to finish the feature and it just suddenly fails. And are you mounting the file system into the

• 32:08Speaker 02

  pod? Because this is not your machine anymore, right?

• 32:10Daniel Whitenack?

  Yeah, yeah.

• 32:11Speaker 02

  Typically when pods die. And, and then how do I, like, even if you get to the end and it finishes, like, what if somebody looks at that and says, this is a feature? Like, this is not what I wanted. So how can I go back and evaluate, to your point? Like, how can I go back and evaluate, hey, could I have done this faster, cheaper, better? If I had done an experiment of using GLM maybe instead of G P D? Or maybe I, I wanted a different tool call. Maybe I should have given it so many tool calls, it got confused. Maybe just let's give it two calls and start again. So.

• 32:44Daniel Whitenack?

  Yeah, it's, uh, and I think that would be especially true if you are moving from this, like, this is my personal agent, which I can look at the output and then maybe provide, you know, I, I can actually commit to memory or skills or that sort of, they're like, this is how I want it to go. This is how you should do things. This is, but it's another thing if these are more autonomous, they're operating in the enterprise environment. Or like, I'm a software vendor and I'm creating my own, you know, my own agent, which is my product, my I P. And this shouldn't just be like a general purpose agent, right? Like, there should be an opinionated take on, on how to do these, these things in this

• 33:27Daniel Whitenack?

  vertical that operate better because I'm infusing actual opinions into the harness, right?

• 33:33Speaker 02

  Yeah. Actually, that is the biggest unsolved problem, right? Then like, I, I mean, you've probably been building agents for years now. So I still am terrified updating my agent in production. I'm terrified because I have no idea. I have literally no idea, given the entropy in that system.

• 33:49Daniel Whitenack?

  Yeah.

• 33:50Speaker 02

  How I can even adding a word to the system prompt, what would happen? And that terrifies me in a world where you have hundreds and hundreds of millions of these things running in enterprise in flight, right? And suddenly you have to you have to task the poor agent developer to grok all of these context states. It's a very stateful application and try to make an educated guess as to how to update this system in a way that wouldn't break for all your customers. It's almost impossible, right? Without given the right observability, given the right ability to go back in time and introspect and, you know, try to run some of these experiments. I it's like you need to have some sort of a simulated. I keep saying the physics words entropy is just simulations.

• 34:34Speaker 02

  So, yeah.

• 34:34Daniel Whitenack?

  And and that's some of what you're doing with the Kitteroo, right?

• 34:38Speaker 02

  Yeah.

• 34:39Daniel Whitenack?

  Which is this, could you explain kind of maybe just introduce a little bit of the the idea behind that? And I know there's some really interesting things around replay, for for example. But yeah, I would love to hear more about that.

• 34:53Speaker 02

  So, yeah, Kitteroo comes from this concept of, you know, so it still uses XAML. So XAML is our open source product that's been running in production for enterprises for five years now. So we've really, we sort of know how to do workflow orchestration now. So we, our task this year was how do we convert that into using that engine in a way that's ergonomic to agent builders? And sort of try to answer, try to take opinions on some of the things that we just spoke about. So Kitteroo is is built on top of XAML. It's a new SDK and a new a new UI. That works almost from the harness backwards because I think very important is the harness layer. Because that's what most people are is the entry point to agent building.

• 35:35Speaker 02

  And a lot of the opinions taken in that harness layer, actually, as as we just talked about in this episode, it do matter a lot at the infrastructure layer. So our our goal is to build an open runtime that allows you to take any harness, any and and deploy it in a way that you don't have to think about some of these problems. And then on top, it gives you all these goodies of replay. And how we end up doing that is turns out if you hook into the harnesses, there's certain checkpoints in state that you can snapshot.

• 36:05Speaker 02

  And store while you're running it on, let's say, a Kubernetes cluster. That will make life a lot easier once those inevitable failure modes do kick in. So the first, first order of business was how do we get the best world-class adapters for all the popular harnesses? Like Entropic Aegis SDK, OpenAI Aegis SDK, that you can just drop Kita ru as a runtime in. And it gets it from your laptop out to the production. And then when it's, once it's running, it makes it resilient to tool calls failing in the middle of an agentic loop. By storing that state in an external database or, or blob storage. And once we have that state, then we have all sorts of interesting questions.

• 36:49Speaker 02

  Like, you know, like, like we already talked about, what if you want to mock a tool call? Or if you want to change the replay, like replay a different model and drop that in the middle of your trace? And this is a very interesting problem anyway, like from a scientific perspective. Because if you change the model midway in a multi-turn conversation or a multi-turn situation. Then what ends up happening is that you never really know. That the model, that the agent would have ever gotten to that point if you had started with a different model anyway. So it's, it's anyway a broken experiment. And also the problem is that when you replay from the middle.

• 37:27Speaker 02

  You have certain, like if you swap out GLM with Claude code, like the harness differences also make it just hard to just start again from that point anyway. So, so there's a lot of code that we have to write in order to make that experiment somewhat, you know, feasible. And again, it's not perfect and we're still working with our customers to, to make it good. But given that you're grounded in your production traces, production executions, and you have a big enough sample size. You could then, like we have seen early signs of where you could just say, oh, I should have just used an open source model or a smaller model. Or I, I should have just swapped out some of the tool calls and made it a bit easier. And I would have gotten a better and cheaper result. In in place of that.

• 38:10Daniel Whitenack?

  Yeah. Do you do you view this as what's the way to put this? So like I think of, you know, taking it out of the AI agent world into like manufacturing. You have like a pipeline in in your manufacturing or a line in your manufacturing plant. There's always going to be a bottleneck in there, right? And so often what people recommend, right? Is you find the bottleneck, you ignore the other things, you address that bottleneck, and then you have a new bottleneck, right? So one way to approach this would be that that way. You could also approach this and look at, well, there's all these things happening, right? And it's really, yeah, it there's, of course, all sorts of optimization theory on like how to optimize things.

• 38:54Daniel Whitenack?

  But like, there's all sorts of things that are actually coupled together, right? If I optimize this, then that gets faster, but creates a different, you know, a different problem. So like, what have you noticed after actually having? Which I think a big piece of this, to your point, is like actually getting visibility into into what's happening. But then comes the next question of, well, now now what, right? Like, what are what are the best? And maybe it's a part of like, we don't know the best practices around addressing some of these things yet.

• 39:28Speaker 02

  Yeah, yeah, I mean, exactly like exactly how I imagine it as well. It's like playing a game of vacuum all and trying to sort things out. So when when you fix one thing, another thing breaks. I I think the first order of business, as I said, checkpoint everything. So that's the durability aspect, right? So, I mean, obviously, there are latency concerns, you need to be efficient about it. But checkpoint everything so you can look back. And then I think it's almost humanly impossible to to all the experiments. So you've got to get agents doing that, right? So I I think our goal really in the future is how can we close the loop from like once? Because you sort of know the outcome, right? You want a cheaper, better, faster model agent at the end of the day. So we have the production executions, the traces.

• 40:09Speaker 02

  And the first thing I ask people to do when they're using Gitara, I'm like, run it for a week. And then after a week, just filter for the most expensive traces. Which were successful watered by your customers. Go through the checkpoints and see the bottlenecks, just like you said. And figure out the common failure modes and and then, you know, go from there. But obviously, my goal is not that humans do this. Our, you know, we we shipped an MCP CLI day one of Kitaru because we know that eventually that problem needs to be solved by other agents that sort of are embedded in that loop. So my ultimate goal is you have every time you launch an agent, you have a companion, like a like a companion, a nurse agent.

• 40:54Daniel Whitenack?

  Trainer or yeah, it's a loaded term, but.

• 40:58Speaker 02

  Exactly. I mean, something like a trainer that's, you know, that's constantly looking at, okay, what is this guy doing wrong? And then running experiments in the back, replaying things and constantly editing that thing. I mean, we're very far. I mean, to be clear, this is a this is a this is a dream because then, you know, your agent builders can start from a really state and get to a very efficient state very quickly. But I think we need more tooling and infrastructure to make that possible because that is eventually the future.

• 41:25Daniel Whitenack?

  Yeah, and I guess that does get to, you know, a really good place for us to kind of start wrapping up the conversation, which is as you see the current state and you look to the future, maybe on both sides of this, like, what's one area where you're really encouraged and, you know, it it it excites you of like, hey, things are going in this direction and I I'm really happy about that. And this is maybe what we could expect. And then maybe something like you said, oh, if we could solve this, if we could move this big rock, like that would open up a lot of a lot of opportunities. Any any thoughts?

• 42:00Speaker 02

  So I I'll start with the the thing I'm less optimistic about for now as an industry because and, you know, I'm you catch me at a very opportune time. Right after this, I'm going to go to the AIE conference, right? So I'm going to be at the World's Fair and walking around the expo. It strikes me how many people have different takes on solving the same four or five problems. So I think we're at this fever pitch of like we were in ML Ops in 2021 when Chip Hewen, like, maybe we can drop that article in the in the show notes. It's a very good. It's a very good. It's great. Yeah, like when we had an explosion of tools when you had these ML ops problems and investor money came in.

• 42:38Speaker 02

  And then, you know, it was just extremely confusing to figure out the modern ML op stack or on top of the modern data stack. And I think that this is why, I mean, we're sort of contributing to that noise, right? By being, by taking opinions, but I feel like we are very early in deciding the canonical ways of separating things like the harness and the infrastructure and the deployment paradigms. And I'm less encouraged by that, by the model providers taking such aggressive stances on how to deploy things and how aggressively they make things hard to, for other vendors to, you know, be interoperable. Although there are some good signs sometimes, but you know, it's just they're pushed into the harness. And when you're pushed into the harness, you want everything running on your infra.

• 43:21Speaker 02

  And I, I think there, this is, this is, this is a slight negative economical impact. But on the other side, okay, here, here are things where I'm, where I'm very optimistic. We have amazing open efforts with open source models.

• 43:36Speaker 02

  Take like Minimax or, or like take the Kimi models or, or, or, or take something like GLM coming out of like China right now, but hopefully also the US and, and Europe. With Pistril that, that have made it very economically viable to start, like, start, like, the, the enterprises have started to really take it seriously to replace that, the, the bigger model providers with their own systems. And I think that's going to be eventually where, when they start thinking, okay, we need to have system engineers that think of that problem. And, and start playing around with and fiddling with the other pieces like the harness and the durable runtime.

• 44:19Speaker 02

  And I'm very encouraged by, by the fact that we have now performing models that are just, this is like two weeks old, right? The GLM is as basically 95% of Opus 4.8, which is just phenomenal. Like we, we are, we were at a trajectory that that was very far away. But in a world where you can have sort of the same performance with open models, then I think you see more investment in creating internal platforms that can deploy those agents. And then open harnesses will be a huge thing, in my opinion, because a thing like Pi or a thing like Open Code or something like that. Or even specific harnesses for law or for science or something will probably explode in growth. And suddenly you have this thing where people will start realizing

• 45:01Speaker 02

  that actually investing in internal infrastructure at the enterprise level is probably going to be the competitive differentiator in a world where the tokens regress down to the cost of electricity and the models become commoditized. And this is, I'm extremely, extremely encouraged by the recent progresses around that.

• 45:20Daniel Whitenack?

  Yeah, that's, that's, I think, a great perspective. And I think it encourages people also that are listening to the show. Please check out these, these open projects around open harnesses. A lot of them are even linked in the ZenML website and integrations that they have. I encourage you to check out the ZenML website, you know, try some things in your own environment. It's, it's never been easier to, to spin up some of your, your own tools and, and infrastructure and try things and would encourage that. You'll, you'll find some links in the show notes. Thank you so much for joining us, Hamza. It's been great.

• 45:56Speaker 02

  Thank you for having me.

• 46:03Speaker 00

  All right, that's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X or Blue Sky. You'll see us posting insights related to the latest AI developments and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.