Creating tested, reliable AI applications

It can be frustrating to get an AI application working amazingly well 80% of the time and failing miserably the other 20%. How can you close the gap and create something that you rely on? Chris and Daniel talk through this process, behavior testing, and the flow from prototype to production in this episode. They also talk a bit about the apparent slow down in the release of frontier models. Sponsors: Fly.io https://fly.io/ – The home of Changelog.com — Deploy your apps close to your users — global Anycast load-balancing, zero-configuration private networking, hardware isolation, and instant WireGuard VPN connections. Push-button deployments that scale to thousands of instances. Check out the speedrun https://fly.io/speedrun/ to get started in minutes. Timescale https://www.timescale.com – Purpose-built performance for AI Build RAG, search, and AI agents on the cloud and with PostgreSQL and purpose-built extensions for AI: pgvector, pgvectorscale, and pgai. Eight Sleep https://www.eightsleep.com/changelog – Up to $600 off Pod 4 Ultra Go to eightsleep.com/changelog https://www.eightsleep.com/changelog and use the code CHANGELOG. You can try it for free for 30 days - but we’re confident you will not want to return it (we love ours). Once you experience AI-optimized sleep, you’ll wonder how you ever slept without it. Currently shipping to: United States, Canada, United Kingdom, Europe, and Australia. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: MLOps Community “Agents in Production” event https://home.mlops.community/public/events/aiagentsinprodUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:16Speaker 00

  Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, and accessible to all. If you like this show, you will love the Change Log. It's news on Mondays, deep, technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at fly.io. Launch your AI apps in five minutes or less. Learn how at fly.io.

• 0:57Speaker 01

  What's up, friends? I'm Kurt Mackey, co-founder and CEO of Fly. As you know, we love Fly. That is the home of changelog.com. But Kurt, I want to know how you explain Fly to developers. Do you tell them a story first? How do you do it?

• 1:11Speaker 02

  I kind of change how I explain it based on, almost like, the generation of developer I'm talking to. So like, for me, I built and shipped apps on Heroku, which if you've never used Heroku, is roughly like building and shipping an app on Vercel today. It's just it's 2024 instead of 2008 or whatever. And what frustrated me about doing that was I didn't, I got stuck. You can build and ship a Rails app with a Postgres on Heroku, the same way you can build and ship a Next.js app on Vercel. But as soon as you want to do something interesting, like as soon as you want to, at the time, I think one of the things I ran into is like, I wanted to add what used to be like, kind of the basis for Elasticsearch. I want to do full-text search in my applications. You kind of hit this wall with something like Heroku, where you can't really do that. I think lately, we've seen it with like, people wanting to

• 1:54Speaker 02

  add LLMs, kind of inference stuff to their applications on Vercel or Heroku or Cloudflare, whoever these days, they've, they've started like, releasing abstractions that sort of let you do this. But I can't just run the model I'd run locally on these black box platforms. That are very specialized. For the people my age, it's always like, oh, Heroku was great, but I outgrew it. And one of the things that I felt like I should be able to do when I was using Heroku was like run my app close to people in Tokyo for users that were in Tokyo. And that was never possible. For modern generation devs, it's, it's a lot more Versal based. It's a lot like Versal is great right up until you hit one of their hard line boundaries. And then you're kind of stuck. There's another one. We've had someone within the company. I can't remember the name of this game, but the tagline was like five minutes to start forever to master.

• 2:39Speaker 02

  That's sort of how we're pitching fly is like you can get an app going in five minutes, but there's so much depth to the platform that you're never going to run out of things you can do with it.

• 2:47Speaker 01

  So unlike AWS or Heroku or Versal, which are all great platforms. The cool thing we love here at Change Love most about fly is that no matter what we want to do on the platform, we have primitives, we have abilities, and we as developers can charge our own mission on fly. It is a no limits platform built for developers and we think you should try it out. Go to fly.io to learn more. Launch your app in five minutes. Too easy. Once again, fly.io.

• 3:32Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. In this fully connected episode of the show, Chris and I will keep you fully connected with everything that's happening in the world of AI and discuss some of the latest trends and share some learning resources for you to level up your machine learning and AI game. I'm Daniel Whitenack. I am CEO at Prediction Guard, where we're creating a private secure AI platform. And I'm joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin.

• 4:10Chris Benson?

  Doing very well, Daniel. I know you're out traveling and ironically, I think I'll be I'll be where you are next week, but I think you'll be gone by then.

• 4:20Daniel Whitenack?

  Swapping places.

• 4:21Chris Benson?

  There you go. We're trade. We're trading geographies here.

• 4:23Daniel Whitenack?

  Yeah, yeah. It's a I don't know why November always seems to be heavy conference event summit onsite month for me. I don't know exactly why that is. It's sort of the last last little bit before the end of the year, maybe. I don't know.

• 4:44Chris Benson?

  It's to make you earn that vegan turkey that you're gonna enjoy, you know?

• 4:47Daniel Whitenack?

  Exactly, exactly. Yeah, I got it all picked out, so we're ready for Tofurkey on Thanksgiving for sure.

• 4:55Chris Benson?

  Excellent.

• 4:57Daniel Whitenack?

  And speaking of other things to celebrate, I wanted to mention before we hop into other discussions today that our good friends over at the ML Ops community, so Demetrius and his crew, they've run a series of virtual conferences, one about L L M's and production and another about data engineering and A I. And their latest in this series is called Agents in Production, which sounds very exciting. As you're listening to this episode, if you're listening to it right when it goes live, you can still probably catch the event live, but you can also catch the content afterwards, I'm sure.

• 5:41Daniel Whitenack?

  As it's recorded, it looks like an amazing conference talking about A I agents moving from R and D to reality. Are you ready? Question mark. So go check it out. Their events are always are always great. So I wanted to mention that upfront at the beginning of the show. Chris, do you have any active A I agents in your life?

• 6:05Chris Benson?

  You know what? I actually don't right now, but I probably should. I feel bad that I'm not that I can't say yes to you on that. But you know what? I've also read recently that the the uptake on agents has been a lot slower

• 6:18Daniel Whitenack?

  Yeah.

• 6:19Chris Benson?

  than was expected. You know, there was kind of one of those hype things and I think it's pretty hard. So maybe something to discuss.

• 6:27Daniel Whitenack?

  Yeah, maybe along those same veins. I see a lot of news articles, especially as related to, I think, what, you know, over the past, I guess, couple of weeks or whenever it was that people realized that OpenAI wasn't going to release GPT-5. I don't know if there's expected this year and not really, anyway, not released on the timeline that that people thought. And also some indications that maybe that that next jump in the functionality of these AI models is proving more difficult than than was originally thought. One question I I had as related to that, Chris, was,

• 7:11Daniel Whitenack?

  let's say we never get GPT-5. We're just stuck with all the models that we have now. So no, no more models were made in the in the world. What do you think the value of AI and kind of its integration across the the enterprise and business and our personal lives? Do you think it would still have the, you know, hype transformative effect that people are are talking about or?

• 7:37Chris Benson?

  No, I mean, we've and we've talked a little bit about that, you know, in terms of hype cycles and stuff on previous episodes. If you postulate that we're hitting a ceiling right there, I don't think it's no more models. I think what happens is all the is that there's more open source that comes along and and, you know, kind of at least catches up to where some of the the leading ones are out there and you end up having the value of a commercial model is less because you have more open source options that are out there. And we're seeing that in industry anyway. I mean, you know, not everybody wants to pipe their data out to OpenAI or, you know, some of the other organizations doing the same thing. And so I think the

• 8:19Chris Benson?

  availability of open models is going to happen regardless in terms of the uptake on that. And I think that would just kind of force that to happen sooner rather you you if the leading models are are no longer, you know, new ones coming out that are better and better to chase. They catch up and it kind of commoditizes the whole space even faster.

• 8:38Daniel Whitenack?

  Yeah, I think um, one of the things I was wondering was, let's say that regardless of whether it's an open model or closed model. So if I told you, Chris, you're an AI engineer actively integrating Gen AI functionality. And I told you the best model you're ever gonna get on the open side, if you're using open models, maybe Llama 3.1 or whatever. And on the closed source side, maybe it's the latest Claude or GPT 4.0 or whatever, whatever that might be. So if that were the case, would that be like, uh, man, I, I, I don't think we're gonna be able to do all of what we had hoped to do with AI?

• 9:22Daniel Whitenack?

  Or do you think it's more of a, yeah, what, what is the level of transformation that you think we could still get with the current generation of models, let's call it?

• 9:31Chris Benson?

  So it's kind of funny, and I know we've talked about this, you know, and some of our listeners will remember some of the previous conversations. But there's a lot more to AI than just the Gen AI models, you know, they've gotten all the spotlight the last couple of years. But there's a lot you can do and honestly, you know, without going into detail, the things I think about all day, every day. Gen AI is not the center of it. It's not the stuff in AI that I care the most about. It's not what's making me most productive. So are there many things you can do with Gen AI to be productive? Sure. And we're still learning how to do that. And I think that's harder than people realize to get there.

• 10:10Chris Benson?

  And I think that's one of the reasons it's plunging down into the trough of disillusionment in the hype cycle as people are frustrated. But we will have lots of Gen AI things, but I think it reminds us, as we've said recently, to look at the larger landscape of AI capabilities out there and other things that we used to be excited about are incredibly productive these days. And yet we're not talking a lot about them, you know, deep reinforcement learning remains amazing in what it can do. And, you know, if you combine that with robotics and other areas, there's lots of really productive work being done out there, but it's not getting much media attention.

• 10:49Daniel Whitenack?

  Where my mind goes is that the current models that are available, if you think about that general purpose reasoning GenAI model, are good enough, in my opinion, to do kind of most tasks at the sort of orchestration layer. And what I mean by that is, let's say that you wanted to do time series forecasting. I don't think that, you know, whatever model you look at on the GenAI side, it's not the best time series forecaster. However, you know, there's really good tools for that that already exist. So, you know, Facebook Prophet or something like that. And you can use the GenAI model as almost the front-end

• 11:32Daniel Whitenack?

  to tools like that, right? You can say, hey, I want to know what my revenue is going to be in six months, do a forecast or something like that, and use the model to extract the data that's needed to make that forecast and maybe call a tool like Prophet or something to actually do the forecast and get something back. So I think even if we got stuck with the models as they're out there, to your point, there's a variety of purpose-built tools and non-GenAI tools out there, whether they be just rules, rule-based tools or APIs or machine learning models or statistical models or whatever, that can do a variety

• 12:15Daniel Whitenack?

  of the really important tasks that we want to do. And the GenAI models that we have now could serve as a way to orchestrate between those tasks and to, you know, create some really appealing workflows and automations and flexible interfaces and all of this stuff. So where my mind was going with that is, I'm not that concerned if it takes a while for GPT-5 or the, in my opinion, when I think about the rest of the time, I'll have to be a developer and AI practitioner. For the rest of my career, I could keep myself busy, no problem with all of the things I have access to, and create some really interesting

• 13:00Daniel Whitenack?

  products and tools and features and integrations and et cetera.

• 13:04Chris Benson?

  I think that's a great insight right there is the fact if you look at all of the different jobs and workflows people have out there in their careers and you know, if you think of all these tools that we currently have today, like without having to go forward, I would argue that it would be a very practical next step, you know, for practical AI to be able to start assessing your workflows, assessing where these different tools and the models we have can make a difference and doing some process reengineering to figure out how you can do that. And I think that the vast majority of organizations out there have not done that sufficiently. They might have done that with a workflow or two, but they haven't gone through, you know, if they're especially large corporations

• 13:48Chris Benson?

  have thousands and thousands of them. There are so many places where productivity can be enhanced by finding the places where people are struggling through their own workflows and where those align well with the capabilities in in these models, that might be a place they want to invest a little bit and get a great long-term benefit out of it. But the next model coming, whatever model we're talking about, whichever line of models, family of models, always gets the attention rather than the kind of grunge work of going through your processes and finding where you can save a whole bunch of effort, a whole bunch of time in a matter of of moments, you know, to increase productivity.

• 14:26Daniel Whitenack?

  Yeah, I think that there will be just a dry parallel, maybe. I know that people have drawn comparisons between this wave of AI technology and the onset of the internet and the web, that sort of thing. I think you could see that the basic components or many and many of the most impactful web-based technologies that have shaped culture and shaped our lives, the building block components of those were around from the very early days of the web. And there are sort of generational jumps, right? Like the, you know, the advent of streaming and all of what we now consume via streaming would not have been possible

• 15:11Daniel Whitenack?

  over certain types of internet connections and technology, right? So there's certainly generational shifts.

• 15:18Daniel Whitenack?

  But the the building blocks were enough and and probably some of the people in those early days, working with those building blocks, could not have imagined the transformative and kind of culture defining effects of those basic building blocks. And so I think we're in a similar scenario where the the building blocks of what we have with AI, whether that be Gen AI or non Gen AI, are enough to, I I don't think it would be too far to say transform certain elements of our culture. Sounds sort of grandiose, but I think that's sort of what's coming. But there will likely be the kind of generational jumps, whether that be GPT-5 or another language family or whatever. There will likely be

• 16:02Daniel Whitenack?

  generational jumps that we also don't anticipate yet. But the tooling that we already have, the building blocks we already have are enough to create transformative technologies and products and systems.

• 16:15Chris Benson?

  I would agree. I think maybe there's another show where we talk about what we think the transformation of of society and culture is in the future. I don't think that's the show right now. But I would agree with you. With what we have today, we can go a long way. And to your point, I was in college when the web came into being. So I do remember exactly, you know, those very early building blocks and trying to imagine. And this is not the same world that we live in today that it was then.

• 16:56Speaker 01

  Okay, friends, I'm here with a new friend of ours over at Timescale Aftar, Suithan. So Aftar, help me understand, what exactly is Timescale?

• 17:05Speaker 05

  So Timescale is a Postgres company. We build tools in the cloud and in the open source ecosystem that allow developers to do more with Postgres. So using it for things like time series analytics and more recently, AI applications like RAG and such and agents.

• 17:19Speaker 01

  Okay, if our listeners were trying to get started with Postgres, TimeScale, AI application development, what would you tell them? What's a good roadmap?

• 17:28Speaker 05

  If you're a developer out there, you're either getting tasked with building an AI application, or you're interested and you're seeing all the innovation going on in the space and want to get involved yourself. And the good news is that any developer today can become an AI engineer using tools that they already know and love. And so the work that we've been doing at TimeScale with the PGAI project is allowing developers to build AI applications with the tools and with the database that they already know, and that being Postgres. What this means is that you can actually level up your career, you can build new interesting projects, you can add more skills without learning a whole new set of technologies. And the best part is, it's all open source, both PGAI and PGVectorScale are open source.

• 18:11Speaker 05

  You can go and spin it up on your local machine via Docker, follow one of the tutorials on the TimeScale blog, build these cutting-edge applications like RAG and search without having to learn 10 different new technologies and just using Postgres and the SQL query language that you probably already know and are familiar with. So yeah, that's it. Get started today. It's a PGAI project, and just go to any of the TimeScale GitHub repos, either the PGAI one or the PGVectorScale one, and follow one of the tutorials to get started with becoming an AI engineer, just using Postgres.

• 18:43Speaker 01

  Okay, just use Postgres, and just use Postgres to get started with AI development, build RAG, search, AI agents, and it's all open source. Go to timescale.com/ai, play with PGAI, play with PGVectorScale, all locally on your desktop. It's open source. Once again, timescale.com/ai.

• 19:34Daniel Whitenack?

  I've been usually teaching workshops again. I'll be at QCon SF next week. So those of you that are around QCon, I'll look forward to seeing you. It looks like a good event. But some of what's come up, I think, in workshops for me recently is how to think kind of about your AI workflows going from prototype to some level of production. And I think these are things that we've talked about on the show before, prior to Gen AI, in terms of how you want to be testing and monitoring and thinking about deployment

• 20:17Daniel Whitenack?

  of AI-based workflows. But I'm guessing there's a lot of people maybe joining the show from different backgrounds after this Gen AI phase. And I connect this to the sentiment that people often with this technology are able to get to a point really quickly where they see an amazing workflow kind of come to shape, right? And it works amazing in like some of the time, like half the time. And then the other half of the time, it fails miserably, right? But they get sort of the taste of the goodness and they maybe don't know how

• 20:56Daniel Whitenack?

  to get the rest of the way. And I thought of an interesting parallel because some people are using these kind of low-code, no-code AI workflow builder tools, right? Whether that be something like Flowwise or Gumloop, Diffy, which is like, you know, these little interfaces, you can string things together, or it's like built into tools like Alterix or something like that, that's maybe a little bit more enterprise-y focused. But they build out this workflow and it sort of does this thing. It works like half of the time and not the other half of the time. And maybe they're using AI calls as part of that. And

• 21:36Daniel Whitenack?

  it struck me that this is sort of like, I don't know if you remember, Chris, back in the day, we had a phase of our AI Podcast life where we were really trying to convince people that they shouldn't run notebooks in production. Do you remember these days of data science?

• 21:57Chris Benson?

  Yeah, I do. It's that's a little ways back, but yes, I do indeed.

• 22:02Daniel Whitenack?

  So for those that aren't familiar, when I say notebook, I'm referring to like a Jupiter notebook. So this is a, an interactive web-based code editor. And if you imagine, maybe some of you that have used Mathematica in the past is similar, but you kind of go into the screen, there's a cell of, that you can put code in, you can execute that cell, you can take notes, you can execute another cell of code. And all of that state is saved. So you can like execute cell one and then go down to cell five and re-execute cell five and then go up to cell three and re-execute cell three and then go down to cell seven and re-execute cell seven.

• 22:43Daniel Whitenack?

  And what happens is, so if we just rewind our mind back to the olden days, I'm a data scientist, I'm building a model or creating a workflow and I'm doing this in a notebook. That sort of workflow generally is good for experimentation and produces really, really terrible code, just by its nature. So it's really good for experimentation, but if I'm hopping around all the time, I don't really understand what the state in the background is. I'm like hopping around between cells. I can give you the same notebook and you could never reproduce what I did. Even though it's the same exact code, you could never reproduce my exact sort of steps.

• 23:26Daniel Whitenack?

  And I just was struck by the fact that the way that people, it's almost like we forgot that that doesn't work that well. And now we're just doing it not in notebooks, we're doing it in these low-code, no-code tools or with agents that jump around between various tasks, right? And part of this is the same reason why notebooks are really terrible at producing good, reliable code is the, is the same reason, I think, why people are taking these AI workflows. From tools that they're using and aren't able to make them robust and reliable. So that's my hot take for the day. What's your thought, Chris?

• 24:10Chris Benson?

  No, I think that's great. First of all, I gotta say, boy, it's already making me feel aged again in a different way. The fact that it wasn't that long ago that we, it was all the hotness of Jupiter notebooks, you know, that we were talking about, and that was the cool thing.

• 24:25Daniel Whitenack?

  Yeah, and even like products managing all your notebooks and such.

• 24:29Chris Benson?

  Yeah, it feels like you just gave a eulogy, you know, for Jupiter notebooks to some degree. And so it's a reminder that things are changing constantly. So you bring a great point that, you know, we're taking some of the same challenges that we had in that environment and we're just recreating them in the newer tools that are out there. There was another company I worked at, I won't name the company, before I was at Lockheed. And at that company, I remember thinking there were people at the time that knew, they knew the AI modeling bit, and there were people that knew the software bit, but they never seemed to cross over. And when you raise the point about your kind of in-process development workflow, and then how do you actually get that to some level of production?

• 25:13Chris Benson?

  I think there's a lot of people out there that are gonna know that. And unless times have really changed in that area, and my gut says they probably haven't, people tend to focus on the thing that they want to do. What is the right development workflow? And how do you start getting to that production environment? I know you've gotten tons and tons of experience at that in recent years. How do you think about it? Can you frame it a little bit before, or do you dive into it?

• 25:37Daniel Whitenack?

  Yeah, well, I was thinking about it in light of this parallel to what we went through in the data science world with notebooks and these kind of ad hoc workflows that execute some other time and not other times, depending on how you execute them. And in reality, the answer to running that code in production is not the file, you know, download as Python script thing, because that just will never work because the state and the workflow is not preserved. How that actually gets productionized or would be productionized in the past is taking the logical steps that are being executed in that workflow.

• 26:22Daniel Whitenack?

  And taking those out of the notebook and embedding them in actual code. In this case, you know, Python code in functions or classes. And attaching tests to those functions or classes, just like a software engineer would do, because this is software engineering. And then figuring out, you know, again, doing the testing on the front end of that, whether that's a UI system, or that's an API, or whatever it is. To make sure that the behavior that you were you were testing in your notebook actually works. And that kind of sucks because it's a reimplementation, right? And to some degree, maybe you don't have to throw everything out, like you got something working in your notebook

• 27:07Daniel Whitenack?

  and you can bring it through and and have it work. But it does take actual work to go from that notebook state to the production code. And so I think if you just look at one of these tools and I do think some of these tools, the kind of low code, no code, assistant builders, workflow builders with AI stuff are useful. And can be useful in your maybe building out um nice workflows for your personal life, right? Like email assistant stuff for automations to, you know, turn news articles into podcasts or what whatever the like the thing is, right? That you want to do.

• 27:51Daniel Whitenack?

  But ultimately, these tools have their own opinionated way of tracing and testing and you debugging, right? Yeah, the same way that like debugging Jupiter Notebook. You have slightly different tooling, you have a slightly different workflow than if you were debugging regular code. And so I think part of the answer, unfortunately, and I guess this is my this is my hot take, if there is one, is I think that we're gonna see a similar dynamic.

• 28:21Daniel Whitenack?

  In the AI engineering world where like a business person, except I think the roles are different here. So similar way to like before, the data scientist would build a workflow in a Jupiter notebook. And maybe a software engineer would integrate that into actual code that's tested and has some form that resembles actual code. And the data scientist is probably interacting with the business person. In this case, it's slightly different role wise because the data scientist almost isn't there. But the business person might go into a tool like GumLube or Flowwise or Diffy or whatever

• 29:02Daniel Whitenack?

  and build out a tool that takes market analysis things and generates, I don't know, articles or summaries that go into some emails that are sent out to the company or whatever workflow that they created, right? And it has a series of steps and they're like, yeah, this like, this works. But it kind of does work, but it kind of doesn't work because they haven't thought about all the edge cases and it's hard to debug, it's hard to know when it's down. And so I think now it's like that business person bringing that workflow. If it really truly does need to be scaled across an organization or released as a product on its own, you just sort of have to take those steps out and actually put them in functions,

• 29:46Daniel Whitenack?

  put them in classes in your code that can be tested. And we can talk about like the methodology of testing here in a second. So I think that the low code, no code things are cool and awesome and have their place, just like notebooks have their place. And I still use primarily Google Colab, not Jupiter locally, but Google Colab notebooks. I still use notebooks. I just realized their limitations, right? Maybe sometimes better than other times, but I realize their limitations and then I eventually write software. So I think it's a similar thing with these tools that are up and coming is they're great

• 30:28Daniel Whitenack?

  and they allow for quick prototyping and business people to get their workflows and their ideas into a workflow that operates. But ultimately this has to become software. If the intention is to make it a feature that you release or something that scales across your organization or something.

• 30:46Chris Benson?

  Let me ask you, I'm just curious, when you're in Colab and you're doing that and you decide it's time to write software, how do you make your own transition? What do you do? Having done this for so long, what's your transition look like? Are you staying in Python? Are you converting some of that over into Go or something else? Or how do you think about it?

• 31:07Daniel Whitenack?

  I think if it's, it depends, of course, case by case, but generally, I would say if I'm doing it in Colab, it probably means that I'm doing something that requires Python. And so that's, you know, I don't know, doing something in Langchain or PyTorch or whatever the thing is. Then I think when I'm ready, I essentially have two ideas in my head of where that's going to live. Either it's going to live in a REST API, because you're going to have to make this functionality available to the rest of your software. So it's either going to live in a REST API, it's going to be integrated into some software you're already supporting, right? So that already has a code base.

• 31:52Daniel Whitenack?

  Or it's going to be run as a script, kind of an offline script at a certain, you know, K8 or something like that. And so if it's the API scenario, you know, I have a bunch of code that I've written over time with Fast API. I can just copy one of those projects, rip out the stuff that is irrelevant and put in the stuff that's relevant, kind of copying over from the notebook. If it's more of the native software integration, I think that really depends on, so if it's what kind of application it is, what the architecture is, that sort of thing. And so that might involve a change of language or a change of the type of infrastructure

• 32:35Daniel Whitenack?

  that you're using or the type of database you're connecting to or whatever that might be.

• 32:39Chris Benson?

  And that's an area that's I'm keenly interested in because, you know, historically, we have, we've been, you know, developing these models in Python and then deploying them in Python mainly because that's where the tools and stuff are still at. But there's a number of use cases, especially as we go forward and we're looking at autonomy and we're looking at robotics and things like that, where in many of those cases, Python is not the best language for the platform that you're deploying to. And so you have this incongruity between the development environment and a distinctly different production or deployment environment that you're trying to target. And so, you know, in my world, there are many things that might start in Python, but

• 33:23Chris Benson?

  probably should end in something like Rust, given what we're, you know, what we're trying to accomplish here. So I think that that still remains a very immature deployment, you know, kind of arena to work in. And I'm rather hoping that in the years to come, maybe we see more tools from tool providers and open source in that arena that can actually cross over from one language to another to make sure that it's always the right one for what you're dealing with.

• 34:09Speaker 01

  What's up, friends? I love my 8Sleep. Check them out, 8sleep.com. I've never slept better. And you know, I love biohacking. I love sleep science. And this is all about sleep science mixed with AI to keep you at your best while you sleep. This technology is pushing the boundaries of what's possible in our bedrooms. Let me tell you about 8Sleep and their cutting-edge Pod4Ultra. So what exactly is the Pod? Imagine a high-tech mattress cover that you can easily add to any bed. But this isn't just any cover. It's packed with sensors, heating and cooling elements, and it's all controlled by sophisticated AI algorithms.

• 34:51Speaker 01

  It's like having a sleep lab, a smart thermostat, and a personal sleep coach all rolled into one single device. And the Pod uses a network of sensors to track a wide array of biometrics while you sleep. It tracks sleep stages, heart rate variability, respiratory rate, temperature, and more. And the really cool part is this, it does all this without you having to wear any devices. The accuracy of this thing rivals- What you would get in a professional sleep lab. Now, let me tell you about my personal favorite thing. Autopilot recap. Every day, my eight sleep tells me what my autopilot did for me to help me sleep better at night. Here's what it said last night. Last night, autopilot made adjustments to boost your REM sleep by sixty two percent.

• 35:36Speaker 01

  Wow, sixty two percent. That means that it updated and changed my temperature to cool to warm and helped me fine tune exactly where I wanted to be with precision temperature control to get to that maximum REM sleep. And sleep is the most important function we do every single day. As you can probably tell, I'm a massive fan of my eight sleep, and I think you should get one. So go to eightsleep.com slash change log. And right now they have an awesome deal for Black Friday going from November 11th through December 14th. The discount code change log will get you up to $600 off the PUD4 Ultra when you bundle it. Again, the code to use is change log. And that's from November 11th through December 14th.

• 36:21Speaker 01

  Once again, that's eightsleep.com slash change log. I know you'll love it. I sleep on this thing every night and I absolutely love it. It's a game changer and it's going to change your game. Once again, eightsleep.com slash change log.

• 36:44Daniel Whitenack?

  Well, Chris, I kind of started talking about the testing and integration of some of these workflows and how I I see that playing out. From my experience in talking with people, there's some general confusion around how to. So let's assume that you're convinced that I want to rip out these various pieces of workflow that have maybe been prototyped in a low code, no code tool. And I want to put them into some software that's an API or a UI or a script, a data pipeline, whatever that is. Let's assume that you're convinced of that. Then the question comes, well, okay, now I have this function or I have this class in code

• 37:28Daniel Whitenack?

  that executes some sort of AI call, a call to an AI model. How do I test that? And what sort of considerations might need to be in place around that? And I, I often find that this sort of breaks people's mind. And this is also something that I, I think we dealt with for a long time in the data science world, which is, it's just very, I guess, overall, it's very interesting to me that these same types of things are popping up, but with a new audience. So I don't know if you remember back in the days of, of data science, when there were data scientists, they would create a model and that model has a certain level of performance, right?

• 38:13Daniel Whitenack?

  Like a 90% accuracy or something, right? So it's gonna be wrong some of the time. So you put that model into, maybe it's a fraud detection model, fraud or not fraud, right? You put that model into, into production, you integrate it into a software function. And now the question comes, well, how do you test that model? Cause it's not always going to give the same response and it's not always gonna be right. I don't know if you remember these discussions happening a lot in, in the data science world.

• 38:46Chris Benson?

  Yeah, I think you just wrote a eulogy for data science as well.

• 38:51Daniel Whitenack?

  Just.

• 38:52Chris Benson?

  Oh my goodness.

• 38:54Daniel Whitenack?

  But uh, this one always really intrigued me because my background is in physics. And so if we said, oh, this model is not deterministic, right? So we can't test it. If we took that approach in physics, we basically wouldn't have any of the technology that we have today cause it's all based on quantum mechanics and everything is a probability distribution.

• 39:19Daniel Whitenack?

  So there is a way to test things that behave non-deterministically like AI models. And maybe that people just sort of need a bit of a reminder about that. And I often kind of break this down into a few categories. But yeah, I don't know. Do you come across people with this sort of mindset, especially in integrating kind of LLMs or something, something like this?

• 39:46Chris Benson?

  All the time. I think there's a lot of people out there. I think everyone's still kind of figuring that out, quite honestly. If they're not in the business that you're in where you're dealing with that, you know, constantly, I think that's one of the big unknowns in, in, you know, with folks in general is how do I go about testing this? I want to get in the workflow. I don't think people know.

• 40:06Daniel Whitenack?

  At least at a, you know, you can only do so much in a 45 minute podcast. But at least to sketch out maybe a good framework for people to think about is number one, I think you should have tests in your code for each step of the process. Right? So if you're doing, if you have an LLM based workflow and the first step of that is translating something into Spanish and then the next step is summarizing it in one sentence and the next step is embedding that one sentence in a template. And then the next thing is, you know, generating an image for whatever kind of string of things you have going on.

• 40:46Daniel Whitenack?

  You should have tests for each of those kind of subtasks in the chain of processing, which partially also gets to why testing agents is hard, which is, I think, an interesting thing to maybe come circle back to at the end.

• 41:01Chris Benson?

  I think that's just good software, you know, engineering, what you're describing. And, you know, if you, if you took it out of the AI world and talked about software functions, you want to, you, each one is a discrete function that does something and you want to test it on its own, even though they're all connected together to do something. I think that's just really sensible.

• 41:19Daniel Whitenack?

  Yeah. And the, you know, the agent stuff makes this maybe a little bit more difficult, which we can come back to. But let's, let's assume that you have a workflow that you just want to execute over and over again, which is, is probably. Most enterprise use cases. So you split that up into subtasks, right? You have subtasks that you can, that you can test. The next thing I would recommend is to have people think about creating a set of tests in three categories. And this comes kind of from the ideas of behavioral testing. And this is, like, just take the fraud detection piece for a second. So you're, you're asking fraud or not fraud, right?

• 42:04Daniel Whitenack?

  So the first category of tests you want to think about is minimum functionality tests, which would be, this, you know, is the most fraudulent thing I can think of. It should always, like, be most fraudulent, Russian characters, uh, you know, Nigerian prince, whatever, you know, pick, take your pick, right? Should always be labeled fraud, right? A hundred percent of the time, that is a minimum functionality. These are not the most in-depth of tests, but they should pass one hundred percent of time, no matter what you do to the model, no matter what you do to your system. These are like a hundred percent pass. And you can do the same thing with L L M's.

• 42:45Daniel Whitenack?

  You know, even though it's not a classifier, you can say, I'm creating a, a bot that asks, you know, it gives all the information about Prediction Guard. If I ask, who is the CEO of Prediction Guard? They should always return the same name, right? That's a minimum functionality. That's a pretty easy question. It should be embedded in the knowledge. You know, these are things that should one hundred percent be returned and that you could test for deterministically, right? Like, does that name appear in the response? That sort of thing. The second category would be, in fancy terms, might be called invariant perturbations, but basically, in non-fancy terms, changes in the input that don't produce a change in the output.

• 43:29Daniel Whitenack?

  So the classic example of this is, if I ask an L L M maybe to do a sentiment analysis of a statement. And the statement is, I love the United States. It is so amazing. It is so great. I get positive sentiment return. If I change the United States to Turkey, I say Turkey is so great. It is amazing. It is wonderful. Right? In theory, regardless of what you think about, what you personally think about the United States or Turkey, that should always return positive sentiment, right? That is invariant change in the input. So you can make changes in the formatting, you can make changes in the ordering of things, and all of these should produce invariant changes.

• 44:13Daniel Whitenack?

  And then, of course, the final one would be the necessarily variant changes, meaning a change in the input should definitely produce a change in the output, right? Like if I change, I love the United States to I do not love the United States, I should actually have a change in the output, and that's a very easy thing to see, right? And so what you do is you create a table of minimum functionality tests, a table of invariant tests, a table of variant tests. And if you have that, those full tables, you can basically probe the behavior of the model and the sensitivity of your model to changes.

• 44:55Daniel Whitenack?

  And the sensitivity is really the thing that people get hung up on with these workflows. They don't realize how sensitive the models are to small changes in the input. And so this allows you to gauge the sensitivity of your system to a real number from passing these tests, and then work systematically to improve that.

• 45:15Chris Benson?

  Interesting. So I'm trying to just kind of put all that together for my own learning purpose, and I'm trying to think how, how we can apply that to workflow directly. Like, how, how do you, how do you actually fit that in to the nuts and bolts of moving it into production? Where do you do that in your workflow?

• 45:31Daniel Whitenack?

  Yeah, so I would take kind of the steps, I, let's say I have a workflow with five steps. I take each of those steps and I produce, you know, five functions or five classes, or however that fits into your code, right? And for function one corresponding to step one, I create a table of each of those tests, with table meaning just input equals output, the same as you're testing an API. Like, if I get this input into my API, I should definitely return this, right? And so you create that table and a set of unit tests or whatever testing framework you use to go over each one of those examples in your table and check the output to make

• 46:13Daniel Whitenack?

  sure it corresponds with what you expect to get either a passing or a not passing score. So the steps would be I have my five steps of my workflow. I split each of those up into a function or class or whatever the relevant programming object is. And then I develop these sorts of tests for each of those functions or classes. Now, the one question that might come up here is, well, should my model always pass? Right. Or should that function always pass all of those tests? And what I tell people is, well, it should always pass 100% of the minimum functionality tests. Because you've defined those from the start as minimum functionality. So if you're not having minimum functionality, then your software shouldn't be released. Right?

• 46:57Daniel Whitenack?

  And then the other ones, basically, I would say, you should never be regressing in those, right? They give you a sense of what the sensitivity of your model is in those variations. And you would not want to regress. You would want to systematically make those better. So some people might treat those as a certain percentage or a threshold that needs to be above that or, or something like that.

• 47:22Chris Benson?

  Gotcha. That takes us back to your point earlier. It just sounds like data science, good data science right there.

• 47:29Daniel Whitenack?

  Yeah, yeah. Well, it's, it's interesting because the roles have shifted, right? I think we went through those phases of data science being very Wild West to all the way up to like good engineering practices and testing and all of that. And we kind of now throw out the data scientist in the middle and we have business people developing these workflows and trying to integrate them into software. And yeah, there's a lot of reminders and learning, I think, that needs to be done. And that connects, you know, to where we started out the day, which is agents and production. Agents are hard to test because you don't know the workflow upfront, right? An agent determines-

• 48:11Daniel Whitenack?

  What steps it's going to accomplish on the fly. And so if you don't know that workflow upfront, then uh then there's some interesting things that you might need to do to test those, but maybe we'll save that for another episode. And or people could join the great learning opportunity that is the agents and production event from the ML Ops community.

• 48:33Chris Benson?

  Oh, absolutely. Yeah, you know what? You started the show asking what agents I had now, and I had to, I had to say, no, I didn't have any going. You got me thinking. And now that we're talking about workflow and testing and getting those agents working, we're gonna have to come back to this topic. I'm gonna have to bring something, though, to discuss.

• 48:51Daniel Whitenack?

  Yeah, we'll, we'll come up with some agent ideas and maybe, maybe work through the testing of those. I like that.

• 48:57Chris Benson?

  Sounds good. Okay. Thanks a lot for the insights today.

• 49:00Daniel Whitenack?

  Yeah, thanks, Chris. Hope, uh, hope you have a good rest of the day and we'll swap places geographically next week.

• 49:08Chris Benson?

  There you go.

• 49:08Daniel Whitenack?

  Our way towards Tofurkey.

• 49:10Chris Benson?

  Perfect. Sounds like a November.

• 49:20Speaker 00

  All right, that is our show for this week. If you haven't checked out our change log newsletter, head to changelog.com/news. There you'll find 29 reasons. Yes, 29 reasons why you should subscribe. I'll tell you reason number 17, you might actually start looking forward to Mondays.

• 49:40Speaker 06

  Sounds like somebody's got a case of the Mondays.

• 49:43Speaker 00

  28 more reasons are waiting for you at changelog.com/news. Thanks again to our partners at fly.io, to Breakmaster Cylinder for the Beats, and to you for listening. That is all for now, but we'll talk to you again next time.