Abundant Intelligence at the Midwest AI Summit

In this special episode, we bring you highlights from three talks at the Midwest AI Summit. Heidi Porter explores how to build reliable AI-augmented workflows using the right tools, evaluations, and guardrails. Kevin Solorio shares why building AI with frontline workers, not just for them, is key to earning trust and driving adoption, emphasizing collaboration and continuous feedback. And Chris Shinkle and Andrew Dunlap of SEP discuss how software teams can find the right level of AI autonomy based on their goals, constraints, and verification needs, sharing lessons from teams using AI to build software more effectively. Featuring: Heidi Porter – LinkedIn https://www.linkedin.com/in/heidi-porter-68a79549/Kevin Solorio – LinkedIn https://www.linkedin.com/in/kmsolorio/Chris Shinkle – LinkedIn https://www.linkedin.com/in/chrisshinkle/?isSelfProfile=falseAndrew Dunlap – LinkedIn https://www.linkedin.com/in/andrew-dunlap-sep/?isSelfProfile=falseDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: Moyer Group AI https://moyergroup.ai/SEP https://sep.com/Focused https://focused.io/Sponsors: Hostinger https://www.hostinger.com/in: Create your website today. Go to hostinger.com/practical and use practical for 10 percent off now. Resources and Events: Register for upcoming webinars here https://practicalai.fm/webinars! Prior Webinars from our partner Prediction Guard https://www.youtube.com/playlist?list=PLRpRdFmBfdF6boDgAD3nNaLm8Yx4KN2VtMidwest AI Summit 2026 https://midwestaisummit.com/Chris’s new book The Art of Swarming will be published soon, written to be the seminal foundational book for the fledgling swarming industry, defining the principles and architecture that enable autonomous agentic UxS swarms with emergent unified embodied intelligence.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:41Daniel Whitenack?

  I had the privilege of being the emcee at this year's Second Annual Midwest AI Summit, and it was amazing to see everyone there at the summit just engage in a really useful, practical way around subjects like AI agents and decision models and AI security. And I arranged with the conference organizers for us to be able to share a few snippets from speakers at the Midwest AI Summit. So today's episode is going to be a very special one where you'll get to hear a selection of a few snippets from speakers at the Midwest AI Summit. We'll have Heidi Porter, who is a principal at Moyer Group AI, and then Chris Schinkel,

• 1:24Daniel Whitenack?

  Andrew Dunlap, who are with SCP, and then Kevin Solorio, who is a staff software engineer at Focus. I hope you enjoy the content. I know I certainly did. And yeah, would love to see you at next year's Midwest AI Summit. We'll be sure to update our audience here when the details of that are shared from the conference organizers. So enjoy the selection of a few talks from the Midwest AI Summit. We have with us Heidi Porter. She's a technology professional with decades of experience in networking, data analytics, machine learning, AI, and on-premise cloud deployments at large companies. Her company, Moyer Group AI,

• 2:09Daniel Whitenack?

  provides education, strategy, and consulting on AI tools and where it belongs in your tasks and workflows. And it's those workflows that she's going to talk about how to make durable AI augmented workflows. Help me welcome Heidi Porter.

• 2:26Speaker 02

  All right, thank you. I have a clicker here. Good morning. I'm Heidi Porter. Quick show of hands. Who has had an AI pilot that did not make it past the demo stage or stalled at the start of production? So there's a good amount, and congratulations to all the rest of you who just, you know, hit it on the first time, right? So a lot of times, these workflows fail because it's not because the model was the problem. It was because the LLM was asked to do too much. And Don and Jeff just gave great talks about when to say no to AI and when to say yes to AI. And so sometimes people ask it to hold the data, be the system of record. It's not a database.

• 3:10Speaker 02

  Make the decision and check its own work. The question I want you to come away with today is to look at the task in the workflow and decide, what is the right mechanism for the task, okay? And then how to evaluate right answers and how to catch it when it's wrong. So evaluation, guardrails, et cetera. But before we begin to look at workflows, we need to say a word about data. So the data that goes into your workflow, I think everybody knows this a bit, but we have to talk about it. It sets the ceiling on how reliable your tasks and workflows can be. And the question isn't whether your data is good. It's whether, you know, good in general. It's whether it's good enough for the task you're doing.

• 3:54Speaker 02

  So we check four things, you know? Is it accurate, complete? There's a whole list here. Is it consistent? And as Jeff talked about, maybe your agent can help with some of this. But, you know, if the data's stale, it'll throw an error and just quietly give you the wrong answers. I don't know if an agent can help with that. So when there's an LLM in the loop, you need to ground it in your data at runtime. And we talk about systems of record. So you're grounding your LLM in your data. You're fetching the facts. You're using a query to query a database or some kind of knowledge graph. You're letting the evidence drive the answer, and you have it cited sources so people can check if they need to, right? Grounding is only as good as the data behind it.

• 4:38Speaker 02

  So fix the data later first because everything downstream inherits its quality, right? So now let's get into what a working system needs behind good, clean, integrated data. So four words we'll use for the rest of the talk. Mechanism, that's what we're going to call what actually performs a step, whether it's rules, classical ML, a decision model like Jev, and I'll talk about what that is, or an LLM. And an LLM has a large range of, you know, directly querying, you know, using an LLM to generate, using it to call tools, using it in an agentic architecture like Jeff talked about, or using it with human review. And you pick that step per task, not per project. And then there's three safeguards

• 5:23Speaker 02

  that surround it, right? And so you have guardrails that are runtime controls on each output. They can catch problems escalated to a person, constrain answers when something, you know, and then recover when something gets through anyways. And then evals are repeatable tests. Eval's are really important, and they're not talked about as much as they should be because they are determined whether the answer you're getting is right, whether you get a good outcome. And one of the things I want to call out is when you're looking at a workflow, don't just look at the averages because averages can hide a bad result. And you need to know how bad your bad is so you can fix it. And then observability, we won't have too much time to get into today.

• 6:08Speaker 02

  But Jeff talked about this a bit, being able to recover and replay and catch drift before an incident. So build each step right and then safeguard every one. And what we're going to do today is show some examples of workflows across manufacturing, agriculture, and healthcare that actually worked and the lessons they learned. But first, let's look at building them. So for every task, there's a lot on this slide, I know. There are really three decisions. So the first one is what performs it. And we just talked about that here. And we talked about the fact that the LLM mechanism has a wide range of how you can implement it. Second is how the work is coordinated. So if you know the steps,

• 6:53Speaker 02

  you probably just want to use a fixed workflow with programming, right? Or you can use an LLM or decision model to pick a branch at a decision point. Or you can have an agentic architecture where the agent works and is constrained and bounded by your cost or time, et cetera. And we'll show one example of that. And then there's third, this is where we start to get into guardrails. So what's allowed to execute? And think of it as in a dial, right? So if it's consequential on certain actions, a person approves it. If it's high volume with mixed confidence, let the system act above a confidence threshold and escalate the rest, or vice versa. Time-sensitive work, act now and give a person

• 7:38Speaker 02

  a window to veto or undo it. And automate outright only what it's cheap to undo. So, you know, using like an agentic architecture. And whatever you pick first for these different questions, test it, treat it as a hypothesis. And on the output of each of these tasks, you need to think about the guardrails, right? That's what we said was very important. To make sure that you get the outcomes you're looking for, you need guardrails around every mechanism. They raise the odds of a good result, and you won't get a perfect one. So you need to measure quality against your own baselines, okay? And the first thing to really realize, and it's been talked about a bit, is that a rule written in a prompt

• 8:20Speaker 02

  is not deterministically enforced. A guardrail is actual machinery, a gate, a check, a human, a limit in code. And on this slide, you can see that the guardrails scale with the mechanism. And I want to call out a couple of these. One is, you know, you think, well, programming, we don't need guardrails. Well, you should do. Traditional programming has always needed guardrails, you know, not to push a bug into the code, you know, to have some change control. And then another one is we think, well, you know, if we have a human review the LLMs, and Jeff talked about this, how there'll be thousands of things to review, the human can get tired, right? They can just have rubber stamping fatigue,

• 9:04Speaker 02

  and, you know, that's not a good guardrail. So you have to be really careful about these kind of things. And we've listed some, you know, possible mitigations, et cetera. And then the column I really want to call out is recover because something will get through eventually. And recover is how you contain it and get back to a safe state. Roll back, retract the output, undo the write, et cetera. And so you should make actions reversible where you can and where you can't require approval before they run and be careful of that human fatigue. And then the valves sit over all of this. Like I said, this often isn't talked about enough. They tell you whether it's working and they gate everything towards more autonomy.

• 9:48Speaker 02

  So if the valves aren't working, you don't want to promote to production because you will soon not have a working system. So I'm going to show you some examples in real deployments that are kind of fun. So this is a published, peer reviewed study. So it's not a company, you know, it's not from a company. It's not marketed. It is a peer reviewed study of a rubber hose production line. And so a camera watches the line and a modified, you only look once vision model finds the defects. And if you take a look at what they're looking at, you can appreciate the difficulty of this task, especially at speed. So it detects the defects and there's no LLM in this. And then a detected defect triggers the diagnosis.

• 10:32Speaker 02

  The system then retrieves evidence from a knowledge graph and it searches relevant SOPs and maintenance history. The LLM then just gets the evidence and from the systems of record, right? Not the whole knowledge base and it drafts a candidate explanation and corrective actions. But here's the thing, now we have this additional guardrails, right? Where they use rule based checks, traditional programming to check whether the draft stayed within the evidence that it pulled. So it's checking the LLM. And if it looks thin, if the evidence doesn't have enough support, it sends it to a human. So there's two different guardrails on this. The other thing is about this example is that operators can

• 11:15Speaker 02

  accept override and comment on any of these, not just the ones that are escalated. So they have that control. So what did we learn? You know, notice what they didn't do. They didn't ask the LLM to look at the hose and figure everything out. The LLM explains and there's additional checks and guardrails. And the results on this are good, but they're not perfect. So, um, you know, grounding still needs checking. And what they did as far as observability was they monitored this for 52 weeks to make sure it was working right. So let's look at an ag example, and I like this one a lot. So I think this, actually, I'm going to add this little note that I think this use case could be useful across a wide variety. So think about it for some of your situations.

• 12:00Speaker 02

  So this is a seed company called Syngenta that has a big seed catalog. And sometimes farmers select the wrong seed for their field. So the salespeople or the grower bring this solution, Cropwise AI onsite, and they do onsite seed selection based on local conditions. So the person asks a question in plain language and then LLM interprets it. Good use of LLM. Before anything else, the user is authorized by the identity service in AWS, an application code, not in the LLM. Then the Bedrock agent routes the request to knowledge bases, APIs, Python actions, data agents, and it pulls weather, soil, product and agronomic data. And when the question needs a prediction, Syngenta's existing

• 12:45Speaker 02

  seed and agronomy models produce it. The LLM does not do agronomy. Okay. Its second job is explain the result in plain language and the rep or grower makes the decision. So this agent, you know, is pulling from systems of record, getting the agents it needs and it's sending out a subage. And we'll see a diagram in a second. So if you look at this, you can see blue is where the LLM or agent works, gray is rules and/or ML and yellow are the guardrails. So in this case, the LLM is the interface, the router and the explainer, not the agronomist. And this is not as nice a picture as Jeff had of, of the knowledge system, but you can see that, um, you know, there's

• 13:30Speaker 02

  several knowledge bases, lots of documents is pulled in. And the reason I wanted to show this kind of diagram here is because of one of the big lessons learned in this. We talked about the first two already here. But the third is that, uh, the way they did not design that system, it doesn't just survive a model change, you know, swapping out a model. You can also swap out the knowledge bases, um, or some of the other, uh, data sources. And so that makes that really, you know, you have to reevaluate afterwards. And, you know, if you look at the evaluation note here, they had a hundred question and answer pairs from actual sales rep, scored it by human reviewers, using LLM as judge. So they really, you know, did all these valves to make sure

• 14:15Speaker 02

  the workflow was successful. But it is a useful thing to know that you can swap out these different pieces, um, pretty easily. So that's our takeaway here is design so the model and knowledge bases are the easiest part of the system to replace. Now, the next one I also really love is from healthcare. It's Stanford healthcare built a tool called Surgical Co-Management Navigator. And it screens next day elective surgeries to see which patients need surgical co-management. So surgical co-management, I had to look it up, is when clinicians share the practice of patient care responsibilities during pre-op, inter-op and post-op. And before launch, routine human triage missed half of eligible

• 15:00Speaker 02

  patients. And so, you know, if I'm having surgery and I'm eligible for co-management, I, I think everyone in here would want that oversight there. So this system is really a big deal for them, for the patients especially. So the new system pulls the pre-op anesthesia notes and the medication list from the electronic health record. And it, the criteria it uses are a rubric that the institution owns, written from their own literature, their own faculty expertise and fixed in the prompt. Okay. And we talked about how the prompt is, you know, readable, not enforcement. But in this case, the LLM's only job is to apply the rubric and say yes, maybe or no. So can't hallucinate anything.

• 15:44Speaker 02

  Yes, maybe or no, that's the answers. Again, just like rubber hose, there, every case shows up in the EHR list, flagged or not. And the physician can, you know, look at all of them and confirm decline or override. The clinician keeps the decision here. So before launch, the other thing they did was they benchmarked five different LLMs against consensus chart review and picked O3 Mini because it performed competitively at a lower cost. The model is again, the replaceable part, not the center of the system. And what's really interesting about this case I like is in the analysis of false negative cases, this is a small sample, but there were a minimal amount of LLM errors. What happened is they found a lot of other errors in their

• 16:27Speaker 02

  process and then they were able to work to fix some of those other errors.

• 16:33Speaker 02

  And of course, look at the LLM errors as well. So three lessons from Stanford and it's all about who owns the protocol. So the institution owns the criteria. They compared candidate models. And then third, they checked, this is the one I want to call out, is they checked why it disagreed. Right? And so what they typically did was overflag, you know, um, you know, be more false positives. And when they looked at the no's, the physicians agreed 97% of the time. So it overflags by design and physicians declined about four in 10 flagged patients, basically saying it wasn't complex enough. So these are three running workflows and an overview of using

• 17:16Speaker 02

  the LLM for the right task or the orchestration piece. And so, you know, back to these four words we were talking about, the four parts we started with, we can map these to something you just saw in these examples. Right? So does each step use the simplest thing that works? Syngenta used the agronomy models, guardrails, the hose line escalated when evidence was weak, the system withholds the answer and escalates. And then third, the valves, you know, we just talked about Stanford benchmarked, uh, five models, Syngenta tested against a hundred real questions. And fourth, observability, we talked about the rubber hose, uh, line, they observed it for 52 weeks.

• 17:59Speaker 02

  So if you can answer yes to all four of these questions, you have a system that most likely keeps working after the demo. Thank you very much.

• 18:08Daniel Whitenack?

  Some, thank you so much, Heidi. That was, that was great. I love, I love how you digged in or dug into each of those, those workflows. Super practical.

• 18:20Daniel Whitenack?

  Whether you're an early stage startup or a Fortune 500, it's critical that you have an amazing website that looks handcrafted, but where you ship that website, updates to that website, landing pages, et cetera, in minutes, not days to keep up with SEO and AEO and to make sure that your content is fresh. Framer is the AI website builder that helps creators, teams and businesses ship production ready sites faster than ever while getting every detail right. Prompt, inspect, edit and publish in one place at a whole new pace. Learn how you can get more out of your site from a Framer

• 19:04Daniel Whitenack?

  specialist or get started building for free today at framer.com/practicalai for 30% off a Framer Pro annual plan. That's framer.com/practicalai. Rules and restrictions apply.

• 19:23Daniel Whitenack?

  I'm really excited for our next speaker here, Kevin Solario. He's a staff software engineer at Focus. They're one of our sponsors out here. I got to meet Kevin, yeah, probably a couple of years ago.

• 19:40Daniel Whitenack?

  He was here last year at the Midwest AI Summit. Their team at Focus is an amazing just set of AI engineers that they build agents all day, every day. They engage with a variety of types of companies and actually get agents into production, evaluate those agents as we learned about this morning. Kevin is a self-taught engineer and AI consultant. Again, building custom agents mainly using the Langchain stack. Today, he's going to be sharing with us how to build AI with workers, not for them, which is part of that earning trust on the frontline. So please join me in welcoming Kevin.

• 20:22Speaker 03

  My name is Kevin Solorio. Most of this talk I am going to be speaking from is coming from one of my most recent clients at Focused. We started a new engagement and the CEO immediately told us, adoption is key. No matter what we build, adoption has to be the number one thing that happens. And we're building an app for some blue collar workers, which he upfront said are notoriously slow to adopt new technology. They don't trust it, they don't like it, they don't see it, they don't want it. So in order to accomplish that, we wanted to take a few minutes and say, okay, we need to get this group of users to use a tool that they may or may not want to use.

• 21:05Speaker 03

  So let's dive in to discover why some of these tools may have failed in the past. And so surprisingly, some of the first things that come up is it doesn't work. So when you talk to someone and they start using a tool and it doesn't meet their needs, they don't like it, they don't use it. But it doesn't work is very vague. So diving into that with them, you start uncovering, well, it's slow, it crashes. It's, it might not meet their needs. It might do the thing that it's supposed to do, but that doesn't mean it's doing the thing that they need it to. So sometimes there's just a, just lack of connectivity there. The other thing that might happen is that the tool's not really helping them.

• 21:49Speaker 03

  Sometimes technology is adopted from several layers up in the management stream and it's there to solve a different problem or to track a thing that they have going on. And it has nothing to do with their day-to-day job. And if anything, it just adds more work for them to actually go into the system and update it with all the information they have to. The other thing is cumbersome. So if the tool is cumbersome to use, they're not going to, they don't want to. If they have to go into three or four different systems and extract information and then come back to this system and put it in there and then do that whole process, it's just not going to be something that catches on. And so that's one of the other things that happens with tools, specifically when they are built for someone.

• 22:35Speaker 03

  So all of these things are symptoms when you have tools that are being built on behalf of someone else. So it didn't work means typically we didn't research the requirements enough. We didn't dive into the use cases. We didn't understand exactly what we needed to build in the first place. It was not built for them. If it's for someone else's, what is it, KPIs, OKRs, any of those other metrics? Like if it's not really there for them, it's just in their way, they'll use it because they have to, but they might do it like at the end of the day. Oh, I'll just go log everything at the end of the day, which means they've forgotten half of what's important to log. So it's not getting into your system anyway. And then the last one is the cumbersome.

• 23:19Speaker 03

  It's likely because it was designed by someone that wasn't doing the job. They didn't understand how much of a pain it would be to go to four different systems and then go into yet another one. So if we stop building for someone and then we actually start, well, sorry, going back to that a little bit more. The big thing with building for is it's always through a proxy. Most of the time, when you have a user that you're working with, like a product owner, they're interpreting the results, they're having conversations with users, but they're adding their interpretation of it, which may oversimplify things. It may take things out, it may miss very important use cases that they just gloss over.

• 24:02Speaker 03

  It's also, again, going back to when you build for someone, you're kind of sending a signal of, I know it's best for you. This is how this work should be done, which if you're anything like my partner, she does not like it. If I say, hey, do this thing, but if I ask her nicely, no problem. So it's one of those situations where you come in and you say, do this thing. And this is how you're going to do it. You're kind of coming back at saying, I know how to do your job better than you. So it just sends a weird signal with that. And then also this tends to codify the standard operating procedures that might just be like those in theory types of cases. So in theory, all of our requests fall into one of these nice little buckets and you never go outside of that, which is rarely what happens.

• 24:44Speaker 03

  Contrasting that with building with someone, so when you start building with someone, you're going to start with a goal. Instead of saying exactly what you're going to do, you're going to start by addressing what problem it is that you're really trying to solve. And by having them in the room and starting to share information across both the silos, you start to build respect and trust with each other. So I know that they know their job well. In this particular case, we were dealing with field technicians that were repairing equipment from meat grinders to ATMs, like literally the same person being dispatched to fix your meat grinder might be the same person dispatched to fix your ATM. So they have a wide breadth of knowledge. That is stuff I do not have.

• 25:27Speaker 03

  And when I build a system, especially using an AI agent that's going to generate something, it will generate for sure. It will give me an answer. Is it right or not? I have no idea. So I really need to lean on them to help correct me where I'm wrong. And this is where those in-practice things come in. Now, historically, when building with people in the day of web apps and pre-AI, so like three years ago, this would largely take the form of looking at paper prototypes. You'd get in the room with them, you might do some drawings on a whiteboard or some big markers and just start drawing things out. What we're really trying to define here is what information needs to be surfaced? At what time, at what fidelity? Are we getting the right information to them?

• 26:11Speaker 03

  Are we giving them escape patches? Is there a way for them to drill down into details that might not be there right now?

• 26:19Speaker 03

  Now, when you start going into agents, it's less about layout because a lot of us, if you've seen anything, it's a lot of chat boxes. So there's not a lot of layout to discuss. You're going to work with an agent. So how do you make sure that it's doing the right thing? And with the agent, your goal is the right behaviors. It's calling the right tools at the right time. It's using the right guardrails, it's respecting the boundaries there. So it's not about the design as much as far as like the layouts and things like that. It's really diving in to, is it doing the thing I want it to do? And evals really provide the perfect seam for that.

• 27:00Speaker 03

  So this might be a bit of a bold statement, but an agent without evals is a proof of concept. And this doesn't matter what infrastructure it's been deployed onto. If you do not have evaluations, you cannot confirm for sure that the agent is behaving as expected. And so in this case, I'll always say that's more of a proof of concept at this phase.

• 27:27Speaker 03

  And the nice thing is with evals, one of the nice, you can actually start testing your agent again and again and again. When I typically kick off a project like this, I will maybe synthesize a data set where I'll take what I know about it and then create a data set to just get something wired up and in their hands. I want someone to be able to look at it, touch it, tell me if it's working right or not, and give me that feedback.

• 27:55Speaker 03

  The interesting thing, working with blue collar workers, that they don't have traces as much. If you're working with white collar workers, you might be able to get by doing most of this without truly interacting with them, though I do highly recommend having them involved. But with blue collar workers, there's not that trail. There's not the list of emails, there's not the list of traces. You don't have everything that you would normally have if you were working with a white collar worker. And so to really start building for people, the first thing is to get everybody in the room, start talking about it. And going back to what I was saying earlier, that synthesized data set, something for them to look at. I want to be wrong here. One of the things I've found as a consultant is getting information from people can sometimes be very hard.

• 28:41Speaker 03

  But if you show them something that's wrong, they are very quick to tell you where it is incorrect. And so if you start with something knowing that, hey, this isn't going to be right, and nothing's right on the first pass, so you go ahead and start with an idea. This is what I have proposed that we work on here. Let it be wrong. It's okay. Get that feedback, get that flowing in. One of my early mentors used to tell me, marry the problem, not the solution. And that was because the problem is what's important. If you get too hung up on a solution, especially if it's one that you created, when someone comes up with a new idea that might be better or more efficient, you might feel a little hurt that your solution wasn't the one that was selected. So in this process, it's really important to realize that you're there to solve a problem

• 29:25Speaker 03

  and not just promote what it is that you came up with. So as you present that idea, be wrong, take the feedback and begin to iterate on it. And that's where that comes together. And I feel like this is where you really start to build that trust, is when they can see how wrong it was at first. And then through some level of interaction with them, they start giving you that feedback. You can then make it so that your agent is now performing the task that you're expecting. Or it's like starting to get there, the responses are making a lot more sense. With this, you are going to start to want to get that feedback from them regularly. So in this case, you might want to make it as easy as possible for them to give you that feedback. So if you have a data set that you want them to label, give them a Google spreadsheet,

• 30:09Speaker 03

  they can go and label it on their own time, get it ready. Or if you have the time, vibe code up an app that's fun, make it gamify it, do something, get them a way to get that feedback for you. And then you can use that in your evaluations. And then this is the part where You're going to iterate a lot with them, but then you might broaden the circle. So initially, you might start with a handful of representatives that are going to showcase their knowledge and help you get on the right track. But they might not see every case. So then you're going to widen that circle a little bit. So this is where you're going to want to start creating those data flywheels. If you've not used a data flywheel before, what that is, is you create an offline data set, and that's what you're going to use for your development practices.

• 30:53Speaker 03

  This is what you're going to use to develop your agent against. You're going to measure against that data set. And then when you deploy it, you're going to have online evaluators that are consistently going to look at a sample of traces and kind of score them. And that could be a code-based evaluators. It's very deterministic. Or it could be an LLM as judge that can then help you find and flag certain cases that were not covered by your agent before. So those that get added to an annotation queue. And in that queue, you can review and discover what went wrong on that particular case. You can then add that back to your offline data set. And now you have a constantly flowing data set that is evolving with the use cases.

• 31:34Speaker 03

  So you get that flywheel set up, and then you invite more people to the platform, and you start seeing how they're using it. And you rely on that data flywheel to capture those additional cases and keep building. And then when you're ready, you're confident in the product, you can go ahead and release it. The, the big thing here, keep the loop running. You're always going to iterate. You're going to find cases that you didn't anticipate through the initial discovery. You're going to find cases that you didn't anticipate through the initial release, through the first couple of rounds of beta users, through the widening of that. And so here is where it's really important to get that going. And then the people that participated in this, if you've ever heard of that IKEA effect,

• 32:19Speaker 03

  that is where you really add a lot of value to something that you helped build, specifically if it went well. You know, those that IKEA furniture, that's your masterpiece in your living room. You actually prescribe a lot of value to that thing because you got your hands dirty. You built it. You were part of it. So that's one of the things that can happen with this. And so then the people that you had helped build it become your champions. They're going to really be happy with what happened. They're going to be like, oh, yeah, see this tool, I helped build this. I contributed to it. I made this better.

• 32:51Speaker 03

  Now, one of the things with blue collar workers specifically is that they're craftspeople. They want to see that you respected them and that you also have a craft. So with this, dive in. These evaluation data sets are typically very curated. You're not going to have millions of them. You're going to have enough that represents the use cases and make sure that you can cover all the things that you expect the agent to do. You dove into the details. There's a lot of nuance specifically in blue collar work. It might be a thousand different things that come up, and you really need to dive in and understand what made this case different from another one. And taking the time to dive into their process and understand their flow, and then be able

• 33:35Speaker 03

  to come back and show them that the system understands it because you took that time to say like, hey, let me hear you out. Share with me what you've learned. And also show it improving. Touching base on a frequent basis where you're able to show like, hey, I heard you. I made those changes. Here's the feedback. Let's keep this going. That all ends up building a really nice, robust trust between the engineers and that team. And I've been fortunate enough, like, right now, we have a former designer. I would say former designer. He still does designing, but he's now one of our lead AI strategists. And he comes in and he really listens and helps just like helps look for these nuances and helps lead that process. And it's been really great seeing how, when you start working with people and really

• 34:20Speaker 03

  meeting where they are, that you can start building a system that everybody's really proud of and actually starts using. That is all I have, and I'm the last thing before you in lunch, so I will not keep you here for much longer.

• 34:34Daniel Whitenack?

  That was great. Thank you so much, Kevin. I love how focused this was on, on building for people. AI agents are everywhere now, and it's so easy to get access to one. However, if you're an enterprise and you're considering having this digital workforce of agents within your organization, there's a whole new level of things to consider, both around human identities and agent identities. Like, let's say Bob, an employee, is on boarded into my AI platform. How do I scope his access to various models, MCP servers, tools within those MCP servers?

• 35:19Daniel Whitenack?

  And then if he creates an agent, can he delegate access to that same scope of things? And what if he shares that agent with Alice, who has a greater level of scoping? Can she expand the scope of delegation to this agent? And what happens if Bob leaves the organization? Does that agent identity survive? These are a lot of the challenging things that are addressed if you have a powerful, sovereign, self-hosted control layer like Prediction Guard. I'm so proud of what my team that I'm leading at Prediction Guard has built and has deployed in production at various organizations that helps you manage your fleet of agents with

• 36:04Daniel Whitenack?

  complete confidence, knowing what they have access to, what they have accessed, how their runs have developed, how they've interacted with human identities, and how you have intervened through control mechanisms in their behavior and their various runs. I encourage you to check out what we're doing at predictionguard.com-practicalai. Go ahead and schedule a demo with myself and the team to figure out how you can scale your digital workforce of agents without losing control. That's predictionguard.com-practicalai. Predictionguard.com-practicalai.

• 36:49Daniel Whitenack?

  So this is, this is a talk from two folks from SEP, Chris and Andrew. If you're not familiar with SEP, they've been building projects with clients across aerospace, medical, healthcare, finance, and, and other industries. So welcome, Chris and Andrew from SEP.

• 37:08Speaker 04

  The clicker right here. Okay. All right, welcome. I'm sure a lot of you by now have seen the, the demos, the videos of people operating what they call a dark factory, right? Hands off the keyboard, agents writing all the code, agents reviewing the code, agents reviewing the agents, agents testing the code, agents shipping the code. And some of that's pretty impressive and, and makes sense in some situations. But I'm guessing somewhere, somebody in your organization has seen that demo and has asked you like, hey, when are we going to start working like that?

• 37:49Speaker 04

  Like, hey, when do we turn off the lights and kick up the dark factory, right? And so that's, that's what we're going to talk about today. And I know the picture looks like the lights are on in the factory. They're not. They're not. It's the glow of the crazy robots in there.

• 38:02Speaker 03

  There's no humans.

• 38:03Speaker 04

  Yeah. So my name is Chris Schinkel. I'm the Director of Innovation at SEP, and this is Andrew Dunlap. He's our, one of our senior AI engineers. And today, we really want to leave you with two things. One is sort of a sense-making framework, a mental model for how to make sense of how your teams should work with AI. We've, we've heard talks already about how people are incorporating AI into the product. We're talking a little bit about how do you equip your teams to better make use of AI to write software? So we want to give you a sense-making framework. And then the second thing we really want to do is give you a set of principles to help you get started. So if you're going into a team and it's like, hey, we, we want to use more AI, or we want to do something different, how do you get started? Where do you go?

• 38:48Speaker 04

  We're going to give you a set of principles to hopefully help with that. Now, if you've paid attention, there's lots of studies around AI. Here's three relatively recent ones. The 2025 door report is the oldest, but lots of studies about how people are using AI and making use of AI. And I think it's interesting that these three sort of have the same pattern, or what comes out of it is the same. And they all sort of essentially say, AI is not going to fix your team. It amplifies what's already there. That's really important for, for us. It's a question we have to ask when we start enabling teams with AI. I see, we build software for companies where being wrong matters.

• 39:32Speaker 04

  Looking around, a lot of regulated space. We work healthcare, pharma, aerospace. And so we think long and hard around what it looks like to enable a team with AI. And that's where the title comes from. We don't believe that every team needs a dark factory. We, we don't think you need to be all the way there to have some significant productivity gains from, from AI. And in fact, I would argue, if you, if you're trying to move there and you're operating in the wrong mode, even the best of teams are going to struggle, struggle. It's going to feel a little clunky. And so what, what does it look like to struggle? What does it look like that you're operating in the wrong mode?

• 40:12Speaker 04

  So you may hear something like this, we're, we're shipping AI slop and nobody knows what right looks like. Or we should be using more AI. Well, I don't know what more, you know, is, but like, we should be doing more. Or the joy's all gone. All I do all day is review code. And if you've heard one of those things in the last week or been in a meeting, heard something like that, like you, you are in the, the right session. When the way you work doesn't fit your situation, like more AI is not going to fix that. More AI won't fix sort of the AI fit.

• 40:54Speaker 04

  And Andrew is going to walk through three examples from three different customers of ours, three different teams that are all using AI in a slightly different way or, or enabled with AI in, in different ways. And we're going to get into a little bit like why, why that is. And we, I think all three are operating where they should be.

• 41:20Speaker 05

  Thanks, Chris. So let's talk about a client we have. They're a transmission operator. And so they have these simulations, and the simulations have these models that convert from one form to another. This is a legacy system. It's been used by the business for a long time. It gets the job done. But they want to improve it. And so they had a clever engineer. One day said, hey, I got these input models. I got these output models, and I got a whole bunch of agents. So I can just reverse engineer them and figure out the process in the middle. Makes sense. Two days later, lo and behold, he has a working prototype. So they want to get the prototype to primetime. Got to get to production. That's when we get the call. We bring in a team, they get started. A week passes.

• 42:05Speaker 05

  We haven't delivered any software yet. What happened? What happened was there's a new force to that play. There's some new constraints the team's facing.

• 42:16Speaker 04

  Yeah, so you're saying the constraints are what impacted the team's ability to to move as fast as we wanted?

• 42:24Speaker 05

  Exactly. So there's six constraints that we found all our teams want to review when they're pinpointing the forces acting on them. So we have token budget. This is a really common one. Maybe an organization is all in on AI, but how comfortable are they spending $10 a day for tokens? 100, 1,000, 10,000? That changes the conversation for the team. Organizational readiness. Do you have a sponsor? Do you have someone that's willing to back this effort? Goal definition. Do we know where we're going with this? Are we trying something out? Are we experimenting? Domain knowledge. How much of the domain is written down? Or how much is in people's heads? That will affect the outcome when agents get to play a lot.

• 43:10Speaker 05

  Verifiability. Can you automate? How if you've gotten to write yet? Do the agents know? Do you need subject matter experts? And then code oversight. This is a big one for regulated industries. If you need eyes on the code, that's going to change how you interact with the agents, right? So if we go back to our two day prototype, what happened? At organizational level, they had some token budget constraints. When we moved to our team, they had a budget that only allowed a couple of days of work before the devs ran out of tokens. At a product level, there was too much domain knowledge in heads. So they weren't able to have the agents get to what the right answer was because you had to ask a person.

• 43:55Speaker 05

  Then at the product level, they found out, yeah, the prototype could be verified by one person, but when we scaled it to production, now we had millions of input parameters. And we didn't have a great way to scale that up to verify every output was good. So what these end up being are kind of knobs or dials that your team can can hone in on. Now, some of the dials are going to be fixed. You may not have a say over token budget, but pinpointing that constraint brings up conversations that happened for our team, highlighting that these token budgets were an issue, highlighted to the organization that this is something that they need to weigh in on. For the domain knowledge, it changes who becomes interacting with the with the project.

• 44:37Speaker 05

  Okay, now we need subject matter experts here because we don't have enough information to the level of automation with the agents we want. So pinpointing these constraints is the first step of what leads into the next section, which is what mode the team operates in. So there's four modes that we commonly find our teams operating in. And now there may be more, there may be less, but this is a good delineation of very distinct different modes of operation. And they're spread out by a level of agent autonomy. So on the left, we have this individual mode where there's going to be a human in the loop. Quite often, they're going to be doing their own thing. As we move to the right, you can see we land on the dark factory.

• 45:19Speaker 05

  The humans have left, the light's turned off, the robots are running the place, and our product's coming out the end, full autonomy. So for our two day prototype, they're working in individual agent to coding mode, right? Because they were just getting a prototype out. They didn't have to worry about scaling too much. They could verify one output and say, yeah, we got a process working. Once our team scaled, that kind of broke down, right? That was bad fit. There was there was friction there now. So let's look at a different example for a different mode. So the team I'm working on, we work on some software that designs building components for a client that manufactures them. So building components, this room's actually fantastic.

• 46:02Speaker 05

  If you look around, we got some concrete blocks and there's some steel beams. And if you look up, there's some Joyce and HVAC units, so building building components. So high cost of failure, right? We don't want buildings falling down. How do you weave agents into that? Because I bet you won't be too happy if I told you, yeah, the building will stand up because our agents verified it. So we started in an individual mode and we wanted to move to the shared mode. And to do that, we had to build a shared pile of information. So what that looks like is we have domain elements, architecture. We may have workflows, rules for how to implement code, plan. But it's all in one body of knowledge that everyone has access to.

• 46:46Speaker 05

  Now, the the developers day to day may still kind of work on their own. They may pair with the agent, but now we know what we're doing, right? We're all we all have the same domain rules. So the point of this is now we're giving the agents a little bit more autonomy here. And the shape of that organization differs based on the constraints that our project allowed. We could have gone further, but we didn't want to because verifiability was really important for us, right? Buildings don't fall down. That's the goal. So we weren't comfortable going further. We made that choice. This was a good fit for us. Okay, what if we wanted to go further? We have a team that did that. So we have a team that worked on a mobile banking app for forum. If you're not familiar, forums, a local credit union here.

• 47:30Speaker 05

  So they had a banking app. It worked. It was in C sharp and they said, hey, we wanted to go to iOS and Android native. Yeah, great. Wonderful. What about tokens? Oh, we don't care about tokens. Spend all the money you want. Let's just see what we can do. Okay, so you have all the requirements set up here. Just want to come go to another code base. There's no cost. We have backing by the organization. So we built something that looks like what you see on the screen now. Now you can see that we're losing individuality. We're talking about a team here and a workflow of agents. The agents are starting to take on roles of their own. They're talking to an orchestrator. The team is providing intent, maybe some context. They're checking on the repo.

• 48:14Speaker 05

  But all the individually has been lost. And what we saw here was we had a greater level of autonomy and that was reflected in a six X reduction of effort for the project. So you can kind of see there how we're moving up in the steps, right?

• 48:30Speaker 04

  Right. So the goal then, you went from individual to shared or like the goal is to get to dark factory. That's what we're trying to do, right?

• 48:36Speaker 05

  Uh, maybe, maybe not necessarily because the goal here is fit, right? That's how we started. Constraints lead to modes. The mode is your fit. So what we do with our teams is we have our six constraints and we have them evaluate them. Depending on how the constraints evaluate, you may have a growing level of area that you can operate in. More constrained areas, you're going to work in a more individualistic mode. Maybe you're working on a prototype. Maybe you have a lot of constraints. Maybe you need to make sure every line of code is right. Maybe you just don't have access to the internet. Probably doing something more individual. As your constraints grow, we get that higher level autonomy, like we saw with the forum team.

• 49:19Speaker 05

  But it's about identifying those constraints and picking the mode that best operates for you, right?

• 49:25Speaker 04

  So you might be asking like, what what does this look like for me or my team? What what? What should I be considering if I understand these constraints? How does that actually play out? I spent many years coaching teams and clients on lean and lean software development. It turns out lean has an answer to this problem. Lean, first off, would say there is there's no single blueprint. There's no one right answer for everybody. Again, the reason why we don't all need a dark factory. And so Toyota never told anyone to hang an end on cord. And you don't know what that is. It's it's a cord they would put into the factory next to a line. And anybody could pull the the cord and stop the line. If they saw a problem manifest it, they could stop the line, fix the problem and and continue on.

• 50:10Speaker 04

  And that sort of tied to Toyota's principle of build quality in like, don't wait to the end to inspect it, but like, stop, fix the problem right now and build quality into the process as we're going. Now, interestingly enough, you may have heard of an end on cord before, may not. But Toyota did not put those on every single line in every factory. It was very constraint based. Based on the constraints of that situation or line, they might implement this. The principle build quality in that that applied everywhere. And so so principles, think about this are true everywhere. The practices aren't. The the practices will vary depending on your, like Andrew said, your your constraints.

• 50:53Speaker 04

  And so in 20 minutes, like, I can't tell you exactly what your practices should look like. But what we can do, I think, is walk through a set of principles that's going to help you get started.

• 51:06Speaker 05

  Exactly. Okay, so we got five principles. Good news, they're easy to learn, hard part, hard to master, but we're going to go through them right here. So we're going to start with the foundation, shared infrastructure. Call back to my team. We went from individual to shared. We started pooling the knowledge into the code base so everyone had access to it. It's shared infrastructure. Next, we have deliberate iteration. So this is a key one. I see a lot of teams fall through. They are doing things. They may even have shared infrastructure, but they're not talking about it. Maybe one person is having a terrible experience with a model. Maybe one person thinks the security audit on the review is just noise.

• 51:48Speaker 05

  Until you schedule a block of time in the week for that and talk about it, it's not going to improve. Next is making it safe, which is verification before velocity. That's exactly what happened to our team in the prototype example, right? We slowed down. That's because the scale changed, the constraints changed, and we needed to make sure that we were getting the right answer before we went fast. Once we're ready to go fast, next we're looking for systematic bottlenecks. Review is going to be a sticking point. If review is a sticking point, you need to make it not a bottleneck. And once you do that, it's going to move somewhere else. You have to keep chasing them. Remember when we called back to the research? The the team gets amplified in all their attributes.

• 52:32Speaker 05

  So the the bottlenecks are just going to be worse than whatever they were before. And you're going to feel the pain. You have to keep chasing those to make the team fast. Finally, remove the toil. Script the things that happen over and over again. Save the thinking for the humans. Save the analytical processes for the L L Ms. It's going to save you tokens. It's going to be faster. It's going to just make life better for the team. Five principles. Three teams, same principles, different modes, different practices. Get it? So that's it. Thank you.

• 53:05Daniel Whitenack?

  Let's go. That was awesome. I think I think basically everyone here who manages a dev team should make that talk required listening. That that was that was awesome. And yeah, the the those sorts of questions are really the questions that we're all asking as we're advancing our team and understanding how to work together, work together again.

• 53:38Speaker 00

  All right. That's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments and we would love for you to join the conversation. Thanks to our partner, Prediction Guard, for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.