While loops with tool calls

Dan and Chris sit down (again) with Jared Zoneraich, co-founder and CEO of PromptLayer, to discuss how prompt engineering has evolved into context engineering (and while loops with tool calls). Jared shares insights on building flexible AI applications, managing tool calls, testing and versioning prompts, and empowering both technical and non-technical users in AI development. Along the way, they dive into coding agents and the “crawl-walk-run” approach to AI deployment. Featuring: Jared Zoneraich – LinkedIn https://www.linkedin.com/in/imjaredz/Chris Benson – Website https://chrisbenson.com/, LinkedIn https://www.linkedin.com/in/chrisbenson, Bluesky https://bsky.app/profile/chrisbenson.bsky.social, GitHub https://github.com/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: PromptLayer https://www.promptlayer.com/Upcoming Events: Join us at the Midwest AI Summit https://midwestaisummit.com/ on November 13 in Indianapolis to hear world-class speakers share how they’ve scaled AI solutions. Don’t miss the AI Engineering Lounge, where you can sit down with experts for hands-on guidance. Reserve your spot today! Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:03Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:48Speaker 01

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris?

• 1:04Speaker 02

  I'm doing just dandy today, Daniel. Getting ready. As we record this, we're approaching Halloween, so I'm getting in the spirit.

• 1:11Speaker 01

  All sorts of sweet treats.

• 1:13Speaker 02

  Thank goodness the listeners can't see me, but yeah, I'm busy growing out my costume in this week beforehand, so I don't look so good. So yeah, glad this is audio only. Thank goodness.

• 1:26Speaker 01

  Well, you're in between. Yeah, you're getting ready for the festive season. That's okay.

• 1:32Speaker 02

  That's right.

• 1:33Speaker 01

  Not quite to no-shave November, I guess. But you could stretch it out through November and...

• 1:41Speaker 02

  I'm starting to become the monster under the bed, maybe. You know?

• 1:44Speaker 01

  There you go. Speaking of sweet treats, our guest today has me thinking about cake because a little while ago, we had Jared Zoneraich, who is co-founder and CEO at PromptLayer, join us. And of course, looking always reminded of their nice cake logo, which is making me hungry right now. I'm wanting some dinner. But welcome back to the show, Jared. It's great to have you.

• 2:09Speaker 03

  Thank you, thank you. I'm excited to be back. And a fun note for you, I'll tell you, we're designing our booths to go to conferences now, and I think we're going to bring real cakes to the booth.

• 2:21Speaker 01

  That's awesome. You should. Definitely.

• 2:24Speaker 02

  Nice.

• 2:24Speaker 01

  You should.

• 2:24Speaker 03

  Steer into it.

• 2:26Speaker 01

  Yeah, yeah, just like have those little, I see at the airport occasionally, they have those little vending machines where you can get cake out of the vending machine. It's in a little like plastic thing you pull open and eat while you're at the airport. So yeah, I think that's a, I think that's a great idea. But I'm just looking, we, we last talked to you March of 2024. So episode 261. Folks, go ahead and can loop back and hear from Jared when we had this initial discussion, which was great. At the time, Jared, everyone was, of course, talking about the, I think we were still in the days of everyone talking about kind of prompt engineers and prompting going crazy.

• 3:11Speaker 01

  Certainly people are still talking about prompting, but maybe they've shifted in some ways focus as you've been kind of at some of the, the center of discussions around prompting. How people are engaging with AI. From your perspective, what has the year been like? How, how have things kind of changed in people's perceptions of, of prompting AI systems? Or maybe even your own thoughts around prompting AI systems? And, and of course we can get into agent stuff later, but yeah, any thoughts?

• 3:44Speaker 03

  Well, we're still called prompt layer. So we haven't changed that yet. I don't think we will. But yeah, so March 2024, a year, a year and a half almost ago. It's almost, it would be hard to even list all the big AI things that happened since then. I mean, the, the one that immediately comes to mind though is reasoning models. So reasoning models, I think OpenAI released O1. I think I saw a tweet that it was like a year from last week or something like that. And that was, I think, the first next generation of prompting. Meaning before, if you recall, probably our last talk, we talked about chain of thought. And now I guess the reasoning model does the chain of thought for you and just the models have been getting better.

• 4:28Speaker 03

  But at the core, the core way to think about LLM applications as an input.

• 4:35Speaker 03

  To an output or input to the LM, meaning the prompt and the model and then the output. That's all the same. I think it's gotten much easier. The models have gotten much better. They they're easier to steer and some of the some of the weirdness of how you persuade the model and yell at the model has changed, has gone away and it's gotten a little bit more straightforward. But I think, yeah, I think prompting is still at the core of everything. And I will have to say, the new word people love to say is context engineering, of course. And to me, they're the same. But I think the main thing, reason people like this word context engineering is it's not just the prompt. It's not just the text. It's how much are you putting in the text? Are you putting too much? Is the model getting

• 5:19Speaker 03

  distracted? Are you using rag? Are you not using rag? Are you throwing in a blob? Are you using multiple models? And I think my like, high level one sentence of what's changed would be, we have way more tools at our disposal now. And we have way more mixing and matching.

• 5:39Speaker 01

  Interesting. Yeah. And and I guess that how would you distinguish or how might you your thinking around kind of context engineering? You know, maybe that is a is a term that's coming up. What do you think is kind of just some jargon that's changing? And what do you think is kind of a more substantive part of what that what that means? And maybe why people are shifting language, if there is one?

• 6:09Speaker 03

  Right. I think the key around context engineering and the reason, well, we're a little financially invested in prompt engineering because we made hats that say prompt engineer on it. So maybe that's where my bias comes from. But maybe we'll make both hats. But I think context engineering really gets to this. Well, one context is much longer. So you can send so much more to a model than a year and a half ago. It's almost unlimited from a user perspective. And the question is, how do you send it? How do you make? Have you squeezed the juice out of the model? Models, I like to think, get distracted like us humans do. So how do you not lead it down the wrong path? And

• 6:51Speaker 03

  The core difference, though, on how we build stuff is I think there's been much more, because models have gotten better and because we can fit more in the context, we've moved from, and we've seen this with our customers, moved from complex DAGs or complex workflows where you say this prompt goes to this prompt and this prompt goes to that prompt, to something a little more autonomous and a little more agentic and a little more of just a while loop with tool calls. Everything's a while loop with tool calls now, but let me, that's a whole, that's a whole different topic, but let me leave it at that.

• 7:28Speaker 02

  I'm curious, just to kind of pick up on that point, though, I think, I mean, I'm guessing a lot of our listeners can really, you know, feel in association with what you're just talking about, about that evolution. I know that, like, in my own job, it's not a primary responsibility, but as an ancillary, I often get asked to talk to groups in our company about how to prompt and stuff, and some of the things we had put out internal videos, like that I put out a year and a half ago that are completely obsolete. And so I'll, I'll come back to a group and I'll be like, that whole thing I told you before, don't do it that way anymore. It's gotten a little bit more capable, a little bit more sophisticated,

• 8:07Speaker 02

  drop the, the structured framework a bit. I'm curious, you know, so I'm kind of backing out of what I told them a year and a half ago at this point, and I'm sure a year and a half from now I'll be doing the same thing with what I'm doing today. So like, how do you address, when you're dealing with end users, how to evolve with the capabilities that the model and the infrastructures are providing these days? Because it's really hard to track that fast, quite honestly.

• 8:35Speaker 03

  It takes me many hours a day of scrolling on Twitter to keep, keep in touch with what's going on. I would say, so I guess for context, for everybody, we're, we're a platform for building and testing L M applications. And people are using us to

• 8:51Speaker 03

  Version, their prompts version, their agents evaluate them, log them and generally just build a AI system that's working. So we get asked this question all the time of like, how do we track it? What should we do today? And also, we have the same problem. We have YouTube videos or interviews where maybe I re-watch it, I'm like, oh, well, that's not true anymore. Now you could just use this.

• 9:14Speaker 02

  Thank goodness it's not just me.

• 9:16Speaker 03

  Yeah, I mean, it's everyone. It's OpenAI too. It's Anthropic too. It's Gemini. I think, I think the answer is to lock into the core truths of building LLM systems. And how we look at it is it's a philosophy thing. It's, I guess, there's, there's two, there's two competing ideas here, and I'm curious also to hear your opinions on this. But there's the academic idea of how do you understand LLMs and how do you understand which the context and how it works? And then there's the tinker, the builder, philosophy, which we push people towards, which is, it's a black box. I don't need to understand it. I just need my input to match to what I want in the output. And I usually give

• 10:01Speaker 03

  a really annoying answer to teams who use us, because we, we get on a lot of calls with our customers. We're not a consulting product. We're, we're a software, but we love to help our customers and we like to share knowledge. So, so we get on a lot of these calls and almost always my answer to a specific question of, should I use GPT-5? Or should I use Sonnet four or five? Or should I use Gemini? Is that I don't know the answer. I don't think OpenAI knows the answer. I don't think Anthropic knows the answer. The only person who can know the answer is you for your own use case by building tests and checking and seeing what works. Basically, it's a black box. You just have

• 10:43Speaker 03

  to try it out and see if it works. And that's the only repeatable motion here. I know it's an annoying answer, though.

• 10:50Speaker 02

  No, it's, it's good. It's, and I think it's, I think it's, it's, it's an answer that flows with the times, because anything that you would say is going to be, you know, anything that you, any of us say today is obsolete tomorrow with the way this thing accelerates. So it's, it's not, not as annoying as you might think.

• 11:08Speaker 03

  It's a framework for figuring out the answer instead of relying on us three. And it's a how do you, how do you think about testing? Because testing them is really hard and figuring out if your output is good and if the prompt works is really hard. So how do you build heuristics and how do you build evals? And how do you even, or how do you skip the eval? Like, just how, how do you, how do you approach this is the interesting thing, at least to me.

• 11:37Speaker 01

  Yeah, and I, I think I, I agree with you, Jared, in terms of like the spectrum that you were talking about. The reality is a lot of people don't need to know what, you know, self-attention, you know, means. For example, to build cool AI automations. I think there is like, on the flip side of that, I guess there would be this kind of like, you need kind of enough of a mental model to understand kind of maybe the coverage of, to your point, like, I think this is partially why the testing is, is so hard. Cause sometimes you, it may not be clear to you like, oh, if I, for example, change the formatting of my prompt without changing any of the words, why would that change anything?

• 12:22Speaker 01

  And kind of these things around prompt sensitivity and, and other things, right? So some of that like, some of that can be built up through tinkering. But maybe also comes even with this kind of mental model, maybe not so much of like what the models are. But I find sometimes like how they operate in generating output is often a trigger for people to know like, oh, well now it makes sense maybe why kind of like, things could go off the rails with, you know, high temperature or, or other, other things like that. So, so yeah, I don't know if you have any thoughts on that, but that, that, that's kind of my thought is that there's like,

• 13:00Speaker 01

  a different kind of mental model or intuition that you need, less so the academic kind and more so like how, how things operate.

• 13:08Speaker 03

  Yeah, a hundred percent agree. I think I think the I think you could have too much academic knowledge of how L M's work and that actually might hurt you in this because you'll try to understand what you're doing. Whereas a lot of these things are pretty hard to understand. And maybe people don't it's it's in the neural net somewhere. But I think what you're referring to, I like to refer to it as a L M idiom. So how do you how do you understand the language of L M's? I think the example you gave was good about formatting, like JSON formatting has been a big topic that people talk about of can you just give JSON to the model instead of a prompt and how will that work? And I think it's a really illustrative example

• 13:50Speaker 03

  because giving asking the model to return JSON or structured format will work really well if you want to return precise numbers or precise values. But if you want to write a love note, it's probably not going to work as well because the model is now in I'm coding gear. It's like if you go into maybe I like to I always compare it kind of the I don't think A I is human, but I think there's some things we can kind of use as metaphors. So it's like if you go into an engineering school and you just give someone a paper and say, write a love note, they're they're thinking about code. They're not in the brain space of of poetry as opposed to maybe they are good at it, maybe they can do it, but you kind of have to step back and say, OK, this is what we're thinking about in the

• 14:33Speaker 03

  same way as if you're asking the model to output JSON and output a key and curly braces. It's probably not going to write as good creative language in one of those values. And I can't write academic proof for you on why this is true. But it is it's a it's like an idiom. It's intuition. It's a way to talk to the L M without using so many words, I think, is where I'm kind of settling around.

• 15:00Speaker 02

  And you may have just identified why my last anniversary with my wife just went off the rails. You know, the JSON output for the love note, I thought I had it, but you know, clearly not.

• 15:11Speaker 03

  Yeah, it's a, it, it, you gotta, you gotta be careful with it. But it's really an important thing of, I mean, when we're talking as humans, we have idioms that we can say one word and you're gonna think of a whole different thing. If I mentioned Beethoven or if I mentioned Kanye West, you're, you're not, you're not thinking of just the person, you're thinking of everything surrounding them. And I think it's the same thing here and it's the same. You're putting it in a probability space. And if you want to be in this part of the probability space, it's gonna be a little bit more challenging.

• 15:48Speaker 01

  So Jared, I do want to dig into something that you said a little bit earlier, which is there has been this progression maybe from static workflows or kind of prompt chaining where you have like a DAG, um, which people, if they're not familiar, like that's sort of a one directional graph of like this logic that the chain of, of calls to the AI system is, is going through. You kind of mentioned that shifting more to kind of a while loop with tool calls. So could you break that down? I mean, first of all, like maybe for those that aren't familiar, kind of what you might, what you mean by kind of tool calling? And then like what, what this means to have a while loop of, of tool calls?

• 16:29Speaker 03

  Yeah, so maybe the best way to start is to like paint the evolution here. So like, uh, you just mentioned DAGs, these one direction, these basically graphs that are just a bunch of nodes that say this node goes into this node, this node goes into this node. The reason we started with that is because models were a little unpredictable because of hallucinations. If you were at the beginning of this LLM craze, let's say two years ago, three years ago, maybe three years ago is too much, maybe it didn't exist. No, yeah, it's something like three years ago was ChatGPT. If you're a United Airlines and you're making a customer support chatbot, you don't want to accidentally give people free flights.

• 17:09Speaker 03

  And to avoid doing that, the best way to do it would be to build these structured paths that LLM can go down. So the first question would decide what the user was asking. And if the user was asking for a refund, it would go to the refund prompt. And this is kind of how, let's call it prompt engineers, context engineers, agent engineers, whatever you want to call it, stopped LLMs from going off the rails. Now what's changed is

• 17:37Speaker 03

  Too big, let's call it innovations. One, just models are better at following instructions and hallucinations really don't happen that much anymore. And the second thing is, models have gotten much better at structured outputs. So before, it was kind of hacky to get the model to return in a way that code can process it. Now, tool calling, which I'll explain in a second, is baked into all the main models. So what tool calling is, is basically, you're telling your prompt, you're giving the instructions to the prompt, but you're also telling the prompt, it has access to a few different functions. So in the United Airlines example, maybe the functions are issue refund, maybe there's

• 18:21Speaker 03

  another function of check user status. When you use chatgpt.com, it has access to search the web, it has access to, like, when you're generating an image, it's probably like a generate image type tool. So this way, as we've built more tool calls, a lot of the models have been built around tool calls and have gotten really good at interacting with them, interpreting their response, sending another message to them. And that's why you see so many more autonomous agents, like, like if you look at Claude Code, or you look at Codex. The reason coding agents are actually good now is because of this paradigm and because they've actually simplified everything and said, instead of this complex DAG where we think through every single step, we're gonna actually kind of give the model a little

• 19:06Speaker 03

  bit more free rein, let the model run things, see if it works, and actually fix it, because models are, turn out to be really good at fixing their own mistakes. And what that has unlocked is a lot more flexibility for the model. So now, if you use Claude Code, which is, or Codex, or any of these coding agents in the command line, they have only access, i- it's- The, we wrote a few blog posts on how it works behind the hood, but the simple way to explain it is it's one loop that says continue until the AI is done and then ask the user for input. So I'll say, make my application work. And then now it'll start the loop and it has access to just write in the terminal like a human.

• 19:49Speaker 03

  So it'll write something, it'll get the output and then it'll decide, do I wait for a user response or is it, or do I want to run another tool call? And this simple loop is much easier to develop, much easier to debug and kind of just the way everybody's gone. Of course it has disadvantages, but that's kind of the way I see it in terms of where we've gone.

• 20:13Speaker 02

  I'm curious, as we talk through this and, and you know, as we've transitioned into this agentic world, especially since we talked to you last time, you know, over the last year and a half, it's really, you know, come on strong, you know, versus where our conversation was back then when that was a very new thing. What kind of incumbencies does this, this whole new set of capabilities bring to the user when they're trying to think, you know, we've talked a little bit already in this conversation about kind of the evolution of prompting context and, and you know, the rapidity of that. But you know, as we move into this agentic world, any thoughts around, around what the user's new responsibilities are to be effective in that?

• 20:56Speaker 03

  Yeah. The user as in the builder of an AI application or?

• 21:00Speaker 02

  The user in this case is in the human who's listening to our podcast right now and is going to turn to their, their system at the end of the show and go, I'm going to go try that.

• 21:10Speaker 03

  Totally, totally. So a little both maybe. I think if you're just a user on chatbd.com or you're using an AI application that's available. Codex, cursor, whatever. It's capable of doing much more and working for a much longer time and staying on track more. And you kind of can have a, it can do a better job, AI now of figuring it out, let's call it. So if you, if you want to do a general task or do an exploration, because of this new concept that people are using to build AI applications, these AI agents are able to try something.

• 21:52Speaker 03

  If it doesn't work, try something else and do the exploration that humans would do. And we're really just trying to make them act more like a human. And the DAG way, the old way that we're talking about, where you have a bunch of nodes and you have a structured way, that's not how humans work. When you, when you, when you give a, an intern a task, you're not giving them the exact flow chart of how to solve it. You're telling them generally what tools they have at their disposal and they're going to keep working and using the tools and they figure it out. Now for the builder behind these applications, what it gives them is it makes it a little bit harder to test. It makes it a little bit harder to keep things on the rails, but it makes it much quicker to build something that succeeds.

• 22:34Speaker 03

  We built a, a agent using Claude Code that updates our docs every day. Basically, it looks at all the code our team has written over the last 24 hours and decides if it should be in the docs and then updates it. Took me two hours to build because all I said was download these repos, read the commits and then check our docs and see if it should change. And then it gets to figure it out. Now, is this going to be good for a production system? Maybe it needs a little bit more work, but for something simple, it, it opens a lot of use cases.

• 23:07Speaker 01

  And what would you say, I guess, as you mentioned, you're focused on kind of a platform around kind of building, testing, versioning, improving AI workflows or agents. With these kind of while loop tool calling things, right? If you have kind of a variety of tools that, that could be called under the hood and you have kind of, part of the complexity of the system, I guess, is in the tools that you, that you can call, how does that? How does that influence, I guess, like, the way that you should version or test these systems, or does it even, in the sense that you kind of now, whereas before it felt like

• 23:52Speaker 01

  there was kind of a single function that I'm calling into with a prompt, right? And that function produces output which may or may not be useful. Here, you have kind of this recursive function that feeds into itself and could call any number of things. And so, like, when you look back, if you're just looking at the input and output, I guess is what I'm saying, then any number of things could have gone wrong in that kind of recursive loop, or in that while loop that you're talking about. So how does that, I guess, how does that influence kind of the prompts that you would version, how you iterate on those, how you test and improve the systems from your perspective?

• 24:32Speaker 03

  Totally. It makes it interesting. So at the end of the day, the tool calls that are being run, the functions, are still those input-output things that can be unit-tested and that can be really thoroughly and rigorously tested. The hard part is this while loop. So I think the core master prompt, the one that runs the loop, that can be tested in a lot of ways. You can run sanity checks, you can test against old data, you could see how things changed. But what you're getting at is this really interesting problem that's kind of been created, which is, how do you test a flexible agent where flexibility is kind of one of the keys to it, what makes it good? I think there's kind of, the heuristic I've developed is something I want to call, maybe

• 25:17Speaker 03

  I'll write a blog post about it, like, agent smell. So if you run an agent, what sanity checks can you see to see if it smells a little funky? Like, is something, is it raising any red flags? And I'll give you some examples. So if I was building an agent to, let's say, to fix errors in my application, so like, if I had a database error and I wanted to build an agent that would go and fix the code automatically, I would want, if I wanted to test this agent, what I would test for is, how many tool calls is there? First, I just want to surface these statistics. How many tool calls is it running? How many times is it retrying the tool calls? How long does the agent take? And these are kind of surface-level things, they're not end-all be-all, but...

• 26:01Speaker 03

  You first want to start simple, so you first want some sort of smell test where you can say, "Hey, um, this new version is behaving very differently than my old version. Maybe better, but maybe worse." And then that's when you go in and break down like a state by state test. So the most useful tests we see our users doing are individual states or full conversation simulations. So individual states would basically be saying, "Here's a conversation history. Now run that through a tool and what's the next step?" And we're just checking if the next step is the same. Of course, that's only one part of the picture. The other part of the picture is, "Here's the initial instructions. Now simulate the whole conversation and see

• 26:44Speaker 03

  if the final output's correct." And then combine that with the smell and it doesn't give you a full picture, but I think the core learning we've had, at least since the past year and a half, is that you don't need to have 100% coverage when you're evaluating these things. If anything, if you're trying to make a perfect test for your agents, you're probably never going to ship and you're probably gonna, it's just gonna, you're never going to do it. Uh, the better thing is to make it good and have heuristics of figuring out when it's regressed before it does.

• 27:19Speaker 01

  So Jared, we talked through a little bit of this agentic stuff and I know like you set up some of the conversation where you as a company are enabling kind of non-technical users to kind of work on prompts, embed their domain knowledge, kind of have that kind of non-technical connection to the prompts under the hood, which are maybe embedded in various systems or tools and that sort of thing. I'm wondering about your perspective on, obviously, one of the things that's been talked about in recent months a lot is this kind of like 95% of AI pilots failing and that sort of thing, which um, we've talked

• 28:03Speaker 01

  about on the show, uh, the, the report from MIT and gave some thoughts. But I'm, I'm wondering how you think that. intersects with the way that the tools that people are using to manage their they're prompting their AI systems, maybe the rigor that needs to be there that's not. Or maybe like there's another side of this where kind of some of those engineering principles need to be brought into the into the picture. Yeah. What is your thought kind of from working with a lot of non-technical users on a platform like this? You know, what have you learned over time to be those kind of key? Key pieces of making sure that those people coming to a problem don't

• 28:47Speaker 01

  end up just wasting a lot of their time working on something that doesn't actually get results.

• 28:53Speaker 03

  Right. So maybe 95% of AI pilots fail, but they're not using prompt layers. So that's why it's.

• 29:02Speaker 01

  Yeah, exactly. That was the that was the setup for the question.

• 29:07Speaker 03

  0% of AI pilots in prompt layer fail.

• 29:10Speaker 01

  Yeah, exactly. Same with prediction guard. Yeah.

• 29:14Speaker 03

  Exactly, exactly.

• 29:15Speaker 01

  Exactly. Yeah.

• 29:16Speaker 03

  They're not using the right tools. I would say. So as a platform, how we look at it is prompt layer. We have a large diversity of teams ranging from super technical and all engineers to basically no engineers and everything in the middle. And what we are trying to do is build a rigorous process for building these applications and expose it to the people who know if the outputs are correct or not. So what I mean by this is rigor in terms of versioning and knowing which versions are working. Rigor in terms of being able to test, like we're just talking about testing agents, testing prompts.

• 29:57Speaker 03

  And also rigor in terms of logging and seeing what's happening in production, what's going on? Because you can only test so much in development in AI and you kind of need to expand the surface area of how people use your product. The the reason we focus a lot on getting these domain experts involved in our in the process is because we believe, actually, from a business perspective, that's how you win as an AI company. If you're building legal AI, you win by having a lawyer involved. The example I always give, and I love going back to, so I come from a family of psychologists. It skipped me, I'm an engineer, but uh but but I have some familiarity with it.

• 30:42Speaker 03

  And if you want to go to

• 30:44Speaker 01

  But now you're just psychoanalyzing the language that's going into models. So I I guess your your family can be proud.

• 30:53Speaker 03

  I I hope so.

• 30:54Speaker 01

  Yeah.

• 30:55Speaker 03

  I hope so. I'm working on it. But uh if you want to see a shrink, there's like six on the block, right? I live in New York City. There's a lot of them. And the assumption I'm making here is that there's no global maximum. There's no one correct answer to psychology. Uh, you know, you have different methods, you have CBT, you have ayahuasca retreats. You have a lot of different ways to treat people. And same with medical doctors, same with education, same with a lot of things. And what's the core differentiator between the different psychologists on the block that my office is on? The taste and how they choose to practice their field. They they're all going to the same education, maybe some have a little bit more knowledge than others.

• 31:41Speaker 03

  But how they implement their product, let's say, and in the same way as if you're building an AI therapist. How you win as a business is the the non-engineering taste that's been put into the AI product. And the way it's using the context that you provided and what you've told it to do. And uh going back to your question of how do you? How do you take that knowledge of needing the non-technical engineer? Or like like we think an AI engineer should be non-technical. But how do you bring in those engineering principles so the pilot doesn't fail? And a lot of times we see engineering actually owns the product, the AI product.

• 32:26Speaker 03

  So we're usually talking to VP of Engineering or CTOs or something like that to get prompt layer installed. Because we're all engineers in our team, we're bringing in these principles. We think even if it's a non-technical expertise, you have to do it in a organized and systematic way. And that's why you see the skill of prompting, I almost think, is not quite the same Venn diagram as the skill of coding, because it's really a skill of tinkering. And not all coders are tinkers, but not all writers are tinkers either. So there's some new type of algorithmic thinking that overlaps very highly.

• 33:07Speaker 01

  Yeah, it's almost like being a negotiator, almost.

• 33:11Speaker 03

  Exactly.

• 33:11Speaker 01

  Yeah.

• 33:12Speaker 02

  It is, but it also, you know, maybe you have organizations that are leaning one way or the other. You know, you've just kind of described that spectrum of skill and expertise that apply. And so possibly for whatever problem your organization is trying to solve, trying to find the right place on that spectrum to bring the right resources together. And that, like even aside from what we're talking about here, that's an easy place for businesses to fall down anyway across it. And so in a sense, it may not be that different for many other business problems that companies are trying to face.

• 33:48Speaker 03

  Yeah, totally. It's like, what can we learn from non-AI world to ship these things better? To us, the big thing that, the big mistake people make is they try to boil the whole ocean at the beginning. And they try to do too much. And really, you want to do the whole crawl, walk, run when you build these systems. And you want to maybe instead of the pilot being, hey, we're going to add a billion dollars of revenue with our AI product, you want to say, all right, we're going to make a beta version. Maybe we'll only release it internally, maybe we'll do this. Maybe it won't do every, maybe it won't be the while loop with all the tool calls. Maybe it'll just be one tool call. And that's also true with how you test these things and how you build them.

• 34:32Speaker 03

  So it's not just what the product does, but it's A lot of companies, a lot of teams get stuck at trying to build tests because they try to build perfect tests. And like we were saying, it's, it's, it's hard and maybe even impossible.

• 34:45Speaker 02

  So, and there's a learning process, which that kind of implies crawl, walk, run is that it gives, it gives companies a chance to, to not crater too hard when they're first starting out, you know, as they're trying to get something done, keep it small enough scope so that they can actually achieve something small but positive and learn from that and kind of build up toward what their true aspirations might be.

• 35:08Speaker 03

  Exactly. Exactly. I mean, we're all learning.

• 35:10Speaker 01

  Yeah. And I don't know, when you said crawl, walk, run, Chris, it made me think like some of the problem might be that people don't understand what, what is crawling, what is walking, what is running? Like, it all just looks like AI tasks. Like, it's a big soup of things that you can sort of quote, do with AI. And it's unclear, you know, how do I pick out the crawling task? Because I don't know which of, I don't know which of them it is. I, I, I see that kind of paralysis a little bit. I don't know if, if you see that as well, Jared, or have any suggestions for like how people can think about, you know, picking apart, because, because you're, you mentioned like those domain

• 35:53Speaker 01

  experts who are coming into, you know, prompt layer and you're connecting those, those people with the business knowledge into prompt layer, you know, how is it that they come upon the knowledge to know what is a kind of crawl task or a feasible task to like start with and, and, and play around with? Any, any thoughts?

• 36:13Speaker 03

  Yeah, it's a good point. I think the most successful teams, AI teams I've seen, work in collaboration between engineering and domain experts. So if you're, if you're just domain experts or you're just engineers, you can succeed and I've seen it succeed, but the, the, the most common way and what we recommend is it should be a joint effort. The engineers often know how to ship a product and how to do agile or iterative design and the, the non-technical understand what makes it good. And you need both of these. I think how to break it down, it's, it's almost the, the crawl task. It's almost you have to just step back and say, what

• 36:55Speaker 03

  is my heuristic here as a, let's talk about testing, just what's the crawl task in testing of our AI application? The hard examples are like something like summaries where there is no ground truth. So what is the crawl task of evaluating a AI note taker? Well, you kind of have to step back and say, as a human, what is a good summary? All right, what's the simplest thing? Maybe the simplest thing is just saying, is the summary in English? Maybe doing another LM's judge where we say, does it use Markdown? And then maybe another one that says, is it less than a page? And these are all obviously not end all be all tests, but it's the crawl. And then, once that's working, you can check for hallucinations. And then maybe you can check for a style.

• 37:40Speaker 03

  But it's very use case dependent. So it's hard to give a one size fits all there.

• 37:46Speaker 02

  So I'm gonna, I wanna turn things as we're, we're starting to get closer toward the end. And I wanna ask a kind of a fun question for you. I know you like Claude Code. And so I wanted to ask some of the things that you're playing around with and, you know, what you're doing and, you know, what's got you excited on it?

• 38:02Speaker 03

  Yeah. And, and I will say, I, I like Claude Code, but I also like Codex. I also like Cursor. I also like AMP. AMP is doing really cool stuff with the, the free coding agent too. So I switch between all of them. I think I give Claude Code a lot of credit for being the first really useful coding agent that I've used. What am I doing? So a lot, we, we, we redid our whole engineering philosophy around these coding agents at Prompt Layer. Basically, the hard part about building a platform, as you guys likely know, is all these little things and the death by a thousand cuts. So we have to work on big features, but also the UX of like, this button here doesn't have

• 38:43Speaker 03

  a loading state or this is not dragable here is how you, you could fail if you don't fix those and that list piles on. So now we have a new rule in our company. If it takes less than two hours to do using Claude Code or Codex or something, just do it. Don't ask anyone. Don't prioritize it. And it's helped a lot, honestly. Our customers have literally told us like, wow, you guys are shipping so much faster now. So I think everyone who says, oh, it actually makes you a slower coder is just, it's just full of it. You know, it's so good. And I use it for non-technical things too. If I want to go through a CSV, I'll tell you, I'll tell you one thing I did recently that that is pretty interesting and non-technical.

• 39:28Speaker 03

  I went to an event and I won't say which one because then people will be like, oh, that's why I'm getting spammed. But I went to an event and everyone was pretty interesting on the event. And I basically copied and pasted the list of users there. I actually I gave the HTML to Claude Code. I said, make it into a CSV of all the people there who clicked going and whatever social media you have. And then I put it into Prompt Layer and I actually like added new columns. So I like did a batch of like find where they work, find their whatever. And then you could go back to Claude. I went back to Claude Code and did some data processing and saying like, who should I contact?

• 40:09Speaker 03

  And just doing like random tasks like that and batch prompting of like, I combined Prompt Layer with it, of course, but random stuff like sending emails, creating random UIs for like to understand a company. I use it for everything. I'm constantly on it. I constantly vibe code these days. And we sat down with everybody on our team and Claude coded with them just so they can see how good it is. Because a lot of people are skeptical and it's great.

• 40:38Speaker 01

  That's great, Jared. I love the tie in to even this sort of like combination of things that it's like when, when Anthropic released Quad Code, I don't know that they imagined all of these like trickle down effects of like things in the way that people are combining it with other things and the like even the non-technical things that you mentioned. So it's really cool to see how that plays out. As you, as you kind of look towards the future, as we, we get to the kind of wrap wrapping up here. Tell us a little bit about kind of what excites you kind of moving into this next year. And you know, when we talk again in a, in a, in the next year and a half, or whenever it is,

• 41:21Speaker 01

  what, what, what you're excited about during that period of time, like related to prompt layer and related to kind of things, things in general and how the ecosystem is evolving.

• 41:30Speaker 03

  I'm excited about a lot. The simplest one, these, we're talking about it, these coding agents as a headless tool. So using them in your workflows to run things. That's exciting. I'm excited about especially non-technical uses of these things. I think you're going to, right now they're in a terminal and you're going to be using them for so much more. And then Claude code aside, I'm very excited about the whole, how the space is evolving to a place where you have a lot of different tools at your disposal. Some models are really good at writing. Some models are really good at coding. And the consumer is just having more options than ever to building their product. I think we're not in a world of one model rules at all.

• 42:15Speaker 03

  There's a lot of ways to solve a problem. There's a lot of variability on how you build your product. And I think that's a good thing. I think I'm very, I think the future is great. I'm, I'm excited for AI to take over. I think, I don't, I'm not worried about it at all. And yeah, I think, I think, and I'm honestly really excited. I know this is a little bit of a, a shill because this is what our company does, but I'm very excited about unlocking AI engineering for people who didn't study computer science. And I think this has been something people have talked about for so long of how do we democratize coding and get more people coding and maybe people aren't going to be coding anymore.

• 42:56Speaker 03

  But the way to like, people have expertise and now they're going to be able to build AI products around it and do AI engineering around it. And it's really Anybody's going to be able to distribute their work to almost infinite levels. So that's what keeps me up at night in a good way.

• 43:14Speaker 01

  That's awesome. And yeah, I would, I would definitely recommend, of course, we'll include links to prompt layer in the show notes, but also as, as Jared mentioned, they have a, they have a great blog. You know, they've, they've released some, some excellent articles. They, they have great learning resources out there, so check out everything that they're doing. Really appreciate the way that you all are contributing to, to the ecosystem, Jared, and definitely keep up the good work and we'll look forward to, to talking with you again next time you're on the show.

• 43:47Speaker 03

  Amazing, amazing. Thanks for having me and anyone can reach out on Twitter or on email or sign up for prompt layer and get started for free. So excited to see what people build.

• 43:57Speaker 01

  Yeah, definitely. All right. Talk to you soon.

• 44:00Speaker 03

  Thank you.

• 44:08Speaker 00

  All right. That's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X or Blue Sky. You'll see us posting insights related to the latest AI developments and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.