Prompting the future

Daniel & Chris explore the state of the art in prompt engineering with Jared Zoneraich, the founder of PromptLayer. PromptLayer is the first platform built specifically for prompt engineering. It can visually manage prompts, evaluate models, log LLM requests, search usage history, and help your organization collaborate as a team. Jared provides expert guidance in how to be implement prompt engineering, but also illustrates how we got here, and where we’re likely to go next. Sponsors: Shopify https://www.shopify.com/practicalai – Sign up for a $1/month trial period at shopify.com/practicalai https://www.shopify.com/practicalaiFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Jared Zoneraich – LinkedIn https://www.linkedin.com/in/imjaredz, X https://x.com/imjaredzChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: PromptLayer https://promptlayer.comUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whiteknecht. I am founder and CEO at Prediction Guard, and I am joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris? I'm doing fine. How's it going today, Daniel? It's going great. I'm pretty excited to prompt our guest today and hear what he has to say. We're joined today by the prompt master, Jared Zoneraich, who is founder at Promptlayer. How are you doing, Jared?

• 1:17Speaker 02

  I am doing well. Excited for this.

• 1:19Daniel Whitenack?

  We're excited to have you. It seems like, maybe from my perspective, there was kind of the release of all of this generative AI stuff, and then there was this realization that there's kind of a new skill needed around this thing called prompt engineering. And then it seems like some people have kind of like, I don't know if they've moved past the term, or they've tried to develop other terms. I see other terms being developed around like AI engineering and such, and as related to generative AI. So could you just give us a sense of like, from your perspective, who, you know, is obviously building things for prompt engineering, maybe

• 2:02Daniel Whitenack?

  to start out, kind of what is prompt engineering? And from your perspective, how have you seen it develop as a skill over the past year since people have been thinking a lot about prompting generative models?

• 2:16Speaker 02

  Yeah, the last year since the word was invented, or maybe a little more than a year ago. Uh, yeah, no, I think it's a really good question. I think this question of what does prompt engineering even mean? Is a is also a good question. Um, and I think I'll tell you how we think about it. Uh, so. Honestly, so for prompt layer, we consider ourselves a prompt engineering platform. So we've really steered into this term that is kind of overloaded. We've embraced it for sure. And it was half by accident, and I would say half realizing it was kind of beneficial to us because the fact that it has no definition means anyone who's kind of

• 2:56Speaker 02

  getting into L L Ms and getting into prompt engineering says, oh, there's a prompt engineer platform, maybe I need that. And they don't even necessarily know what it is. So it kind of helps us a little bit there. I guess prompt engineering, I first started hearing that term, I'd say G P T three days, kind of a little bit before Chat G P T, maybe G P T two. Back when everyone was just using the OpenAI Playground. And you'd start to hear a little bit about prompt engineering and stuff like that. And it was kind of cool, but that was, I guess, in the days before it clicked for everyone. Uh, maybe, maybe you can call it that. Uh, days before Chat G P D came out and before people really realize how much potential this technology has. So that was when I think prompt engineering first became

• 3:40Speaker 02

  a word and then kind of got even more, you know, scale AI. I think is famous for maybe the first one to publicly hire someone with a role prompt engineer. And uh, from there, we called their platform a prompt engineering platform. At the beginning, people brought a lot of people to us who had no idea what prompt engineering is. And kind of the definition we've started to roll with is as a company, we consider prompt engineering to be the tuning of the inputs to the L L M. So the prompt is the main input, but it also includes what model are you using? What are your temperature? What are your other hyperparameters? But the whole process of prompt engineering to us is what goes in?

• 4:24Speaker 02

  And what comes out and that specifically is and I could talk a little bit more about this if it's interesting, but specifically a little bit different to the MLOps definition or the the standard machine learning definitions of hyper parameter tuning and standard traditional ML and uh specifically we call ourselves a prompt engineering platform and not an L M Ops platform for that reason because I do think there is slightly a difference. So I don't know if I answered the question fully, but uh that those are my thoughts on prompt engineering.

• 4:59Chris Benson?

  As we all dived into it, like you talked about, you know, the OpenAI playground originally that I think everybody kind of dips their toe into first, at least before the chat GPT release days. With that, uh, one of the things I discovered as other models started coming out was some of the skills I was developing for prompting did not always translate as I expected across to other models. Have you uh, have you seen uh, one of those things where every model had its own kind of variations of what seemed to work from a a productivity and output standpoint. Any thoughts around that? Like, how do you guys? How do you guys see that with all these different models coming out and little variations across them in terms of how they respond?

• 5:38Speaker 02

  Yeah, that's for sure. A good observation. I think uh, it's become clear that each model, I mean, each model is made a little bit differently. So you have to talk to it a little bit differently. Uh, I think these differences are gonna maybe get a little less significant over the future. Uh, you know, when I guess chat GPT came out and the big thing was if you were nicer to the model, you're gonna get a better answer, maybe because stack overflow questions that are nicer get better answers or something like that. And I think now you have people talking about, oh, if my grandma's gonna die, you need to answer this, or I'm gonna tip you a hundred dollars if you answer this. These are all, I think, tricks that work today.

• 6:20Speaker 02

  They're not gonna last forever. These are like little things people figure out. But the part you mentioned of talking to different models differently, I think is not going away. These are, a lot of these models are made very differently. We, think it's pretty conclusive now that we're not gonna live in a world with an open AI monopoly of language models. Uh, I think just a week or two ago, I saw a good tweet like, now we have Mistral, Claude, GPT-4, and a few others that are all really good. And they're all made somewhat differently, and there are intricacies. And I think our philosophy and what we talk to our users about, uh, regarding prompt engineering is think about it as a black box. It's almost helpful to be kind of

• 7:01Speaker 02

  a little bit naive and stupid here, and not try to understand how an LLM works, and just try to track the inputs to the outputs, if that makes sense.

• 7:09Daniel Whitenack?

  Have you found any difference from people that are coming maybe from a deep sort of like data sciency background where they like are over analyzing everything, and then other people who are coming maybe from a not technical background, but they're maybe domain experts and they're getting into um, developing these prompts. Have you seen different struggles on each side of that spectrum where, because you have sort of this very interesting mix of people that are trying to be like prompt engineers, quote unquote, some of which I've kind of seen are very much just like non-technical domain experts who are really good at even like psychology and

• 7:51Daniel Whitenack?

  writing narrative instructions, you know, um, being articulate. And then you kind of have this other side, which is the data science side, and they're really into modeling and wanting to analyze all of the outputs and that sort of thing. So have you seen different struggles on both sides of that in terms of being effective prompt engineers?

• 8:13Speaker 02

  I think you put it well that there's kind of these two groups coming at L M's. Traditionally, I I mean, this is what makes L M so cool to me is that traditionally M L, traditional machine learning standard like Matthew machine learning kind of need a PhD. Uh, maybe you don't need it, but PhD is very helpful. Uh, it's intense. How you're building these models, doing a lot by hand. Still, you're doing a lot of this tweaking by hand in traditional M L. And then OpenAI came out with this amazing A P I and I've done a lot with DevRel in the past and hackathons and stuff like that. And helping companies make A P I's and in my opinion, OpenAI's A P I, it's like the best docs I've ever read in my life. It's so simple. You just give it text and you get text out.

• 8:58Speaker 02

  And like you said, this kind of opened up this completely new technology that's just so much better than everything else. To non-technical people, you don't need a PhD to be able to understand how to communicate. And I think it brings about this new skill set, which is prompt engineering, which in my opinion is kind of a mixture of communication. And like being able to write succinctly, but also uh, being able to think algorithmically. So I don't know the exact word for thinking algorithmically. I've heard Stephen Wolfram use that word. I like that way to talk about it. But kind of just a scientific method. Do you know how to think in terms of creating a hypothesis, trying it out, tracking it? Are you strategic about this? And I think it's the same challenge on both sides.

• 9:41Speaker 02

  I think some people try to overcomplicate it if you're coming from an M L background and you're trying to understand why a certain token gives you an output. And I almost think these things are getting more complicated, not less. You kind of just need to take the naive approach and say, hey, I'm just gonna talk to it. I'm gonna try to get the output I want. I'm gonna keep trying stuff till it works.

• 10:02Chris Benson?

  As someone who's used like the A P I's a lot, you know, and you're talking about, you know, OpenAI is being so good. So many folks listening to this may have only uh, have only used things like the normal chat interfaces on each of the models. You know, when they've gone and tried them out or pay for a subscription for the the top end and stuff and have never touched the A P I's at all. Could you take a moment and just talk about what an API experience is like to someone who has done a lot? I'm kind of taking advantage of you as an expert in the area. Share that a little bit with listeners, just so they kind of get the other side of that, because not everybody, probably the vast majority of them don't.

• 10:39Speaker 02

  I'll explain in that, just to sidetrack myself for a sentence. I think the fact that you use the word expert is so funny. It's such a new field. It's it's almost amazing. I'm I tell everybody you could become an expert in this thing very easily. Like, nobody really knows what's going on. You kind of just need to study for a week and you're an expert, which is a very unique place to be. But, uh, anyway.

• 11:00Chris Benson?

  It makes it fun to be able to dive into something and and get as deep as the leaders in the field, so.

• 11:05Speaker 02

  Yeah, a hundred percent. And just to be on the cutting edge and know that nobody really knows what they're doing here. Some people do, but there's very few of them. So regarding the APIs, ChatGPT, the way I think about it and the way I usually explain it is it's basically a very thin wrapper on top of the API. Every time you talk to ChatGPT, behind the scenes, it's using this LLM technology. It's sending your message and a little bit of a preamble before your message. And the preamble is basically, we can call it the prompt. And it's basically saying, hey, you're an AI assistant. Make sure to be helpful to the user. Maybe don't be controversial. You can use a calculator if you need, stuff like that, and then giving the user messages. And that's all it is.

• 11:49Speaker 02

  And the process of prompt engineering is how do you tweak that preamble? And how do you how do you get it to respond in the way you want to respond by telling it what to do? When I talk about the API, I'm talking about the things OpenAI has exposed to let you build your own ChatGPT and let you build your own products on top of it. And I just think OpenAI has done, if you want to get started and you haven't touched these APIs today, the best, best thing to do is just go to OpenAI's docs and just read the Getting Started tutorial. It's really well done. And I think as someone who's running a developer tools company, I'll tell you how hard it is to write good docs. I think our docs, from my perspective, I think they should be much better.

• 12:31Speaker 02

  Other people say they're great, but well, there's always like room for improvement and it's just a very hard thing to do. So that's where you should go.

• 12:40Daniel Whitenack?

  From your perspective, over the last year, as people have gotten into doing this practice of prompt engineering, have you generally seen people that are engaging with your platform be sort of more informed coming in? Like, they've done that experimentation with ChatGPT or the APIs, and they are like using words like "few-shot", whatever, blah, blah, blah, and like this stuff? Or is it still kind of like people coming in that say, "I know, I think I need to have a prompt engineering platform, but I'm kind of don't know where to start"? Have you seen that shift even over in this last year?

• 13:21Speaker 02

  I would say we're still very early in this, and there's a few leaders, but everything's up for grabs in terms of AI products. I don't buy the whole notion of it's only the incumbents who are going to be able to use AI. There's so much. The AI products people are making today are just scratching the surface. Having said that, I have seen a bit of a change since we launched our product. We launched our product January of last year, so January of 2023. I guess that was a few months after ChatGPT came out. These things were, I think we were, I think we were the first prompt engineering platform. Maybe there's some argument there, but we were one of the first at least. And when we launched our product, we had a lot of indie hackers and individual hobbyists using our platform. That was the

• 14:04Speaker 02

  whole community back then. That lasted for a little. Then we kind of moved on to AI-first startups. So, like, one or two-person startups, these are like the really cutting-edge ones. We used one of them, it's a great company, to actually refactor our whole codebase into TypeScript. A lot of really cool stuff. And then from there, I would say, starting in the fall, and I've heard this from a few other founders in the space, it felt like there was, started to become a real shift where... Real companies, and when I say real, I mean, maybe companies that actually make money, uh, started actually getting serious about AI and getting serious about LLMs.

• 14:43Speaker 02

  And I think we're still seeing that maturation, uh, continuing where these real teams are building AI products that they care about and not just, but like Twitter demos are one thing. Prompt layer is interesting for a Twitter demo, but prompt layer becomes really useful for a team that is serious about building their product and has multiple stakeholders and wants to collaborate. And so I am seeing that this shift is still, is happening. More and more companies are getting serious about their LLM products and getting value and revenue from them. But we're still at the very, very beginning of the curve. So a lot, a lot more to come.

• 15:20Daniel Whitenack?

  I think that was a great sort of intro into prompt engineering and the state of prompt engineering. I'm wondering if you could help us maybe understand, yes, it's good if people get hands on with these models, kind of gain some intuition about how they behave and different ways that you prompt them. What is it about this discipline of prompt engineering that needs sort of systematic ways of managing your prompting methodologies? Um, and how is that different to or the same as different sorts of engineering in the past? You know, we've always had version control and that sort of thing. What's kind of unique and not unique about this discipline of prompt engineering in

• 16:04Daniel Whitenack?

  terms of how you need to approach it systematically?

• 16:07Speaker 02

  Just starting from first principles, there's one fundamental thing that's changed. And it's that we're now building upon a probabilistic technology. So we're now building on a technology that sometimes gives us some answers, sometimes gives us another answer and is trending to be more confusing on why it gives one answer and not less. And confusing, I mean, yes, theoretically it is deterministic. You can really dive into the weights and maybe figure it out, but nobody's really practically going to do that. It's virtually too hard for 99.9% of use cases with models. So we're at a place where you're working with a black box now. And yes, some servers, some architectures become black boxes if you use bad code,

• 16:50Speaker 02

  but that's a different type of black, that's not a real black box, but we're building technology.

• 16:55Daniel Whitenack?

  That's the code I write.

• 16:56Speaker 02

  Yeah. Me too. That's why I'm going to get banned from our repo soon, but. Yeah, so you're building technology on this black box and you need to think about it differently. And I think this is a big philosophy we have at Prompt Layer. I think we have the philosophy of we built a lot of great stuff in traditional software and traditional machine learning and had a lot of great learnings. Git is a fantastic tool. Version control is important. Access controls are important. Test-driven development is important, but do we necessarily want to take one to one of everything? Not really. I think the biggest difference between LLM-based development and building AI applications

• 17:40Speaker 02

  and versus building standard software is who are the stakeholders? Like we were talking about earlier, now you can have subject matter experts who are not necessarily software engineers, but rather these prompt engineers, these AI whisperer, like these people who are able to talk to the black box and able to communicate with the AI, we could call it in a little sci-fi sense. But we have this new stakeholder in the process of software engineering who is not going to jump into the code and not going to jump into Git. And that's why, at least for Prompt Layer, we've taken a very first principles, ground-up approach where we're saying, hey, what can we learn from normal software and how people work together on software and do version control and collaborate?

• 18:22Speaker 02

  And how can we take that into LLMs and bring in new stakeholders and bring in new collaborators and let people actually build on this black box technology in a systematic way? So hopefully that makes sense.

• 18:36Chris Benson?

  It does. I'm curious, you said some things there that I'm really kind of piqued my interest that you kind of were contrasting it with deterministic programming, you know, that we've all kind of grown up with. And now we're in this new age and we have these non-deterministic things and you can give it the same prompt and it may or may not, on any given day, give you the same, the same answer back. So how has that kind of fundamentally changed software development in general? And I'm encompassing all the things when I say software development, both the AI and the systems around it, defeat it, you know? Because when you're dealing with that potentially unexpected return and that non-deterministic black box that you're talking about. How do people handle that when they're trying to devise and say, hey, I want to use a model

• 19:21Chris Benson?

  in the thing that I'm building? How do they change in the way they think about that?

• 19:26Speaker 02

  I don't know how people think, but I can tell you a symptom of how they think, which is we work with a lot of L M teams, of course, because they're using our platform and we talk to them, we try to, we're always trying to talk to people, trying to figure out what to improve. And one of the big pieces we've come to with prompt layer is that the iteration cycle of prompt engineering is different than software engineering. So what I mean by that is your code deploys and continuous integration and that whole sort of thing is happening at a different cadence, almost always immature software than prompt engineering because because of a lot of reasons. Maybe you're updating the prompt frequently. Maybe, again, different stakeholders are updating the prompt.

• 20:09Speaker 02

  That's why we encourage people to have their prompts in a C M S or we call it a prompt registry. You can put it in a postgressql database, something like that. But you don't want to block your prompt engineering cycle on edge deployments. And that's, I think, the symptom of this new thought pattern of how you're building, let's call it black box software, and how you're kind of thinking through these problems where they are different problem sets also. They're not, you're not solving the same type of software problems. You're solving, let's call it language problems or things that can employ this new type of software. And so it's not just a different like way to build, but it's also a different way to think about it.

• 20:51Speaker 02

  And then we can also get into like, how do you code now? How that's different and how I think maybe this is maybe a more contract. I don't, I don't, I don't like to predict things, but because I think it's kind of a fool's errand. But if I had to predict one thing, I think as a lazy programmer, there's something really nice about just having a little block of L M do something for me. So for example, parsing strings or parsing chunks of text and reordering it. You can do that deterministically and it'll probably be better. But there's probably a world where models are going to get cheap and quick enough where maybe it's worth the engineering time not to build it well and just outsource it to A I. But maybe that's a whole different tangent.

• 21:37Daniel Whitenack?

  Yeah, and maybe you could get into a little bit of the implications of these prompts in a registry, right? Because you could have like these sort of random tasks that I don't know if you all saw the Devon thing that's been going around where it's like a junior engineer agent that can, you know, write scripts and interact with software documentation. And today I was thinking about that, similar to what you were saying, Jared. It's like, hey, well, I could go and take all of these strings and go and interact with the API to translate them into another language, which is

• 22:20Daniel Whitenack?

  what the task was that I was doing. Or I could just like say, hey, write a script that does this and do it for me. So there's those random things. But then there's another type of prompt, which as soon as you're starting to expose a system to end users, then small changes in the prompt like on the backend could produce very different changes in the behavior to your actual end users and cause like actual, you know, problems. And so could you talk a little bit about maybe the implications of changes in your prompts and like best practices that you've found around? I know we're all talking a lot about prompt versioning and registries around those prompts. And I know that prompt layer is thinking about more than that around evaluation and other things.

• 23:05Daniel Whitenack?

  But yeah, maybe before we go to those other things, could you talk about what you've seen in terms of how people are managing these different prompts, some that have very low risk in terms of changing them very often, and maybe some that have actually a large risk in terms of maybe small changes in the prompt?

• 23:23Speaker 02

  Yes, I think there's a life cycle here for how you care about your prompt. And I think every prompt in any mature product and any, if you have a product, an AI product that's making your company $10 million a year or something like that. You're gonna care about any change to it. And I think there's that cycle. So let's say that's the end stage. At the beginning, you probably are just gonna ship any prompt. You're just gonna write a prompt. It's gonna kind of work. You're gonna try it once or twice. Say, all right, good enough. Let's get the MVP out. In that case, maybe the prompt is in your code. Maybe you don't care so much about what you were just saying about like updating, having a breaking change. All right, whatever. I gotta get it out. Let me get five people using it. All right, so you you've done that.

• 24:08Speaker 02

  What's next? You probably now have like five different prompts on your system. They're scattered everywhere. I was just talking to a founder who was visiting us, our office today, who was actually had this exact same story. And then he moved all his prompts to a text file. Or actually, in this case, it was just like a .ts file on his system. So now you have your prompts in one place. That's the next step, but still in your code base, it's still linked to Edge deploys. And like you said, you still have no way of knowing if you push a new one, what happened, like, is it breaking 20% of our use cases? That not great. But I would call this stage, I like this word for it, the like vibe-based prompt engineering.

• 24:50Speaker 02

  So this is the vibe-based prompt engineering stage where you're kind of writing a prompt. You're testing it in Playground, you're doing it once or twice, and you're just judging like, you're just looking, okay, yeah, it's pretty good. This lasts a little bit. The time when this is no longer good enough is usually when your product's either getting to a greater level maturity. Maybe you're rolling it out to GA. Or you're adding more stakeholders to the team. Maybe you're adding a PM, maybe you're adding a content writer or a subject matter, like we were talking about earlier, a psychologist or a lawyer or something like that. And you need more people involved. So now you have a non-technical person writing your prompts who isn't really capable of building out the whole dev workflow to test it out.

• 25:32Speaker 02

  You really wanna like make sure it doesn't break everything. So there's a Few things I've seen people, few strategies I've seen people employ here. One is, of course, traditional software, let's borrow some stuff, A B testing. Let's release it for some people. If we're monitoring user feedback, maybe if users are giving us a thumbs up, thumbs down, we should be able to see it pretty quickly. Also, having prod staging, dev staging, different like, we call them release labels, a lot of words for them. Let's call that category slow releases. So that's one way. Then there's two other big ways of solving it. Uh, second way, regression tests. So again, another concept borrowed from software engineering.

• 26:13Speaker 02

  Let's find cases where it's failing and see if we succeed in this test case. Third, which is also kind of like regression tests, I guess, is backtesting. Let's just run it on on old examples and see if it changes. I think in a lot of L M use cases, the really hard part about this problem is that you don't know what the ground truth is. Let's say we're making a summary. There's no correct answer to a summary. There's a good summary, a bad summary. It's almost very hard to understand if it is good or bad. We actually, there, there's a user of ours that is doing this exactly, and they're trying to figure this out. And they're using a combination of human graders and whatnot, but we can talk more about that in a second.

• 26:53Speaker 02

  But in this case, often the best thing to do is just rerun it on old responses and see how much changed. And, oh, I updated the prompt and 50% of my responses changed. Hmm, maybe I should look into those. Only like one out of a thousand changed, probably good enough. So it's all about, uh, trade offs in this world and everything's, again, it's a new way of thinking. It's non deterministic. So how do you trade off how much it's changing versus how much you need to make sure it doesn't change? Maybe you want to force a specific output that is deterministically graded. So for example, you're giving a JSON or a Boolean output. So there's a lot, there's a lot of strategies here.

• 27:35Speaker 02

  But, uh, if I were to kind of give it a one line answer, you should be deciding at what cadence you update prompts, by what stage your product is at, and how bad these issues are going to be.

• 28:07Speaker 04

  You know, when we started podcasting back in 2009 and online stores just the furthest thing from our minds, now we have merch.changelog.com and you can go there right now and order some t-shirts and that's all powered by Shopify. It's so easy, all because Shopify is amazing. Shopify is the global commerce platform that helps you sell at every stage of your business. From the launch your online shop stage to the first real life store stage, all the way to the did we just hit a million dollar stage. Shopify is there to help you grow. Whether you're selling security systems or marketing memory modules, Shopify helps you sell everywhere from their all-in-one ecommerce platform to their in-person POS

• 28:49Speaker 04

  system, wherever and whatever you're selling, Shopify has got you covered. Shopify helps you turn browsers into buyers with the internet's best converting checkout, up to 36% better compared to other leading commerce platforms, and sell more with less effort thanks to Shopify Magic, your AI-powered all-star. You know, nothing gets me and Jared more excited than when our guests get that coupon code in their email when their show ships, or to everyone out there who loves Changelog Podcasts and can go to merch.changelog.com and get your favorite threads to support our podcasts. It is just the best thing ever. From stickers to threads, all that is at merch.changelog.com. And did you know that Shopify powers 10% of

• 29:31Speaker 04

  all ecommerce in the US and Shopify is the global force behind Allbirds, Rothy's, and Brooklinen, and millions of other entrepreneurs of every size across 175 countries. Plus, Shopify's extensive help resources are there to support you and your success every step of the way. Because businesses that grow, grow with Shopify. Sign up for a $1 per month trial period at Shopify.com/PracticalAI, all lowercase, go to Shopify.

• 30:03Speaker 04

  Shopify.com slash Practical AI now to grow your business no matter what stage you're in. Again, Shopify.com slash Practical AI.

• 30:34Chris Benson?

  If you're looking at like a large, out on the edge, system of systems in the sense of you have a number of models deployed and they all do specific things and some of them are generative and some are not. And for the generative ones, they may be trying to address very specific functions that they're doing. With a system like that, you've got it in production and maybe you've done kind of that minimal viable product approach on getting it up. But when you get things to where they're kind of stable in production, you are starting to kind of address some of that. I know that I'm grappling in my own head with like how to think about being able to make those tweaks and changes to prompts in any given model in the system

• 31:17Chris Benson?

  and detect that. Is there like a best place for me to start? Because I'm still trying to kind of grapple with the larger picture and really understand it. And so if I want to change something, but I don't want to impact the larger stable system, what would you kind of be like, what if it was you in that position, like one, two, three, try this, try that, try that, just to give me a good hands-on takeaway from that. And I apologize for the selfish nature of the question, but it's hard to do that.

• 31:45Speaker 02

  No, I think it's good to have the selfish type of question here because one thing that I actually think a lot of people get wrong in this space is that every prompt is kind of different. And the answer to this question is really very unique to what you're actually trying to do and the task you're trying to solve. There isn't really I mean, a lot of people are trying to sell it. So I like to say that maybe I'll change my opinion in a month or a week or a day.

• 32:14Chris Benson?

  I do that all the time as I learn more. So no worries. You're allowed to.

• 32:18Speaker 02

  It's good to change your opinion, but uh, right now, I think a lot of these eval sets that people produce are not that useful for building real products because you're trying to evaluate your prompt for your real application, not for some pie in the sky, financial data set or something like that. Having said that, I think the first question to ask, and maybe we'll use this example. So I would modularize it. I would think about it on the prompt level. Uh, so I guess there's two ways to think about it. We could think about modular tests and then end to end tests. We should be doing both. For this case, do we have a ground truth? I think is the first question I'd ask. Is there a ability to make a data set with ground truths that we can compare to?

• 33:02Speaker 02

  Or is it like a summary type example where there's no answer?

• 33:05Chris Benson?

  In the case that I'm looking at, I think you could establish ground truth. I don't know it would be easy, but you probably could. Because you could bypass what you're seeing and you could have a, a human kind of assess what the generative AI was, model was trying to assess as well. So you could get a ground truth that is a human analysis of it as a proxy.

• 33:27Speaker 02

  Excellent. So your life is now 10 times easier.

• 33:31Chris Benson?

  That's always good.

• 33:33Speaker 02

  It is great. Yes. Step one is uh, whether you do it yourself, whether you hire some people on M Turk or QA or however you do it. Step one is kind of build, it doesn't have to be big, Bill. Build a small data set and try to get into this method of test driven prompting or eval driven prompt engineering. I don't know if we have a word for it yet. Maybe we need to define one, but try to build some sort of metric we can evaluate our test on. So you don't have to be just one by one trying these examples out. So what I mean by this is build a, let's say 10, let's say 15, let's say 100, whatever it is,

• 34:13Speaker 02

  depending on the use case of input variables to your prompt. So your prompt is probably having, hey, your task is to do this. Here's this data, blah, blah. So the data is an input variable in that case. And then get a human to give you the output. And then start every time you test the prompt, let's run it on that over time, we can. Let me, I could talk about over time, how to make that better, but does that make sense?

• 34:37Chris Benson?

  It does, it does.

• 34:38Speaker 02

  Thing I'd add is over time, how you make that better is you start connecting that back with real data and how your users use it. So again, you can make your life 10x easier. If you have good user feedback and a way to know if the production inference, if the production LLM run actually worked or not. So user feedback's a way to do that. Say a user gives you a thumbs up, thumbs down. Now you can take all those thumbs ups, thumbs down. Make a new data set out of that and now, now you're really going and now you're building this whole feedback loop. And I think I'll say like our biggest goal with prompt layer, our MO

• 35:18Speaker 02

  is to shorten the prompt engineering feedback loop. I think that's what everything boils down to in this world.

• 35:24Daniel Whitenack?

  Maybe along with that element of feedback, I'm wondering if you can talk a little bit because we've talked a lot about evaluation, prompt versioning. There's the other element of this, which I know you all are thinking about deeply, which is sort of logging and monitoring. And there's certainly cases where, oh, I have this chain of LLM processing or even loops that could happen like, you know, LLM as a judge or something that kind of or critic kind of elements of LLM prompts that actually could loop until something happens or a certain number of times. And the way that you develop your prompts, both in terms of their length, in terms of how effective they are, could drastically

• 36:10Daniel Whitenack?

  impact your latency of processing. It could impact your cost in terms of how much text you're putting into models, especially if they're charging you for how much text you're putting in. So yeah, could you talk a little bit about maybe the highlights of some kind of best practices around logging and monitoring and how you think about that at prompt layer?

• 36:33Speaker 02

  Yes. So I think, not to sound like a broken record here, but

• 36:39Speaker 02

  The thing I like to, or I go back to a lot is, everything is use case dependent. So you brought up that some people are very concerned about latency, some are very concerned about cost. I know teams that are concerned about neither of those, and their only concern is, are we getting the right answer? For example, code generation type startups. A lot of times latency doesn't matter. You're giving them a task. Hey, again, in our case, a prompt layer, hey, can you? We worked with a company called Grit, where we said, can you move our whole code base into TypeScript? It could take a week, I don't care. So latency and cost don't matter then. But then there's a lot of other cases where, most cases probably latency and cost matter.

• 37:20Speaker 02

  So in that case, or in either case, why logging is important here is just debugging. Honestly, I'll be honest, logging and observability is kind of the most boring part of our platform. Not because it's not useful, but because it's obvious. I think we started with observability. This is table stakes to shortening that feedback loop and that high level goal. This is table stakes because you need to collect the data. You need to collect the data to build these evals, to see when it's not working. To be able to triage issues, say someone, one of your users tells you, hey, I got a weird error. This happened to me, actually. I was using Superhuman AI, and it didn't work. And I told them about it. I said, I got like a weird output, maybe you guys want to debug it?

• 38:05Speaker 02

  And they asked me what prompt I gave it to do the output. I don't remember. Don't you have a logging system? Maybe you should use prompt layer. And they, but yeah, you should be able to figure out why something broke. You should be able to step through, step by step into the chain, see which version of the prompt it used. Maybe you have multiple versions in production. And our logging is just, logs each request and lets you integrate it with metadata like user information.

• 38:33Daniel Whitenack?

  It's one thing to like log a bunch of things, but like, let's say that I want to improve latency or cost or something like that. And I maybe have like 17 different prompts that I'm using across my system. How have you learned how to present that information to users so that they can kind of, especially from a, you mentioned kind of the skill of prompt engineering, having this algorithmic thinking kind of piece to it. But there's also a lot of people, you know, there's coming in, maybe that's the part of their brain that they're building up and they're bringing in these other skills with them.

• 39:15Daniel Whitenack?

  So how have you found it useful to present this sort of information to people, to give them the right sorts of feedback along their kind of journey of optimizing things?

• 39:24Speaker 02

  So I think latency and cost are the easiest things to figure out how to, they're the easiest metrics you get out of the box when you're doing prompt engineering, because you're always getting latency and cost. The harder metrics is, is my answer correct? Is it rude to me? Is it mean? Is it, does it have API? I don't know. Microsoft Bing releasing, we have a, we did a prompt engineering tournament last night. The first round was, can you avoid a PR disaster like Microsoft Bing? But that's the hard part is making sure your answer doesn't go off the rails or isn't wrong.

• 40:03Speaker 02

  But latency and cost and those type of logings, like base level logging, specific things, those are the tractable metrics. So we give you latency and cost for each prompt template. We give you it broken down based on version. And then we also have a full analytics page that actually we revamped the other week for one of our customers who, they were going to have to build that their whole, like a whole BI dashboard, because I think something about their, either the founders of the company or the investors were worried about some users spending too many credits. So we kind of revamped our analytics page to just save them some time there. So you can use our analytics page to see which prompt templates are costing you the

• 40:43Speaker 02

  most, which users are costing you the most, maybe if you're segmenting things to prod, maybe you're segmenting based on geo and just kind of filtering down based on that sort of thing.

• 40:54Chris Benson?

  I want to ask a question. As you have kind of pioneered this whole space, jumping into prompt engineering, quite honestly, before anyone really knew what it was, as you pointed out earlier, and you've been building out this capability. As you look to the future and you know, the future is changing so rapidly right now, you know, we're all in this massive acceleration of things coming out, you know, Daniel and I every week are trying to figure out of all the things happening, what do we actually talk about, you know, it's it's getting harder, whereas it used to be there was something that happened last week and we'll talk about it. Now there's so many. As you're operating a business in this, um, you know, kind of intense increasing environment,

• 41:34Chris Benson?

  where do you think this is going? Uh, from a prompt engineering standpoint, what will prompt engineering become as we become increasingly multimodal and you know, all the fantastic things that are happening on a weekly basis? I would imagine that would be fairly hard to try to plan ahead uh, on, you know, where the industry is going and the technology and where to put your business. How do you see the future? What's the next one year, two year, five years in your head look like?

• 42:00Speaker 02

  That's a billion dollar question, right? Um, I think we try to do two things. We try to not predict the future because it's too hard and we try to build something useful that is built on first principles that make sense. And I think that's how we try to stay ahead of the curve there a little bit. So I can give you some examples. So for example, kind of just the whole process of iterating, of testing for evals. We procrastinated a little bit on building that part of our platform. We always needed, knew we needed it. It's been the buzzword in the industry for like six months now, but we really wanted to know how to build that correctly. And I think

• 42:43Speaker 02

  we spent a lot of time talking to a lot of teams and saying, how do you do evals today? And every team we spoke to did it in their own way, in a Google spreadsheet, building out some weird, unique things. So our eval product, if you try it out, it looks like a spreadsheet for that reason. And it's very much inspired by the robustness of a Microsoft Excel type product where it seems very simple, but you can take it in a lot of different ways. So I think we are trying to Become future-proof by avoiding taking strong opinionated wins. We want to support best practices and build best practices for the community, especially in a space like this. But we want to do it without pigeonholing people into different ways of doing things.

• 43:28Speaker 02

  And I think it's been funny seeing how people like the Hive Mind has changed their opinion on what the future is. I remember we were, a year ago, we were talking to investors, obviously not the investors that are on our team right now. And there were investors like, oh, prompt engineering, AGI is just going to take over. We're not going to have any infrastructure anymore. And a lot, a lot had that opinion. I don't think many have that opinion anymore, let's just say. And I think it's very obvious to us, and it's been obvious to us, that prompt engineering is the process of giving inputs to the LLM and choosing which model you're using. And even with the most advanced LLM ever, let's say the LLM is as advanced as a human.

• 44:13Speaker 02

  You still have to tell a human what you want. You still have to tell the intern what task you want him to do. And that's prompt engineering. And there's always going to be a process of inputs there. So that's how we think about the future and lack thereof.

• 44:29Daniel Whitenack?

  Jared, thank you so much for coming on to share your insights today. We definitely appreciate that you and the team at PromptLayer are thinking deeply about these things and building really good tools to support the community. And I encourage everyone in the audience to check out the show notes, follow the links, find out more about PromptLayer and the cool stuff that they're doing. And I hope we can have you back on the show in another year, when I'm sure prompt engineering will look very different than it does now. But thank you so much for joining, Jared. It's been a pleasure.

• 45:02Speaker 02

  Yes, thank you for having me. This has been fun.

• 45:12Speaker 00

  All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to PracticalAI.fm for all the ways. And join our free Slack team where you can hang out with Daniel, Chris, and the entire changelog community. Sign up today at PracticalAI.fm/community. Thanks again to our partners at Fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.