The new AI app stack

Recently a16z released a diagram showing the “Emerging Architectures for LLM Applications.” In this episode, we expand on things covered in that diagram to a more general mental model for the new AI app stack. We cover a variety of things from model “middleware” for caching and control to app orchestration. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Emerging Architectures for LLM Applications https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/ Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another fully connected episode of Practical AI. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll cover some of the latest news and we'll cover some learning resources that'll help you level up your machine learning game. I'm Daniel Whitenack. I'm the founder of Prediction Guard and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 1:12Chris Benson?

  Doing very well today, Daniel. How are you?

• 1:14Daniel Whitenack?

  I'm doing great. I am uncharacteristically joining this episode from the lobby of Hampton Inn in Nashville, Tennessee. So if our listeners hear any background noise, they know what that is.

• 1:29Chris Benson?

  You have a built-in audience right there.

• 1:31Daniel Whitenack?

  Built-in audience. The people in this lobby are unexpectedly learning about AI today, which I'm happy to do. Yeah, out here visiting a customer onsite. And yeah, it's nice to sit back and take a break from that and talk about all the cool stuff going on.

• 1:51Chris Benson?

  Excellent. Well, I'll tell you what, you know, we have had so many questions and about kind of sorting out all the things that have happened the last few months and over the last year. And we've done a couple of episodes where we're trying to kind of clear out like generative AI, what's in it, what LLMs are, how they relate and stuff like that. What do you think about taking a little bit of a deep dive into large language models? And kind of all the things that make them up? Because there's a lot of lingo being hurled about these days.

• 2:23Daniel Whitenack?

  Yeah, yeah. I think maybe even outside of LLMs, there's this perception that the model, whether it be for image generation, or video generation, or language generation, that the model is the application. So when you are creating value, the sort of model, whether that be, you know, Llama 2, or Stable Diffusion XL, or whatever, that somehow the model is, is the application. Like, it's providing the functionality that your users want. And that's basically a falsehood, I would say. And there's this whole ecosystem of tooling that's developing around this.

• 3:07Daniel Whitenack?

  And one of the things that I sent you recently, which I think does a good job at illustrating some of the various things that are part of this new ecosystem, or this new generative AI app stack, was created by Andreessen Horowitz. They created a figure that's like emerging LLM app stack. We'll link it in our show notes. I think it goes, though, maybe more generally than LLMs. And, but that provides maybe a nice framework to talk through some of these things. Now, of course, they're providing their own look at this stack, especially because they're invested in many of the companies that they highlight on the stack.

• 3:50Daniel Whitenack?

  But I think regardless of that, they're trying to help people understand how some of these things fit together. Have you seen this picture?

• 3:57Chris Benson?

  I have, and I appreciated when you pointed it out a while back. There, it definitely is an interesting. I haven't seen anything quite like it in terms of putting it together. And some things they seem to dive into more than others in the chart. It will be interesting to see how we parse it going forward here.

• 4:14Daniel Whitenack?

  Yeah, and maybe we could just take some of these categories and talk them through in terms of the terminology that's used and how they fit into the overall ecosystem. So, you know, we can take an easy example here, which is one of the things that they call out, which is playground. Now, I think this is probably the place where many people start their generative AI journey, let's say. So they either go to, I think, within the playground category, there would be like, ChatGPT might, might fit in that category where you're prompting a model, it's interactive,

• 4:55Daniel Whitenack?

  it's a UI, like you can put stuff in, you can put in a prompt and get an output, right? Now, ChatGPT is maybe a little bit more than that because there's a chat thread and all of that. But there's other playgrounds as well, so you could think of spaces on Hugging Face that allow you to use stable diffusion or allow you to use other types of models. There's other proprietary kind of playgrounds that are either part of a product or are their own products. So OpenAI has their own playground within their platform. You can log in and try out your prompts. There's nat.dev, which is a cool one that kind of allows you to compare one model to the other.

• 5:39Daniel Whitenack?

  There's other products, like I would say something like ClipDrop, which is a tool that lets you use stable diffusion. And you can just go there, there's, you can try out prompts for free. You can pay up if you need to use it more. So there's a limit to that, but there's a lot of these playgrounds floating around and that's often where people start things.

• 6:01Chris Benson?

  It's funny, the playground itself as, as a category has a lot of subcategories, I think, to it because, you know, you've already kind of called out kind of the diversity of what you've, you know, in the cloud providers, for instance, all the big cloud providers have their own playground areas. Nvidia has a playground area. I think it's almost becoming a ubiquitous notion. And of course, all those playground areas for the commercial entities are focused on their products and services, definitely, but trying to, you know, trying to bring some cool factor to it, so.

• 6:31Daniel Whitenack?

  Yeah, it's almost like a demo or experimentation interface. If we define this playground category, it's usually, but not always, a browser-based playground or a browser-based interface where you can try to prompt a model and see what the output is like. I think that would kind of generally be true. Maybe there's some caveats to certain ones, like Midjourney, for example, is a Discord bot, or there's still a Discord bot that you could use. Maybe that fits into the playground. But generally, these are interactive and useful for experimentation, but not necessarily useful to like build an application.

• 7:15Chris Benson?

  Yeah, I agree. And another thing to note about it from a characteristic standpoint is not only is it really, it's not made for you to go build your own thing. It's made for you to try the kind of the stuff of whatever organization is doing, but they do do it, they provide the resources. So by being in a browser, you don't have to have a GPU on your laptop.

• 7:37Daniel Whitenack?

  You don't have to have resources. Yeah.

• 7:38Chris Benson?

  Yeah, you don't have to have all the things. Through various means, they set up all that for you on the backend, whether it be just calling a service, or whether it be creating a temporary environment through virtualization. But it is a good way to either to test out a new product line, or to, or to just get your toes wet a little bit. If you want to try some stuff out, maybe you've been listening to the Practical AI Podcast for a little while, and you want to, a particular topic grabs you. That would be a good place to go.

• 8:06Daniel Whitenack?

  Yeah, and I think within that same vein, you could transition to talk about this other category, which is not unique to the generative AI app stack, let's call it. But it's still part of the stack, which they have called out app hosting. So that's like very generic, right? So in here would fit things like Vercel, or I would say, you know, generally, like the cloud providers, right? And the various ways that you can host things, whether that be in Amazon with ECS, or AppRunner, or whatever that is, or in Even your own infrastructure, your own on-prem infrastructure, if you host things.

• 8:46Daniel Whitenack?

  Now, there are, I would say, a number of hosting providers that are kind of cool and trendy and people that are building new AI apps, they seem to gravitate towards like, let's say, Vercel. And a lot of front-end developers that use Vercel, which I think it's an amazing platform, so cool. That hasn't traditionally been like a data sciency hosting way of doing things, but it represents, I think, this new wave of application developers that are developing applications, integrating AI, and you see some of those now kind of coming into or being exposed in this kind of wider app stack.

• 9:29Chris Benson?

  Which is a good thing, because we've talked for a long time, even as we opened this conversation up, saying the model is not the app, you know, you have to wrap the model with some goodness to get the value out of it, to be productive with it. And so I personally like the fact that we're seeing the model hosting and the app hosting are starting to merge, because I think that's more manageable over time. It's less being in its own special category and it's more about, okay, every app in the future is going to have models in it. And so, you know, we're accommodating that notion. So I like seeing it go there. I've been waiting for that for a while.

• 10:05Daniel Whitenack?

  Yeah, and to really clarify and define things, you could kind of think about like the playground that we talked about as an app that has been developed by these different people that illustrates some LLM functionality, but it's usually not the app that you're going to build. You're going to build another app that is exposed to your users that uses the functionality, and you'll need to host that either in ways that people have been hosting things for a long time or new, interesting patterns that are popping up, like things that Modal is doing, or maybe things that front-end developers really like to use, like Vercel and other things. But there's still that app hosting side.

• 10:45Daniel Whitenack?

  Now, where I think things get interesting is you have the playground, you have the app hosting. But regardless of both of those, what happens under the hood? And this is, I think, where things get quite interesting and where there's a lot of differences in the kind of emerging generative AI stack compared to the maybe more traditional non-AI stack. In the middle of the diagram that we're talking about, this emerging LLM app stack diagram, which I think also is, again, more general, is this layer of orchestration. So I don't know about you, Chris, but I'm old enough, I guess, you don't have to be that old,

• 11:30Daniel Whitenack?

  I don't think, to, when someone says orchestration, I think of like Kubernetes or like container orchestration.

• 11:37Speaker 03

  Yeah.

• 11:38Daniel Whitenack?

  Maybe that's my own bias coming from working in a few microservices oriented startups and that sort of thing. But this is distinctly not the orchestration that's being called out here. In the generative AI app stack, there's a level of orchestration, which in some of my workshops, I've been kind of referring to as almost like a convenience layer. Think about like when you're interacting with a model. Let's give a really concrete example. Let's say I want to do question and answer with an LLM. I need to somehow get a context for answering the question. I need to insert the question and that context into a prompt.

• 12:23Daniel Whitenack?

  And then I need to send that prompt to a model. I need to get the result back and maybe do some like cleanup on it. Like I have some stop tokens or I want it, you know, to end at a certain punctuation mark or whatever that is. That's all convenience. What I would consider sort of this convenience and what they're calling orchestration around the call to the model. And so this orchestration layer, I think, has to do with prompt templates, generating prompts, chains of prompts, agents.

• 13:01Daniel Whitenack?

  Uh, plugging in data sources like plugins. These are all things that kind of circle around your AI calls, but aren't the AI model.

• 13:13Chris Benson?

  Yeah, I mean, it's the software around it, you know, just to simplify a little bit.

• 13:17Daniel Whitenack?

  Yeah, and maybe tooling.

• 13:19Chris Benson?

  Yeah.

• 13:19Daniel Whitenack?

  Orchestration tooling, yeah.

• 13:21Chris Benson?

  Yeah, it's the stuff you have to wrap the model with to make it usable in a productive sense. And from the moment that I saw that word, that was almost the very first thing that grabbed me. You know, those, you know, little psychological quirk where you kind of notice the thing that sticks out?

• 13:35Daniel Whitenack?

  Yeah.

• 13:35Chris Benson?

  That's the thing that stuck out was they, it's a big bucket that they're calling orchestration, which is a loaded word that can mean a lot of different things depending on what it is you're trying to do. And the examples that they list in that category are all somewhat diverse as well. I think that was the first point where I thought, well, it's a chart with the creator has a, has a bias there. What are some of the ways, I'm just curious, when we think about this kind of orchestration, as they say, wrapping around and providing the convenience? Any ways that you would break that up, like how you think about it? You mentioned convenience and stuff, but they go from something like Python as a programming language

• 14:15Chris Benson?

  to Langchain, to ChatGPT, all three very distinct kinds of, of entities.

• 14:21Daniel Whitenack?

  Yeah, I think that you're kind of seeing a number of things happen here. The first one that they call out is Python slash DIY. So you're seeing a lot of roll your own kind of convenience functionality built up around LLMs. But I do think one of the big players here would be like Langchain and what they're doing. Cause if you look again at those kind of layers of what's available there, you have maybe categories that I would call out. If we just take Langchain as an example, categories that I would call out of this sort of orchestration functionality would be templating. So this would be like prompt templates, for example, or templating in terms of chains.

• 15:06Daniel Whitenack?

  So manually setting up a chain of things that can be called in one call. There's also an automation component of it. Maybe the, this is a way that orchestration kind of fits with the older way the orchestration term is used in like DevOps and other things where some of it could be automation related to with things like agents or something like that, where you have an agent that automates certain functionality. It's not the LLM itself, but it's really automations around calling the LLMs or the other generative AI models to generate an image or what have you.

• 15:50Chris Benson?

  They also kind of have some separate callouts, you know, for APIs and plugins. And then they have, which we can hit in a moment, they kind of have a collection of the, the maintenance items, you know, the things to keep the lights on, if you will, logging and caching and, and things like that. How do you look at that breakdown, the way they have it?

• 16:10Daniel Whitenack?

  Yeah, so I think this is where they kind of have the orchestration piece in the middle there as connecting a couple of different things. One of those would be what I would consider, I think, more on the data or resource side. And then one is more on the model side. So I think we could split it into those two major categories. So what are you orchestrating when you're orchestrating something with LangChain or similar? Well, you're orchestrating connections to resources. I'll use the term resources because it might not be data per se. It might be like you say, like an API or another platform like Zapier or, you know,

• 16:54Daniel Whitenack?

  Wolfram Alpha, something like that. The other side of that is the model side, both the model hosting and some really useful tooling around that. But let's start on the resource side. So as you mentioned, you might orchestrate things like, one of the things that I've found both really fun to do and useful is to orchestrate calls into like a Google search. So if I want to pull in some context on the fly, then I might do, want to do a Google search. That's a call to an API. So that's a resource or a plugin that might be conveniently integrated into your orchestration layer,

• 17:35Daniel Whitenack?

  either via something like LangChain or via your own DIY code. Another side of this would be the actual data and the data pipelines, which are your own data or data that you've gathered or is relevant to your problem. So again, if we're thinking about this sort of set of resources that could be orchestrated into your app, maybe you have a set of documentation that you want to generate answers to questions out of. Or maybe you have a bunch of images that you want to use to fine tune stable diffusion or something like that. Having data and integrating it into models isn't new. And so the things that are called out

• 18:20Daniel Whitenack?

  in this particular image, like data pipelines, those are also not new and are part of this app stack if you're integrating your own data. So things like Databricks or Airflow or Pachyderm or tools to parse data. So PDF parsers or unstructured data parsers or image parsers or image resizing or all of that sort of stuff still fits into the data pipelining piece. And so you've either got your data coming from APIs, which might be a resource that you're orchestrating, or you've got your data coming from your data sources, which might be traditional data sources of any type,

• 19:04Daniel Whitenack?

  from databases to unstructured data.

• 19:13Speaker 00

  This is a changelog news break. It's official. Advancements in computer vision have rendered CAPTCHA's obsolete as new research shows AI bots are 15% more accurate than humans at picking which images have a bridge, or a sign, or a bicycle, or whatever in them. The researchers recruited 1,400 participants to test websites that use CAPTCHA puzzles, which account for 120 of the world's 200 most popular websites. The bots' accuracy ranges from 85 to 100%, with the majority above 96%. Meanwhile, we mere mortals check in at a pathetic 50 to 85% accuracy, and we answer slower than the robots to add insult to injury.

• 19:58Speaker 00

  I've surmised this for months now, as we've been unable to ward off spam account creations on changelog.com, no matter which shiny new CAPTCHA service we tried. There are other efforts in the works besides CAPTCHA in order to differentiate between robots and humans, but so far, the robots are winning. You just heard one of our five top stories from Monday's changelog news. Subscribe to the podcast to get all of the week's top stories, and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 20:43Daniel Whitenack?

  Well, Chris, part of the data piece, or the resource piece, that is kind of unique within this new generative AI app stack is the embedding and the vector database piece. And I have to say, I've just got to recommend that our listeners, if they haven't listened to our very recent episode about vector databases, because that episode goes into way more depth in terms of what a vector database is, and why people are using it, but just for a quick recap. Part of what you might want to do with generative AI models is find relevant data

• 21:28Daniel Whitenack?

  that's relevant to a user query and somehow orchestrate that into your LLM calls either for chat or question answering, or maybe even into image generation or video generation. In order to find relevant data, what people have found is that they would like to do a vector or an embedding search on their own data to find relevant data. And again, you can find out much more about that in our previous episode, but that's called out in this app stack as probably something unique that's developing, which is not just having data pipelines and databases, but having data flow through an embedding model

• 22:11Daniel Whitenack?

  and into a vector database where you're performing semantic searches.

• 22:17Chris Benson?

  I mean, at the end of the day, it's a database that's very, it works well for the kind of operation that we're doing here. Whereas some of the traditional things that we had been working on for years before, there's kind of a context shifting in terms of how you're handling data, what data is, how it's organized. So this makes a lot more sense.

• 22:35Daniel Whitenack?

  Yeah, and it should call out here too, part of the stack here, and I'm glad that they called it out in this way in the schematic that we're looking at, is the embedding model. So a lot of people are talking about these vector databases, but in order to store a vector in a vector database, there is a very relevant component to this stack, which is the actual model that you're using to create embeddings. And not all are created equal. So think about if you're working on an image problem, right? You may use a pre-trained

• 23:14Daniel Whitenack?

  feature extractor type model from hugging face to extract vectors that your images, so put in an image and get a vector out. But if you're working with both image and text, for example, maybe you're going to use something like Clip or one of Those are related model that's able to embed both images and text in a similar semantic space. But if you're only using text, there's a whole bunch of, of course, choices. And all of those don't perform equally for different types of tasks as well. There's, if you search on Hugging Face or just do a Google search for a Hugging Face embeddings leaderboard,

• 23:58Daniel Whitenack?

  there's actually a separate leaderboard. So Hugging Face has a leaderboard for open models and how those score in various metrics. They also have a leaderboard for embeddings and you can click through the different tasks. Let's say you're doing retrieval tasks, like we're talking about here from a vector database. You can see which embeddings perform the best according to a variety of benchmarks in retrieval or in summarization or other things.

• 24:27Chris Benson?

  Do you use that a lot when you're putting models and storing them into vector databases and figuring out the embeddings, do you tend to go and see what is going on? Because right now there's so much happening in that space. Does that make for a good guidepost for you?

• 24:42Daniel Whitenack?

  Yeah, yeah. And I think what is also useful is looking at those performance metrics, but also at least on the Hugging Face leaderboard and some other leaderboards. So if you search for, if you're working with text, one of the major tools for creating these embeddings in a really useful way is called Sentence Transformers. And they have their own table where they have measured and benchmarked various embeddings that can be integrated in Sentence Transformers. That's useful, but it's also useful to look at the columns, whether you're looking in the Hugging Face leaderboard or the Sentence Transformers or wherever you're looking, at the size of the embedding and the speed of the embedding.

• 25:27Daniel Whitenack?

  Because it was called out when we had our vector database discussion, but only in passing. Let's say you want to embed, you know, 200,000 PDFs. So I just ran across this use case with some of the work that we're doing. And it can take a really, really, really, really long time.

• 25:47Daniel Whitenack?

  Depending on how you implement it to both parse and embed a significant number of PDFs. The same would be true for documents or other text or other types of data even. And so when you're looking at that, there's two implications here. One is how fast am I able to generate these embeddings? Do I have to use a GPU or can I use a CPU? Cause there's going to be a different speed on GPUs versus CPUs. And how big are the embeddings? This is another kind of interesting piece, which is if I have got embeddings that are thousand or more in dimension,

• 26:25Daniel Whitenack?

  that's going to take up a lot more room in my database and on disk than embeddings that are 256 or something like that. So there's also storage and moving around data implications to how you choose this embedding space. So there's a lot of, I think, practical things that maybe people skip over here when they're just doing a prototype with Langchain and some vector database. It's easy, but then as soon as you try to put all your data in, it gets much harder.

• 26:59Chris Benson?

  You raise a question in my mind and I'm going to throw it out. You may or may not be familiar with what the answer would be. But when you're looking at vector databases and you're looking at all this, you know, the diversity in embedding possibilities here. And the fact that that has kind of physical layer consequences, you know, in terms of storage and stuff like that. Are we seeing that in vector or other database arenas where they're trying to accommodate this new approach to capturing data? In terms of having embeddings, the way a vector, you know, with the rise of vector databases. It seems that there would be a whole lot of kind of vendor related research on how you do that. Because to your point a moment ago, you're talking about data at such a volume

• 27:45Chris Benson?

  that poor architecture in terms of what's under the hood could have some pretty big consequences there.

• 27:51Daniel Whitenack?

  Yeah, I think that's definitely true. And there was a point that was made in one of our last episodes that the vendors for these things are having different priorities that don't always align. So some are optimizing for how much data, how quickly you can get a large amount of data in. But maybe they're not as optimized for the query speed. Some are optimizing for query speed, but it might be really slow to get data in. And so that's one piece of it. I think another piece of it is, how large of an embedding do you need? And how complicated is your retrieval problem? Right? I would recommend that people do some

• 28:35Daniel Whitenack?

  testing around this because let's say you have 100,000 documents that are very, very similar one to another, or 100,000 images that are very, very similar to one another. And the retrieval problem is actually semantically very difficult. You might need a larger embedding and more kind of power, even like optimization around the query, like reranking and other things to get the data that you need. Whereas if you have, you know, 100,000 images and they're all fairly different, well, maybe you don't need to go to some of those links. So yeah, I think that that's also part of this problem. And people are still feeling

• 29:19Daniel Whitenack?

  out the best practices around some of this, partially because it's this kind of new part of the AI stack, and partially because things are constantly updating as well. So if you use this embedding today, there's a better one tomorrow. And vector databases are updating all the time. So it's also just a very dynamic time here.

• 29:41Chris Benson?

  As we look at the chart here, and there's kind of the three that we referred to earlier that are that are kind of together, and those are LLM cache, logging slash LLM ops, and validation. First of all, could you kind of describe what's encompassed in each of those? And also, kind of, why are they fit together? Why are we seeing those lumped in one category here, one super category?

• 30:05Daniel Whitenack?

  Yeah, so if you think about what we've talked about so far, there's this new generative AI stack, whether you're doing images, or language, or whatever, there's an application side, which might just be the playground, or it might be your own application. There's a data and resources side, which is what we've talked about with integrating APIs and data sources. And then there's like a third arm here, which is the model side. And all of those are kind of connected through the orchestration layer, or the automation layer, or the convenience layer, whatever you want to end up calling that. So on, now we're kind of going to this third arm of the model side. And we can come back to it here in a second, but one side

• 30:49Daniel Whitenack?

  of this is just hosting the models and having an API around them, which we can come back to. But between the model and your orchestration layer, almost as, maybe we could call it like model middleware. I'll just go ahead, coin that. I just coined it on, maybe people are already referring to it that way, and I didn't coin it, but model middleware sits kind of either wrapping around, or in between your orchestration layer and your model hosting. And these are the things that you're referring to around caching, logging, validation. Probably the one that people are most familiar with, if they are familiar with one of these, would

• 31:29Daniel Whitenack?

  be the logging layer, which is, again, something that is kind of a DevOps-y infrastructure term. But here, we might think of very specific type of logging, like model logging, which might be more natively supported in things like weights and biases, or clear ML.

• 31:52Daniel Whitenack?

  Or these other kind of ML ops type of solutions where you're logging requests that are coming in, prompts that are being provided, response time, GPU usage, all the kind of model related things. And you want to put those into graphs and other things. So there may be specific kinds of logging. So how quickly on average is my model responding? What is the latency between making a prompt or a request and getting a response? How much GPU usage is my model using? And do I need more replicas of that model?

• 32:35Daniel Whitenack?

  These sorts of things can be really helpful as you're putting things into production. So that's a first of these middleware layers.

• 33:00Daniel Whitenack?

  So Chris, the other middleware layers, I would say, that have been called out, at least in what we're looking at is our validation and caching. So I'll talk about caching and then we can talk about validation a little bit, which is close to my heart. But caching, let's say that, again, this already happens in a lot of different applications. So think about like a general API application. If someone makes a request for data in your database and you retrieve that data, and then the next user asks for the same data in your database, the proper and smart thing to do would not be to do two retrievals, right?

• 33:41Daniel Whitenack?

  But to cache that data in the application layer in memory so that you can respond very quickly and reduce the number of times that you're reaching out to your database and things like that.

• 33:52Chris Benson?

  I noticed in this chart that some of the examples that they put for caching, such as Redis and SQLite and such, are very typical and long-term players in the app dev world.

• 34:03Daniel Whitenack?

  Yep.

• 34:03Chris Benson?

  So does that beg the question, or at least for me, begging the question, that when you're caching, like you're really talking about for the input, here's an output, whether it goes to a model or not. Is it, is it really just application data that you're caching at that point?

• 34:19Daniel Whitenack?

  So it's caching in that sense, but I think there's maybe implications to it that go beyond kind of normal caching. So if you, you know, running AI models is expensive most of the time because you have to run them on some type of specialized hardware, right? If I've got a model running on two A100s, right? I would rather not have four replicas of that model. I would rather just have one if I can, because I don't want to pay for all those GPUs. So part of it is really related to cost and performance. So it's also for a large model. This is mainly for large models, I would say.

• 35:04Daniel Whitenack?

  You've got a lot of cost, either because you're running that model on really specialized hardware, or because like if I'm calling out to GPT-4, it's really expensive to do a lot of requests to GPT-4, right? So in order to deal with that, if you have a prompt input, you can cache that prompt. And if users are asking the same question, I would rather just send them back the same response from GPT-4 or my large Llama 2 70 billion model, or whatever it is. I'm going to respond to them the same way, based on the same or a similar input.

• 35:44Daniel Whitenack?

  The other implication to this, which in my mind, it sort of fits into caching, but maybe not in the traditional sense. So I normally think of caching as like, oh, I'm going to cache things in memory or locally at the application layer. But if you're caching prompts and responses, there is a real opportunity to leverage that data to build your own sort of competitive moat with.

• 36:15Daniel Whitenack?

  Your specific generative AI application. So for example, like you've got a user base. They're prompting all of these sorts of things. All of a sudden, if you're saving all of that data and the responses that you're giving, you're essentially starting to form your own domain-specific data set that you could kind of leverage in a very competitive way in kind of two senses. One is, right now, if you're using a really expensive model to make those responses, maybe you start saving those responses from the really expensive model and you can use that data to fine-tune a smaller model that might be more performant and cost-effective in the long term.

• 36:59Daniel Whitenack?

  So it's an operational kind of play. The other way is, if you're gathering that over time and you actually have the resources to human-label that or give your own human preferences on that or certain annotations on that, that now is your own kind of advantage in fine-tuning either one of these generative models or your own internal model for the domain that you're working in. So it's caching, but that's almost like a feedback or data curation side of things as well.

• 37:35Chris Benson?

  So you mentioned earlier that validation was close to your heart.

• 37:40Daniel Whitenack?

  Yeah, so as our users know, I think part of the tooling that I'm building with Prediction Guard would fit into this category. It would actually span, I think, more categories. It kind of spanned between validation and orchestration and model hosting. So there's kind of a little bit of overlap there. But this validation layer really has to do with the fact that generative AI models across the board, I think people would say are There's a lot of concerns around reliability, privacy, security, compliance, what have you. And so there's a rising number of tools that are addressing some or all of those issues. So whether it be putting controls on the output of your LLM.

• 38:25Daniel Whitenack?

  Again, think about this like a middleware layer. My LLM produces something harmful as output. Or my generative AI model generates an image that is not fit for my users. I want to somehow catch that and correct it if I can, right? Or I want to put certain things into my model, but I want to make sure that I'm not putting in either private or sensitive data. Or I want to structure the output of my model in a certain way into certain structures or types like JSON or integer or float. All of these sorts of things kind of, I personally would break this apart probably into maybe like validation,

• 39:09Daniel Whitenack?

  type and structure, and then like security related things. Cause there's a lot here. There's validation, which is like, is my output what I want it to be? There's security related things, which is, am I okay with putting the current request into my model? And/or sending the output back to my users. And then there's type and structuring things. So with images, like is the image upscaled appropriately for my use case? Or with text, if I'm putting in something and wanting JSON back, is it actually valid JSON? That's more of a structure type checking type of thing. So there's a lot in this category.

• 39:51Daniel Whitenack?

  And I think you can, you're probably getting the fact that I'm thinking a lot about this. I can tell. And there's a lot here, but um, yeah, other things fitting into this category would, I think cool one called Rebuff, which is doing kind of checking for prompt injections. For example, that's like part of that security side of things. There's things like prediction guard and guardrails, guidance, outlines now that do.

• 40:18Daniel Whitenack?

  Type and structure type of things. There is also, I would say, a layer of this, which a lot of people are implementing in the kind of roll your own Python DIY way as well, which in Prediction Guard, we implement some of these, but also people are implementing them in their own systems like self-consistency sampling, like calling a model multiple times and either using the out, choosing between the output or merging the output in some interesting way, or things like that, this sort of consistency stuff. I think a lot of people are are rolling their own too.

• 40:56Chris Benson?

  What do you think as we start winding up here? What do you think are some of the the takeaways from this chart? You know, or what brings top of mind things that people, as they look at it, might benefit from? How would you see it in the large?

• 41:10Daniel Whitenack?

  Yeah, it's a good question. I think one major takeaway, one thing to keep in mind is the model is only a small part of the whole app stack here. In a similar way to like used to when a thing existed called data science, we would say, uh, training a model is only a very small part of the kind of end-to-end data science lifecycle of a project. There's a lot of other things involved. And I think here, you know, you can make a similar conclusion that the tendency is to think of the model as the application, but there's really a lot more involved than there's. Our friends over at Laten Space would say this is really where AI engineering comes into play.

• 41:55Daniel Whitenack?

  This space of AI engineering is, seems to be developing into a real thing, whether you call it that word or not, it is part of what this is. So that's one takeaway. I think the other takeaway is maybe just kind of forming this mental model around these three spokes of the stack. So you've got your app and app hosting, you've got your data and your resources, and you've got your model and your model middleware. And all that kind of middle hub would be some sort of orchestration that you're performing either in a DIY way or with things like Lingchain to connect all of those pieces together.

• 42:39Chris Benson?

  So you're probably hoarse by now because we've pulled so much information out of you. This was a really, really good dive. You know, it's one particular publisher's way of looking at it, but we've never really dived into all the components of the infrastructure of a stack with this kind of. And, and I think most people haven't had a chance to see it yet because so much of this has really arisen in recent months. Thanks for kind of wearing half of a guest hat along the way here and, and taking us through this on, on this fully connected episode.

• 43:13Daniel Whitenack?

  Yeah, and I think in terms of learning about these things, I think people can check out our show notes. We'll have a link to the diagram that we've been discussing here. I would say, learning wise, this helps you organize your thought process, but to really get an intuition around these things, you can look at various examples in this diagram and go to their docs and try out some of that. There's a variety of kind of end to end examples as well that are pretty typical these days. Like in language, if you're doing kind of a chat over your docs thing that involves a model and a data layer and an application layer.

• 43:52Daniel Whitenack?

  So just building one of these example apps, I think, could give people the kind of learning and that sort of thing that they need. But, um, yeah, it's been fun. It's always helpful to talk these things out loud with you, Chris. I, I find it very useful.

• 44:08Chris Benson?

  Well, I learned a lot every time we do this, so thanks a lot, man.

• 44:11Daniel Whitenack?

  Yeah, yeah, well, we'll see you next week.

• 44:13Chris Benson?

  See you next week.

• 44:22Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a long time listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changetalk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.