Controlled and compliant AI applications

You can’t build robust systems with inconsistent, unstructured text output from LLMs. Moreover, LLM integrations scare corporate lawyers, finance departments, and security professionals due to hallucinations, cost, lack of compliance (e.g., HIPAA), leaked IP/PII, and “injection” vulnerabilities. In this episode, Chris interviews Daniel about his new company called Prediction Guard https://www.predictionguard.com/, which addresses these issues. They discuss some practical methodologies for getting consistent, structured output from compliant AI systems. These systems, driven by open access models and various kinds of LLM wrappers, can help you delight customers AND navigate the increasing restrictions on “GPT” models. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Prediction Guard https://www.predictionguard.com/Prediction Guard docs https://docs.predictionguard.com/LLMs in Production II event https://home.mlops.community/public/events/llm-in-prod-part-ii-2023-06-20Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:42Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist and founder of Prediction Guard. And I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 0:59Chris Benson?

  Doing well today, Daniel. It's just continues to be super interesting in this space, in the world of AI. So much change. This has been a year, I think it's a year that's for the history books in terms of the advances and the fact that AI is really making deep impact into the general population, people who normally might not be listening to our podcast as hard as it is to believe that.

• 1:25Daniel Whitenack?

  Yeah, I was just going to say, people like in companies like my wife's company, which is not a large company, it's not a tech company, but they're having conversations about how do we as a company leverage AI or leverage large language models in our content generation or what have you. And it's really permeated all industries at this point, I think. And people are wrestling with the idea of what do we do? Not just, you know, if we do something in relation to AI.

• 1:56Chris Benson?

  Agreed. I think that's a huge issue right now. It is potentially more confusing about how to handle everything that's coming at companies these days from a large language model and generative AI than it ever has been. And the problem is getting harder. And so we want to talk a little bit about that today. And I want to acknowledge a couple of things with our audience. Many of you have been with us for quite a long time. We've been doing this show for about five years on a weekly basis. I know it's been forever. And it's just getting more and more interesting. Over that course of time, something I wanted to share with our audience is I've gotten to know Daniel pretty well. And we didn't

• 2:40Chris Benson?

  know each other super well beforehand. We had met in the in the Go software development community as kind of the two people looking at data concerns. But Daniel, over time, has demonstrated not only the fact that he is repeatedly an incredibly smart man with a lot of capability, but he's also an incredibly good human being. And for anyone who's followed the show for a long time, they know that the idea of just being a good person and AI for good and such things are a huge repeating topic in the show. And I've also learned to trust where he's going and to understand that if Daniel is doing or interested in something, it's something that I want to know about. And so I'm today, we want to

• 3:23Chris Benson?

  hit this large language model kind of in the world and how you manage that. But I also want to acknowledge that we're going to have bits of the show that could be considered conflict of interest. And the reason I say that is we're going to talk about some work Daniel has been doing. And so if that bothers anybody, this is the point where you might want to shut off this particular episode. But I'm hoping that most of you trust us and have been with us for long enough to know that I'm not going to take you down a path that you wouldn't want to go. And I've asked Daniel to talk not only about the space of large language models being brought into production and trying to juggle all the things coming out, but talk about the work he's doing. And so we are unabashedly going to go that direction. And if anyone has hate mail to send

• 4:06Chris Benson?

  Please send it to me because I have demanded, I have demanded that Daniel talk about this. And so, uh, thank you for bearing with us on that. And so, Daniel, if you, uh, you're kind of both the co-host today and the guest, if you will. And if you might lay out a little bit of a landscape for us about, uh, what this looks like as an insider, as someone who spends all your time focusing on this problem, that might be a good way to start.

• 4:34Daniel Whitenack?

  Yeah, thanks, Chris, and thanks for the kind words. Um, I've, over time, learned so much from doing this show and it's shaped a lot of what I think about. And certainly the things that I've been thinking about really since, well, this whole year, since kind of Christmas time, have been focused around these ideas of controlling large language models, guiding them, guarding them, making compliant AI systems. And a lot of that's led into the thing that I'm building right now, which is called Prediction Guard. So that's what you're referring to in terms of what I'm building. So I'm coming at it from that perspective and been thinking about this a lot, been talking about this a lot publicly.

• 5:16Daniel Whitenack?

  And excited to do things like, you know, upcoming, there's an LLMs in production event, uh, put on by our friends at the ML Ops community. That's really cool. I'm giving a talk there on controlled and compliant AI applications. So that's part of what I'll share here today as well. One question maybe that I have for you as we start out here, Chris, is what have you experienced in terms of the people that you're talking to with regard to the pressure that they're feeling either internal to their own company or from like market pressures? Like jump into the AI waters, like implement something, make AI part of our stack. Like, what are you seeing there?

• 5:58Chris Benson?

  So I don't think you'll be surprised when I say this, and we've alluded to this on some previous episodes, but it is a difficult business concern to navigate. Um, I know all of us who straddle into the AI technical realm are incredibly excited. We're trying to Figure out how to do the models and put them out there and everything like that. But if you are not in our shoes, if you're walking on a slightly different path, and let's say you work for a legal department or a compliance department or other business concerns, and suddenly these technologies are coming at you hard and fast, week by week in 2023. And you're trying to navigate that and look at things like licensing

• 6:43Chris Benson?

  on how the data that goes into models is used. And you're looking at compliance concerns and you're looking at protecting your intellectual property. There's a whole host of challenging business problems with essentially no guidance. This is a brave new world. It's a that has to be pioneered through. And so I have talked to a lot of business people in various roles, including attorneys. And this stuff is scary stuff. It is problematic stuff. It is challenging to navigate. And and I definitely want to take you down the path today of talking about the space and prediction guard relative to how you actually

• 7:28Chris Benson?

  get these models out there in a productive way in a business environment. So that people can take advantage of the technology and understand what the pitfalls are and such. So that's the big thing that I've been hearing. I've been getting an earful of it lately, like, Chris, settle down. Stop taking us down this this AI thing. We got to figure some things out first. So I'm coming to you for answers, man.

• 7:49Daniel Whitenack?

  Yeah, it's so tempting, actually, to have really easy to use systems, like, let's say, the OpenAI API, right? I can go to the playground or I can go to ChatGPT or I can go wherever, put in my prompt and get some like magical output, right? It's magical and immediately it triggers in your mind. I can solve real business problems and I can create like actual solutions with this type of technology. Like, it's so quick to make that connection. But what I've seen, both in sort of advising and consulting and conversations that I've been having,

• 8:30Daniel Whitenack?

  is on maybe like a less stringent case, like people are struggling to make that connection to how they can build robust systems out of these technologies. So it's one thing to get text output and look at it with your eyes as a human, right? And say like, extract this piece of data, or give me a summary of this, or something like that. But as soon as you make that programmatic, right, and automated, then how do you know you're getting the right output? And if you actually wanna do something with that, like you're outputting a number, you know, a vomit of text blob out of a large language model doesn't really actually do you that much good

• 9:13Daniel Whitenack?

  if you're trying to implement like a robust system that's making actual business decisions on top of the output of large language models. On the harder side of this, I'm getting feedback from people that either I know or I'm advising or other things that companies are actually telling them, no. There's a full stop on using, quote, GPT models in this organization because of one of a few different reasons. Maybe that's a risk thing around, hey, we're gonna hallucinate some name out of this or something that this person doesn't exist and that's gonna get us in trouble. People are gonna stop trusting our product and that sort of thing.

• 9:56Daniel Whitenack?

  So there's the hallucination or consistency of output sort of thing. There is also, as you mentioned, the IP or PII type of leakage scenario. So it is actually a problem for people to sit in a company. I'm sure this would be true whether you're at, you know, your company or a variety of other companies that I've talked to where I'm sitting there and I'm like, oh, I could solve this problem with ChatGPT. Let me copy and paste this user data into ChatGPT and like have it summarize something or extract something or whatever it might be. It's sort of unclear in murky waters like how that data is actually going to be

• 10:37Daniel Whitenack?

  used by OpenAI and you're kind of leaking IP or company information, PII, to external systems, right? Which is a big, big no-no. Regardless of how that's used in the end, this data, it seems like is going to exist outside of your own systems. And so, on the harder side of this problem, people are being told like, no, you have a full stop. Can't use GPT, can't use large language models. So to summarize how I would kind of think about this problem space, people are feeling the pressure that they need to or really want to implement these systems, either because they feel like they're getting left behind, or there's an actual market pressure for

• 11:22Daniel Whitenack?

  them to do something. But in practice, they don't know how to deal with the outputs of large language models, and they might not even be able to connect to the kind of most common large language models because of these privacy, security, leaked IP type of issues.

• 11:39Chris Benson?

  I think that's really, really widespread. It's funny, you've kind of enumerated a whole set of risks associated with that. Yesterday, just as a thing, you know, I have a particular employer and thinking about public information, you know, well-known public information about the lines of business that we have that is publicly acknowledged in multiple sources out there. I went to ChatGPT, I should have done the 4.0 model, but I forgot, and I just let it default to the 3.5, and I simply asked for our 19 lines of business, which is incredibly public knowledge, and it got it wrong. It got it wrong the first time, and so I tried to steer it a little bit, and it got it wrong the second time.

• 12:20Chris Benson?

  And had I not known better about the intellectual property concerns with licensing, had I tried to put something in that might have been out there in the public. So I run into what you just said all the time, and there are so many risks, and yet... There's so much value to extract from the space. And so I think putting your finger on the fact that if you can find a way to mitigate these risks in various ways, that will unlock a huge amount of value for a lot of organizations and users to do that. But it certainly, from my standpoint, feels like the Wild West right now.

• 13:02Daniel Whitenack?

  Yeah, I would say that that's true. And yet, there's these concerns, like the money that people are able to save operating costs with AI in your business are significant. So I saw the study from Accenture estimating like insurance companies saving $1.5 million per 100 full-time employees. So if you're insurance company A, and you're not trying to implement AI systems in your business, then you're actually introducing a liability, right? Because insurance company B might be doing that, and they're going to slash their prices

• 13:44Daniel Whitenack?

  and undercut you and put you out of business, right? So even regardless of new features that might be implemented in like people's products and that sort of thing, there's this real liability around not considering AI solutions as part of your business strategy.

• 14:04Chris Benson?

  I think that's a huge point, and that's the other side of the coin that I was just talking about. There was the risk of using, and there is the potentially larger risk of not using at all. So we're seeing that in all markets in terms of the need to stay on top of, you know, what is gradually evolving over these past months, and to be able to use that to promote your business. And if you don't do that, the risk is substantial. So the idea of navigating the licensing and the compliance concerns and being able to productively use these outputs is really crucial to being successful in almost any industry going forward.

• 14:46Chris Benson?

  So definitely looking forward to finding out how we might do that.

• 14:55Speaker 00

  I'm Jared and this is a changelog news break. In what appears to be a particularly security unaware move, Google has added eight new top-level domains, two of which are quite concerning: .zip and .mov. Yikes. Ars Technica writes, "While Google marketers say the aim is to designate tying things together or moving really fast and moving pictures and whatever moves you, these suffixes are already widely used to designate something altogether different. Specifically, .zip is an extension used in archive files that use a compression format known as zip.

• 15:37Speaker 00

  The format .mov, meanwhile, appears at the end of video files, usually when they were created in Apple's QuickTime format." Fishers and scammers rejoice, the rest of us beware and be ready to help protect your family and friends from this otherwise completely avoidable new threat vector. The linked Ars Technica article demonstrates a few URLs scammers could now craft and they're darn near indistinguishable from the legit URL, even to someone like myself with trained eyes. One such URL in the example is a Kubernetes release, which yes, is distributed as a zip file. You just heard one of our five top stories from Monday's changelog news.

• 16:22Speaker 00

  Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 16:43Daniel Whitenack?

  If I could summarize some of what's been said, we kind of talked about these two large categories of problems. One was... The structuring, consistency and validation of the output of these models to make them useful in actual business use cases. And the second was maybe compliance concerns, privacy, security concerns, which really have to do with like how a model is hosted or how you access that model. So on the one side, it's how do you process the output of a model? And then on the other side, how do you access or host a model? Both of those things can be pretty big blockers.

• 17:28Daniel Whitenack?

  To kind of dive into the latter of those, the hosting, privacy, security thing, I actually am quite encouraged by where things are headed recently because we've seen this kind of proliferation and explosion of open access models that continue to be released day after day.

• 17:48Chris Benson?

  Mm hmm.

• 17:48Daniel Whitenack?

  The most recent one at the time of recording this, I might be missing one. They seem to come out every week, but one, for example, that came out recently is the MPT family of models from Mosaic ML, which is just really extraordinary. I think that they have up to like context links or like you can think about that as kind of your prompt size for the model of like 60,000 tokens. And they do quite well in various scenarios. So there are these increasing number of open access models, but I would say there's two problems with using these as a business. Let's say I wanted to host one of these and use it internally. Well, maybe three problems.

• 18:32Daniel Whitenack?

  It's always good to have three points, right? Three, three problems. One is like you still have to figure out like the weird GPU hosting and like scaling of that model, which is a challenge, right? The second is in reality, these open access models, at least according to most people, I think it's generally accepted that these aren't quite up to the standards of the larger commercial systems that like OpenAI and others are putting out there, Cohere and Anthropic and others.

• 19:02Chris Benson?

  Sure.

• 19:03Daniel Whitenack?

  So there's like a performance concern, there's the hosting concern, and then the third, which is the same as our other major topic here is you still have to figure out how to use the output of them. They're still just going to vomit up text on you and you have to figure out how to deal with that. This has led some people to strike up these kind of expensive deals to host OpenAI models in Azure infrastructure. That's becoming easier over time. I hope that becomes increasingly easier. It's still a little bit like limited to Azure mainly in my understanding and it's definitely not cheap, I would say. If you kind of compare all the costs and add in the engineering time to do that and and all that.

• 19:47Daniel Whitenack?

  So some people are solving this model hosting issue by either hosting an open access model, maybe with a hidden performance, or implementing a really expensive kind of private version of OpenAI, something like that. And if you don't have that budget, or if you don't know about GPUs or how to host models, you're kind of out of luck in a lot of ways.

• 20:13Chris Benson?

  Not only I agree with you, but I think that that's going to proliferate in terms of the challenges across there. I know, speaking for myself and another friend that I talk to a lot about this stuff a lot, we are experiencing the fact that as model updates come out, new models come out, they have different strengths and weaknesses. There are some things that I might, for instance, go to GPT-4 on. There are other things I might go to Bard on now. And, you know, those are just two. There's a whole bunch of open source ones and we were starting to talk about that. And with the acknowledgement of, for instance, OpenAI has kind of acknowledged that there is a practical limit in terms of how much data you can feed a model and that we need to start looking at other dimensions on that.

• 20:57Chris Benson?

  So with practical limits in sight, the commercial advantage, for instance, may hit that ceiling and open source ones will gradually catch up. And so you're seeing the relationships of utility for a user between different models changing on a regular basis and us users having to make adjustments to that. How does that play into the landscape? Because if you're an organization and you're trying to make investments, like do we bet on OpenAI and Microsoft? Do we bet on Google? Do we bet on open source options? What are the options there? What are the different capabilities that might be available to us for doing that? And acknowledging upfront, Prediction Guard may be one of those.

• 21:40Chris Benson?

  What does the rest of the landscape look like and how does Prediction Guard fit into that? And what are some of the pros and cons that you see?

• 21:46Daniel Whitenack?

  I'll decouple a couple of these things and talk about the general landscape and then prediction guard. So in terms of this problem of the hosting, compliance, privacy, IP leakage, that sort of thing, I think if you're a company of a certain size and you can afford kind of a private, open AI setup in Azure, it's probably a pretty reasonable solution. It will definitely work very well, but it's going to be very, very costly. And again, it's not going to solve this like structuring and usage of the output of language models problem. So you're going to have to put additional engineering effort into helping build layers on top of that that work

• 22:31Daniel Whitenack?

  for your business use cases. You could bet on certain open access models right now, but like you said, things are advancing so quickly, it's hard to say like, I'm going to put all of this effort into one and hosting of the one and build a system around it. I do think that there is advantages if you're going that route to center your infrastructure around kind of model agnostic workflows, like those in Langchain or others, where you actually abstract away the model interface and can connect to multiple large language models with with a lower, lower switching cost than if you kind of have a one-off solution centered around a certain model. So I think there's, there's some things that people could be encouraged about there.

• 23:16Daniel Whitenack?

  In terms of that, though, if you think about, okay, now I'm going to go all in on these open access models, like you say, these models have different characters. So I'm going to want to host maybe multiple of them and generate these model agnostic workflows on top of Langchain and other things. You start to really add up the engineering effort to make this happen. A parallel might be, I could create a data visualization solution for my company by assembling a database and hosting that, making the connection into like a layer that would run like Plotly plots or something like that. And then maybe some UI for my users that those are embedded in.

• 24:00Daniel Whitenack?

  And all of a sudden, I'm now talking about An absolute fortune in engineering costs and support costs over time, which is why products like Tableau or other, you know, I remember a long time ago, I don't know how much people are still using it. One of the companies I was at was using Domo. This is one of these solutions where you quickly suck in data and visualize it and all of that. There's a reason why those products exist. So Prediction Guard, you could kind of think of as taking the best of open source models and the best of this kind of control and structuring of output, which we haven't talked about yet and we can get into here in a second,

• 24:41Daniel Whitenack?

  and assembling those together in an easy to access and cost efficient manner. So people can get quality output out of the latest large language models that's structured and ready to be used in business use cases. And also with a guarantee, if you want it, around using only specific models that are hosted in a compliant way, even compliant in a certain way, like a HIPAA compliant way, or in a data private sort of way, where your data isn't leaked if you're putting data into models. So that's kind of how the landscape works and how Prediction Guard works as this kind of system that assembles the best of large language models

• 25:24Daniel Whitenack?

  with structured and typed output that can be deployed compliant without this whole huge engineering effort to build your, you know, roll your own system.

• 25:35Chris Benson?

  You mentioned structured and typed output. And can you go ahead and kind of talk a little bit about that? Because I think, you know, for many of us that are listening, we're used to using the models that are out there, kind of in the default interfaces on the web, you know, using ChatGPT, using Bard, and we're not really dealing with that. You know, we get an output and, uh, but we're not at the level of sophistication where we're doing APIs and such as that. Can you talk a little bit about what structured output looks like when you're dealing with it from an API standpoint and how you unify that landscape?

• 26:06Daniel Whitenack?

  There's a lot of use cases where this may come up, but let's take one, for example. Let's say that you're doing data extraction. You have a database with a column in it, which is basically, so this scenario has happened at every company that I've been with. So I know that it's very common. There's some database with a table in it, and there's a column that's like a comments column or something. And it's just like text blobs in there that are like notes from people or technician messages or user messages or like whatever it is. It's not structured. And you want to run a large language model over that to extract,

• 26:47Daniel Whitenack?

  you know, maybe it's phone numbers or prices or certain classes of information out of this column. Well, you could run your large language model and set up a prompt that says, you know, give me the sentiment of each of these pieces of text in my database. Well, that prompt, each time you run it through a large language model, maybe once it generates an output that says, "space, positive sentiment." And the next time it creates an output that says, "positive." And the next time it creates an output that says, "this is positive sentiment." And you can start to see there's a consistency problem here. Like, how do I parse all of these strange outputs

• 27:32Daniel Whitenack?

  from my large language model? You can do a little bit of prompt engineering to get around that, but ultimately, it doesn't solve the problem that you could have all sorts of weird output out of your large language model. So ultimately, what you would want in that scenario is a system that lets you constrain and control what types of output you're going to get out of your large language model. So in the case of sentiment, maybe I want to restrict my output to only POS, NEG, and NEU, tags for sentiment. There's only three choices. I always want one of those three, right? I don't want it to say, "this is positive sentiment," right? So I want to actually structure or control the output of my large language model to produce one of these outputs.

• 28:16Daniel Whitenack?

  Another example that's maybe a little bit more complicated would be to say, "I actually want to output a valid JSON blob out of my large language model, or valid Python code out of my large language model." And these are structures that are very well-defined, but you could have all sorts of variability coming out of your large language model. And if you want a specific type coming out of your large language model, maybe it's a float that you can do like greater than or add it to another number. Like, you need that as a typed output, or you need very specific structured output to actually make Automated decisions in your business. And so with Prediction Guard, what we're doing is we're kind of assembling the best of the recent advances

• 29:01Daniel Whitenack?

  in this kind of control and structuring of output and layering it on top of these open source large language models to allow you to say, here's my prompt. I'm going to send it to these five open source or open and or closed. We support OpenAI as well. So open and or closed models. And for each output, I want you to give me a float number. And that's the sort of rich, you know, output that you can get from large language models very quickly with a Prediction Guard kind of prompt, because you can control the models that you're using, either ones that are more privacy conserving or the closed source options, and provide constraints around the output that allow you to actually make business decisions on that.

• 29:46Daniel Whitenack?

  Now, there's additional checks that could go along with that, like factuality checks and toxicity checks, which we also implement. But I've vomited up a lot of information, so I'll pause here.

• 29:59Chris Benson?

  No, no, that sounds fascinating. It's the way I'm interpreting what you're saying is sort of like you have these kind of software filters that are creating boundaries, if you will, on how you structure input and what that output can be so it's usable, which kind of goes back to one of the points that we're often talking about on the show is that the AI is to some degree inseparable from the software that you're using it within. And so you have a best of breed software product that's kind of shaping and constraining what that can be so that it's actually usable on that. So as we look forward at kind of where things are going and you, what are some of the problems that you see going in the space that we have?

• 30:43Chris Benson?

  And like, what are some of the things that you would like to see Prediction Guard starting to address in the time ahead? And I don't mean so much as the far distance, but kind of like you're, you're busy putting this solution together now, works pretty darn well, what you already have. What are some of the challenges when you're in this kind of a fast moving space? Because you're, you're having the world change out from under you on a week by week basis right now.

• 31:07Daniel Whitenack?

  Yeah, I think maybe one of the things that we're thinking about is really at the forefront of our mind is ease of use and accessibility to both data scientists and developers. So the reality is that I think we had Kirsten Lum on the podcast talking about this, like the majority of data scientists out there are super constrained in the time that they have to put into one of these solutions, right? So it's really, really important that there is an ease of use to this sort of controlled, compliant LLM output and generative AI output. Now, what we're seeing, and I want to acknowledge this as well, is there are an increasing

• 31:52Daniel Whitenack?

  number of open source projects that are doing an amazing job at digging into this problem of controlled and guarded LLM output. So these are things like guardrails and guidance from Microsoft and Matt Rickards, RE LLM or RegEx LLM. These projects are doing amazing things at really flexible ways for you to control the output of large language models, but I see this as kind of like a double edged sword a bit. The more flexible you become, it's also possible to become less easy to use.

• 32:31Chris Benson?

  And there's more engineering involved in it.

• 32:33Daniel Whitenack?

  Yeah, so I saw Matt Rickard tweet about this related to his RegEx LLM project, which is that sort of famous quote about RegEx, which is, "I have a problem, and so I decided to use RegEx, and now I have two problems."

• 32:50Chris Benson?

  Yeah, that's been around for a long time, actually.

• 32:53Daniel Whitenack?

  Yeah, it's actually so true, right? And some of these solutions are coming up with their own query languages to kind of deal with this structured output, which I think is great and it's really important. But there's a need for this abstraction layer on top where I know kind of what I want my output to look like. So I should be able to plug that into something and have it constrain the output of my large language model in an appropriate way. So with Prediction Guard, what we've started with is the kind of presets of structuring your output. So I want integer and float and JSON and Python or YAML. I want categorical output.

• 33:38Daniel Whitenack?

  These are things that we support now. Also supporting kind of these hosted models and access in a guarded, kind of controlled way to these models. But let's say that I have a really specialized format that I want to work with. I would rather set up a solution with Prediction Guard, and this is actually what we're actively working on, where they could give examples of the structure that they want. And we actually generate the right constraints for them on the large language model output, which I think is very possible. And our initial work on this, which is kind of in a beta form, is really good. So let's say that I want a specific JSON with these specific fields

• 34:20Daniel Whitenack?

  or a specific CSV output with these specific columns, right? I should be able to give a few examples of that and generate the right underlying constraints for my large language model without the user having to think about special languages or regex or context-free grammar or these things that are a little bit harder to grasp. We'll handle that bit for you and you just get the right structured output from your models. So that's part of where I see us headed is leveraging these rich systems under the hood that are being produced around using context-free grammars, special query languages, regex, all of these things to structure output and combining those in a more automated way for users

• 35:05Daniel Whitenack?

  where they can just say, here's my examples, here's my query, and they just start getting the right formatted output from their language model. So that's kind of thing one is this automation of some of the problem and the constraints. I think the thing two would really be around the validation and checking of output in addition to the structuring. So right now, we support factuality and toxicity checks on the output of large language models.

• 35:33Chris Benson?

  So could you talk a little bit about what each of those are?

• 35:36Daniel Whitenack?

  Yeah, yeah. So let's say that I take a big piece of text and I generate a summary, or I do a question-answer prompt and get an answer, right? It doesn't mean the answer is factual, right? And we all know about the hallucination problems of these models. So the things that we have implemented in Prediction Guard are two things with respect to that. The first is a factuality-checking score, which is built on these trained models under the hood that look at a reference piece of text and your text output to determine a likelihood of the answer being factual. So this is an estimate on the factuality of your output. The other thing that we're doing around hallucinations and factuality

• 36:21Daniel Whitenack?

  is making it really easy for people to do consistency checks. I kind of alluded to this earlier, but we have all of these different language models accessible under the hood. So you could combine the outputs of Camel 5 billion, MPT 7 billion, DALL-E, and OpenAI, restrict the output to say, "Give me the answer, but only if all of these agree on what the output is." If all of them don't agree, then I'm going to flag that as not a reliable output. And so you can actually gain a lot by not just leveraging one model, but ensembling these models together to do a check. The toxicity thing is something that's been studied for a while, and there's models out there, state-of-the-art models,

• 37:06Daniel Whitenack?

  for detecting whether an output is toxic or not, or includes hate speech or not, that sort of thing. So this is another layer of check that you can have on the output. And so if you put the whole pipeline together of Prediction Guard, you've got models on the output which can be deployed compliant with HIPAA or just data privacy, those structured or typed output that you can define. Very easily. And then you can run additional checks on that output for factuality, toxicity, consistency as a final sort of layer in the pipeline towards the output that's used in a business application.

• 37:46Chris Benson?

  I appreciate the explanation. It's a very robust sounding pipeline that you have on that. Let me ask you this with, and this could be whether it's Prediction Guard or whether it's the larger field, you know, one of the challenges is certainly something I've been playing with, but I don't have a good rhyme or reason to it yet. With the proliferation of these models coming out and, and ever more coming, you know, we know this space is going to get larger and larger. How does a user or how would a system like Prediction Guard be able to determine which is the right way to go in terms of which model you want to choose or which group of models? And you talked about the comparisons

• 38:24Chris Benson?

  a moment ago. Like, how do you structure the input and know that you're going to get what you need from an output by putting the right model or collection of models together? And then knowing how to evaluate them against each other? Does that make sense?

• 38:41Daniel Whitenack?

  Yeah, yeah, that makes sense. Actually, early on, when we were building the Prediction Guard backend, this was actually front of my mind and has since kind of evolved a little bit. The fact that there's all of these models and I want to choose the right one for my use case, you can very much automate that process. And we actually, it's actually still implemented in the Prediction Guard backend where you can give some examples and evaluate a whole bunch of models on the backend. I think where this is headed though, and where the Prediction Guard system is headed is making it easier for people to get output from multiple models in a typed way because they know how to do the evaluation. They're

• 39:25Daniel Whitenack?

  familiar with this sort of thing, whether you're a developer doing sort of integration tests or unit tests and you're checking and you're asserting certain values, or you're a data scientist that's running a larger scale test against a test set. People kind of know what they want to do with that sort of thing. What they need is an easy way to get that typed output from multiple models. So like if I have a test set and I'm comparing to scores on the output, like float numbers. I need to get float numbers out of a whole bunch of different large language models to compare them to my baseline or to my test set. Right now, that's very difficult because all of these different kind of structuring guidance

• 40:10Daniel Whitenack?

  control systems work not for all models and they don't work in the same way for all models. And you have to implement it for all of the models. And so it becomes this compounding problem to figure out how to do that. And so how we're approaching that with the prediction guard system is there's a standardized API to all of these different models, along with the typed and structured control on the output. So I can do a query that says, give me the float output for these 100 prompts using these five models. And then I'll just compare all the float outputs and figure out which is the best. That's, that's not the hard problem.

• 40:54Daniel Whitenack?

  It's the getting that structured output from the variety of models in a robust and consistent way. That's actually a more difficult problem.

• 41:03Chris Benson?

  Gotcha. Is it fair, you know, as we're talking about this, it sounds a lot like you're, you like you're also solving one of the bigger challenges we've talked about over time, which is that there's so much domain expertise in the AI space in terms of being able to manage models. But if I'm understanding you correctly, it sounds like with minimally, with some basic software skills, knowing how to use APIs and stuff, you can probably without deep expertise and deep learning, manage to get some fairly productive output through production guard by implementing it that way. In other words, it becomes just another part of your software workflow. Is that, is that a fair characterization, what I'm saying?

• 41:45Daniel Whitenack?

  I would say it is in the sense that there's still some sort of like integration testing and integration that will have to happen regardless, right? But going back to my example before of like the data visualization stack, it's a lot harder to implement the database and the visualization layer and the front end than it is to like login and do the, there's still configuration that's needed in like a Domo type solution or Tableau. It's just a lot more accessible, right? So here we have the language models hosted on the backend. We have the structured guarded way to query those models via something that all developers

• 42:28Daniel Whitenack?

  know how to use, a REST API or a Python client. Maybe there'll be other clients over time. And have the ability to configure that in the way that you want. So I want output from these five models. I want to ensemble them together, or I want this structured output. And so there's still configuration and I think developers and data scientists, they want that. It's just that it's really hard to get all the other pieces in place and we're hopefully making that a lot easier.

• 42:56Chris Benson?

  So, and let me ask one final, I think this is an aspirational question, but I'm kind of curious. One of the things that we've seen with large language models is the ability for people who aren't even developers, you know, I was saying like developers who aren't even deep learning experts, but to have a certain amount of capability producing code, you know, that's the kind of avenue into a no-code world has at least been started on this. It has a lot of maturing to do, obviously. Do you envision a point where someone with very limited skills can also use Prediction Guard in this way and be able to kind of generate apps using large language models

• 43:37Chris Benson?

  that then kind of feed into a more mature workflow like what you've described? Do you think that that's attainable at some point in the not-so-distant future?

• 43:46Daniel Whitenack?

  It's hard to say how far this kind of automation will go. I think a lot of the... A lot of the agents that we've seen produce good demos, right? But they have an additional layer of this sort of additional problems around automating these various steps of the process. I think that in terms of what we're looking at, this sort of automated structuring of output is a step in the right direction in terms of I don't have to define a special query language or a special specification, but I can say what sort of structure I want output and that gets output. I think then if you layer that on top of the agent sort of infrastructure that's in

• 44:30Daniel Whitenack?

  LangChain and the data augmentation, we just had the episode with Jerry from Llama Index, which is super fascinating. So if you layer the kind of structured guarded output with the chaining and agent and automation of LangChain and maybe the data augmentation of Llama Index, I think a lot of things become possible. I hope that that becomes, some of the things that you mentioned become possible. It's yet to be seen, but I am really encouraged that adding in this sort of type safety for outputs and structuring of outputs gives a lot more confidence, maybe in some of the checks that you could do on AI agents over time.

• 45:12Daniel Whitenack?

  And that increases our confidence in sort of releasing AI agents on various parts of the workflows that we'd like them to work on, right?

• 45:21Chris Benson?

  So you've sort of already covered some of the territory, but for our listeners, Daniel and I often, when we're talking to a guest, we'll talk, we'll kind of finish with what we roughly call the future question, you know, kind of wax poetic about where things are going. And so, Daniel, since you're knowing that there's, I've kind of hit some of that already, but what would you be asking yourself? You know, so you've kind of had me throwing these questions at you from a point of somewhat ignorance compared to where you're coming from as the expert on it. What right now would you ask yourself that you haven't covered that you think is worthy of getting in before the episode is over? I'm putting you on the spot.

• 46:04Daniel Whitenack?

  Yeah, I've mentioned open access models quite a bit. And I think hopefully, a lot of us are encouraged by the direction that that's going, that these models are getting better and better. But one thing that maybe I would ask myself, or that I think is important to highlight and encourage people with is these open access models might not quite be at the level of OpenAI, Anthropic, etc. yet. But I think not only will they get there, but already, like, in the space where we're at now, with some of these kind of structured control elements around open access models,

• 46:46Daniel Whitenack?

  you can actually boost the performance of open access models to be more in line with, you know, OpenAI-level output. Because what you can do is say, "Well, I'm going to force my output to this. If I'm not able to produce it, you know, I can re-ask the question, or I can try a variant of my prompt." And these kind of wrapping layers around open access models actually provide a way for you to operate in a data-private, compliant way with open access models that boost their performance closer to what these kind of closed and maybe more suspect in terms of IP leakage and that sort of thing, systems are doing.

• 47:28Daniel Whitenack?

  So I think that's an encouragement that I've found recently, and I hope that's encouraging to others, is we are really seeing a proliferation of these models, and they're all going to have a little bit different character. But the ways that we wrap them and the way that we present them is a lot of, provides the majority of the value of those models. And I think we'll see not only Prediction Guard, but other systems as well, coming out that wrap these models and use them in really intelligent manners that boost their performance in a way that isn't reliant on sort of a centralized API.

• 48:05Chris Benson?

  I appreciate that. I think you're right. And I am deeply appreciative of you not only telling us about Prediction Guard, but actually kind of laying out the space. Even if someone is not chomping at the bit the way I am to use Prediction Guard, they hopefully kind of understand what some of the problems are that need to be addressed, whether by you or others out there. So, thank you for allowing me to twist your arm and do this episode today. I appreciate you letting me go there. So, anyway, thank you very much to my good co-host and my guest today for coming on Practical AI.

• 48:45Daniel Whitenack?

  Thanks so much, Chris.

• 48:51Speaker 00

  Prediction Guard is waiting for you. Check it out at predictionguard.com. There you'll find all the features, pricing info, and docs to get started. The link is also in your show notes and chapter data for easy click-ins. Thank you for listening to Practical AI. Help us help more people by sharing the show with your friends and colleagues. Word of mouth is still the number one way people find podcasts they love. Thanks once again to our partners Fastly, Fly, and TypeSense for helping us bring you awesome pods each and every week, and to Breakmaster Cylinder for producing all the beats on all Changelog podcasts. That's all for now. We'll talk to you again next week.