E28: Keeping the AI Revolution on the Rails with Shreya Rajpal of Guardrails AI

Nathan Labenz sits down with engineer Shreya Rajpal, the creator of Guardrails AI, a new Python library that allows developers to add a layer of output, validation and correction to their code. Practically guardrails can ensure a reliable interface between language models and more traditional deterministic software systems. At the same time, mind-bending and potentially risky use case frameworks like Guardrails allow developers to ask and answer entirely new kinds of questions. Talking to Shreya really reinforced just how early we are in LLM’s impact on the software industry. We were introduced to Shreya’s work when recording our interview with Matt Welsh, the CEO of Fixie. AI (Ep 19: How AI Agents Will Change How We Work). LINKS: Guardrails AI: https://shreyar.github.io/guardrails/ We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. TIMESTAMPS: (00:00) Episode preview (05:00) Why Shreya built Guardrails AI (08:33) Common ways LMs can “go off the rails” and how Guardrails can correct it (13:45) Discussion of validators (15:31) Sponsor: Omneky (18:48) Business and creative use cases of Guardrails AI (25:00) What can be achieved by Guardrails AI that cannot be achieved by traditional code (32:44) How agents work vs how Guardrails works (35:34) AI as shepherd vs delegating to AI and the role of human understanding (39:54) Trust deficit and risks (46:33) Is it realistic to imagine GPT-4 using Guardrails? (52:00) How Shreya thinks about security (57:02) How Shreya thinks about embeddings (1:05:05) How Shreya thinks about problemsolving with LMs (1:07:50) Shreya on OpenAI’s Evals Library, Anthropic’s Constitutional AI (1:12:55) Discussion of determinism (1:15:00) Recommendations for developers to minimize overhead (1:26:00) Predictions about providers in the space of LMs (1:29:30) Shreya’s favorite AI tools (1:30:00) Would Shreya get a neuralink implant (1:31:50) Biggest hopes and fears for AI TWITTER: @CogRev_Podcast @ShreyaR (Shreya) @labenz (Nathan) @eriktorenberg (Erik) SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected]. Music Credit: MusicLM More show notes and reading material released in our Substack: https://cognitiverevolution.substack.com

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  It's kind of insane to see just the amount of like activity and excitement around the space. Like there's, you know, people training and fine tuning deep learning models that, you know, weren't even in the space like a few months ago. And that's really awesome, right? And that I had like people who were like, oh, I I really like guardrails, I really like OpenAI, but it's just too expensive for what I'm trying to build. And so I can you make this work with, you know, an open source model as an example. Um, so I I do think that we're going to see a lot of that that proliferation happening of, you know, great performing models from like different price points, different latencies, different providers, et cetera.

• 0:34Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Levens, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, my guest is Shreya Rajpal, a former machine learning engineer at Apple and founding engineer at Predabase, who is now best known as the creator of Guardrails AI, a new Python library that allows developers to add a layer of output validation and correction to their code.

• 1:18Nathan Labenz?

  As anyone who's spent time building AI-powered products over the last two years will attest, validation and even more so reliability are key challenges. LLMs simply don't always follow instructions and sometimes go entirely off the rails. Better models have helped tremendously, it's true. GPT-4 can follow instructions far more reliably than earlier models and Claude V 1.3 is also very impressive. But with this elevated capability also comes expanded developer ambition. And so it seems that for the foreseeable future, the problem of LLM reliability will remain both critical and ubiquitous.

• 1:59Nathan Labenz?

  Trey's work tackles this problem in many ways and at multiple levels. Super practically, guardrails can ensure a reliable interface between language models and more traditional deterministic software systems. Validations like, did the language model return data with the right type and format? Or, did the model choose a value from the list of allowable values that we provided? These are very familiar questions for developers and are still powered in guardrails by traditional code. But at the same time, and for me, this is clearly the more novel, exciting, mind-bending, and potentially risky use case. Frameworks like guardrails allow developers to ask and answer entirely new kinds of questions.

• 2:43Nathan Labenz?

  Assessments of things like the quality of a summary or translation, or whether a given piece of text contains any redundancies, inconsistencies, or gaps in logic. These are the sort of things that, until recently, developers simply had no way to validate. Thus, for product and engineering teams around the world, guardrails is both a solution to a very practical problem at hand, and a sort of introduction or bridge to an emerging paradigm of AI-first software development that goes well beyond the copilot-style autocomplete or even chat interfaces that we've recently seen, and begins to use AI functions not just as development tools, but as components of the production technology stack itself.

• 3:27Nathan Labenz?

  Talking to Trey really reinforced for me just how early we are in LLM's impact on the software industry. The core AI capabilities needed to transform software, as far as I can tell, mostly already exist. What remains, though, is the work of reimagining not only how software is built, but how it functions now that intelligence can be baked in at any point. Personally, I believe that this paradigm shift could ultimately unlock bigger productivity gains and more user value than the current generation of tools that increase developer speed but don't yet attempt to change the kind of software that they're building.

• 4:08Nathan Labenz?

  At the same time, this is also something to be approached with real care and caution. The delegation of AI output validation to other AI models is not a step to be taken lightly or taken for granted.

• 4:21Nathan Labenz?

  I hope you enjoyed this thought provoking conversation with Shreya Rajpal of Guardrails AI. Shreya Rajpal, welcome to the cognitive revolution.

• 4:30Speaker 00

  Yeah, really excited to be here. Thanks for inviting me.

• 4:33Nathan Labenz?

  My pleasure. So this is the first time that I've invited a guest, basically as soon as I got off the call, recording a previous episode, it was Matt Welsh, CEO of Fixie.ai, who mentioned your new project, Guardrails AI. And I was immediately like, okay, I have to learn everything I can about this. So I'm really excited to dive into it with you. I guess let's start with what made you say, you know, a couple of months ago, I need to build a system to help people keep their language models on the rails.

• 5:09Speaker 00

  Yeah, yeah, I think it's, I was really solving my own pain points and my own problems. So I had been, end of last year, I'd kind of been like doing some tinkering on my own, where I was building some applications. And even as I was building them, I was like, yeah, this is really cool. I think it was not, nothing very exciting, you know, a lot of like what you're seeing on, on Twitter about, you know, chatting over proprietary documents, etc. And I was building that and I realized, yeah, this is pretty cool. I can see the potential of this. But even as I'm kind of like, you know, as a developer, like testing it out, like I can tell that it doesn't, you know, get me lively, the desired experience, the experience that I want, you know, to achieve. And so I think it was this big problem where these language models are really potent and

• 5:52Speaker 00

  they're really powerful. But they're also inherently very stochastic and very hard to control. The other kind of like, what makes this very interesting is that like, unlike traditional machine learning, how this is different is that as a developer, you haven't really trained the model. So you can't just, you know, throw more data, like make it work really well for your use case. And then separately, the only knob you really have as a developer is, you know, like, here's this prompt. And if you wanted to, you know, like, maybe do something or not do something, like how people, how developers typically deal with that is just, you know, adding like a lot of verbiage in the prompt and maybe a lot of like exclamation marks, etc., like make it listen to you, right? And that just seems like woefully inadequate.

• 6:33Speaker 00

  Um. And so guardrails is this idea of, uh, uh, you know, a specification framework where uh, as a developer, you know what the right output for an L L M looks like and you're able to, you know, like decompose that and deconstruct that and, you know, individually validate and verify um, uh, each component of that output. Um. And then, you know, if any of those components fails and if any of those quality criteria that that you impose on it fails, and you know, it gives you a set of tools to address that in a very extensible manner. Um. So I was building this and I was like, yeah, I I need like, I know what responses I want a user to be able to get from this thing that I'm building and how do I ensure that, um, uh, you know, I'm always able to do that for a wide variety of scenarios. And so that was kind of some of the inspiration.

• 7:14Speaker 00

  Um. It also, um, it spelled in Spent some years and, you know, um, autonomous systems and self-driving and, you know, it's a similar problem there as well, where you have, uh, this like really powerful deep learning based uh, perception model that often feeds into, you know, this more, um, a more rule-based like decision-making system. And how do you essentially ensure that the interface between that stochastic system and that deterministic system is, you know, robust and and not brittle whenever like the perception system like maybe doesn't do as well. Um. And so, uh, the idea was, you know, to like, uh, it was, it was inspired by like some techniques you'd kind of see there, but like a built for language models and built in a very extensible way so that it's not very given specifically. Um. Those, yeah, some of the inspiration, yeah.

• 7:57Nathan Labenz?

  Yeah, as much as possible, I love to get super concrete on these things. This is clearly a pain point that a lot of people have. The project has, you know, gone on its own little rocket ship ride of GitHub stars with twelve hundred uh, as of last check at the time of this recording. Uh, but probably a lot of people listening also could use a little bit more of like a concrete example of like, okay, what kind of thing are you looking for and how is it failing? And then, you know, can you kind of tell us maybe a couple of those? And then how does the? Guardrails, you know, come in and save the day in those instances.

• 8:35Speaker 00

  Yeah, yeah, absolutely. I love that question. I love digging deep into the details, so I'm happy to do that. I think when I was prototyping guardrails, my favorite like kind of prototype example was that I have this, you know, Chase credit card agreement, which was like my own credit card agreement. And I want to extract like what are the key terms, et cetera, from that credit card agreement, you know, get a nice JSON out of it. And I want like some, as a user, I know that, okay, if I'm extracting something like an interest rate, it must always be a number, you know, it must be maybe a percentage sign and it, you know, like what is a reasonable range for that? If I'm maybe extracting like the name of like a specific fee and I want this name to be presented somewhere, I know that the name should be, you know, very, very concise, it should be like very verbose, et cetera.

• 9:20Speaker 00

  So I think like one of the, so the common favorite points, et cetera, there is that, you know, like if I want to extract this information and then it needs to be like maybe added to some downstream data. I think like it's hard to do that, you know, reliably, consistently from an LLM because an LLM, you know, doesn't behave reproducibly essentially. So in this like structured data extraction setting, I could essentially enforce constraints of like what I want each extracted entity to look like. So for example, the interest rate must be a number, it must be within the risk range. The, there must be a description with each interest rate and, you know, the description maybe needs to be, you know, some, some length or some, needs to be relevant to like whatever the, whatever the, whatever the entity that it, it, you know,

• 10:04Speaker 00

  is coming along with. So I think like all of those constraints are like what I want to impose on this JSON structure. So that was a use case that I was prototyping it. I think since then, I have a ton of examples on my documentation, but like one of my favorite ones is a text to SQL. So in text to SQL, it's like a wildly different domain, but a lot of the same ideas apply, which is, you know, that you want correctness from any generated like SQL query. So the idea is that as a user, you want to be able to ask natural language queries over your data and get like a SQL query that you can maybe execute, right? So a lot of the ideas are the same, which is that you know, you need to be able to like, what are the constraints you want to be able to impose a priority on that SQL query? So it must be, you know, it must actually work for my database or the environment that I want to execute this query.

• 10:49Speaker 00

  Other, other constraints, right? Like you may not want to return results from any specific, um, you know, tables. So some tables might be private. And if you want to say like, as a customer, you never want to be able to, you know, like query those tables, you can like filter those out. Um, you can add things like only support these specific SQL predicates, like if there's any like drop predicates or you know, like maybe uh, update or insert predicates, you want to be able to filter those out. So the idea is that you can like, as you're setting up this text to SQL task, you can like add all of those constraints. Um, so how the, how Guardrail solves this problem is it takes your, um, it takes your database schema and sets up a SQL sandbox. Uh, for essentially like any SQL query that is generated, it's executed in that sandbox to make sure that, you know,

• 11:35Speaker 00

  um, it's executable. And if it's not executable, you take all of those errors uh, for why the SQL query fails to execute and like wrap those errors into something, send it back to the large language model, you know, like correct itself and get something that actually works uh, for your specific database. And then you can add like other constraints and other restrictions on top of that, like similarly, you know, like filtering out specific tables, filtering out specific predicates, it's a result. I think just this idea of like, um, here's a question, here's like a task I want this large language model to solve, but as a developer, I have like some domain expertise for what, what correctness needs to me in this task and how much I care about that correctness. Like is it, you know, if it's incorrect, then this query is totally useless to me. Or if it's incorrect, I just want to know about it and maybe post hoc and handle it. Like I think

• 12:19Speaker 00

  um, that is the main kind of like idea and uh, and you know, Guardrail like takes that and allows you to like basically implement that um, as a developer. Yeah. So hopefully that was a little more, more grounded in terms of, you know, specific examples.

• 12:31Nathan Labenz?

  Yeah, it's fascinating. One of the things I want to kind of explore the most in this conversation with you is the seemingly we're like opening up a spectrum or a dimension, if you will, where we can kind of go from on the one end, like explicit errors, like the errors that we are used to as developers in code. Uh, you know, and, and that is, could be a, starting with a syntax error.

• 13:01Nathan Labenz?

  Up to, you know, more meaningful errors like, but that could still be sort of, you know, found through kind of traditional software messages like this variable doesn't exist, or you know, all sorts of things like that. But then you've got kind of a whole domain of correctness, which code has never really accessed before. And that is like, what are you trying to do? Is this, you know, does this appear to be a reasonable like approach or output or, you know, from, from a much more sort of semantic or, you know, dare I say like intelligent point of view. But I find myself a little confused or a little bit like lost in that space. And I see that you're kind of covering it

• 13:42Nathan Labenz?

  in a really interesting like mix of, of ways, right? In the, in the library, in the validators, you know, there's a mix of validators. So maybe kind of walk us through, tell us how you think about that, I guess. And again, maybe some examples of like different validators that sit in different, you know, parts of that space would be helpful to help people understand what I'm talking about, if it's already clear.

• 14:05Speaker 00

  Yeah, yeah, no, I think that's a very, I think that was kind of like, one of the very exciting things about the library as I was building this out, which is that, you know, the general framework works even outside of things you can't, you know, like maybe use like an assert statement to verify or something, right? Like it, it's really extensible as a framework where you can have a mix of, you know, large language models and maybe some rule based heuristics, some programmatic checking, as well as like more traditional machine learning models that are, you know, like very maybe high precision classifiers, etc. And you can like ensemble all of these techniques together to get something that is, you know, greater than the sum of its parts and like much more robust and much more, much more reliable compared to just using a pure large language model. So as

• 14:47Speaker 00

  an example of that, like I last week added a bunch of guardrails for summarization. So if you're, you know, summarizing like multiple documents and maybe generating like an aggregated summary from that, you know, there's a, there's a bunch of like requirements that you may have in order to ensure that, you know, that summary is accurate, that summary is, you know, concise, it's not redundant, etc. And so how guardians kind of thinks about this general problem of correctness when. Assert statements aren't sufficient is to like really break it down into like either, you know, a smaller amount task or into smaller like verifiable, you know, heuristics, et cetera, and tries to, you know, like get an aggregate assessment of like how well this works.

• 15:31Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors. I want to tell you about my new interview show, Upstream. Upstream is where I go deeper with some of the world's most interesting thinkers to map the constellation of ideas that matter. On the first season of Upstream, you'll hear from Marc Andreessen, David Sacks, Balaji, Ezra Klein, Joe Lonsdale, and more. Make sure to subscribe and check out the first episode with A16Z's Marc Andreessen. The link is in the description.

• 16:00Speaker 00

  So in the case of summarization, what that looks like is if you essentially want to

• 16:08Speaker 00

  figure out if the summary is, you know, like faithful to the original source text, you can, what, how you can do it is like basically look at like each sentence in the summary. And then, you know, essentially do something like a similarity matching based on, you know, like which parts or which passages or sentences it's most similar to in the source text, which allows you to do much more fine-grained attribution and figure out, you know, where is this sentence that it generally coming from? The other cool thing that you can do on top of this is that you can assign thresholds. So as a developer, you can like do some experimentation, figure out what is my, what is my appetite for, you know, how, how varied I want these sentences to be from the original source text and then set a threshold.

• 16:49Speaker 00

  And then any sentence that, that is below the threshold in terms of like similarity score, you can essentially like filter out and, you know, like not, not include that in the summary. So that allows you to, I think it's just this way of thinking about,

• 17:04Speaker 00

  you know, Text outputs which may seem like, you know, um, like a single unit and it basically kind of like breaks them down into like smaller chunks. And independently tries to like verify that. Um, so, uh, I, I think like other uh techniques you have in summarization specifically is. Um, make sure that there, there's, you know, I want the summary to be concise. So make sure there's no like redundant information. Uh, so essentially within the generated summary, essentially make sure that like each sentence is diverse enough. From, you know, like other sentences within that. And if that's not the case, maybe like filter out, uh, you know, sentences that are too similar to each other, et cetera. Um, so, uh, it, it, yeah, I think like the cool thing about L L M's, it's like, you know.

• 17:43Speaker 00

  Uh, we are looking at like a lot of the first order benefits of that where you can use this and do like a bunch of like really insane tasks, but. Um, I think the second order benefits are that you can really use these models as verification systems, you know, in and of themselves. So you can like use a, a large language model within a validator to like, you know, verify and validate like whatever you're getting is correct. Or you can use, you know, like, um, depending on your access to data and your like latency requirements, et cetera, maybe train like a smaller M L model that can does this for you. Uh, but like, uh, truly like bring together, you know, a bunch of these different verification strategies and we'll get something that's more robust.

• 18:19Speaker 00

  So that's the philosophy behind like how guardrails tries to, um, you know, add guarantees around like what traditionally seem like harder problems to, uh, harder M L problems to verify.

• 18:29Nathan Labenz?

  Yeah, that's really interesting. Um, I'm mapping this onto my own use cases, uh. Here in real time. So I'm wearing my Waymark uh swag today. Waymark is in the video creation space. And we have, you know, it's a multi, excuse me, a multimodal problem, right? Where we ultimately take in some minimal information about a business and then ask the user to tell us what they want to create. And then on the other end, provide a ready to watch video. We kind of do that with like an ensemble of different models working together, but the core one is the language model. That writes the script and kind of gives the direction for what the visual assets should look like and all that sort of stuff.

• 19:15Nathan Labenz?

  And I mean, for one thing, boy, we've come so far in a year. You know, it was just a year and a half ago, basically, that I. That we got the very first fine tuned GPT three to do the task at all, you know, with any sort of like, it wasn't good, but it would at least like respect, you know, the nature of the inputs and outputs. And now today, GPT four can basically do it zero shot and like respect the outputs. And then three point five turbo kind of not really reliably. But then I think, geez, it is like 20 times cheaper.

• 19:55Nathan Labenz?

  So I guess I'm, I'm interested in kind of, what do you see the value drivers being for this sort of thing? To some degree, maybe like there's no other way to do it. But as I'm kind of running through my head, I'm like, so on some of these things like GPT four is just, is pretty reliable at this point. So, you know, people might want to do it for cost savings, or they might want to do it for, you know, I don't know, I guess a lot of different reasons, but cost savings and, and latency are actually one, I would see like benefits to that potentially if I could move to a little bit of a less reliable model, but still like know that my stuff is going to render in the right way for our users. So what, what are you seeing in terms of like the value drivers from your community?

• 20:39Speaker 00

  Yeah, yeah, I think that's a really great question. So I think there's two aspects of it. So cost and latency, you know, maybe being able to use like open source models or, you know, like cheaper models with like the same level of reliability, et cetera. I think like that is like one aspect. But the more interesting aspect for me is, you know, like, what is the task that you're truly using this model for? So, you know, like Wemark, there's a lot of use cases where you're using these models as maybe writing assistants or to, you know, like help you. One of the ones, one of the products I enjoy using is like Notion AI Assist, I think is what it's called, where it's roughed a little bit and, you know, it helps you like maybe shape up an outline, et cetera. So I think for those use cases, the creativity of a large language model is actually really good and it's a desirable trait.

• 21:24Speaker 00

  But I think at the same time, there's this whole other space of use cases that are, you know, like where, that rely on the, the world model that is, you know, like ingrained in these large language models and, you know, it uses these large language models as like software abstraction to do more general purpose reasoning. So I think there's a wide variety of those use cases that are seen. So for example, like one of them is that. Use an L L M to basically, um, you know, uh, as like an A I receptionist essentially. Uh, where you're a small business owner and you, instead of, you know, needing to hire a receptionist, you can use a large language model. Anytime you get calls that, you know, figures out scheduling, figures out who's available, et cetera. Um, so that's a very, um, powerful capability of that model where, you know.

• 22:08Speaker 00

  Creativity is maybe not as good a thing, right? Like you wanted to maybe stick to like some desired workflow. Uh, that you want to be able to ask, you want to make sure that maybe you don't ask for private information if you know, like a customer is calling, et cetera. Uh, so I think like where guardrails is like most useful is within those constraints where you use the L L M. Not just as a text generator, but as like a software instruction really. Uh, and when you're using it in that capacity, that is when like reliability becomes, uh, most useful. I, uh, like, I, I think over the weekend I'd, um, uh, shared something that. Uh, a community contributor had built using guardrails, which was, you know, this. Uh, GitHub action called Auto P R.

• 22:46Speaker 00

  Uh, and what Auto P R does is it, um, you know, uh, takes a GitHub issue and then automatically creates a code request from like GitHub issue like for your code base. Um, and that is, you know, that is one of those use cases that has like a bunch of really strict constraints, right? Like this files must exist. Uh, the code, the, the diffs that are generated must be valid for those files, et cetera. Um, and those constraints are pretty hard to, um. Um, enforce like without having some validation framework such as guardrails on top of it. Uh, so I think like for a lot of those use cases, you know, like creativity is like not as nice a thing and you want like more reliability. Uh, and I think like it, it's still like such a powerful use case of this model that it would be.

• 23:26Speaker 00

  Um, yeah, we would be kind of under like, under utilizing their capability if we don't build software such as that, you know? Yeah, it's, it's like, it's, it's a mix of both. I think even with. Uh, a lot of the creative, creative use cases, I've kind of found that there's still like a bunch of constraints that, you know. People implicitly have that, you know, right now, the way to do it is like be a prompt tuning or prompt iteration. But truly being able to, you know, like encode these constraints and like maybe. Uh, only like instead of like manually having to do this, you know, like have. Uh, a validation that runs and then, you know, only do it like when we did is like also a pretty nice workflow. Uh, where you, you, for example, you might not want any profanity in like scripts that are generated. Or you might, if you're, if you're creating like video content for someone, you might not want to mention, you know, like.

• 24:08Speaker 00

  Um, uh, peer products or like competitors, et cetera. And so I think like those kinds of constraints are also, you know, like useful even with free prompt test. So it's, it's, yeah, it's a mix of both.

• 24:19Nathan Labenz?

  It sounds like, though, you are most excited about something that could not be achieved in traditional code. And I still kind of am really trying to find that line, or like, I guess there's maybe just a lot of overlap. A lot of times when I get to these kind of puzzling moments as I study different aspects of AI, I end up kind of finding that it's like, sort of both in the end. And there's probably a lot of truth to that here, too. I'm thinking, okay, so we're in the GitHub, not whatever, the automatic pull request, Auto PR, it was called, right? I saw this. It is cool. And super technical, to some degree, those responses could be validated presumably by like, existing libraries. Like, I'm sure, you know, the Git

• 25:00Nathan Labenz?

  package itself has like, some way to sort of say, you ain't got your shit together. So, you know, this is not going to work. And I think that's kind of what a lot of people are like, naively doing, is, you know, just kind of implementing that stuff on a case by case basis. Like at Waymark, there is no standard, nobody else has our video standard. Like, we totally define it and own it. So, you know, it was up to us to figure out how do we represent that in text? And then like, what validation comes back from that? So I think one thing we could maybe have tried to do is, you know, we did this before we'd launched the project. But I kind of wonder, like, if you're advising us, you know, and

• 25:39Nathan Labenz?

  we were maybe a little earlier, how would you think about, like, what points of validation we might, you know, ought to do through guardrails? Like, should we be thinking about structure? Or should we be thinking, like, does this copy, like, satisfy the user's prompt? Or, you know, I guess there's probably a lot of room in between, right? Those kind of stake out the like, most rigid versus most kind of semantic, you know, desires or requirements from the model. It seems like you're more interested on some level on the semantic side, but that there is this kind of like fundamental interface with computing where it gets like

• 26:22Nathan Labenz?

  very syntactic as opposed to semantic as well.

• 26:27Speaker 00

  I think it's a good question. I do want to say before I say this, like I want to preface this by saying that like generating something and getting outputs, you know, getting feedback from the end user about, you know, is this good? Is this like, does this meet your original criteria and doing a more like qualitative assessment, uh, taking that feedback and, um, you know, going back to the drawing board. I think that's a very valid way of doing things, right? Like, I think, uh, for a lot of domains, you know, like getting that human feedback and getting that human input and write-off is like essential before you can like truly, um, think that this output is correct and is, is valid for, you know, whatever use case you have. But I think like for, like i- in the guardrails world, um, there's, there's basically this idea that, you know,

• 27:10Speaker 00

  maybe some parts of that human feedback can be done, um, by a combination of, you know, traditional ML, uh, heuristics and maybe more large language models in the loop, right? Uh, and if you are able to, you know, like take some of that, encode whatever that qualitative criteria is into, you know, like codify it into something that is more, um, more specific, um, what you're then able to do is, you know, like get, generate specific like failure, um, error messages, et cetera, that help the model output correct itself, um, right? So in terms of like maybe how that would like, I, I obviously don't know as much about remark, but what that might be helpful in is, you know, like maybe

• 27:48Speaker 00

  doing the number of like back and forths you have to do with your end customer because you're just able to take some of their constraints, you know, run those programmatically. If they're wrong, like automatically, um, um, uh, create like new prompts that, you know, tell the large language model why previous outputs are incorrect and get it to correct itself. Um, so I think this is like one of the net new capabilities that we have, you know, with large language models that didn't exist previously, which is, you know, their, their understand, like the ability to get them to like self-heal or self-correct themselves if you give them enough context. Um, and that is where like guardrails, like I think a lot of the, um, frame, a lot of the like core functionality for guardrails, you know, like, like really harnesses that, right?

• 28:32Speaker 00

  So what, how it, how it functions under the hood is like, it's really good at, you know, Um figuring out like what the relevant context is, you know, like packaging it up nicely, automatically creating a new prompt for you, um reaching, get like getting a new response from the large language model, merging that new response, you know, with the old response that you had, because it only, it's very efficient, it only reasks like things that are wrong, not like your whole previous output, etcetera. So it merges all of that together, um and then, you know, you're then like that is your corrected, validated response, sorry. So I think that is the world where like guardrails is most useful. Um and and like with that said, it is very domain specific, right? Like there are domains where um um let's say

• 29:15Speaker 00

  it's something like, any text that is generated must be funny. Um now that is something that is like next to impossible to validate, and I can't imagine like writing a validator that maybe assigns like a scoring function to humor, right? Like that's just very hard to do, and so you can't do that, and you need to have like a human group. Uh maybe there's like some other domains, right, which have like very high stakes and very high cost to like getting something wrong, where even if like maybe you can do some pre-scoring, etcetera, uh the actual like human, um you know, um um a confirmation that this output is correct is like essential before you can do another like iteration of uh with the large language model, etcetera. So there are those domains where you know this re-asking strategy doesn't

• 29:54Speaker 00

  work. Uh but I do think um just this idea of, you know, like taking what is correct, like what is correctness, um or or what does, you know, like an aligned output mean in your use case and trying to like codify some of that, um is useful, um both in terms of, you know, like prompting the large language model, it allows guardrails to construct like prompts that are, you know, more effective at getting you what you want, and then it also allows you to like post-shot validation. And this, you know, like the loop of, I want to systematically programmatically handle, you know, failures as and when they arrive, uh arise, and you know, maybe that involves like re-asking, that involves like filtering any like incorrect output, etcetera. I think this is a very powerful framework to like

• 30:35Speaker 00

  require less human supervision, and you know, like take a lot of that pain away of like, maybe going into chat G P T and like, you know, writing like this doesn't work for this reason, you know, trying to write something else. Um so that's, that's a world in which, like, that's the hypothesis behind guardrails, yeah.

• 30:50Nathan Labenz?

  Maybe it's just because we all have agents on the brain, but as I'm listening to you describe that, I'm really going to this agent, you know, moment that we're in and sort of really, when you dig into these agent systems, it's usually like, you might think of it more classical as like a multi-agent system. In many cases, like it's the same language model playing the role of different agents. We're seeing all these examples where it's like a simulated town where, you know, GPT-4 plays all of the people, or, you know, a research agent where there's like a planner and a coder and a, you know, retriever that all kind of

• 31:32Nathan Labenz?

  work together and have their own prompts and they're all kind of scaffolded together. And then, of course, you know, these things fail a lot because they are, you know, there's some probability of being wrong at any given point in the chain. And then you're only, in the naive implementation, you're only kind of as strong as the weakest link in the chain. So in a sense, I guess what I'm learning here is like, this guardrails paradigm is like connective tissue between the different roles in a like multi-agent system. And I should probably accelerate how quickly I think that these agents are going to start to work.

• 32:15Speaker 00

  I think it's a, it's, it's a, yeah, agents are very exciting. Like I, I think I've been thinking of them a bunch and trying to think about like, you know, how do you, how do you make them more effective, more reliable? I think the interesting thing about agents versus how guardrails works is guardrails is, you know, essentially the, the, the main problem that is always like, I want to add constraints on this output so that it works for my use case, right? It works for what I wanted to work on. But essentially, it gives developers a lot of agency to think about, you know, like their specific, the specific problems that they're solving and you know what, um, what correctness means to them. I think like contrasting that with the, uh, with agent frameworks where, you

• 32:56Speaker 00

  know, a lot of the goals of agents and a lot of the, uh, the tasks of agents are like configured autonomously like by a large language model itself. So you're in this like interesting setting where. The the involvement of a human or a developer in terms of, you know, like, your ability to kind of like enter something like an agent framework and like add guarantees, et cetera. You you typically like with the agent frameworks that exist today, you typically don't have access to that fine-grained level of, you know, like, um, a task execution or goal setting, right? Because you're you're not the person who's configuring these uh these agents themselves. So I think like there's this um, so so I think like that is the gap between like how agents operate today versus, you know, what what humans would ideally like to have, like ideally,

• 33:38Speaker 00

  uh, if you're if you're a person who wants to, you know, like employ a bunch of these agents and, you know, maybe do research for you, et cetera, you want to be able to say, you know, like, this is the set of allowable things that you're able to look at. Um, but also you want you want that to be configured dynamically based on like where agents are operating. Um, so I do think that like, so so that is a context. And if I were to summarize into like what that means for, you know, agents, uh, I do think like constraints are essential and like correctness specs are essential, right? It's the only way to think about like, how do you how do you assess what these large language models are doing? Like you constrain them and you you know, you evaluate them at each step to make sure that they're not going off the rails. But like, how do you do that dynamically when uh you're not the person who's, you know,

• 34:23Speaker 00

  creating these agents, setting their goals, et cetera? I think like that is the key problem uh to solve there. Uh, so yeah, it's a problem I'm very excited about and I think it's a problem that, you know, uh, willing to get solved before these agents are like employable, before you can truly use them, you know, outside of just like seeing how exciting they are. Yeah, I guess like for agents, you know, uh, I'm always very curious to see people's use cases. So like for Wemark specifically, you know, where, once again, where like in this domain, where, you know, there's like constraints and there's like a finite scope of like what you want the large language model to do, et cetera. Uh, if you've like looked at some of these agents and, you know, like, um, thought about like what is something that you would want to, you know, like add into eh eh, what part of, you know, the Wemark stack could be uh aided with like agents?

• 35:06Nathan Labenz?

  Yeah, it's probably not such a great fit, I don't think, for the Wemark product experience because.

• 35:14Nathan Labenz?

  We do have a lot of structure. I kind of think of these things as this is I'm still working out this framework, but I talked about this with Matt, actually, and this is part of what led to him mentioning your project. So we're kind of bringing it full circle. But I I kind of think of different mo I don't know if this is a spectrum or a binary or multiple categories or what exactly. But in terms of like how we interact with AI systems, there seems to be a like real time co pilot mode where you are doing stuff as a human and the thing is like there to kind of guide you, shepherd you, whatever. And then there's things where you're kind

• 35:55Nathan Labenz?

  of ultimately delegating more because you really don't want to do it. You know, you don't want to be the person in the driver's seat or the the entity in the driver's seat. You want to put the AI in the driver's seat, let it do it, and then look at its work once it's done, or you have some other sort of way to like circle that back into your life. And those workflows right now are largely done with like integrations, you know, various kinds that could be code, no code, you know, Zapier, what have you, with Waymarket's code. And it's a pretty guided experience where like, you know, you are delegating the task of writing a script, you know, choosing all the assets for your video, etc. And then

• 36:39Nathan Labenz?

  what you get to do is watch the output. So there's so much structure there that it's it doesn't feel like we really need an agent to like come in and, you know, mix it up too much. But I do see the agent as sort of the bridge between these two modes where you're like, in real time, I sort of want to send something off. So if I'm in ChatGPT plus today, I can like use a plugin perhaps to look for flights. But I really more might want the thing to be like, go book me the flight for, you know, whatever date like subject to my preferences. And ideally, like, it would kind of figure out all the downstream mess of that. So where I think this is actually really relevant for me. So I'm also working

• 37:24Nathan Labenz?

  with a company called Athena, which is in the executive system. I also talked about this with Matt. And you know, they have those kinds of things all the time. Where you know, a human today is like responsible for executing a lot of web tasks, you know, for a client. And there's kind of, you know, different aspects of the cognitive work there. One is like, understanding what's going on, like translating the language to and you know, the from the client. Like you got a request, right? What does it mean? Uh, understanding what it means. And then the other part is like being able to like actually hit the right buttons to make it happen. And ideally we delegate that whole thing.

• 38:07Nathan Labenz?

  But the agents, you know, that's where we in our kind of testing like along with everybody else. We've kind of found, yeah, it's uh, we may have an, we may have AI that can understand the request. You know, parse it effectively, you know, ask the right follow up questions. You know, or at least suggest some pretty good follow up questions. Demonstrate a really robust understanding of like what the human wants. But we're, we're falling down very much still on like, how do you actually, you know, hit the right buttons? Check out, I mean, God, God forbid you have to like do a payment, you know, process. Or like log into something.

• 38:42Speaker 00

  Yeah.

• 38:42Nathan Labenz?

  Like two factor authentication does still work as a deterrent to AI login. For the time being at least.

• 38:50Speaker 00

  Yeah, yeah. I think that's kind of been like my experience and, and my exploration as well, which is that there's, you know, there's this like, there's this interesting like wedge where agents can be useful, but in order for that wedge to succeed, you need this notion of like grounding, right? Of like being able to like, uh, figure out like, okay, here's how it understands what I'm saying. But like at each step of that execution, you need validation. So I think this idea or the, the problem of why you book me a flight is like very interesting because book me a flight like given my, given my schedule and given my budget and where I'm trying to go, right? And that translates into maybe like some constraints that are then grounded like based on your calendar

• 39:30Speaker 00

  or based on your, um, uh, you know, based on, uh, like where you're trying to go and like what flights are available. And so I think like validation. At each like decision that the large language model makes and each action that it makes like becomes, you know, more important uh for these things to succeed. Um but at the same time, like um you know, having having worked in like the self-driving world a little bit. Like part of it is like, are they able to do it? And part of it is like, can a human, um you know, this is like trust deficit, right? Um where uh even if even if these agents were perfect, there'd be this trust deficit between how much people are comfortable, you know, delegating. Uh and so even in terms of like building that, uh you need, you know, like um you need this like verification system essentially that makes sure that each step of the way.

• 40:15Speaker 00

  You're able to kind of like have control and have some oversight into like how they execute themselves. Uh so I think those frameworks would be kind of like essential before we're able to see their uh adoption, you know, outside of the outside of like demo use cases.

• 40:28Nathan Labenz?

  Yeah, it's funny that you say that. I would actually take the other side of the bet there when it comes to the human behavior. Yeah, I mean, if I understand you correctly, you're saying people won't want to trust these systems unless there's good guardrails in place. I think of you as saving people from themselves because I think people are going to be much more quick to just kind of go ahead and be like, yeah, it seems like it works. And why, you know, what's the worst that could happen, right? And I think we may find out, especially, you know, in a future world, that the the worst could be potentially quite bad, but

• 41:03Nathan Labenz?

  I do kind of expect that behavior. We I just recorded an episode of it um. A medical uh school professor at Harvard, uh Zach Kohane, who has just written this book, The AI Revolution in Medicine. And, you know, this is exactly, I think, the kind of system that he's looking to figure out how to implement into clinical practice, right? Something to kind of help, you know, for now, it's like, of course, the human is in the loop. That's the the official recommendation. But even then, you know, it's just so easy to get lazy. It's so easy to kind of be like overly trusting, especially when the models are getting so good. It's it's fascinating that you can get.

• 41:46Nathan Labenz?

  Better performance just by kind of pointing the model at itself. That's one of his, his big, one of his big findings was our current theme is that G P T four is better at evaluating text than it is at generating. And so, you know, he's starting to develop some of these like self critique things totally on the fly, like in the context of, you know, just exploration in, in clinical setting.

• 42:07Speaker 00

  Yeah, I think this, that's part of like what Garriel's like the validation system is also based on, which is that you don't like it's almost like trust with verify, like you get G P T to generate something with maybe. You don't have it, even, even if you are doing large language model based evaluation, maybe you know, they have a separate step that, that does evaluation separately so that you have, you know, that additional layer of security. And ideally, um, uh, this is, you know, back from machine learning, but like the more diverse, uh, you know, evaluators you have, the more kind of like guarantees are going to be. So just like ensembling is a technique that, you know, I'm, I'm pretty bullish on in this space. It's like, what's going to get us that confidence?

• 42:44Speaker 00

  But the, the medical, um, like A I for medicine, uh, or medical assistance is so fascinating because it's back to, you know, like reminds me of like, uh, once again, like self driving where, you know, like with the autonomous vehicles that are out there today, maybe like Tesla, um, F S D, et cetera. The idea is like, okay, you're, you're still supposed to be very alert, you know, you're not supposed to take, you know, maybe your hands off the wheel or something. I don't have a Tesla, so I don't know, but, uh, even within that, you know, there's this expectation that a human should always be, you know, aware and like present, et cetera. Right. But like as, as humans, it's so easy to get lazy. And so one of the really, one of the things I'm really excited about in the space is like good, um, you know, interface design so that like we are able to like get human involvement when it's most needed and like not, you know.

• 43:29Speaker 00

  Like if you're always notifying them like, hey, pay attention, pay attention, like maybe that, you know, like ceases to have an effect, right? Uh, so it's this idea of like, okay, how do we build this balance between like when human involvement is necessary versus like, you know, when we can like maybe offload some of that, do that, you know, more programmatically or more with code. I think like figuring out that balance and figuring out the interface to, um, to surface that balance or surface that like, um, a division of responsibilities to the human. I think like that is very, uh, very exciting and very, very excited to see like how that evolves.

• 44:00Nathan Labenz?

  I don't have a Tesla either, by the way, but a neighbor of mine does and has the full self-driving package. So knowing that I'm as obsessed with A I as I am, he was gracious enough to take me for a ride and. First of all, it works way better than people I think commonly realize. Like, I got into the car in front of my house. He put his finger on the screen, like, we're gonna go here and hit the drive button. And the car drove. Like, it was not a lot of fuss between just, you know, map, go, and you're riding. But when you talk about like the reminders, that's something I think they've really put a lot of engineering into as well. Like, there's like three progressions.

• 44:44Nathan Labenz?

  I haven't spent a ton of time with it, but there's a, you know, a couple of, I think it's three levels of. First, there was a little maybe visual indicator that like you haven't done anything to the wheel in a while. There's a camera also that watches you from the, from the rearview mirror in some versions. But then after that, it goes to a little sound. And then after that, it goes to a warning and it's like, we're gonna pull over if you don't, you know, keep your eyes on the road. So they're pretty far along in that. And I thought that was a really remarkable experience because it is a very delicate balance. So easy to tune out. Obviously, they have the hammer there eventually of like pulling over. And eventually, they also will kick you out of the program if you pull over too many times. He said, you know, if you're truly sleeping at the wheel, they'll, you know, retract your access to FSD ultimately.

• 45:30Nathan Labenz?

  But it is a fine balance, right? Because you can start to tune these things out. So I mean, how many mornings do we tune out? Crazy. So the next big thing that's kind of jumping to mind is, okay, you've got this developer, like, what is right to me? But the pattern seems to be very quickly like, well, why can't an AI just do that? So I start to then think like, is there a version of this that's like a plugin? Like, you've created a spec. And I'm trying to envision what is it like to use a computer, you know, a year from now as kind of the tooling and the plumbing all matures? One good candidate would seem to be like, it's a chat interface that you can like access the world through and delegate stuff to little adrians that go, you know, do stuff and report back.

• 46:15Nathan Labenz?

  It sure seems like a GPT-4 central process could like use the guardrail spec or something similar to, you know, kind of insulate itself from problems as it like delegates things off to the side. So.

• 46:33Nathan Labenz?

  Do you, I mean, first of all, do you see that as realistic? And do you have a point of view as to kind of how the computing experience might evolve in light of that?

• 46:42Speaker 00

  That's a very, very good question. I think I can go in like a bunch of directions like in my answer. But I think like my, my core belief like for that, um, in that whole space is I think like humans will be very essential in the loop. And so it will be very hard to automate like all of this, right? I think primarily because even what correctness means like very different things to different people in different contexts, right? And so, um, one common thing as an example is like profanity filtering. Um, profanity filtering is like one of the things that like most people can get consensus on is like we leave these, uh, if you know, you're generating facts, like make sure it doesn't have profanity. But I also chatted with people who are building, you know, like chatbots, um, where authenticity, where the chatbots are, you know, interacting with like certain audiences and like authenticity is essential, right?

• 47:27Speaker 00

  Um, and so if the, uh, chatbot is trying to imitate someone or like be in likeness of someone who does use profanity, uh, then you know, filtering out that profanity is actually detrimental to the user experience that they're building. I think it is very hard to figure out like what those constraints are on a global level. Um, and so I think like having, you know, humans and domain, uh, domain experts like be involved, uh, to developers to kind of figure out like, okay, you know, like think way, think ground out from like what they're building, what they want to use. For users to even think about, you know, like what is their desired experience?

• 48:01Speaker 00

  I think like those inputs will like continue to be very important and like there would need to be like some way to configure, um, you know, um, those inputs or those criteria or just that experience that like a user or a developer wants. Um, so I do think that like, I guess like offloading this entirely to the model or to like a provider, you know, is like going to be hard to be able to achieve like because of that, just because of that constraint. Um, but in terms of like what the programming experience, what the developer experience could look like, um, I think one way, and this is like purely hypothetical, I'm, you know, not, not making bets on this at all, but like one way is to like. Really start thinking like almost configuration first, right? Like there's like a configuration system that allows you to like, um, you know, um, uh,

• 48:46Speaker 00

  tune the outputs that you want to see from these L L M's. Uh, and even if the underlying machine learning model stays the same, like how that model is like validated, corrected, like how the outputs of that model are post-processed, like that is all what you can configure. And so when people are working with large language models, you know, it's not just text. It's also this configuration that they kind of pass in every time. Um, and this is like, I'm an engineer and so, you know, I, my, all of my empathy goes to like engineers whenever I'm thinking of like building systems, et cetera. So especially in engineering, like this is a pattern that, uh, I do think will be like very important where, you know, the prompt isn't sufficient by itself, you know, and just, just choosing the L L M and choosing, you know,

• 49:28Speaker 00

  temperature or something isn't sufficient by itself, but it's, you know, this configuration framework of like, like, how do I want this output to be like? And even like, how do I want my input to be, you know, like processed or formatted, et cetera? So, uh, there is like one pattern that I can see emerge, yeah.

• 49:43Nathan Labenz?

  Yeah, security also jumps out to me as like a big driver here because as I was thinking about, you know, where would I use this in all the, the things that I've done? And I'm kind of like, increasingly I see like G P T four just running pretty well and giving me like the format that I want. Um, certainly it has, you know, plenty of errors, but it very rarely goes like, you know, totally off the rails, so to speak. There are, there are a lot of vectors that are as yet totally undeveloped that seem like they're going to come online and, and cause a bunch of problems. So having this sort of S L A guarantee layer, validation layer, you know, seems really smart in light of like

• 50:29Nathan Labenz?

  prompt injection, for one thing, like as users get more, you know, generally sophisticated and kind of, you know, savvy in their adversarial, you know, attempts, like you gain a lot, uh, by having something like this, like implemented ahead of time. Similarly, you know, we've seen some really interesting things even in Bing where a user will change or, you know, not even a user, right? This is where it starts to get weird, like a site owner. We've just, we haven't even begun to see like what the S E O people are going to do in the, in the, you know, in the A I search era. So we, I mean, the battle between.

• 51:06Nathan Labenz?

  Search and SEO, the arms race there, I think is going to be made to look pretty pale in comparison, compared to what is going to happen now that you can like try to trick the language model at runtime with whatever kind of content. Then there's all these like, you've got model risk too, where you really probably do want to, as awesome as GPT-4 is, and my general strategy right now is like use it for everything, you know, get the quality to an acceptable level and then think about, you know, maybe taking out cost or taking out latency opportunistically or as necessary or what have you. But that's going to come in due time too. And then people are going to be like,

• 51:47Nathan Labenz?

  well, you know, what about like, I heard like alpaca whatever was just as good and, you know, we can just sort of drop that in there. But now you're just in a world where you have no idea what's going on, right? I mean, your OpenAI, your Anthropic, they have a certain SLA. And I think one huge misconception that people have is like that that SLA sort of is inherent property of language models, when in fact like it's not at all. And they've worked really hard to get to the level that they have and like, you know, you definitely cannot take for granted that your, you know, bootlegged llama, fine tuned on whatever is going to be, you know, anywhere as safe or friendly to users.

• 52:29Nathan Labenz?

  I don't know, I'm going on and on, but that seems really important, but it's all that stuff is just starting to kind of emerge from the mists. I guess how much did that motivate you? Do you see other things like that? What do you make of all that kind of emergent security stuff that you're you're kind of helping people get in front of?

• 52:45Speaker 00

  Yeah, yeah, I think like, I think security is, you know, a big, big operational risk of, you know, working with these models, especially like prompt injection. I think we all kind of, you know, when Bing Chat was released, I think Sydney and you all kind of saw how easy it was to, you know, kind of manipulate these models in like specific ways. So.

• 53:07Speaker 00

  Yeah, I think like my, like how I think about this is very okay, like how do you decompose the problem, right? Um, I truly think that um, this isn't a problem that will be solved by machine learning specifically. I think like, you know, um, a lot of what OpenAI has managed to achieve is, you know, built on like this concept of like scaling laws, where you know, like as you scale the data, et cetera, you just kind of start to see in scale the models, you try to start to see, uh, you know, emergent properties. But um, this is, you know, just from like all of my experience working in machine learning, like there's, uh, these are fundamentally stochastic systems and it's very hard to like, there's a, there's a big long tail of, you know, like, what these stochastic systems like can't have guarantees for, right?

• 53:45Speaker 00

  Like you cannot have guarantees, or you cannot have data points for all of the different exciting and weird ways that people are going to use these models. And so as a consequence, it's very hard to like add that validation like from the model piece itself. And so, security becomes, you know, an essential thing because that isn't something that you can leave like up to the stochastic system, right? You essentially need to have like, um, more determinism like around it. And so, my way of thinking about it is to kind of like break it down into like, okay, the model is stochastic. Like what can we do around the model that then adds that, you know, those like, um, those like watertight guarantees? So for prompt injection, for example, like the um, things that are really exciting to me are both like on the input and output side, right?

• 54:29Speaker 00

  You sandwich the L L M A P I call with like, you know, input validation, output validation, to essentially ensure that you have like multiple layers of security that, you know, make sure that your model isn't behaving in ways that you don't want it to. Um, so some of the ways I, I like, you know, familiar with like some of the, um, more exciting like, uh, new developments in, you know, like protecting against prompt injection. But also, you know, on the, um, on the output side of things, like I think a lot of those, uh, corrections are like an input validation. But like on the output side of things, if you know that there's behaviors that you don't want this large language model to exhibit, it's possible to do that, you know, as like secondary checks on in terms of output validation, right? If you want, for example, um,

• 55:11Speaker 00

  If you have like known patterns of, uh, you know, prompt injection that people tend to follow, you can do that on input validation to essentially make sure that, you know, there's not, you're only, uh, you're almost like gatekeeping, like queries and interactions that users can have to the set of things that you can kind of support. So I think like solving, i- it's, it's back to this problem of like, kind of what are the, um, like decomposing this whole problem of security into, you know, specific domains, specific applications, and then for those domains, thinking about like, what is the, um, uh, just like, what is the end goal that I want my users to have? And then, you know, like adding constraints to filter out like everything that is outside of that goal, that is not kind of related to that goal. Um, so like, I would, I would be very surprised to see if we,

• 55:56Speaker 00

  I, I, I, for example, I chat with a bunch of teams who have been doing this. And like, I think that's a common design pattern, right? Like where you can't simply rely on the L L M. Like, there's a reason, for example, like we don't have like end-to-end machine learning systems, um, right? Like you, you kind of have like, um, some sub competent be machine learning and then some checkpoint that has like either human or like more deterministic, um, you know, a component like verifying or validating the output of the machine learning system. And then like another downstream thing that may be machine learning, but, but decomposing it and, you know, like adding, um, adding different layers of security on either end of like the M L system is like, yeah, the, the pattern that it, uh, that most people, when they try to like production these things, we're kind of like seeing happen.

• 56:38Nathan Labenz?

  Security in depth, defense in depth is definitely, uh, I think could have become really important. Another version of this too is models talking directly to other models in vector form. Um, seems like it's going to become a huge trend. Uh, we had the authors of Blip two on the show a while back and, you know, that I think of that as such an emblematic. Foreshadowing of what's to come where they're able to take a frozen vision model and a frozen language model and very quickly train a connector model that essentially converts

• 57:19Nathan Labenz?

  the encoding of the image to something in some unspeakable, literally unspeakable thing in the language embedding space in a way that then allows you to have like dialogue with the language model about the image.

• 57:36Speaker 00

  I actually love that, yeah, sorry, I love that example because this like literally is like research that I've done, you know, in my masters and that, you know, I've worked with some collaborators and I have a paper about this that does like these kind of joint embeddings in language, in vision space for, I think we did this in the fashion domain where you have some fashion embeddings and then some texts associated with it. How do you project it into a single space or into a single like, you know, projection space so that you're able to like figure out similarity between, you know, like essentially like using language, figure out what are similar images, et cetera. So it's a, yeah, it's a body of work that is like pretty interesting and like opens up like so many, so many cool capabilities.

• 58:18Nathan Labenz?

  I mean, the fact that that's going to work seemingly across, you know, I'm still educating myself on this, but I really just see so many examples of like even just simple linear maps from one space to another that people are kind of able to, you know, now bridge these different encoded or latent spaces, whatever you want, you know, embedding spaces. I think that's going to be a huge trend that will bring a lot better performance out of a lot of systems because like, why would you go through this natural language bottleneck if you have the, you know, a much richer representation of an image or for that matter, a medical scan or, you know, a sound or whatever.

• 59:02Nathan Labenz?

  Like you have the sound of the bird in an audio file, you're not going to project that down into language long-term, you know, and be like. A sound of birds is heard or whatever, and then feed that into the, you know, the language model. You're going to figure out how to represent that in the language model's space, but in like a way that is truly unspeakable. But I just see so much force going in that direction. But then the obvious worry there is like, well, now you've opened yourself up to God knows what, you know? Like the space of possibility there is so vast, it's totally untestable. It's like incomprehensible on the input side. So what can be done about it? I think, again, this is like a really good answer. Like, start validating your outputs, people. This is really, this is really important. And it's only going to get more important.

• 59:47Nathan Labenz?

  Like you could, I think one thing people are probably really underestimating is they can create a system today that behaves pretty predictably. And the world underneath it is going to change in such a way that it may start to expose their vulnerabilities over time. I mean, in some sense, software kind of always works that way. You know, we've seen like Windows patches way, you know, way downstream after launch. But I don't know, this seems a little bit different.

• 1:00:12Speaker 00

  Yeah, I think, I think it's a very interesting point. I do think, I, I think it's like this fundamental idea of like, you know, more sensory, like input from more sensors. If you're able to have that and code that, I think that just leads to like better performance. Uh, and it's also like going back to this idea of like grounding that I was talking about earlier. Like you have like some M L system that like projects it in some space. How do you make sure that that projection is correct or not? And, and if you have this idea of grounding where you have like, you know, this other sensory input that is also supposed to be projected in something, you can use that as like a, as, as we, like you can use that as like self correction or like self verification systems, right? So we kind of like saw this like while I was a, um, you know, a driveway AI, which was the software I started that I worked at.

• 1:00:55Speaker 00

  I think we had this where, uh, the performance with using just LiDAR versus using, you know, like if you have cameras in the, as part of your inputs as well. In addition to LiDAR, you just kind of like have better, a better understanding of the state that the car is in. Uh, and you're able to, you know, kind of like have better, just, um, just make better decisions overall. So, um, I think it's a, it's a, in my opinion, I think it's a net positive thing because it allows us to also like enforce these covers, you know, across like multiple dimensions, right? Because, uh, it allows us to kind of figure it out, like where things might break down and then add like checks in the systems there. Okay. So, yeah, I'm, I'm excited to see a world where we can have like all of these vectors embedded in the same space. And, and there's a lot of like interoperability between them.

• 1:01:39Nathan Labenz?

  So essentially, you're describing a validation step that is in coherence of

• 1:01:48Nathan Labenz?

  like semantic interpretation of different input signals. Trying to think of any example of that that I've seen in the wild. I guess you're pointing to one in the self-driving car area. Maybe I want to just kind of throw a few other kind of safety, control, et cetera, sort of paradigms at you and just have you react to those, not in a like better or worse sort of way. I mean, I think, obviously, again, defense in depth, like there's, there's a place for probably all of these different things. Maybe in just kind of comparing and contrasting some of these other approaches to what you're building. Again, it can help enlighten people as to, you know, how you're thinking about it and how, you know, what kinds of use cases, you know, your project is best suited for. So I've got a handful.

• 1:02:32Nathan Labenz?

  One is the, like, most people listening to this will be most familiar with OpenAI, obviously. So they're moderation endpoint, which briefly, you can like take an output, bump it up against the moderation endpoint. It gives you like a flag of, hey, you might have one of these like finite number of, you know, problematic content types. And then I believe you can really do with that what you will still. I don't know that they even enforce any, you know, particular action downstream of the moderation endpoint. But essentially, you're classifying stuff into, you know, into one of these categories. That's obviously pretty simple. So where do you think that kind of falls short?

• 1:03:14Speaker 00

  I think it's great. I think it's very finite in scope. So if you were to take something like that and then expand that out into, you know, like any use case that you want to verify or validate, but you know, that, that is like at least programmatically verifiable or verifiable with a machine learning model. Flag specific things within that output that may be problematic for that. And then also allow you to configure how you want all of those, you know. Invalid outputs to be dealt with, right? I think that is how I think about Godreal. So moderation, so if you think about moderation, I mean specifically like profanity filtering. It's either like one of the validators out of like a number of validators within the library.

• 1:03:58Speaker 00

  So it, you know, you, you take some L M output, you give it to that profanity filtering validator. It'll tell you whether it is profanity or not in your L M output. And depending on how you configure that validator, it'll also like correct that for you. So for example, generate that text without profanity. Or you know, maybe just filter those sentences that have profanity in there. But you can just take that pattern and apply it across a bunch of other use cases. So if there's code that is generated that you know, maybe, maybe is incorrect or you know, is not executable. You can do the same thing for code. If there's like summaries that are generated from some source text, but you know, those summaries are incorrect or invalid for some reason, you can do that. If there's, you know, like structured data that you've extracted, which is like specific parts of it are incorrect. You can basically apply that same paradigm.

• 1:04:40Speaker 00

  So it's taking that moderation and fine, but you know, making it like a very general, very accessible thing that you can use.

• 1:04:46Nathan Labenz?

  In practice, what do you see people doing most to fix on fail? Like you gave a number of different kind of choices there where one could be like, you know, popping outright error. The other could be like rerun the whole call and just like hope for better the second time. But you're getting pretty granular in between as well with like maybe just fix a little bit or like snip the profanity. How do you, how would you like advise people to think about which option they should take in that moment of a problem? And what do you see people actually like doing today in the community?

• 1:05:22Speaker 00

  I think that's a, that's a hard problem to answer because like I can't tell people what their, um. I guess what their pain tolerance is like for their applications, right? Like I think fundamentally, um, you know, there's like some cases where you're like, oh yeah, if this, if this doesn't pass, like this. L M output is of no use to me, right? And so I want it to either be corrected or just like filter it out. I can't use it in this like in between state. Um, and so re-asking is like one of the things that was, that is pretty valuable. Like it, it allows you to, you know, kind of like just, uh, from the user perspective, it's like a sing, like I don't want to say one shot because it. There's like, it's an overloaded term because you know, ML, one-shot, et cetera, but like, it's like from a user perspective, it's a single API call that you know, like,

• 1:06:08Speaker 00

  on the backend might make you know, like, multiple API calls, et cetera, to the large language model, but like, give you like, either a collected output, or it'll tell you like, hey, this is just like incorrect and I can't handle it. So, so for very important use, or for very high stakes use cases where if a validation check fails, you can't use it, I think like, re-asking is probably the most effective thing that I see people use a bunch. Uh, I think like filtering is another one, so specifically for like summarization or for profanity, et cetera. If you know, there's like some sentences that don't, um, you know, uh, aren't information dense or that have profanity, you would be able to, you know, have like that granular control of like filtering out those specific things.

• 1:06:49Speaker 00

  Um, and I think like another one, which is like the default setting if you're using guardrails is like a no-op. So essentially you would still run all of the validations on an LLM output. Uh, but you would only like, if validation fails, you wouldn't do anything, you would just like return the output as is to the user. So same as if they were using, um, an LLM without guardrails, but it would just like log everything that went wrong. Um, and you know, you can access that log and like figure out, you know, do I want to like, do I want to iterate on my model or my prompt or something using that? Um, so I think those are probably some of my favorite ones, but like, there's also like raise an exception or like deterministically fix it if it's possible to deterministically fix. So, yeah, there's a bunch of, there's a suite of options within the, uh, within the framework.

• 1:07:34Nathan Labenz?

  OpenAI's, we'll go with OpenAI's first. They also recently with GPT-4 launched this evals library. I'm sure you've studied that a little bit. I guess my surface understanding is that's more of like a ben- architected, it's like a benchmarking suite versus like a runtime aid. But maybe, you know, what, what have you learned or what, what did you think was smart in the evals, uh, implementation or approach to?

• 1:08:03Nathan Labenz?

  Validating language model outputs.

• 1:08:06Speaker 00

  Yeah, yeah, I think my my I like evals, I think my interpretation is the same, which is that it's like this offline, you know, benchmark that allows them to internally test out um how well um you know their LNs are doing for for tasks that people care about. And so I I think like one of my favorite things about it is just the um I guess this is maybe not technical, but maybe more from like a product or go to market standpoint. I really like how they um it's almost like this way to crowdsource, like how people want to use their large language models. Right? Like all of the ways that people are using it and like all of the ways that it's going wrong. Uh take that and almost make it part of your like your training data set or your evaluation framework so that the models themselves are

• 1:08:48Speaker 00

  you know, are are trying to serve those use cases better. Uh so I think that's probably my favorite thing. Uh I've seen like other folks use it to just like scour through it and maybe figure out, you know, interesting prompts uh that they can borrow. So uh I've I've like uh seen some people do that, which is pretty interesting, which I think is like a nice way to like mine that library. But uh yeah, that's probably the um I I see it as a totally different framework, basically.

• 1:09:13Nathan Labenz?

  I thought it was a nice touch also that they tied uh or essentially created a a separate lane for GPT four API access for folks who contributed to the to the evals library. How about like Anthropic's constitutional AI approach? Again, it seems like there's a certain that's obviously a training protocol as compared to a runtime validation, but do you see commonalities there or have you taken inspiration from uh their approach?

• 1:09:42Speaker 00

  Yeah, the constitutional AI stuff is interesting, but uh I think it's back to like some of the some of the things I said earlier, which is that like it is essential to, you know, have like deterministic checks, right? Deterministic post hoc validation, because like just by training itself, like the models are not, you know, super sufficient. So um I think like Yeah, I think like runtime verification, runtime validation is like the most exciting kind of problem to me. So that's how I think about it, but it is, it is pretty interesting. I think this other aspect of it is just like configurability, you know, so also a similar thing that I touched on earlier, but being able to configure, you know, like what, uh, what correctness means to you and like enforce that specifically,

• 1:10:24Speaker 00

  rather than, you know, how it is globally understood and redeveloped, like standard of correctness. Um, so I really like that world where, you know, like developers like have that agency to configure what that means to them. So I think that's also being to do that in this post hoc setting, where you don't have to go and train a model every time your definition of correctness changes or every time your use case changes, I think is like, you know, um, it, it like opens up access um, uh, to a much broader audience.

• 1:10:50Nathan Labenz?

  Yeah, I feel like there's a, there's an opportunity though to sort of bring the constitutional critique to runtime. I, I listened to the, um, the guy, his name is escaping me right now, but one of the authors, one of the lead authors on the diplomacy paper, the Cicero model that came out of Meta, he was describing a general strategy of like, how can we bring more compute forward to runtime? Like the old systems of like, you know, Deep Blue in chess was like super intensive, you know, compute at runtime to the point of being like borderline largely a tree search, uh, more so than an AI, or as we think of it today anyway.

• 1:11:33Nathan Labenz?

  But like with our language models, you know, it's not been quite as obvious how to do that. And it seems like, again, this, this sort of framework of rather than lean on all that embodied compute and training, like bring some of it somehow more forward to runtime and, you know, apply like.

• 1:11:54Nathan Labenz?

  All these checks that essentially like the more compute you can spend like the more likelihood you have of of getting like really good output essentially is is his mega, you know, observation. You talk about like deterministic, but some of these things, they're not, they're still not really deterministic, right? Like if I, the more I critique, like I still have this kind of inherent, you know, it's layered non-determinism, right? More than, which hopefully kind of limits my my problems, but still there is some like, in most cases, right? There's still some amount of stochastic stuff that is not like reduced.

• 1:12:31Speaker 00

  I think that's a very good point, and I should, I should, I should like clarify, um, you know, the, I should, I not clarify, I should qualify what deterministic means. Um, so I think there's, uh, there's essentially different ways to like evaluate or to to validate outputs, right? And and I think like inherently, like basically one of those ways is, you know, like taking an L M output, creating a new prompt that, you know, asked G P T four again, like, okay, is this output correct or not, given these criteria, and maybe give me like a yes or no, or like maybe some some binary response that allows me to assess this, right? And that is like one way of doing validation that is not deterministic at all, uh, but it is, you know, like additional layer of security. Uh, but I think like the, there's also like

• 1:13:16Speaker 00

  a bunch of other techniques and like, um, uh, yeah, other like validation rules that people can use. So some of them are more kind of like rule based, um, or heuristic based. Some other of them are like, you know, not using, um, L L M A P I's, but using, you know, like maybe smaller, high, high precision models that are, you know, like trained on like subset of data. So even if they're not deterministic, they're A like trained on your own data. And because you have control over the model, you can, you know, maybe do things like set random seeds, et cetera, so that you essentially get, even if it's an output that is, you know, like generated by a machine learning model, it's an output that you have control over and you can control its randomness on, right? Either in terms of like changing the seed or changing some of the parameters, or just like being able to like generate more training

• 1:14:00Speaker 00

  data. So that, uh, both of those frameworks are, you know, like, One is like purely deterministic and then one is also like more more deterministic in terms of, you know, being able to look for randomness. Um, so when I say deterministic, I I I'm truly referring to, you know, like an ensemble of all of these techniques. And so some of them are, you know, basically all over the map in terms of like what their randomness kind of looks like. But that was a good, that was a good like little cash. Yeah.

• 1:14:26Nathan Labenz?

  One obvious question from a developer's standpoint is what overhead does this create? And, you know, you could measure that in a number of ways, right? Like token overhead, which is cost overhead, maybe like complexity overhead, latency, you know, increase. Uh, and then maybe some like just tensions also in a product experience. Like I think about the Bing experience where fascinating, honestly, that uh, that one of the biggest corporations in the world went with this as their approach.

• 1:15:02Nathan Labenz?

  They spit out the token, you know, in a streaming token by token way and then retract it from you if they determine that it went off the rails. So like, I mean, that is crazy to me that Microsoft like launched with that paradigm. Uh, I I do get why, of course, because like people don't want to sit there and wait for the whole thing to be generated before they can start to see what's happening. There's a really powerful draw to the streaming experience. Uh, so much so that, you know, Microsoft did this, but like it's crazy that, you know, the these uh, the biggest corporation, the one of the biggest corporations in the world is willing to knowingly like set up a system where they're gonna, yeah, sometimes emit some, you know, toxic content or whatever. And then, well, I'll just kind of swipe that.

• 1:15:47Speaker 00

  Swipe it off. Yeah. Yeah.

• 1:15:49Nathan Labenz?

  We're in a brave new world, uh, for sure.

• 1:15:54Nathan Labenz?

  So, yeah, I guess, you know, all those dimensions, how do you think about kind of overhead? And, you know, again, how would you kind of guide developers toward? Minimizing that overhead, are there things that could be done in parallel? Or you know, what's what's kind of the smart version of this so that you know, the the because a lot of people are going to be like, okay, yeah, but the CEO says we can't wait for that.

• 1:16:13Speaker 00

  Yeah, I think it's really, it's a really good question, honestly. Interestingly, before I before I started, you know, talking about how I think of overhead, you would be surprised by how for specific applications, like if you're not in the chatbot world, like how comfortable people are with latency, as long as you know, if they're getting like high quality output at the end of it. So I, when I was kind of initially building this, like latency was a concern and like some of the design decisions, you know, support that hypothesis, like re-asking, for example, you know, it does not piece me of re-asking, like aggregate stuff and then re-ask, so it's a one-shot kind of like request. But but people are pretty okay waiting for, if they're in this world where correctness

• 1:16:57Speaker 00

  matters to them, people are pretty okay, you know, like attacking on some additional latency in order to have that correctness. But with that said, I think like overhead is a pretty important question. I've actually found like token overhead is an interesting one because I found like this interesting balance between, you know, like how efficiently can you like write a prompt if you're using the structured prompting strategy that CardReel offers versus how efficient are your prompts when you know, everything is in words, right? And so like for some use cases, interestingly, I have seen that like being able to structure your prompts, you know, even if you would think that it's more tokens, it's actually

• 1:17:39Speaker 00

  more efficient because like all of that structure and all of that, those constraints, you know, which are like now maybe in symbols or in some domain-specific language, end up, you know, being represented in like words, which is way more, you know, just way more expensive in some cases, right? And and that's a similar, like, I've seen that with complexity as well, where if you're trying to get your LLM outputs to be like structured in some way, or to be, you know, to have some behavior, et cetera. Right now, the only way to do it is, you know, to like prompt it yourself and like do a bunch of prompts, et cetera. But Guardrails kind of like abstracts some of that complexity and some of that, you know, some of that exploration that you have to do away from you, right?

• 1:18:20Speaker 00

  Because here's this like, um, here's this DSL that like is tested and like works across a bunch of these LLM providers. And you essentially, as a developer, you don't have to think about like, how do I write a prompt that will give me this output that's structured this way? Like, I just, you know, I just write it in this like known way. Um, and that, you know, is like almost easier in terms of development. Like, I have, um, tweeted this, uh, some time ago. And, um, also a club for my Twitter, follow me at, uh, Shreya R. if you, if you don't already. And I basically talk about guardrails and NAI all, all the time there. But, um, I tweeted this, which is that like, even if you're in this setup where you don't care about validation, if you just care about like getting structured outputs, like it, it's a pain to kind of like think about how to do that, right? So just use guardrails for like getting those structured outputs across like LLMs.

• 1:19:05Speaker 00

  And like, it's just, it's useful in and of it, in and of itself, right? Because you don't have to like write prompts and like iterate on prompts and so on. So.

• 1:19:13Nathan Labenz?

  How, how people understand a little bit more like that complexity that you're taking out? And, and how you're doing it on the just first question of like, can I get the desired format back?

• 1:19:26Speaker 00

  Yeah, good question. So, um, I think how guardrails kind of like the entry point for developers is like creating this spec, which is, um, you know, a markup language, uh, where you're able to specify here's my, here's like an output schema, right? So if you're, if you, if I want an output, here's, you know, like all of the different components that I want in that output. If that's a JSON, then you're able to kind of configure that. If it's, you know, just a string with like some additional validation on top of it, like it's just a snake or string that you're generating. And you want some additional validation, you're able to kind of configure that, right? But in this case where, um, but, but that is like totally separate from your prompt. That is just you thinking about like.

• 1:20:06Speaker 00

  What is the output that I want and like, how do I write that output in like a schema definition perspective? Um, in the real, so guardrails basically has real specs, reliable AI market language, uh, that's the specification framework. And so in a real spec, in addition to the output schema, you can also have like a prompt separately. And then all you need to add on the prompt essentially is like the high level task description, right? Because everything that describes like, make sure my output is this way and make sure my output has these constraints and make sure my output is formatted that way. All of that you do in a schema, which is like a specific, like it's a programming language, right? So you can think of it in terms of like, if you can write like XML or you can write markup, you can write that and you don't have to convert that into English.

• 1:20:48Speaker 00

  And then, you know, do a bunch of experimentation to make sure that your L M A I will like, understand what that means. Um, so it, it like, it allows people to like, basically, yeah, it allows people to only think about like the pain of prompt engineering essentially goes away a little bit. And it allows you to think about like interacting with these L L M's in a more programmatic fashion, um, to some, to some degree. So that's, you know, that's the complexity that gets, um, that guardrails kind of takes on. Um, so one of the exciting things about that is that the contract between the user and uh, guardrails is basically the spec, right? And then it's guardrails's job, um, to translate that spec into a prompt.

• 1:21:28Speaker 00

  So the user gives, you know, like a high level prompt and everything, but you know, the output schema definition is translated into a prompt, uh, based on guardrails. So, um, if you work with these L L M's, you would know that, you know, they basically like change, the, the version internally gets updated like all the time. Uh, and as, as part of like the, the behavior of the large language model changes, right? And experience some of that, uh, as I've been building this library, but you would essentially kind of find that like, even if the model version changes, as a developer, you have the same, uh, rail spec. And then guardrails just like compiles that rail spec into different prompts, uh, depending on, you know, like what is most effective, uh, for a specific L M.

• 1:22:09Speaker 00

  So, so I think like that is also, you know, like from a developer point of view, you don't have to, you know, um, uh, wrangle like model version updates and like model quality issues to some degree.

• 1:22:19Nathan Labenz?

  That, that sounds really useful. I was just doing some stuff yesterday where I'm doing some task automation. And I use the format trick, right? Everybody kind of who uses language models much comes across the format trick, uses format in the response. As far as I know, that came out of OpenAI. As far as I've been able to track that back through Riley Goodside, you know, who like very much popularized it. He attributes it to Boris Power, who's at OpenAI. Before that, I don't know. Maybe Boris just came up with it on his own. But I do run into this stuff where I'm like, well, how intricate do I really want this format to be? A lot of times I'm just like, use this format, just like XML open tag, content, XML close tag.

• 1:23:03Nathan Labenz?

  That way I can at least kind of easily just use a regular expression, just kind of parse out whatever's within those tags. And then if it has a prefix or a suffix, or an I hope this was helpful to you, you know, whatever, I can like get rid of that cruft and have what I want. But then I usually haven't gone that much farther than that. Because I'm like, then I'm getting into like hand coding XML and like the playground or whatever. And that's not great. So right off the bat, I think that is maybe something we underemphasize at the beginning of the conversation is just like, there's just a straight convenience factor associated with, at least from a developer standpoint,

• 1:23:45Nathan Labenz?

  being able to write something in a little bit higher level abstraction and have the thing kind of translate to, you know, a more detailed prompt that will get you the parseable result that you want.

• 1:23:59Speaker 00

  No, yeah, totally. I kind of found that as well. When I was doing a lot of my prototyping, I prototyped it with GBD three. And then I had a similar experience with, I think anybody who's worked with GBD three and three point five knows that, you know, you have like, here's the output you were looking for and like, hope that was helpful, et cetera. Like all of these statements, right? Which are like, oh, just give me the JSON, which is what I want. And then I think like with Godreals, since release, I've had like a community contributor actually push this effort to, you know, like add instruction tags and do a bunch of experimentation to make sure like what works well with GBD three point five. And so from a developer's standpoint, you write like one spec. And then depending on if it's GBD three or three point five, you know, and basically like compiles it a little bit separately.

• 1:24:44Speaker 00

  So that, you know, it just, it just works. Like I think even with GBD three point five, I've had a bunch of people testing it out and they just get the, get the thing that they're looking for without a lot of that. Um, extra, um, you know, filler tax that maybe that makes it not work as well.

• 1:24:59Nathan Labenz?

  Yeah, it's interesting that you're that you are also then translating that to other models, which the angle that I was going to mention a second ago was it seems like there's a dynamic here where as people kind of try to think about like, where's all where's all this going? You know, how is the market going to shape up? Who's going to lead? Are we going to have like one AI to rule them all? Or is it going to be, you know, many, you know, or so an oligopoly of large language model providers? Oh, my God. So where's this all going? One force that I see kind of pulling everybody in line with OpenAI is just the fact that all of these things are getting developed against OpenAI's state of the art behavior at the

• 1:25:41Nathan Labenz?

  time, almost like across the board. And so then if you're a different language model, large language model provider, you know, if you're an Anthropic or a Cohere or a, you know, Google or a, you know, Alpha out of Europe or like whatever, then it seems like you have a really strong incentive to basically try to be as much like OpenAI as possible when it comes to like supporting all these things. But then that kind of just sounds like, well, then how are you going to compete on that, right? If like, if you've got to expend all this energy to basically just make sure that like people even can switch to you without things that they're kind of currently assuming will work breaking.

• 1:26:27Nathan Labenz?

  You know, that's not a great position to be in, you know, to be spending all your time just like trying to catch up in that way. But I don't know that there's any way around that for other providers. So do you see that dynamic similarly? And do you, does this experience like lead you to think that we may see kind of concentration, if not necessarily providers, at least in terms of like? How language models behave?

• 1:26:53Speaker 00

  Yeah, I think it's a really great question. Um, I do think I'm I'm pretty like, I personally believe that. Um, there would be a big diversity in terms of model providers. I think like. We are in this almost, it's it's kind of insane to see just from like, I I worked in a mall my whole life, but. Um, or my whole adult life, but um, but it's it's kind of insane to see just the amount of like. Activity and excitement around the space. Like there's, you know, people. Training and fine tuning deep learning models that you know weren't even in the space like a few months ago. And that's really awesome, right? And that. That just that, just that amount of like demand like makes it so that there's, you know, more need and more providers that will come up.

• 1:27:33Speaker 00

  Um, I had like people who were like, oh, I um, I really like guardrails, I really like opening high, but it's just. Too expensive for what I'm trying to build. And so I, can you make this work with, you know, an open source model as an example? Um, so I I do think that we're going to see a lot of that that proliferation happening of, you know. Great performing models from like different price points, different latencies, different providers, et cetera. Um, so I'm pretty, pretty um, excited about that world. I think the other interesting aspect of this is that like, because. Um, a lot of this was unlocked by OpenAI, we're in this space where um, people are, you know. Building these frameworks, et cetera, around what is the most performant. Um, provider, what is the most performant model?

• 1:28:18Speaker 00

  And because like those frameworks are maybe like indexing too high, highly on OpenAI, like there's, you know, this incentive for. Other model providers to, you know, offer interoperability with with for those specific functionality, right? So I do think that like, um. Uh, the the standards are evolving at the same time as like the models are getting better. So there's this like interesting kind of dynamic between them. So. At least for the foreseeable future, there's like this big incentive for other other model providers to also, you know. Provide some more functionality or at least not have regressions, right? So that incentivizes people to, you know, like switch over to them. Um, for a variety of reasons, but I do think that like. Just the pace of innovation means that, you know, there will be um, some um.

• 1:29:03Speaker 00

  Just like a more level train field at some point and then it'll be very interesting to see like, what are the specializations that different, uh, different model providers offer, or even like the different open source models, like what that specialization is.

• 1:29:15Nathan Labenz?

  Okay, here's my three quick, you can answer this quickly or not at all, whatever it is you want. Uh, but I typically, if there's time, I usually ask these three final questions. One is favorite AI apps, experiences, tools, whatever that you are loving that you would recommend to others.

• 1:29:31Speaker 00

  Uh, I'm, I'm gonna be very basic and say Copilot. Um, I think Copilot is awesome. I have, I use it extensively and I recently like, um, had to set up my dev environment from scratch and wasn't like, didn't have Copilot and I was like, what am I missing? What is, why is my, why am I coding slower? So yeah, definitely Copilot.

• 1:29:51Nathan Labenz?

  I can totally relate to that. When they went paid and it stopped working, I was like in a panic until I realized that I could just pay for it and make it come back. Uh, but yeah, I was like, I'm not gonna be doing this without this now, am I? That's like, uh, not to be endured. Uh, okay, second one. Hypothetical situation.

• 1:30:13Nathan Labenz?

  Sometime in the future, a million people already have the Neuralink implant. If you get one, then you can control your computer devices with your thoughts. Would you be interested in getting, uh, a Neuralink implant at that point?

• 1:30:31Speaker 00

  I would not. Um, I can imagine that like, I don't know, as like a, as a weird like strong, what if I'm in like a presentation or something and I'm computing, you know, controlling my laptop and my brain and I get distracted, right? Like, does that mean that my slide deck now is all like whatever I'm distracted by? Like, I think, I think there's, you know, like, there's, again, like having guardrails and like having layers of security between, you know, uh, thinking versus like, something being executed on a different system. I think it's important to kind of have like some, some filtration mechanism there. Uh, so I would skip on it, but I would be very excited in that role. Like I would, yeah, I, I would like, I would, I would like inhabiting that role, even if I'm not actively participating in it.

• 1:31:14Nathan Labenz?

  Great answer. I love it. Um, final one, just zooming out, you know, as wide as you can. You know, and thinking as far in the future as you have any sense of what might happen, what are your biggest hopes for and fears for society at large as this AI moment continues to unfold?

• 1:31:34Speaker 00

  I think it's a great question. Um, let's see, I think my biggest concerns, once again, I don't think I'm very, I have any special insight here, but I do think like job displacement is something that I think about quite a bit. Um, and thinking about, you know, like, what are the, you know, just like, what are going to be the, what is the, what is the amount of work, you know, that would still be valuable in this future where a lot of knowledge work can maybe be at least assisted or like a lot of knowledge workers can be made like more efficient, right? And uh, than than what they are like today. Um, so what does that mean for the future of work? I think that is something that I think about. Um, and let's see, I think the hope is that I am able to live a life where I just don't have to do any of these like mundane

• 1:32:17Speaker 00

  things that end up taking so much of, you know, our parts of life. So, um, in in in the future, like maybe I don't have to do my own taxes and we, you know, I don't have to book my own flights. I can just say like, uh, book this flight for me at this date and just find me the best price, etc. So, being able to, you know, like automate away a lot of those parts and then just do like whatever fun parts of life are, I think like that would be something that I'm excited about and hopeful for. Yeah. Uh, go check out the package, uh, if you're building a large language models and if you're facing like any issues where you're like, I was just working and it's not working anymore. How do I, how do I kind of fix that? Check out Garriels, um, and, um, yeah, uh, follow me on Twitter.

• 1:32:58Nathan Labenz?

  Yeah, it's a beautiful vision. We can all hope for it. Shreya Rajpal, thank you very much for being part of the cognitive revolution.

• 1:33:05Speaker 00

  Yeah, absolutely. Uh, thank you again for inviting me. Really enjoyed spending my afternoon chatting with you.