Controlling AI Models from the Inside

As generative AI moves into production, traditional guardrails and input/output filters can prove too slow, too expensive, and/or too limited. In this episode, Alizishaan Khatri of Wrynx joins Daniel and Chris to explore a fundamentally different approach to AI safety and interpretability. They unpack the limits of today’s black-box defenses, the role of interpretability, and how model-native, runtime signals can enable safer AI systems. Featuring: Alizishaan Khatri – LinkedIn https://www.linkedin.com/in/alizishaan-khatri-32a20637/Chris Benson – Website https://chrisbenson.com/, LinkedIn https://www.linkedin.com/in/chrisbenson, Bluesky https://bsky.app/profile/chrisbenson.bsky.social, GitHub https://github.com/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:03Speaker 00

  Welcome to the Practical AI Podcast where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind the scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:48Speaker 01

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I am joined as always by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris? Hey, doing great today, Daniel. How's it going? It's going well. Done a little bit of snow shoveling on the ground. As we speak, we're kind of headed into winter break or the holiday Christmas season here in the in the US. And I think this this episode will be released in the new year. So if you're listening to this, this you're listening in the future. To be honest, I'm really excited about talking about the future, because our guest today is really thinking very

• 1:31Speaker 01

  innovatively about how we can secure our AI models and have safety as we move into that future. Really excited to welcome to the show today, Ali Khatri, who is founder of Rinks. Welcome, Ali.

• 1:45Speaker 02

  Thanks. Thanks for having me, Daniel.

• 1:47Speaker 01

  Yeah, yeah, it's we met earlier this fall, really fascinated by your kind of line of work and technological innovation at Rinks. But I also know that you've been thinking about these topics around AI safety, guardrails, disallowed content, etc., for for quite some time. Could you give us a little bit of a background of how you got into these topics and what you've done in the past?

• 2:14Speaker 02

  Yeah, so I have been in the machine learning for AI.

• 2:20Speaker 02

  Safety or anti-abuse use cases in general for the past eight or so years of my career. I spent about three years at Meta where I built infrastructure that serves about half of the world's population. Basically, any safety, anytime you type a message on Facebook, it goes through tens of safety checks which are powered by thousands of models. I built the infra that these models run on. And then I moved on to Roblox where I built AI-powered, like where I built systems to protect about $3 billion in payments against fraud. So I've been in this space, I've been using AI models to sort of protect against abuse.

• 3:03Speaker 02

  And during this time, I realized that the models that I'm using themselves are susceptible to abuse. So that's what led me to Faming Rings.

• 3:14Speaker 01

  And I know that now you're thinking about those actual models. So I often tell people, also on the AI security side, there's kind of AI for security and then there's security for AI. It sounds like there's something similar kind of in, I guess, model or our safety anti-abuse space. Could you give us a little bit of an understanding of, like, when we're talking about the safety or security of AI models, could you kind of define that for us? Like, what does that, what do you have in mind as the kind of bad case scenarios or worst-case scenarios of what a model could do, why it's not, why it might not be secure or safe?

• 3:59Speaker 02

  Yeah, so thanks for making that distinction between AI for security and security for AI. And they're two very different things, right? Like, AI for security basically means using AI to solve existing security challenges in a more effective or a better way, right? So that's a very different and a very linearly separable body of work from security for AI, which is. Which focuses on making the AI models themselves and like AI-based use cases secure. As models have entered the tech stack, they also bring in a bunch of security challenges. And that is what security for AI solves.

• 4:39Speaker 02

  Uh, now, in terms of the safety aspects that we talked about, models are today will generate anything. We'll literally tell like these these generated models, like if it's a text model, it'll generate any any vile, any form of vile content known to man, it could tell you like OpenAI got into trouble for like where a teen was asked to commit suicide, right? Like was or was encouraged to commit suicide. So you have self-harm, you have different other categories of harm where you can, you know, generate pornographic content, you can generate other forms of inappropriate content like violence, gore. And sometimes it doesn't even have to be inappropriate because safety is very context-specific, right? Like as a law firm, safety looks very different from you than for you than

• 5:23Speaker 02

  what it does for a medical shop versus say a customer service setting versus like a co-generation environment. So each one of these uh use cases come in with a set of permissible and non-permissible behaviors. And that's kind of what safety really is, that making sure that the technology works in ways that you intended to and minimize the unintended outcomes of it.

• 5:47Speaker 01

  And I guess people are addressing, I mean, these are uh known issues in the sense that at least a uh a segment of the people that are working on these mo- models know about these issues. Could you give us a little bit of a sense of like as we speak? How these, at least in in a production sense, like what's capable now? Like how how how does the landscape look in terms of capabilities to uh defend against or align with uh with with certain um certain policies or allowed content, disallowed content? What what are our choices right now?

• 6:30Speaker 01

  If I'm looking at the availability currently of uh both kind of open source projects and what's in maybe closed platforms, what's available to me to deal with this issue?

• 6:42Speaker 02

  Okay, um, so before I answer that, I'll use, I'll start with an analogy which will help make the rest of the response much more clear and add some context. So imagine you're in a

• 6:55Speaker 02

  Like a story, like a built, like a giant apartment building with over, like I say, a thousand condos, right? Now, let's say your neighbor, for some, for some reason, or no reason at all, really decides to pull out a golf club and starts violently assaulting you with it. Now, for a situation like this, will the good guys be able to protect you just by checking IDs at the gate?

• 7:19Speaker 03

  Obviously not.

• 7:20Speaker 02

  That's kind of where we are.

• 7:21Speaker 03

  You're past that point.

• 7:23Speaker 02

  So that's, that's kind of, that's kind of where we are with AI safety today. Uh, that's kind of how jailbreaks work, right? So here, the giant thousand-story building is the model. So today, what we are able to do is just today's solutions, analyze what's going into the model, also known as the prompt, and analyze what's coming out of the model, which is the response. But by then, uh, the damage has already been done. So now, if you're using video generative models, right? You can put in a text prompt, get a video output. Your video output is gonna be, like, it's too expensive to analyze, number one. And the video has already been generated, so you've already spent a larger amount of compute generating that bad content. Right? Uh, again, if you're talking with audio models, you can trick audio models into generating bad content.

• 8:08Speaker 02

  Like a seemingly innocuous-looking prompt can be tricked today using tech, multitude of techniques to generate really malicious output. So now, um, like, unless you have visibility into what's going on inside of the model, you're not gonna be able to catch a lot of these things. That's where jailbreaks come from. That's where adversarial machine learning comes from. If you look at like, if you look at it through the context of predictive models versus generative models. It's essentially the same core phenomenon where operating these models as black boxes, and we have no idea of what's going on inside of them. So we're, we're trying to change that.

• 8:46Speaker 03

  Looking at that, it seems like in the way that you just kind of phrased it with the context there, it seems like a very intractable problem is a follow up to Dan's question like, how should I as a new, as a new user potentially, or someone coming into a use case in my company, where a model is desirable, but I'm worried about whatever, whatever bad or abuse means in the context that I'm operating in, how should I start off thinking about that? Like, like, what's my starting point? Because I, I gotta say, you know, coming into it, I'm not even sure where to start. So like, can you level set that a little bit in terms of how, you know, what, what's, what's square one?

• 9:26Speaker 02

  Yeah, so that's a good question. So normally what you do is, the way I like to think about it is, there's a general category of bad stuff that no one really wants, or the law doesn't allow, and things like that. So there's like a general category of stuff like that, right? Like here, I'm including porn, hate speech, yada, yada, yada, which take a general category of undesirables that no one wants on the platform, like child safety, for example, that's non-negotiable, no matter what context you're in. Now, there's also another aspect of categories, which is about context-specific safety. So now, if you're in a banking use case, you've got to think about, okay, money laundering. You might not have to think about money laundering, say in a code generation setting, for example, right?

• 10:08Speaker 02

  So you want to think about these very specific categories of risk or issues that come from your use case. And people tend to usually have a very good idea of that. Like, if you're, like, if you understand your use case well, which most people do, right? Which is why they're exploring models and they're trying to solve some problem within their use case. So within that use case, you also would understand the problems that you're facing. And that's another category of risk that you want to think about. Then, once you've thought broadly about these two, then you've got to figure out about, once you've identified both of these categories, then you want to think about mitigations, detections, and things like that.

• 10:48Speaker 01

  I guess there's like, if we're just kind of defining some terms for people that they might have heard of, this approach that you talked about, related to kind of guarding the the gate to the to the apartment complex, right, or the inputs or outputs, managing the prompts and the outputs, is that what people refer to as like guardrails, safeguarding? What is some of the the terminology that's being used? And then I know that you all that you have a different way of of thinking about this. I guess just setting the stage jargon wise, how do we define these kind of guarding the gate things? And then as

• 11:29Speaker 01

  we filter into actual safety within the apartment complex or within the model, what kind of terminology and I guess like there's a body of research building up to to what you're doing, what terms are used to describe that? And like as people wanted to research that, what would they look for?

• 11:47Speaker 02

  Yeah, so guardrails essentially like are a catchall term. They can refer to prompt and response filters. Now today, one of the ways of doing like, there are like multiple guardrail type solutions out there. There are guard models out there. Meta has one, IBM has one, Google has one, OpenAI has one, like, and I'm talking about public releases, right? Internally, most people have their own. But these essentially are prompt and response filters. So they look at the data going in, they look at the data coming out. Now, so that is like one thing that guardrails are used to refer to. Less commonly, guardrails are also used to use, refer to like static checks, where you just look at the output

• 12:30Speaker 02

  of the model and say something like, okay, the word, like forbidden word, let's say the F word, right? The F word appeared in output, this is not permissible. So that's a simple reject filter that you can use. So that would also be called a guardrail. In terms of like, looking at the internal state of the models, there's a whole bot, like a whole field of research, area of research that's developing, it's called interpretability. There's a subset of that called mechanistic interpretability, where they try to figure out what subcomponent of the model led to

• 13:05Speaker 02

  This particular output and try to change it at the source or try to alter or modify behavior while it's happening as opposed to after or before.

• 13:29Speaker 04

  Well, friends, here's your hot take of the day. Your team's AI tools, they might be making collaboration messier, not faster. You probably know this. You feel this. Think about it. You've got AI literally everywhere now summarizing, generating, suggesting, but if there's no structure, no shared context, you're just creating more noise, more outputs, more stuff to wade through. The gap between having a great idea and actually shipping that idea, that gap isn't a speed problem. It's a clarity problem. Well, this is where Miro comes in. And honestly, it shifted how I think about team workspaces. Miro's innovation workspace is not about brain dumping everything into an infinite canvas and just hoping for

• 14:13Speaker 04

  the best. It's about giving your work context, intentional structure. So your team knows what to focus on and where to find what they need without playing detective across 12 different tools, clicking and moving and tabbing. Just, just too, too messy. And the AI piece, well, Miro AI actually gets this right. They've got these things called AI sidekicks that think like specific roles, like product leaders, agile coaches, product marketers, reviewing your materials and recommending where to double down or where to clarify. You can even build custom sidekicks that are tailored to your team's exact workflow if you desire. And then there's Miro Insights. It sorts through sticky notes, research docs,

• 14:57Speaker 04

  random ideas in different formats and synthesizes them into structured summaries and product briefs. And Miro prototypes, they let you generate and iterate on concepts directly from your board. Test 20 variations before you ever touch your HiFi design tools, saving you time, giving you Ideas and getting it right. This whole thing is built around the idea that teamwork that normally takes weeks can get done in days, not by going faster, but by eliminating the noise and the chaos. So help your teams get great done with Miro. Check it out at Miro.com to find out how that's M I R O dot com again, Miro.com.

• 15:44Speaker 01

  So Ali, you were just getting into these ideas of interpretability mechanistic interpretability. I think you called it. We've, we've talked about interpretability on the show before. I think mostly in relation to like trying to figure out why a model made a certain decision in relation maybe to certain concerns around bias or other things. So like if I have a risk model for approving insurance or something like that, then I might need to have some interpretability around that. Or maybe in the case of healthcare, there's a burden for interpretability of like how decisions were,

• 16:27Speaker 01

  were made here, it sounds like kind of the interpretability is being applied to, I guess, why the is a good way to put it, why the model generated some problematic output? Or is there a better way to think about that?

• 16:44Speaker 02

  So there's multiple overlapping aspects here, right? Interpretability is like an umbrella research area. It's an umbrella term. So what, what you alluded to earlier is more of, it's also described as explainability. So why was this credit card denied? Why? So you're trying to explain it in human concepts. That is a part of interpretability, no doubt. It's a subset of it. There's another subset, which is how was this generated? Right? So for example, if your gendered model outputs, say,

• 17:18Speaker 02

  Like if you say, how are you? Hi. And the model says, how are you? You want to care about like, how was this generated internally? How were these tokens produced? And the reason you care about that is you want to know if, instead of saying, like responding to, how like responding with how are you? It could respond with howdy or with something else, right? So you want to know what caused those differences. You want to be able to control that. So that aspect is also interpretability. Now, where this interplays with safety is when you have these prompts, which look bad, which look good to a human, but bad, but result bad out resulted bad outputs, which is how jailbreaks work. When you analyze how it how the data flows inside of this black box,

• 18:02Speaker 02

  you're able to, you're able to like control it and stop it at the source. So think of this as continuing that analogy from earlier on. Think of this as cameras at every gate or every path. So you know that, okay, well, this is what's happening in this hallway and we got to stop it. We got to put an end to it. So it's a very different class of defenses.

• 18:23Speaker 03

  As you're saying this, you're, you're actually talking about manipulating the internals of the model and the flows that are there, kind of the cameras on the doors and stuff like that, and making it maybe a gray box rather than a black box to some degree. As opposed to kind of the more traditional guardrail approach, where you have programmatic, you know, you use the word guardrails around the model's inputs and outputs to try to, to try to handle things that way. So it's kind of a whole different thing about, about instead of treating the model as a black box, you're saying you're diving into it and trying to, to affect an improvement there.

• 18:59Speaker 02

  Yeah. So intervening is one, like stopping generation or modifying that is one form of intervention. Right? Intervention does not have to necessarily be in that form. It can take various other forms. So once you understand, today, we have no idea what's going on inside of the model. We put an input to the, like, we provide an input to the model, get an output. We have no idea what's going on. So. This, what we're trying to build, or what interpretability tries to do is understand what's happening inside. Now, you could control it, or you could use that to make a risk quantification and use that downstream. You don't have to do anything in the moment necessarily. It can be leveraged downstream. So now, it's basically like, you have a whole new set of

• 19:43Speaker 02

  information, or a whole new set, like a whole new class of data points, that you can leverage in creative ways downstream. This is something that's not available today, and this is what interpretability builds really at runtime.

• 19:55Speaker 01

  And, am I correct? So, part of my assumption in the past, and I'm fascinated by this whole subject, is like, some small changes in the, for example, the weights, or individual layers, or individual pieces of the model, can produce very large changes in the output behavior of the model, which, it's kind of, I'm blanking on this, what is the thing, it's like butterfly flaps its wings and... Oh, the butterfly effect. Yeah, the butterfly effect, or whatever. So, like, these, because you can make a change,

• 20:36Speaker 01

  whether that's quantization, or other things to the model, and that, you know, may produce unclear, and sometimes catastrophic, changes in the, in the behavior of, of the output. And so, if I'm understanding what you're saying, right, Ali, it's one way you could try to use the information about how the model is producing certain outputs, is to intervene by actually making a modification, or preventing something in the model. But that could produce other changes that you may not want, I'm assuming. But you could also instrument the model to understand, potentially, when it is kind of firing those certain neurons, or, or lighting

• 21:20Speaker 01

  up in a certain way that is indicative of problematic behavior. Am I, am I understanding that right, in terms of like... Various ways of, I guess, either intervening or instrumenting. I don't know if I'm using the right terms.

• 21:34Speaker 02

  We are instrumenting, like the way we're approaching this is we are trying to understand what happens inside of a model at runtime. Models are a monolith, right? But we're breaking it down into different spaces or subspaces. And we look at the subspaces that get activated, that get activated during bad generation. So now, uh, when you're, let's say, generating a non-permitted content versus permitted content, right? Different subregions of the model get triggered. So we're building visibility into that and we're trying to identify them at runtime. So now there are some subregions that you wouldn't care about, right? Like, for example, in a like, if you take a general purpose L L M, it's trained on everything ranging from

• 22:17Speaker 02

  Python code to 15th century Chinese poetry, right? Now, when you're using it in a customer service setting, you care about neither one of them. And if they're, if those subregions are get of a model are getting activated, then you want to be able to like arrest it while it's happening. So this is similar to like, if you go back to the analogy that I had, like made about the apartment building, you want to have visibility at all times into what's going on at each level, right? So you understand, like you find out that a bad thing is going to happen way before it actually happens. Like, for example, if someone's going to conduct a bank robbery, people just don't get up and conduct a bank robbery, right? There's some searching going on. There's, you know, planning, cycles of planning going on. There's

• 23:00Speaker 02

  purchases of firearms or whatever going on. So now if you stop them at these bad activities at different levels, it does not have to get, the, the police don't have to deal with a shootout situation in a bank at the very end. So similarly, like defense works in depth, right? And we're building a whole new layer of safety that hasn't been tapped into just yet.

• 23:20Speaker 01

  Yeah, that's fascinating. And I'm, and I'm guessing certain folks are probably wondering, like I am out of curiosity, like This might be the first time that they're hearing about such an approach to this kind of problem, and they might be thinking, you know, how is this possible? Like, how? So, one thing, I think the general concept makes sense, right? Like, instrumenting the interior of the apartment complex, understanding what's happening, kind of retrieving that, you know, that intelligence for you to make decisions or determine if you want to mitigate something. What are, what are the, I guess, have there been multiple attempts to try this sort of

• 24:05Speaker 01

  thing? And from your perspective, in terms of how you all are approaching it, what is kind of needed, I guess, from the customer side to create this sort of instrumentation? So, I could imagine, like, in one scenario, like, I could train, I could tell the customer, "Well, we're not going to train on Chinese poetry or code. We're just going to train a whole new model." And that burden on the customer is very, very heavy, right? And on another scenario, you could say, "Oh, take this model off the shelf and do this to it." And then, you know, that's a less burdensome. So, there's probably a spectrum of here.

• 24:48Speaker 01

  Could you help us understand, like, the, I guess, the burden and what might be required to get to this instrumentation?

• 24:54Speaker 02

  Yeah. So, the way we're approaching this today is we do not modify the model. We do not even build models, nor require the customer to build a model. We take an off-the-shelf model, and we build a safety module that sits on top of it. So, essentially, for the customer, it's like a very low-friction approach, where they take a model that they use and love, like Lama, or, you know, Gradit, or any Mistral, or any of the open-weight models for image generation. You have WAN, and you have a host, other host of Chinese models for the audio/video setting. So, any model that you love, we sort of make it more secure and tailor it for your context. So, now, again, remember, if you're like a lower law firm, right?

• 25:38Speaker 02

  An off-the-shelf Lama or off-the-shelf Mistral is not going to have the protection that you had. Like if you're like, say, a shoe company, uh, let's say you're Adidas or Nike, right? You want, you as a user want to talk about Nike, but not talk about Adidas, right? You can't expect the model maker to put that in for you because the model maker is trying to sell to everybody. So we helped build that customization and we do that without changing your primary model. Like your primary model will continue to be as it is. If you make any modifications to it, uh, like fine tuning or anything like that, that's on you. You control that. We don't require you to do it, but even if you do that, we can still support you.

• 26:19Speaker 03

  Could, could you, uh, so totally recognizing that there's proprietary stuff that you're not going to dive into and, and, and respect that. Could you talk a little bit about, um, just kind of clarifying as we were kind of talking earlier about kind of the buttressing with guardrails on the external side versus the going into the model. And as you're talking about adding a component like, so in my confusion, it seems a little bit like it's on the outside. Can you talk a little bit about what you mean by that without diving into places you can't go?

• 26:49Speaker 02

  Yeah. So I'll, I'll give you a very, um, I'll try to address that as much as possible without going into the specific.

• 26:56Speaker 03

  Fair enough.

• 26:57Speaker 02

  Specific details. Uh, so today what you have, right? You have these filters which analyze the inputs and the outputs. Now analyzing the, the economics here is all messed up. Like if you were to analyze video or audio, right? Those tend to be very expensive computationally. Those models are very expensive. So now if your inference itself costs X and if you're expecting someone to pay another X to analyze it. Number one, it's slow. Number two, it's like paying someone a thousand dollars to guard a hundred dollar bill. You're just not going to do that, right? So what people end up doing is people end up like shipping unsafe models. So today I've, I've tested, like we have tested models from different audio companies.

• 27:39Speaker 02

  We've tested models from different video companies, different image generation models. Each and every one of them with little to no trickery, which means that average user can just go there and ask them to generate bad stuff and they will do it. There are little to no guards there, uh, and that's because the economics does not make sense. The way thing is today. So what we've built is a scientific, like we've sort of. Had a research breakthrough where we can build safety about thousand times cheaper. So just to give you some concrete numbers, what we've done with Llama, we've taken a Llama model, which is like an eight billion parameter off the shelf Llama model. And today, if you had to protect it, you would have to use Llama Guard three, which is an eight billion parameter model. And assuming it generates 10 tokens, that is,

• 28:23Speaker 02

  you're running about 80 billion parameters of inference at runtime. Now, if you do that on your prompt and response both, that number balloons to 160 billion parameters of inference. That is two extra GPUs or one extra GPU, depending on how you've

• 28:40Speaker 02

  wired the, or set up the, deployed the models. So now what we are essentially doing is we're analyzing the internal states of the primary model as it makes the prediction. So in doing so, we don't need any of those two extra GPUs. And that 160 billion of parameter, billion parameters of inference that I counted, we have succeeded in bringing it down to 20 mil with an M. So we're essentially a rounding error today because of this expensive safety profile. You cannot even deploy them on edge devices, like on edge devices, guardrails are non-existent. Because you can barely like edge, when people are working on the edge, they work really hard, a lot of people work really, really hard

• 29:21Speaker 02

  to squeeze that one device through techniques like quantization, that one model onto the limited memory of the device. So you have no room to deploy a safety model, right? So that's why we've like built tech, which literally is like a rounding error. 20 million parameters on eight billion is nothing. And we can sort of deliver safety, like comprehensive safety and our safety performance is comparable to a standalone guard model and it is significantly faster. Because our late, like our latency just doesn't exist. It's parallelized and it, it becomes in practice, the latency of the primary model is the latency that the user sees. With today, you have to like sort of account for the latency of the primary model. And you also have to account for the latency of the response filter and the prompt filter.

• 30:04Speaker 02

  Also, the response filter cannot kick in until the primary model is finished generating. So you're looking at like very high latencies. So from the perspective of the end user, you're looking at a lot of added friction in terms of slow speed. Uh, you're looking at increased costs because ultimately the cost will be passed on to the user. Right. So you're paying for two extra GPUs that you don't have to. And your quality will be substandard. Again, remember you're just, all these models are able to do is check IDs at the gate. So that is the protection you're getting.

• 30:35Speaker 01

  So Ali, it's a very fascinating and encouraging the, the, the results that you're seeing and, and what you're able to do with this sort of approach. I'm also wondering like the size or latency or behavior that you just described. That is certainly a, a huge component of, of what people are thinking about and why they can't use guardrails in certain cases. Another question that might come up though, and I actually think that you, you can validate me, but I think you have a, a very good answer to this is, is what, what about the kind of uh, accuracy or reliability of kind of one approach or the other?

• 31:18Speaker 01

  So a, a person I guess could argue and say, well, if I have a guard at the gate and he's 100% accurate and making sure no, no gun ever gets into the building, then there'll never be a, you know, there'll never be a shooting in the, and that's a very robust guardrail or, or something like that. But I think, uh, I think what you're saying is you still wouldn't know what happens in, inside the building with 100% certainty. So could you address that side of things like the accuracy side or the quality side? I guess of the performance of guarding and safety with this kind of instrumented model approach versus kind of an exterior guardrail?

• 32:03Speaker 02

  Yeah. So an exterior guardrail, as you pointed out, is limited in visibility, right? So even if they do 100% great job of checking someone's uh, IDs, they only have limited information. There's only so much you can do, right? What you're, the, the kind of defenses you're expecting is out of the scope. Like you're limited by information there. Like, if you, if you don't know how to drive a car, you're just not going to be able to do it, right? No matter of, like, no matter how fit you get, no matter how, uh, how much you train, you're not going to become a race car driver if you cannot drive a car. Similarly, like in this security setting, like if the example that I use where

• 32:45Speaker 02

  somebody, you know, decided to pick up a golf club for one, some reason or no reason at all, you weren't checking for, like golf clubs are permitted items to go bring into a home. So the security have done their job right. They, they haven't done anything wrong, but they're just, there's a fundamental limitation that exists here. So you can only do so much looking at artifacts. So there's a whole layer of safety that is untapped or unaddressed or inaccessible because of, because of scientific limitations. But that's changing fast and we're sort of at the, like at the bleeding edge of it.

• 33:20Speaker 01

  And just to make sure that I, that I have it right. So would it be a good way to describe it that, let's say that I have, or I want to prevent, you know, toxicity or something coming out of the model of a certain type, right? There are a variety of inputs to the model that could result in that type of output, but I'm never going to know all of them, right? Uh, or, or there's always a, there's always an edge case, right? And so by instrumenting and saying, this part of the model lights up when toxicity is being produced, then I no longer have to worry that I have all of the

• 34:05Speaker 01

  possible inputs in the world put together that, that might, might trigger toxicity. I just know when there's toxicity. Is, is that, is that a appropriate?

• 34:17Speaker 02

  Yeah.

• 34:17Speaker 01

  Way to put it?

• 34:18Speaker 02

  Yeah. So, um, when you're, when you're on the defense, when you're playing defense, right? You don't have, like when you're defending models against abuse in any scenario, whether you're defending models or just protecting a platform in a classical trust and safety sense, you're never going to have an exhaustive list of the million ways in which things go wrong. But you can gauge, you can develop a fair understanding, uh, through past examples or through data point that you have. That's the benefit of machine learning, right? But with, um, Well, the way jailbreaks work or with the way adversarial examples work is that they do certain things inside of a model that are not possible to predict in a different model, which is being used as a guard, right? So the guard is model type A. The primary model that you're predicting is model type B.

• 35:03Speaker 02

  So the guard is not going to be able to predict what's happening in model type B, simply because they don't have visibility into what's happening inside the model. So without like, without information, you're not going to be able to do anything, right? Like, for example, if you're the SEC and somebody takes away your access to bank accounts, you're not going to be able to prevent money laundering, no matter how many books you've written on that subject. There's only so much you can do with guns and badges, right? You, you would need visibility. If you want to try preventing money laundering, you will need visibility into the financial system. So that's kind of what we're building here. Uh, in terms of accuracy, the numbers speak for themselves. Like, we're able to be like match and beat the performance of standalone guard models over thousand times our size.

• 35:48Speaker 02

  Um, and that's because we are exploiting this unique insight.

• 35:51Speaker 03

  Yeah, it sounds a lot like what is an analogy that in like neuroscience would be, if I'm, if I'm can pronounce it right, an electro cellulogram, an EEG where they monitor the synaptic connections and they can see it lighting up, you know? Um, since I can't pronounce the word, I'll just describe it to the best of my ability. It's a long word. I actually had it written down. I was like, oh crap, I still can't pronounce it here. So, um, so it sounds like that. As you're thinking, uh, you know, about like how you can apply that, are there any ways that that can kind of tie back also into the hybrid approaches to see? Like, if we talk about some of the other, uh, more traditional forms of guardrails that are out there,

• 36:35Speaker 03

  can you combine them into sort of a hybrid approach where you do have, um, different types of guardrails, uh, that, that are in place, but people can add this kind of capability from you into it and thus enhance their overall security model? Like, what does that, what would that world look like in your, in your view if that's valid?

• 36:55Speaker 02

  Yeah, uh, I mean, we, I do, I'm a firm believer in defense in depth. So one, one product does not miraculously solve everything. Just like with our human society, right? Like, if you think about law enforcement, it's a very good parallel. Where for national security, you need army to protect you. You need different forms of the military to protect you from external threats. You need the border police to make sure that entrance is regulated. But at the same time, you also need state and local law enforcement. You also need federal civilian law enforcement. So you need different levels of that to make sure that security on the whole is insured. Similarly, in our, in the setting of, in the context of AI models. Yes, you have guardrails which look at prompts and responses. They are valuable, right? But then there's ways to do them efficiently.

• 37:40Speaker 02

  And then you also oftentimes would need to combine, say, system level features with model level features. So we build those model level features. So you could do something complex, like, let's say you're, you know, running some sort of customer service bot and a customer has a history of refunds, right? And then in that, we detect, let's say, we detect some sort of misrepresentation or some sort of lying. So you can say, okay, block if lying score. You can compose rules like saying block if, you know, lying score greater than point eight and this customer has refunded more than thousand dollars worth of merch. So you can, there's potential to combine, mix and match as well. And I think that's how you improve the overall safety profile of any system.

• 38:25Speaker 02

  It's always in depth and those layers have to work together. If you, I can go on, like, I can just look at web applications in parallel, right? Like, you need static code analyzers and you need AI firewall. They don't replace each other. If anything, they complement each other and build an overall robust system.

• 38:43Speaker 01

  And before we leave the subject and maybe, maybe look towards the, the future a little bit. I do want to highlight, I think, one of the fascinating things that, that is, comes out of the work that you're doing, Ali, is. Some of the, I guess, customization that's possible in the sense that we've talked a lot about kind of the, quote, "traditional" types of things that you would want to prevent, whether that's jailbreaking or toxicity. But there's also, like you were saying, in every industry and actually at every company, there might be custom types of policies that they want to enforce or certain things that they want to instrument. Does this, I guess,

• 39:28Speaker 01

  how extendable is this approach to those sorts of situations?

• 39:34Speaker 02

  Yeah, so this is very extendable to those. I mean, our approach is designed for these situations. We've identified this niche in the market with models, you're shipping a one-size-fits-all solution. So, to put it a different way, the model needs of most companies are similar-ish, but the safety needs are dramatically different. So, you cannot have a one-size-fits-all safety stack that works for everybody. Yeah, sure, there's a general category of undesirables that everybody would want to keep off their platform, but that's a very small subset. With generative models, which are capable of doing so much,

• 40:16Speaker 02

  you need to be able to customize safety as well. So, that's the space we thrive in, and that's what we're building for.

• 40:25Speaker 03

  Well, that's very interesting to me. I've learned a lot today. As you're thinking, you know, Dan, kind of telegraph, we like to kind of finish up by asking about the future. And as you're thinking about the future, both in terms of kind of your specific approach that you're doing, but also, you know, kind of like where security is going for models in general, in the large, what kind of guideposts do you have that you're thinking about? You know, I like to say, at the end of the day, when you're taking a shower or you're lying in bed at night about to go to sleep and your mind's just kind of going loosely, you know, where do you see things going and what are you really

• 41:09Speaker 03

  passionate about pursuing or exploring going forward and in time? You know, what's that aspirational, "hey, I'd like to go do that" that you have in mind?

• 41:19Speaker 02

  So I think when you look at safety, when you look at like runtime safety, like when you look at safety, right? There's different aspects. So there's build time safety, which is a whole different class of safety products. But in terms of runtime safety today, runtime safety only exists at the data layer, which is at the prompt and response layer. The model layer is missing. My aspiration is to sort of build that out. So that has been my vision. And that is the guiding vision behind Rynx, where we want to build model native safety. And I see that will start, that will have to exist for models to, you know, be adapted, be adopted into different settings. Like if you're in healthcare, for instance, today, like it's very hard to use a public L L M, right?

• 41:58Speaker 02

  Because of data concerns and like no one in their right minds today would fine tune a L L M on D I I data. That's because you, you just can't, it's just not possible. So it's locking a lot of people out of the ecosystem. Uh, and that's, that's the problem we exist to solve. So I want to become like, my vision is to sort of build that de facto model safety layer for no matter what model we are using. I want to build model, become the go-to for model safety.

• 42:29Speaker 01

  That's awesome. Well, I definitely encourage people, check out the show notes and I, I should have said this at the beginning. But um, just to make sure people, uh, Rynx, W R Y X?

• 42:41Speaker 02

  N X.

• 42:42Speaker 01

  N X, sorry. I, I even missed W R Y N X. Um, so check out Rynx, we'll have the, the link in the show notes and um, yeah, just really fascinating work. And uh, from the community, uh, Ali, just thank you for, for digging into this topic and, and bringing a fresh look at things. Um, it's awesome and hope to see you back on the show to find out where, where things have advanced.

• 43:06Speaker 02

  Yeah. Thank you for having me on the show, guys.

• 43:08Speaker 01

  We'll talk to you soon.

• 43:09Speaker 02

  Thanks.

• 43:10Speaker 01

  Bye.

• 43:18Speaker 00

  All right, that's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats, and to you for listening. That's all for now, but you'll hear from us again next week.