GenAI risks and global adoption

Daniel and Chris sit with Citadel AI’s Rick Kobayashi and Kenny Song and unpack AI safety and security challenges in the generative AI era. They compare Japan’s approach to AI adoption with the US’s, and explore the implications of real-world failures in AI systems, along with strategies for AI monitoring and evaluation. Featuring: Rick Kobayashi – LinkedIn https://www.linkedin.com/in/hironori-kobayashi-5371/Kenny Song – LinkedIn https://www.linkedin.com/in/kennysong/Chris Benson – Website https://chrisbenson.com/, LinkedIn https://www.linkedin.com/in/chrisbenson, Bluesky https://bsky.app/profile/chrisbenson.bsky.social, GitHub https://github.com/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: Citadel AI https://citadel-ai.com/Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:03Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:48Speaker 01

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I am joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How you doing, Chris? Doing great today, Daniel. How's it going with you? It's going great. I traveled a bit over the weekend to run a mini marathon, which was fun, but I am back safe at home. And of course, safety is something that hopefully we can talk a little bit about today with our guests, who are Rick Kobayashi, who is co-founder and CEO at Citadel AI, and Kenny Song, who is co-founder and CTO at Citadel AI.

• 1:33Speaker 01

  Welcome.

• 1:34Speaker 02

  Thank you for having us. Yeah, thank you.

• 1:36Speaker 01

  Yeah, it's great to have you both here. Good to get introduced to you both. I'm really excited about this conversation because, of course, security and safety related to AI is very close to my own work. And so it's always great to connect with people in this space. And yeah, I'm also interested to hear a little bit, and I even saw some things on your website, you know, talking a little bit about some of the AI in Japan, some of the, I guess, regulations. Or guidelines for for businesses that have come out there. So maybe before we get into the the nature of what you're building and what you're doing.

• 2:18Speaker 01

  Any thoughts on for those out there that might be in the US or in Europe and and be constantly exposed to the things that are going on in AI in those jurisdictions? Any thoughts about, you know, what is the same or what stands out about what's happening with AI in Japan?

• 2:37Speaker 03

  So yeah, anyway, thank you very much for joining us. And my name is Rick. And as for the Japanese conditions, basically, I'm afraid to say it's around one, two years behind the US situation, basically, up to now. But on the other hand, surprisingly, as for Gen AI, I think that Japan is one of the most advanced countries. Not the development of the foundation models, but the utilization of the Gen AI applications. And probably it might be related to kind of Japanese kind of animation or cartoon.

• 3:18Speaker 03

  So, you know, like there are lots of, say, robot or some of the, say, animations which specify some of the AI or robot technologies. And, you know, Doraemon, or there are lots of animation over there. And people are very, like, getting familiar with such a kind of a, say, talking with robot or some of these, say, advanced computers from the child phone. So in that sense, I think that many people do not have any hesitation to work with or talk with, like a chatbot or any Gen AI. So in that sense, the hurdle to introduce LLM or Gen AI in Japan is very low in that compared with US or other countries.

• 4:01Speaker 03

  So in that sense, development side or technology side, frankly speaking, Japan is behind the US situation, but as for the usage of such a kind of Gen AI, I think, I hope, Japan is one of the most advanced countries.

• 4:17Speaker 01

  Yeah, that's that's really interesting to hear the sort of. Perception side of things. And of course, with more, maybe even more adoption or adoption that's ahead of maybe other places in the world. It could be that, you know, users or usage of the technology has hit some bumps along the road or has hit some problems, which I know is a lot of what you all are working on. What has been the situation in Japan around kind of regulation and business usage of AI?

• 4:57Speaker 01

  Is it, is it on the sort of more regulated side or less regulated side in terms of the government and I guess, regulation in terms of security or safety, privacy, these sorts of things?

• 5:13Speaker 03

  I think Japan is in the middle between EU and US situation. So they say that it's soft approach. So there is no strict, say, the regulation in Japan, but there are lots of kind of guidance in Japan. And people are, say, I've already noticed some of the issues on the security side or safety side of AI. And on top of that, because they are, say, trying to introduce LLM applications for like a contact center applications. So they are concerned about more, say, kind of reputational risks rather than security risks.

• 5:55Speaker 03

  So in that sense, they understood the importance of the safety of the AI or trustworthy of the AI. But more say concerned about reputational risk rather than security risks.

• 6:08Speaker 04

  I'm curious, as we're, as we're kind of talking about the adoption and the environment around that, as you kind of mentioned that, that in Japan, it's very, you know, people are really getting into the utilization of LLMs and generative AI. Do you have any, any kind of thoughts around, you know, what is it, what is it that's driving that, you know, compared to other? Countries that you've observed if you're looking at that kind of adoption in Japan versus the US or versus Europe or whatever. Any thoughts around that? Because I had I had observed that as as someone in the US that Japan seemed to be implementing. And I was kind of curious if there was, you know, what was the what was kind of the driving force there that that got the

• 6:51Speaker 04

  utilization up, especially among just, you know, your typical average everyday folks there not, you know, outside of the AI industry. I would love your thoughts about that.

• 7:01Speaker 03

  I'm not so sure the the background reasons why Japan is more aggressive to introduce such a kind of L and L applications. Probably, as I said, people has lower, say, risk of the perception, I mean, say, in some cases in Japan, like a chatbot or kind of say new technology, might be a kind of friend in essence.

• 7:26Speaker 04

  Sure.

• 7:27Speaker 03

  In US side, such a kind of a new AI is kind of enemy of a human.

• 7:33Speaker 04

  Yeah, so they're kind of a little bit of a cultural difference in terms of how that openness to adoption and stuff. So that would make sense. That would make sense to me.

• 7:43Speaker 01

  Yeah, and I guess sometimes your your friends can potentially hurt you even if they even if they're not not trying, which I know you you all are kind of involved in both, you know, the evaluation of these systems, you know, running metrics against these systems and kind of building trust in a very real way. Kenny, I'm wondering if we bring you in here and maybe just, you know, safety and security around AI is such a broad topic now with so many people addressing it from so many different perspectives. I'm wondering if you could help us zero in on

• 8:22Speaker 01

  on the maybe the kinds of problems that you all are are exploring and how those fit kind of more generally into the landscape of, I guess, security risks or or threats as related to AI.

• 8:36Speaker 02

  Definitely. Yeah. So thank you for having us on the podcast. My name is Kenny. I'm the co-founder and CTO of Citadel AI. So what we do at Citadel is we build software tools to help organizations test, monitor, and govern their AI systems.

• 8:52Speaker 02

  And in the world of L L M's, when customers come to us, they usually have some proof of concept that they've been developing internally. So they've been using some foundation model, they're building some chatbot or some agentic workflow. And they come to us when they want to make that proof of concept production ready. And usually their main problem is they want to mitigate some of the risks that they see. Right. So things like hallucinations, toxicity, or users trying to prompt inject the system. They're they're looking for a solution to these types of problems. And that's where we come in as a tool provider.

• 9:30Speaker 01

  Yeah. And how does that, I guess, how are you seeing when you're interacting with your customers or they're coming to you with these problems? What is, what is the impact of those of those problems like hallucination or injection? Is that something that is causing real problems or something that they've sort of heard about that they're concerned about, but it's maybe not causing a real problem? Or, you know, what are you seeing there in terms of, I guess, the the impact? Let's say I just want to throw caution to the wind and ignore these things. What's, what's the bad side of this that could happen? Were I to kind of take that more, more loose approach? Any thoughts?

• 10:13Speaker 02

  Yeah, I think it's usually a mix of both. And it also depends on the the sort of the risk appetite of the company that's developing the system. And typically, our customers are larger enterprise companies, both inside of Japan and outside of Japan. And so they have very mature risk management practices. And before they, you know, launch these P O Cs into a production service, whether that's internal facing or external facing. They want to make sure that they, you know, have appropriate controls in place to manage the risk and they've properly identified the potential risks, reputational, data, security, and so on.

• 10:54Speaker 02

  So yeah, I think for the customers we talk to, it's generally a big concern for them. And they come to us with the problem of mitigating some of these risks that they've already identified.

• 11:07Speaker 04

  I'm curious, as you've kind of identified, you know, what that kind of the strata of customers that you're looking at there, what is kind of, is there, is there something that definitively kind of separates that kind of mature, larger organization from some of the smaller ones? Do they have a different set of problems that they're coping with? Or maybe just haven't gotten far enough along in terms of maturity and risk management? What like, at what point do you see the uptake kind of falling off within maybe smaller organizations? You know, what's, what does that look like as you, as you move from that upper strata into the mid-tier?

• 11:48Speaker 02

  I think my, my general perspective is that there's two ends of the spectrum of like startups to enterprise company. And for startups, the, the risk is fairly low because you don't have a brand to protect. You don't have an existing business to think about. And so you can just launch these P O Cs out to your customers very quickly and you can iterate quickly. For enterprise customers, they tend to be more guarded. They, they have a closer eye on potential risks to the business. And those are the customers that sort of really want pretty robust testing before deployment and also monitoring and potentially real-time guardrails sort of filtering after deployment.

• 12:32Speaker 02

  Yeah, I think it depends a bit on the size of the company.

• 12:35Speaker 01

  Yeah, I, I'm wondering with that, like, as you're kind of looking to those types of organizations, I'm, I'm curious a little bit of the backstory of Citadel. Maybe, Rick, you could tell us a little bit of kind of how you came to the, these problems? Cause you seem like you, you have customers now, you've developed some of these things. When did that happen? How early was that in kind of, was that before Gen AI? Was that, you know, as Gen AI was coming up? And, and how did things develop in terms of your, your thinking around these problems? And how you would bring something to the, to the market in terms of addressing them?

• 13:14Speaker 03

  So in that sense, it was before Gen AI. And actually the person. Who came up with such a kind of idea is not me, but Kenny. So Kenny has worked in Google Brains and he was one of the core member of the Tensor4 team. So he's taking a leadership to develop the most advanced AI technologies in Google. And found that there should be a lots of risk about the trustworthiness or say, safety of the AI. And he believes, probably he is the best person to explain, but anyway, he reached the idea that

• 13:54Speaker 03

  some of the tools which protect such a kind of a safety or security issues would be, say, should be become popular, especially in the enterprise companies where they may not have a bunch of the AI engineers inside. So that is a background and our company name Citadel itself is say, shown the art what we are focusing. So Citadel is as you know, saying like a vault or castle. So we are the company to say, protect such a kind of risk, I mean, human from AI risks. Such kind of as a concept is say, based on our, with our company name.

• 14:35Speaker 01

  Yeah, anything to add there, Kenny?

• 14:37Speaker 02

  Yeah, that's a pretty good overview of the background. When we started the company, it was at the end of 2020. So it was before the era of large language models. And we were initially really focused on helping organizations monitor their traditional predictive AI models, so tabular models, vision models, that kind of thing. And about two years ago, we started getting a lot more interest in our, from our customers in L L M's. And how do we, you know, reliably test this new technology? How do we integrate it into our workflows and our, our business applications? And so these days, I think a lot of our new customers come to us with L L M types of, of questions.

• 15:19Speaker 02

  And then we still have a lot of existing customers that work more on the predictive AI side.

• 15:25Speaker 04

  So Kenny, I'd like to follow up as you guys were kind of developing the idea, going back to that a moment for the company. And Rick mentioned that you had been there at Google Brain and that you were on the TensorFlow team. I'm curious, for me, as someone who's used TensorFlow a bit, and so that was like, oh, wow, you know, one of the people who helped put that together. I'm curious what parts of the experiences you had there led into the formation of Citadel in your mind, as you had some of the insights that you might have developed in your previous employment.

• 16:05Speaker 04

  Did any of that, you know, how did that lead to what Citadel does? How did that kind of bring forward when you and Rick started the company up?

• 16:14Speaker 02

  Sure. So a bit more about my personal background. So in 2017 to 2020, I was working at Google Brain as a product manager. And my team was responsible for building machine learning infrastructure at Google. So this included TensorFlow and also other platforms like TFX, TensorFlow Extended, and some of the work on Google Cloud's AI platforms and so on. So sort of the software foundations that power a lot of Google's machine learning applications. And I think Google tends to be pretty ahead of the curve in AI adoption. And I think back then it was more called machine learning rather than AI.

• 16:54Speaker 02

  But basically what we worked on was making models at Google more reliable at Google production scale. And that usually meant building these pretty sophisticated pipelines that not only target training the models, but also serving them in production. So we had pretty robust systems for monitoring data drift and validating data that enters production models, monitors the output of these models. And, you know, at Google, you can afford to have hundreds of platform engineers build out this kind of infrastructure to use internally. But for most other types of organizations, they can't make that level of investment in internal platforms.

• 17:35Speaker 02

  And we felt that there is an opportunity to sort of build some of these model monitoring and data validation tools for other companies. And so that's where the idea of Citadel AI kind of started. When we started the company originally, we focused a lot on adversarial attacks, actually, back in 2020, since it was a very hot research topic. So basically, you know, designing noise and images and other types of input data to trick models into making the wrong predictions. We sort of found that after a few months, this wasn't really a problem that companies were interested in. It's like a very interesting research problem, but less interesting commercially.

• 18:19Speaker 02

  So after that, we pivoted towards more observability and testing of these predictive models. And these days, we focus a lot more on LLM testing and monitoring.

• 18:29Speaker 01

  And maybe you could just give us, you know, going back to you, Rick, maybe you could just give us a few examples that stand out of kinds of companies and the kinds of models that they're running and the way that, or the, some of the things that they would be interested in tracking or detecting or observing. A few examples, concrete examples might help our, our audience kind of grasp some of the, the cases that you're working with.

• 18:57Speaker 03

  One of the largest customers in, in Japan is the financial industry, surprisingly. So like a banks or insurance companies or security companies are very aggressive to introduce LLM applications into their core operations, such as like a contact center or internal applications. And when they say they can go through POC stage, and when they get into commercial operations, they start caring about some of the kind of safety issues. So because those financial industry is governed by the, the government itself.

• 19:41Speaker 03

  So in that sense, different from regulation-related AI, they have already been kind of regulated by traditional, say, financial regulations. So if, say, LLM applications behave badly, that damaged their core businesses. So to introduce such a kind of a AI application is to differentiate or advance their services compared with their competitors. But by introducing new technologies, if their core, a core business is damaged.

• 20:20Speaker 03

  That would be very, say, huge negative impact. So, and also, uh, if the AI behave badly, as I said, there, say, the Japanese government may, say, in some cases, call or punish them. So that is one of the most largest risks for them to introduce LLM general applications. So what we are doing is to making sure that such kind of, say, bad behavior will not happen in the application on that. And making sure that they can safely introduce, say, go into commercial operation. So, uh, some of the leading bank company or insurance company is our, say, our customers.

• 21:05Speaker 04

  Gotcha. I'm curious if I could follow up on that for a second. With, when you're, we've kind of talked about the, the notion of risk management in general that that industry is, is dealing with. How do they, given the risks, uh, for a large, you know, organization to implement LLMs, do you, do you have any insight into how they're making the, the risk management judgment on the benefits of implementing maybe a new chatbot versus the potential downside where, you know, as you pointed out, could be damage to brand, damage to core operations, damage from a regulatory standpoint? That's, as I'm listening to you, that does sound very risky, you know, from that perspective. How do they make, how do they evaluate that?

• 21:50Speaker 04

  Like, say that they know that they want to come to Citadel and get that help. What, do you have a sense from your customers what a typical balance on that is of the benefit of the LLM utilization versus the downside of if things go off? How do they, what are they thinking when they come to you in that way?

• 22:07Speaker 03

  So, in that sense, they fully understand that balancing is very important. So to protect the risk, uh, so much may, say, uh, delay their, say, the service advancement. On the other hand, say, without having any security or safety test, uh, they may get into bad situation. So they need to, how to balance such a kind of risks and benefit. And they have set up several like internal organization to manage such a kind of a risk inside. So, yeah, as you can easily imagine, financial industries, they are very structured

• 22:50Speaker 03

  company. So there are some like a risk management department inside. And in some cases, there are like AI governance team inside or something like that. So they jointly work together and try to manage such a kind of a risk. And at the same time, they try to say introduce the advanced system so that they can differentiate themselves from others. And as for the contact center, probably the same situation in the US. But when the the consumer, end user, try to reach out to banks or insurance company and call it a free free dial, in many cases, they

• 23:34Speaker 03

  can't easily to say reach to the contact center person. They have to wait thirty minutes or something like that. So because of that situation, they are so aggressive, introduce such a kind of applications into contact center and the internal processes.

• 23:51Speaker 01

  And I I'm just thinking about the scenario where you have the contact center, you're introducing whatever it is, a chatbot, a voice assistant, that sort of thing. And then I'm thinking back to what Kenny was talking about the about kind of how the how the company started evaluating models that, you know, were not Gen AI models yet. And I'm wondering, Kenny, if you could help us think about, you know, what what needs, because if I'm understanding, you know, part of what you all are doing, part of it is evaluating the risks of a particular model or system, and part of it is observing those and monitoring those in in real time. And if

• 24:32Speaker 01

  I think about like a traditional model, let's say a model that detects tumors in medical imagery, right, I can Have a very nice sort of ground truth data set and you know, maybe it's hard to get because there's some privacy concerns, but I can still get it. I need to get it to train my model. And I, you know, have very specific metrics, right around whatever it is, accuracy, F1 score, etc. I can sort of grasp what the, you know, performance of that model is, maybe even compare it to human performance. With something like a call center, in some ways, people might struggle

• 25:13Speaker 01

  to connect that to real metrics that make sense, right? Because it's like, oh, well, people could say anything, right, to the, to the chatbot. How do I, how do I know one what's gonna come in, you know, either from a normal usage or malicious usage or whatever? And how do I connect that to any sort of metric around a model? So I think sometimes people struggle with this idea of metrics and gen AI models or gen AI systems. Could you help maybe clarify, like, what, what are some of the relevant metrics that, that people could think about in terms of these systems that might help them understand, you know, how the, how the systems are behaving?

• 25:55Speaker 02

  Sure. Yeah, that's a very spot on question. I guess before I talk about specific metrics, I'll just take a step back first. And if we sort of think at a high level, what is the same between predictive AI and generative AI? I think the structure of how you maintain reliability is basically the same, right? You need testing before deployment and you need monitoring after deployment. And it's also very similar to like traditional software applications, right? Where you have automated tests and automated monitoring. And so I think that part is the same. But the part that's much trickier for generative AI is that usually, as you mentioned, you don't have ground truth in the same way that you do for a classification data set,

• 26:39Speaker 02

  for example. And so the metrics that you used for evaluation are not as well defined. So you can't measure accuracy, you can't measure precision or recall. And the output of a generative AI model is also much more complex than just like a probability score. And so in that environment, it's it's very hard to determine how do we actually evaluate these things in a quantitative and objective way. So the approach that most of our customers take and most of the industry has gone in is basically using LLM as a judge. So you can craft these evaluation prompts that ask an LLM to evaluate some quality

• 27:23Speaker 02

  of some generated text. So it could be, you know, a very simple example is sentiment. So you evaluate the sentiment of of of some text. You could do that with traditional, like a sentiment classifier as well. But there are more sophisticated metrics such as, you know, detecting hallucinations against some ground truth document or measuring the relevance of the answer relative to the question. Or you might have more, we call them custom metrics that are sort of designed to be domain specific. So if you have like a refund chatbot, you can design a metric that measures if the chatbot adheres to your your company's refund policy.

• 28:08Speaker 02

  And so these metrics, they're very flexible because you can design the evaluation prompt in natural language. And in our tools and our open source libraries, we have a set of built in metrics. It's like a library of metrics you can choose from. But for many of our customers, they also extend those built in metrics to customize them to fit their business applications.

• 28:30Speaker 01

  So Kenny, you're mentioning this sort of idea of LLM as a judge, which is, you know, using a model to evaluate the model within in some sort of axis of a performance or some some quality, which definitely seems like a flexible option. But also, some people, you know, some people might be thrown off by this sort of circular using a model to evaluate a model. Also, there's sort of this, you know, you're you then have a model that's evaluating the model. So how do you evaluate the model? That evaluates the model and you kind of get in this.

• 29:11Speaker 01

  How have you all navigated that side of things, both kind of in terms of using using the larger model, making sure that the evaluations are are sound and also kind of transferable, you know, one model to the other. And, you know, maybe benchmarking the system over time because also the models you might want to use as evaluators might change over over time.

• 29:34Speaker 02

  Yeah, also a very good question, and it's a question that we get from our customers quite a lot as well. And the way that we generally approach the the evaluation workflow, which includes designing these metrics is not usually human judgment and taste is treated as the gold standard. But the problem with having humans evaluate every experiment with your LLM system is that it's very expensive and it's very slow. And so in the ideal world, you would design these LLM as a judge metrics that can mimic human preferences. And so in our software tooling, this is what we design specific workflows to help users to do.

• 30:15Speaker 02

  Usually when they, when a customer starts on a evaluation project, they'll, you know, of course, think about the evaluation criteria that's important, but then they'll also have humans do a small set of that evaluation. So maybe like 50 to 100 of these manual annotations. And from there, you can design LLM automated metrics and measure their correlation and accuracy against the human judgment. You usually need to iterate a few times to get that custom LLM metric as close as possible to the human judgment. But then, once you have that, it's very powerful, right? You have this automated metric that is a very good proxy for human judgment.

• 30:57Speaker 02

  And it's automated, which means you can run it at scale, you can deploy it during evaluation, but also in monitoring as well. And you can also potentially use that as a production guardrail in our firewall.

• 31:09Speaker 04

  Rick, I was wondering, I know we've kind of alluded to it that the two sides of the equation in terms of testing the models and then monitoring, could you talk a little about Citadel Lens and Citadel Radar, how you bring them to customers and what the relationship is between those two products that you're bringing to your customers? And how do you go to, how do you present them when somebody is interested in being able to bring this level of security to the models that they're interested in?

• 31:43Speaker 03

  Sure. First of all, now we are saying merging radar features into Lens. So Lens can provide both testing function and monitoring functions together at this stage. And as for the balancing or difference between testing and monitoring, as Kenny mentioned, especially in the case of LMM, how to customize human, say, setting back. So our system is in the concept that the system should follow the human, rather than human has to follow the system.

• 32:23Speaker 03

  In a sense, we set the human annotation or human judge as a first priority and try to our metrics to be customized to the human judge. So that is a very important point. And to make it happens, we need to, or the customer need to go through testing phase first, so that all the, say, metrics should be aligned with the human judgment. So that, and we can make use of the same custom metrics during the, say, the monitoring phase or firewall stage.

• 33:04Speaker 03

  So in a sense, even though final goal might be a monitoring or firewall, but the two, before going to that stage, how to test and customize a matrix is very important, I mean, very critical. To protect the safety and security and reputational risk. So we recommend, strongly recommend to start from testing phase first. So testing phase is not just testing, but customize your metrics into your, say, professional, the persons. So that is a testing phase. And after that, they can go into monitoring phase. So that is our approach to the customers.

• 33:47Speaker 01

  And I'm wondering, you know, either one of you could answer this, but why is it, and this may be obvious to maybe more of the software engineering type crowd, but maybe less so to some others outside of that crowd. Why is it important to, once you've tested your model, to actually monitor it online for maybe the things that are potentially problematic inputs? Whether that's a security thing, like a prompt injection, or maybe something outside of the, you know, some type of input that you want to filter out, like IP going in, or something that doesn't fit your policy?

• 34:28Speaker 01

  Why is, why is it important to have that monitoring piece and not just the testing piece? Because if I test my model and I convince myself that it can't be prompt injected, which I'm saying that sort of in jest because, you know, as you all know, every model, there's no perfectly aligned model. There's no, every model is vulnerable to various things. But let's say that I convince myself of high performance in one of these areas. Why then is it useful and necessary then to monitor that over time or in real time?

• 35:04Speaker 03

  Technically, probably Kenny is also, again, the best person. But even if the customers can go through the testing phase, the market condition, or say, Kandala, say, the human reaction may change over time. So if we were safe right now, but if anything, say, something new happens today.

• 35:30Speaker 03

  What we say guaranteed today may not apply tomorrow. So it's a very general things. But say, in that sense, keep monitoring is very important to protect our customers. And even if the market condition or the world condition changed or economy condition changed.

• 35:51Speaker 02

  Yeah, and just to give a concrete example of why you may want monitoring. So we really view them as complementary and you really need both if you want to make a system reliable. So for example, if you have like an answer quality metric that measures how how high quality an answer is, you should of course use that for testing to make sure that it meets some like, you know, eighty, ninety percent bar. But then in monitoring, you actually want to measure quality of the real answers that your chatbot is giving to real customers. Right? So from a quality perspective, it makes a lot of sense. From like a safety and risk reduction perspective, another example is that, you know,

• 36:32Speaker 02

  as you mentioned, Dan, during testing, you might test a bunch of prompt injections against your system. But then in in deployment, you have real users. Some of them are adversarial, some of them are actually trying to prompt inject. They may do it in creative ways that you haven't tested before. Right? And so you may want some some kind of guardrail that will automatically detect those attempts and filter them out, even if you're sure that the model is robust to, you know, ninety percent of these attacks.

• 37:00Speaker 04

  I'm curious, you guys have some open source available out there. I know one of the tools is LangCheck. Can you talk a little bit about your approach to open source?

• 37:12Speaker 02

  For context, LangCheck is a is our open source Python library that contains a suite of metrics that are built in that you can use for evaluating the quality of text. One of our motivations for creating this library is that, I think we launched it in October of twenty twenty three roughly. And around that time, there weren't a lot of sort of industry standard metrics and practices for evaluating text, particularly in non-English languages as well. So there was some focus on, you know, these metrics in English. But we work with a lot of customers that have Japanese texts or Chinese or German and these other languages.

• 37:54Speaker 02

  And we wanted to make a library of these metrics that anyone can use. And we view this as a as a pretty good starting point, right? So if you're, if you just need like one or two metrics, you're comfortable writing code, you can use line check and integrate that into your task pipeline or your monitoring system. But then if you want something production scale and you want an easy workflow to design custom metrics and test them against manual annotations, that's where our commercial product comes in.

• 38:23Speaker 01

  Makes sense, yeah. And as we get a little bit closer to the end here, there's so many things and this is such a, you know, so many in-depth areas to go in, which is why it's great that there's, you know, wonderful people like yourselves exploring the topic. But I'm wondering if we could maybe talk just a little bit as we close out here about what you're excited about, about kind of, yes, Citadel, but maybe the kind of general ecosystem that you're a part of. As you look to the future, you know, what's, what's exciting to each of you about how the ecosystem is developing, what's becoming possible with the technology?

• 39:06Speaker 01

  Yeah, what's inspiring to you or what are you thinking about in terms of the, the future? Maybe I'll start with you, Rick.

• 39:13Speaker 03

  Okay. So, yeah, now the chat GPT five is released and but when we say look back today from, say, five years later, I believe that, oh, this is a very premature, say, although something like that. So in that sense, the technology advancement in the AI field is so rapid. And in that sense, yeah, there may be a lot of risks coming in, but on the other hand, there are mostly infinite opportunities. So you can find that huge, say, variety of possibilities, not only just, say, the AI

• 39:58Speaker 03

  directory rated technologies, but I believe some of the material products or machines or anything will change, maybe within five or 10 years. So in that sense, we are in the midst of the, say, period where anything can change in that sense, especially technology rated.

• 40:22Speaker 03

  So there are lots of, say, risks may coming and we like to protect such a kind of risk as a company. But on the other hand, people can find many possibilities, opportunities for you to try. So I strongly believe that we, so even though there are lots of issues in the world, but people can enjoy or say, say, can, let's say, make best use of this opportunity, everybody.

• 40:51Speaker 01

  Yeah, that's great. What about yourself, Kenny?

• 40:54Speaker 02

  Yeah, I think as a consumer of AI in both my personal life and work, from a consumer perspective, it's really exciting to benefit from all the advancements in these new AI tools and models. I really loved O3 as a model in ChatGPT and I love using Cursor and I'm excited for these tools to become more and more agentic over time. And I think that's sort of the trend that you see. If you just look at ChatGPT, originally it was just like 3.5 and GPT-4 that just answer a question based on a forward pass of the model. But now these models will search the internet and it'll sort of reason and think about what to search next. And as a result, the outputs have become a lot better.

• 41:39Speaker 02

  So really excited for that to improve even more from a consumer perspective. And then from a business perspective, I'm really excited to help bring these capabilities to our business customers and help them use AI more reliably and more effectively in their business.

• 41:56Speaker 01

  That's great. Yeah. Well, thank you both for taking time to chat with us today. And thank you both for the work and thought that you're putting into the tools that you're building and the open source projects that you're putting out there. It's a great benefit to the, to the community and to the business world, of course. So thank you for the work that you're doing. And yeah, we'll look forward to, to keeping, keeping an eye on what you, what you evaluate and protect us from next. So appreciate you both and I hope you have a great evening. Thank you for joining.

• 42:32Speaker 03

  Thank you very much.

• 42:34Speaker 02

  Thank you for the conversation.

• 42:42Speaker 00

  All right, that's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Brake Master Cylinder for the beats, and to you for listening. That's all for now, but you'll hear from us again next week.