#296 Yeop Lee: How Coxwave is Redefining AI Evaluation

This episode is sponsored by AGNTCY. Unlock agents at scale with an open Internet of Agents. Visit https://agntcy.org/ and add your support. How is Coxwave Redefining AI Evaluation? In this episode of Eye on AI, host Craig Smith is joined by Yeop Lee, Head of Product at Coxwave. Together they explore how teams move beyond accuracy-only metrics to outcome focused evaluation with Coxwave's Align. We look at how Align measures satisfaction, trust, and task completion across chat, email, and voice, how LLM as judge pairs with human review, and how product teams search conversations to find hidden failure patterns that block adoption. Learn how leading companies design an evaluation stack that guides prompts, agents, and UX, which pitfalls to avoid when shipping updates, and which metrics matter most for success, including completion rate, CSAT, retention, and cost per resolution. You will also hear how to run experiment tracking with model and prompt change logs, set up governance that prevents regressions, and choose between SaaS and on premise deployments that meet security and compliance needs. Stay Updated: Craig Smith on X: https://x.com/craigss https://x.com/EyeOn_AI Eye on A. I. on X: https://x.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Cox Wave, our product align is an analytics solution for LLM-powered conversational AI products. So the way we think about it is we're the Google Analytics for products like ChatGPT or Claude. We've always like visited the US to understand, okay, what are the sort of developments in Silicon Valley? How are builders approaching building these new types of products and so forth? But I think this year and then especially next year is when we'll more actively push into the North American markets.

• 0:27Craig S. Smith?

  Build the future of multi-agent software with AGENCY. That's A-G-N-T-C-Y. Now an open-source Linux Foundation project, AGENCY is building the Internet of Agents. A collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on AGENCY. Including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

• 1:11Craig S. Smith?

  AGENCY also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation AI infrastructure together. AGENCY is dropping code, specs and services, no strings attached.

• 1:49Craig S. Smith?

  Visit agntcy.org to contribute. That's A-G-N-T-C-Y dot O R G.

• 1:59Speaker 00

  Hey, I'm Yaup. I'm the Head of Product at Cox Wave. I joined Cox Wave, I think... A little over two years back and but I knew the team since they were founded in 2021. The way that I got to know Cox Wave was actually so I met the founder of Cox Wave, his name is Gijang, at a government competition. So this was when Cox Wave was just starting off and at that time I had just left Reed and I had started my own company and so we were both just getting started off.

• 2:32Speaker 00

  And that that was sort of during the peak of COVID as well, so this government program was like a three day program, but none of the participants were allowed to interact with each other. Everyone was sort of locked in everyone's sort of separate room and we had to watch sort of each other's presentations over screen. But you know, Gijang had a wonderful presentation sort of outlining the vision of Cox Wave and I had given my presentation and during the award ceremony, we connected just, you know, swapping business cards. And during that competition, Gijang Cox Wave got first place, my company at that time got second place. And so that's sort of how the relationship started.

• 3:14Speaker 00

  And then we sort of went our separate ways, building up each, you know, I built up my own company, he was continuously building up Cox Wave. And then I had the opportunity to sell my company in 2023. And that was also when Gijang got the offer from a public crane company to sell two of the Gen AI products that he was building. So he had built Gen AI products since 2021. And this Korean public company, because they saw sort of the rise of ChatGPT and the potential of Gen AI, wanted to acquire these two products to sort of leverage internally as for themselves. And that's sort of when I joined Cox Wave

• 3:57Speaker 00

  because Gijang and Cox Wave was transitioning from B2C to the B2B space. So that's sort of the journey, yes.

• 4:05Craig S. Smith?

  Yeah, and the products that he sold, you mean Cox Wave no longer owns them? Or do you mean they were their customers of Cox Wave?

• 4:15Speaker 00

  Um, so we no longer own those two products. So I can give a brief introduction of those two products as well. The first one was a product called Hama, which is an image editing tool. So you know, like sort of background removing, et cetera. And so it was used by a lot of companies in Korea that were sourcing, like for example, foreign products. And they wanted to edit out different sort of labels and you know, edited in Korean. So that was sort of a big use case. And the second product was a product called EnterPix. EnterPix was a search engine for generative AI, you know, created images. So it was an image search engine.

• 4:56Speaker 00

  And so the company that acquired these products, they're Korea's largest font company. So they're sort of like the default font company in Korea. And they wanted to leverage this technology for their own sort of B two C use cases.

• 5:11Craig S. Smith?

  Oh, um, and the, so, so the premise of Cox Wave today is a, a platform that L L M

• 5:25Craig S. Smith?

  companies can use to evaluate their models? Or explain exactly what Cox Wave is doing.

• 5:34Speaker 00

  Um, so Cox Wave, our product Align is an analytics solution for L L M powered conversational AI products. So the way we think about it is we're the Google Analytics for products like ChatGPT or Claude. So what we focus on specifically is the conversations between a user and an L L M. So we actually focus on the interaction that's happening to help businesses and product managers understand. Is the chatbot working properly to our users having a good or bad experience? If not, why are they having a bad experience? How can we fix this and so forth? And so I'm sure you know, you've seen a lot of these evaluation platforms that have been coming up.

• 6:16Speaker 00

  And I can sort of explain a little bit about the way we differentiate with these sort of other evaluation platforms. So these evaluation platforms that exist today, they focus on an area called objective evaluation. So this is sort of, you know, a product manager intends the L L M to say a certain thing. And you evaluate, does the L L M actually say that in a bunch of different scenarios, right? Cause you don't want the L L M to just, you know, go go off rails. Uh, where we focus on is, okay, let's say that the L L M is already saying what the product manager intends the L L M to say. But that response may be a good response for a certain user, but also maybe a really bad response for someone else, right?

• 7:00Speaker 00

  Um, I'll just give you a quick example, like, let's say there are two users, they ask the same question, what is the best food in Korea? And the L L M responds, oh, the best food is uh, samgyeopsae, which is a type of pork, right? Um, but one user says, oh, thank you so much for letting me know. But the other says, oh, but like what what restaurant sells this, you need to tell me the restaurant, right? So in that case, for the second user, the response wasn't sufficient enough. So it wasn't the best sort of most ideal response, right? Um, so that's sort of the areas where we pinpoint and analyze to help businesses understand. Okay, is this certain L L M response really a good response for this user? How, what is an optimal response and so forth?

• 7:45Speaker 00

  So that's sort of an area we called subjective evaluation and feedback analytics.

• 7:53Craig S. Smith?

  Yeah, it's an interesting area, uh, because initially.

• 8:01Craig S. Smith?

  You know, there's reinforcement learning with human feedback to try and guide L L Ms uh to answer appropriately and not hallucinate. Uh, and, and that's been successful to, to a degree. And then there are, you know, now there are systems, uh, multiple models that talk to each other and uh for providing an answer. And the reasoning models and all of that. Uh, and, and for a time,

• 8:38Craig S. Smith?

  evaluation relied on benchmarks and there were all these benchmarks. Uh, that got increasingly complex, but users figured out or the market figured out pretty, pretty quickly that you can train to a benchmark. So, you know, Grok three comes out and they show their chart and they, they do, you know, so much better on this or that benchmark than any of the others. It doesn't mean that the user experience is better. So this is, uh, as you said, did you call it a subjective evaluation?

• 9:20Speaker 00

  Correct, yeah.

• 9:22Craig S. Smith?

  Yeah. So, uh, you know, from the point of view of a user, do you, do you train, uh, I would guess that this is done by, by evaluation L L Ms that talk to the primary L L M. And then there's some scoring or something. Is, is that right? Or how much human intervention is there in the, in the judging?

• 9:50Speaker 00

  Yeah. So, um, we also leverage like a lemon as a judge as well. But then we do have like human interactions that are built into our evaluation system. So for example, when our customer is onboarding, um, our system will ask, will do sort of pro, um, initial sort of judging and it'll sort of show those examples like. Is this a sort of like an example of user dissatisfaction or not? And the reason why we do this is even if, for example, we just use sort of like a standard model, depending on the conversational product, right? It's very different. And so what we realized is this area of subjective evaluation ultimately is personalization, right?

• 10:35Speaker 00

  To the user. But that also means our platform and our product has to be personalized to our customers as well. And so that sort of type of interaction is built in. And one of the areas where we focused on a lot is our search engine, which is a search engine for conversational data. So we have a natural language interface where if you type, for example, I want to find messages where the user is expressing distrust against the chat box, then our search engine will go across the conversational data and identify that. And this is for even conversations where the user isn't explicitly saying, I don't trust you.

• 11:17Speaker 00

  But any sort of implicit feedback that you can extract from the conversation will identify as well. And so that's sort of the system that we've built in place for our analytic system.

• 11:29Craig S. Smith?

  Yeah. And when you say there's a conversational interface, is that for the customer,

• 11:38Craig S. Smith?

  for Coxwave customer to query

• 11:44Craig S. Smith?

  your platform about the L L M it's evaluating? So that's another layer? Is that right?

• 11:51Speaker 00

  That's, that's correct. That's correct.

• 11:54Craig S. Smith?

  Yeah. And you would have to evaluate that L L M as well.

• 11:59Speaker 00

  Correct, yeah. So what we do is

• 12:02Craig S. Smith?

  It's kind of a house of mirrors.

• 12:05Speaker 00

  That's true, yes. So we work very closely with our customers to help them understand the performance of our search engine as well as our copod. So that it ultimately is up to par to their expectations. And initially, like honestly, like right out of the box, like, for example, the accuracy is, let's say, eighty percent. But we all, but the system gets better and better as our customers leave more feedback as well. Because our search engine, right? For example, same example of user is expressing distrust, right?

• 12:40Speaker 00

  There may be new cases of user distrust, but our system needs to learn as well to understand, okay, for this company, for this use case, what is distrust? What does it look like? Right? So we always tell our customers, if you use our product more, if you give more feedback, it'll get better and better and so forth.

• 13:01Craig S. Smith?

  Yeah. And how does someone use this? Is this a SaaS product that they access through the web? Or is it an API that they build into their their system? Describe how that works.

• 13:18Speaker 00

  Yeah. So we have a SaaS product. As you mentioned, they can just go on the web. They can log in and just leverage our platform. We have an SDK that helps our customers ingest the conversational data into line so that it's available for analytics. The other is we have a on-premise self-hosted solution as well. So these are for our enterprise customers, you know, who value privacy, security. They have more rigid data governance requirements, et cetera. So we'll provide that option as well. And we are looking to expand into an API option too, but it's in our product roadmap.

• 13:54Craig S. Smith?

  Yeah. And how, how many conversations? Does this run live? As well, if you run a customer service

• 14:08Craig S. Smith?

  AI system, will it monitor the reactions that were the exchanges live and then maybe signal to somebody that

• 14:22Craig S. Smith?

  that it's not the customer service bot is not answering properly or is creating problems? And, and then the more important question is after the customer knows this. Uh, what do they do? How do you correct it?

• 14:41Speaker 00

  Yeah, um, so we, for our SaaS product, it runs near, near real time. For enterprise ones, we usually run it in batch, um, especially because there are some limitations on the hardware, et cetera, right, of our enterprise customers as it's on premise.

• 15:00Speaker 00

  For the fixing part, so our job is one to sort of identify, help them monitor, analyze the impact, to help them understand and prioritize what are the issues that they should really be focusing on, right? And then in terms of the fixing part, because like our customers, they have many different ways that they're building these systems. Some are just fine tuning all the models, some are just, you know, using prompt engineering and so forth. So we can provide guidance. We have a separate consulting service that helps provide this guidance. But other times our customers just go on and fix it themselves. And they'll understand when that sort of fix has been made. And they can add that data into the metadata that they're logging into line.

• 15:44Speaker 00

  So they'll understand, okay, after this change has been made, did the sort of problems disappear? Or are these problems still happening and so forth?

• 15:54Craig S. Smith?

  Yeah, uh, how many uh, conversational interactions? Do you typically screen with a customer? Uh, is, and, and then it, it's an ongoing thing, uh, presumably from your point of view, that's what you guys want it to be. Uh, or do companies get to a point where they feel pretty confident? In their model and, uh, and stop evaluating?

• 16:26Speaker 00

  Um, so like we, it's such a huge range, like from thousands of conversations to like millions of conversations per day. Um, especially because like, um,

• 16:38Speaker 00

  Companies that are running like contact centers, for example, with AI, like there's a lot of volume that's happening. There are other companies that are running character chats that have a lot of daily usage as well and so forth. Up till now, the customers that we've engaged in, they haven't left us. And the reason is this sort of evaluation and analytics is it's never over. Especially because new LLMs come up, things that are working before stop working, right? And as users, more and more users come into the platform and there are new sort of user engagements. This type of new user behavior leads to new problems that need to be analyzed, need to be fixed and so forth.

• 17:22Speaker 00

  And again, these sort of new insights can propel new feature ideas, new sort of interaction ideas and so forth. And these are constant monitoring, analytics and evaluation.

• 17:36Craig S. Smith?

  Yeah, the

• 17:40Craig S. Smith?

  the this issue, are you work, you're working, I presume primarily with application companies, companies that are building up applications on top of either

• 17:58Craig S. Smith?

  open source model that they've fine tuned or that they've hooked up to a

• 18:05Craig S. Smith?

  vector database with a RAG system or

• 18:09Craig S. Smith?

  or do you also work with the foundation model companies themselves, the companies that are are coming out with their own models, listing them on Hugging Face and and hoping to get a piece of of that that pie that's dominated obviously by OpenAI and Anthropic and.

• 18:32Craig S. Smith?

  A handful of others.

• 18:34Speaker 00

  Yeah. So as you mentioned, we mainly work with the application layer companies. And these are also enterprises that are building these applications for their internal use cases too. But we've begun to expand our work with foundation model companies as well. You mentioned Anthropic, so we're one of Anthropic's closest partners in Korea. So a couple of months back, we actually hosted Asia's first Builder Summit. So we had the CPO of Anthropic, Mike Krieger, in Korea as well. And we had this sort of huge event that we had hosted. So that sort of work is something that is ongoing as well. As Anthropic is also looking to expand its presence in Asia and Korea.

• 19:18Speaker 00

  As well as a company called 2AI, they are a foundation model company with a presence in Korea, India, the US. And they have a consumer product called Chatzutra. Also, they offered their foundation model to as via API to enterprise customers as well. So we work closely with them too.

• 19:38Craig S. Smith?

  Yeah, yeah. I've had 2AI on the podcast. Yeah, and are most of the applications that you're working with? Are the models conversing in Korean?

• 19:56Craig S. Smith?

  Or is it predominantly English, even though your market presumably is East Asia?

• 20:03Speaker 00

  So most of the language is actually in English. And very interestingly, when we first launched Align, we made the decision initially to not support Korean. And so our product was just in English. And our main target market was actually, we started off with India. And so, most of our customers, enterprise customers as well, are based out of India. And our sort of support for the Korean markets, including like localizing our product, supporting the Korean language and so forth, started early this year. Like before that, we have Korean customers, but they were kind of forced to just use the English version of our product.

• 20:49Speaker 00

  But yeah, now we're seeing more and more, you know, Korean customers come on board. And the reason for this was actually quite, quite clear. It was because when we first launched our product, there just weren't as many Korean Gen AI products in the market. And so we had to, as a startup, right, pick and choose and really focus. And so that sort of led us to the Indian markets.

• 21:13Craig S. Smith?

  Build the future of multi-agent software with AGENCY. That's A-G-N-T-C-Y. Now an open source Linux Foundation project, AGENCY is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on AGENCY, including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

• 21:57Craig S. Smith?

  AGENCY also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than seventy-five other supporting companies to build next-generation AI infrastructure together. AGENCY is dropping code, specs, and Services, no strings attached. Visit agntcy.org to contribute.

• 22:39Craig S. Smith?

  That's A G N T C Y dot O R G. And and you mentioned Align, Align AI is the flagship product from Cox Wave, is that right?

• 22:53Speaker 00

  Correct.

• 22:54Craig S. Smith?

  Um, and and the what L L M do you guys use? Uh, on the conversational side and then on the evaluation side and in uh in looking at the the conversational uh data that comes in from from a customer.

• 23:22Speaker 00

  Yeah. Um, so we use a bunch of different models, but then our main is actually Anthropic's Claude models. Um, also explains kind of why we have a good relationship partnership with Anthropic. Um, we also have our own custom embedding models that we've trained to be very specifically good at retrieving conversational data. Um, so that's sort of our own internal technology. And also, there are like a bunch of different like smaller analytics tasks. And depending on the task, we've sort of found which model works the best. And we've um sort of put that L L M inside. Um, and also some like do run on OpenAI uh models as well. Yes.

• 24:07Craig S. Smith?

  Yeah, and uh you know, that the foundation field is changing quickly. Uh, I mean, it seems to have slowed down a little bit, but

• 24:21Craig S. Smith?

  uh, do you do you keep uh sort of rotating in the latest foundation model? Uh, so that you're

• 24:32Craig S. Smith?

  Keeping up with the market and, you know, not getting stale. I've always wondered how companies handle that.

• 24:39Speaker 00

  Yeah, so that's always like a big challenge as well, but what we do is we don't always switch out for the newest model, but we do in like very rigorous internal testing to understand, okay, with these new models, how do they perform against the current models that we're leveraging on these specific analytics tasks? But because what we found is, you know, like even these like newer models, like sometimes they're not the best at the analytics task that we need completed. Also for our enterprise customers, like switching out for the newest models, like every single time that change always introduces new risk as well. So we'll always go through that internal evaluation process.

• 25:22Speaker 00

  And sometimes we'll make that switch over and sometimes we'll stick to the model that we're using.

• 25:28Craig S. Smith?

  Yeah, you've you've been around, what did you say, since 2021?

• 25:39Speaker 00

  Yeah, so with the company I started off, I think, early 2023. Yeah, and then this company has, Cox Wave has been around since 2021.

• 25:49Craig S. Smith?

  That's what I meant, yeah, yeah. The

• 25:56Craig S. Smith?

  initially Cox Wave,

• 26:01Craig S. Smith?

  the two products that they sold, was that Hama and EnterPix?

• 26:06Speaker 00

  Yes, correct, Hama and EnterPix.

• 26:08Craig S. Smith?

  Is that's what you said, yeah. And and you've recently raised money

• 26:16Craig S. Smith?

  and and are putting all of your your focus on

• 26:23Craig S. Smith?

  On Align AI, is that right?

• 26:26Speaker 00

  That's correct.

• 26:27Craig S. Smith?

  Yeah, and is this a crowded market? I mean, everyone's talking about evaluation, but I haven't spoken to many companies in the evaluation space.

• 26:40Craig S. Smith?

  Is it a growing market? I mean, how do you guys see the market globally?

• 26:47Speaker 00

  So it definitely is a growing market, especially as like enterprises and more and more companies start leveraging generative AI, not only for their internal use cases, but for customer-facing use cases. So it is growing and the need is very clear, especially because as a company and builder, if you can't trust the output of your LLM, then you can't really introduce it to your customer, right? Especially for larger brands. The way we differentiate, though, is that area of subjective analytics. And honestly, we haven't seen that many players in that space yet. But as the sort of world moves more and more into, you know, hyper-personalization,

• 27:28Speaker 00

  where these LLM systems are hyper-tuned to that specific individual, right, and very flexible, adaptable, this area will become more and more important.

• 27:39Craig S. Smith?

  Yeah. Is it?

• 27:43Craig S. Smith?

  How affordable is the platform for companies? Is it something that's within reach of a relatively small startup? Or is it really focused on, you know, Fortune 500 or Fortune 1000?

• 28:02Speaker 00

  Yeah, so we do have a SaaS plan. That's $4.99 per month. So it is more affordable. And then we also do have like a design partner plan for earlier companies, especially because at that time, these companies are just starting to understand how their users are interacting with their products. And we want to be good sort of growth partners for them as well.

• 28:26Craig S. Smith?

  Yeah, uh, the, uh, is there, why did you settle on Anthropic as your

• 28:34Craig S. Smith?

  base model? Uh, is it through an evaluation? Uh, or was it based on business relationships and that sort of thing?

• 28:43Speaker 00

  Yeah, so it was on both. Um, so one, we internally evaluated the Claude models, and they performed really well on the analytics and evaluations task. And then the second was um, the sort of shared philosophy that both our companies have on, you know, building safe, responsible, reliable AI. Um, especially Anthropic, you know, they have the concept of constitutional AI as well, right? And so those are very important values, especially when we were going to our enterprise customers. Like, we wanted to have a foundation model partner that shared the same values as us, that we could trust, and so forth. So that also played a big role.

• 29:21Craig S. Smith?

  Yeah. Uh, do you work with, like, Hugging Face to, uh, provide evaluations there? Uh, I mean, it seems their leaderboard is really crowdsourced, but there's all kinds of problems with crowdsourcing. Uh, I mean, can be gamed pretty easily. Is that, uh, is, do you work with Hugging Face, or, or, or do you, uh, publish rankings of, uh, L L L L M's that are in the market?

• 29:55Speaker 00

  Um, so we don't, uh, work with Hugging Face yet, or publish that. Um, especially because what we're analyzing is not a specific model, but we're, yeah, evaluating the product itself. Um, so yeah, not yet. But one thing that we do do, though, is we work very closely with consulting companies. And so one of our closest partners is P W C India. Um, they were our first enterprise customer, as well as enterprise partner. Especially because, for example, I'll just give you the example of PWC India. They have thirty thousand employees in India alone. And they have so many verticals that they're working on. And it was very clear that generative AI can be very transformative,

• 30:40Speaker 00

  not only for their internal use cases, but they're also seeing a lot of, you know, demand from their customers as well on how they can leverage generative AI. And so we've been a very close partner to them, providing not only analytics and evaluation, but also, as I mentioned earlier about, okay, if these problems are there, how can we fix it? So we provide that sort of consulting layer service as well.

• 31:05Craig S. Smith?

  I see. And so where is this going? I mean, this field is, as you said, it's growing. It's becoming increasingly integrated into the global economy. The number of LLM-based applications has exploded. I mean, is it just from your guy's point of view right now? Just working to build market share? Or is it, or is there something on the horizon that you see as a new product that Cox Wave is looking to launch?

• 31:47Speaker 00

  Yeah. So increasing market share, of course, is very important, but another piece is a multimodal analysis. So nowadays, these, right now, we focus mainly on text. So most of these are chatbots. But then now, you know, with even with like ChatGPT, you go across different modalities. And so providing that layer of analytics for these multimodal products, helping businesses understand, for example, okay, for this user, like, when do they prefer one modality over the other? Are there some modality features that aren't performing as well and so forth? To help these companies make those sort of feature design choices as well. So that's a very important thing on the roadmap.

• 32:30Speaker 00

  Another thing is this thing about personalization, right? The reason why I think it's such an interesting problem, but also very complex. Is let's say you get all this sort of user feedback that's coming in and you understand this user. But with that information, you can't always like make that change immediately into the product, right? Um. So for example, I'll just, um, like, for example, when I use chat GPT, sometimes there's this problem of with chat GPT memory where it's just remembering everything. There are things that I don't want it to remember and actually make that change. And some of my colleagues, they've made the decision to just shut off the memory feature.

• 33:13Speaker 00

  Um, because that that gets really annoying, right? So that type of like, um, decision. And how we can sort of help product managers make that decision in a more informed way, right? What is, what are things that should be remembered and changed? What are things that users probably don't want changed? Um, and actually helping them make those decisions, that's a very important thing on the horizon for us.

• 33:37Craig S. Smith?

  Yeah, actually, that's interesting. You mentioned that, but that seems to, that problem.

• 33:43Craig S. Smith?

  Seems to be relatively new and getting worse. Um, uh, even if you start a new.

• 33:54Craig S. Smith?

  Instance or new conversation or

• 33:59Craig S. Smith?

  a new session.

• 34:02Craig S. Smith?

  Is sometimes, uh, it'll, it'll mix the, the answers from your previous conversation into the current one.

• 34:12Speaker 00

  Yeah.

• 34:13Craig S. Smith?

  Uh, which you're right, it can be, uh, can be annoying and you've have to repeat to the model that I've stopped talking about that. I'm talking about something new.

• 34:24Speaker 00

  Yeah.

• 34:24Craig S. Smith?

  Uh, yeah, forget about that. Uh, I didn't realize you could turn off the memory feature, but, uh.

• 34:31Speaker 00

  Yeah.

• 34:31Craig S. Smith?

  Uh, have you guys looked, um, excuse me, like?

• 34:38Craig S. Smith?

  Uh Grok three, I mean, do you have customers that are building applications on uh on the Grok models?

• 34:47Speaker 00

  Um, so far off the top of my head, I don't think we have customers building on the Grok models. Uh mainly OpenAI Anthropic and then for um our enterprise customers, they're usually building on top of like Lama models or

• 35:04Craig S. Smith?

  Right.

• 35:04Speaker 00

  models. And then there are like large enterprises in Korea that have their own models. So LG has their own model. Samsung has their own Gauss model. Um SKT has their own model as well. So they've been trying to figure out, okay, how can they leverage these internal models that they've built? And just bring up the sort of performance on that.

• 35:26Craig S. Smith?

  Yeah, uh does uh Korea have a national model? Some countries are are building uh, you know, sovereign models, whatever you want to call them.

• 35:37Speaker 00

  Yeah. So right now we don't, but our uh recently elected president, uh Lee Jem Young, he's pushing for this as well. So I think there is like a new, almost like a hundred billion dollar um AI fund, like eighty to hundred billion dollars AI fund, that'll be used to build um sovereign AI as well as, you know, just supporting startups ecosystem so forth.

• 36:00Craig S. Smith?

  Yeah.

• 36:01Speaker 00

  Yeah.

• 36:01Craig S. Smith?

  Yeah. Uh and I mean, I've been impressed by the AI, the Korean uh startups I've always spoken to because there's so much um government support for startups, as you were saying, that's how uh you met the founder. Uh is is that increasing? Uh and and do you pay much attention to how things are organized in the United States where it's it's largely um a private?

• 36:37Craig S. Smith?

  You know, a private exercise or private initiatives and fund AI startups.

• 36:47Speaker 00

  Yeah, yeah. So we pay attention to the US as well. And then the government support has been increasing too. Especially with the sort of rise of AI. The government is looking for new partners that they can work with to help, you know, not only build sovereign AI, but just provide this level of support to the startup community. So recently, the, there was a new government position created. I think it's like a policy advisor position for AI. But the person who's currently in that advisory position, he used to work for Naver. And I believe he worked on the foundation model at Naver. And so he has an understanding of the role startups should play,

• 37:31Speaker 00

  how this ecosystem can play out and so forth. So just by having that new position created in the government, that also provides like a very strong channel and voice that startups can, you know, present to the government as well. Another thing that's, that we're doing as well is we, we had a couple of other companies, including Ritten, which is Korea's largest B2C GenAI company, Liner, et cetera. We created the Generative AI Startup Association. And so this is an association of GenAI startups and companies coming together. So we organize a bunch of different events to share knowledge, learnings across one another, as well as, you know, discussing with government policymakers to understand, okay, what are their priorities?

• 38:16Speaker 00

  What are things that us as startups really want to see, you know, changed in the market? How can we work together to, you know, create sort of the good future that we all envision?

• 38:28Craig S. Smith?

  Yeah, uh, the, um, are most of the startups focused on the Korean-Japanese markets? Uh, or are most of them focused on the, uh, English language speaking market?

• 38:45Speaker 00

  Yeah. So I think traditionally it has more been focused on Korean, Japanese, Asian markets. But now you're seeing more and more sort of, um, English-speaking, US-facing products as well. And I think the reason for that is actually because of these like foundation models and how good they are with languages. In the past, like the language of, you know, Korean has been a big barrier for large companies from abroad coming in, right? But now with these L L Ms like Claude speaking Korean is just so natural. Um, that language is no longer a barrier. So for these Korean companies, now if you don't go to the US markets or these larger markets, there is just no way of surviving. So it ultimately comes down to survival.

• 39:31Craig S. Smith?

  Yeah. And on that, on the, the, the multilingual, uh, L L Ms, uh, I'm, I use, uh, uh, because I spent a lot of my life in China. I, I, I work between English and Chinese in, uh, in the, with, with various L L Ms. Is the Korean, uh, to English as, I mean, can Claude, uh, two, is it 2.7?

• 40:01Speaker 00

  3.7.

• 40:02Craig S. Smith?

  3.7, I always get that wrong. Uh, does Claude 3.7 speak Korean as fluently as, for example, it does Chinese?

• 40:12Speaker 00

  Uh, yeah, it's, it's so fluent that last time, like an anthropic researcher came over for the Builder Summit. Um, we asked him like, why, like, does anthropic just have a lot of like Korean teammates? Um, like, how is Claude so good at Korean? Um, and even like they're surprised that it's so good at Korean. Um, but where in, where in China did you live? I lived for like 13 years in China as well.

• 40:41Craig S. Smith?

  Oh yeah, I saw that you had spent time in Shanghai. I was, yeah, I was in, I was the New York Times bureau chief in Shanghai for quite a while and then I was in Beijing

• 40:55Craig S. Smith?

  running the business side of the New York Times. Since my audience is primarily, not only, I mean, there are 150 countries or something that listen, but is this, but most of the listeners are in the US and Canada and the UK.

• 41:20Craig S. Smith?

  I mean, the English speaking markets. How big has your push been into these markets?

• 41:28Speaker 00

  So up till now, we've been focusing primarily on the Indian markets as well as Korea. But now we're starting our push for the US markets. So up till now, like we've always like visited the US to understand, okay, what are the sort of developments in Silicon Valley? Like how are builders approaching, building these new types of products and so forth? But I think this year and then especially next year is when we'll more actively push into the North American markets.