Ep 80: CEO of Surge AI Edwin Chen on Why Frontier Labs Are Diverging, RL Environments & Developing Model Taste

Edwin Chen is the founder and CEO of Surge AI, the data infrastructure company behind nearly every major frontier model. Surge works with OpenAI, Anthropic, Meta, and Google, providing the high-quality data and evaluation infrastructure that powers their models. Edwin reveals why optimizing for popular benchmarks like LMArena is "basically optimizing for clickbait," how one frontier lab's models regressed for 6-12 months without anyone knowing, and why the industry's approach to measurement is fundamentally broken. Jacob and Edwin discuss what actually makes elite AI evaluators, why "there's never going to be a one size fits all solution" for AI models, and how frontier labs are taking surprisingly divergent paths to AGI.

• 0:00Speaker 00

  As the CEO of Surge, Edwin Chen gets a front row seat to everything happening in the foundation model ecosystem. Surge, which is worth a reported $24 billion, works super closely with all the top labs on improving their models. And Edwin had a really interesting perspective on a bunch of different things. I'm Jacob Effron, and today on the supervised learning, we talked about Edwin's learnings from working with the top labs and the divergent approaches he's seen them take so far. We talked about where models are today, his view on RL, RL environments, RL as a service, and that whole startup ecosystem. We talked about what it means to have model taste and what really good researchers do. And we also hit on how Edwin's changes to use from there being one model to rule them all, to there being a constellation of different models. Just a fascinating conversation with someone who gets to see the future being created every day. Without further ado, here's Edwin.

• 0:44Speaker 00

  Edwin, really excited to have you on the podcast. Thanks so much for coming on. There's a lot of things I'm excited to explore today with you. But one place I figured I'd start, I listened in prep for this to a few podcasts you've been on, and a lot of things you said that were pretty interesting. One thing you've talked about pretty consistently is just the consequences of choosing bad benchmarks as model builders. And I think you've specifically called out LM Arena and some of the other things that folks are optimizing for. And so I'd love if you could just elaborate a bit on the kind of consequences of some of these optimizations.

• 1:15Speaker 01

  I think the way you should think about LM Arena in particular, is that when you optimize for LM Arena, you're basically optimizing for clickbait. The mental model you should have in your head of LM Arena is that users go on to LM Arena, they issue a prompt, and then they get two responses. And then their goal is to basically vote on which one is better. But they actually aren't carefully reading their responses. What you're doing inside is, okay, they just scroll through the two responses. After two seconds, they just pick which one is better. And so they're not carefully reading them. They're not seeing, oh, yeah, this one followed all of my instructions. This one was completely accurate. This one was well researched. It was high quality and so on. They're not doing any of

• 2:00Speaker 01

  that. What you're doing inside is again, they're reading through very, very quickly, like maybe one, two seconds. And they're like, okay, which one impressed me the most? What they're going to optimize for is, okay, which response had more emojis? Which one caught my attention? And yeah, the thing that caught your attention will be a lot of emojis. It'll be a lot of formatting. So responses that are, that just contain a lot of markdown, a lot of headers, and so on, so on. They will just naturally prefer responses that seem that are that are longer. Like, yeah, if it's longer, then sure, it has a sheen of expertise to it. I do not, do not, do not reading these things through. Like, for example, I was actually looking through one of the LMS's data sets recently. So they've published a bunch of these online. And I just saw all these incredible errors.

• 2:41Speaker 01

  So one of the errors was like, I think the prompt was, tell me the, tell me all the divisors of 1452. And one of the responses said, oh, there's only one, there's only one divisor of 1452. And that's the number one. And the other one got it completely correct. It was, I was like, I think the problem was actually, what are the, what are all the divisors under, under six of 1452? And obviously, one, two, three, four, and six are all divisors. And so that was the only response. But if you look at a data, guess which one the reader preferred, I guess, which user preferred? The user preferred the one that was completely wrong. And again, like, if you even think about it, like, this is a fairly simple mathematical question. But if you ask a more advanced prompt, what, are you going to, are you going to research it? Are you going to go fact check it yourself?

• 3:25Speaker 01

  No, that's just not what you're going to, you're going to do as a user. And so what ends up happening is that users, again, they're, they're basically optimizing for whichever one caught their attention. It's almost like a tabloid.

• 3:37Speaker 00

  And it's interesting, it seems like something that pops up in, in a bunch of different consumer use cases. I remember there was like, maybe a study of ChatGPT for medical responses. I forgot, maybe a year ago or so. And they, you know, the responses rated way higher than physician responses. But I think when they actually unpacked why, it was like literally just because the responses were longer.

• 3:54Speaker 01

  Yeah, exactly. That's one of the, the big things that we've seen. It's that even when you haven't intentionally trained on LMS's data, like, maybe you put a bunch of your models on, on, on the, on the site. And then, okay, you have 10 different models that are on the site. And you're just going to, you know, A, B, test which one and pick the one that performs the best. Again, users just naturally prefer responses that are longer. It's like the easiest way to think that response is high quality. And so we've seen a lot of models. I just think when you go through that process, you just naturally end up responses that are two, three, four X as verbose as ones that aren't.

• 4:28Speaker 00

  And obviously, like, you know, LMS, you know, isn't the only thing that people are optimizing for probably and, and, you know, leaving things on the table. What are some other examples you've seen of, you know, where these, where not optimizing for the right thing really leaves a model on the wrong track?

• 4:42Speaker 01

  I think the problems especially arise when.

• 4:47Speaker 01

  You are both optimizing on the wrong kind of, like you're training on the wrong kind of data, you're optimized for the wrong objective function. And then in conjunction with that, you don't have the right measurements in place. So for example, earlier this year, we were actually starting to work with a new team. A lot of the researchers were telling us like, yeah, they were suspecting that their models were getting worse, but they didn't have any quantitative evidence of it because they didn't have the right measurements in place. So we basically dug into the models for them. And what we found is that actually over the past six to 12 months, their models had actually regressed. And so what happened was the data that we're gathering from supposedly expert coders, the

• 5:30Speaker 01

  data we're gathering, the raiders were just, they weren't executing the code. They weren't checking the code carefully to see that it was, it was actually correct. And so what they were doing instead was they were, again, like very similar to the LMS use case, they were giving training data that was essentially full of flowery language and grandiose claims. Like, oh yeah, like here I produced this amazing program for you that does ABC things. And they had taken the time to execute it. You know, executing can actually often be, times be pretty hard. You have to solve these libraries, you have to have all these interest filters in place, you have to make sure that you actually understand the language and so on, so on. And so the code was actually just completely, either completely wrong or just full of all

• 6:15Speaker 01

  these subtle bugs that people wouldn't notice until later. And so at the end of six to 12 months, because they didn't have any, actually, they didn't have any actual measurements in place to see whether or not the, the models were actually improving or not, they were, they were optimizing for this. And it's kind of crazy because really the, the whole industry is moving forward, coding is such a, such a, such a hot area where everybody else is progressing. And then whatever teams is basically making negative progress because you don't have the right data, you don't have the right measurements. I think it's a real concern where. Enough people in the industry just aren't paying attention to the quality of data that they're receiving and whether or not they're measuring the right things.

• 6:54Speaker 00

  I mean, it's interesting how things can actually go south, given all the model improvements that are happening if you're taking shortcuts on the data side. And then also, you know, I was struck by in that example, a big part of the problem is just not even knowing for six months if your models are getting better. And I think this is something interesting that we've seen across the industry now, where it's not always clear. Like, you know, I think there was a generation of model improvements where it was like blindingly obvious that models were getting better in some ways. And now it feels like it's a little more challenging to tell at times. What are like the best companies doing to figure out like, hey, you know, on a month-to-month basis, like, are our models getting better?

• 7:29Speaker 01

  So historically, all of the companies that paid really, really hard attention to benchmarks, like these very academic benchmarks that have been created by the research community. And not all the labs realized the problems with basically benchmarks. It's almost like very easy to optimize for benchmarks. Like, models, at the end of the day, they're very good at hill climbing, very concrete, very specific, very narrowly-defined objective functions. And so what would end up happening is that the models would make a ton of progress on the benchmarks. And sometimes it would be fake, because the benchmark data would actually be in the training data and people wouldn't realize. Or even if it wasn't fake, per se, like, they were actually proven on the benchmarks, what they wouldn't realize is that, it's like, because you've narrowly focused on benchmarks,

• 8:13Speaker 01

  but don't have measurements in place outside of the benchmarks, people wouldn't realize that, yeah, sure, they got really, really, really good at some narrowly-defined problem, but their models were actually getting worse, kind of like on more real-world problems. Maybe an easy example would be, imagine optimizing for SAT. Like, you, as a high school student, sure, you can spend thousands, like, whatever it is, hundreds of hours optimizing for the SAT. A SAT is a very, very narrowly-defined set of problems. Like, sure, it's like, what is it? It's like reading comprehension and analogies and like vocabulary. But it's not really measuring your ability to, I don't know, it's like to write well, or your ability to perform complex problem-solving in your own world, or your ability to do all these other things that are outside of the SAT system.

• 8:58Speaker 01

  And so what ended up happening is like, again, like we would actually see companies, uh, probably a lot of frontier labs. They would suddenly get all these impressive scores on the benchmarks. And, uh, we would just play around with the models ourselves, but like, no, your models only got a lot worse. Like, uh, like, like, like even before they told us that, um, that they were optimized for the benchmarks. It, it became prevalent enough that every now and then, every now and then, we would start playing around with a, like a model that one of the frontier labs would give us. And like, did, like, why did this suddenly drop in quality? And like, oh, you guys probably gathered a lot of synthetic data to improve on your benchmarks. And lastly, yeah, is that what you guys have seen? They're like, yeah, like we suddenly doubled the performance on XYZ benchmark.

• 9:41Speaker 01

  Um, and we didn't realize all, all, all of the other consequences. So I, I think that's a real concern. And then, um, like basically what the best relatives have realized is that the, kind of like the only way to measure their, uh, the performance of their models is to run, uh, proper human evals. And the way it works is you're essentially asking people to, again, like high quality people, people who are very, very diligent, people who are, but paying attention to the content of the, uh, of the responses, but then are also really sophisticated and have a lot of taste to make sure, yeah, like this is a fun personality. This is the kind of style that, uh, maybe like the frontier labs want their, when our models emulate.

• 10:18Speaker 01

  And it's basically mimicking the, kind of like the, the real user, real world experience, like all the diversity, all the complexity, all like the messiness of the real world. Without the artificial constraints of benchmarks. And then it's also laying on the quality from, again, like really, really, um, experienced and really, really trusted raiders as opposed to just anonymous people who aren't paying attention all on this. Like this, this, uh, this, uh, process of going through these rigorous human evals, that's, that's been a gold standard for all the frontier.

• 10:47Speaker 00

  I mean, beyond, I guess, paying a little more attention, maybe like, what have you noticed about what makes a really good evaluator have taste?

• 10:53Speaker 01

  So at high level, it might be the three, these, uh, the following three things. One would be. Certainly expertise in itself. Like if you're judging a, yeah, like a lot of our, like a lot of our evaluation sets, they're very, very advanced. So they might be measuring a model's ability to perform algebraic topology research, or they might be measuring a model's ability to like use PyTorch. And so first of all, you just need like really, really intelligent people with a lot of expertise to the domain that they're evaluating. So that's one piece. Another would be kind of this notion of sophistication and taste.

• 11:37Speaker 01

  It's like, okay, at the end of the day, you want models to be, like when you're writing, when you're asking to write code, you don't just care about correctness. You care about is a code well-written? Is it well-designed? Or like in the area of like, you know, essays or creative writing? Yeah, was this a really, really well-written essay that introduced new ideas, had great pros, and just doesn't feel like a A.I. lab? So there's this notion of sophistication. Another one would be this notion of kind of creativity. I think people often underestimate how important prompts are. Like people think of evaluating, like evaluating models. It's like, okay, yeah, you're evaluating the response.

• 12:19Speaker 01

  But in order to measure the models properly, you need to have kind of prompts that span the entire distribution of what you want your models to be good at. And so like within creative writing, for example, if you just ask the model a thousand, like if your prompt set is literally just a thousand, thousand stories or a thousand essays, and they're all phrased in the same way, as opposed to, you know, like the, the, the long, the long tail of how people actually interact with models in the real world, that's just not going to cut it. And I think people often underestimate, often underestimate how difficult it is to

• 12:54Speaker 01

  Create good prompts and to be creative. It's almost like that phenomenon where, what is it? If you just ask me to like name 50 foods. It's like, it's actually really, really hard for me to do it until that must have been fairly hard or I forced myself to maybe constrain, constrain the list of foods in various ways. In the same way, it's like very, very hard surprisingly to be creative in this fashion. So anyways, like I'll take creativity as another piece.

• 13:15Speaker 00

  Yeah.

• 13:16Speaker 01

  If I think the fourth piece is just like the ability to follow instructions. So like when we are asking you to, or when French rulers are asking you to evaluate the models, they often have very specific criteria that they have in mind. Like, okay, they have a certain, I don't know, style guide or they have a certain personality they want the model to follow. Or to care about XYZ grid here more than, you know, ABC grid here. And so oftentimes these are very, very complex instructions. And so people just need to be

• 13:40Speaker 01

  very, very good at following them.

• 13:41Speaker 00

  You know, I guess moving to one of the main themes, I feel like of, of model improvement these days. It feels like RL environments, reward models around them are all the rage these days. Can you talk a bit about that kind of transition? And, you know, how you've thought about supporting your customers with it?

• 13:57Speaker 01

  Ah, yeah, definitely. So the way I think about our environments is that they're kind of the continuation. I mean, there's almost just like the next step in, in training paradigms. So in the same way that historically a lot of work has gone into like SFT and then R adjust and then verifiers. RL environments are kind of just like the next step in that progression. And, you know, I, I, I do believe there will be, there will be other steps in the future. And I think our environments are also really, really interesting because like for example, we've actually been working on our environments for quite a while now, like maybe one or, one or two years. And so it's actually interesting that the Rust industry has only just started picking up with them.

• 14:40Speaker 01

  Like, like a lot of the teams that we work with, especially like there's a really, really amazing agents team at Meta, we've actually been working with them on our environments for over a year. So like it's a team that created the Gaia benchmark. This is basically their agents team that also just basically just open sourced their agent research environment, agent R environment platform. And so they, I think, really, really saw the, the, the wave of, of the future. So like our environments are kind of like interesting, different. If I think about what we've had to build in order to support our environments. Like what you need when you're creating environments is, our environments is the following. So like one, you need to have these really, really rich worlds that are basically simulations of, you know, the real world as best as you can.

• 15:23Speaker 01

  So just in the same way that, as I mentioned earlier, that creating prompts can be surprisingly hard because you want them to be creative and diverse and rich in real world as opposed to, you know, these very synthetic prompts that you find in benchmarks. Or you, or like just maybe what will naturally happen is if you don't have any constraints on diversity and creativity, you just let people create prompts really nearly, you just don't get much diversity. So in the same way, when you're creating these worlds, you also really need to have a bunch of really, kind of like underlying entities supporting the world. Because when we create these worlds, we have, we basically populate them with people and

• 16:06Speaker 01

  businesses and tools and interactions between these people. And, you know, messages, Slack, Slack messages, emails, calendar events, et cetera, et cetera. And we want all of these to basically mimic the real world in very interesting and complex ways. And so like a big part of our efforts are, okay, so how do we build the tooling? How do we build the infrastructure? How do we build the quality control measurements? How do we build the data measurements to make sure that this is happening? And then, once we create these worlds, we need to basically create all of the tools that exist in these worlds. So these could be the MCP servers that models are accessing. These could be the browsers or the code that the models are executing. So we just need to make sure that we have the, like, the underlying infrastructure

• 16:49Speaker 01

  for the models to basically run within this world and to be able to execute the prompts. And then we need to, you know, create the prompts themselves. We need to actually test how models are performing on these. We want to make sure that we're basically coming with tasks that test the limits of all of these frontier models. There's also like a very, very big measurement and almost like an introspection aspect where once we discover that a model has failed, we want to dig in and understand why. So yeah, there's, I think, there's a lot of really, really interesting infrastructure and tooling that we had to go to support this.

• 17:22Speaker 00

  I mean, you've been doing this work for a while now. I guess what have you learned along the way or what's surprised you or maybe something that you initially got wrong in the way you set these things up and have gotten better at?

• 17:31Speaker 01

  One interesting maybe viewpoint that we've tried to kind of propose is that it's actually very important to

• 17:43Speaker 01

  Pay attention to model trajectories to understand why they are succeeding and why they are failing. People often underestimate the amount to which models can reward hack themselves to the correct answer.

• 17:58Speaker 00

  I feel like there's a lot of very funny examples of that.

• 18:00Speaker 01

  Yeah, like the, I think we've, like, one of my favorite things, like, looking at, looking at some of these examples, because they often just perform in all these crazy ways. And then similarly, I think people have actually underestimated the kind of myriad of different ways that models can fail, and what that says about the model's underlying capabilities. Like, I think people have this, like, maybe notion in their head that, okay, you know, I'm just going to give a final reward, and also, almost like, train an F R award, everything's going to work out. I think what happens in practice is that, again, models can just deviate in these very, very odd ways, or they can show different types of intelligence, depending on what the underlying capabilities are. And if you

• 18:43Speaker 01

  don't shore up those underlying capabilities enough, models may seem to perform well in the short term, but, like, you're basically going to face a lot of problems down the road.

• 18:53Speaker 00

  This feels like a theme in a lot of the, the work you do with the labs, right? Like, there's, there's easy things to hill climb on, but if you're not, like, doing it in a deeply thoughtful way with the right evaluations, it doesn't actually serve the end purpose you're going for.

• 19:04Speaker 01

  All these things tie together, because it's often a thing that we've seen with benchmarks, where, like, people will kind of end up hacking these benchmarks, and so they think that they're making progress, because, you know, these, these numbers that everybody's paying attention to are going up, when just the underlying model isn't becoming materially more intelligent. So, yeah, like, I think all these things tie together.

• 19:28Speaker 00

  And then, obviously, you know, I think alongside R L environments becoming a more consensus way to improve models, there's been, like, a flurry of, of net new startup activities, right? I feel like there's probably 30 Y C companies trying to do, build R L environments. There's a lot of these, like, R L as a service companies. You know, I wonder what you, what you make of that activity, one, and I guess as a follow-up to that, it feels like every time there's a transition in maybe the main type of ways labs are improving their models, some people see that as an opportunity to, to step in, but obviously, there's persistence from people that were there in the previous wave. And so, wondering, you know, how you kind of reflect on that?

• 20:00Speaker 01

  I think one of the things that always drives me crazy about Silicon Valley is that there's this pivot culture that people have in their minds, where people are

• 20:11Speaker 01

  Just constantly pivoting to whatever seems the latest hot topic, or whatever seems to drive the highest valuations, instead of building things that they really materially believe in. And it's almost like this, like this funny thing where I think a lot of people in Silicon Valley, they'll maybe, you know, don't talk shit about Wall Street because, oh, Wall Street, what are you doing? You're just chasing money at the end of the day. But what is Silicon Valley doing? Yeah, you're just chasing the same thing. You're chasing valuations, you're chasing VCs. You're pivoting not because you have some amazing, great idea,

• 20:55Speaker 01

  but just because that's what YC told you to do in order to achieve market fit and get that, you know, get some revenue that you can show VCs when you're fundraising. So I think I really, really dislike the culture.

• 21:08Speaker 00

  Well, and what I'm struck by is obviously it seems, you know, it's not clear to anyone how long, you know, RL environments are the flavor of the day to improve models. And maybe that persists for a while, maybe it doesn't. But, you know, I think what you've clearly shown the ability to do over time is like, whatever the way that models are being improved, like you have an offering that helps support that. And obviously, I'm sure a lot of that comes from being just deeply embedded in your customers. I could imagine some folks saying, look, building an RL environment is so different than the first few acts of surge. Like, you know, to what extent do they have a right to go do that versus the 30 new entrants? I'm curious like how on the ground that's felt.

• 21:47Speaker 01

  Like, if you think about what our environments involve, they involve the following, following three things. So one is, again, like our environments are just like the next iteration or the current iteration of the data that is needed to train, to enable AGI. And so that, I mean, that just fits, obviously, with our fundamental thesis. Like, I don't know, we just want to create whatever data is needed to enable that. So that's a big piece of it. Second, our environments require a lot of tooling, like I mentioned earlier. So it requires the tooling to create all these tools, to run the models, to measure them, to analyze them, and so on and so on. And

• 22:32Speaker 01

  I mean, that's not any different from the fact that, yeah, for RHF, you needed tooling as well. So sure, like, it's a different kind of tooling, but all of those pieces are the same. Like, even when we're, even when we've done RHF, yeah, you need a lot of tooling to ensure that you're able to analyze the models as they progress throughout a conversation. You need to understand the wins and the fails. You need to make sure that, or searchers, like, the people who are creating these prompts and evaluating responses, you need to make sure that they're able to do all of these things in a really high-quality, diverse way. Yeah, so, like, you obviously need a lot of tooling to disport all this. Again, this is, like, very different from, I think, a lot of the other companies in our space,

• 23:12Speaker 01

  where they are essentially just, like, staffing agencies, and so they historically haven't built any technology. But, like, we've always been a technology company first and foremost. And, yeah, so it's just another type of tooling in the same way that you're sure any technology companies build tooling. And the third piece of it is that I think people, maybe some of these other startups, they haven't quite realized that creating R environments is all about getting really, really rich, complex, creative data, and there's just no other way to do it beyond using humans. Like, if you think about a lot of the, like, even think like Sweetbench.

• 23:55Speaker 01

  What is Sweetbench? Well, it's a collection of PRs that were created by humans. And then, like, Sweetbench itself wasn't quite clean enough, so people needed to build Sweetbench verified. And that's, you know, people basically taking Sweetbench's problems and evaluating them and cleaning up in various ways. And so, just in the same way, I think we fundamentally, that creating R environments is a human data problem that just requires a lot of technology.

• 24:20Speaker 00

  I think some people probably just assume quality is synonymous with, like, credentials, right? And they're like, "Well, if I have a PhD in something labeling, you know, or spending time on evaluating something, like, of course, that's high quality." But, like, maybe what, help me understand, like, an example of something where you actually have someone that seems credentialed on paper to improve models, but it's not happening in practice.

• 24:38Speaker 01

  An example I always love to give is that, take, you take Hemingway. Hemingway didn't have every PhD. I don't even know if he completed college. And I mean, what we're looking for is like, we want the greatest people in the world at every scale, regardless of their credential. So even just think about like, who works at Google, right? Google doesn't just hire people based off of what school they have in their resume or degree they have in their resume. And that's not how you progress at a company like Google or any other company. The way you progress is based off the actual work that you do. And one of the interesting things about us as a platform is like, yeah, we have a technology

• 25:22Speaker 01

  platform that looks at all the data they are creating and then measure it, measures it. And so we gather, you know, basically millions of signals on or workers every day. We see the types of data they are producing. And so it's almost like the most meritocratic thing possible that you can imagine. So as opposed to, okay, you progress because you happen to have a degree from Harvard. Like, no, like, we are going to actually measure what you do and advance you based off of that. And sure, like, we have a ton of Harvard students on our platform. We have a ton of PhDs on our platform. I think we're probably the biggest, like, biggest, like, source of PhDs in the world.

• 26:05Speaker 01

  But that just isn't sufficient, and it's not sufficient for two reasons. So one is, like, even if you think about, like, take coders. If you're like an MIT grad who has a computer science degree and you're actually really, really good, well, you're probably not, you're probably not actually going to try to create really good data to train these models. Instead, you're probably just going to try to cheat the system, right? Like, okay, you're a really good coder, you're like fascinated by like right teaming systems, you're fascinated by like adversary or attacks. Like, what you're going to try to do is you're going to try to find a way to cheat. And then another part of it is, like, even if I think about all of the

• 26:45Speaker 01

  People who are in my class at MIT, or if I think about the number of people that I've interviewed from MIT first for it itself, like honestly, half of them can't even code, right? Like, there's a very, very big difference between reading about something in a textbook and then having the street smarts to execute it. Like, again, this almost like ties back to the other things I was saying about performance on benchmarks versus performance in a real world. Like, a big problem that the frontier labs have had is that the models are almost like too textbook intelligent instead of, instead of having like the, the street smarts to, to do things in your world.

• 27:19Speaker 00

  You know, you, you alluded to earlier, I think you mentioned one of the, you know, labs is having you, you know, be at their, at their internal big conference today. And obviously, it speaks to just how closely you work with those teams. You know, obviously, you had, I guess, a few months ago, you know, met and decided to buy one of the vendors they, they work with really closely and, and scale. And, and given the proximity of those relationships, I know part of that was also, you know, bringing some talent into the organization. But I'm curious, like, your reaction to that? And, and do you think over time, like, some of those things are, are natural to happen, given just the proximity of relationships?

• 27:51Speaker 01

  I mean, I think the, the scale acquisition was actually amazing for us because up until then, we'd been pretty under the radar. And so I think all of the top researchers of all the labs already knew about us. So they knew that we had the highest quality by far. And they knew that we were the biggest and the, and the fastest. But the AI field has just been progressing so large that, you know, more and more people are entering the field every single day. And so that basically the scale acquisition just put a spotlight on us that was really, really helpful for, for expanding. Like, I think we, we just, we got so much like new demand from all these new teams overnight. And so that was really, really beneficial, beneficial for us.

• 28:36Speaker 00

  You kind of mentioned this, this difference in, you know, maybe the culture of, of folks that are attracted to these types of companies and the things that, that people are optimizing for. I'm wondering how that manifests itself and maybe the different decisions you've made for search today, and also maybe the different trajectories that these businesses go in over time.

• 28:53Speaker 01

  Good question. So I think it changed and, and, or shaped us in a bunch of ways. And maybe one of the most prevalent ones is in hiring. If I think about the culture that we're trying to still and the type of people that we're trying to hire. It's people who are kind of like researchers at heart. And people who just fundamentally care about data and AI. And when I think about us as a company and what we are trying to build, I think of ourselves as a lot more like a research lab than like, you know, just another Silicon Valley startup that's trying to chase money and hype and evaluations. And one of the concrete ways that manifests is,

• 29:36Speaker 01

  again, if you think about this, this idea of hiring people who are fundamentally interested by research and data and enabling AGI as opposed to this type of Silicon Valley person who's

• 29:50Speaker 01

  kind of embodied by, uh, embodied by growth hackers and embodied by people who are just doing whatever it takes to increase your revenue. Like if you think about the incentives of those types of people, what they will often do is they'll basically try to sell you things that you may not need, that they don't think will actually improve your models. I mean, essentially, they'll just try, they'll just try to act like salesmen. As opposed to digging deep into what you need, they can dig deep into the problems of your models, trying to make sure that you understand all of the, uh, all the different ways that you should be measuring your models to make sure that they're actually improving. As opposed to kind of just like selling you like what's essentially snake oil.

• 30:33Speaker 01

  So I think that, uh, that belief in caring more about

• 30:41Speaker 01

  model progress over revenue, I think that that would probably just shake the company.

• 30:45Speaker 00

  I guess, speaking of model progress, like, how, how do you articulate? Like, how, what's the path to models getting better from here? And like, does it feel like there's consensus among, you know, most of the top labs around this? Or do you actually see some pretty divergent approaches?

• 30:59Speaker 01

  Yeah, I think there's been a lot more divergence than we expect. For all the training paradigms out there, it almost feels like every frontier lab has their own take on it. Sometimes those takes are wildly different, sometimes they're just like maybe slight variations on each other. But there's a lot more divergence than I would have expected.

• 31:14Speaker 00

  What are like some of the key vectors where that divergence exists?

• 31:18Speaker 01

  I think at a high level, there are two ways in which the companies diverge. One is, once they choose their objective function, is they know what type of training algorithm, what type of training data they're already going to gather. I can't speak too much about that, but I think an underestimated difference between all the frontier labs is their choice in what they optimize for and what they pay attention to. And so I can give a couple of examples. So one example is, so I've mentioned LLM, LLM Arena earlier. And I think one of the fascinating things to me is that some frontier labs, they've just chosen not to pay attention to it at all. And I think those frontier labs have done better.

• 32:02Speaker 01

  Because the frontier labs who have had to pay attention, what researchers at those labs have often told me, it's like, okay, like the researchers tell me, I hate LLM Arena. They like, these researchers understand all the ways in which optimizing for LLM Arena will lead to this negative progress. It will lead to models that hallucinate because LLM Arena users don't care about hallucinations. If anything, they love them because when your model hallucinates, it just makes it sound wild and crazy and kind of like a very fun and enticing way. Again, just like a tabloid. And so the fact that frontier labs, like some frontier labs have had the, something to fortitude and the, just like this underlying thesis, underlying belief

• 32:47Speaker 01

  in what they see as the path to AGI, as opposed to Feeling like they need to chase publicity and chase

• 33:00Speaker 01

  chase like this, like very hyped up and publicly visible leaderboard. Like, I think just the fact that certain frontier labs have felt a freedom and authority to not pay attention to that because they had sorts of core belief and what they're going for instead. I think that that's actually really, really shaped a lot of model progress. And then I think another interesting divergence is, again, in the choice of objective that the frontier labs are trying to optimize for. And I think you can see this clearest difference between OpenAI and Anthropic. Like, you think about OpenAI, what are they optimizing for now?

• 33:35Speaker 01

  It's almost like they are leaning more towards optimizing for user engagement. So really long sessions or amount of daily users as opposed to a company like Anthropic, who might be optimizing for something more akin to productivity and like how much value you can extract, like how much, almost in GDP, or how much productivity or like time savings you can extract by interacting with the model. And I think that shapes the type of products that they build, it shapes the type of types of people that they attract, it shapes the capabilities of their models. So I think it just really shapes them in many ways. Yeah, we're starting to see that.

• 34:17Speaker 00

  It's an interesting point. So I think it feeds into this larger question of, you know, obviously, I think a lot of model progress to date has all kind of, you know, come back to one core, large model that can do lots of different things. And, you know, to your point now, it feels like you could imagine models optimizing around consumer engagement and then a productivity model. I mean, I think you've you've seen this with voice, right? There's like the enterprise voice people want, and then maybe more of like an engaging consumer voice. Over time, do you think that, like, there'll be one model that just is, you know, is able to context switch across like whatever the thing being optimized for, or do we actually end up in a world where it's like, no, you probably want like this enterprise model or this even per industry models for finance, legal, or other things?

• 34:59Speaker 01

  Yeah. So I think this is one place where my thinking has diverged a lot. So I used to think that there would essentially be one model to rule them all. Because, yeah, okay, sure, you have some super

• 35:08Speaker 00

  Like the ultimate ASI vision.

• 35:10Speaker 01

  Yeah, like you have some super intelligent model and should be able to context switch and adopt whatever, whatever you want to do. But actually, I think over the past year, I've started to realize that

• 35:22Speaker 01

  it's almost like every company should have a thesis on, like, the world is just so rich. There's never going to be a one-size-fits-all solution. Instead, every company or every lab or every, like, AI, it needs to have a thesis underlying it that, uh, of like, what will be useful in the real world? And, uh, what kinds of AI will best serve people? And that thesis will shape how the model behaves. Like, sure, like, two models can be, it's almost like two models can be just as intelligent as each other, but they'll have different personalities, they'll have different biases for how they answer particular questions. They'll have different ways in which they converse with you, uh, and so on and so on.

• 36:07Speaker 01

  And so just in the same way that, um, like, if you think about, I don't know, Google and Facebook, if Google were to build a social media platform, it's going to be very, very different from the way Facebook built social media platforms. If Facebook were to build a search engine, it would have a very different take from how Google would build a search engine. And so there's no, like, right or wrong answer, um, per se, it's just that different companies, different people, um, they have different fundamental beliefs and in like the things that are like useful and good for the world. And so I think the same thing will happen with AI.

• 36:39Speaker 00

  So what does that mean for, for your take on how many people should be building models? Or like, do you think we'll see more model players over time or conversions from the folks we do have today?

• 36:48Speaker 01

  I definitely think that there should be more people building AI because, um, to that point, like, I think that many, many, many different types of theses are needed on what kinds of AI will be useful for the world. And I just don't think anybody's figured that out yet.

• 37:04Speaker 00

  Should companies be training their own models? Well, as in like, you know, uh, I'm a really large finance firm or a really large, you know, uh, healthcare organization.

• 37:12Speaker 01

  So I definitely think that eventually every, every company should be training their own models. And that's because these models will be so important to the world and you want to, you'll want to eventually deploy them to like ninety-nine point nine nine nine nine percent of use cases, right? And if you simply rely on models from the frontier labs.

• 37:34Speaker 01

  What you're optimizing for may not be what you're optimizing for. And just again, just because AI will be so important and you want to deploy it everywhere. I think if you want to get the best value, best performance possible, you should be training them.

• 37:46Speaker 00

  Can you achieve that optimization through like just some good prompting or some light fine tuning? Or do you think it actually requires like building somewhat from scratch?

• 37:54Speaker 01

  So I think this again goes back to what I said earlier about having a fundamental thesis on how AI should serve your customers and what types of AI you want to build. So if you have a strong thesis, it's almost like having like a product thesis. But if you have a strong product as opposed to just building some commodity product, if you believe that you have some unique take on how AI should behave, then yes, I think it makes absolute sense.

• 38:18Speaker 00

  Obviously, we'll be, we'll, we'll be curious to see how the costs of doing that, you know, change over time because certainly the, the trade off of, of making that investment or having that unique take, you know, today, it's, it's kind of relatively prohibitively expensive to get to the state of the art. But I imagine a lot of people could build these opinionated models with a thesis and still be six, 12 months behind state of the art and, and be okay.

• 38:38Speaker 01

  Yeah, exactly. Like I, I don't think companies are quite ready right now, given the state of, I mean, both the companies themselves and the state they're on. But as AI gets better and better, I think it will be increasingly increasing for it.

• 38:49Speaker 00

  And obviously it's very clear models are getting way better coding and these easily verifiable domains. You know, I think there was a time where, you know, you, you'd use chat GPT and a new model would come out and it'd be blindingly obvious that the models have gotten better. You know, I don't know if I would necessarily say that's been the case over the last three, six months. You're, you're obviously on the inside of this stuff. Do you feel like models are still getting better outside of coding right now?

• 39:10Speaker 01

  Yeah, I definitely think they do. And in part, that's through all the evaluations that I run where we see this constant progress. But it's also, it's also true that I think just the other day, I started using Claude a lot more for, for writing in particular. And I was just shocked by how. How much better was today compared to a couple of months ago?

• 39:33Speaker 00

  I do want to make sure I hit on some of the multimodal models that are being built, whether it's video, robotics, stuff being done in bio. I'm sure you've thought about some of this stuff. To what extent is that interesting to you? Is it kind of like a similar set of problems? Or how would you characterize what's similar and different there?

• 39:50Speaker 01

  Yeah, so I think all these modalities are fascinating to me. I think one of the things that people don't realize is that we actually work very, very heavily across all of these spaces already. Like, I think maybe 50% or more of our work today, it's actually in domains outside of pure text. So I actually do think it's fascinating. And again, if you think about our thesis and what we're trying to enable, it's like this idea of we just want to enable AGI no matter what it takes. And yeah, if we want AI that is useful out there in a real world and needs to understand all these capabilities, needs to operate across all of these domains. And so we just want to do whatever it takes to make that happen.

• 40:30Speaker 00

  What's like quality mean in the video context? I could certainly imagine, I get like these text use cases, but for video, what's that mean to you guys?

• 40:39Speaker 01

  Yeah, I think people often underestimate how important quality is across even across these modalities that people think are surprisingly important. So one example is, again, even in the types of prompts that you're creating, you need a lot of creativity and kind of like technology to make sure that you're exploring the full distribution of the space. I think people often underestimate that because they just think that people can kind of create prompts out of thin air that target the full distribution of a model's capabilities when it's actually spiraling to the goal. One of our big goals is that we often try to teach people or try to try to make sure that our customers understand.

• 41:24Speaker 01

  It's that when you think about quality, you need to go beyond robotic instruction following and robotic correctness to think about all of these other implications of the prompt itself.

• 41:35Speaker 01

  So yeah, I think people often underestimate what quality means.

• 41:38Speaker 00

  Yeah. And what makes like either a video evaluator or like video itself like higher or lower quality? If you know, as you think about maybe the problems that text models have run into with Elmerina, I imagine there's similar traps on the video side. Like, what have you learned around that?

• 41:52Speaker 01

  Uh, yeah. So again, I think it boils down to this notion of taste and sophistication that I've mentioned before. It's like, okay, sure. Like you ask, uh, Scorsese to film a video, create a video about, uh, I don't know, a fish. Then you ask your high school arts graduate to do the same thing, like, you know, just pull a grand boss on our street. Sure, both people can create a film about a fish, but Scorsese is probably going to have a much better film about the fish. And, um, like, that's where that notion of taste and sophistication and creativity and just like going above and beyond.

• 42:29Speaker 01

  Because when you think about what you want from like, uh, from models, it's not just the ability to, to literally follow your instructions and do whatever you say. It's to, um, it's to kind of craft something that will blow your mind, something that feels imaginative and creative. And, uh, kind of raises, raises the bar. That, that, that's what we're trying to do.

• 42:53Speaker 00

  And then like robotics and bio and these spaces that require like a hardware component for data collection. Do you think those are natural extensions for companies like you guys, or maybe you're already working in them? Or is that like a totally separate, you know, set of companies that might pop up and do that?

• 43:06Speaker 01

  The way I think about it is we want to do whatever it takes to enable the data that, uh, that is going to help accelerate AGI. And sometimes involves, uh, sometimes involves building new tools, sometimes involves buying new hardware equipment. Sometimes involves, you know, uh, expansion to whatever space. We are, yeah, like we're a technology company and we're going to do whatever it takes to, to make that happen. As opposed to being some, you know, like narrowly constrained company, um, because we just pivoted into the area. And, uh, we're not really thinking longer term about, like, like we are thinking, like we, uh, in contrast to some of the other companies, we are thinking longer term about everything that's needed to achieve all these things.

• 43:45Speaker 00

  Amazing. Well, we always like to enter interviews with just a quick fire round where we get your take on, on, on the standard set of topics. And so maybe to, we've hit on this actually a few times, but I'm curious, one thing that you've changed your mind on in AI in the last year?

• 43:56Speaker 01

  Probably the biggest thing is this idea that I used to think that it would be one model to rule them all. And now I actually do see how these different product opinions, AI opinions will shape AI going forward.

• 44:11Speaker 00

  Obviously, it seems like search has just been a series of incredible wins and you've built like an amazing company. But I'm curious and reflecting back, what's the biggest mistake you've made in building the business?

• 44:20Speaker 01

  So my background has always been in research and data science. And so I used to love publishing and I used to love blogging. I used to love sharing all of our insights. And we did that early on and then I somehow just got too busy to do it.

• 44:34Speaker 00

  And I can imagine how.

• 44:36Speaker 01

  And I really miss that, like this idea of like teaching the world and like kind of sharing, sharing our viewpoints on the industry. And one needs to change or one needs to happen in order to make sure that we're on a good path. And so I think the biggest mistake is that we kind of stopped, stopped publishing as much in the past like two or three years. And so I'm hoping to fix that now.

• 44:54Speaker 00

  And yeah, if you were, if you were to like, you know, get a week of vacation where you could just sit back and write like a really long piece. Like what, what would you write about or what's kind of like most top of mind for you?

• 45:04Speaker 01

  So the most top of thing, top of mind thing to me really is this concept of objective functions. And what every frontier lab is optimizing for. Like, I think it's surprisingly subtle and has surprisingly far reaching consequences. Like, you know, are you optimizing for engagement? Are you optimizing for usefulness? Are you optimizing for a number of users or optimizing for GDB? Like, like whatever, I think that concept has very, very far reaching consequences for the industry and for AI at large.

• 45:36Speaker 00

  What would you optimize for if you were running one of the labs?

• 45:40Speaker 01

  So I would optimize for this, I think quite crystallized it yet. But it's this notion of a month later, would you be happy? That you had this interaction with this model? Would it have almost like changed your life in some way?

• 46:03Speaker 01

  Like the more moments they have been get like that and could have changed your life because it maybe you were asking about a vacation and introduced introduced you to serendipitous new new location that you've never thought about before. Or maybe you had a medical question with it and it and you didn't quite maybe you didn't quite know how to phrase a medical question, but then the AI is serendipitously like notice something and you know, like taught you something that you would have figured out otherwise. Like that, I think is one of the one of the things I'm thinking about right now.

• 46:34Speaker 00

  Yeah, I'm struck by like how much of this of like these questions we're asking AI just bring up like challenges that we've always had as society, right? I mean, you were alluding to the SAT challenge earlier and like that's certainly, you know, it's an imperfect way of measuring intelligence and but we've never really found much better ways. And similarly, even here, it's, you know, there's been lots of talk about what technology should be optimized on and what we should be trying to improve in people's lives. And, you know, it's a hard, a hard question to answer, but obviously an ever important one as these as these models do get better and are going to hill climb on whatever it is we are we are optimizing for.

• 47:06Speaker 01

  Yeah, exactly. I think like a lot of the way I think about AI is and maybe like the worries and consequences of AI is analogous to parallels with social media.

• 47:16Speaker 00

  Totally. Well, this has been a fascinating conversation. I want to make sure I leave the last word to you. Where can our listeners go to learn more about you, about Serge and anything else? The mic is yours, wherever you want to point, folks.

• 47:27Speaker 01

  Yeah, so I would definitely suggest or blog. So we're starting to blog a lot more, starting to share a lot more insights, analyses. So definitely check that out.

• 47:34Speaker 00

  Amazing. Well, thanks so much. This was a ton of fun. Thanks so much.