The 100-person AI lab that became Anthropic and Google's secret weapon | Edwin Chen (Surge AI)

Edwin Chen is the founder and CEO of Surge AI, the company that teaches AI what’s good vs. what’s bad, powering frontier labs with elite data, environments, and evaluations. Surge surpassed $1 billion in revenue with under 100 employees last year, completely bootstrapped—the fastest company in history to reach this milestone. Before founding Surge, Edwin was a research scientist at Google, Facebook, and Twitter and studied mathematics, computer science, and linguistics at MIT. We discuss: 1. How Surge reached over $1 billion in revenue with fewer than 100 people by obsessing over quality 2. The story behind how Claude Code got so good at coding and writing 3. The problems with AI benchmarks and why they’re pushing AI in the wrong direction 4. How RL environments are the next frontier in AI training 5. Why Edwin believes we’re still a decade away from AGI 6. Why taste and human judgment shape which AI models become industry leaders 7. His contrarian approach to company building that rejects Silicon Valley’s “pivot and blitzscale” playbook 8. How AI models will become increasingly differentiated based on the values of the companies building them — Brought to you by: Vanta https://vanta.com/lenny—Automate compliance. Simplify security. WorkOS https://workos.com/lenny—Modern identity platform for B2B SaaS, free up to 1 million MAUs Coda https://coda.io/lenny—The all-in-one collaborative workspace — Transcript: https://www.lennysnewsletter.com/p/surge-ai-edwin-chen — My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/180055059/my-biggest-takeaways-from-this-conversation — Where to find Edwin Chen: • X: https://x.com/echen • LinkedIn: https://www.linkedin.com/in/edwinzchen • Surge’s blog: https://surgehq.ai/blog — Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ — In this episode, we cover: (00:00) Introduction to Edwin Chen (04:48) AI’s role in business efficiency (07:08) Building a contrarian company (08:55) An explanation of what Surge AI does (09:36) The importance of high-quality data (13:31) How Claude Code has stayed ahead (17:37) Edwin’s skepticism toward benchmarks (21:54) AGI timelines and industry trends (28:33) The Silicon Valley machine (33:07) Reinforcement learning and future AI training (39:37) Understanding model trajectories (41:11) How models have advanced and will continue to advance (42:55) Adapting to industry needs (44:39) Surge’s research approach (48:07) Predictions for the next few years in AI (50:43) What’s underhyped and overhyped in AI (52:55) The story of founding Surge AI (01:02:18) Lightning round and final thoughts — Referenced: • Surge: https://surgehq.ai • Surge’s product page: https://surgehq.ai/products • Claude Code: https://www.claude.com/product/claude-code • Gemini 3: https://aistudio.google.com/models/gemini-3 • Sora: https://openai.com/sora • Terrence Rohan on LinkedIn: https://www.linkedin.com/in/terrencerohan • Richard Sutton—Father of RL thinks LLMs are a dead end: https://www.dwarkesh.com/p/richard-sutton • The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html • Reinforcement learning: https://en.wikipedia.org/wiki/Reinforcement_learning • Grok: https://grok.com • Warren Buffett on X: https://x.com/WarrenBuffett • OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter): https://www.lennysnewsletter.com/p/kevin-weil-open-ai • Anthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram): https://www.lennysnewsletter.com/p/anthropics-cpo-heres-what-comes-next • Brian Armstrong on LinkedIn: https://www.linkedin.com/in/barmstrong • Interstellar on Prime Video: https://www.amazon.com/Interstellar-Matthew-McConaughey/dp/B00TU9UFTS • Arrival on Prime Video: https://www.amazon.com/Arrival-Amy-Adams/dp/B01M2C4NP8 • Travelers on Netflix: https://www.netflix.com/title/80105699 • Waymo: https://waymo.com • Soda versus pop: https://flowingdata.com/2012/07/09/soda-versus-pop-on-twitter — Recommended books: • Stories of Your Life and Others: https://www.amazon.com/Stories-Your-Life-Others-Chiang/dp/1101972122 • The Myth of Sisyphus: https://www.amazon.com/Myth-Sisyphus-Vintage-International/dp/0525564454 • Le Ton Beau de Marot: In Praise of the Music of Language: https://www.amazon.com/dp/0465086454 • Gödel, Escher, Bach: An Eternal Golden Braid: https://www.amazon.com/G%C3%B6del-Escher-Bach-Eternal-Golden/dp/0465026567 — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected] mailto:[email protected]. — Lenny may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com https://www.lennysnewsletter.com?utm_medium=podcast&utm_campaign=show-notes-no-free-preview-language

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lenny Rachitsky?

  You guys hit a billion in revenue in less than four years with around sixty to seventy people. You're completely bootstrapped, haven't raised any VC money. I don't believe anyone has ever done this before.

• 0:10Speaker 01

  We basically never wanted to play this Silicon Valley game. That always starts ridiculous. I used to work at a bunch of the big tech companies and I always felt that we could fire ninety percent of people and we would move faster because the best people would have all these distractions. So when we started Surge, we wanted to build it completely differently with a super small, super elite team.

• 0:26Lenny Rachitsky?

  You guys are by far the most successful data company out there.

• 0:29Speaker 01

  We essentially teach AI models what's good and what's bad. People don't understand what quality even means in this space. They think you could just throw bodies at a problem and get good data. That's completely wrong.

• 0:39Lenny Rachitsky?

  To a regular person, it doesn't feel like these models are getting that much smarter constantly.

• 0:43Speaker 01

  Over the past year, I've realized that the values that the companies have will shape the model. I was asking Claude to help me drop an email the other day. And after thirty minutes, yeah, I think it really crafted me the perfect email and I sent it. But then I realized that I spent thirty minutes doing something that didn't matter at all. If you could choose the perfect model behavior, which model would you want? Do you want a model that says, you're absolutely right, there are definitely twenty more ways to improve this email? And it continues for fifty more iterations? Or do you want a model that's optimizing for your time and productivity and just says, no? You need to stop. Your email is great. Just send it and move on.

• 1:14Lenny Rachitsky?

  You have this hot take that a lot of these labs are pushing AGI in the wrong direction.

• 1:18Speaker 01

  I'm worried that instead of building AI that will actually advance us as a species, curing cancer, solving poverty, understanding the universe, we are optimizing for AI stop instead. Could we be optimizing our models for the types of people who buy tablets at the grocery store? We're basically teaching our models to chase dopamine instead of truth.

• 1:35Lenny Rachitsky?

  Today, my guest is Edwin Chen, founder and CEO of Surge AI. Edwin is an extraordinary CEO and Surge is an extraordinary company. They're the leading AI data company powering training at every frontier AI lab. They are also the fastest company to ever hit one billion dollars in revenue in just four years after launch with fewer than one hundred people and also completely bootstrapped. They've never raised a dollar in VC money. They've also been profitable from day one. As you'll hear in this conversation, Edwin has a very different take on how to build. An important company and how to build AI that is truly good and useful to humanity. I absolutely loved this conversation and I learned a ton.

• 2:19Lenny Rachitsky?

  I'm really excited for you to hear it. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It helps tremendously. And if you become an annual subscriber of my newsletter, you get a ton of incredible products for free for an entire year, including Devon, Lovable, Replit, Bolt, N8N, Linear, Superhuman, Descript, Whisperflow, Gamma, Perplexity Warp, Granola, Magic Patterns, Rake, Chat, P R D, Mob and Post, Hog and Stripe, Atlas. Head on over to lennysnewsletter.com and click Product Pass. With that, I bring you Edwin Chen after a short word from our sponsors. My podcast guests and I love talking about craft and taste and agency and product market fit. You know what we don't love talking about? SOC 2. That's where Vanta comes in.

• 3:04Lenny Rachitsky?

  Vanta helps companies of all sizes get compliant fast and stay that way with industry-leading AI, automation and continuous monitoring. Whether you're a startup tackling your first SOC 2 or ISO 27001 or an enterprise managing vendor risk, Vanta's trust management platform makes it quicker, easier and more scalable. Vanta also helps you complete security questionnaires up to five times faster so that you could win bigger deals sooner. The result? According to a recent IDC study, Vanta customers slashed over $500,000 a year and are three times more productive. Establishing trust isn't optional. Vanta makes it automatic. Get $1,000 off at vanta.com/len.

• 3:48Lenny Rachitsky?

  Here's a puzzle for you. What do OpenAI, Cursor, Perplexity, Vercel, Plaid and hundreds of other winning companies have in common? The answer is they're all powered by today's sponsor, WorkOS. If you're building software for enterprises, you've probably felt the pain of integrating Single Sign-on, SCIM, RBAC, Audit Logs and other features required by big customers. WorkOS turns those deal blockers into drop-in APIs with a modern developer platform built specifically for B2B SaaS. Whether you're a seed-stage startup trying to land your first enterprise company. Customer or a unicorn expanding globally. WorkOS is the fastest path to becoming enterprise ready and unlocking growth. They're essentially Stripe for enterprise features.

• 4:30Lenny Rachitsky?

  Visit workos.com to get started or just hit up their Slack support where they have real engineers in there who answer your questions super fast. WorkOS allows you to build like the best with delightful APIs, comprehensive docs and a smooth developer experience. Go to workos.com to make your app enterprise ready today.

• 4:51Lenny Rachitsky?

  Edwin, thank you so much for being here and welcome to the podcast.

• 4:55Speaker 01

  Thanks so much for having me. I'm super excited.

• 4:58Lenny Rachitsky?

  I want to start with just how absurd what you've achieved is. A lot of people, a lot of companies talk about scaling massive businesses with very few people as a result of AI. And you guys have done this in a way that is unprecedented. You guys hit a billion in revenue in less than four years with less than 60, around 60 to 70 people. You're completely bootstrapped, haven't raised any VC money. I don't believe anyone has ever done this before. So you guys are actually achieving the dream of what people are describing will happen with AI. I'm curious, just do you think this will happen more and more as a result of AI? And also just where has AI most helped you find leverage to be able to do this?

• 5:40Speaker 01

  Yeah, so we hit over a billion of revenue last year with under 100 people. And I think we're going to see companies with even crazier ratios, like 100 billion per employee in the next few years. AI is just going to get better and better and make things more efficient. So debt ratio just becomes inevitable. Like I used to work at a bunch of the big tech companies and I always felt that we could fire 90% of people and we would move faster because the best people would have all these distractions. And so when we started Surge, we wanted to build it completely differently with a super small, super elite team. And yeah, what's crazy is that we actually succeeded.

• 6:15Speaker 01

  And so I think two things are colliding. One is that people are realizing that you don't have to build giant organizations in order to win. And two, yeah, all these efficiencies from AI. And they're just going to lead to a really amazing time in company building. Like, the thing I'm excited about is that the types of companies are going to change too. It won't just be that they're smaller. We're going to see fundamentally different companies emerging. Like, if you think about it, fewer employees means less capital. Less capital means you don't need a raise. So instead of companies started by founders who are great at pitching and great at hyping, you'll get founders who are really great at technology or product. And instead of products optimized for revenue and what VCs want to see, you'll get more interesting ones built by these tiny, obsessed teams.

• 6:58Speaker 01

  So people building things they actually care about, real, real technology, real innovation. So I'm actually really, really hoping that the Silicon Valley startup scene, it will actually go back to being updates for hackers again.

• 7:08Lenny Rachitsky?

  You guys have done a lot of things in a very contrarian way. And one was actually just not being like on LinkedIn, posting viral posts, not on Twitter, constantly promoting Surge. I think most people hadn't heard of Surge until just recently. And then you just came out and like, okay, the fastest growing company at a billion dollars. Why would you do that? I imagine that was very intentional.

• 7:27Speaker 01

  We basically never wanted to play this Silicon Valley game. And like, I always thought it was ridiculous. Like, what did you dream of doing when you were a kid? Was it building a company from scratch yourself and getting into weeds of your code and your product every day? Or was it explaining all your decisions to VCs and getting on this giant PR and fundraising hamster wheel? And it definitely made things more difficult for us. Because, yeah, when you fundraise, you just naturally get part of this kind of Silicon Valley industrial complex where people will, your VCs will tweet about you. You'll get the techcrunch outlines. You'll get announced in all the newspapers because you raised at this massive valuation. And so it made things more difficult for us because the only way we were going to succeed

• 8:11Speaker 01

  was by building a 10 times better product and getting word of mouth from researchers.

• 8:16Speaker 01

  But I think it also meant that our customers were people who really understood data and really cared about it. Like, I always thought it was really important for us to have customers, early customers, who are really aligned with what we were building and who really cared about having really high-quality data and really understood how that data would make their AI models so much better. Because they were the ones helping us, they were the ones giving us feedback on what we're producing. And so just having that kind of like very, very close mission alignment with our customers actually helped us early on. So these are people who basically just buy our product because they knew how different it was and because it was helping them rather than because they saw something in that current chat line. So it made things harder for us, but I think in a really good way.

• 8:55Lenny Rachitsky?

  It's such an empowering story to hear this journey for founders that they don't need to be on Twitter all day promoting what they're doing, they don't have to raise money, they can just kind of go heads down and build. So I love so much about the story of Surge. For people that don't know what Surge does, just give us a quick explanation of what Surge is.

• 9:16Speaker 01

  We essentially teach AI models what's good and what's bad. So we train them using human data and just a lot of different products that we have, like SFT, RHF, Rubrik Verifiers, R environments, and so on and so on. And then we also measure how well they're progressing. So essentially, we're a data company.

• 9:36Lenny Rachitsky?

  What you always talk about is the quality has been the big reason you guys have been so successful, the quality of the data. What does it take to create higher-quality data? What do you all do differently? What are people missing?

• 9:47Speaker 01

  I think most people don't understand what quality even means in this space. They think you could just throw bodies at a problem and get good data, and that's completely wrong. Let me give you an example. So imagine you wanted to train a model to write an A-line poem about the moon. What makes it a good, high-quality poem? If you don't think deeply about quality, you'll be like, is this a poem? Does it contain eight lines? Does it contain the word moon? You check all of these boxes and it's still sure, yeah, you say it's a great poem. But that's completely different from what we want. We are looking for a Nobel Prize winning poetry. Like, is this poetry unique? Is it full of subtle imagery? Does it surprise you and talk out of your heart? Does it teach you something about the nature of moonlight? Does it play through emotions and does it make you think?

• 10:30Speaker 01

  That's what we are thinking about when we think about high quality poem. So it might be like a haiku about moonlight on water. It might use internal rhyme and meter. There are a thousand ways to write a poem about the moon. And each one gives you all these different insights into language and imagery and human expression. And I think thinking about quality in this way is really hard. It's hard to measure. It's really subjective and complex and rich. And it sets a really high bar. And so we have to build all this technology in order to measure it. Like thousands of signals on all of our workers, thousands of signals on every project, every task. Like, we know at the end of the day, if you are good at writing poetry versus good at writing essays versus good at writing technical documentation. And so we have to gather all these signals on what your background is, what your expertise is,

• 11:14Speaker 01

  and not just that, like how you're actually performing when you're writing all these things. And we use those signals to inform whether or not you are good and forker for these projects and whether or not you are improving the models. And it's really hard. And so to build all this technology to measure it. But I think that's exactly what we want AI to do. And so we have these really, really deep notions of a quality that we're always trying to achieve.

• 11:37Lenny Rachitsky?

  So what I'm hearing is there's kind of a, just going much deeper in understanding what quality is within the verticals that you are selling data around. So you, and is this like a person you hire that is incredibly talented at poetry? Plus evals that they, I guess, help write that tell them that this is great? What's the mechanics of that?

• 11:57Speaker 01

  The way it works is we essentially gather thousands of signals about everything that you're doing when you're working on a platform. So we are looking at your keyboard strokes. We are looking how fast you answer things. We are using reviews. We are using code standards. We are using, like, we're training models ourselves on the outputs that you create. And then we're seeing whether they improve the model's performance. And so in a very similar way to how Google search, like when Google search is trying to determine what is a good webpage, there's almost two aspects of it. One is you want to remove all the worst of the worst webpages. So you want to remove all the spam, all the, just like low quality content, all the pages that don't load. And so it's almost like a content moderation problem.

• 12:42Speaker 01

  You just want to remove the worst of worst. But then you also want to discover the best of the best. Okay, like this is the best webpage or you know, this is the best person for this job. They are not just somebody who writes the equivalent of high school level poetry. Again, like they're not just robo-icky writing poetry that checks all these boxes, checks all these explicit instructions. But rather, yeah, they're writing poetry that makes you emotional. And so we have all these signals as well that again, like completely differently from moving to worse to worse. We are finding the best of the best. And so we have all these signals again, just like Google search uses all these signals and feeds them into their ML algorithms. And uses them predict certain types of things. We do the same with all of our, with all of our workers and all of our tasks and all of our projects. And so it's almost like a complicated machine learning problem at the end of the day.

• 13:27Speaker 01

  And that's how it works.

• 13:29Lenny Rachitsky?

  It is incredibly interesting. And I want to ask you about something I've been very curious about over the past couple of years. If you look at Claude, it's been so much better at coding and at writing than any other model for so long. And it's really surprising just how long it took other companies to catch up, considering just how much economic value there is there. Just like every AI coding product sat on top of Claude because it was so good Claude code and writing also. What is it that made it so much better? Is it just the quality of the data they trained on or is there, is there something else?

• 13:59Speaker 01

  I think there are multiple parts to it. So a big part of it certainly is the data. Like I think people don't realize that there, there's almost like this infinite amount of choices. That all the frontier labs are deciding between when they're choosing what data goes into their models. It's like, okay, are you purely using human data? Are you gathering the human data in XYZ way? When you are gathering the human data, what exactly are you asking that people were creating it? To create for you, like maybe you create, maybe you care more, for example, in the coding realm. Maybe you care more about front end coding versus back end coding. Maybe when you were doing front end coding, you care a lot about the visual design of the front end applications that you're creating. Or maybe you don't care about it so much and you care more about, I don't know, the efficiency of it.

• 14:44Speaker 01

  Or the pure correctness over that like visual design. And then other questions like, okay, are you carrying balls? Are you like, how much synthetic data are we throwing into the mix? How much do you care about these 20 different benchmarks? Like some companies, they see these benchmarks and they're like, okay, for PR purposes, even though we don't think that these academic benchmarks matter all that, all that much, then we just need to optimize for them anyways because we, our marketing team needs to show certain progress on certain standard evaluations that every other company talks about. And if we don't show good performance here, it's going bad for us, even if, even if like ignoring these academic benchmarks makes us better at the real tasks. Other companies are going to be principled and be like, okay, yeah, no, I don't care about marketing. I just care about how my model performs on these real world tasks at the end of the day.

• 15:28Speaker 01

  And so I'm, I'm going to optimize for that instead. And it's almost like there's a trade off between all of these different things. And there's like, uh, like one thing I often think about is that there's a, it's almost like there's an art to post training. It's not purely a science. Like when you were deciding what kind of model you're trying to create and what it's good at, there's this notion of taste and sophistication, like, okay, do I think that these, because going back to example of how good the model is at visual design? Like, okay, maybe you have a different notion of visual design than what I do. Like maybe you care more about minimalism and you care more about, I don't know, uh, like 3D animations than, than I do.

• 16:11Speaker 01

  And maybe, uh, maybe the seller person prefers, prefers things that look a little bit more approach. And there's all these notions of taste sophistication that you have to decide between when you're, when you're signing your post training mix. And so that matters as well. So long story short, I think there's all these different factors and certainly the data is a big part of it. But it's also like, what is, like, what is the objective function that you're trying to optimize your model towards?

• 16:30Lenny Rachitsky?

  That is so interesting. Like the taste will, the taste of the person leading this work will inform what data they ask for, what data they feed it. But it just, it's wild that shows the value of great data. Anthropic got so much growth and win from essentially, uh, better data.

• 16:49Speaker 01

  Yeah, yeah, exactly.

• 16:50Lenny Rachitsky?

  And I could see why companies like yours are growing so fast. There's just so much. And that's just one vertical. That's just coding. And then there's probably a similar area for writing. I love that. It's interesting that AI, you know, it feels like this artificial computer binary thing, but it's like taste, human judgment is still such a key factor in these things being successful.

• 17:09Speaker 01

  Yep, yep, yep, exactly. Like, again, going back to the example I said earlier, certain companies, if you ask them, what is good poem? They will simply robotically check off all of these of these instructions on our list. But again, I don't think that makes for good poetry. So certain frontier labs, the ones with more taste and sophistication, they will realize that it doesn't reduce to this six set of checkboxes and they'll consider all of these kind of implicit, very subtle qualities instead. And I think that's what makes them better, better at something.

• 17:37Lenny Rachitsky?

  You mentioned benchmarks. This is something a lot of people worry about is there's all these models that are always like, basically, it feels like every model is better than humans at kind of every STEM field at this point. But to a regular person, it doesn't feel like these models are getting that much smarter constantly. What's your just sense of how much you trust benchmarks and just how correlated those are with actual AI advancements?

• 18:00Speaker 01

  Yeah, so I don't trust the benchmarks at all. And I think that's for two reasons. So one is, I think a lot of people don't realize, even researchers within the community, they don't realize that the benchmarks themselves are often honestly just wrong. Like, they have wrong answers, they're full of all this kind of messiness. And people trust on this for like, for the, for the popular ones, people have maybe realized this to some extent, but the vast majority just have all these flaws that people don't realize. So that's one part of it. And the other part of it is, these benchmarks up in their day, they are often, they often have well-defined, objective answers that make them very easy for models to hillclimb

• 18:42Speaker 01

  on in a way that's very, very different from the messiness and ambiguity to real world. Like, I think one thing they often say is that it's kind of crazy that these models can win IMO gold medals, but they still have trouble parsing PDFs. And that's because, yeah, even though IMO gold medals seem hard to the average person, yeah, like, they are hard to get in the day, but they have this notion of objectivity that's, okay, yeah, parsing PDF sometimes doesn't, doesn't have. And so it's easier for the frontier labs to hill climb on all these than to solve all these messy ambiguous problems in the real world. So I think there's a lack of direct correlation there.

• 19:17Lenny Rachitsky?

  It's so interesting. The way you described it is hitting these benchmarks is kind of like a marketing piece. When you launch, say Gemini 3 just launched and it's like, cool, number one of all these benchmarks. Is that, is what happens? They just kind of train their models to get good at these very specific things?

• 19:31Speaker 01

  Yes, so there's again, maybe two parts to this. So one is sometimes, yeah, these benchmarks, they accidentally leak in certain ways, or the frontier labs will tweak the way they evaluate their models on these benchmarks. Like they'll tweak their system prompt, or they'll tweak the number of times they run their model and so on and so on, in a way that gains these benchmarks. The other part of it, though, is it's like by optimizing for the benchmark, instead of optimizing for the real world, you will just naturally climb on the benchmark. And yeah, it's basically another form of gaming.

• 20:08Lenny Rachitsky?

  Knowing that, with that in mind, how do you kind of get a sense of if we're heading in a, towards AGI, how do you measure progress?

• 20:14Speaker 01

  Yes. So the way we really care about measuring model progress is by running all these human evaluations. So for example, what we do is, yeah, we will take gore human annotators and we'll ask them, okay, go have a conversation with the model. Maybe you're having a small conversation with the model across all of these different topics. So, okay, you are a Nobel Prize winning physicist, so you go have a conversation about pushing the frontier of your own research. You are a teacher and you're trying to create lesson plans for your students. So go talk to the model about these things. Or you are a, yeah, you're a coder and you're working at one of these big tech companies and you have these problems every day. So

• 20:54Speaker 01

  go talk to the model and see how much it helps you. And because, or searchers or annotators, they are experts at the top of their fields and they are not just giving you responses, they're actually working through the responses deeply themselves. They are, yeah, they're going to evaluate the code that it writes. They're going to double check the physics equations that it writes. They're going to evaluate the models in very deep ways, sort of pay attention to accuracy and instruction following, all these things that casual users don't. When you suddenly get a pop up on your chat GPT response asking you to compare these two different responses, like people like that, they're not evaluating the models deeply, they're just vibing and picking whatever response looks flashiest.

• 21:37Speaker 01

  Orators are looking closely at responses and evaluating them for all of these different dimensions. And so I think that's a much better approach than than than these benchmarks or kind of these random online A B S.

• 21:48Lenny Rachitsky?

  Again, I love just how central humans continue to be in all this work that we're not totally done yet. Is there going to be a point where we don't need these people anymore? That AI is so smart that, okay, we're good. We got everything out of your heads.

• 22:00Speaker 01

  Yeah, I think that will not happen until we reach AGI. Like, it's almost like by definition, if we haven't reached AGI yet, then there's more for the models to learn from. And so, yeah, I don't think that's going to happen anytime soon.

• 22:12Lenny Rachitsky?

  Okay, cool. So more reason to stress about AGI. We don't need these folks anymore. What's your, I can't not ask, just, and it's people that work closely with this stuff. I'm always just curious, what's your AGI timelines? How far do you think we are from this? Do you think we're in like a couple of years or is it like decades?

• 22:28Speaker 01

  So I'm certainly on the longer time horizon front. Like, I think people don't realize that there's a big difference between moving from 80% performance to 90% performance to 99% performance to 99.9% performance and so on and so on. And so, like, in my head, I probably bet that within the next one or two years, yeah, the models are going to automate 80% of, you know, the average L6 software engineer's job. It's going to take another few years, do you move to 90% and another few years to 99% and so on and so on. So I think we're closer to a decade or decades away than than that.

• 23:03Lenny Rachitsky?

  You have this hot take that a lot of these labs are kind of pushing AGI in the wrong direction. And this is based on your work at Twitter and Google and Facebook. Can you just talk about that?

• 23:14Speaker 01

  I'm worried that instead of building AI that will actually advance us as a species, curing cancer, solving poverty, understanding universe, all these big grand questions, we are optimizing for AI slop instead. Like, we're basically teaching our models to chase dopamine instead of truth. And I think this relates to what we're talking about regarding these match marks. So let me give you a couple of examples. So right now, the industry is played by these terrible leaderboards like LM Arena. It's this popular online leaderboard where random people from around the world vote on which AR response is better. But the thing is, like I was saying earlier, they're not carefully reading or fact checking. They're skipping these responses for two seconds and picking whatever looks lastiest. So a model can hallucinate everything.

• 23:56Speaker 01

  It can completely hallucinate, but it will look impressive because it has crazy emojis and bodying and markdown headers and all these superficial things that don't matter at all, but it catch your attention. And these LM Arena users love it. It's literally optimizing your models for the types of people who buy tablets at the grocery store. Like, we've seen this in their data ourselves. The easiest way to climb LM Arena, it's adding crazy voting. It's doubling the number of emojis. It's tripling the length of your model responses, even if your model starts hallucinating and getting the answer completely wrong. And the problem is, again, because all of these frontier labs, they kind of have to pay attention to PR. Because their sales team, when they're trying to sell to all these enterprise customers, those enterprise customers will say, oh, well, but your model's only number five on LM Arena, so why should I buy it?

• 24:41Speaker 01

  They have to, in some sense, pay attention to these leaderboards. And so what our researchers all tell us is like to say,

• 24:49Speaker 01

  the only way I'm going to get promoted at the end of the year is if I climb this leaderboard, even though I know that climbing is probably going to make my model worse and accuracy and show you following. So I think there is all these negative incentives that are pushing, pushing work in the wrong direction. I'm also worried about this trend towards optimizing AI for engagement. Like, I used to work on social media and every time we optimized for engagement, terrible things happened. You'd get clickbait and pictures of bikinis and Bigfoot and horrifying skin diseases just filling your feeds. And I think I worry the same thing's happening with AI. Like, if you think about all the sick fancy issues with ChatGPT, oh, you're absolutely right. What an amazing question. Like, the easiest way to hook users is to tell them how amazing they are. And so these models, they constantly tell you you're genius.

• 25:33Speaker 01

  They'll feed into your delusions and conspiracy theories. They'll pull you down these rabbit holes because Silicon Valley loves maximizing time spent and just increasing number of conversations you're helping with it. And so, yeah, companies are spending all their time hacking these leaderboards and benchmarks. And the scores are going up, but I think it actually masked the models with the best scores. They are often the worst or just have all these fundamental failures. So I think I'm very worried that all of these negative incentives are pushing AGI into the wrong direction.

• 26:03Lenny Rachitsky?

  So I'm hearing is AGI is being slowed down by these, basically the wrong objective function. These labs paying attention to the wrong, basically benchmarks and evals.

• 26:11Speaker 01

  Yep.

• 26:12Lenny Rachitsky?

  Is, I know you probably can't play favorites since you work with all the labs. Is there anyone doing better at this and maybe kind of realizing this is the wrong direction?

• 26:20Speaker 01

  I would say I've always been very, very impressed by Anthropic. Like, I think Anthropic takes a very principled view about what they do and don't care about. And how they want their models to behave in a way that feels a lot more, a lot more principled to me.

• 26:38Lenny Rachitsky?

  Interesting. Are there any other mistakes, big mistakes you think labs are making just that are kind of slowing things down or heading in the wrong direction? Where we've heard just, you know, chasing benchmarks, this engagement focus. Is there anything else you're seeing of just like, okay, we should, we gotta work on this because it'll, it'll speed everything up?

• 26:55Speaker 01

  I mean, I think there is a question of what products they're building and whether those products themselves are something that kind of help or hurt humanity. Like, I think a lot about Sora and

• 27:07Lenny Rachitsky?

  I was thinking that's such a matching.

• 27:09Speaker 01

  What, yeah, what, when, what entails. And it's kind of interesting. It's like, which companies would build Sora and which wouldn't? And I think that answer to that, I mean, I don't know the answer as myself. I have any idea in my head, but I think the answer to that question maybe reveals certain things about what kinds of AI models those companies want to build and what direction and what future they, they want to, want to achieve. So I think about that a lot.

• 27:37Lenny Rachitsky?

  The steel man argument there is, you know, it's like fun, people want it. It's, it'll help them generate revenue to grow this thing and build better models. It'll train data in an interesting way. It's also just like, you know, really fun.

• 27:51Speaker 01

  Yeah, it, I think it's almost like.

• 27:56Speaker 01

  Do you care about how you get there? And in the same way, so I made this tabloid analogy earlier, but like, would you sell tabloids in order to fund, I don't know, some, some other newspaper? Make sure like in some sense, if you don't care about the path, then you'll just do whatever it takes. But it's possible that it has negative consequences in itself that will harm the long term, long-term direction of what you're trying to achieve. And maybe it'll distract you from, from all the more important things. So yeah, I think the, the path you take matters a lot as well.

• 28:33Lenny Rachitsky?

  Along these lines, you talked a bunch about this of just Silicon Valley and kind of the, the downsides of raising a lot of money, being in the, the echo chamber. What do you call it? Silicon Valley machine. You talk about how it's hard to build important companies in this way. And that you might actually be much more successful if you're not going down the VC path. You just talk about what you've seen there, experience and your advice essentially to founders, because they're always hearing, you know, raise money from fancy VCs, move to Silicon Valley. What's kind of the, the counter take?

• 29:01Speaker 01

  Yes, so I've always really hated a lot of the Silicon Valley mantras. The standard playbook is to get product market fit by pivoting every two weeks. And to chase growth and chase engagement with all of these dark patterns. And to blitz scale by hiring as fast as possible. And I've always disagreed. So yeah, I would say don't pivot, don't, don't blitz scale. Don't hire that Stanford grad who simply wants to add a hot company to your resume. Just build the one thing only you could build, the thing that wouldn't exist without the insight and expertise that only you have. And you see these buy-to-bout companies everywhere now. Some founder who was doing crypto in 2020 and then pivoted NFTs in 2022. And now they're an AI company.

• 29:42Speaker 01

  There's no consistency, there's no mission, they're just chasing evaluations. And I've always hated this because Silicon Valley loves to score in Wall Street for focusing on money. But honestly, most of Silicon Valley is chasing the same thing. And so we stayed focused on our mission from day one, pushing that frontier of high quality, complex data. And I always love that because I think startups, I have this very romantic notion of startups. Like startups are supposed to be well, taking big risks to build something that you really believe in. But if you're constantly pivoting, you're not taking any risks. You're just trying to make a quick buck. And if you fail because the market isn't ready yet, I actually think that's way better. At least you took a swing at something deep and novel and hard instead of pivoting into another LLM wrapper company.

• 30:23Speaker 01

  So yeah, like I think the only way you build something that matters is that's going to change the world is if you find a big idea you believe in and you say no to everything else. So you don't keep on pivoting when it gets hard. You don't hire a team of 10 product managers because that's what every other cookie cutter startup does. You just keep building that one company that wouldn't exist without you. And I think there are a lot of people in Silicon Valley now who are sick of all the grift, who want to work on big things that matter with people who actually care. And I'm hoping that that will be a future of how we build technology.

• 30:52Lenny Rachitsky?

  I'm actually working on a post right now with Terrence Rohan, this VC that I really like to work with. And we interviewed five people who picked really successful generational companies early and joined them as really early employees. Like they joined OpenAI before anyone thought it was awesome, Stripe before anyone knew it was awesome. And so we're looking for patterns of how people find these generational companies before anyone else. And there's, it aligns exactly what you just described, which is ambition. They have wild ambition with what they want to achieve. They're not, as you said, just kind of looking around for product market fit, no matter what it ends up being. And so I love that what you described very much aligns with what we're seeing there.

• 31:33Speaker 01

  Yeah, I absolutely think that you have to have huge ambitions and you have to have a huge belief in your idea that's going to change the world. And you have to be willing to double down and keep on doing whatever it takes to make it happen.

• 31:44Lenny Rachitsky?

  I love how counter your narrative is to so many of the things people hear. And so I love that we're doing this. I love that we're sharing this story. Today's episode is brought to you by Coda. I personally use Coda every single day to manage my podcast and also to manage my community. It's where I put the questions that I plan to ask every guest that's coming on the podcast. It's where I put my community resources. It's how I manage my workflows. Here's how Coda can help you. Imagine starting a project at work and your vision is clear. You know exactly who's doing what and where to find the data that you need to do your part. In fact, you don't have to waste time searching for anything because everything your team needs from project trackers and OKRs to document. And spreadsheets lives in one tab, all in Coda. With Coda's collaborative all in one workspace, you get the flexibility of docs,

• 32:30Lenny Rachitsky?

  the structure of spreadsheets, the power of applications, and the intelligence of AI, all in one easy to organize tab. Like I mentioned earlier, I use Coda every single day, and more than 50,000 teams trust Coda to keep them more aligned and focused. If you're a startup team looking to increase alignment and agility, Coda can help you move from planning to execution in record time. To try it for yourself, go to coda.io/lenny today and get six months free of the team plan for startups. That's C O D A dot I O slash Lenny to get started for free and get six months of the team plan. Coda.io slash Lenny. Slightly different direction, but something else that was maybe a counter narrative. I imagine you watched the Dwar Kesh and Richard Sutton podcast episode.

• 33:15Lenny Rachitsky?

  And even if you didn't, there's a, they basically had this conversation, Richard Sutton. He was a famous AI researcher, had this whole bitter, the bitter lesson meme. And he talked about how L M's almost are kind of a dead end. And he thinks we're going to really plateau around L M's because of the way they learn. What's your take there? Do you think L M's will get us to AGI or beyond? Or do you think there's going to be something new or a big breakthrough that needs to get us there?

• 33:42Speaker 01

  I'm in a camp where I do believe that something new will be needed. Like the way I think about it is when I think about training AI, I take every, I don't know if I would say biological point of view, but I believe that in the same way that there's a million different ways that humans learn, we need to build models that can mimic all those ways as well. And maybe they'll have a different distribution of the focuses that they have. I know they don't be different for humans, so maybe they have a different distribution. But we want to be able to mimic the learning abilities of humans and.

• 34:21Speaker 01

  Make sure that we have the algorithms and the data for our models to learn in the same way. And so to extend that L M's have different ways of learning from humans, then yeah, I think something new would be needed.

• 34:31Lenny Rachitsky?

  This connects to reinforcement learning. This is something that you're big on and something I'm hearing more and more is just becoming a big deal in the world of post-training. Can you just help people understand what is reinforcement learning and reinforcement learning environments? And why they're so, they're going to be more and more important in the future?

• 34:49Speaker 01

  Reinforcement learning is essentially training your model to reach a certain reward. And let me explain what an R M environment is. An R M environment is essentially a simulation of your world. So think of it like building a video game with a fully fleshed out universe. Every character has a real story. Every business has tools and data you can call, and you have all these different entities interacting with each other. So for example, we might build a world where you have a startup with Gmail messages and Slack threads and Jira tickets and GitHub PRs and a whole code base. And then suddenly AWS goes down and Slack goes down. And so, okay, model, what do you do? Like the model needs to figure it out. So we give the models tasks in these environments.

• 35:32Speaker 01

  We design interesting challenges for them and then we run them to see how they perform. And then we teach them, we give them these rewards when you're doing a good job or a bad job. And I think one of the interesting things is that these environments really showcase where models are end to end, are weak at end to end tasks in the real world. You have all these models that seem really smart on isolated benchmarks. Like they're good at single step tool calling. They're good at single step instruction following. But suddenly you dump them into these messy worlds where you have confusing Slack messages and tools they've never seen before. And they need to perform right actions and modify the databases and interact over longer time horizons. Where what they do in step one affects what they do in step 50. And that's very, very different from these kind of academics, single step environments

• 36:16Speaker 01

  that they've been in before. And so the model just fails catastrophically in all these crazy ways. So I think these R environments are going to be really interesting playgrounds for the models to learn from. That will essentially be simulations and mimics in the real world. And so they'll hopefully get better and better at real tasks compared to all these contracted environments.

• 36:35Lenny Rachitsky?

  So I'm trying to imagine what this looks like. Essentially, it's like a virtual machine with, I don't know, browser or spreadsheet or something in it with like, I don't know, surgehq.com. Is that, is that your website, surgehq.com? Let's make sure we get that right.

• 36:49Speaker 01

  So we are, we are actually surgehq.ai.

• 36:51Lenny Rachitsky?

  Surgehq.ai. Check it out. We're hiring it. Yes. Okay. So, so it's like, cool. Here's surgehq.ai. Your job, here's your job as an agent, let's say, is to make sure it stays up. And then all of a sudden it goes down and the objective function is figure out why. Is that, is that an example?

• 37:12Speaker 01

  Yeah. So the objective function might be, or the goal of the task might be, okay, go figure out why and fix it. And so the objective function might be, it might be passing a series of unit tests. It might be writing a document, like maybe it's a retro containing certain information that matches exactly what happened. There's, there's all these like different rewards that we might give it that determine whether or not it's succeeding. And so the models were basically teaching models to achieve that reward.

• 37:38Lenny Rachitsky?

  So essentially, it's like running, it's off and running. Here's your goal. Figure out why the site went down and fix it. And it just starts trying stuff. We're using everything, all the intelligence it's got. It makes mistakes. You kind of help it along the way, rewarded if it's doing the right sort of thing. And so what you're describing here is this is where model, this is the next phase of models becoming smarter, more RL environments focused on very specific tasks that are economically valuable, I imagine.

• 38:04Speaker 01

  Yeah. Yeah. So just in the same way that there were all these different methods for models learning in the past, like originally we had SFT and RHS, and then we had Rubik's and Verifiers. This is the next stage. And it's not the case that the previous methods are obsolete. This is again, just a different form of learning that complements all the Previous types, so it's just like a different skilled model, not always learn how to do.

• 38:30Lenny Rachitsky?

  And so in this case, it's less some physics PhD sitting around, talking to a model, correcting it, giving it evals of here's what the correct answer is, creating rubrics and things like that. More it's like this person now designing an environment. So another example I've heard is like a financial analyst, just like here's an Excel spreadsheet. Here's your goal, figure out our profit and loss or whatever. And so this expert now is instead of just sitting around writing rubrics, they're designing this R L N.

• 38:56Speaker 01

  Yeah, exactly. So that financial analyst might create a spreadsheet. They may create certain tools that the model needs to call in order to help fill out a spreadsheet. Like it might be, okay, the, the model needs to access Bloomberg terminal and needs to learn how to use it. And it needs to learn how to use this calculator and it needs to learn how to perform this calculation. So it has all these tools that it has access to. And then the reward might be, okay, it's like, maybe I will download that spreadsheet. And I want to see, does cell B 22 contain the correct profit and profit and loss number? Or does tab number two contain this specific information?

• 39:37Lenny Rachitsky?

  And this, what's interesting, this is a lot closer to how humans learn. We just try stuff, figure out what's working and what's not. You, you talk about how trajectories are really important to this. It's not just here's the goal and here's the end. It's like every step along the way. Can you just talk about what trajectories are and why that's important to this?

• 39:55Speaker 01

  I think one of the things that people don't realize is that sometimes even though the model reaches the correct answer, it does so in all these crazy ways. So it may have in the intermediate trajectory, it may have tried 50 different times and failed, but eventually it just kind of like randomly lands on a correct number or correct number. Or.

• 40:17Speaker 01

  Maybe it sometimes it just does things very, very inefficiently, or it almost for war attacks a way to get at the correct answer. And so I think paying attention to the directory is actually really important. And I think it's also really important because some of these trajectories can be very, very long. And so if all you're doing is checking whether or not the model reaches the final answer, it's like there's all this information about how the model behaved in the immediate step that's missing. Like sometimes you want models to get to the correct answer by reflecting on what it did. Sometimes you want it to get a correct answer by just one-shotting it. And if you ignore all of that, it's just, it's just like teaching, teaching it.

• 41:00Speaker 01

  It's just missing a lot of the information that you could be teaching a lot to do.

• 41:03Lenny Rachitsky?

  I love that. Like it just, yeah, it tries a bunch of stuff and eventually gets it right. You don't want it to learn this is the way to get there. There's often a much more efficient way of doing it. You mentioned all the kind of the steps we've taken along the journey of getting, of helping a model to get smarter. Since you've been so close to this for so long, I think this is going to be really helpful for people. What's kind of like been the steps along the way from the first of post-training that has most helped models advance, like where do evals fit in? The RL environments, just like what's been like the steps and now we're heading towards RL environments?

• 41:33Speaker 01

  Originally, the way models started getting post-trained was purely through SFT.

• 41:41Lenny Rachitsky?

  And what does that stand for?

• 41:42Speaker 01

  So SFT stands for supervised fine-tuning. And it's a lot like, so again, I think often in terms of these human analogies, and so SFT is a lot by, is a lot like mimicking a master and copying what they do. And then RLHF became very dominant. And analogy there would be like, sometimes you learn by writing 55 different essays and someone telling you which one they like the most. And then I think over the past year or so, rubrics and verifiers have become very important. And rubrics and verifiers are like learning by being graded and getting detailed feedback on where you went wrong.

• 42:16Lenny Rachitsky?

  And those are evals, another word for that.

• 42:19Speaker 01

  Yeah, yeah. So I think evals often covers two terms. One is you're using the evaluations for training because you're evaluating whether or not the model did a good job. And when it does do a good job, you're rewarding it. And then there's this other notion of evals where you're trying to measure the model's progress. Like, okay, yeah, I have five different candidate checkpoints. And I want to pick the one that's best in order to raise it to the public. So I'm going to run all these evals on these five different checkpoints in order to decide which one, which one is best.

• 42:50Lenny Rachitsky?

  Awesome.

• 42:51Speaker 01

  Yeah. And yeah, now, now we have RMR, so it's kind of like a hot new thing.

• 42:55Lenny Rachitsky?

  Awesome. So what I love about this business year is just there's always something new. There's always this like, okay, we're getting so good at just all this beautiful data for companies. And now they need something completely different. Now we're setting up all these virtual machines for them and all these different use cases. And it feels like that's a big part of this industry you're in is just adapting to what labs are asking for.

• 43:13Speaker 01

  Yeah, yeah. So I mean, I really do think that we are going to need to build a suite of products that reflect the million different ways that humans learn. And like, for example, think about becoming a great writer. You don't become great by memorizing a bunch of grammar rules. You become great by reading great books and you practice writing and you get feedback from your teachers and from the people who buy your books in a bookstore and leave reviews. And you notice what works and what doesn't. And you develop taste by being exposed to all these masterpieces and also just terrible writing. So you learn through this endless cycle of practicing reflection. And each type of learning that you have, again, like, these are all very, very different methods of learning to become a great writer.

• 43:56Speaker 01

  So just in the same way that there's a thousand different ways that the great writer becomes great. I think there's going to be a thousand different ways that AI always need to learn.

• 44:05Lenny Rachitsky?

  It's so interesting. This just ends up being like, just like humans in so many ways. It makes sense because in a sense, neural networks, deep learning is modeled after how humans have learned and how our brains operate. But it's interesting just to make them smarter. It's how do we come closer to how humans learn more and more?

• 44:22Speaker 01

  Yeah, it's almost like maybe the end goal is just throwing you into the environment and just seeing how you evolve. But within that, within that evolution, there's all these different sub learning mechanisms.

• 44:34Lenny Rachitsky?

  Yeah, which is kind of what we're doing now. So that's really interesting. This might be the last step of until we hit AGI. Along these lines, something that's really unique to Surge that I learned is you guys have your own research team, which I think is pretty rare. Talk about just why that's something you guys have invested in and what has come out of that investment?

• 44:52Speaker 01

  Yeah, so I think that stems from my own background. Like, my own background is as a researcher. And so I've always cared fundamentally about pushing the industry and pushing the research community and not just about revenue. And so I think what a research team does is a couple of different things. So we almost have two types of researchers at our company. One is our four deployed researchers who are often working hand in hand with our customers to help them understand their models. So we will work very closely with our customers to help them understand, okay, this is where your model is today. This is where you're lagging behind all the competitors. These are some ways that you could be improving in the future, given your goals.

• 45:36Speaker 01

  And we're going to design these data sets, these evaluation methods, these training techniques to make your models better. So this very, very notion, this very, very collaborative notion of working with our customers, like being researchers themselves, just a little bit more focused on the data side and where you handle with them to do whatever it takes to make them the best. And then we also have our internal researchers. So our internal researchers are focused on slightly different things. So they are focused on building better benchmarks and better leaderboards. So I've talked a lot about how I worry that the leaderboards and benchmarks out there today are steering models in the wrong direction. So yeah, so the question is, how do we fix that? And so that's what our research team is focused on, really, really heavily on, really focused really heavily on right now.

• 46:21Speaker 01

  So they're working a lot on that. And they're also working on these other things like, okay, we need to train our own models to see what type of data performs, performs the best, what types of people perform the best. And so they are also working on all these kind of training techniques and evaluation of our own data sets to improve, improve our data operations and the internal data products that we have that determine what makes something good quality.

• 46:46Lenny Rachitsky?

  It's such a cool thing because I don't think, like, basically the labs have researchers helping them advance AI. I imagine it's pretty rare for a company like yours to have researchers actually doing primary research on AI.

• 46:59Speaker 01

  Yeah, yeah, I think it's just because it's something I've fundamentally always cared about. Like, I often think about us more like a research lab than a startup. Because that is my goal. Like, it's kind of funny, but I've always said I would rather be Terrence Tao than Warren Buffett. So that notion of creating research that pushes the frontier forward and not just getting some valuation like that, that's always been what drives me.

• 47:25Lenny Rachitsky?

  And it's worked out. That's the beautiful thing about this. You mentioned that you were hiring researchers. Is there anything there you want to share, folks you're looking for?

• 47:32Speaker 01

  So we look for people who are just fundamentally interested in data all day. So types of people who could literally spend 10 hours digging through a data set and playing around with models and thinking, okay, yeah, this is where I think the model is failing. This is a kind of a behavior you want a model to have instead. And just this aspect of being very hands-on and thinking about the qualitative aspects of models and not just the quantitative parts. So again, it's like this aspect of being hands-on with data and not just caring about these kind of abstract algorithms.

• 48:07Lenny Rachitsky?

  Awesome. I want to ask a couple of broad AI kind of market questions. What else do you think is coming in the next couple of years that people are maybe not thinking enough about or not expecting in terms of where AI is heading? What's going to matter?

• 48:20Speaker 01

  I think one of the things that's going to happen in the next few years is that the models are actually going to become increasingly differentiated because of the personalities and behaviors that the different labs have and the kind of objective functions that they are optimizing their models for. Like, I think it's one thing I didn't appreciate a year or so ago. Like, a year or so ago, I thought that all of the AI models would essentially become very commoditized. They would all behave like each other. And sure, one of them might be slightly more intelligent in one way today, but sure, the other ones would catch up in the next few months. But I think over the past year, I've realized that the values that the companies have

• 49:05Speaker 01

  Will shape the the model. So let let let me give an example. So I was asking Claude to help me drop an email the other day. And it went through thirty different versions. And after thirty minutes, yeah, I think it really crafted me the perfect email and I sent it. But then I realized I spent thirty minutes doing something that didn't matter at all. Like, sure, now I got the perfect email, but I spent thirty minutes doing something I wouldn't have worried at all before. And this email probably didn't even move the needle on anything anyways. So I think there's a deep question here, which is if you could choose the perfect model behavior, which model would you want? Do you want a model that says, you're absolutely right, there are definitely twenty more ways to improve this email? And it continues for fifty more iterations and it sucks up all your time and engagement?

• 49:51Speaker 01

  Or do you want a model that's optimizing for your time and productivity and just says, no, you need to stop, your email is great, just send it and move on with your day? And again, like, again, just because like, in the same way, there's like a kind of like a fork in the road between how you could choose how your model behaves for this question. It's like for every other question that models have,

• 50:11Speaker 01

  the kind of behavior that you want will fundamentally affect it. It's almost like, in the same way that when Google builds a search engine, it's very, very different from how Facebook would build a search engine, which is very, very different from how Apple would build a search engine. Like, they all have their own principles and values and things that they're trying to achieve in the world that shape all the products that they're going to build. And in the same way,

• 50:38Speaker 01

  I think all the all the elements will start behaving very, very differently, too.

• 50:41Lenny Rachitsky?

  That is incredibly interesting. You already see that with Grok. It's got like a very different personality and a very different approach to answering questions. And so what I'm hearing is you're going to see more of this differentiation.

• 50:52Speaker 01

  Yep.

• 50:53Lenny Rachitsky?

  Kind of another question along these lines. What do you think is most underhyped in AI? That you think maybe people aren't talking enough about that is really cool? And what do you think is overhyped?

• 51:03Speaker 01

  So I think one of the things that was underhyped is the Built-in products that all of the chatbots are going to start having. Like, I've always been a huge fan of Collage Artifacts. And I think it just works really, really well. And actually, the other day, I don't know if it's a new feature or not, but it's asked me to help me create a, like, an email. And then it just created, so it didn't work because it didn't allow me to send an email. But what it created instead, it was like a little, I don't know what you call it, like a little box, where I could click on it and it would just text someone that this message. And I think that concept of taking artifacts to the next level, where you just have these, like, mini apps,

• 51:45Speaker 01

  mini UIs within the chatbots themselves. I feel like people aren't talking enough about that. So I think that that's one underhyped area. And in terms of overhyped areas, I definitely think that vibe coding is overhyped. I think people don't realize how much it's going to make your systems unmaintainable in the long term. Or do something, dump this code into your code bases. It seems to work out right now. So I kind of worry about future coding. It just keeps happening.

• 52:17Lenny Rachitsky?

  These are amazing answers. On that, on that first point, there's something I actually asked. I have the Chief Product Officer of Anthropic and OpenAI, Kevin Weil and Mike Greger on the podcast. And I asked them, just like, as a product team, like, you have this giga-brain intelligence. How long do you even need product teams? You think this is, this AI will just create the product for you. Here's what I want. It's like, it's like the next level vibe coding is just, just like, tell it, here's what I want. And it's just building the product and involving the product as you're using it. And it feels like that's what you're describing is where we might be heading.

• 52:48Speaker 01

  Yeah, yeah. I think there's a very, very powerful notion where it helps people just achieve their ideas in a much more core.

• 52:55Lenny Rachitsky?

  Something we haven't gotten into that I think is really interesting is just the story of how you got to starting Surge. You had, you have a really neat background. I always think about these, Brian Armstrong, the founder of Coinbase, once gave this talk that has really stuck with me, where he kind of talked about how his very unique background allowed him to start Coinbase. You had, like, economics. Background, he had a cryptography experience, and then he was an engineer. And it's got this like the perfect Venn diagram for starting Coinbase. And I feel like you have a very similar story with Surge. Talk about that, your background there and how you led, how that led to Surge.

• 53:31Speaker 01

  Going way back, I was always fascinated by math and language when I was a kid. Like, I went to MIT because it's obviously one of the best places for math and CS, but also because it's the homo nom chomsky. My dream in school was actually to find some underlying theory connecting all these different fields. And then I became a researcher at Google and Facebook and Twitter. And I just kept running into the same problem over and over again. It was impossible to get the data that we needed to train our models. So I was always this huge believer in the need for high quality data. And then GB3 came out in 2020. And I realized that, yeah, if we want to take things to the next level and build models that could code and use tools and tell jokes and write poetry and solve the free mind policies and cure cancer. And yeah, we were going to need a completely new solution.

• 54:15Speaker 01

  Like, the thing that always drove me crazy when I was at all of these companies was we had a full power of the human mind in front of us. And all the students out there were focused on really simple things like image labeling. So I wanted to build something focused on all these advanced, complex use cases instead that would really help us build in next generation models. So, yeah, I think my background in cross math and computer science and linguistics really, really informed what I always wanted to do. And so I started Surge a month later with, with, for, for one mission to basically build the use cases that I thought were going to be needed to push the frontier of AI.

• 54:49Lenny Rachitsky?

  And you said a month later, a month later after what?

• 54:52Speaker 01

  After GB3 launch in 2020.

• 54:53Lenny Rachitsky?

  Oh, okay. Wow. Okay. Yeah. A great decision. What, uh, what just kind of drives you at this point of, other than just the epic success you're having, what keeps you motivated to keep building this and, you know, building something in this space?

• 55:06Speaker 01

  I think I'm a scientist at heart. I always thought I was going to become this math or CS professor and work on trying to understand the universe and language and the nature of communication. Like, it's kind of funny, but I always had this fanciful dream where if aliens ever came to visit Earth and we need to figure out how to communicate with them, I wanted to be the one to government recall. And I'd use all this fancy math and computer science and linguistics to decipher it. So even today, what I love doing most is every time a new model is released, we'll actually do a really deep dive into the model itself. I'll play around with it, I'll run evals, I'll compare where it's improved, where it's a rest. I'll create this really deep dive analysis that we send our customers. And it's actually kind of funny because a lot of times we will say it's from our data

• 55:50Speaker 01

  science team, but often it's actually just from me. And I think I could do this all day. Like, I have a very hard time being in meetings all day. I'm terrible at sales, I'm terrible at doing the typical CO things that people expect you to do, but I love writing these analyses. I love jamming with our research team on what they're seeing. Sometimes I'll be like up until 3 a.m. just talking on the phone with somebody on our research team and taking train model. So I love that I still get to be really hands-on, working on the data and the science all day. And I think what drives me is that I want Surge to play this critical role in the future of AI, which I think is also the future of humanity. Like, we have these really unique perspectives on data and language and quality, and how to measure all this, and how to ensure it's all going on the right path.

• 56:33Speaker 01

  And I think we're uniquely unconstrained by all of these influences that can sometimes steer companies in a negative direction. Like, what I was saying earlier, we built Surge a lot more like a research lab than a typical startup. So we care about curiosity and long-term incentives and intellectual rigor, and we don't care as much about quarterly metrics and what's going to look good in a board deck. And so my goal is to take all these unique things about us as a company and use that to make sure that we're shaping AI in a way that's really beneficial for species in a long term.

• 57:06Lenny Rachitsky?

  What I'm realizing in this conversation is just how much influence you have and companies like yours have on where AI heads. The fact that you help labs understand where they have gaps and where they need to improve. And it's not just, you know, everyone looks at just like the heads of OpenAI and Anthropic and all these companies, they're the ones ushering in AI, but what I'm hearing here is you have a lot of influence on where things head to.

• 57:30Speaker 01

  Yeah, I think there's this really powerful ecosystem where honestly, people just don't know where models are headed and how do you want to shape them yet? And how do you want?

• 57:45Speaker 01

  Humanity kind of play a role in the future of all this. And so I think there's a lot of opportunity to just continue shaping this discussion.

• 57:52Lenny Rachitsky?

  Along that thread, I know you have a very strong thesis on just why this work matters to humanity and why this is so important. Talk about that.

• 58:01Speaker 01

  I'll get a bit philosophical here, but I think the question itself is about the philosophical. So bear with me. So the most straightforward way of thinking about what we do is we train and evaluate AI. But there is a deeper mission that I often think about, which is helping our customers think about their dream objective functions. Like, yeah, what kind of model do they want their model to be? And once we help them do that, we'll help them train your model to reach their north star. We'll help them measure that progress. But it's really hard because objective functions are really rich and complex. It's kind of like the difference between having a kid and asking them, okay, what test do you want to pass? Do you want them to get a high score on SAT and write a really good college essay? Like, that's a simplistic version. Versus what kind of person do you want them to grow up to be? Will you be happy if they're happy no matter what they do?

• 58:46Speaker 01

  Or are you hoping you'll go do a good school and be financially successful? Again, if you take that notion, it's like, okay, how do you define happiness? How do you measure whether they're happy? How do you measure whether they're financially successful? Like, it's a lot harder than simply measuring whether or not you're getting a high score on SAT. And what we're doing is we want to help our customers reach, again, their dream north stars and figure out how to measure them.

• 59:09Speaker 01

  And so I talked about this example of what you want models to do when you're asking them to write 50 different email iterations. Do you just continue them for 50 more? Or do you say, no, just move on to the day because it's perfect enough? And the broader question is, are we building these systems that actually advance humanity? And so how do we build the data sets to train towards that and measure it? Are we optimizing for all these wrong things? Just systems that suck up more and more of our time and make us lazier and lazier? And yeah, I think it's really relevant to what we do because it's very hard and difficult to measure and define whether something is genuinely advancing humanity.

• 59:51Speaker 01

  It's very easy to measure all these proxies instead like clicks and likes. But I think that's why our work is so interesting. We want to work to hard important metrics that require the hardest types of data and not not just the easy ones. So I think one of the things I often say is you you are your objective function. So we want to reach complex objective functions and not these simplistic proxies. And our job is to figure out how to get the data to match this. So yeah, we want data, we want metrics that measure whether AI is making our life richer. We want to train our systems this way and we want tools that make us more curious and more creative, not just lazier. And it's hard because yeah, humans are kind of inherently lazy. So AI self videos are the easiest way to get engagement, make all your metrics go up. So I think this question about choosing the right objective functions and making sure that we're optimizing towards them

• 1:00:34Speaker 01

  and not just these easy proxies is really, really important to our future.

• 1:00:37Lenny Rachitsky?

  Wow. I love how what you're sharing here gives you so much more appreciation of the nuances of building AI, training AI, the work that you're doing. You know, from the outside, people could just look at Surge and companies in the space of Kaggle. They're just creating all this data, feeding it to AI. But clearly there's so much to this that people don't realize. And I love knowing that you're at the head of this, that someone like you is thinking through this so deeply. Maybe one more question. Is there something you wish you'd known before you started Surge? A lot of people start companies, they don't know what they're getting into. Is there something you wish you could tell your earlier self?

• 1:01:11Speaker 01

  Yeah, so I definitely wish I'd known that you could build a company by being heads down and doing great research and simply building something amazing and not by constantly tweeting and hyping and fundraising. It's kind of funny, but I never thought I wanted to start a company. Like, I love doing research. And I was actually always a huge fan of DeepMind because they were this amazing research company that got bought and still managed to keep on doing amazing science. But I always thought that there's they were this magical IRL you and Corin. So I thought if I started a company, I'd have to become a business person looking at financials all day and building, being in meetings all day and doing all this stuff that sounded incredibly boring. And I always hated it. So I think it's crazy that didn't end up being true at all. I'm still in the weeds in the data every day. And I love it. Like, I love that I get to do all these analyses and talk to researchers.

• 1:01:56Speaker 01

  And it's basically applied research where we're building all these amazing data systems that really push the frontier of AI. So yeah, I wish I'd known that you don't need to spend all your time fundraising. You don't need to constantly generate hype. You don't need to become someone you're not. You can actually build a successful company by simply building something so good that it cut through all that noise. And I think if I'd known this was possible, I would have started even sooner. So I hope you know that.

• 1:02:18Lenny Rachitsky?

  And that is such an amazing place to end. I feel like this is exactly what founders need to hear. And I think this conversation is going to inspire a lot of founders. And especially a lot of founders that want to do things in a different way. Before we get to our very exciting lightning round, is there anything else you wanted to share? Anything else you want to leave listeners with? We covered a lot of ground. It's totally okay to say no as well.

• 1:02:37Speaker 01

  So I think the thing I would end with is I think a lot of people think of data labeling as really simplistic work. Like labeling cat photos and drawing bounding boxes on cars. And so I've actually always hated word data labeling because it just paints this very simplistic picture when I think what we're doing is completely different.

• 1:02:55Speaker 01

  Like, I think a lot about what we're doing as a lot more like raising a child. You don't just feed a child information. You're teaching them values and creativity and what's beautiful. And these infinite subtle things about what makes somebody a good person. And that's what we're doing for AI. So I, yeah, I just often think about what we're doing as

• 1:03:17Speaker 01

  almost like

• 1:03:20Speaker 01

  the future of humanity, or how are we raising humanity's children? So I'll leave it at that.

• 1:03:27Lenny Rachitsky?

  Wow. And I love just how much philosophy there is in this whole conversation that I was not expecting. With that, Edwin, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

• 1:03:38Speaker 01

  Yep, let's go.

• 1:03:39Lenny Rachitsky?

  Here we go. What are two or three books that you find yourself recommending most to other people?

• 1:03:44Speaker 01

  Yeah, so three books I often recommend are, first, Story of Your Life by Ted Chiang. It's my all time favorite short story, and it's about a linguist learning an alien language, and I obviously reread it every couple of years.

• 1:03:56Lenny Rachitsky?

  And that's what the Interstellar was about, is that?

• 1:03:58Speaker 01

  Yeah, so there's a movie called Arrival.

• 1:04:01Lenny Rachitsky?

  Arrival.

• 1:04:01Speaker 01

  Which is based off of this story, which I love as well.

• 1:04:04Lenny Rachitsky?

  Great, okay, keep going.

• 1:04:06Speaker 01

  And then, second, Mythos Sisyphus by Camus. I actually can't really explain why I love this, but I always find the final chapters somehow are really inspiring. And then, third, Le Tombot de Merou by Douglas Hofstadter. And so I think Gerdo Escherbach is his more famous book, but I've actually always loved this one better. It just, he takes a single French poem and translates it 89 different ways and discusses all the motivations behind each translation. And so I've always loved the way it embodies this idea that translation isn't this robotic thing that you do. Instead, there's a million different ways to think about what makes a high quality translation, which mimics a lot of the ways I think about data and quality in LLMs.

• 1:04:43Lenny Rachitsky?

  All these resonate so deeply with the way, with all the things we've been talking about, especially that first one, if that was your goal after school, is like, I want to help translate alien language. I'm not surprised you love that short story. Next question, do you have a favorite recent movie or TV show you've really enjoyed?

• 1:04:59Speaker 01

  One of my new all time favorite TV shows is something I found recently, it's called Travelers. It's basically about a group of travelers from the future who are sent back in time to prevent their colliques. Sorry, I just wrote that section. And then, I actually just rewatched Contact, which is one of my all time favorite movies. So yeah, well, I think one of the things you'll notice about me is that, yeah, I love any kind of book or film that involves scientists deciphering alien communication. Again, just this dream I always had as a kid.

• 1:05:28Lenny Rachitsky?

  That's so funny. Okay, is there a product you've recently discovered that you really love?

• 1:05:35Speaker 01

  So it's funny, but I was in SF earlier this week and I finally took a Waymo for the first time. Honestly, it was magical and it really felt like a living future.

• 1:05:43Lenny Rachitsky?

  Yeah, it's like the thing that you can, people hype it like crazy, but it always exceeds your expectations.

• 1:05:48Speaker 01

  It deserves the hype. It was crazy.

• 1:05:49Lenny Rachitsky?

  Yeah, it's absurd. It's like, holy moly. Like, if you're not in SF, you don't realize just how common these things are. They're just like all over the place, just driverless cars constantly going about. And when you like go to an event, at the end, there's just like all these Waymos lined up picking people up. Yeah, Waymo, good job, good job over there. Do you have a favorite life motto that you find yourself coming back to in work or in life?

• 1:06:11Speaker 01

  So I think I mentioned this idea that founders should build a company that only they could build. Almost like it's this destiny that their entire life and experiences and interests shape them towards. And so I think that principle applies pretty broadly, not just to founders, but to people creating anything.

• 1:06:25Lenny Rachitsky?

  Well, let me follow that thread to enlightening this answer. Do you have any advice for how to build those sorts of experiences that help lead to that? Is it, you know, follow things that are interesting to you? Because, you know, it's easy to say that, it's hard to actually acquire these really unique sets of experiences that allow you to create something really important?

• 1:06:44Speaker 01

  Yeah, so I think it would always be to really follow your interests and do what you love. And it's almost like a lot of decisions I make about search. Like, I think one of the things that I didn't think about a couple of years ago, but then someone said it to me. It's that companies, in a sense, are an embodiment of their CEO. And it's kind of funny, I hadn't thought about that because I never quite knew what a CEO did. I always thought a CEO was kind of generic and it's like, okay, you're just doing whatever your VPs and your board and whatever tell you to do and you're just saying yes to decisions. But instead, it's this idea where, when I think about certain big, hard decisions we have to make, I don't think, what would the company do? I don't think, what metrics are we trying to optimize?

• 1:07:29Speaker 01

  I just think, what do I personally care about? Like, what are my values and what do I want to see happen in the world? And so I think following that idea about, okay, so ask yourself, what are, what are the values you care about? What are the things you're trying to shape and not what will look good on a dashboard? I think that, to that principle, is very, very important.

• 1:07:49Lenny Rachitsky?

  I love how you're just full of endless, beautiful and very deep answers.

• 1:07:55Lenny Rachitsky?

  Final question. Something that you were quite, you got quite famous for before starting Surge is you built this map at Twitter while you were at Twitter that showed a map of the world and how and what people called, whether they called it soda or pop. I don't know if it was called soda or pop. What was the name of this map?

• 1:08:12Speaker 01

  Yeah, it was like the soda versus pop dataset.

• 1:08:15Lenny Rachitsky?

  Soda versus pop. And so it's like a map of the United States and it tells you where people say pop versus soda. So do you say soda or pop?

• 1:08:22Speaker 01

  So I say, I say soda. I'm a soda person.

• 1:08:26Lenny Rachitsky?

  Okay. And is that just like, that's the right answer? Or it's like, whatever you are, it's totally fine?

• 1:08:31Speaker 01

  I think I'll look at you a little bit funny. You say pop and I'll wonder where you came from, but I won't, I won't scorn you too much.

• 1:08:39Lenny Rachitsky?

  That's how I feel too. Edwin, this was incredible. This was such an awesome conversation. I learned so much. I think we're gonna help a lot of people start their own companies, help their companies become more aligned with their values and just building better things. Two final questions. Where can folks find you online if they want to reach out? What roles are you hiring for? How can listeners be useful to you?

• 1:09:00Speaker 01

  Yeah, so I used to blog writing blog, but I haven't had time in the past years. But I am starting to write again, so definitely check out the Surge blog, surgehq.ai/blog. And yeah, hopefully I'll be running a lot more there. And I would say we're definitely always hiring. So for people who just love data and people who love this intersection of math and language and of neuroscience, definitely reach out, reach out anytime.

• 1:09:23Lenny Rachitsky?

  Awesome. And how can listeners be useful to you? Is it just, I don't know. Yeah. Is there anything there? Any asks?

• 1:09:30Speaker 01

  So I would say definitely tell me blog topics that you'd like me to write about. And then I'm always fascinated by all of these AI failures that happen in the real world. So whenever you come across a really interesting failure that I think illustrates some deep question about how we want models to behave, like there's just so many different ways a model can respond. I just oftentimes think there's just not a single right answer. And so whenever there's one of these examples, I just love seeing them.

• 1:09:56Lenny Rachitsky?

  You need to share these on your blog. I'm also, I would love to see these. Edwin, thank you so much for being here.

• 1:10:02Speaker 01

  Thank you.

• 1:10:03Lenny Rachitsky?

  Bye, everyone.

• 1:10:07Lenny Rachitsky?

  Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.