From Code Search to AI Agents: Inside Sourcegraph's Transformation with CTO Beyang Liu

Sourcegraph's CTO just revealed why 90% of his code now comes from agents—and why the Chinese models powering America's AI future should terrify Washington. While Silicon Valley obsesses over AGI apocalypse scenarios, Beyang Liu's team discovered something darker: every competitive open-source coding model they tested traces back to Chinese labs, and US companies have gone silent after releasing Llama 3. The regulatory fear that killed American open-source development isn't hypothetical anymore—it's already handed the infrastructure layer of the AI revolution to Beijing, one fine-tuned model at a time.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Martin Casado?

  This is the first time in computer science I can think of where we've actually abdicated like correctness in logic to it. Like in the past, it was a resource, right? So maybe the performance is different, maybe the availability is different, but like, whatever I put in, I'm gonna get back out. But now we're like, figure out this problem for me.

• 0:16Speaker 01

  You talk to some devs and they're like, you know, I've never been more productive, but coding isn't fun anymore. That's one of the things that we're trying to solve for. It's like amazing new technology, it feels like magic. Never experienced anything like this before in my life. Then the narrative that was fun was like, this thing will just, you know, run our lives for us, or it's gonna kill us all.

• 0:33Martin Casado?

  Total annihilator.

• 0:34Speaker 01

  Like Terminator. And there's just like absolutely no danger that like this thing's gonna, you know, acquire a mind of its own and like try to reach out of the computer and kill you.

• 0:41Martin Casado?

  If you use this every day, right? This idea that this thing could take over the world.

• 0:45Speaker 01

  Yeah. Exactly. That narrative, I think, is largely been dispelled within our circles. But I think that it's sort of like taken on a life of its own in other circles and it's made its way to some of the halls of policymaking in the US. This is the old adage of like, you know, do you blame it on ignorance or malice? I honestly don't know, but it is clearly like nonsensical and I think very much in the the national interests to be still telling this story.

• 1:15Speaker 02

  The United States invented the AI revolution. We built the chips, trained the frontier models, and created the entire ecosystem. But right now, if you're a startup building AI products, you're probably writing your code on Chinese models. Today's guest is Beyang Lee, one of the co-founders of Sourcegraph. Beyang is joined by A16Z's Martín Casado and Guido Appenzeller to talk about the shift he's seeing on the front lines of software development today. Sourcegraph's coding agent, which has hit number one on the benchmark for merge pull requests, runs on open source models. Many of them are Chinese, not because of ideology, but because they work better for what the company needs. Here's the tension. Beyang studied machine learning under Daphne Kohler at Stanford. He spent a decade building developer tools.

• 2:00Speaker 02

  He knows the technology cold. And his view is that we're sleepwalking into a dependency problem, not because Chinese models are dangerous, but because American policy has made it nearly impossible to compete in open source AI. We dig into why the Terminator narrative around AI safety might be our biggest strategic mistake, whether it's already too late to catch up, and what happens when the atomic unit of software isn't a function anymore, but a stochastic subroutine you can't fully control.

• 2:29Martin Casado?

  Behind, thanks for coming and joining. So the topic today is AI encoding. But I mean, I would say you're one of the world's experts on this. And so we would love to kind of do a deep dive and kind of how you view the problem, how you view the solution. Of course, you're a co-founder and CTO of SourceCraft.

• 2:43Speaker 01

  Yeah.

• 2:44Martin Casado?

  And so we'll talk a bit about that as well. Of course, we've got Guido. Thanks for being here. And so maybe just to start, we can do a bit of a background on you and then we'll just kind of dig into details.

• 2:54Speaker 01

  Yeah, so background, I've been working on dev tools for more than the past decade of my life. Started SourceCraft about 10 plus years ago, brought the world's first kind of like production legit code search engine to market and pushed that to, I think, a good portion of the Fortune 500. Prior to that, I was a developer at Palantir. And I guess now it's like the early days, right? That's where I met my co-founder Quinn and we were working on data analysis software and a lot of large enterprise code bases that we're kind of like drop shipped into and realized that there was a big need for better tooling for understanding massive code bases. And then before that, I guess relevant now is I actually did machine learning as a concentration when I was doing my studies. So I did some computer vision research.

• 3:38Martin Casado?

  Know that?

• 3:39Speaker 01

  Yeah, under Daphne Kohler at the.

• 3:41Martin Casado?

  I didn't know that. You're an AI guy.

• 3:43Speaker 01

  Yeah, yeah. OG AI.

• 3:44Martin Casado?

  Dawson Angler, like Tyler Stoff. I thought you were a systems, you talked like a systems guy.

• 3:50Speaker 01

  Yeah, yeah. For me, like this whole phenomenon of like LLMs and coding, it's almost like a homecoming of sorts. Cause I really.

• 3:56Martin Casado?

  I didn't know that. That's awesome.

• 3:57Speaker 01

  Yeah.

• 3:58Martin Casado?

  Yeah.

• 3:58Speaker 03

  Definitely taught me AI as well, so.

• 4:01Speaker 01

  She's a great teacher.

• 4:02Speaker 03

  Yes. That's amazing.

• 4:03Martin Casado?

  I gotta say, that was like the one class I was so happy I comped out of because I didn't think I would do well. If I didn't pass the comp, I thought it would defeat me.

• 4:11Speaker 03

  I think I failed the comp too many times. I had to take it.

• 4:16Speaker 01

  There were those two. I was the TA for that class, actually.

• 4:19Martin Casado?

  121?

• 4:19Speaker 01

  Uh, 228.

• 4:21Martin Casado?

  228. Yeah, that's right. That's what it was. Yeah, yeah. So, cool. Great. So SourceCraft started code search.

• 4:27Speaker 01

  Yeah.

• 4:27Martin Casado?

  Navigation, but then now you've been making waves with AMP, which is like, you know, an agent, what we call it. So maybe talk through a little bit about what you've been working on, maybe pre-AI and now. Just to level set.

• 4:38Speaker 01

  Yeah, so kind of like the history of the company is we were really built to make coding a lot more efficient inside large organizations and to make the practice of actually like building software way more accessible, primarily to professional software engineers. But I think our eventual vision was always to expand the franchise. And we started by tackling like the key problem, which is enabling humans to understand code, because if you've ever worked inside a large code base, you know that that probably takes anywhere from 80 to 99% of the time. And then the remainder is when you actually understand the problem well enough to actually write the code. So that's where we kind of like built up our domain expertise. And then when LLMs sort of matured, it was something that we were always kind of like monitoring in the back of our minds. Originally looked at LLMs and embeddings as a way to enhance the ranking signals that

• 5:23Speaker 01

  we're incorporating into our search engine. And then when things really hit their stride with ChatGPT and all that, it was fairly obvious to us that there was a big opportunity to combine LLMs, which are this amazing technology, with a lot of the stuff that we'd built up to that point. And then I guess to round that out, finally, our latest product is this coding agent called AMP.

• 5:44Martin Casado?

  What's interesting about AMP is it's kind of viewed as like a very sophisticated kind of opinionated view on agents. Do you share that or is that just kind of the outside?

• 5:56Speaker 01

  I would say there's certain things that we're doing that I think are quite unique.

• 6:00Martin Casado?

  It was the top recently on one of these benchmarks, right?

• 6:02Speaker 01

  Yeah, I think there's like some startup out there that compares pull or request merge rates or something and we would manage to claim the top spot.

• 6:08Martin Casado?

  It's awesome.

• 6:09Speaker 01

  Yeah. Yeah, it was very gratifying to see. But again, like I would say that I think we're opinionated on some parts of our philosophy of building agents. And my own take is I think a lot of these opinions will soon become widespread. But there's other elements of what we're doing, which are like, people like to read in a lot to AI these days. And sometimes it's just like, look, we actually did something very simple here, yielded good results and we shipped that and it works very well.

• 6:35Speaker 03

  So I think your focus is really on large code bases. How's that structurally different from, you know, me coding my little homebrew tool?

• 6:42Speaker 01

  Yeah, so it's funny you mention that. Like historically, the company's focus has really been on large code bases. But with AMP, we decided to build it almost completely separate from the existing code. And the reason for that was, one, we built AMP. AMP is really, at this point, like seven, eight months old. So we started AMP in around like February, March this year. And that was right at the wave of this new type of LLM hitting the world, the like agentic tool use LLM.

• 7:11Speaker 03

  And it worked.

• 7:12Speaker 01

  Yeah, finally worked, right? Like after so many demo videos, finally there was a model that could actually do robust tool calling and compose that with reasoning. And our original tack was like, okay, let's build this into some of the existing things that we've created. But the more we started playing around with the technology, the more we sort of came to the conclusion that this was actually truly disruptive. And we should actually start from first principles to see, you know, build the agent from the ground up and see what tools we really need. So what we've arrived at is the coding agent, which works, I think, very well in large code bases. Because again, we push this to a lot of our customers. But it's also great for like hobby coding. Like I spun my dad up recently on it and he's been using it to create these like iPad games for our kid.

• 7:56Speaker 01

  Because, you know, typical Asian dad fashion, trying to teach him math, right? I want to teach him arithmetic and whatnot. And so my dad, who's never written a single line of code in his life, is able to just like, hey, make a simple game that has him count the numbers. And then if he gets it right, the little rocket ship blasts off. So it's kind of interesting. It's a really interesting time to be building because even if you're building for professional developers as we are, a lot of the technology ends up being just kind of widely accessible.

• 8:24Speaker 03

  This is the new parenting. You're taught as a parent not to write the games for your kids that are age appropriate. They're well into great of curriculum what they have supposed to learn.

• 8:31Speaker 01

  Yeah.

• 8:31Speaker 03

  I love it.

• 8:32Martin Casado?

  Another thing that's been kind of made kind of splash is you've recently decided to go to an advertisement based model. So like, on one hand, so I've got this dissonance internally, which is on one hand, I'm like. This is the boutique.

• 8:44Speaker 01

  Yeah.

• 8:44Martin Casado?

  Sophisticated. On the other hand, I'm like, and it's also for everybody with ads. And so like, how do you kind of reconcile?

• 8:49Speaker 01

  It's really funny because I think we, we had this sort of reputation for being like the primo agent.

• 8:54Martin Casado?

  Yeah, totally.

• 8:55Speaker 01

  The super intelligent one, but we never had like a flat rate pricing model. We did pure usage based pricing and that also meant that there was never any incentive to switch to a cheaper model for our users. So our tack was like the most intelligence and you just pay for the inference cost. But as we built more and more, we kind of realized there's sort of this efficient frontier that you can draw. This two by two grid and one axis is intelligence, but the other axis is latency. And there's multiple interest points along this trade off curve. It's not just that having the smartest model makes your experience the best. The smartest model often tends to be a significant amount slower than other models on the market. And so we felt that there was like an opportunity for us to create like a faster top level agent

• 9:40Speaker 01

  that couldn't do as complex of coding tasks, but could do these like targeted edits. And when we started to play around with these small, fast models, we realized that, hey, actually the inference costs are significantly lower. And that got us thinking, like going back to folks like my dad, right? Like he's just doing this stuff on the side. He doesn't want to spend hundreds of dollars per month to create these kind of like simple games. We're like, hmm, maybe there's like a model here. I think it started as a joke. Someone was like, we should just do ads and see how that works. And everyone was like, nah, that'll never work. But then it just kind of kept coming back up and at one point we're like, all right, let's just try it and see how it works. And we launched it and it's been growing very quickly since then.

• 10:17Martin Casado?

  Can I dig philosophically into this just a little bit? So I had a conversation with somebody that works on Cloud Code, which is a very successful CLI tool. And this person is like, you know, what we've done over time is we've literally just removed, you know, stuff between the user and the model. Like that's it. Like that's like, kind of like the way that we improve things.

• 10:35Speaker 01

  Yeah.

• 10:35Martin Casado?

  Are we just like do less?

• 10:37Speaker 01

  Yep.

• 10:37Martin Casado?

  And let the model do more. And so I guess that makes, you know, s- It sounds kind of intellectually or intuitively interesting.

• 10:47Speaker 01

  Yeah.

• 10:48Martin Casado?

  It kind of makes sense, but on the other hand, it seems expensive.

• 10:53Martin Casado?

  You're like, here is this state of the art model that costs a billion dollars to train. And like, now it's just the user and the model. And so it's almost like that statement is almost contrary to an advertisement based model. Or like what you're talking about, like, you know, like a fast model or smaller models.

• 11:09Speaker 01

  Yeah.

• 11:09Martin Casado?

  Like, are we seeing two parallel paths in the industry?

• 11:14Speaker 01

  I, so there's definitely, you can, there's definitely different like working styles, right? Like depending on the task or maybe depending on the person, you talk to people using coding agents. And some of them are like, I just want to write a paragraph long prompt and then have, have the agent go figure it out. I want to come back to something that's like mostly working. And then there's other people who say like, actually, I don't want to do that because half the time I myself don't have a clear idea of what I want yet. The creative process is sort of one where you kind of like figure out what, what the, the software looks like as, as you go along. And sometimes it's the same person saying both things, right?

• 11:56Speaker 01

  Like when I go, there's some features where it's like implement billing where I'm like, okay, I know exactly what, what protocols we need to support. And the Stripe integration, I know what feedback loops we need to hit. Then it's like, okay, big prompt, agent, go at it. But then there's other, other types of development where it's like, you know, I want to build a new, a brand new feature. We, we just shipped this code review panel in our editor extension. And that was a, a kind of like situation where I was like, I don't actually know what this review experience should look like. Because it's not me reviewing other people's code, it's me reviewing agents code, which is like a new workflow. And for that, I kind of did want like a more interactive back and forth interaction between me and the agent.

• 12:35Speaker 01

  So I don't think it's necessarily like, these, these two things don't have to be completely separate products, but they are distinct working modalities.

• 12:46Martin Casado?

  Interesting. That's a great way to put it. How do you think about the difference between like using somebody else's model? Like one of the Sotolabs versus?

• 12:55Speaker 01

  Yeah.

• 12:56Martin Casado?

  Building your own model versus, you know, using an open source model like,

• 13:02Speaker 01

  Yeah.

• 13:02Martin Casado?

  How does that fit in your philosophy?

• 13:04Speaker 01

  Yeah, so I would say our philosophy is is not model centric. It's more agent centric. So we view the model as an implementation detail.

• 13:12Martin Casado?

  Yeah, I don't know what that means.

• 13:13Speaker 01

  OK, so let me explain. So like when you're interacting with an agent.

• 13:17Martin Casado?

  Yeah.

• 13:18Speaker 01

  At the end of the day, you care about how that agent is going to respond to your inputs. You know what tools it's going to use, what sort of trajectories it's going to take, what sort of thinking it does. A lot of that goes back to the model, but it's not solely dependent on the model. There's a lot of other things that can influence how an agent behaves. There's the system prompt, there's a set of tools that you give it. There's a tooling environment, there's a tool descriptions. There's the sort of instructions that you give it for connecting to feedback loops. And with the same model, with wildly different like tool descriptions and system prompts, you actually get like, you know, completely different behaviors out of that model.

• 13:52Martin Casado?

  Is that true in both directions? Like with the same prompts and two completely different models, we'll get different behaviors?

• 13:56Speaker 01

  Oh, for sure, for sure. It's like if you have like a an agent harness, like a set of tool descriptions, and you swap out the model, then there's no guarantee that that thing is going to work well with the model that you swapped in.

• 14:07Martin Casado?

  Yeah.

• 14:08Speaker 01

  And so what we view as like the kind of atomic composable unit is not the model. It's it's this thing called the agent, which is essentially this contract of like user puts text in and gets certain behaviors out. And that agent is really a product of both the model plus all these other things that I just listed. And so when when it comes to like figuring out what models we want to use, it's not so much like, hey, we want to use like the latest quote unquote frontier model from XYZ lab. It's really about, hey, what behavior do we want the agent to take? Or in some cases, the sub agent? And how do we find the right model that enables that agent to do its job?

• 14:52Martin Casado?

  This is so hard to me. Like this is the first time in computer science I can think of where we've actually abdicated like correctness in logic to a. Like in the past, it was a resource, right?

• 15:01Speaker 01

  Yeah.

• 15:02Martin Casado?

  Like, whatever, it's not logic. It's like, okay, so maybe the performance is different. Maybe the availability is different, but like, whatever I put in, I'm going to get back out, whether it's a database or compute or whatever. Like, these are like, you know, but now we're like, figure out this problem for me, right? So you're kind of abdicating like, you know, core logic and correctness.

• 15:20Speaker 03

  Your unit test comes back with works 45% of the cases.

• 15:23Speaker 01

  Yeah. The non-determinism is something that people struggle with a lot. But I, so for me, I actually do think, you know, like historically, pre-AI, like when you think about computer systems, the, the basic unit of composability is like the function call in, in programming, right? So it's like, when you think about your system, it's like this function calls out to these other functions and those other functions delegate to these other functions. I do think there, there's still an analog to that in the agent world. Like the agent is really the analog of the function, but just updated or generalized to AI.

• 15:54Martin Casado?

  Can I just push on this? Cause I, I mean, listen, call me a traditionalist.

• 15:58Speaker 01

  Yeah.

• 15:59Martin Casado?

  But for, for me, like, like computer infrastructure is computer network and storage, right? And like databases, and these are resources that are abstracted. They're like, like, so give me storage, give me network.

• 16:09Speaker 01

  Yep.

• 16:10Martin Casado?

  But like, the semantics, like what actually happens, I write, right? That's like my code. Right here, we're like, figure it out for me. It's like we're abdicating actual like, like logic and correctness. It just feels like in a way, like a little bit, you know, like in your case, for example, if you pick up, you know, let's say you're using model V 2.1 and then you go to model V 2.2, like you're gonna have wildly different answers, right? It's almost like a new instruction set or something.

• 16:38Speaker 01

  Yeah, you, you might have different answers, but I think if, if you construct the agent right, you're not, they're not going to be wildly different. So like, for, for instance, like we have a sub-agent that's designed to search for things, like uncover relevant context. And, you know, it's, it is a bit of a dice roll every time, right? Like it, it takes a slightly different trajectory, it might search for different things. But it's to the point now where if I want to find something in the code base, I have like 99% confidence that this thing will eventually be able to kind of like stochastically iterate to the right answer. And so in, in that, uh, in, in that way of thinking, it's like, yeah, how it gets there might vary, but if I wanted to do a specific thing, it's reliable enough that I can, I can invoke it.

• 17:22Martin Casado?

  It feels like there's kind of a backlash right now in the industry to evals. Like, do you view like this as an eval problem or like a runtime system problem?

• 17:30Speaker 01

  Yeah, so you know, my take on evals is evals are definitely effective as a sort of like unit test or a smoke test.

• 17:40Martin Casado?

  Yeah.

• 17:40Speaker 01

  Because if you push a change your agent and it breaks something, you want to know, right? Like, if there's like an important workflow that you're like, hey, this should work reliably well, because if this doesn't work, then probably a lot of other things break. Then that's a great instance where you want you want an eval that will alert you when it goes from green to red. I think where it gets hairier is treating evals as a kind of like optimization target. Because any eval set, like, what are you trying to capture? If you're building an end user product, at the end of the day, what you care about is the product experience. And so you construct the eval set to kind of proxy the vibes of the user using the product. And by definition, that means your eval set is always like lagging a little bit from

• 18:23Speaker 01

  the frontier, because it takes time to like distill what is a good product experience into a set of evals. And we've had multiple times in our past where we've picked a number, just to take an example, like with, you know, back in the kind of like code completion days of, you know, 2023 or whatnot, we had a coding tool that would do coding autocomplete and the kind of like banner top line metric there was completion acceptance rate. You know, like, given that I suggest this change to the user, what is the likelihood they're going to accept? That seems like, you know, bulletproof, right? But actually, like, I think in building that, we ended up over optimizing that to a certain extent, because there's, and like, any metric you choose, there's going to be a way to game it.

• 19:06Martin Casado?

  Well, I mean, even in this one, like, okay, so like, the developer accepts it, but do they end up committing it?

• 19:12Speaker 01

  Yeah.

• 19:12Martin Casado?

  Oh, they committed it, but it, you know, like, whatever, did like- Did it pass code?

• 19:15Speaker 01

  Yeah, exactly.

• 19:16Martin Casado?

  Did the PR get-

• 19:17Speaker 01

  Exactly. There's like a subtle bug introduced or whatnot, or, yeah.

• 19:22Martin Casado?

  You know, did it get merged into main? Like, I mean, it just feels like, you know.

• 19:24Speaker 01

  Yeah, yeah.

• 19:25Martin Casado?

  You know, like, this is kind of an adjacent topic, but something that Guido and I discuss a lot is to what extent the market is Pareto efficient on the Pareto frontier. Like, if you can trade off, like, let's say, performance for cost or intelligence for costs, like- Will the market kind of adopt that uniformly? Or does it just optimize only for speed or only for correctness?

• 19:52Speaker 01

  Like, yeah.

• 19:52Martin Casado?

  Being on the front lines, we would love.

• 19:56Martin Casado?

  You know, your sense on this.

• 19:57Speaker 01

  Here's a simple question.

• 19:58Martin Casado?

  Yeah. We ask this question a lot and nobody seems to know.

• 20:00Speaker 01

  Like, is the question here, like, what matters more? Speed or intelligence?

• 20:04Martin Casado?

  It's whether the parrot frontier is what matters? Or if it's kind of, there's points on the parrot frontier that matter, right? So you can imagine.

• 20:10Speaker 01

  Okay, yeah.

• 20:11Martin Casado?

  So traditional pricing psychology is you're the expensive one or you're the cheap one.

• 20:15Speaker 01

  Yeah.

• 20:16Martin Casado?

  Right?

• 20:16Speaker 01

  Yeah.

• 20:16Martin Casado?

  And everything in the middle is called the value gap, which people don't use, right?

• 20:20Speaker 01

  Yep.

• 20:20Martin Casado?

  And so originally, we were like, oh, the parrot, like, that happens here. So either you buy the most expensive one or you buy the cheapest one.

• 20:26Speaker 01

  Yeah, yeah.

• 20:27Martin Casado?

  But actually, as we kind of look in the market, it actually feels like most of the frontier is pretty full. Like, developers are pretty sophisticated, like, you know, different, you know, there's different cost sensitivities, different price sensitivities, so.

• 20:37Speaker 01

  Yeah. So, you know, it's funny that you mentioned this, like, you know, the cheap option versus like the premium option. It just so happens that AMP has two top level agents. There's a smart agent and there's a fast agent.

• 20:49Martin Casado?

  That's interesting.

• 20:50Speaker 01

  And the fast agent is the one that's ad-supported, like, that we can offer for free. And the smart agent is the one where we're like, okay, we're not, uh, we will always only do usage-based pricing for that because we want to keep that at the frontier of smartness. But that being said, like, I don't know, like, maybe there's like a third point in there that could make sense. It really just comes out of the vibes at the end of the day, like, as we use this more heavily and see the usage patterns emerge.

• 21:13Martin Casado?

  The mid-agent.

• 21:15Speaker 01

  Yeah, the mid-agent. Like, I honestly, yeah.

• 21:21Speaker 01

  Well, if you put it that way, and they're like, oh, like, yeah, the galaxy brain idea is you either want, you know, smart or fat.

• 21:29Martin Casado?

  Cool. So I would love, I mean, if you're open to it, I'd love to dig into a bit on kind of your view on open source models.

• 21:36Speaker 01

  Yeah.

• 21:36Martin Casado?

  Do you use them?

• 21:38Speaker 01

  Yes.

• 21:38Martin Casado?

  You know, do you think that they are an important part of the ecosystem?

• 21:43Speaker 01

  Yeah, so we do use a variety of open source models. You know, we use both closed source and open source models quite heavily. But the open source ones, I think, are. Are becoming a bigger theme now for a couple of reasons. One is with an open source or open weight model, you can post-train them, right? Which means like if you have a domain-specific task, like AMP has a growing number of subagents that are specialized for a specific task like contact retrieval or like extra reasoning, library fetching. Those are more constrained tasks where you don't necessarily need like frontier general intelligence.

• 22:26Speaker 01

  If anything, you want faster, right? And so the benefit of having open weight models is you can look at the thing that you're trying to optimize for, like what that subagent needs, and post-train the model to accomplish that more effectively. And the other element of open weight models that's very appealing is just the pricing aspect a bit. Like, there's now more and more like effective open weight models that are emerging on the scene that are actually quite robust at agentic tool use. You know, the landscape has changed immensely since like June of this year. We've gone from like, you know, there was really only one really good agentic tool use model to now there's like- Could you name that?

• 23:11Speaker 01

  I mean, it'd be great to actually, I mean, I open- Yeah, I mean, like, so, you know, originally there was Claude, right? Like Sonnet or Opus. That was the first agentic tool use model. And that sort of, you know, ushered in in the current agent wave. But now, you know, there's GPT-5, there's Kimi K2, there's Quen3 Coder, GLM. Are these, are these open source models like on par or pretty close? It depends on the workload. So I would say in our evaluations for kind of like the top-level smart coding agent driver,

• 23:50Speaker 01

  we still tend to prefer Sonnet or GPT-5. But for kind of like quick targeted edits or specific subagents, I think more and more we're preferring smaller models because they have better latency characteristics and because the complexity of the task isn't high, like you reach a ceiling. It's like- Once you reach a certain level of quality, there's diminishing returns and then you start optimizing for latency because that gets you more you know, interactivity.

• 24:28Martin Casado?

  What's the smallest models you can use for an effective agent?

• 24:33Speaker 01

  I mean, for, for an agent right now, it's probably still fairly large, like talking to probably like hundreds of billions of, of parameters. For, for kind of like a top level agent, but for like search agents, you could go smaller than that. And then we also have a model that does kind of like edit suggestions. So, you know, for those times where you still have to go into the code and manually edit stuff, this thing suggests the next edit that you'll make. And for that, we use a very small model, like, you know, single digit billions of parameters.

• 25:07Martin Casado?

  So do you train your own models?

• 25:09Speaker 01

  Yeah, we, we do.

• 25:10Martin Casado?

  Oh, wow.

• 25:10Speaker 01

  But I would say we, we don't train them from scratch.

• 25:14Martin Casado?

  No, no pre-training, it's mostly.

• 25:15Speaker 01

  No pre-training.

• 25:17Martin Casado?

  And I'd be dumb.

• 25:17Speaker 01

  Yeah. At this point, it's just like, it would be fiscally irresponsible, right?

• 25:23Martin Casado?

  Probably pointless.

• 25:24Speaker 01

  Yeah.

• 25:25Martin Casado?

  Are these for special use, special use cases? Like a lot of the products that we work with, let's say just outside of coding, just like a lot of products that we work with, you know, it just, I mean, here's this general view. Pre-training is done.

• 25:37Speaker 01

  Yeah.

• 25:38Martin Casado?

  Right? Paying people to create data, we've hit economic equilibrium, right? It's like.

• 25:42Speaker 01

  Yeah.

• 25:42Martin Casado?

  You can keep, you can keep paying people, but like, you know, we're hitting diminishing returns there because you need kind of more expensive people and like,

• 25:49Speaker 01

  Yes.

• 25:49Martin Casado?

  need 10 times more data. And so at some point you hit equilibrium.

• 25:52Speaker 01

  Yep.

• 25:52Martin Casado?

  But, you know, like, there's a lot of product data out there and there's a lot of users out there and like, you know, the, the, the, the, the solution domain is enormous. And so you can start building smaller models and, you know, so it's like, you know, like A, is that correct? And B, you know, like the models that you train, do they kind of fit in that general pattern of? In specific smaller models?

• 26:12Speaker 01

  I think that's spot on, actually. It's like the very large generalist models were great, and they still are great for experimentation, because it's almost like you train this thing on all sorts of data, and it's almost like a discovery process, where the training team themselves don't quite know what behaviors might emerge. But once you map those to specific workload, specific agents that you want to build, then you have a much clearer target. And it's widely known that a lot of the model labs do this now behind the scenes. Like, they might expose an API that's like one model, but behind the scenes, they're routing to smaller models. And you can also do that at the application layer. Like, if you have an agent architecture like we do,

• 26:57Speaker 01

  there's all sorts of specialized tasks. Like, we've broken down the process of software creation to various tasks, like context fetching, or debugging, or things like that. And once you have a specialized agent for each, then you take a look at what the agent needs to succeed, and you try to get the model as small as possible while still maintaining the requisite quality bar.

• 27:20Speaker 03

  So it's not just a Pareto frontier of quality versus cost?

• 27:25Martin Casado?

  There's also multiple graphs.

• 27:29Speaker 01

  Yeah. It's basically per agent. Like, every agent maps to a workflow.

• 27:36Speaker 03

  Yeah.

• 27:37Speaker 01

  It's emulating some workflow.

• 27:38Speaker 03

  Yeah.

• 27:40Speaker 01

  That maybe approximately maps to something that a human used to do, maybe it doesn't. But it's a subroutine. This is why I go back to the function analogy.

• 27:49Speaker 03

  Yeah.

• 27:51Speaker 01

  And so for any given agent...

• 27:53Martin Casado?

  It's a subroutine where you abdicate the logic.

• 27:57Speaker 01

  It's a stochastic subroutine.

• 28:00Speaker 03

  Even weirder. I mean, now we have parameters like how much reasoning do you want? It's a tunable sort of thing. It's like, how powerful do you want to make this? What's your budget?

• 28:07Speaker 01

  But there's like a mini Pareto frontier for each of these tasks, right? And then the optimal point along that frontier is different for each task.

• 28:14Martin Casado?

  So I actually want to dig into, you know, like the open source models, the implications. I mean, I know that you've got opinions on that. We've got opinions. It's an interesting topic. But before we do that, so in 10 years, are we using an IDE or are we using agents on a CLI? What happens to software engineering in 10 years?

• 28:35Speaker 01

  Uh, okay, so.

• 28:36Martin Casado?

  I mean, listen, you're like one of the people that has stared this in the face for quite a while.

• 28:40Speaker 01

  I do have a take on this.

• 28:41Martin Casado?

  You're probably literally like the world expert on this question. I'm serious.

• 28:44Speaker 01

  Yeah, so here's my take, like, um, I don't think it, it's not going to be an idea that looks like any idea that exists today. And it's not going to be like a terminal that looks like any terminal that exists today. My view is that, um, and I don't think this is like a particularly unique view. It's just that, you know, the effect of AI on every single knowledge domain, including coding, is that it's just going to enable the human to level up, right? So the job that you do already, like that, like my job has changed so much in the past year. Like, I think about all the kind of like toilsome, like line by line editing that I did like a year ago today. It seems like completely foreign. I like honestly don't think I could go back at this point.

• 29:26Speaker 01

  Now, when I'm doing stuff, it's more at the level of like telling the agent to make the specific edits or execute like a specific plan. And I'm really playing the role more of like an orchestrator. Now and then you still have to like pop in and make some manual edits when it gets stuck. But increasingly, like, I would say like by sheer like lines of code volume, probably more than 90% of the code that I write these days is through AMP.

• 29:52Speaker 01

  And I think it's only going to get higher and higher level over time. And so when we think about like the the interface that a human will interact with primarily, I think the future looks like, uh, something that allows you to orchestrate the the job of multiple agents. And crucially, uh, something that allows you as the human to understand the essentials of what these agents are outputting. And I actually think that's probably the limiting bottleneck today.

• 30:21Martin Casado?

  Of course, comprehension is like the human comprehension just on like,

• 30:24Speaker 01

  Yeah.

• 30:25Martin Casado?

  Does it have to like my understanding of like?

• 30:27Speaker 01

  Exactly.

• 30:27Martin Casado?

  The problem needs even at like a business size.

• 30:30Speaker 01

  Yes.

• 30:30Martin Casado?

  Because there are fundamental trade offs in the system.

• 30:32Speaker 01

  Yes. Yes, but I think.

• 30:34Martin Casado?

  You can't wish those away.

• 30:35Speaker 01

  You can't wish them away and the human is the bottleneck, but I think the human is still essential and will still remain essential 10 years from now in software engineering. Because it's fundamentally a creative process.

• 30:43Martin Casado?

  No, no, that's what I mean. I'm sorry. I just want to make sure we're talking about the same thing.

• 30:46Speaker 01

  Oh, yeah.

• 30:47Martin Casado?

  Like a human has in their head of what they want to accomplish.

• 30:49Speaker 01

  Yes.

• 30:49Martin Casado?

  And only the human has that in their head.

• 30:51Speaker 01

  Yeah, yeah.

• 30:51Martin Casado?

  And so like often that's going to require. Choosing a point between two trade offs.

• 30:56Speaker 01

  Yes.

• 30:57Martin Casado?

  Like, whatever that is.

• 30:58Speaker 01

  Yes.

• 30:58Martin Casado?

  So like, there has to be some way that this articulation happens.

• 31:02Speaker 01

  Yes. And, and when you, when you talk to like practitioners today, like a lot of them are very, uh, it's like bittersweet. Cause like, on the one hand, it's like, oh my God, like agents, they're writing all this code and they're actually pretty good at it. On the other hand, it's like, oh, I'm spending like 90% of my time, like essentially doing code review now. Uh, which is, you know, there's like the one, one in 100 dev, uh, that you talk to that says like, I really love code review. The rest of us are like, oh man, it's like such a drag.

• 31:30Martin Casado?

  Yeah.

• 31:31Speaker 03

  While becoming middle managers of, of coding.

• 31:33Speaker 01

  Yeah, yeah, exactly. I mean, you talk to some devs and they're like, you know, I've never been more productive, but, uh, coding isn't fun anymore. And so, you know, that's one of the things that we're trying to solve for.

• 31:41Speaker 03

  The beauty, the elegance is gone. It's now all looking at implementations requirement.

• 31:46Speaker 01

  It's, yeah, it's, it's, it's, it's that, but also it's just like the task of like reviewing code, I think is, is a slog. And like classical code review interfaces, um, are, are just not that good. Like, I think they were never that good.

• 31:58Speaker 03

  Yeah.

• 31:58Speaker 01

  But it wasn't like blindingly obvious because the, the rate at which like lines of code were shipping was.

• 32:04Martin Casado?

  It's remarkable.

• 32:05Speaker 03

  It's a super simple example, right? Today, if I review code from pretty much any coding agent out there, typically it's just like file by file by file by file.

• 32:13Speaker 01

  Yeah, yeah.

• 32:14Speaker 03

  Like grouping this by task or something like that, or explaining it.

• 32:17Speaker 01

  Yeah.

• 32:17Speaker 03

  A couple of arrows with little bubbles.

• 32:18Speaker 01

  Yeah, exactly. You are literally like.

• 32:21Speaker 03

  There's so much low hanging fruit.

• 32:23Speaker 01

  Yes.

• 32:23Martin Casado?

  It seems like.

• 32:24Speaker 01

  So we, we, we launched a review panel in our editor extension last week. Uh, that's, it doesn't get all the way there, but I think it's the first step. And it's already like, it's way better than like an existing like code host review tool. Uh, like it's, it's mind boggling to me that like, we live in an age where like, you can literally have a robot, like, you know, one shot, a very large change. And then you pop over to like, you know, GitHub PRs and you're clicking, you know, expand hunk, expand hug, expand hug. No code intelligence, can't edit.

• 32:53Speaker 03

  No diagram.

• 32:53Speaker 01

  Yeah, yeah. Like, it just feels like, you know, it's like, we have, we have like a Ferrari engine. Uh, but then part of our workflow still requires like strapping it to this like horse and buggy style, uh, thing. So anyways.

• 33:07Martin Casado?

  It's like a kind of microchip and then I give you an oscilloscope.

• 33:12Speaker 01

  Yeah, exactly, exactly.

• 33:14Martin Casado?

  All right, so listen, we're moving on on time here, so I actually want to get more into the policy side. Because I do think like, listen, a lot of the way this goes is the way the model goes.

• 33:22Speaker 01

  Yep.

• 33:23Martin Casado?

  The open source ecosystem, we see it all over the place, not even talking about Source Draft. But I would say if a company walks in now, that's a product company, that's decided that they need to post train their own models.

• 33:36Speaker 01

  Yep.

• 33:36Martin Casado?

  It's going to be on an open source model.

• 33:38Speaker 01

  Yep.

• 33:38Martin Casado?

  And more and more, these are Chinese models.

• 33:41Speaker 01

  Yep.

• 33:42Martin Casado?

  And so you mentioned that you do use open source models and Chinese models. So like, how do you think about that as far as like, A, maybe just like, the implications of the dependency? And then B, what does this mean like?

• 33:54Speaker 01

  Yeah.

• 33:54Martin Casado?

  Maybe more holistically with the United States and the ecosystem?

• 33:57Speaker 01

  Yeah, so like, first off, like, in terms of our production setup, like, every model that we hit is hosted on American servers. So from like, an information and security point of view, I think this is like, best practice across the industry. It's like, you don't hit models that are hosted in China or... Yeah, so like, from that part, it's fine. I would say, though, if you take a step back, it is fairly concerning. Because my view is that as the model landscape evolves, you're going to start to see a flattening in terms of model capabilities, right? Like, there's going to be a healthy competition at the model layer, and there's going to be a number of options for choosing a model at, you know, a given point in the Pareto Frontier.

• 34:42Speaker 01

  And with that flattening, there's a strong incentive for application builders to, you know, at a given capability level, use the one that's open for the reasons stated before. And because the most capable open weight models right now are of Chinese origin, it essentially means that, like, application builders around the world are choosing to post train on top of these models. And so if the US open weight ecosystem doesn't catch up, we're kind of in danger of, you know, the world migrating to a world where most systems are heavily dependent on models of Chinese origin.

• 35:24Martin Casado?

  Do we have competitive US open source models right now?

• 35:28Speaker 01

  Um, I mean.

• 35:31Martin Casado?

  We have a competitive non-Chinese. If you look at Europe.

• 35:34Speaker 01

  You know, we've sampled like a good portion of the model landscape. Because again, like we have all these subagents and agents who want to find the best ones for the job. And frankly, like the ones that we find most effective at agentic workloads, they're almost all. I would say they are all of Chinese origin right now. And that's not to say that like there haven't been like good efforts by American companies. It's just that, you know, when you plop those into like an agentic application. You know, the tool use isn't quite robust enough. It's not quite there yet.

• 36:09Martin Casado?

  Do you think this is a

• 36:12Martin Casado?

  a result of of policy or funding or like?

• 36:18Speaker 01

  I think probably all of the above.

• 36:20Martin Casado?

  I mean, the easy answer is is like, yes, you know, it's a regulatory thing, this and that. I just don't know how true that is. I mean, it just turns out there's very sophisticated like.

• 36:29Speaker 01

  You know, so it is interesting. Like, it's like, you know, the AI revolution was basically like born and created in the West. Right? And I think.

• 36:37Martin Casado?

  Down the street.

• 36:38Speaker 01

  Yeah, down the street. And the US still holds a lead in basically like every part of the stack. You know, whether it's like chips or, you know, frontier intelligence. Like basically every place except Openweight Models.

• 36:53Martin Casado?

  And robotics, yeah.

• 36:55Speaker 01

  I guess like that's the manufacturing aspect of it.

• 36:58Martin Casado?

  Yeah, yeah.

• 37:00Speaker 01

  But, you know, from where I stand, it's like, you know, if you go back to the quote unquote, early days of the AI revolution back to, you know, 2022 or so. I feel like the narrative that was told that was like the dominant narrative was this one of like AGI at that point. Where it was kind of like this, it's like amazing new technology. It feels like magic, right? Like never experienced anything like this before in my life.

• 37:25Speaker 01

  And then the narrative that was spun was like, hey, AGI is nine. What does AGI mean? Well, either one, it's like utopia. All our problems are solved. This thing will just, you know, run our lives for us. Or it's going to kill us all.

• 37:37Martin Casado?

  Total annihilation.

• 37:39Speaker 01

  Like Terminator.

• 37:41Speaker 01

  A sly outcome.

• 37:42Speaker 03

  Skynet, yeah.

• 37:43Martin Casado?

  I love, I love the Balaji, the Balaji view of this. He's like, there's this very Abrahamic view of it. It's either like God or the devil, right? And then he's like, I'm Hindu. He's like, we've got a bunch of gods. Some are capricious, some are nice.

• 37:59Speaker 01

  Yeah.

• 38:00Martin Casado?

  And I've chosen the Hindu view of this.

• 38:03Speaker 01

  Yeah, arguably that view of the model landscape was, was the right one.

• 38:06Martin Casado?

  Right, yeah, yeah.

• 38:07Speaker 01

  In, in retrospect, and, and I think at the time, like, people like, like using these models directly, kind of realized this, right? It's like, you use the models, um, they can, um, they can emulate intelligence of, of a certain kind, but it's like mostly pattern matching. And there's just like absolutely no danger that like this thing's gonna, you know, acquire a mind of its own and like try to reach out the computer and kill you.

• 38:31Speaker 03

  If you use this every day, right? This idea that this thing could take over the world.

• 38:34Speaker 01

  Yeah.

• 38:35Speaker 03

  It's funny.

• 38:35Speaker 01

  Exactly.

• 38:36Speaker 03

  That's right.

• 38:37Speaker 01

  So like, and now if you talk to practitioners, uh, like anyone who's building it, and increasingly anyone who's using it, right? Cause like now, you know, ChatGPT has been out for like threesome years and everyone and their mom has used it. Uh, like people kind of understand what the, the limitations are. So like that narrative, I think is, uh, is largely been dispelled within, uh, within our circles. But I think that it's, it's sort of like taken on a life of its own in, in other circles. And it's made its way to some of the halls of policymaking in the US.

• 39:05Speaker 03

  I mean, is part of the problem that not every policymaker is using LM's day to day? To put it, to put it carefully.

• 39:13Speaker 01

  Yeah, I don't, I don't, you know, there's the old adage of like, you know, do you blame it on, uh, you know, ignorance or, or malice?

• 39:21Speaker 03

  Or opportunism or?

• 39:22Speaker 01

  Yeah, like I, I honestly don't know. Like it's, it's a black box, but it, it is like, uh, it is clearly like nonsensical. And I think very much in the, the, the national interests to be still telling this story because it, uh, one, it, it, it leads to kind of like overemphasis on, on like the model as the end all be all of, of AI. Where in reality, it's like pushing this, pushing the models into like all these different application areas where like the rubber meets the road and things become useful.

• 39:50Speaker 03

  Building systems, yeah.

• 39:51Speaker 01

  Yeah, um, but then also like when you're, when you think about making laws and regulations for this sort of stuff, if, you know, you've been sold on this sort of like Terminator style narrative. That's going to put you in a very different mindset with respect to how much risk tolerance you're willing to take on, how much innovation you're going to allow in the ecosystem, and your tolerance for open sourcing model weights.

• 40:13Martin Casado?

  So, you know, you use a bunch of open source models. And there's a question that we actually debate quite a bit, which is, assume the policy environment exists as it is, even with like infinite funding and infinite talent, could you still actually build competitive models? Or like now are we at a place that like we're just at a disadvantage just because of like, is it, is it, is it too late to actually assume that we can do it without actually changing policy?

• 40:38Speaker 01

  Like, like build adequate open, open way models?

• 40:42Martin Casado?

  Well, let me just give you an example. I don't know why OpenAI released the open source models the way they did, but it seems like they were very, very sensitive to what data was in them. And I presume this is kind of a concern around copyright. I would just, I don't know the answer to this. I'm just, I just assume that.

• 40:58Speaker 01

  Interesting.

• 40:59Martin Casado?

  We haven't seen something come out of Meta in quite a while.

• 41:01Speaker 01

  Yeah.

• 41:02Martin Casado?

  Like, are there even any open source model? You know, so like, like, it's just very unusual for the United States not to do this. And like, the efforts that have done it have seemed to be like handicapped in one way. And so like, there's one view of the world that like, this isn't a tech problem. It isn't a money problem. We were already in the overhang of policy. Like, that's one view. And so like, I guess my, my specific question is, do you think that is the case? Or do you think we've just kind of, you know, haven't, haven't kind of gotten to it yet? And we're going to come up with open source models?

• 41:29Speaker 01

  You know, I, I honestly don't know. Like, I don't, I don't have like inside knowledge of what goes on inside a lot of these research organizations.

• 41:36Martin Casado?

  But it's just super remarkable that like, here we're the US. You know, we were the first with open source models. We had Lama 3. And like, now he's like, like, listen, you're, you're using Chinese models.

• 41:45Speaker 01

  Yeah.

• 41:45Martin Casado?

  Like, where are the US models? And then they've, you know, and they, why aren't they there? And I guess my best guess, I mean, again, you know, you both can gut check me on this is like, actually, there's like, like all of the rhetoric around like developer liability, even though it didn't happen, but there was rhetoric around it, all of the policy stuff, all the copyright stuff, all the lawsuits. My guess is, is that, you know, a lot of these folks are gun shy.

• 42:09Speaker 01

  Yeah, I think that could very well be the case. And I think that the way that the regulatory landscape is evolving doesn't help at all as well. Because there was an effort earlier this year to have kind of like a federal set of standards for AI model layer regulation. But that, I think, fell apart. And so now we're kind of like slow walking, in some cases, fast walking towards this like patchwork quilt of state by state regulations.

• 42:40Speaker 03

  Some of that state regulation writes in that it applies to anybody making a model available in the particular states. And theory, you're right. It's one state, every state tries to drive policy for all the United States.

• 42:53Speaker 01

  It's very vaguely worded and it leaves a lot of room for interpretation, which is never good, I think, for.

• 43:01Speaker 03

  A lot of that hasn't been litigated either, right? So it's, it massively increases complexity. I think for a small startup to build an open weight model at this point is extremely hard. Who wants to take that risk?

• 43:12Speaker 01

  It reminds me like back in the day when we were like looking at GDPR compliance. When that was the first thing. And like I was talking with like our legal team and external counsel and trying to like read the text of that regulation and figure out like, oh, you know, is this thing, you know, technically in violation? It seems kind of high level. And the answer that I got was like, look, honestly, these are underspecified and it's really like, it's going to come down to some decision maker within, you know, that bureaucracy. And they're going to make a judgment call. And hopefully they lean towards going after, you know, the bigger fish in the pond before they come after you. So, you know.

• 43:53Speaker 03

  Paradoxically, this is the greatest case you could have ever given to the large social networking giants. So the only ones that actually could have the legal teams and the policy teams navigate this stuff. And we saw this up close as investors. We're like, as soon as these things came up, it basically entrenched the incumbents.

• 44:06Speaker 01

  Yeah.

• 44:06Speaker 03

  Who could come play. Last quick topic, if you did have like some recommendation on like how we should think about policy going forward to kind of aid in.

• 44:16Martin Casado?

  You know, open source efforts for the United States.

• 44:18Speaker 01

  Yeah.

• 44:19Martin Casado?

  What would you guide?

• 44:20Speaker 01

  Do as much as possible to ensure a dynamic and competitive AI ecosystem within the US. I mean, the best thing that we can do, I mean, we're America. Like, the best thing we can do is to take a step back and let the free market function. And so to that end, like, ensuring there's kind of like a standard, like, you know, nationwide set of regulations that's, you know, clear and, you know, like, well-specified to be going after, like, specific applications and application areas rather than, you know, like, general, you know, existential risk at the model layer. That would be good. And then, two, just ensuring that there's, like, competition at the model layer, avoiding any sort of, like, anti-competitive behavior.

• 45:04Martin Casado?

  Regulatory lock-in.

• 45:06Speaker 01

  Yeah, regulatory lock-in, that sort of thing. You know, essentially, like, you know, don't let the, like, Internet Explorer versus Netscape thing play out the way it did in, like, Internet 1.0 with, like, the AI ecosystem.

• 45:18Martin Casado?

  Yeah, we were very, very lucky that actually academia and the broad industry ended up erring on the side of openness. Let's hope this happens this time, too.

• 45:26Speaker 01

  Yeah.

• 45:29Speaker 02

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at A16Z.substack.com. Thanks again for listening, and I'll see you in the next episode.

• 45:54Speaker 02

  As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com/disclosures.