Ep 33: CTO and Co-Founder of Sourcegraph on Current Landscape and Future of Software Development, How to Make RAG Better, and Building Towards the Agentic Future

On this week’s Unsupervised Learning, Pat and I sat down with CTO and Co-Founder of Sourcegraph, Beyang Liu. Sourcegraph is a leader in the AI coding space, and recently launched AI coding assistant, Cody. Beyang shared with us his view on the current landscape of AI coding and the future of coding and software development. He also shared how Sourcegraph has tried to make RAG better, and their model eval approaches.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Coding is one of the best use cases we've seen for AI so far. There's over a million paying users for GitHub Copilot. We've seen tons of tools evolve. And today, we have one of the leaders in this space, the CTO and co-founder of SourceGraph, Beyond Liu. Beyond's built some really interesting products for SourceGraph that help developers better code using AI. We talked about a bunch of practical tips for folks building AI products, including how SourceGraph does model evaluations, the different search techniques they've used to make Retrieve Logcent's generation better, how they think about dividing the team between building features that work in AI today and planning for a future where agents and other things unlock new capabilities. And then we also hit on what AI means for software engineering and what coding will look like for the decades to come. Definitely stay tuned for Pat and my debrief. This was a really interesting conversation. And without further ado, here's Beyond. Thanks so much for coming on.

• 0:45Speaker 01

  Cool. Thanks for having me.

• 0:47Jacob Effron?

  You know, I guess to start, I'm sure no one has ever asked you this question before. So really, you know, going to innovate from the beginning here. But I feel like anytime someone talks about AI coding, you know, the question always becomes, okay, what does this mean for the future? And particularly, I think one angle of that is, you know, what does it mean for teenagers today? Or folks that are in school and thinking about what their careers look like. And so I'm curious, like, what advice you'd have for, like, you know, someone starting high school on how they should think about whether to study coding, how to study coding, and how that will impact their career, you know, when they started a decade later.

• 1:19Speaker 01

  Yeah, that's a great question. It's sort of like the classic, you know, what should I do with my life?

• 1:22Jacob Effron?

  And you can tell Pat and I the answer to that.

• 1:24Speaker 01

  Yeah, if you know, then let us know. No, it's a good question. I would say, you know, pre-AI, you know, the standard answer to that was like, you know, learn to code because software is going to eat the world and all that. Now, people aren't so sure. Now AI eating software. Yeah, exactly, exactly. Like, you know, the programming language of the future is going to be English or whatever your native language is. And I think that's very much true. You know, like, a lot of code will be generated for you either through, like, an English prompt or just, you know, some sort of automated system that takes a look at the state of the world and suggests a change. Personally, I think there's never been a better time to get into software and coding. So my recommendation to, you know, if I were to talk to my younger self living now in the moment,

• 2:09Speaker 01

  I would say, yeah, dive into software and learn to code. Because if anything, I think... What AI is going to do for software is it's just going to expand the horizons, expand the ability to generate and create really high-quality software. And there's still going to be a lot of leverage you get from kind of like technical thinking. Like, when I think about the essence of software creation, there's really, you know, the high end and then the low end that matters. The high end is connecting what you're building to, you know, some end-user application, delivering value, sort of like the product side of things. And then the lower end is really, okay, what are the underlying pieces I'm putting together? You know, it's really about like the bits and bytes of computing.

• 2:54Speaker 01

  You know, what sort of manipulations can you have at almost like the atomic or chip level in terms of, you know, the underlying data structures and algorithms that you're working with? And so the essence is really just, you know, connecting those two. And I think right now, there's a lot of middleware that has been built because it's a very complex task to connect the end-user application with the underlying, you know, abstractions and things like that. I think a lot of the middleware needs to be rethought, but as far as like the entire picture of like, hey, how do I build something that's actually useful utilizing the underlying technologies that we have today? That's never been more important and will only get increasingly important over time.

• 3:34Speaker 02

  We'd love to ask you a bunch of questions about AI at SourceGraph. And maybe before we dive in, can you tell us for the listeners kind of about some of the AI products at SourceGraph and, you know, Cody and how you guys are using AI today?

• 3:46Speaker 01

  Yeah, totally. So SourceGraph has probably like two main end-user products that are going to be like recognizable to developers. One is a code search engine, which is a thing that we built to enable human developers to understand code. So basically, how that works is you have a vast code base. In order to get anything done, anything built, you have to first understand what exists in that code base and how to fit your, you know, new feature into that big picture. So what do you do? You go and search through a lot of code. You kind of acquire understanding of what's going on. You click the equivalent of like hyperlinks in code a bunch of times, go to a definition, find references, sort of like, you know, walk the reference graph, the knowledge graph of code. That's how you acquire context. That's what you need to do to become effective.

• 4:29Speaker 01

  And without Sourcegraph, that can often be like a very tedious task. So that's product number one. Product number two is Cody. Cody is an AI coding assistant that does many things that you expect from an AI coding assistant. So it does inline completion, so completing your kind of thoughts as you type code in the editor. It does chat, so you can ask high-level questions about code, like how do I do this? How do I do that? Generate some code that does X, Y, Z. And then there's kind of like a menu of commands that target specific like actions and really like sources of toil, like things you'd rather not do, but you kind of have to do in the inner loop of software development. So, you know, write a doc string for this function or, you know, write a high-quality unit test for this function, things like that.

• 5:12Speaker 01

  And how those two are kind of connected is the differentiator for Cody is really,

• 5:18Speaker 01

  we do all the things that you kind of expect from an AI coding assistant with the benefit of context and awareness about your codebase. So in the same way that Sourcegraph provides context to, you know, human brains that are trying to make sense of code, it also provides context to Cody, which is sort of like an AI brain that's trying to do stuff in the code. And we found that the combination really matters because as soon as you start trying to apply AI within the context of your codebase, it becomes really important to be aware of that environment because you're no longer just taking like the median or whatever is out there in open source or on Stack Overflow, like you care about using this specific framework or this specific library or deploying into your specific production environment.

• 6:00Jacob Effron?

  Right. And so I imagine you guys obviously do a lot of retrieve along with the generation to, you know, tie whatever you're kind of querying on to the actual codebase that the customer has. I'm sure the search stuff you've done has evolved over time as, and I imagine we'll get into all that, but it's a really cool suite of products.

• 6:17Speaker 01

  Yes.

• 6:17Jacob Effron?

  I mean, I think one thing that's interesting from your purview is you must be thinking through constantly all the different things AI can do in code. And I know we see lots of flashy demos on Twitter and sometimes that's different than what's actually possible.

• 6:28Speaker 01

  Yeah.

• 6:28Jacob Effron?

  Maybe it's just level set for our listeners, like where are we today in like AI coding? Like what works consistently? What works sometimes? What doesn't work at all? How are you kind of seeing the landscape?

• 6:39Speaker 01

  Yeah, it's a great question. Like, what is hype and kind of hope for the future? And what's actually getting day-to-day usage today? I think the current state, as far as like day-to-day usage, are really these sort of like inner-loop accelerants. So when we talk about software development, there's an inner loop and there's an outer loop. The outer loop is like the whole kind of like software development lifecycle, all the way from like ideation to deploying your code in production. But in the middle there, there's this inner loop, which is really the loop that a developer goes through, hopefully many times per day of like, "I want to do X. Let me figure out how to do X in the code. Let me write the code. Let me play around with it, test it, make sure what I did is valid. And then I'll do the next X that follows that."

• 7:23Speaker 01

  So really, the things that are in heavy use day-to-day right now are these like inner-loop tools that kind of accelerate the things that you are doing as a human and driving forward as a human like very frequently. So it's things like, "I want to write this function." Especially if the function that you're trying to write has probably been written before, or it's like a commonplace function. You probably have code completion, or maybe like a code-based aware chat do that for you. Especially if it's something that's quote-unquote like "not interesting", or like, "Oh, like, I know someone has written something like this, so let me just have the AI kind of like autocomplete its way there." And there's a bunch of different manifestations of how that can work in the UI.

• 8:06Speaker 01

  Like, sometimes it's like a keyboard shortcut, sometimes it's like typing something into a text box. But those are all kind of like inner-loop accelerants. I would say the things that are more on the horizon are use cases of more and more automation. So I don't think anyone has cracked the nut yet of like how to do kind of like multi-step, like bot-driven development, where it no longer feels like you as the human are guiding it along every step of the way, but it's actually the bot driving, and then you're just kind of advising along the way. Even the tools that are more kind of like fully automated, like the Devins of the world, they show the promise, but I would say you still want a human that's kind of in the loop, watching what it's doing, checking what it's doing.

• 8:51Speaker 01

  And the lion's share of issues still requires some sort of assistance.

• 8:56Speaker 02

  And what will move us from kind of the inner-loop Iterating and you know that that type of workflow to automation is that.

• 9:06Speaker 01

  Yeah.

• 9:06Speaker 02

  Is that moving from GPT four to GPT five and getting better models? Or maybe people are just continuing to build better applications of the models themselves?

• 9:16Speaker 01

  Yeah, so the the way I like to think about it is you really want to work backwards from what you want. So the end result of what you want is a patch to the code that satisfies something you're trying to accomplish at a high level. So it's really the pull request. Okay, so like if you work backwards from that, like the final step to doing that is going to be some kind of evaluation of language model to make a code change. And that code change is going to get you to the state of like, okay, this pull request works, right? So how do you make that last kind of invocation of the LM work? Like, how do you have confidence that that's going to be correct?

• 9:53Speaker 01

  And really, the answer is, you know, what are the inputs that, you know, second to last step? And it's really the combination of the context that you're able to fetch from the code. So awareness of the structure of the surrounding code and existing, you know, other patterns. And then recollection of the sequence of actions that you took to try to get there. So it's kind of like this feedback loop where you try a bunch of different things. You observe a result. You learn from your mistakes and then that provides kind of this history of context from which an intelligent enough LM can kind of pattern match its way to, okay, this is my next attempt and the next attempt at getting the right answer is actually going to be correct. So what does that mean?

• 10:37Speaker 01

  It really comes down to you need a way to kind of try things, trial and error, you know, make changes of the code.

• 10:46Speaker 01

  And observe the results. So you need some sort of kind of like virtual execution environment. And you need really, really good context fetchers because those are going to be the limiting factors of how much juice you're you're going to be able to squeeze out of every kind of like step along the way. So, you know, if if your if your thing takes, you know. A thousand cycles to get to the right answer. Like, a thousand cycles could be very expensive and it's going to take a very long time at this point. If you're able to shorten that through kind of like better observation, better learning from your actions, better prediction of what the next action will be, or just better like context retrieval, like, I'm doing this. So like, these are the relevant pieces of code that I want to pull in automatically for that.

• 11:30Speaker 01

  That will, you know, shorten that by, you know, maybe of an order of magnitude. And that's how I think we bring this from like, oh, here's a demo that kind of works, but you're not going to use it day to day because it takes like an hour to execute or whatever, and works, you know, very kind of like infrequently, it's not like, not reliable enough to be used in production, to the point of like, being within the kind of threshold of like, yes, I can actually depend on this. It might not be perfect, but I can bring this into my workflow.

• 11:57Jacob Effron?

  It's really interesting because I think relative to other problems for agents, obviously, code has a much easier eval in some senses of like, hey, did this work or did it not? And so I think it is such a natural place, like you're saying, for the exploration of many potential steps and then learning from, did that work? But to your point, if you have to go explore a thousand, that's probably time and cost prohibitive versus, you know, being able to be much smarter and providing much more context to get a few.

• 12:20Speaker 01

  Yeah.

• 12:21Jacob Effron?

  I guess in the short term, does, you know, because I think the first code completion models, you know, that kind of took the world by storm was GitHub Copilot and Codex, which I guess was, you know, a GPT, a refined version of GPT-3.

• 12:32Speaker 01

  Yeah.

• 12:33Jacob Effron?

  Like, as these models have gotten better, so going from three to four, like, has it unlocked a new set of things for you guys? Like, were there things that were possible with three that weren't with four? And like, what happens when a new model comes out? Is there a set of things that you're like, oh, we would love to do this if it worked? And you test the new models against it? Or how do you think about both what's been unlocked by the models getting better and then what you hope is, you know, when GPT-5 comes out, what you'll be testing day one?

• 12:54Speaker 01

  Yeah, I would say like, certainly there have been a lot of new things that have been enabled with the model advancements over the past year. In fact, I went back just the other day. So in Cody, one of our like, design principles is like, freedom of choice. So in our chat view, we actually give you the choice of selecting among the different frontier models, you know, open and closed. So GPT-4, Claude 3, Mixtral. You can select, choose your own LM, basically, and try those out. You know, some are smarter, some are faster, and it really depends on your use case, you know, like what.

• 13:31Jacob Effron?

  What do most people choose?

• 13:32Speaker 01

  I would say most people probably stick with the default of Claude 3 right now, but we have a strong GPT-4 contingent. People who like, yeah, I like the the style of this better. And then Mixtral is really good for use cases where you just want something blazingly fast. It's just like super, super fast and still reasonably good at integrating the context that we provide. We have that model selector and one of the options is still GPT-3.5. So I went back and actually like selected it the other day just to see like, you know, okay, how much does the model choice matters? And it did a significantly worse job at kind of like integrating the, you know, search results and the additional code context that we pulled in.

• 14:13Speaker 01

  The model managements have made certain use cases like invoking Cody Chat a lot more reliable, like more able to more consistently get to like a wow moment through regular usage of the chat. And that's in part because the model's ability to incorporate the context given and also integrate that into like a working code example has has gotten better. Another demo that I like to give is sort of the like app from scratch demo, which is like a standard thing, but like our variant on that is like, can you can you build an app from scratch that uses the specific libraries that you want to use? So there's a demo I give where you're trying to build a simple like stock tracking app.

• 14:57Speaker 01

  And so you want to track, you know, Wells Fargo, Bank of America, JP Morgan, and you want to use React as your framework. You want to use the AlphaVantage API to pull stock tick data and you want to use recharts as as the library. And both Claude 3 and GPT-4 with the power of, you know, the SourceCraft context can essentially zero shot like an app that like pulls the stock data and displays it in kind of like a nice line chart. And that's pretty cool. That was not possible with, you know, original GPT-3.5.

• 15:32Jacob Effron?

  How do context windows affect what you guys are doing? Like, if you could send the entire code base of context to these models, like, how big of an impact is that?

• 15:42Speaker 01

  Yeah, I mean, context windows are another dimension where it's super, super helpful because the more context that we can stuff inside the window, the more potential information can be incorporated into the model's answer. So for the questions that really involve tying many different concepts together, the context windows help a lot. I will say that it's still not at the point where you can just shove the entire code base into the context window and expect that to be as good. I think a lot of the benchmarks that currently exist for long context windows are kind of like recollection-driven. Can the model point out that this thing exists somewhere in the context window?

• 16:27Speaker 01

  And as soon as you try to have it attend to more than one thing, like, can you recall if these two things exist? Or can you compose them in some way that's interesting? The performance of that use case is still much lower than the performance of, like, can you recall it or not? Long context windows alleviate some of the problems or some of the challenges that you would address with a good RAG engine or retrieval engine. But I think the best application architectures will always combine the context window with more tailored information retrieval mechanisms.

• 17:01Jacob Effron?

  Yeah, so you're not getting rid of RAG?

• 17:02Speaker 01

  No, not, yeah, no plans to do that.

• 17:05Jacob Effron?

  Makes sense. I guess, what will, like, so the day GPT-5's released, like, what will that look like at Source Graph? Like, what are you guys gonna, you know, test it against? How do you figure out, like, you know, are there new things you're thinking about maybe it could help? And how do you think about even evaluating, you know, how good it is?

• 17:20Speaker 01

  Yeah, I mean, I think the day it's released is hopefully the day we can turn it on in Cody. I think it's just a matter of adding the right model strings into the different pieces, you know, layers of the application. And then we're gonna roll that out, give users the option. We'll, of course, you know, run our own benchmarks to see if we should make any changes in the default Selection of of models. But a lot of it will come down to observing. You know how it affects product usage metrics, you know, whether we find more people using it or opting into it. And in hearing people's feedback about what it's good at. So I think it'll be a combination of like, let's get out the door, let's get in the hands of of users because.

• 18:04Speaker 01

  I think the space is evolving so fast that. One of the things we want to enable our users to do is just stay at the frontier like we're not going to be kind of like a gatekeeper or. I don't know, an arbiter of, you know, what is, what is the best model for all use cases? We'll do our job, you know. At the 80% of what we think are good defaults, but there's such a long tail of use cases that people explore. And we really want to enable people to find that out for themselves and, you know, send us that feedback so we can.

• 18:29Jacob Effron?

  I love that, rather than you guys, you know, internally do all the evals, figure out exactly what the context in which it works and doesn't like. You just get it out to your users and I'm sure.

• 18:38Speaker 01

  Yeah, exactly.

• 18:38Jacob Effron?

  With that volume, you'll, you'll soon, soon enough hear where it does and doesn't work.

• 18:42Speaker 01

  Yeah, and I would say that like what we found too is. Eval's are really, really important. They they keep you honest to a certain extent, but they're also not. You shouldn't use them. As the ultimate meter stick and we we've made model selection choices. In the past where we've actually chosen a model that. You know, is not at the the top of like a benchmarks list, but we found in like day to day use that it seems to you. Generate like better results. So there's, there's a certain like value and still like putting your finger to the wind and and. Gut checking the reported benchmarks. Because there's, there's so much nuance that is. That affects.

• 19:25Speaker 01

  End user experience that's not captured in a, you know, well defined.

• 19:30Jacob Effron?

  That's so interesting. What I like to met like, so you're not even doing your own internal different eval data set. It's like you're putting in the hands of users and and just observing. Yeah. How often they accept things or use.

• 19:39Speaker 01

  Yeah, exactly. So like, I think as far as like metrics go, the most authoritative metrics are still product metrics. Because those tell you. Like how much engagement you're getting.

• 19:51Jacob Effron?

  Harder, harder to gain.

• 19:52Speaker 01

  Related value is much harder to game, right? Because like. If a person is not accepting completions from a given model, that's strongly suggested that they're not getting value, even if that model is best in class at human eval or whatever standard benchmarks.

• 20:07Jacob Effron?

  I'm surprised more companies don't just like, you could just A B test product metrics from different models versus doing all the eval stuff yourself in-house.

• 20:15Speaker 01

  Yeah.

• 20:16Jacob Effron?

  I feel like there's these questions of, a big question people have been asking, these AI tools that are coming out, do they benefit people that are maybe earlier developers or mid-career developers or the top 10%? I think Pat and I always talk about this, like that BCG study they did where it was like the bottom half of consultants that seemed to benefit the most and they didn't go up to the top.

• 20:38Speaker 02

  Yeah, Jacob and I are benefiting a lot.

• 20:39Jacob Effron?

  Exactly. We were like, that's a little weird that we're enjoying this so much. And the rest of our colleagues were not.

• 20:44Speaker 02

  No wonder we're getting so much better.

• 20:46Jacob Effron?

  But curious if you've observed any segmentation in the types of developers that seem to benefit more or less from these tools today.

• 20:54Speaker 01

  I would say there's no crystal clear segmentation, but there are some correlations that are weak, but still, I think, significant. So one thing that we've observed is between, I would say like inline completions and chat. Those are probably typically held up as the maybe two prime examples of end-user AI use. And it seems to be that the inline completions tends to be enjoyed more by developers on the junior end or less experienced developers, at least among our user base. And the folks that get more value out of chat tend to be more like senior engineers.

• 21:39Speaker 01

  Now, that's not to say that there isn't a lot of cross-cutting, right? There's tons of junior engineers that get a lot of value out of using the chat as well. But I would say like,

• 21:49Speaker 01

  There is a strong camp of like senior engineers who are like, I hate the completions because it's not smart enough, enough of the time and sometimes I have to rework it. And I know exactly what I want to write and this is getting it wrong. We've been able to address some of that just by improving the context quality. So like that's, that's one of the kind of trigger signals for us. Like someone complaining about, say like copilot completions, then we're like, oh, hey, you know, what if, you know, those completions were higher quality. But it's still at the point where it's like, you know, I think when you're experienced, you, you develop maybe sort of an aversion to like something coming in and suggesting something that's not exactly right.

• 22:32Speaker 02

  Yeah.

• 22:32Speaker 01

  Because it kind of wastes your time. And so we hear a lot of that feedback. Like, I don't like the, the standard AI code completions because it's disruptive and it takes me out of flow and I would write better code anyway as a senior developer. But the junior developers, on the other hand, you know, coming from a point of, you know, less experience, they find it really useful because it's almost like a pedagogical tool.

• 22:53Speaker 02

  Yeah.

• 22:53Speaker 01

  You know, I want to do XYZ, show me like the kind of median way of doing it. And if you think about what an LLM is, it's really a machine for kind of like pattern matching or almost like bullshitting its way to a correct looking answer. And that's actually helpful if you're, if you're coming from, from a point of inexperience because it, it gives you like, at least like an example to, to use as a starting point.

• 23:20Speaker 02

  It's not BSing, it's like, learn from the collective human knowledge, you know?

• 23:24Speaker 01

  Yeah, exactly. Well, the analogy I like to draw is like, you know, you're back in college and, you know, what, what a really good student does is you, you understand the material from first principles, you build a world model and you go in and ace the test, right? And you come out of that not just being able to ace the test, but you have internalized the concepts you're supposed to, you know, take away. What a less good student does, but you know, one who cares about grades a lot is you cram the night before the exam, you try to like page in as much context into your like brain's working state and then come exam time, you just kind of like regurgitate and like stitch together. You go for that partial credit, so to speak. If someone's really going through

• 24:05Speaker 01

  your answers with a fine tune, they'll find non sequiturs and logical fallacies. But you know, at the service level, it seems right. Turns out like there's a lot of tasks that we do day to day as developers that are like amenable to that approach. And so if we have a machine that can do that, we should just let the machine do that.

• 24:22Jacob Effron?

  Obviously, these models are kind of like, you know, it seems like, you know, a summation of that is that they're, you know, kind of the median developer of the median, you know, like contribution of humanity. And so, you know, you're going to have, and I know you guys have, I think, been explicit about not like training your own models or fine tuning your own models to date. I think mostly have been using like models with like kind of rag on the code context, if I'm correct.

• 24:48Speaker 01

  I would say, so we're actually doing some fine tuning now.

• 24:50Jacob Effron?

  Oh, amazing.

• 24:51Speaker 01

  But I would say like our philosophy is, you know, what's going to add the most end user value and what's allow, what's going to allow us to move the needle for the user experience most quickly. And when we took a look at what levers we had to pull initially, our sense was that like fine tuning would become important, but it was better, like we always like to do the dumb thing first, right? Like, and this comes from like, I did machine learning research way back in the day. I'm not, I'm not like a AI.

• 25:22Jacob Effron?

  You worked with Daphne Gohler, right?

• 25:23Speaker 01

  Yeah, yeah. She was my advisor. I would not characterize myself as like a deep learning expert or anything like that, but like I have some research experience and one of the like lessons, painful lessons that every, you know, new researcher learns, you always want to do the fancy thing because that's sexy. You want to have the line item in your resume.

• 25:41Jacob Effron?

  Of course.

• 25:41Speaker 01

  And be like, I trained like this fancy, you know, model and, you know, improve the state of art, blah, blah, blah. But like you always go with the baseline, you know, you always do a mixture of Gaussians or even simpler, like baseline, brain dead, it's probably not going to work very well, but it's brain dead simple. I can get this out in like a day. Just establish a baseline because if you don't do that, oftentimes you spend all this time trying to do something really fancy and then. At the end of the day, you go and revisit and it performs worse than like the brain dead thing. There was a paper in Nature a couple of years back where the report was, we trained

• 26:23Speaker 01

  this large neural network to predict weather patterns. And it was like, oh, that's really cool. Like, you know, weather is really hard to predict and, you know, we applied all this fancy machine learning to try to predict, you know, how the climate was going to evolve in this particular. So like that was published in Nature, which is like, you know, the pinnacle of research journals.

• 26:41Jacob Effron?

  Good line item for the resume.

• 26:42Speaker 01

  Yeah, exactly. A year later, there was another report that was basically like, we trained a neural net that beat the previous, you know, weather predictor model. It was one node. So, like, that's our principle, right? Like, we take a look at a problem, we take a look at what we're trying to solve, and then we try to do the simplest thing possible that we think will lead to a good result for the user. And like, RAG is that, right? Like, RAG, you can iterate super rapidly, you can evolve your approaches, you can try out different, you know, context retrieval mechanisms very quickly, all without having to go through the hassle of like, gathering a data set and, you know, paying for a lot

• 27:26Speaker 01

  of flops and then like, you know, futzing with, you know, how the loss function to all that. Now, the point at which you want to fine tune, in my opinion, is where like, okay, you tried the RAG thing, you tried a bunch of different approaches, and you're like, okay, this is not really getting there. We do need something at the model layer that improves.

• 27:43Jacob Effron?

  Yeah, so what got you over the, what was it for you that made you say, all right, time to fine tune?

• 27:48Speaker 01

  It's really about learning models that replicate kind of like specific, so specifically, the opportunities we see right now are language-specific code generation. So if you look at the performance of LLMs across different languages, I think generally the best performing language is Python, first of all, because that was the language of machine learning, or is arguably still. And, you know, JavaScript and, you know, other languages that are well represented in the training set. The things, the languages that perform less well. Like Rust does not perform as well typically Ruby, probably because there's just like a large variety of DSLs in that language.

• 28:31Speaker 01

  And then of course, like the long tail of of things like, you know, MATLAB, for instance.

• 28:38Jacob Effron?

  So you basically, obviously you embedded RAG first, see where it works well and doesn't, and then where it doesn't, you can kind of fine tune.

• 28:44Speaker 01

  Exactly, exactly.

• 28:45Jacob Effron?

  You know, obviously, we're saying these models kind of revert to the median developer like, I feel like there's one version of the world where you're like, let's find all the the best code out there, the highest quality stuff and build the model or fine tune just on that like. That on your radar or something you guys think about or?

• 28:59Speaker 01

  Oh, yeah, totally. So we have a feature in Cody called Cody Ignore. It's like a file format that makes Cody ignore certain files in your code base.

• 29:09Jacob Effron?

  Yeah.

• 29:09Speaker 01

  So you're not going to use it as context to generate any sort of completion or or response. And initially, the impetus for that was there's certain like sensitive files that some of our customers didn't want, you know, going over an API. So we built that for that initially, but I think now we're starting to see more requests of like, hey, can you extend this because? Look, like not all the code in my code base is good code.

• 29:33Speaker 02

  Cody ignore Jacob's code.

• 29:35Jacob Effron?

  Yeah, exactly. I was like, I hope you don't report back to the developer team whose code is being ignored and who's.

• 29:40Speaker 01

  Yeah, yeah. So I think where where that evolves is just giving both the individual dev and engineering leaders more levers of control over what code and context gets used to generate responses and allows you to. Almost, I think this would be a great lever for engineering leaders, actually, because I think one of the challenges you have as an engineering leader is like, how do I herd the cats in our direction that actually? Improves the quality of the code, you know, while they're trying to ship features out at the door, like no one gets paid to write very nice quality code. People get paid because you had user impact, you got the feature out the door. And so I think a future lever to actually for the people who care about code quality, because it is.

• 30:24Speaker 01

  It is an organization, a matter of organizational concern.

• 30:29Speaker 01

  One lever we can provide to engineering leaders is the ability to control what gets used.

• 30:37Speaker 01

  As context for automated code generation.

• 30:41Jacob Effron?

  How has inference costs played into how you think about the product roadmap? Whether are there things that were maybe you would want to do, but might be too expensive? Or how top of mind is kind of the costs that go into serving these AI models?

• 30:55Speaker 01

  I would say it's like you can't completely ignore it because you know, if you're getting a lot of usage, then you know, especially usage of features that rely on the large models, then the costs add up. We have not prioritized over optimizing how cheap we can make things because our view is that the costs will go down over time significantly. And so I essentially want to do the bare minimum to keep costs under control for now and then take the rest of our mental energy and bandwidth and focus on like, how can we add value?

• 31:33Jacob Effron?

  Yeah. Have you done anything to rein costs in? Like limit the number of suggestions you surface or you know, use smaller models? And obviously you've used it for performance, but have you made those trade-offs on the cost side?

• 31:45Speaker 01

  We have rate limiters and there is a, we have both a free and pro tier. But I would say the rate limiters, the main thing we use that for is to counteract abuse. So it's not so much that like legitimate users are using it so much that they're exceeding the cost threshold. It's that, you know, if you have any sort of like API endpoint that's accessible to the internet in some way, and you have a free tier that gives you access to that endpoint, you will have people out there that try to exploit that. So that's the main thing that we've done.

• 32:21Jacob Effron?

  Yeah. And then have you thought about like pricing the AI products? Like, obviously, I think this is a topic everyone seems to be still figuring out, like bundle it into what you already have, you know, price it as a separate kind of tier. Like, I'm sure you guys have had a lot of discussions around that.

• 32:34Speaker 01

  So the pricing model that we settled on for Source Graph, you know, on the code search side, and that we've also kind of settled on for Cody for now is, is an active user per month model. So it's kind of like a variant on the traditional, you know, seat-based pricing model with, with a twist of like, you only pay if a user logs in in a given month and actually uses the product. Because a common complaint we hear from our customers is like, I don't want to like over allocate the number of seats that I have.

• 33:06Speaker 01

  And like, we certainly have frustrations with vendors of our own where it's like, you know, we paid for this many seats because we thought, you know, they would use it and turns out like a fraction of people actually go and log in. So we try to adjust our pricing so that's correlated with the amount of value that the customer is seeing. And then if we see low usage, then that's the incentives are nicely aligned there.

• 33:23Speaker 02

  Did the introduction of AI change the pricing at all or just something you bundled into the existing pricing tiers yet?

• 33:28Speaker 01

  Uh, I would say we thought about usage based pricing. I think that's where a lot of people's minds go. I think there is a possibility that we introduce that at some point, especially for features that we think will be more expensive, but also like very valuable. For now, I think we've gone with the active user pricing just because it's like simple in, in the end user and customer's mind.

• 33:51Speaker 02

  And is there a separate charge for Cody? There's like for code search would be one subscription and then Cody would be a second one, or is it all bundled?

• 33:58Speaker 01

  It's, uh, I mean, so if you're just, uh, if you're using the cloud product and you're just like a developer, uh, you can, you can elect to use either product. So you can, you can, uh, use Cody for free without adopting Source Graph. You can upgrade to Cody Pro without using Source Graph. Source Graph is kind of like, uh, an option you have. Like if you see that, you know, it's providing value to the AI code generation. If you just want that, you can just have that. If you also see the value of like, oh, I want the same kind of like context providers for me as a human, uh, then there's kind of like an option to, uh, buy a Source Graph instance, uh, and connect Cody to that and it'll get better. I would say that, you know, one of, one of the things that we struggled with, um, you

• 34:41Speaker 01

  know, prior to Cody was really figuring out what kind of like, uh, bottoms up, uh, adoption model for Source Graph, the code search engine, uh, looked like. And that's something that we're still iterating on. I'm, I'm hoping at some point in the future that like, maybe we can use Cody as almost like a vehicle to find a way to make that kind of like bottoms up adoption of, of the code search engine easier. But for now, our solution has just been like, Cody's awesome, people love it. All it takes to start using it is go to the VS Code Marketplace or the Jetbrains, uh, plug in an extension store and, and install it from there.

• 35:16Jacob Effron?

  That's awesome. Hopefully I don't have to pay extra to exclude all of Pat's code from our retrieval. That's in the enterprise tier. Yeah, whatever that tier is, we'll be contacting you from the red point side. One thing I'm curious about, because I feel like a lot of companies are thinking through this, is you obviously, it seems like AI is such a core part of what you guys do at Sourcegraph now. You obviously, you've been doing AI throughout, I'm sure with much more effort as the models have gotten more powerful. How have you kind of like structured the teams internally? You have like separate AI teams? Is it just the folks that were working on different parts of the product are now incorporating AI into what they're shipping? Like, what does that look like in your organization?

• 35:53Speaker 01

  I'll preface this by saying that no org chart is perfect, and we're still in the process of trying to iterate on ours. I think at a high level, so we have a team that's working at the model layer. So that's like a kind of like distinct team. And then as far as like the rest of the product engineering org, there's kind of like two major buckets. There's people working broadly speaking on code search and people broadly working on Cody in some form. I think that will come together over time because we see so many like integration and synergies between the two that they're kind of like pulling themselves together.

• 36:34Speaker 01

  There's like certain context APIs that really makes sense to be built into, you know, Sourcegraph as like a kind of like server product that would be great for Cody to have. And vice versa, you know, there's this great opportunity to improve Sourcegraph's search quality through the use of like better ranking mechanisms that are powered by AI. But broadly speaking, it's, at least for now, it's code search Cody, and then we have a model layer team.

• 37:02Jacob Effron?

  Yeah, what's the model layer team mostly focused on?

• 37:05Speaker 01

  Two things. Fine tuning models for better code generation in specific use cases and defining quantitative benchmarks.

• 37:16Jacob Effron?

  Yeah, that makes sense.

• 37:17Speaker 01

  Yeah.

• 37:17Jacob Effron?

  And then, like, how many people does it take to ship something like Cody? Like, I mean, they were an initial version. Like, how many folks did you have?

• 37:24Speaker 01

  Uh, initial version was two people and then it grew to, uh, maybe like five or six, uh, maybe like this time last year. Um, and now, I mean, I, we still kept the client team relatively small. Still probably like a gang of, it's definitely like less than 10 people that are working on, like, the Cody client stuff directly, but there's a lot of other people supporting it now in the form of, like, producing better context fetching mechanisms and things like that.

• 37:51Jacob Effron?

  So, I feel like we always ask this question. I'm waiting for, like, someone's gonna say, like, 200 people or something. It doesn't bode well for our business that you can do this with, uh, two engineers.

• 38:00Speaker 02

  There's, uh, there's better AI coding tools, so they, they scale better.

• 38:03Jacob Effron?

  Yeah, exactly.

• 38:04Speaker 02

  You, you started Sourcegraph in, in 2013 and there's been this just explosion of AI since then and kind of generative AI has taken off. And obviously, you know, you guys have moved really quickly to launch Cody and kind of be in, in the center of this movement. Was there a singular moment that made you think, oh shit, we need to go all in on this? Or, um, we see the impact that coding, or that AI is gonna have on coding? Or how did you think about kind of navigating that, um, that change?

• 38:34Speaker 01

  I would say, for, for us, the early proof was incorporating some AI signals into, uh, like the search ranking. And I was like, oh, here's some value. It, uh, is a vector of improvement for something that really matters, the core product. And, uh, we were just kind of like dipping our toes into, uh, kind of like inline code generation, um, driven by context. Cause like, one of the things that we saw early on was like, uh, with, with Copilot, for, for, for instance, um, like, that was, that was the first kind of like mainstream, very popular, uh, AI coding tool, uh, that got to market. And so we were trying that and it...

• 39:17Speaker 01

  It worked very well for what it was good at, but the places where we saw it falling short were really around places where like the generated code really needed to be more aware of the surroundings. And so we're like, huh, you know, maybe there's something to that. Like, we have mechanisms that can search the codebase for your usage examples, that can kind of walk the reference graph, and it seems like high level, there's maybe something there. You know, whether we incorporate that data at inference time, or perhaps through some like, training procedure, it was unclear to us at the moment. So we were like, experimenting with that. But then, I think, similar to a lot of companies, when ChatGPT was released, like, that was

• 39:56Speaker 01

  a big accelerant, because it, like, the whole world, it's like the winds just shifted. And so, any sort of discussion that previously would have required kind of like a lot of thinking, and, you know, should we do this, or should we not, it was kind of like, when the wind shifts, you just kind of like raise your sail and try to catch the wind. So like, that was a big moment for us. And fortunately, we had like, things in motion that we were kind of just able to like, put more wood behind, and that ended up driving us to this sort of like MVP release of Cody early last year. And then, of course, like, actually getting in the hands of our users and customers and

• 40:41Speaker 01

  seeing the feedback there, that was a big wow moment and point for us as well.

• 40:45Jacob Effron?

  Yeah, it's so interesting. It feels like a lot of the biggest beneficiaries of this, like, current AI wave are companies that had been around and had built, obviously, you'd built like, code search and had built so much around understanding your codebase, which feels then perfectly set up to, you know, retrieve against and provide context into some of these new completion tools.

• 41:02Speaker 01

  Yeah, I would say, in my view, beneficiaries of AI include companies that have sort of like, unique access to data, or a unique way to structure some data so that's valuable. And for us, that's been, you know, part of our foundation from the very beginning.

• 41:19Jacob Effron?

  Yeah, no, it's really interesting. And I guess like, you know, what what metrics do you use? Obviously, it sounds like you look at like the percent of code completion recommendations that are suggested. But what metrics do you guys use to say, hey, is this working or not? And then anything you can share about what the impact you've seen from these AI features?

• 41:35Speaker 01

  Yeah, so it's a mix of metrics. Completions on the completion sides, we look at acceptance rate and overall volume a lot. So overall volume gives you like a sense of like how much value you're adding. And acceptance rate gives you a sense of like, how often is this like actually helping versus like wasting someone's time? And that's important because completions is kind of like a implicitly triggered action. So like engagement, for instance, not necessarily as useful as it is there versus like an explicitly triggered feature. Because it's not up to you to decide when you want a completion. So those are the main metrics on the completion side on the chat and kind of like generate tests. Those are explicit actions. So we tend to look at engagement or derivatives of engagement there as well as, you know, overall usage.

• 42:19Speaker 01

  And I don't know, in terms of impact, I think it's it's the true impact really comes across into like the anecdotes that you hear from users and customers when someone comes to you and says like, holy crap. Like, I was able to like build up this entire application in like a day or two. Previously, I would have had to budget like a week or maybe even longer to get all the pieces in place. And I think that's that's really the magic here. It's like the things that have been done that maybe you don't necessarily know how to do well, but have been done many times over in your organization or in the world at large.

• 43:00Speaker 01

  Now those are are much easier to do because you can kind of let the AI get you to that point of like, okay, all the baseline pieces, all the boilerplate, the boilerplate plus plus is in place so that you can actually focus on the things that actually. You know, are your point of differentiation or your unique point of value of what you're building here?

• 43:22Jacob Effron?

  And what about like, I mean, one thing I noticed that you guys have focused on recently, or at least written about is, you know, letting people run models locally. Like, where do you see that playing? Is that something that your users are asking for? Like, how do you think about, you know, both like focusing on that and then also where that fits into things long term?

• 43:39Speaker 01

  There's a growing contingent of the Cody community that is all about local inference models. And so a lot of people who use Cody use it in conjunction with something like Ollama or LM Studio. So they run the model on their local hardware and because Cody by design is able to integrate, you know, new LM's easily, it was, you know, I think it was a community contribution, actually, that first added the Ollama support into Cody. And so that's something that we definitely want to nurture. And I think there's going to be, you know, some really, it's definitely not going away. And I think it will become a growing part of inference to do it locally.

• 44:21Speaker 01

  I think like the first kind of like killer use case, at least for me, is just the ability to use a coding system on an airplane where the network connectivity is not. Cause like-

• 44:36Speaker 02

  Maybe with Starlink, you know, you'll have a big model in the cloud, you never know.

• 44:39Jacob Effron?

  I like, I like that it's like, would the airlines have gotten wifi faster? Would we have built incredibly powerful local models?

• 44:44Speaker 02

  That can generate all code of humanity on your laptop.

• 44:47Jacob Effron?

  Yeah, it turns out the latter happened faster than the former.

• 44:49Speaker 01

  Exactly. So like in the short term, it's really about like having availability in places where you don't have network connectivity and that helps a lot. I think longer term, the reasons, you know, why would you want to do inference locally? Or why would you want to do any computation locally? It's like privacy and cost, I think, mainly. And I think developers care a lot about that. You know, having something that is contained within the walls of my local machine. All things being equal, I'd rather just have it run on my local machine rather than have to talk to network.

• 45:20Speaker 02

  Yeah, I also could see a world, you were talking about the inner loops. You want those to happen super fast, right? And they're often less complex. So maybe the inner loops, there would be something that's more local. That, or you don't know where it's running and that runs really, really quickly locally. But then if you want to do something complex, then you go send requests to the cloud. It comes back with something smart.

• 45:39Speaker 01

  Yeah, exactly. And I think also over time is like-

• 45:43Speaker 01

  GPUs become faster and also models become faster. I think you, because like right now, the, the, the source of, the, the largest source of latency in most AI-based systems is inference time. And so you don't really care too much about like network round trip as much as you do in other applications of compute. But I think inference will continue to improve and at some point we'll get to the point where it's like, oh, actually, that's no longer the bottleneck. The bottleneck is like, I gotta wait for the, the round trip from, from me to wherever the, the server is hosted. At that point, like you can feel the difference between, like, developers are very latency sensitive, to the point of like, you can feel-

• 46:27Speaker 02

  I think we all are in this TikTok generation.

• 46:28Speaker 01

  Yeah. But like, you can feel the milliseconds. Like, there's certain, like, editors, like, for example, like VS Code is probably like the most popular editor today, but like, I can feel the difference between VS Code keystroke latency and say like, you know, Emacs my terminal latency. And I still use Emacs for like, a good chunk of development because for me, it's just something that I perceive and I find that I, having to deal with latency, it is, it detracts from my feeling of joy while coding and that in turn affects, you know, my output or, or my propensity to want to code.

• 47:04Speaker 02

  You have to check out Z, uh, Zed, shameless plug.

• 47:07Speaker 01

  Yeah, so-

• 47:07Speaker 02

  Yeah, yeah, there you go.

• 47:08Speaker 01

  Zed is a great-

• 47:09Jacob Effron?

  There it is. You waited, you waited a long time before plugging one of your four codes. I'm impressed.

• 47:12Speaker 02

  I didn't want to plug it too early, you know, but yeah, uh, yeah, check out Zed.

• 47:16Speaker 01

  Yeah, we, we love the folks at Zed and I really like, uh, what they're doing. And I think that is, that, uh, that is just a proof for, for, for why latency matters, uh, uh, a lot. And it's an argument for, you know, local inference, uh, uh, when, when inference times goes down, uh, on, on local hardware. And it's, it's proof that like shaving milliseconds still really matters to developer experience because, um, I, I don't know how, how exactly to describe it, but you, you just, you can feel it and it, you can feel it slow, slow you down. Like if your brain has to like block on something external, it just, it, it kind of like changes the psychology of, uh- Of development work.

• 47:57Speaker 02

  And you talked a little bit about the model evaluation in terms of product metrics, right? You'll check all the different product metrics and how often people are accepting the completions. Are there other offline evaluations that you guys do when something ships of, I don't know, test sets internally or is that something that's top of mind or is product kind of the focus?

• 48:18Speaker 01

  We have a set of like offline benchmarks that we run new models through. We're currently in the process of standing up a few more specifically tuned at like or aimed at I would say like retrieval based models because that's really important. That feeds into search quality as well as AI quality. So yeah, that is an important muscle that we're.

• 48:40Speaker 02

  Have you looked at like using a tool for that or?

• 48:43Speaker 01

  We've taken a look at several tools. The there is a number of them out there. They all seem very promising. I think my personal, this is just like a personal bias of mine, but like I always like to with things that are like very new like this.

• 49:00Speaker 01

  My philosophy is that like let's, let's do it ourselves first because like that will like maybe doing it ourselves will be good enough and we won't have to look at external dependencies. And if not, then at least we'll have learned a lot through that process that will then in turn like allows to make a very smart decision there. So we did this in kind of like the early days of Cody too, when we were building up the rag engine, we decided to build that in-house because it was, it was just so early that the amount of confidence I had in any of the kind of like external solutions being significantly better was low.

• 49:44Speaker 01

  And any sort of time you outsource something like that, it comes at the cost of an abstraction boundary, a sort of like API dependency.

• 49:56Speaker 01

  And almost like a mental block for your own team to kind of acquire knowledge that might contribute to your differentiation in the marketplace.

• 50:05Jacob Effron?

  So interesting. What vendors do you use now for the RAG?

• 50:08Speaker 01

  Nothing major.

• 50:09Jacob Effron?

  Yeah.

• 50:09Speaker 01

  Yeah.

• 50:09Jacob Effron?

  I guess.

• 50:10Speaker 01

  Mostly in-house.

• 50:11Jacob Effron?

  Yeah, I'm curious on that because it feels like there's, you know, there's so many different ways you can customize and make RAG better, right? There's obviously you can choose and purpose build different embeddings models. There's all sorts of like chunking strategies, you know, there's databases specific to vectors you can use.

• 50:26Speaker 01

  Yeah.

• 50:27Jacob Effron?

  And it's funny, I think we talked to like, I feel like 99% of the people we talked to are not doing any of those customizations today. It's like, let's just figure out if, to your point earlier, like, let's do the dumbest thing first and just see how it works. Like,

• 50:39Speaker 01

  Yeah.

• 50:39Jacob Effron?

  Have you guys, you know, you're heavy RAG users. Have you evolved like the way you've done that over time in other places where you're kind of customizing, you know, different parts of this? Or have you found that kind of off-the-shelf RAG, for lack of a better term, works pretty well?

• 50:52Speaker 01

  You need to think a lot about the RAG pipeline. If I like the, the retrieval mechanisms matter a lot. I think it's very, I would say two things. Okay. RAG, RAG is really just like a, it's a generalization of the search problem.

• 51:08Speaker 01

  And so if you think about like what makes for a good search engine, it's like, it's both simple and also very complex, right? It's really just a matter of like, can you not forget anything that's important and bump up the things that are important to the top?

• 51:28Jacob Effron?

  Quickly.

• 51:29Speaker 01

  Quickly.

• 51:31Jacob Effron?

  Spoken like someone who worked on search.

• 51:34Speaker 01

  And so you need to, it's, that's actually, it's very hard to do. I would say like in, in most domains, that is a very hard technical challenge. And so there is a very deep rabbit hole of optimizations and improvement that you can make that drive, improvement in the end user experience. That being said, you know, where do you start? I think the baseline that I recommend to everyone who asks this question of me is like, start with like classical keyword search because that is very easy to implement. It's very easy to understand what it's doing and it's very easy to.

• 52:14Speaker 01

  Kind of iterate on and improve. There's no like black magic to it. You kind of can understand like why it's finding certain things and not others and why it's ranking them more highly. Uh, and that's what we did. And for a good portion of time, like Cody, uh, was and still is like the best, has the best context quality on the market. Um, and for a long time, a lot of, uh, that engine was actually mainly relying on keyword search with, you know, some kind of like clever, uh, code aware, uh, uh, chunking strategies. That's no longer the case, but like that, that put us on the frontier very quickly.

• 52:54Speaker 01

  And meanwhile, we saw a lot of other players in the market, uh, trying things that were very, uh, kind of like sophisticated. Um, you know, like, uh, trying to, uh, utilize different embeddings models and deploying them, uh, or, or using them in like, you know, various vector databases. And there's a bunch of like fancy names out there, uh, for different, uh, uh, AI assisted search techniques. Um, but in practice, like they didn't, they, they worked very poorly. Um, we, we did the same thing, by the way, cause like, you know, we're like, oh, we can't resist. Like, we gotta try this out.

• 53:31Speaker 01

  But what we found is like the naive, uh, kind of like rip the embeddings out and then dump it into a vector database for nearest neighbor retrieval.

• 53:43Speaker 01

  I guess it's better than nothing, but, um, oftentimes it performs less well than keyword search and it's very noisy. Like you get some results that are like, oh, cool. It, you know, it made this connection. These, these two concepts are kind of synonymous, but they don't share any keyword overlap. You do get those, but there's a lot of noise. So you miss out a lot on a lot of quote, like obvious answers. And there's a lot of, uh, um, things that it thinks are highly relevant. They're, you know, very close in embedding space that don't actually map to what you're looking for from your retrieval engine or, you know, as, as a user of end user search.

• 54:16Speaker 02

  How have you guys thought about?

• 54:19Jacob Effron?

  Building, because you have code search and the RAG stuff.

• 54:22Speaker 01

  Yeah.

• 54:22Jacob Effron?

  How have you thought about kind of using one for the other? Or what you learned in code search and then applying that to RAG or vice versa? Like, how do you think about the search stacks for those two different pieces?

• 54:33Speaker 01

  Yeah, so I mean, on the RAG side, we have a lot of, we have context providers outside of search, right? There's like a bunch of different ways to surface context. It's basically like, anything a human does to acquire context, you could conceivably automate and then use that as like a strategy or a meta algorithm for context for AI. But that being said, like, search is probably still the most prominent form of context retrieval. And so the way we're running it now is we have two parallel threads, essentially. Like, we have a really talented leader on search, who's just like focusing on the problem, treating it as kind of like an end-user search problem. And then we have another person who's driving forward, thinking more of it

• 55:17Speaker 01

  as like a context optimization problem. And what I expect will happen is, in doing that, we will develop pieces that end up being useful for, you know, cross-cutting concern. So like, if you build me like a really awesome code search, like, I can for sure use that as a context retriever. And if you build me like a really awesome, like, user intent-aware context engine, I can for sure use that in the search stack. You know, like, I can adjust my search strategy based on my inferring of like what the user is trying to get out. Like, what flavor of search are they doing here?

• 55:51Jacob Effron?

  I've seen people spending so much time trying to figure out, like, you know, what models to use and what contexts. And I feel like so much less time figuring out, you know, what search strategies to use for retrieval. And it feels like that, obviously, has been a huge driver of your improved performance. It feels like something we'll probably see in the years to come much more.

• 56:05Speaker 01

  Take a look at like how Google search works, right? Like, or the path that they took. I think in the very early days, they had like a key technical insight that, you know, moved the needle very much, that was PageRank. But if you look at like, you know, anecdotally, what people have said like matters for search quality over time, especially like now, it's

• 56:26Speaker 01

  Like, what percentage of the search ranking is page rank at this point? It's like so much of just like, look at what users do, like, what do they actually click on? And, you know, partition the space by use case and, and there's like a million different things that happen on that search path. So I think the search problem is still like a very meaty one. It's definitely amenable to kind of like data-driven approaches. Like, let's just, you know, train a model to learn the best search ranking. But there's still, I think, a place for kind of like human judgment calls to be made.

• 57:05Jacob Effron?

  One thing I definitely want to make sure we hit on is like agents too. And I think particularly as we're talking about like the inner loop and the outer loop, obviously, like there's, you know, I think a lot of excitement around what might happen in the outer loop, but you know, probably the stuff doesn't yet work today. And so I'm wondering then for you, you know, at the homeless source graph, how do you think about like getting ahead, I guess, of the Sagic future? Like, obviously, there's, there's one world where it's like, hey, let's just wait until the stuff works and then we'll like figure out how to put it into the product. Or there's another world where you're like, well, let's make sure we kind of know how we might incorporate this in or, you know, the second, we know the second it does work so we can put it in. Like, I guess, how do you think about that?

• 57:41Speaker 01

  There's a couple of different ways of thinking about it. So at any given point in time, we have like a portfolio of experiments that we're running of things that we think like could work out. And there are certain, like, we're currently building systems that are targeting like specific sectors or problems, I would say, that we think are amenable to full automation. And so, you know, maybe those, some of those will see the light of day at some point. I certainly think so. We're seeing a lot of promise in that. So basically what we're doing is we're taking the existing context provides that we have, combining that with kind of like an execution loop and throwing that at like a set of problems that we think are amenable to something that gets kind of like recurring usage.

• 58:31Speaker 01

  And so that is a current effort that's underway. We're actively working on that and it's driving. It's nice because like solving for that problem adds value to like even interloop cases. Cause a lot of the subroutines you end up developing for that, you can also reuse in sort of like an interloop tool, right? Like one subroutine, for instance, is like generating a test function or some sort of like verification mechanism. Because the name of the game for the end to end thing is like generate a good verifier. The verification mechanism needs to be more or less correct and then suggest a way to like make that pass. And then if it doesn't pass, then you take that context and you feed it back to the loop. But like you need a verification mechanism.

• 59:16Speaker 01

  So like test generation is really important for agents. It's also really important for interloop activity. Cause like, you know, that's what human developers do. It's like, how do you know your code works? Well, if you haven't tested it, you don't really. At the same time, like philosophically, I think our view is that some innovations are going to come from us and there's going to be some innovations that are going to come from the market. Like we're not going to have a monopoly on like every cool new idea that comes out at the intersection of AI and code. Where I think we can add value is like no matter what new end user use case emerges, you know, whether it's some sort of like agentic UI or whether it's some sort of like new

• 1:00:01Speaker 01

  interloop user experience. I think high quality context providers are sort of like crosscut all those different use cases. So it's almost like as these new kind of UIs or user experiences emerge, how can we plug our context engine into those to make whatever application is in that kind of like vertical the best in class? And it doesn't necessarily have to be like a Source Graph first-party application either. I think where our mind goes is increasingly toward how can we provide maybe like one or two

• 1:00:46Speaker 01

  end-user applications, but beyond that, like I said, a building blocks that other people can assemble. Because one, there's going to be a lot of players in the market that are exploring new use cases, and so like these primitives would be useful building blocks for them. And then two, our customers ask us about this incessantly. There's like, you know, when's the Cody API coming online? I want to plug this in. I want a script against this. I want to plug it into like my CICD pipeline in some form. And I think philosophically, it would be really cool to enable a sort of like ecosystem to emerge that enables like any developer out there to take advantage of the power

• 1:01:31Speaker 01

  of AI. And you know, it's not necessarily like a separate class of developers that builds AI-enabled development tools. We want to give every developer in the world the ability to build on top of AI. And we've already done this to a certain use case, sorry, we've already done this to a certain extent in the form of custom commands. So like, one of the features of Cody is if there's a command or if there's an action that you want to do that involves, you know, asking LM to do something, say like, you know, generate a commit message for, you know, a recent change or something like that. You know, Cody doesn't do that out of the box today, but if you wanted to do that, we actually have like a simple framework in the extension itself where you can go and describe what context you want to pull in, what the prompt is to the LM, and how you

• 1:02:14Speaker 01

  want that output to be rendered, like whether you want it to render in line or in the chat box. That's really cool. So we increasingly think about like, okay, what are the building blocks that are reusable here and how can we get them in the hands of people outside the company, either customers or other builders out there? So yeah.

• 1:02:32Jacob Effron?

  So I mean, there's so much there. I thought so. I love this idea that like in working on the outer loops, you're actually improving the inner loops and that like, um, you know, you're because to make that stuff work requires a lot of the inner loop stuff working more effectively. And then, you know, it sounds like you've just basically picked a set of problems that you're like, if we could solve this, this would be really valuable. And maybe some of them will hang out there for five years and, you know, who knows on this exponential curve where we're going, but like, at least good to know, uh, good to know when. I mean, I am curious, like, what are the milestones in like AI coding that would be meaningful to you? Like if I said, okay, you know, you wake up one morning and like X has happened, you know, what are, what are the milestones that, you know, you'd be kind of blown away by?

• 1:03:11Speaker 01

  I think it's, it's probably in the form of like, what cannot be done today that, you know, all of a sudden becomes, uh, possible. Um, and, and maybe I would partition, maybe, maybe I would partition that by, uh, like, uh, use case or like a, a set of things that I would want to be able to. So one way to partition this is like, of all the issues that out there that you might want to resolve, you know, you take all the entire taxonomy of like different feature requests, uh, you know, does it evolve the UIs, is it middleware only, you know, all, all the variations of like bug fixes. Um, it's like, which classes of problems, uh, are solvable by like a fully automated system today? Like right now in

• 1:03:54Speaker 01

  production, uh, it's, it's zero or close to zero.

• 1:04:00Jacob Effron?

  Yeah.

• 1:04:00Speaker 01

  Uh, right. Like everyone's still kind of like very much like human must be in the loop. Um, and so I think we'll hit milestones where it's like, oh, you know, um, simple bug fixes where maybe you take like the context pulled from some like production log that's showing a stack tracer and error and you plug that in and like all of a sudden like, you know, 80% of those become automatically fixable. Like that would be a significant milestone, um, I think. And then the other dimension of thinking about that is just, uh,

• 1:04:31Speaker 01

  I don't know. I guess model quality, that remains like a driver of what's possible. Um, I think there's like a ton of interesting work um in both like new model architectures and also uh how to train the existing architecture to be um more controllable and tunable. Um, and so there's a bunch of things like you always end up engineering around the limitations of uh the model. Um, but the more powerful the models become or the more controllable they've become like that that uh maybe eliminates some of the need for some of like the surrounding tools and guardrails that you put in place.

• 1:05:14Speaker 02

  If your average startup has X engineers today, what percent of X do you think they'll have in 10 years?

• 1:05:21Speaker 01

  Uh, average startup?

• 1:05:23Speaker 02

  Yeah, or average company. So they have X engineers today. Do you think kind of for the same? We only think in startups. We only think in startups, but the, do you think, do you think they'll have kind of X engineers or 2X? Or maybe 0.5X or um like any take on where the, the number of engineers is going?

• 1:05:41Speaker 01

  Uh, so in aggregate, I think the number of engineers is gonna grow. Uh, I think it's gonna grow uh maybe at the existing velocity for a little while and then accelerate in the medium to long term. But I will caveat that by saying that the, the definition of what engineering is or like the day-to-day lived experience of being a software developer is gonna change drastically. So the way I think about it is like, you know, the, the, the kind of um provocative of question here is like, you know, is AI gonna steal my job?

• 1:06:19Speaker 02

  Yeah.

• 1:06:20Speaker 01

  Uh, and

• 1:06:21Speaker 02

  We already know we're doomed, but we're asking for it.

• 1:06:24Speaker 01

  The VCs are gonna be the first to go for sure. Yeah, yeah. Conversation, you know, coming up with uh great interview questions.

• 1:06:31Speaker 02

  That, these came from, where do you think these came from?

• 1:06:36Speaker 01

  This is why you, you wanted to do it in person.

• 1:06:38Speaker 02

  Yeah. Yeah, exactly.

• 1:06:39Speaker 01

  So yeah, the answer to that question is both yes and no. Like, yes, it is going to take your job. But the good news is you will still have a job. It will just be a very different job than it is today. And I think by and large, it will be a better job. Because if you ask what, ask an average developer, like, you know, how do you spend your time? And most will tell you that they spend very little of their time actually producing software and shipping features. A lot of it goes to understanding the existing codebase and really context acquisition and, you know, communication overhead and things like that.

• 1:07:20Speaker 01

  And, I mean, a big part of the impetus for us starting Sourcegraph was kind of bridging this gap of what we all know software development could be, because we've all experienced that spark. Anyone who's developed like a simple hobby project knows this like feeling of awesome of like seeing your creation come to life. But then contrast that to like the day-to-day experience of being a professional software engineer. It's like, how often do you get to experience that magic moment? And the fact of the matter is, is that like most of your job is not the magic part right now. It's kind of like the toil and the drudgery.

• 1:08:01Speaker 01

  And so, yeah, I want AI to come in and steal that work away from me, so I have more time to do the fun things.

• 1:08:07Jacob Effron?

  Yeah. How do you like, I mean, this AI coding market, I feel like there's a ton of companies popping up every day.

• 1:08:12Speaker 01

  Yeah.

• 1:08:13Jacob Effron?

  Like, how do you think about, you know, I mean, how do you even like differentiate among like the different segments of that market today? And then do you think, like, how many are there room for, I guess, long-term? Like, are we going to have, you know, one incredible coding model subsume the entire market? Or is there like, you know, where do you think this kind of all heads?

• 1:08:30Speaker 01

  First off, I think it's amazing that DevTools is viewed as kind of like a hot investment area these days. It's a very, it's like night and day from when we first started the company back in, you know, the ancient time of 2013. Or four years before attention is all you need, as we call it here. Yeah, right, right. Four B, A.

• 1:08:57Speaker 01

  Because in those days, if you said the word dev tool to an investor, 99% of them would kind of like immediately run away. Fortunately, we're enough to connect with the good folks at Redpoint, Scott Rainey, and he had the conviction, I think, early. Like, he was a big early investor in some great developer-driven companies, like Grokust, right?

• 1:09:18Jacob Effron?

  Yeah.

• 1:09:18Speaker 01

  And Sourcegraph. I think where it goes is the process of software creation is becoming kind of like a market unto itself. So, the previous generation of tech is really about like software eating the world. And during that era, like, the process of like software creation itself was kind of like, it was not like a market to chase after itself. In fact, like, most dev tools advancement was driven by people trying to build platforms that they're trying to drive people onto. So like, you know, your desktop operating system, or your mobile operating system. You know, like, the history of tech has largely been defined about like, you know, what is the new platform?

• 1:10:00Speaker 01

  I want to become the platform that captures as much of the developer mindshare, and then people are forced to build on top of me, and I can sort of like extract value from that to my heart's desire, assuming I win that space. But I think now we're in this age where the process of software creation itself is becoming a thing, like a mainstream thing. And the amount of people involved in software creation is going to grow. Like, I still think that, you know, there's this notion of like software literacy. I think like the literacy will continue to expand. AI is an accelerant to that, because it puts the power of software creation into many more people's hands. So like, the marketplace is huge.

• 1:10:44Speaker 01

  And I think that means that there is going to be a lot of room for a lot of different applications that help people working in different domains of software, who have different styles of building software, and that's great. I think where we see ourselves playing a role there is we want to ensure that the ecosystem that emerges is open and preserves sort of like freedom of choice, both for the individual developers and the companies that are building software. So, we don't want to... Anytime there's like a big technological disruption, it's kind of like, you know, what is the new world order that will emerge from this? And I think we're very lucky to have come of age at a time where the predominant

• 1:11:27Speaker 01

  computing model was was very open, thanks to the web and the dominance of Linux on the web. It was a great time to be a software developer, right? Because you could build on top of those open platforms and they were they were friendly, developer friendly, right? They enabled a ton of value to be created because they were easy to work with and they didn't restrict your freedom of choice. And I think one question in our minds, and I think that should be in the minds of like every single developer, is like, what is the next era of software creation look like? Because there is a very real reality that, you know, the outcome of all this is, you

• 1:12:09Speaker 01

  know, some large players in the market use this as a an opportunity to kind of like vertically integrate the market and force everyone onto a proprietary development platform of sorts. And that would be very much to their self-interest, because now they have the ability to capture value. And that's not the world that we want to live in. It's not the world that we want to facilitate as a company. We think we make the most money by building a tool that gives individual developers freedom of choice, you know, in terms of models, in terms of code hosts, in terms of technology stack. And also enterprise companies, they care about that a lot as well. Because, you know, if you think about, how do you build a sustainable business? It's hard to build a very profitable, sustainable business if you are entirely dependent on

• 1:12:52Speaker 01

  a single vendor and that vendor, you don't have an alternative to that vendor because of, you know, high switching costs or whatnot.

• 1:13:00Jacob Effron?

  And the board fires the CEO of that vendor randomly on a Friday?

• 1:13:04Speaker 01

  That was not actually the example that I had in mind when I was saying that. But like, like, we, for instance, like, I don't think we have like much control over like how the the model landscape evolves, but we are happy that there are a lot of like model vendors out there, both open and closed. Like, we work very closely with OpenAI and also Anthropic. And we love, we use models from Meta Research and Mistral. And it's just, it's great to see that the field is vibrant currently. It's great for application builders and end users as well. And I think for society at large, that's kind of like the picture that we want to keep going as we move forward, because it'll just mean things remain vibrant and very innovative.

• 1:13:49Jacob Effron?

  Well, we always like to end our interviews with a quickfire round where we get your quick takes on overly broad questions that probably should be standalone, but we lump into the quickfire anyway. I guess the first one we always start with is, you know, what's one thing that's overhyped and one thing that's underhyped in AI today?

• 1:14:03Speaker 01

  I think this notion of AGI, you know, it's it's kind of like underdefined. So it's hard to pin down, but like, I think this like vision, almost like messianic vision that some people have, either of it solving all our problems or, you know, killing us all, is hugely overblown. I can, there's multiple layers to why I think that's not the case, but like, simply scaling transformers, I just, that would be like, it's it's it's a take that is so wrong to me that it's it's it's definitely not right and it's almost not even wrong, if that makes sense. If you think about like what a transformer is and what it does and like what you're

• 1:14:47Speaker 01

  doing when you're like minimizing the loss. Anyways, so yeah, I think that's probably overhyped. Obviously, it's a useful technology and will deliver a lot of value to humanity. What's underhyped? I would say like six months ago, I would have said RAG, but I think now RAG has had its moment. As far as what's underhyped today, I think there is a ton of value in building things that are complementary to AI and specifically large language models.

• 1:15:27Speaker 01

  And one thing that I think is complementary are essentially like formal specifications or formal languages. And so I think a lot of people think that AI maybe replaces a lot of these. I think the two are complementary because if you want to do useful things, There is a very real place for formal specifications of formal languages. Like, I do not, for instance, think that in the future, there will be no programming languages at all. Because I think like, natural language is not, it's not specific enough to describe what you want, and you need a medium to describe things with precision.

• 1:16:12Speaker 01

  I mean, that's why like, math exists, right? Like, in the early days, if you read like, Newton's letters from back in the day when he was corresponding with that other guy, the other inventor of calculus, I don't know.

• 1:16:23Speaker 02

  Yeah, I forget.

• 1:16:24Speaker 01

  Yeah, I was gonna make a guess. I want to say Leibniz, but like, I'm really... Yeah, Leibniz. The podcast.

• 1:16:29Speaker 02

  When they were like... I'm way off with Gauss, I know nothing.

• 1:16:31Speaker 01

  When they're corresponding... He sounds like someone famous, too.

• 1:16:33Speaker 02

  It's a mathematician.

• 1:16:34Speaker 01

  It's like, a hot take these days is like, you know, why would I write code if I could write English? Well, it's kind of like, well, why do we have math? It's like, in the earliest days, like, mathematicians corresponded in English, or whatever their local language was, because they didn't have another option. But they came up with math because English is a very precise and very verbose way of describing what you mean. So I think formal languages, programming languages, still have a place, and I think they will become an increasingly important complement to like, Norvig-style AI.

• 1:17:03Speaker 02

  Very cool. What's been the biggest surprise in building LLM features at Source Graph? Maybe one thing that you thought would work that didn't, or one thing that you thought would work and ended up not?

• 1:17:15Speaker 01

  I would say that there were a bunch of things that we thought would be easy on the context side that ended up being a much, like, deeper rabbit hole, in terms of like, number one, how do we find a model that is good at integrating this context into something that works well? And then, how do we like, I guess it's a form of prompt engineering, like, how do we place that context in the correct format so that the model actually uses it in the way that we want? So that's been much harder than we anticipated. Like, if you asked me a year ago, like, where would we be in the process of like, constructing this context engine? There's some, I definitely thought we'd be further along in some dimensions, and we're still kind of working on that.

• 1:17:58Speaker 01

  What's been easier? I don't know, like, I think the pace of advancement in model output quality and efficiency continues to be, like, very happily surprising. I feel like with every new kind of like generation of model releases, like, oh, wow, like, this actually does move the needle significantly. And so, you know, a bunch of the things that we were thinking about, we'd have to hack around or design around, you know, maybe we don't have to build that anymore. And so that's been, that's been a nice surprise.

• 1:18:33Speaker 02

  Yeah. What do you think of all the buzz around cognition?

• 1:18:36Speaker 01

  Kind of like Devin.

• 1:18:38Speaker 02

  Yeah.

• 1:18:39Speaker 01

  I think, I mean, it's great. I, I think the, the team there is, is really talented. I think Devin showed the world the potential of, of what kind of like a fully automated AI UI might look like. I think they had a lot of good insights in terms of how they built that, that were spot on. And I respect the heck out of the team there. And I think, I think it's, the next step is just shipping a product to market and getting people to, to use it every day. And so, yeah, I'm, I'm excited to see how that, how that, how that plays out.

• 1:19:19Speaker 02

  How ubiquitous do you think open source models will be in the future?

• 1:19:22Speaker 01

  I think they will be very widespread. So, I, I, you know, I honestly don't think it'll be like, you know, just open or just closed. I think there's always going to be a place for proprietary model providers out there. Either they have like a unique, unique way of combining different models or, you know, training their models, either in the training regime or maybe the model architecture or maybe the data sources, proprietary data sources that they've constructed or gathered. But I think open source models will probably become... More prevalent, certainly in terms of usage, if not value capture.

• 1:20:08Speaker 01

  Just because, I don't know, it's like the cat's out of the bag. And most of the key research, I think, is publicly accessible. And so it's kind of just like a matter of time before the world at large kind of understands how to train high quality models well. And as soon as that becomes a thing, then.

• 1:20:32Speaker 01

  Yeah, already there have been like stellar open source models. We use a lot of them within Cody. Our primary code completion model actually is StarCoder, which is open source.

• 1:20:44Speaker 01

  And there's just a ton of advantages of having access to a model at the model layer. Like you do things like fine tune them. You can do things like peer into what the state of attention is. And there's a bunch of different things you can that are useful at the application level. If you can look at like the attention weights during inference that are behind an API wall with kind of like proprietary models.

• 1:21:06Jacob Effron?

  Yeah. So what are examples of those like that you've experienced?

• 1:21:10Speaker 01

  Oh, I mean, it's something as simple as like.

• 1:21:14Speaker 01

  Here's an example, like.

• 1:21:17Speaker 01

  One of the things you want to do when you're constructing a context engine is like determine what's relevant information and what's not. And so one way you can do that is just like look at it and hand label it. Make that determination as a human. Another way you could do it is put that in the context window of an LM, ask it a question and see which weights get activated.

• 1:21:41Speaker 01

  Or another idea that we've had, which we haven't got around to implementing is like if you wanted to give the user some notion of confidence in a response, maybe at like the token or character level. Uh, it might actually be interesting to see like, is this completion, you know, what part of it is highly confident versus what is more kind of like, eh, you know, maybe this is right, maybe it's wrong.

• 1:22:03Jacob Effron?

  If I remember correctly, I think you're like a star testimonial on the fireworks website. And so I'm curious, like, uh, how do you think about choosing inference provider? Like, what happens to that market long term?

• 1:22:14Speaker 01

  Um, first of all, I just want to say like fireworks has been really awesome to work with. Um, and we took a look at a variety of inference providers and we went with them because they were the ones who treated it most like a business. We're, we're like in a, at an interesting phase of like AI, uh, right now, where I think there's, you know, there's, there's obviously like a ton of hype. And, and I think that it's important for, uh, founders to realize that, uh, at the end of the day, you want to build a business and the business is all about serving your customers. And so what really impressed us with fireworks was just their attention, their customer focus essentially. Um, and they were able to help us spend something of, uh, very quickly and they were helpful

• 1:22:55Speaker 01

  with a lot of, uh, advice on, you know, what, what would improve the quality and, and speed of inference. Um, where the market goes, uh, gosh, I don't know. Um, I honestly, uh, have, have no idea. I think from, from our point of view, uh, we build an application layer. So it's like a piece of infrastructure that we use. And so our calculus is, you know, which vendor out there provides, uh, the best inference. And there's really three dimensions that matter there. It's, uh, cost, speed and, and quality. Um, and I don't know. You guys are the investors. You tell me.

• 1:23:34Jacob Effron?

  I'm, I'm having, I would love you to shamelessly do my job. If AI is not, you can take it and then AI can take it afterwards.

• 1:23:40Speaker 02

  You buried the hard questions or chat GPT, I guess, buried the hard questions.

• 1:23:45Jacob Effron?

  I just showed up here 30 seconds before. Obviously you seem to be having a great time at Sourcegraph, but, uh, if we teleported you, you were now running any other AI startup out there. Uh, what do you think would be most fun or exciting to run right now?

• 1:23:58Speaker 01

  I think there's a lot of, so, uh, maybe like two things seem very promising or exciting to me. One is I think it's, AI presents an opportunity to think how work is done in any given knowledge domain. And so I think there's like a ton of interesting opportunities. In building essentially like a vertical SaaS, or not like an operating system in the Windows sense, but operating like a knowledge work domain operating system for given domain. Because like you can do things a lot differently with the benefit of the technology that we have today. And so there's a great opportunity for rethinking how work should be done in a bunch of domains. And I think that there will be some really cool startups

• 1:24:42Speaker 01

  built around rethinking the way that, you know, I don't know, like financial services, professional investing, healthcare consulting. I don't know, like the list of knowledge work domains is very long, so that's a big opportunity. The other thing that I'm still waiting on, but I'm still am very excited about is like the consumer impact. I'm still waiting on the, I don't know what the standard way to describe this is, but I think of it as like, you know, the concierge as a service thing. Or another way to think about it is like Google, but it actually does things for me rather than just respond to queries and questions. So something like that would be very useful to me. I personally, I'm like a developer at heart.

• 1:25:27Speaker 01

  So this whole like GUI thing where I have to click or tap, you know, it's like a necessary evil that I am forced to deal with because the existing markets for users like me, the market for users like me is much smaller than, you know, humanity as a whole. And humanity as a whole has traditionally had trouble with command like, command line like utilities, but I think maybe that changes with L M. So I'm excited at some consumer paradigms emerging, which are more just like, I tell it what to do and I don't have to like scroll or tap or get distracted.

• 1:25:59Jacob Effron?

  Your concierge sounds far more practical. Pat and I are just excited with the AI music. You know, we have tons of fun with that.

• 1:26:05Speaker 01

  The AI music is pretty awesome. They're very cool.

• 1:26:07Jacob Effron?

  Well, we always like to leave the last word to you. I mean, I think this has been a fascinating conversation. I'm sure folks will want to pull on different threads. Like where can they go, where can they go to learn more about you, about SourceCraft, Cody, all the above?

• 1:26:17Speaker 01

  Cody is cody.dev. That's a kind of our landing page. You can download us from VS Code Marketplace or the IntelJ plugin store. We also have a EVM extension. You can go to sourcecraft.com if you want to search for code or pop into our Discord. So that's pretty easy to find from any of our kind of like major sites. And we get a lot of users from, you know, companies, large and small, and around the world popping in to say hello and give us feature ideas and feedback on product. And we'd love, love to say hi.

• 1:26:50Jacob Effron?

  Awesome. Well, this was a ton of fun. Thanks so much for, for coming on.

• 1:26:53Speaker 01

  Yeah, likewise. This was, it was awesome. And, and thanks for having me on.

• 1:26:57Speaker 02

  Well, that was super fun. What, what stood out to you?

• 1:27:00Jacob Effron?

  Yeah, I gotta say that was one of my favorite, like, post podcast moments I think we've ever had where for, for listeners won't be able to see this. But what did happen is as we were walking out after a pretty long recording, our fearless producer, V, mentioned he was a user of Cody. And beyond, you know, halfway out after, you know, having put his company on hold for two hours to spend with us was like, well, wait a second. Like, what feature requests do you have? And immediately, user interview happened in this teeny podcast studio, which I think is just, I mean, if that's not the, the heart of a founder, I don't know what is.

• 1:27:29Speaker 02

  It was, it was awesome. Yeah, one thing that I thought was, was super cool. I mean, there's, there's so much to unpack in the episode. I think him talking about how kind of senior engineers are using AI versus more junior engineers, with maybe more junior doing the code completion thing and then chat being more prevalent for the senior engineers. I thought that was really interesting because we've heard a lot about it in the market. But I think he kind of summed it up in a way and obviously has more visibility into this stuff than, than almost anyone. And so I thought that was really fascinating.

• 1:28:00Jacob Effron?

  Yeah, I mean, this idea of people maybe with less skills benefiting more from these tools. I mean, I feel like maybe, maybe I can finally catch up to you. I don't know. We'll see.

• 1:28:07Speaker 01

  I was fucking carried away.

• 1:28:11Speaker 02

  Yeah. And I also thought it was interesting. He was, we were asking about the, you know, GPT four ships or GPT five ships and then what happens. And It's kind of cool. I was trying to think of other examples where a new technology comes out and is moving that quickly and almost making his product better and better as as the new technology comes out, like GPT-5 ships and they can just immediately integrate that. And there's, you know, Cody gets better. I thought that was

• 1:28:38Jacob Effron?

  Totally.

• 1:28:39Speaker 02

  A pretty interesting

• 1:28:39Jacob Effron?

  I want to say not to shout out like another podcast on our podcast, but I was listening to Sam Altman on 20 Minute VC and he had this, you know, Harry Seibers was asking him like, what's this heuristic you use for like whether a startup, an application startup's going to investment? And I thought this pithy line he had was just, well, if our models get way better, are they happy or are they sad? And it's like, you can tell that basically like Source Graph is in this great position because the better these models get, they're the ones marrying it with the developer experience and kind of, you know, the core code base. And so I thought that was really interesting.

• 1:29:06Speaker 02

  Yeah, that's cool.

• 1:29:07Jacob Effron?

  And I love like, you know, other part I found fascinating was this idea of like, you know, evals and how obviously they're doing something around evals, but I love the idea of just like, well, we could spend all day trying to evaluate this internally. Why don't we just put it in our users' hands and give them choice and see what they, you know, they will know far better. Like we talk about the vibes eval a lot. Like you can actually get like concrete metrics on, and I'm surprised more companies don't do that. Just like give your end user a choice of different things. Now, obviously, I think they benefit from the fact that code completion is already an established AI category, but you could imagine someone using a not as effective model to start and just never trying an AI feature again. But for for features that clearly folks will use, I love the idea of just giving them the choice and then you can see the context in which in which people actually like using them.

• 1:29:50Speaker 02

  Totally. I think it's, I think about it kind of similar to how the the two approaches in self-driving cars, there's Tesla out there in the world driving with human assistance and launching all of these self-driving features. And then there's a Waymo approach of go into a lab and build this thing end to end. So it actually works. And, you know, obviously there are the same consequences on coding that there are and things in the real world, but it was still interesting to see how he thought about shipping the models and the the different approaches.

• 1:30:18Jacob Effron?

  Totally. I mean, and one theme that really resonated from this episode for me is like, I think he said it multiple times, just like, do the dumb thing first.

• 1:30:25Speaker 02

  Yeah.

• 1:30:25Jacob Effron?

  It kind of reminds me of like the episode we did with Arvin from Perplexity where he's like, you know, you can think about all the Complex stuff later, like build, like just get something out there, like see if people like it. Um, and I think in their case, you know, it's like, try the simple thing first. See if that a lot of times is going to work. And then on the edges, you can figure out where, uh, you know, where it makes sense to fine tune a model or do something more complex. But I think obviously there's always such a temptation. It's can be sexy to like, build your own model or do something, you know, from scratch. And it's like, a lot of times you don't need to do that.

• 1:30:55Speaker 02

  His takeaway that many times keyword search is really good for rag. I thought was also interesting, especially because they come from the search world, right? Like he built out all of this code search and then now rag is basically semantic search. And they're still using keywords for part of that among some, a bunch of other different data sources. And so I thought that was, that was really cool because he's a search expert. Rag is a search problem and you know, we got to see how kind of a search expert of the world would, would approach it.

• 1:31:24Jacob Effron?

  Yeah, it was intimidating being among two search experts. I felt like a fish out of water here.

• 1:31:28Speaker 02

  I'm far from an expert, especially um, with, with him. But yeah, it is, it's cool to see some of the, these techniques around um, keyword search or semantic search. Uh, you know, start to become very kind of cool. I guess cool is relative term, depends who you ask, but.

• 1:31:43Jacob Effron?

  You don't think we, if we find it cool, it's not just inherently.

• 1:31:45Speaker 02

  I find it cool. I found it cool before, but um, but the, you know, like vector search used to suck, right? It used to not be very useful and now the vectors are a much better representation than they were before. And um, it's, it's cool to see people uh, and companies like Sourcegraph put that, put that into practice.

• 1:32:03Jacob Effron?

  Totally. What I took away is that we are very early innings of, you know, again, people are doing the, the basic thing and that makes sense. But there's going to be so much more complexity in terms of what type of search you actually use for retrieval.

• 1:32:12Speaker 02

  Totally. I was happy. He said AGI was under defined and not under hyped. Because, um, because I would, I would push back on the fact that AGI is under hyped. But um, but yeah, I thought uh, his take on kind of where intelligence was going. And um, we're doing kind of the closed loop thing for developer, the inner loop thing. And then moving to the outer loop and then to more automation. And I thought um, there's, there's an interesting. Progression there.

• 1:32:41Jacob Effron?

  I love that because I think it's so relevant. So many of our portfolio companies are thinking like about, you know, obviously the inner loop work you do, like it works today and it makes the product better. I think there's this question of how much time do you spend on the outer loop when it's not ready yet? Do you just wait till the tools get better? And I loved his idea of like one, you know, two things I took away from what he was saying. One is we're always testing. We have a few things we know will be so impactful in the product that we want to make sure we know when they work. And then this idea that even in improving the outer loop, even if it's not ready for prime time, they make the inner loop better. I thought was such an interesting insight.

• 1:33:13Speaker 02

  Totally. Is there anything else you want to hit in the episode? Should we move to news?

• 1:33:18Jacob Effron?

  Yeah, I'd say AI news and any Taylor Swift updates that we needed to hit.

• 1:33:22Speaker 02

  I mean, Taylor Swift, the new album. I'm a big fan of So High School, her song about Travis Kelsey and Down Bad. Those are, those are my favorites. So for, for both Taylor Swift, the inner. Actually, that's probably not true. There's probably a lot of Taylor Swift listeners that, you know, maybe listen to us. I don't know.

• 1:33:44Jacob Effron?

  Yeah, I don't know if on our YouTube recommended, like, you know, Shake It Off comes right after this podcast.

• 1:33:49Speaker 02

  Well, and maybe now, and maybe I'm influencing the algorithm.

• 1:33:51Jacob Effron?

  If we say Taylor Swift enough, maybe it just happens.

• 1:33:54Speaker 02

  Yeah.

• 1:33:55Jacob Effron?

  But no, in the AI world, I feel like it's been, you know, besides the, the what 31 song Taylor Swift album dropping. I feel like there's been a lot happening in the AI world. You know, maybe most notably the Lama three release. I mean, again, a really powerful model. And I think, you know, they're still training the 405 billion parameter model. So this was just the interim step with the 70B one. But it definitely feels like, you know, the open source momentum really continues.

• 1:34:19Speaker 02

  Yeah. What'd you think of Zuck's new look with the, the hair and the chain and making people really bullish on their AI strategy based on?

• 1:34:26Jacob Effron?

  Totally. Well, I mean, this will probably be outdated when, when we release it. But I mean, their stock was down, you know, a large amount. I think, you know, I think there's still this question that Meta has around how much, you know, revenue are they really recognizing from AI? All the while, while Zuck, to keep Lama models, you know, competitive with OpenAI Anthropic is saying, hey, I'm going to spend tens of billions of dollars on GPUs. And so I think, you know, it's only a founder like Zuck, I feel like that could go. Have a public market company and be like, I'm just gonna spend this money and not get totally hammered for it. But it's interesting to see the, you know, I'll be interested to see whether public market investors are cool with him making this massive investment in something that it's not so clear how it monetizes. What else do we see? We saw obviously cognition incredible, incredible demo. You know, raising it $2 billion six months in.

• 1:35:11Jacob Effron?

  It's not, not bad for six months of work. But I think just reflects the pace and the excitement around. You know, this whole code space, which we hit today with, with Sourcegraph and, you know, there was a announcement of AugBen to raise this $250 million round. Like, it's kind of becoming clear in my mind, like if, you know, LLMs powering chat your docs and customer support were the first market. Feels like code, AI code, we're like about to hit some big inflection point in terms of just enterprise adoption. Like these tools, you've seen the metrics from Copilot, they're just so good.

• 1:35:40Speaker 02

  Code generation, code completion and support, like those are some of the first markets and we definitely saw the, the proof points for for GitHub Copilot. And now we're seeing the next generation of that, you know, obviously Copilot's growing a ton, but then also the, you know, end to end code generation that's, that's now coming out. It'll be, it'll be cool when these, these companies start shipping their models.

• 1:36:02Jacob Effron?

  I'm trying so hard not to shout out my portfolio company, PoolSuck, because I always give you shit for shouting out your portfolio company.

• 1:36:07Speaker 02

  I know, I know, but you had to get it in.

• 1:36:08Jacob Effron?

  I had to get it in. I mean, we can't talk about the AI code space and not say something.

• 1:36:12Speaker 02

  Next time I'll have to bring it up for you and then.

• 1:36:14Jacob Effron?

  Yeah, exactly. And then I can give you shit again.

• 1:36:16Speaker 02

  Yeah, exactly.

• 1:36:17Jacob Effron?

  I want that to be a shtick. But yeah, no, that was a, that was a ton of fun. Thanks for listening to the episode. If you enjoyed it, definitely share it with a friend who you think might too. We have an awesome set of guests coming up, so make sure to subscribe to see all the episodes that come through and we'll see you in a few weeks.