Coding in Collaboration with AI with Sourcegraph CTO Beyang Liu

Coding in collaboration with AI can reduce human toil in the software development process and lead to more accurate and less tedious work for coding teams. This week on No Priors, Sarah talked with Beyang Liu, the cofounder and CTO of Sourcegraph, which builds tools that help developers innovate faster. Their most recent launch was an AI coding assistant called Cody. Beyang has spent his entire career thinking about how humans can work in conjunction with AI to write better code. Sarah and Beyang talk about how Sourcegraph is thinking about augmenting the coding process in a way that ensures accuracy and efficiency starting with robust and high-quality context. They also think about what the future of software development could look like in a world where AI can generate high-quality code on its own and where that leaves humans in the coding process. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @beyang Show Notes: (0:00) Beyang Liu’s experience (0:52) Sourcegraph premise (2:20) AI and finding flow (4:18) Developing LLMs in code (6:46) Cody explanation (7:56) Unlocking AI code generation (11:00) search architecture in LLMs (16:02) Quality-assurance in data set (18:03) Future of Cody (22:48) Constraints in AI code generation (30:28) Lessons from Beyang’s research days (33:17) Benefits of small models (35:49) Future of software development (42:14) What skills will be valued down the line

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Sarah Guo?

  Hi listeners and welcome to another episode of No Priors. This week, we're talking to Beiyang Liu, the co-founder and CTO of SourceGraph, which builds tools that helps developers innovate faster. Their most recent launch was an AI coding assistant called Cody. We're excited to have Beiyang on to talk about how AI changes software development. Welcome.

• 0:24Speaker 01

  Cool. Thanks, Sarah. It's great to be on. Thanks for having me.

• 0:27Sarah Guo?

  Yeah. So you guys founded SourceGraph all the way back in 2013, right? I feel like I met you and Quinn at GopherCon either that year or the year after. Do you remember?

• 0:37Speaker 01

  Yeah, I think that's right. We met at one of those after-conference events. And I remember you asked me a bunch of questions about developer productivity and code search and what we were doing back then.

• 0:48Sarah Guo?

  Many listeners to the podcast are technical, but can you describe the core thesis of the company?

• 0:52Speaker 01

  Quinn and I are both developers by background. We felt that there was kind of like this gap between the promise of programming, being in flow and getting stuff done and creating something new that everyone experiences. It's probably the reason that many of us got into programming in the first place, the joy of creation. Then you compare that with the day-to-day of most professional software engineers, which is a lot of toil and a lot of drudgery. When we kind of drilled into that, you know, why is that? I think we both realized that we're spending a lot of our time in the process of reading and understanding the existing code rather than building new features because all that is prerequisite for being able to build quickly and efficiently. And that was a pain point that we saw again and again,

• 1:35Speaker 01

  both with the people that we collaborated with inside the company we were working at at the time, Palantir, as well as a lot of the enterprise customers that Palantir was working with. So we were kind of dropshipping into large banks and Fortune 500 companies and building software kind of embedded with their software teams. And if anything, the pain points they had around understanding legacy code and figuring out the context of the codebase that could work effectively was 10x, 100x of the challenges that we were experiencing. So it was partially scratching our own itch and partially like, hey, like, the pain we feel is reflected across all these different industries trying to build software.

• 2:14Sarah Guo?

  Yeah, we're going to come back to context and how important it is for using this generation of AI, but I want to go actually go back to like some roots you have in thinking about AI and your interning at the Stanford AI Research Lab way back when.

• 2:28Speaker 01

  Yeah.

• 2:29Sarah Guo?

  Like, that wasn't the starting point for Source Graph. It was more like, oh, we need like super grep, right? Like, we just need a version of search that works in real environments and is useful for getting to flow.

• 2:40Speaker 01

  Yeah.

• 2:41Sarah Guo?

  When in the story of Source Graph did you start thinking about how advancements in AI could change the product?

• 2:47Speaker 01

  My first love in terms of computer science was actually AI and machine learning. That's what I concentrated in when I was a student at Stanford. I worked in Stanford AI Lab with Daphne Koller, she's my advisor, mostly doing computer vision stuff in those days. And it was very different in those days. We're now living through the neural net revolution. You know, we're well into it. It's just like neural nets everywhere. And in those days, it's still kind of like the dark ages of neural nets, where it was after the first initial successes they had in like the late 80s and 90s doing OCR with them. But then after that, the use cases sort of petered out. And by the time that I was doing it, the conventional wisdom, the thing that they told us in, you know, machine learning 101

• 3:32Speaker 01

  was like, you know, neural nets were this thing that we tried, you know, a decade or so ago, but it didn't really pan out. So these days, we're mostly focused on graphical models and statistical learning techniques, you know, really trying to be explicit about modeling the probability distribution of what we're trying to represent.

• 3:48Sarah Guo?

  We actually had Daphne and one of her other former students, Lucas Bewald, from Now Weights and Biases, on the podcast as well. And both of them were also like lamenting the dark ages when like neural nets were like this weird niche thing, and we're going to work on graphical models instead. But it's very cool to see so many people who have like a, you know, interest and technical passion in this, like emerge the other end and be like, aha, like now is the time. So at what point were you like, okay, like, I'm going to look at this and we're going to try to work on it at Source Graph?

• 4:20Speaker 01

  Yeah, it's great. It really feels like a homecoming of sorts. And I think we're very fortunate that a lot of the underlying skillsets, I think, do transfer pretty well. I mean, it's all linear algebra and matrix operations underneath the hood, and that stuff is still applicable. And a lot of the intuitions, like the value of sparsity and things like that, still are kind of applicable. I'm still waiting for the statistical learning and maybe some of the convex optimization stuff to reemerge. I wouldn't count it entirely out yet. I feel like the pendulum always swings back the other way. It swung away from statistical learning and convex optimization and models now. But I think they'll reemerge, especially as we try to get deeper into interpreting

• 5:01Speaker 01

  how and why neural nets and attention is as good as it is. But to answer your question, when did we start thinking about this at SourceGraph? I want to say it was like circa 2017, 2018 that we started to kind of like revisit some of this, because I think 2017 was when the attention paper came out and you started to see more applications of LLMs in the space of code. I think Tab9 was one of the earliest to market there with the LLM-based auto-apply. I remember chatting with someone who had essentially implemented that on top of GPT-2 at the time. And it wasn't nearly as good as it is now, even then, you know, like two or three years ago.

• 5:45Speaker 01

  But we ran some early experiments applying LLMs, specifically embeddings, to code search, and that yielded some interesting results. Again, the quality wasn't at the point where we were ready to productionize it yet, but it was certainly enough to keep us going. I think things really picked up September or October of last year. It was a confluence of factors. I think, one, our internal efforts just kind of reached a level of maturity where we started being more serious, devoting more time to it. Second thing is I went on paternity leave, so I was able to step away from kind of like the day-to-day stuff a little bit, and that gave some time and room for kind of experimentation. And then, of course, at the end of November, ChatGPT landed, and that just changed the

• 6:30Speaker 01

  game for everyone. And there was a ton of interest and excitement that really gave us a big kick to start exploring in depth the efforts that we already had underway.

• 6:42Sarah Guo?

  Awesome. And so, explain what Cody is today.

• 6:44Speaker 01

  Cody is an AI coding assistant. It integrates into your editor, whether you're using VS Code or JetBrains. We also have experimental support for Neo Vim. And as an Emacs user, Emacs is on the way. We've also integrated it into our web application. So if you go to sourcegraph.com and go to a repository page, there's an "Ask Cody" button that allows you to ask high-level questions about that codebase. And in terms of feature set, it supports a lot of the features that other coding assistants support: inline completions, high-level Q&A, informed by the context of your codebase, kind of specific commands like "generate unit test" or "fix this compiler error" that are kind of like inline actions in the editor.

• 7:27Speaker 01

  And our main point of differentiation is, across that feature service area, we augment the large language model that we're using underneath the hood with all the contexts that we can pull in through SourceGraph and through techniques that we've refined over the past decade, building a really awesome code-understanding tool for human developers.

• 7:49Sarah Guo?

  Okay, so you have said, and I think it is a more interesting point of view now, that there is an argument that choosing and structuring large repo contexts is the key unlock for code generation and AI code functionality. Can you explain how you guys approach it?

• 8:05Speaker 01

  Yeah, so, in many ways, the context problem, so, context, another word for it is retrieval augmented generation. The basic idea, I mean, listeners of your pod are probably familiar with this, but just for the ones that are tuning in and unfamiliar, the idea is that large language models get a lot smarter when they're augmented with some sort of context fetching ability, the most common of which is typically like a search engine. So there's a number of examples out there of doing this. Bing Chat is one example. Perplexity is another example. They're building Google competitors where they integrate the large language model with a web search functionality. And fetching search results into the context

• 8:45Speaker 01

  window of the language model helps basically anchor it to specific facts and knowledge that helps it hallucinate less and generate more accurate responses. We essentially do the same thing for code, using a combination of code search and also something we call graph context. To pull in relevant code snippets and pieces of documentation into the context window in a way that improves code generation and high-level Q and A about the code. And so on the code search end, we're essentially incorporating the technologies that we've built to help human developers over the past decade. So if you look at the core feature set of Source Graph, the bread and butter really

• 9:26Speaker 01

  is you have code search, which allows you to go from, you know, I'm thinking of a function, or I'm thinking of an error message, and quickly pinpointing the needle in the haystack in a giant, giant universe of code. And then from there, it's sort of this walking the reference graph of code. So go to definition, find references in a way that doesn't require you to, you know, set up a development environment, or, you know, tangle with any build systems. It just all kind of works. So the analogy there is like, we want to make exploring and searching code as easy as it is to explore and search the web. That's a huge unlock for humans being able to take advantage of the institutional knowledge embedded in that data source.

• 10:08Speaker 01

  And it turns out, those same actions, the code search and then the walking the reference graph, turns out to be really useful for surfacing relevant pieces of context that you can then place into a language model's context window. That makes it much better at generating code that fits within the context of your code base, and also answering questions accurately without making as much stuff up.

• 10:27Sarah Guo?

  Actually, I'm very interested, do you do both, let's say, other traditional information retrieval approaches, like ranking along with AST traversal?

• 10:36Speaker 01

  Yeah.

• 10:37Sarah Guo?

  Or like, is there, is there information missing from the graph context that's also useful either for your humans using search, or for the models using search?

• 10:45Speaker 01

  Yeah, there's a ton of data sources. But let's start with the search side, which the search problem is really like, hey, the user asked a question, now find me all the pieces of code or pieces of documentation that could be relevant to answering that question. We really view that as a generalized search problem. It has a lot of like parallels to end-user search, with the difference being, you know, for human search, it's really important to get the quote-unquote "right" result in the top three, otherwise people will ignore it. Whereas with language models, you actually have a little bit more flexibility, because, you know, you have a context window of, these days, at least, you know, 2,000 tokens, in some cases, much longer, right? And then in terms of how you do that fetching, the overall architecture is very similar to how you would design a search engine.

• 11:30Speaker 01

  So you have a two-layered architecture. At the bottom layer are your kind of like underlying retrievers. So the base case here would be just keyword search. Or, you know, the fancy way of saying that nowadays is sparse vector search. If you use the kind of like one-hot encoding, where ones correspond to the presence of certain dictionary words. Anyways, that's just keyword search. It actually works reasonably well. I think if you talk to a lot of RAG practitioners, you'll find that the kind of like dirty secret is that keyword search can probably get you more than 90% of the way there. Let's talk about embeddings in a little bit. But on keyword search alone, there's a lot that we do.

• 12:13Speaker 01

  It's a combination of classic keyword search, combining that with things that work well for code, like regular expressions and string literals. Also really important is how you index the data. So what you're treating as, quote unquote, like the document in your keyword search backend, we found that it's absolutely essential if you're searching over code to parse things. And so you can extract specific functions and methods and classes along with the corresponding doc string and treat those as separate entities in your system rather than indexing at the file level or trying to do some more naive chunking. So there's the keyword search.

• 12:54Speaker 01

  We also have embeddings-based search or dense vector search, where you basically run those same documents, those functions and symbols in code through an LLM, take out the kind of internal representation, the embeddings vector, and then do nearest neighbor search against that. There's a couple of other techniques that can use to surface, you know, relevant context to, like matching file names and things like that. Anyways, you have this basket of underlying kind of retrievers. And the goal of retrievers is just to get Preserve 100% recall. So make sure you don't miss anything. But also get the candidate result set down to a size where you can use a fancier method to bump the really relevant stuff up in the context window.

• 13:40Speaker 01

  And that's where the second layer of the architecture comes into play. And the second layer is the reranking layer. Again, if you're implementing a search engine, this is how you do it, right? Like you have, after your layer ones proposed, like all the candidates up, you have a fancier, you know, reranking layer that would be too slow to invoke across the entire document corpus. But once you've kind of scoped it down to a smaller set, you can take the reranker and the purpose of the reranker is really to bump the right result or the most relevant results up to the top. So optimizing for precision over recall. So that's kind of like the general architecture of the search backend that powers Cody.

• 14:18Sarah Guo?

  Awesome. Yeah, I think one of the things that I believe and we believe at Conviction is that people are going to build pipelines that look like search pipelines attached to a large language model in many more domains. And like you should treat that entire, like you guys are building a very sophisticated version here, having worked on search for a while, but that parts beyond the language model itself are quite important. For example, like the embeddings model and your chunking strategy. And they're actually pretty data specific.

• 14:50Speaker 01

  Yep.

• 14:50Sarah Guo?

  Right? We were just talking about this and I think people are going to end up with domain-specific and even fine-tuned embeddings models from companies like Voyage or Inhouse, because there's a, I think there's a lot of headroom on performance there.

• 15:03Speaker 01

  Yep, absolutely. I think the Voyage folks are doing really interesting stuff, working on an embeddings model for code. We're kind of collaborating with them at the moment. They're a really smart set of folks. And I think you're absolutely right. There's so many components in these AI systems that are outside of the quote-unquote "main" language model that are really important. And we found that the most important things are, I mean, really what this comes down to is like a data quality and data processing pipeline, which has been something that people have realized for a long time, right? Like, your model architecture can only go so far if your data is garbage. So you really need a high-quality data pipeline. And that means not only having

• 15:47Speaker 01

  You know, in our domain, high-quality code that can serve as the underlying data to use, but also a way to structure that data in a way where you can maximize your ability to extract signal from noise.

• 15:59Sarah Guo?

  Do you take into account the quality of code in this pipeline in some way? Because, you know, you're working on customer code bases, like if there are anything like the code bases I have interacted with, like there's, you know, there's a variance of quality, but that's the real world. So like, what do you mean by, you know, high-quality code here?

• 16:17Speaker 01

  I mean, we kind of implicitly do right now because we, built into Cody is this notion of like, you know, which code is it referencing? It's going to reference the code in your code base first. And that's probably the most relevant code if you're trying to work on day-to-day tasks in a private code base. We're probably going to release a feature soon. This is something that our customers have requested. Basically, the ability to point Cody at areas of the code base that are better models of what good looks like. We've talked with a lot of enterprise customers where when we say like, hey, you know, Cody has the context of your code base, it will go and do a bunch of code searches when it's generating code for you. Their initial reaction is like, can I tell it to ignore large parts of the code?

• 17:01Speaker 01

  Because there's certain parts of the code where like, yeah, those are anti-bad, we're trying to like deprecate that or migrate away from that pattern. And we're like, yeah, absolutely. That's actually like a very easy thing to do at the the the search layer. And the nice part of this too is when you're doing RAG, you can, you can be very explicit about the the type of information, the type of data you're fetching into the context window. You basically like can give someone like a lever that they could turn on or off or like a slider at query time that kind of controls what you tag in as context. So, you know, maybe sometimes you really do want the full code base as context when you're doing something like a completions or you're just trying to, you know, get something out the door. Other times, maybe you want to be a little bit more thoughtful about what context

• 17:44Speaker 01

  you're attending to because you have another goal in mind. Not only do you want to ship the feature, but you also want to up-level the quality of your code or make it look more like some golden copy you have somewhere in your code base.

• 17:59Sarah Guo?

  You just mentioned completions, and then there's the other sort of user experience model that we've seen, which is chat in terms of how people interact with code generation capabilities.

• 18:09Speaker 01

  Yep.

• 18:10Sarah Guo?

  Where do we go from here? Is it agents? Is it more reliability? What do you want to build Cody into?

• 18:18Speaker 01

  Yeah, so I think there's kind of like the short term and the long term to think about. In the short term, I think there's a ton more surface area in the developer inner loop and kind of like human in the loop use cases.

• 18:29Sarah Guo?

  Sorry, describe what you mean by inner loop.

• 18:31Speaker 01

  When you think about the software development lifecycle, this kind of iterative cycle through which we build software, there's kind of like an inner loop and the outer loop. The outer loop is kind of like the entire ring of like you plan for a feature, you decide what you want to build, you go and actually implement the feature, you write the tests, you submit it to CI, you submit to code review, and then provided you pass all that, then it's time to deploy it in production. Once it's in production, you've got to observe and monitor it and react to any issues that happen along the way. So that's kind of like the outer loop. That sort of happens at the team level or maybe the organizational level. The inner loop is the kind of cycle that a single developer iterates on potentially

• 19:16Speaker 01

  multiple times per day. And this is really the engine of how you iterate to something that is like a working patch that actually delivers the feature. So in one invocation of the outer loop, there's many inner loops that you go through because as a developer, unless you're like a superstar genius who's already written this feature before, the first attempt at implementing new feature, you're going to get a lot of stuff wrong, you're going to kind of like figure stuff out along the way, you're going to acquire more context and realize, oh, there's this other thing that exists that I should be using. And so it's that kind of like learning process that you want to accelerate as much as possible. And so if you look at the landscape of code AI today, the systems that are actually in production and in use.

• 20:01Speaker 01

  They're all inner loop tools. So anything that is in your editor, doing inline completions or chat, that's kind of assisting you in the process of writing the code and assisting you in accelerating your inner loop as a developer. And there's just a ton of opportunity there. And we think of it mainly in terms of, beyond chat and completion, there's these specific use cases that represent forms of toil or are just a little bit tedious or repetitive or just non-creative that we can help accelerate. And so we've broken those out into distinct use cases that map to commands and coding. So there's a command to generate a unit test, informed by the context of your code base. There's a command to generate doc strings.

• 20:46Speaker 01

  There's a command to explain the code, again, pulling in context through the graph and through using code search that we think can be targeted. Basically, these are like laser beams that allows to focus on key pain points in the developer inner loop, things that disrupt you, slow you down and maybe take you out of flow. Ton of stuff there, that's all near term. In the longer term, I think the vision that we and a lot of folks are working toward is, hey, can we get to the point where the system can write the feature itself? The code writes itself, so to speak.

• 21:21Sarah Guo?

  An AI engineer, yeah.

• 21:22Speaker 01

  An AI engineer, exactly. The kind of interface for that, the way we describe it is, can you take an issue description? It's either a bug report or the description of a new feature that you want to add. And can your system generate a pull request or a change set that implements the spec that you provide without a human intervention or human supervision in the actual process of writing the code? And so in the long term, we are working towards that. I think we're still a little bit aways from getting there. There will be kind of like a range of issues that can be supported in terms of complexity, right? Like, there's certain bugs and issues

• 22:06Speaker 01

  that, in whole, are kind of a form of toil. Like, no one wants to do them because it's kind of like busy work, even though it might be really important busy work, like keeping your dependencies up to date and things like that. Those are probably the things that we'll tackle first, be able to completely automate first, and then we'll slowly work our way up towards more sophisticated features.

• 22:27Sarah Guo?

  Migrating database schema.

• 22:29Speaker 01

  Yeah, exactly. There's probably maybe like a two by two you want to draw here between like, how tedious is it and how high stakes is it? And, uh, you know, we'll slowly try to migrate up into the, the, the upper right quadrant.

• 22:43Sarah Guo?

  You don't trust my AI to do that yet? Actually, I, I do want to talk about the constraints because like I've been thinking a bunch about this too. And, um, like one, if you take inspiration from the iterative process of real humans writing code, and I'm like, okay, like, you know, there's, there's pseudo code in my head and I'm going to test something and then I gotta like remember how something works. There's now one, within a small community of people working on this, like, um, increasingly interesting vein of thought, which is like, okay, we're going to invest more in, sometimes people call it system to thinking, or, you know, variations of test time search, like generate more samples and because it is code, do different types of validation, right?

• 23:23Speaker 01

  Yep.

• 23:23Sarah Guo?

  There's another school that's just like, make the model better, right? Like, we don't even need validation. We just need more reasoning, right? I don't know if there's others that you think about, but like, are those the right dimensions of constraint? Like be more right in terms of what we show the end user or just, you know?

• 23:38Speaker 01

  Yeah, yeah, yeah. So, I mean, just to restate what you just said, um, I think that, I think that's a good way to slice it. Um, like the, the two examples you mentioned is like, okay, like is the approach, uh, just integrate validation methods into the kind of like, uh, chain of, of thought and execution. Um, and, and maybe we can get by with like small dumb models as long as like there's a, a feedback loop and, uh, work.

• 24:06Sarah Guo?

  Or what we have today, right?

• 24:07Speaker 01

  Or what we have today.

• 24:09Sarah Guo?

  Yeah.

• 24:10Speaker 01

  And then, uh, another school thought would be like, hey, we, we really need just like much smarter models who, who don't make the same sorts of like stupid mistakes, uh, as, as are made today. Um, I think that's an interesting way to slice it. Another way to slice it that, um, has been kind of top of mind for me is if your goal is issue to pull requests. Uh, one way to do it is you could take a model of whatever size and basically like decompose that task uh down into subtasks. So if you're trying to implement this complex feature, you know, which files do you need to edit? What functions do you add to each file? And what unit tests do you need to validate that functionality? You can kind of keep decomposing it till you're at the level where, you know,

• 24:54Speaker 01

  today's language models can solve that. And then you kind of chain them together, uh, right? So that's, that's kind of one way is kind of like the break it down and then build it from the bottom up. Um, the other way to do this is just, I mean, you could just say like, the first way is wrong, like the first way is how humans do it. But you know, it's not necessarily the case that the, the best way to do it for a machine is the way that humans do it. Another way you could do it is just say like, hey, let's expand the context window of the models that can attend to, you know, a large chunk of the existing code base and then just ask it to generate the diff. And uh, you know, if that is reliable enough, um, it'll probably be unreliable. But if it's reliable enough such that, you know, it works 1% of the time,

• 25:37Speaker 01

  uh, then you could just roll the dice a hundred times. And as long as you have like a validation mechanism, you know, as long as it outputs the unit tests, uh, which you can kind of like quickly review, then you just roll the dice a hundred times. And chances are at least one of them will be correct. And that's the one that you go with. Um, and that's, that's kind of the latter approach is, is kind of the approach that papers like, you know, Alpha Code uh, or systems like Alpha Code take uh, when they're trying to tackle these like programming competition type problems. So the limiting factor in the first approach, the bottom sub approach is uh, what percentage of time does like a single step uh, in, in your, your whole process work? Because you're essentially rolling the dice, you know, N times.

• 26:21Speaker 01

  And if each, your success rate each time is, you know, like 90%, uh, then it's, it's basically like, you know, decays to zero the longer your chain of execution is. So the more steps that are required, the, the more, the exponentially less likely you're going to get all the way to full success is. And right now, I think like the- The fidelity of today's systems far less than 90% for each step. So I think this is the issue that everyone building agents in that way is encountering is like,

• 26:49Sarah Guo?

  You have compounding failure.

• 26:51Speaker 01

  You have compounding failure. And then, I mean, you have a kind of similar issue on the other side of things, which is like, if you're trying to do the alpha code thing, you've gotten that to work decently well for programming language competition style problems. But working, like building a new feature within the context of a large code base, if you try to zero shot it, I think the number of times you'd have to roll the dice would be basically cost prohibitive or time prohibitive. For both approaches, I think context quality can play a key role. Because what we found is, for Cody, for example, when our context fetching engine works, the quality of code generated by Cody, it's like night and day.

• 27:31Speaker 01

  The ability for today's LLMs to kind of like pick up on patterns in the existing code, understand what the existing APIs are, are in use, pick up on like the testing framework they're using, it's like really, really good. And so it raises the kind of like reliability level up from, you know, this is a complete dice roll, we definitely need to keep the human in the loop, to the point where you're like, okay, maybe if we improve this context fetching engine just a little bit more, we can get to the point where we can start chaining these like two, three, four step workflows together into something that works. So I guess the short answer to your question, like, how do we get to more reliable agents? For us, the answer relies heavily on context quality and fetching the right context

• 28:16Speaker 01

  for the right use cases quickly.

• 28:19Sarah Guo?

  Yeah, I guess I have a lot of optimism when, if you look at this as just like a, it's just an engineering problem with a pipeline that has a bunch of different inputs, each of which you can improve. From here, and you're doing trade offs against improvement in any part of that pipeline. And like that, that could include how we turn that natural language issue into something that a model can plan from, or that we decompose, right? To what the context quality is to solve that, to what is the, you know, efficiency trade off of like, go sample new solutions from the language model versus what is the quality of your feedback from runtime evaluation?

• 29:04Sarah Guo?

  And there's different types of feedback you could get.

• 29:06Speaker 01

  Yeah.

• 29:06Sarah Guo?

  Like, I assume that there's like some, for any given level of language model quality, there's like some optimal pipeline. And I think we're like very far from that today, and then all of the dimensions are improving.

• 29:18Speaker 01

  Yeah.

• 29:18Sarah Guo?

  So I still kind of think the AI engineer is going to come sooner than, rather than later.

• 29:22Speaker 01

  Yeah, I'm optimistic. You see very promising signs, especially when, when the context engine works. And I think you raise an interesting point. I think it, it still is a bit of an open question. I, I think maybe the question comes down to like, you know, this system, this AI engineer, how much of the architecture of that system is going to be captured at the model layer? You know, embedded in the, the parameters of some very large neural network, or, or something that looks like a neural network versus how, how much of it is going to be, is going to be in, I guess, a more traditional, uh, uh, software system? Kind of like a, a traditional, uh, you know, boxes and arrows, uh, architecture?

• 30:06Speaker 01

  And, uh, yeah, I, the, my honest question is, I, I'm not exactly sure. Like, it's not, it's not like we don't have any model layer stuff going on at all. It's certainly something that we're interested in. But I think our philosophy is, we always want to do the simplest thing, or like what feels like the simplest thing first. I think, you know, when I was doing machine learning research, there's like a, a principle that I took away. Because doing the simple thing, it establishes a baseline. Like oftentimes, you'll find that like the doing the fancier thing is often sexier. And certainly these days, it's like trendier, right? Because you can kind of claim the mantle like, ah, you know, I made a, I made my own LLM,

• 30:49Speaker 01

  beyond LLM, and you know, I trained it on my own data. Now I'm, I have, you know, AI or ML street cred because I did something at the model layer. But the lesson I took away from, from my research days was really the importance of establishing a baseline because oftentimes, if you do the fancy thing first, you might have something that looks like a good result because it, you know, it's going to work to some degree. But then someone else might come along and do a much dumber, simpler thing, cheaper, and one that can be improved more iteratively. And it's going to work as well or better than, than your solution. There's like many examples of that. I think there was a, the most recent example that comes to mind was, um, there was some

• 31:30Speaker 01

  paper in Nature that was published where some, a research group trained a very large neural network to do like climate prediction, you know, very important problem predicting the weather. It's very tough, right? And the thought was like, you know, using the power of magic of neural networks, we could, we could actually train something to predict the weather. And lo and behold, you know, like it generated good predictions and it was published in Nature. And then a year later, there was another paper that was published in Nature where another research group trained a neural network for this exact same application. But in this case, the neural network was one neuron.

• 32:07Speaker 01

  It was literally just like a single aggregator and that performed as well as, as the gigantic neural net. So it basically established a baseline first. And that was kind of like what informed our initial prioritization of RAG over fine tuning. It's not that we don't think that there's value in fine tuning or there's value in training at the model layer. It's that, you know, RAG helps you establish a baseline. And I think you're still going to want to do RAG anyways. Like even if you have fine tuned models in the mix, RAG is still sort of this like last mile data or context. And so you'll want to do that anyways. So why not do that first and establish a baseline that will actually inform where you want to invest in at the, the kind of training layer?

• 32:53Sarah Guo?

  I absolutely agree with that characterization. And I'd say if you approach RAG first, you'll benefit from improvements at the model layer, internal or external, right?

• 33:06Speaker 01

  Absolutely.

• 33:07Sarah Guo?

  One question for you before we zoom out from some of the technical stuff. Does the offering of small models, like the 7 or 8x7B size that are quite capable, I think surprised a lot of people from Mistral. Do small models that show higher level reasoning change your point of view at all, or how you guys approach this?

• 33:32Speaker 01

  We're very bullish on small models. We've actually integrated Mistral into Cody. You can use Mistral as one of the models in Cody Chat as of last week. And it's just amazing to see the progress on that side. I mean, there's a lot to like about small models. They're cheaper and faster, and if you can make them approach the quality of the larger models for your specific use case, then it's a no-brainer to use them. I think we also like them in the context of completions. The primary model that Cody uses for inline completions right now is StarCoder 7 Billion. And with the benefit of context, that actually matches the performance of larger proprietary models.

• 34:16Speaker 01

  And we're just scratching the tip of what's possible there with context fetching right now. So I think we're very bullish on pushing that boundary up even further. And again, with a smaller model, inference goes much faster and it's also much cheaper, which means we can provide a faster, cheaper product to our users. What's not to like there? I think there is a question with the smaller models, specifically in the context of RAG, because I think there's been some research that shows that the in-context learning ability of large language models is a little bit emergent. It emerges at a certain level of model size, or maybe a certain volume of training data size.

• 35:00Speaker 01

  And if you fine-tune a medium-sized-ish model, sometimes it loses the ability to do effective in-context learning. Because I think the intuition is it's voting more of its parameter space to memorizing the training set, so it can do better road completion, rather than... Have something that approaches kind of like general reasoning ability. So that's something that we're kind of watchful for. And it does mean in certain use cases, chat, for instance, Cody still uses some pretty large models for chat. And we have seen better results with models that have more of a kind of like general reasoning ability because they're able to better take advantage of the context that's fetched in.

• 35:42Sarah Guo?

  We can't, at this time of year, not make predictions. So one is just, you have thought about software development and how to change it for literally a decade now, probably longer since you had to like think about it to start the company. What does it look like five years from now?

• 35:57Speaker 01

  That is a great question. Where my mind goes is, well, I guess to answer where software development will go in the next five years, maybe it's it's kind of informative to look at how it's evolved over the past. There's a seminal work called The Mythical Man Month that was written in the 70s about software development that today, oddly enough, despite all the technological changes, still rings very true. And the core thesis of that book is that software development is this strange beast of knowledge work that's very difficult to measure. The common mistake that people make again and again is to treat it as some sort of

• 36:37Speaker 01

  like factory style work where, you know, commits or lines of code are kind of commodities and the goal is just to try to like ship as many of those widgets out as possible. Whereas, you know, anyone who's spent, you know, a month inside a software development or working as an actual software creator knows that there's such a high variance in in terms of the impact that a line of code can make. You know, you have some features that eat up many lines of code that have very little business impact. And there's also kind of like one line changes that can be game changers for the product that you're building. And so when I look

• 37:20Speaker 01

  Forward at how software development is going to change, I like to place it in the context of solving a lot of the challenges that that book called out in the 70s that still exist today. And I think the core problem of software development is one of coordination and visibility. So to develop the volume of software that we need in today's world requires teams of software developers, often large teams, building complex systems, features that span many layers of stack. And a lot of the slowness and a lot of the pain points and a lot of the toil of software development comes from the task of coordinating

• 38:03Speaker 01

  human labor across all these different pieces among many different people in different areas of specialization and also different incentives at play. And I think the real potential of large language models and AI more generally is to bring more cohesion to that process. And I think the gold standard is to really try to get a team of software developers to operate as if you are of one mind, you know, one really, really insanely intelligent, productive person with kind of a coherence of vision and a unity of goals

• 38:44Speaker 01

  and a clarity of focus. And so there's a couple of ways in which AI can do that. Well, specifically two. One is, you know, working from the bottoms up, making individual developers more productive such that more and more scope of software can be produced by a single human. If a single human brain is producing that software, then of course there will be more of a coherence of vision because it's just you that's building everything and you can kind of ensure there's a consistency of experience and code quality there. The other way of doing this is giving people responsible for the overall execution of software engineering team, you know, the team lead.

• 39:26Speaker 01

  Or an engineering leader, director, visibility into how the codebase is changing, actually helping you keep up to date with the changes that are happening across the area of code that is your responsibility. I don't know of a single director or VP level of engineering today that reads through the entire Git commit log of their codebase, because doing so would be just literally so tedious and so time-consuming that you wouldn't have time for any other parts of the job that are very critical as an engineering leader. But with the benefit of AI, I think now we have a system that can read a lot

• 40:10Speaker 01

  of the code on your behalf and summarize the key bits and sort of grant engineering leaders, at long last, the sort of visibility and transparency into how the system as a whole is evolving, so they can attend to the parts that need attention, and also make visible to all the other people on the team how things are evolving, so that everyone has kind of the context of the overall narrative that you're trying to drive when you're kind of shipping day-to-day and making changes to codebase.

• 40:41Sarah Guo?

  I'm just gonna take this to its logical conclusion, Bang. So like, Brooks's Law from this book was that adding manpower to a late software project makes it later, right? So I think the future is just me and like, you know, like a Jira linear shortcut interface, a really good spec, and like, one sprint later, my AI engineer is done, because I didn't have to communicate with my team. That's it.

• 41:05Speaker 01

  Yeah. If your goal is to build software as it exists today, then yes, I think in the future, a single human will be able to build applications that today require large numbers of people to coordinate. On the other side of things, though, I think that the demand for software, we're nowhere close to reaching the demand for good, high-quality software, and I think human beings have a tendency To take any system or technology that we're given and kind of push it to the limits or stretch it as far as we can. So I think the other thing that's going to happen is that our ambitions as a species

• 41:47Speaker 01

  for building complex sophisticated software are going to kind of grow with the capabilities that we have. And so I still think we will have large teams of software developers in the future. They will just, you know, each individual will be responsible for far more feature scope than they are today. And the system as a whole will be more sophisticated and more powerful. But people will still have to coordinate.

• 42:10Sarah Guo?

  So what do you think will matter in that like future in terms of how, like what software engineers need to know how to do, right? And the little bit of color I'll give you here is we ran this hackathon early in the year for a bunch of talented undergrads who had built like, you know, they're working on startups, who had built like really good machine learning demos or done interesting research or something. So there are people who are like, I learned to code around AI tools, which is a wild idea to me. Right? Like I started on Cursor, it was my first IDE or whatever. And a point of view that was a little surprising to me, and I think like March of this year was like, we just don't need to learn to code anymore. Right? And I'm like, like, how could you say that?

• 42:54Sarah Guo?

  Like, you know, like they don't even teach a garbage collection anymore. Like grumpy old man. Like, where's the CS fundamentals? Like, what do you think people need to know? Like, what will be valued?

• 43:05Speaker 01

  So my take on this, and you know, here's the advice I would give to myself or, you know, a younger sibling or my child, you know, if they were, you know, at that age where they're trying to determine what skills they should invest in. I think coding is still going to be incredibly valuable skill moving forward. I think in the limit, the things that are going to be valuable, that are going to differentiate humans operating in collaboration with AI. If you think about like layers through which software delivers value, you know, at the very top. You have kind of like the product level concerns, the user level concerns, like how do I design the appropriate user experience?

• 43:47Speaker 01

  How do I make this piece of software meet the business objectives that I'm trying to achieve? And then you have at the very bottom, the very low level, okay, like what data structures, what algorithms, what sort of specific things underneath the hood are happening that are going to roll up to the high level goals that I want to achieve? And then you have like a lot of stuff in the middle that is really just mapping the low level capabilities that you're implementing to the high level goals that you're trying to achieve. And I think what AI will do is it will compress the middle because in the middle is really

• 44:27Speaker 01

  just a lot of like abstractions and middleware and other things that are today necessary and today, you know, require a lot of human labor to implement. It's more boilerplatey, it's more tedious, repetitive, non-differentiating, it's more mechanical, but it's all necessary today because you gotta connect the dots from the high level goals to low level functionality. But the actual like creative points, the real linchpins around which software design turns are really going to be the high level goals, like what you're trying to achieve, and then the low level capabilities.

• 45:10Speaker 01

  My maybe a bit contrarian hot take here is that CS fundamentals, if anything, are going to grow in importance. You know, the stuff you learn in the coding boot camp, maybe that gets, you know, automated away, but the fundamentals of knowing, you know, which data structures, what their properties are, how you can compose them creatively into solutions that meet high level goals, that is kind of like the creative essence of software development. And I think humans will have the ability to spend more time connecting those dots in the future because they'll just need less time spent on kind of like that middleware piece.

• 45:53Speaker 01

  So I still think CS fundamentals are very important and also domain expertise. So, you know, if you're trying to build software in a given domain, really understanding like what moves the needle in that domain is going to be really important.

• 46:03Sarah Guo?

  Awesome. Bang, I think we're out of time. It was a great conversation. Thank you so much for doing this.

• 46:08Speaker 01

  Thank you so much for having me. This was really fun.

• 46:12Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.