Code Context is King: Augment’s AI Assistant for Professional Software Engineers, with Guy Gur-Ari

In this episode of the Cognitive Revolution, Guy Gur-Ari, Co-Founder and Chief Scientist at Augment, explores the transformative impact of AI on the software industry. Highlighting Augment's unique approach, Gur-Ari discusses the challenges and solutions associated with integrating AI into large codebases, the nuances of maintaining context in AI-driven coding tools, and the evolving economics of AI-driven businesses. He shares insights on the company's focus on reinforcement learning from developer behaviors, future trends in software development, and offers advice for junior developers entering an AI-enhanced industry. The conversation also touches upon the vital role of user data, the complexities of vector databases, and the potential of agentic flows to revolutionize coding processes. SPONSORS: Box AI: Box AI revolutionizes content management by unlocking the potential of unstructured data. Automate document processing, extract insights, and build custom AI agents using cutting-edge models like OpenAI's GPT-4.5, Google's Gemini 2.0, and Anthropic's Cloud 3.7 Sonnet. Trusted by over 115,000 enterprises, Box AI ensures top-tier security and compliance. Visit https://box.com/ai to transform your business with intelligent content management today Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers. OCI powers industry leaders like Vodafone and Thomson Reuters with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before March 31, 2024 at https://oracle.com/cognitive Shopify: Shopify is revolutionizing online selling with its market-leading checkout system and robust API ecosystem. Its exclusive library of cutting-edge AI apps empowers e-commerce businesses to thrive in a competitive market. Cognitive Revolution listeners can try Shopify for just $1 per month at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (04:38) Introduction and Welcome (04:46) The Software Supernova Series (05:23) Augment's Unique Approach to AI in Software Development (06:16) Challenges in Large Code Bases (07:32) Understanding Augment's Customer Base (09:19) Context Management in AI (11:46) Technical Insights and Blog Highlights (13:16) Context Management and Code Indexing (19:35) Sponsors: Box AI | Oracle Cloud Infrastructure (OCI) (22:40) Developer Workflows and AI Integration (29:04) Vector Databases and Retrieval Systems (Part 1) (33:25) Sponsors: Shopify | NetSuite (36:13) Vector Databases and Retrieval Systems (Part 2) (37:29) Best Practices for Building RAG Applications (50:36) Establishing a Solid Process for Model Evaluation (51:01) Optimizing Experimental Iteration Time (53:10) Exploring Reinforcement Learning from Developer Behaviors (54:14) Challenges and Benefits of User Data in AI (01:05:34) The Economics of Running an AI Company (01:14:16) Future of Software Development and AI Integration (01:18:51) Advice for Junior Developers in the AI Era (01:23:58) Conclusion and Final Thoughts (01:24:50) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, my guest is Guy Girari, co-founder and chief scientist at Augment, a company using the full range of AI strategies from autocomplete to RAG to chatbots to autonomous coding agents to transform the practice of software engineering in large enterprise codebases. While our first episodes in the Software Supernova series looked at vibe coding platforms that allow anyone to prompt their way from zero to a proof of concept or basic app, Augment, which was founded in 2022, back when OpenAI's codex models and early autocomplete tools were still mostly just foreshadowing a very different way to code, is tackling a harder, but potentially more economically transformative, challenge.

• 0:44Nathan Labenz?

  How do you 10x productivity for professional engineers who bring their considerable human expertise to bear on vast, messy, legacy codebases, which often have millions of lines of code spread across multiple projects that can vary in age, coding style, and underlying technical infrastructure? Unlike personal projects where one can often simply copy an entire codebase into Gemini's context window, as Guy explains, the enterprise challenge requires serious technical firepower at all levels of the stack. And so, Augment has spent the last three years deeply exploring multiple different approaches to code understanding and has ultimately developed a sophisticated, retrieval-heavy approach from the ground up. Their RAG stack includes a custom-built vector database

• 1:29Nathan Labenz?

  capable of real-time updates, proprietary retrieval models designed specifically for large codebases, code search that fires on every single keystroke for every single user, custom code generation models trained with a technique they call reinforcement learning from developer behaviors, and multiple different product paradigms for delivering code to users, all of which is intensively optimized for both accuracy and speed and available across a number of the most popular development environments. The results are quite impressive. As you'll hear, Guy reports that he personally hasn't written a line of code in months. These days, the coding agent, which I had the chance to use in preview and which will be released to the public very soon, handles all of that,

• 2:14Nathan Labenz?

  leaving Guy to focus on higher-level issues, including how he and the team can continue to improve the agent so that it can eventually run for extended periods, take on larger projects, and even go beyond explicit user instructions to infer and address unstated needs.

• 2:29Nathan Labenz?

  The economics of the business are fascinating, too. Augment's pricing is pretty conventional today with $30 and $60 a month plans. But Guy was quite candid about the fact that some powered users already cost them a whole lot more than that to serve. And especially as agentic workflows consume more and more compute, pricing in the AI space in general is very much a live question. It helps, of course, to design pricing that aligns company and customer interests, but it's less clear how best to do that, considering that enterprise customers also value stable pricing and predictable costs. The good news for Augment is that having raised some $250 million in investment capital, they do have some time and financial cushion to figure that out.

• 3:10Nathan Labenz?

  There is a ton of technical depth in this episode, but arguably the most valuable part is Guy's super practical, down-to-earth advice for AI builders. While he and the Augment team have repeatedly invented new technology to solve hard problems, he recommends starting new projects simply by creating small evaluation data sets of just 10 to 20 high-quality, hand-labeled examples that you understand deeply and can quickly test new solutions against. And then optimizing for the speed of iteration by pursuing the simplest available strategies first, and then exhausting what's available in the market before building custom solutions in-house. All advice that, as regular listeners will know, I wholeheartedly endorse.

• 3:52Nathan Labenz?

  Toward the end, I asked Guy if Augment is currently hiring junior engineers, and more broadly, what advice he has for today's early career engineers and CS students. His answer, I think you'll agree, reflects the current moment in the software industry: a sense of excitement and opportunity for the foreseeable future, but also a recognition that nobody can see the future more than two to three years out. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends or write a review, and we always welcome your feedback and suggestions, either via our website, cognitiverevolution.ai, or by DMing me anywhere you like. Now, I hope you enjoy this deep dive into the hard tech powering AI coding assistance

• 4:33Nathan Labenz?

  for enterprise software engineers with Guy Girari, co-founder and chief scientist at Augment.

• 4:40Nathan Labenz?

  Guy Girari, co-founder and chief scientist at Augment. Welcome to the Cognitive Revolution.

• 4:45Speaker 01

  Great to be here. Thanks for having me on.

• 4:47Nathan Labenz?

  My pleasure. I'm excited about this. So we've been doing a little series that I'm calling "The Software Supernova," which is just a nod to how much the software industry is changing. And we're coming at that from a bunch of different angles to try to understand it as deeply as possible. I think you're going to provide a really differentiated and interesting angle because a lot of the stuff that we've looked at previously has been kind of people want to create an app out of nothing. And there's a growing number of products out there now that actually can take you quite far if you show up with just an idea and you want to go from prompt to app in seconds is often the promise. But you guys are coming at the software industry from basically the other end,

• 5:29Nathan Labenz?

  which is targeting larger organizations with big code bases and things in production, long-lived projects. And so I think this will be a really interesting compare and contrast to understand the different challenges that that poses and the different solutions that you're bringing to the market. So maybe for starters, give me a little just kind of introduction to the company and that core challenge. I think folks that follow this feed are like paying attention to AI. I'm not actually sure how many have been in larger software organizations and would be familiar with the particular challenges that those organizations face.

• 6:04Speaker 01

  Yeah, for sure. So Augment was founded with that vision of bringing AI to bear on real software engineering challenges that show up when you're working on a large team, when you're working on a large existing codebase. Because those are the challenges that the vast majority of professional developers face day to day. Definitely the larger the organization, the larger the codebase, the less zero to one projects people do and the more ongoing maintenance, still feature development, product development, but all has to work in the context of a large codebase. And our premise was we could see that AI technology was crossing the threshold of becoming useful. So back when Augment was founded, we had autocomplete.

• 6:49Speaker 01

  As a product that was out there, but ChatGPT still did not exist. But we could see how these models were getting rapidly better and we felt that AI could play a big role. And we also felt that as a startup, if we went after these hard problems of allowing software developers to be productive in their code bases, we could differentiate because it requires a lot of context understanding. And I guess that's something we'll dig into more.

• 7:16Nathan Labenz?

  Yeah, there's a lot of, um, a lot of dimensions to the problem. Do you want to give me a little sense of like, how big a typical augment customer is? I mean, you could measure that in employees or, you know, number of repos or lines of code. But, you know, yeah, just how big of organizations are you guys targeting?

• 7:34Speaker 01

  Yeah, so we typically target organizations that have hundreds of developers. We do stretch higher, so we have some customers who have thousands of developers. In terms of how many repos, that really varies from customer to customer. So some customers use monorepos like we do internally. And some customers use many different repositories, maybe one per microservice. So that varies a lot. In terms of lines of code, I think it starts probably with millions of lines and then it goes up from there.

• 8:05Nathan Labenz?

  So that highlights obviously one immediate challenge right off the bat, which is, you know, when I do my own little projects, my default workflow, unless I'm, you know, testing something else or whatever is, I'll usually have the AI write a little script to print the entire code base to a single file. And then, you know, for a while at least, I can just copy that entire file, put it into the context window and ask the AI for help. So I'll take it to, you know, a one pro, maybe to do some planning or Claude. Or now I've got Gemini 2.5, which can take me farther. But, you know, it still maxes out at a million tokens, which is obviously not going to handle the whole code base. What do you feel is kind of?

• 8:45Nathan Labenz?

  Not done well or sort of missed by, maybe take Copilot if we want to pick on one, but you could say like other offerings in the market that are, you know, and maybe you could kind of cash that out to like, what are the frustration points or that, you know, the places where you see developers just kind of like not getting the value that you, you, you know, that augment can in fact deliver. Like what's, what's the, because I mean, we've all seen these sort of like one-shot examples and oh my God, it wrote this function for me and all that kind of stuff. But where does the conventional approach like break down in practical terms?

• 9:21Speaker 01

  Right. So from what we've seen, again, if we're operating inside a codebase that doesn't fit into the prompt and today, even if we have a million tokens of context length, the ratio is roughly 10 to one. So 10 tokens per line of code. And so still that only gets us to 100,000 lines of code, which in industry is still considered a small project. And there are other downsides as the project grows to actually putting all of that in the context that we can talk about separately. But I'd say the problem there is when you're working in a large codebase, you really have to keep in mind, not just the work that you're doing as a developer, what you're focused on, but also the context. And that could be very obvious things like I need to call a few APIs,

• 10:06Speaker 01

  I need to call them correctly, and I need to put in the right parameters in there. And I also need to call them in a way that respects the conventions. Maybe there are multiple ways to call them. Maybe there are multiple ways to achieve the task that I'm trying to achieve. And we want to be respectful of the conventions that are in the codebase. These are all things that if you're a developer inside an organization, if you've worked there for a while, you're already familiar with the right way to do it. But when you ask a model to do it, an AI model, and you don't provide it with all of that context, it's going to struggle. Basically, it's going to give you bad predictions, whether it's a bad completion or a bad chat answer. And so since at AUGMENT, we've prioritized context from the beginning.

• 10:48Speaker 01

  We have full codebase understanding built in by default to every feature. And so if you're getting a completion, it's going to take the context into account, whether it means looking up the function you're calling or looking up other examples of its usage, things like that. If you're asking chat the question like, where is this function that I used six months ago and I can't remember what it was? It's going to search through your whole repository. And with agents, we see that this actually matters even more because when you're trying to get these models to achieve more and more complicated tasks in a codebase, the context becomes ever more important because, simply because there's less supervision from the developer as the agent is working.

• 11:31Nathan Labenz?

  Yeah, context, as Tyler Cowen says, is that which is scarce. He said that before we got into the LLM era, but it feels like it's, you know, 10 times more applicable to the LLM agents that we're all trying to figure out how to make work for us than it ever was for the humans. So I guess I'd love to just dig into sort of how you're making this work. Because I mean, one of the things that has really stood out to me, I guess a couple of things have really stood out to me as I've studied the company and used the product a little bit. One is the blog is outstanding. There's a lot of technical information shared on the blog and that's really an excellent resource for people to get a good sense of what you're doing. And I guess a theme throughout many of those blog posts

• 12:14Nathan Labenz?

  is just really pushing hard on bringing a lot of resources to bear for an individual user. So like one way in which that manifests, I understand, is if I'm not mistaken, like literally every keystroke that I make fires off a thing to the server, which then, you know, begins to search the codebase to try to figure out like, what am I doing right now? Where am I and assemble the useful context? So maybe, you know, let's get into the sort of context management like. You could take this in many different directions, because I know there's a lot to it. But you know, from maybe describe like from the second that I sort of opened the app. And we can also talk a little bit about it's an extension of VS Code, not a fork.

• 12:56Nathan Labenz?

  And there's a whole, you know, sort of debate going on there as to like, what's the right way to go to market? So you take that one even first if you want. But then, you know, when I opened the thing and I like, okay, here's my repository and I'm new. Like what's happening behind the scenes as it's indexing and, you know, getting me ready to like, put me and the app together in a position to like, really use a lot of compute at runtime.

• 13:17Speaker 01

  Right. So we've actually in the beginning, we explored several different approaches to code-based understanding. I think the approach we landed on was the third one that we tried. And each one of these was a multi-month research project to try to figure out, could we make it work? What we landed on at the end could be described as as rag. And so what happens behind the scene is we upload the code. We have our own custom-trained retriever models that we trained for the purpose of code-based understanding. And then we index code using these models. That's what happens when you open augment and it says indexing your code base.

• 13:57Speaker 01

  Once that's done, yes, on every keystroke and on every chat request, we send a request to the model. And part of processing that request is figuring out which parts of the code base are most relevant to show to the model so it can make the best possible prediction for the user. And there is quite a bit of optimization then that goes, the speed optimization that goes into making all of that fast. Because it's one thing to index a large code base in the background. But then it's a whole different story to say, okay, this completion request needs to finish within, let's say of the order of 300 milliseconds. And that needs to account for both retrieving everything that's relevant from the code base

• 14:38Speaker 01

  and actually doing the language model call request to generate the completion. And so we prioritize both quality so that the retriever is good. And end to end, it actually feels like it understands your code base and also speed. Because to us, speed is a super important feature, basically, of the product.

• 14:59Nathan Labenz?

  Yeah, can you, you know, you can obviously calibrate exactly how much you want to share about the details here, although I recently did an episode also with Andrew Lee of Short Wave, and they have a pretty similar approach where, you know, you sign up at the first thing they do is like ingest your entire Gmail history, which can be a lot, right? And then that goes into their database and he's like a database guru. And he told me, and I don't know if you'd feel the same way, he's like, well, yeah, we could pretty much tell all our secrets because by the time anyone like figures out what we've done and tries to recreate it, we'll have like a whole new generation. So I'm not sure if you, you know, feel quite as confident on that dimension and would be willing to share all the current way things work.

• 15:41Nathan Labenz?

  But to the degree that you can, long preface, I love to understand a little bit better, like, how are you chunking code? Because I think people have like broadly kind of come to frustration with RAG where, and I think there's a number of different reasons for this, but sometimes, you know, I mean, you can kind of fail at every step, right? So like, how are you chunking, you know, when you get a hit on a chunk? Are you then expanding out to, you know, make sure you have the surrounding context that's needed? So it's not just that one, you know, function out of a broader class, kind of loaded in isolation, you know, there's context management and then there's effective context management, I guess, is what I'm really trying to get at. So how do you make it not just fast, but actually like good? So it has the right information that it needs?

• 16:25Speaker 01

  Right. So I'm, I probably can't be as open because I do believe that there is quite a bit of secret sauce in what we do. I mean, it is true, getting RAG to work well is very challenging. And as far as I, in my experience so far, getting it to work well on code is even more challenging than in other domains. So just to give an example of why that is, let's say I'm starting to type something. And I'm starting to type a piece of code. And let's say there's enough context there to understand kind of what it is that I'm trying to do. Maybe there's a comment or something, although often there's not even that. And my cursor is sitting there and I'm trying to, you know, we need to get a prediction out of the model.

• 17:10Speaker 01

  And we need to know what pieces of code are relevant to make that completion. This is a very different situation from chat or question answering system, where the user is asked to provide the context for the request, right? You start with an instruction or you start with a question. You have basically a lot of context for understanding what it is that you're going to be looking for in your knowledge base. With code, at least with chat, you kind of have that. But with completions, it's more passive. We kind of are trying to both infer what it is that the developer is trying to do, and then try to figure out which code it is. So just to give an example, let's say that we figured out that we need to call a function,

• 17:54Speaker 01

  or the model figured out that it needs to call a function. Then the question becomes, okay, what pieces of code are most relevant to help the model make that function call correct? We could pull up the function signature. We could pull up example usages of that function. We could pull up other pieces of code that maybe serve as counter examples. Another thing about code bases is they evolve over time, and we kind of see a snapshot of the code base. And so if we're pulling up examples, those examples could be new, or those examples could be obsolete, and they're just left around in the code base. And actually, the developer doesn't want to call them that way. So it's an extremely challenging problem. What I can say is that we use a mix of different techniques.

• 18:38Speaker 01

  We use RAG, we use some amount of static analysis on code. There are multiple models at play to provide the best possible context to the model. And we also let the user steer often, because these systems are not perfect. And so we need a way for the user, especially in chat, to say, okay, I'm actually pointing at this directory. I'm pointing at this file, and this can also kind of indirectly help steer the retriever. Yeah, there are multiple things at play. I'd say on chunking.

• 19:09Speaker 01

  Yeah, there's definitely better and worse ways that you can do it. And it's true that, yeah, code has more structure that you can hang on to. What I can say is this is more, yeah, I can say that improvements in chunking are more on to solve problems that are in the tail. Maybe I can say it like that. If you have strong retrievers and strong models, chunking shouldn't be a blocker. Yeah, I think I can say that.

• 19:37Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:42Nathan Labenz?

  So one thing that jumps out to me there is just how you started with the sort of assumption that the user is like typing code into an IDE in the traditional way. And this maybe can also tie back to like the go to market as an extension of VS Code as opposed to a fork. I'm so AI-pilled myself and just kind of always trying to do two things at once. Usually I'm like trying to accomplish some goal in some project, but then also like learn about the latest AI capabilities or, you know, use augment or whatever, right? So I'm always kind of looking for these two for ones. And I think that probably puts me, I'm realizing, in maybe a very different pattern of behavior than what you typically see.

• 20:26Nathan Labenz?

  So like, as I've been using it over the last few days, I've done it entirely through the sort of chat panel and have, I basically don't really ever, almost never, like actually get in and start like typing, you know, functions myself anymore. Where are people on that today generally though? Like, you know, what is the sort of balance of approaches that you're seeing from, and by the way, like sort of a mid programmer, which is maybe why I'm so, you know, drawn to the chat experience. But, you know, for the pros, are they?

• 21:01Nathan Labenz?

  What's the balance between how many are like, kind of working the old, you know, traditional way of like, being file by file and then getting this assistance, you know, sort of proactively served up to them versus those that are actually saying, okay, I want to like, interact with an AI and have it help me, you know, but I'm going to like, give it an assignment in a sort of chatter or agent type of paradigm?

• 21:20Speaker 01

  Yeah, there's definitely a distribution. So I think when we were talking about completions in chat, we did notice that there do seem to be two camps of developers. And of course, there's a lot of overlap. Like, I doubt there are many people who only use completions or only use chat, but there are certainly developers with a preference to being a lot closer to the code, I would say, who don't even use chat much, but really love completions. And now also next edit, next edit is kind of a way to, okay, you get completions, they might be away from your cursor, they can delete code and edit code, not just add code, but it fits in very nicely with the workflow of developers who kind of want to keep their

• 22:05Speaker 01

  focus on the code. And then we see a lot of developers who really only use chat. I mean, that is fairly common. The thing that's changing now is, as we're building agent mode, you can take another step away from the code and really let the model edit multiple files, run your tasks, and you're kind of taking another step away and supervising everything. And then you can dig into the code when needed. This is something that we've seen. When you work on a large code base, you, we pretty often have to go back to looking at the code and making some changes yourself. That's pretty frequent, which is quite different from the zero to one experience that I think we talked about before. So I

• 22:47Speaker 01

  would say for large code bases, as far as I can tell, most developers are comfortable being in chat a lot of the time. And yes, using completions, next edit, but using chat a lot of the time. The switch or the move to full autonomous agentic flow will take longer. I think agents and models will need to improve before that becomes kind of the default mode for enterprise developers, let's say. It will take longer, but I feel like that's the direction we're going in.

• 23:21Nathan Labenz?

  Yeah, certainly. And is the, you know, this is, it can get into, you know, sort of almost ideological territory very quickly, but I'm sure for you, it's much more a practical question around extension of VS Code or do a fork. Is that a matter of just kind of meeting developers where they're comfortable and like not asking them to change too much? Or are there other, you know, big decision drivers there that have you in the extension paradigm?

• 23:52Speaker 01

  Yes, I think it starts from meeting developers where they are. And so we have a VS Code extension. We also have JetBrains's extension and we have Vim support. This is really about, we don't want to force developers to change how they work. I think with the forks, these are all VS Code forks because VS Code is open source. And so you can say that if you switch from VS Code to a fork, you're not changing your workflow that much. But if you're asking a JetBrains's developer to switch to a VS Code fork, that's a pretty substantial change to their workflow. So that's one. I would say there are also other considerations with the fork, which is that doing a fork means you need to keep track with updates, especially security patches, which then becomes extra maintenance work that you have to do.

• 24:37Speaker 01

  And then especially if you're selling to enterprise, these security considerations can matter. Now, the downside of not having a fork is that there are certain UI things that are harder to do or sometimes impossible to do, although I have to say that with the VS Code API, we've been able to do a lot within VS Code. I don't think this has been like a very substantial limitation. Sometimes we've had to work harder because we can't just go and change the VS Code code. This is another place where I suspect that the more we move to agentic flows, the less we have to do kind of inside the text editor. And once you're building an agent inside VS Code,

• 25:20Speaker 01

  you have a lot of freedom of what to do because you can open panels, you can put web views in there and you have full control over what's happening. So my sense is that this distinction is going to become probably less important over time. But, you know, I can't promise we won't do a fork at some point. There's certainly a trade off there.

• 25:39Nathan Labenz?

  Yeah, that's interesting. I mean, the point about just security and being able to kind of piggyback on, you know, all the hard work that Microsoft has already done to establish trust definitely makes a lot of sense. I've had enough experience with the security review processes at enterprise customers and not nearly as much as you've had, but I've had enough to know that it's not where I want to be spending my time. And, you know, to the degree that you can shorten that process, it certainly has a lot of appeal.

• 26:06Speaker 01

  Yeah, exactly.

• 26:07Nathan Labenz?

  So going back to the just retrieval, and again, you can kind of calibrate your answers however you want, but practical guidance, you know, for other people building their own RAG apps. Do you have a favorite vector database?

• 26:23Speaker 01

  So we actually built our own vector database. I can explain why we did that. There was nothing out there that we found at the time that addressed all of our requirements. So, you know, what do we want the user experience to be, right? We want the user to feel like the model understands their whole code base. And we wanted, we wanted to feel like it understands the current state of the code base, right? So if I just wrote a function in a file or had chat write it for me, and now I ask chat, let's say, okay, implement the tests. Or I go to a test file and I start typing a completion or I start typing a test. We want the model to understand that this is something that I recently did and have that all kind of indexed and available.

• 27:06Speaker 01

  And so that means giving every developer or giving the model a real time view of every developer's code base. So it has to be real time or feel like real time. And it also has to be different for every developer because if I'm a developer on a team, I work on my feature branch, you work on your feature branch. We cannot have those things mix. That's also a security requirement. So that means that in terms of a vector database, you need something that allows almost real time updates to the index, which is already a significant requirement from a vector database. And it also needs to be able to have queries based on different views, right?

• 27:48Speaker 01

  I have a slightly different set of files that I'm retrieving from than you, but we still want to deduplicate. We still want to have like one database that captures a repository and not like deduplicate that for every user on a team. We did not know of a product that did all of that. And there's a technical reason for it. So typically the way Vector databases work. When you query, it's pretty expensive to do a full query every time. And so you apply some kind of statistical algorithm. Maybe you cluster your embeddings and you search in the cluster. I mean, there are all kinds of ways to do that. But taking that kind of approach, or one of these standard approaches, means that it doesn't work well

• 28:29Speaker 01

  both with indexing or updated indexing, because updating the clusters can be expensive, and also views or queries based on views is hard, because if you're doing a statistical query and you only have a subset of the files that you're retrieving from, you might miss them completely when you're doing a statistical query. And so it was certainly a difficult engineering problem to build a vector database for us. And we still keep iterating on it, especially as we have customers that have larger and larger code bases. We need to keep scaling up solutions. So there's a project ongoing right now, addressing scale requirements with repositories for us. But yeah, we ended up building our own.

• 29:12Nathan Labenz?

  It's fascinating. I mean, I guess you started in 2022, is that right?

• 29:17Speaker 01

  Yes.

• 29:18Nathan Labenz?

  Yeah, so I want, and it may or may not be different today. I wonder, you know, this sometimes strikes people as crazy when I float ideas like this, but that almost sounds like a product unto itself. Have you thought about that?

• 29:33Speaker 01

  It's come up. The thing is, when you, it feels like when you build an AI lab and then an AI product on top of that, you run into many things that can become a product on their own. And one of the challenges is to stay focused and have a concrete vision of what we're trying to accomplish. So this comes up, and oh, this could be a product. This comes up of, oh, should we fork? And then, well, who are our users really are? Who are we catering to? And are they going to want to fork? Or are they going to want to prefer their ID? There are questions like this that come up all the time, especially with something like AI, where it's a completely new technology that keeps improving rapidly, and you keep kind of having to keep up with what's happening and make the right bets.

• 30:18Speaker 01

  Short story, yes, I think this could be a product, but we're trying to stay focused on building the best AI assistant we can for developers.

• 30:27Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. Are there any like, you know, I'm guessing you probably haven't kept up with the evolution of other vector databases, but for people who are trying to pick one, you know, because so many people right now are at the stage of like, either we're embarking on a sort of rag app for our business, you know, probably for internal use, or maybe we made one and it's like not quite working well enough, you know, and we want to like, take some, you know, next level step with it. Are there any sort of general guidelines that you would give people for how to make this part of the system work? Like, one that I have in

• 31:10Nathan Labenz?

  mind, it sounds like you kind of have a version of it, is just like, I personally would, I think, almost always insist on some sort of hybrid, like structured query plus vector, as opposed to, you know, for, at the beginning of this like rag wave, people were just like doing pure vector search, and that seemed to be, you know, kind of a mess. So having some ability to do like, classic SQL style where clause, along with the vector similarity, whatever, seems important to me. But I wonder, you know, what would your sort of lessons or guidance for the masses be based on all this experience?

• 31:48Speaker 01

  Yeah. So I would treat this as a research problem and start with, I mean, unless there are like, pretty clear engineering requirements that would preclude, like, we have, I'd say, pretty special requirements, because it all has to be low latency and so on. I don't think most RAG implementations need all of that. And so I would probably start with some off the shelf vector database, and I would focus more on the quality. And I think for the quality, one thing that's pretty important is to have an evaluation data set that you trust. It doesn't have to be a huge data set. You can start with even 10 to 20

• 32:28Speaker 01

  samples labeled by hand. That's how we start most projects. Actually, most research projects will start with collecting 10 to 20 samples labeled by hand. And then start with some baseline, taken off the shelf retriever, whatever is easiest to use, run it on your evaluation and get a baseline of how are we doing? Are we solving 20% of samples? Are we solving 80% of samples? Probably it's going to be somewhere in between. Is that good enough? And start iterating from there and hill climbing on your evaluation dataset. And when the evaluation gets saturated, so basically when you've managed to solve it, expand it, add more samples, make them more diverse, make them harder.

• 33:11Speaker 01

  I think this is coming up with good evaluations and being diligent about running those evaluations is, I would say, in some sense, it's one of the hardest things to do in research, not because the work is so hard, but because it can be pretty tedious. But this is the way to get to good results. And so, starting with that, things become straightforward in terms of, oh, should we just do vector? Should we do structured queries? Well, let's try it. Everything becomes an experiment. Let's try it on the eval set and the eval set will tell us because we're basically reducing the problem to hill climbing on an eval set. That is the ideal situation.

• 33:56Speaker 01

  I will say, yes, certainly doing pure vector, let's say it like this, real world retrieval systems are almost never a single thing. It's almost never, oh, I'll just do embeddings and I'll work on the embeddings really hard and I'll get the best embeddings and they will solve the problem. That almost never happens. It's usually a mix of different techniques. So vector, structured, could be other, any other signal that you can bring to bear on the problem. And the models today are good enough that you can actually throw a lot at them in the context and they will kind of deal with it. And so, in some sense, recall, recall becomes more important than precision.

• 34:41Speaker 01

  You want to make sure that the right chunks are in there in the context. That's really today, with modern models, that is the thing to prioritize. Now, I described like the ideal situation where you can kind of come up with a data set that you trust and all you're doing is hill climbing on that data set. It's very important to start like that, but at some point necessarily your evaluation data set is going to diverge from optimal user experience. And again, I've never seen it. This has happened in every one of the projects I think that we've done. It's really hard to capture user experience and map that to one number, not just because

• 35:24Speaker 01

  there are multiple axes, but also because we don't really know how users use AI products. There's a whole distribution of what they put into the prompt box. There's a whole distribution of what they expect to get out. You can't really boil it down to a number. And so, once you have something, dog fooding is crucial for understanding where you are. And then, once you have users, user feedback is crucial. You have to take all these things into account. So, I would recommend starting with an eval set, but then understanding that you also need these other sources of feedback to iterate. So, that, I think, is a summary, kind of like a quick summary of best practices for how to go about this. Once you're able to reduce all these questions, basically the place you want to get to is

• 36:09Speaker 01

  can you reduce all these questions of what to do, turn them from philosophical questions to experimental questions that you go test, then that's when you can iterate and really boot fast.

• 36:21Nathan Labenz?

  Yeah, that's great. It is striking to me, and for whatever reason, I find myself doing more projects where there isn't quite a ground truth that's so easy to hill climb on. Like with my company, Waymark, we do video creation for small businesses and there's not like a single answer that's like, you know, what is the right or best script or voiceover script, you know, or whatever to, or selection of images for this particular small business. There's definitely better and worse and, you know, sometimes it's like very obvious and other times it's the subject of disagreement. We've definitely had plenty of cases where we like ask two different people and they see, you know, one is better than the other. It's not like unanimous in most cases.

• 37:05Nathan Labenz?

  But it is definitely an enviable, you know, from that perspective of having these sort of almost irreducibly vibey sort of tasks, the idea of being able to just, you know, climb a hill is quite attractive. But in both cases, I do think it's really important for people to keep in mind, you can start and you should start with like a pretty modest sized data set. I have a whole presentation that's much more about the social side than the technical side of like getting your team on the same page on what 10 instances of a task that are really well done look like. And it's amazing to me how often like that ends up becoming the stumbling point. I think it is often because of what you said, it's just kind of tedious. And there's like, you know, they all also like, don't have any chain of thought, you know, which is sometimes

• 37:50Nathan Labenz?

  really helpful if you want to do a, you know, supervised, whatever, I'm on the verge of the soapbox. But yeah, 10 examples, people will take you far. If it's objective, great, even if it's just like a vibe task and you're just demonstrating what a job well done looks like, 10 examples, you know, it's the first place to get. And then from there, you know, world can open up like a lot, lot more.

• 38:12Speaker 01

  Yeah, just to add to that, 100%. So these few examples, the big advantage of having so few before you go to like hundreds, right? If you can, is that you can, you become very familiar with them. You can hold them all in your head. And so the labels, the ground truth is less important. I'd say you can, you can still hill climb on 10 samples, even if the evaluation procedure is completely manual. So I just trained a new model, or I have a new RAG setup, I will run it through the 10 samples, I will run the before model and the after model, and I will compare them by hand. I don't have to have a, I can boil it down to a number,

• 38:55Speaker 01

  but I can also go based on vibe. So I agree with you that the real minimal thing to start with is the 10 samples. The number can, yeah, the number can come later. I totally agree with that. One other thing you said that I thought was worth just reemphasizing too, is the key thing is make sure the model has what it needs. Worry less about.

• 39:19Nathan Labenz?

  Other considerations like distracting it with wrong information. You know, and this is of course evolving quickly because that used to be, I think, a much bigger problem not too long ago. And as you said, modern models, you know, that word modern is important. Everybody should be using modern models, but like our expectations aren't necessarily always keeping up with what the latest models can do. One way I kind of generalize that for people is just like turn your hyper parameters up, you know, in general, like you have a choice typically in a lot of these rag type setups of how many chunks am I going to take? You know, I'm going to take the top end chunks or if I'm going to expand out like how from a chunk, like how much should I expand out? And I'm always kind of telling people from what I see developers doing. They're leaving all those sort of settings too low.

• 40:03Nathan Labenz?

  The, the right thing to do is usually turn them up. Yes, that might make it slightly slower. It, you know, will make it a little more expensive, but you know, it's always like, at least I don't know if I can think of any exception where turning those things up didn't more than pay for itself, even with those marginal cost increases in the sense of the time savings that you get to getting to something that's like working, you know, better. So I don't know if you have any, uh, you know, exceptions you would, you would put on the turn your hyper parameters up rule of thumb, or is that also, I mean, it's from the blog. It does seem like you guys are differently like, how can we sort of jam and use all these things? You know, to roughly the maximum, but yeah, interested in your take on, you know, any nuances you would add to my simple rule.

• 40:48Speaker 01

  No, I, I fully subscribe to that rule. You just have to be aware of the trade off between, um, yeah, latency, cost and quality. It's, it's really as simple as that. If you're okay with the extra latency and cost in rag context, it's just better to show more because these days the models have been trained to deal with it. It didn't used to be the case, like definitely not two years ago, maybe not even one year ago. I'm not sure, but roughly around that time is when models got the rag training to be able to deal with a lot of distracting information. And so that scales really well, and I expect will just continue scaling because the attention mechanism in transformers is basically built to do that.

• 41:33Speaker 01

  It's basically built to sift through all the noise and focus on the relevant parts. And so with sufficient training, it makes sense that it will work. I think I've also seen this bias toward putting less, less in there, especially if you're coming from a background of using models the way they were two years ago or, or before, there were, you had to be a lot more careful with your tokens, but that has changed. I think the place where, so if we're focused on, if we're talking about RAG, then yeah, I think that's just the right answer. And also certainly way easier than on the research because making, improving the recall, so that is improving the ability of the model to find the right chunk within the first like 50

• 42:16Speaker 01

  or 100 or something like that is doable. Improving the recall, if you have fewer chunks, like if it needs to land in like the top 10 or something, or top five, that task becomes exponentially harder, basically, the less context you have. So if you can give it the room to do it, then the research task becomes much easier. I can say the place where adding more context doesn't seem to scale yet is with instructions. So giving the model tons and tons of instructions that you expect it to follow, in my experience so far, this actually doesn't scale that well. And it will start ignoring instructions if there's too much in there, skipping steps you asked it to do, things like that. But that's not a RAG problem.

• 43:00Speaker 01

  That's just a different kind of prompt scaling problem where models are not yet good enough.

• 43:05Nathan Labenz?

  Yeah, okay. It's a good point. How about, you, you mentioned training, you've trained your own retriever models. This is something that I think, I don't want to bias your answer too much, but my sense is that a lot of software engineers, AI engineers are sort of attracted to the notion of, you know, well, and of course, they're not going to pre-train from scratch, but you know, we'll grab some off-the-shelf thing and we'll, you know, customize it for our own purposes. I wonder what guidance you would give people on when that is in fact a good idea. I once made a meme, you know, the bell curve meme of like, you know, what's genius and what's dumb and what's in the middle.

• 43:49Nathan Labenz?

  And I, my thing was on the extremes is just like, just use OpenAI embeddings. And in the middle was, you know, well, we'll do this, you know, complicated thing. We'll fine tune our own, blah, blah, blah, blah, blah. It's worth it for some. You're in that situation where, you know, you're have like, A lot of resources and a very ambitious project. But where do you think it starts to become worth it to take on that sort of challenge versus just like, you know, using the best off the shelf thing you can find?

• 44:18Speaker 01

  Yeah.

• 44:21Speaker 01

  That is a very task dependent question. Yeah, I think, I mean, anytime you do research, I think there is just a general human tendency to reach for complicated solutions too quickly. My recommendation would be to try hard to bias towards simplicity, like really the simpler, the better and rely on your evaluation, either like vibes based evaluation or numerical evaluation to guide you. Yeah, I'd say, again, it's hard to give general answers, right? Because the answers are going to be task specific.

• 45:04Speaker 01

  So I'd rely more on establishing a solid process for finding the right answers for your particular use case. And that starts with an evaluation set that you're comfortable with and that you mostly trust and that you can run through different iterations of your model or your system to test. The other thing that I think is important to optimize for is iteration time. Because the faster you can run experiments, the more likely it is you'll find something that's good enough or something that's better than what you currently have. So experimental iteration time is something that's very much worth thinking about. Taking experimental time down from hours to minutes can have a lot of impact on not

• 45:48Speaker 01

  just how fast you get to a solution, but actually, do you even get to a solution? Because cranking through 100 experiments versus cranking through 10 experiments, the chances of you finding the right thing in those 100 experiments is just much higher. It's kind of like the RAG problem. If you like, what's your chance of finding the right chunk in the top 100 versus top 10 is just much higher. It's also like that with experiments. So if you can afford to run 100 experiments. Then yeah, you're gonna try a hundred different simple things. Maybe you're gonna try SQL style queries. Maybe you're gonna try five different open source models and find that like, maybe you'll try the OpenAI embeddings plus other things. And you'll find that one of them, for some reason that was really hard to predict, actually works better for your use case.

• 46:31Speaker 01

  So I'd prioritize experimental iteration time and being able to actually trust the result of an experiment with an eval set to tell you the answer. And once you've tried some simple things and nothing seems to work, like, I think another thing that process gives you is a kind of feel for, uh, it looks like nothing out there is really doing what I want. Maybe I should start thinking about fine tuning an open source model. Or, oh, okay, this is like, it's not exactly there, but it's kind of close. It's not that far. Probably if I keep going this way, I will be able to make it good enough without fine tuning. That's the sort of information you get by doing a lot of experiments. So my, my, my suggestion would be to do that and then let the experiments

• 47:15Speaker 01

  kind of tell you which way you need to go and when.

• 47:18Nathan Labenz?

  Okay, cool. That's great. There is so much that I wanted to cover and I don't think we're gonna get to all of it. So I'm gonna have to start to pick and choose and then I'll refer folks to the blog for some deeper dives on stuff we don't get to. But what I definitely want to cover is reinforcement learning from developer behaviors. Obviously reinforcement learning on language models in general is having a moment.

• 47:42Nathan Labenz?

  The floor is yours. Tell us about reinforcement learning from developer behaviors.

• 47:46Speaker 01

  Yeah, so one advantage that we have as a company that both does research and builds a product for users is that we are very close to our users. We get feedback from them on Slack, on Discord, and they also send us their data. Now, for enterprise customers, we do not look at that data, of course. Everything is audited and behind access controls and so on. But we do have a free community tier. That's for anyone who wants to use Augment. It could be on open source. It doesn't have to be on open source, but anyone who's comfortable with us looking at their data and also using it for improving our own models. So there's a very clear separation between those two things.

• 48:28Speaker 01

  But on our free tier, certainly we find value from that data because One of the things that's just universally challenging about building AI products is that we don't really know the, let's call it the, the input distribution or the task distribution. What do users want to do? How are they trying to use the product? What do they expect to get out of it? And this is through collecting this data from the free tier, we get a glimpse into that. And in fact, for coding, we get more than a glimpse. So that, that is one of the nice things about coding is that it's quite different from a chat interface, right? In a chat interface, the user asks a question or they assign some task, they give an instruction,

• 49:13Speaker 01

  they get an answer, they can continue steering, but we don't know what the ground truth answer was. By comparison, if they're working on code in their IDE and we follow what's, what's happening in the IDE, we eventually know what they were trying to do because this is where they actually work. And so the way this connects to reinforcement learning is that the idea with reinforcement learning is that you are not just training the model by showing it examples of what to do, you're actually showing it contrasting examples. Every sample contains input and then it contains a better and a worse output.

• 49:54Speaker 01

  And the model learns from that contrast to do better over time. This is a very powerful paradigm because it means that it's not just that there are correct and incorrect answers, there are better and worse answers. And you see that in coding, just like anywhere else, you know, the answer could be correct in the sense that the algorithm is correct, but it could have the wrong style. Maybe that's not what the developer prefers, or maybe the style does not align with the rest of their code base, right? There are actually multiple axes of in which a sample can be better or worse. And that's the kind of signal that reinforcement learning tries to capture. And so we've applied that technique initially to the completions feature.

• 50:37Speaker 01

  I mean, that was kind of our first reinforcement learning project where we use examples from the model and we use what we know from the user in order to improve the model and align it better with what users expect through reinforcement learning. And we ended up with a better completion model because of that. Yeah, because I mentioned that we train retrievers, but we actually also train the generation models that we use for completion and extedit.

• 51:03Nathan Labenz?

  Yeah, there's a lot today. I'm interested in kind of digging into more deeply there. I guess for one thing, like, have you benefited from like DeepSeek and other recent algorithm releases? Has that stuff, I mean, I think it's safe to say, you know, the GRPO broadly has like blown a lot of people's minds. Did it blow your minds? Or did you feel like you already had a pretty good read on what was working such that it wasn't such a revelation for you?

• 51:34Speaker 01

  The algorithm itself, I don't think was, I'd say, yeah, I think there were known problems with existing RL algorithms that every subsequent iteration kind of addressed. So, yeah, I don't think GRPO for us was a, it was more of an iterative improvement, I think. The DeepSeek work was remarkable. I mean, in the sense of, yeah, they actually implemented a chain of thought reasoning training. It was super impressive work and very nice paper. I wish they shared more details on how they did it, but like, I still enjoyed reading the paper.

• 52:11Speaker 01

  We do benefit a lot, so not from that particular work, but I can say we don't train models from scratch. We made a bet very early on that open source models will rapidly become better. And that was at a time where open source models were really not good. This was long before Llama. I previously worked at Google, worked on some of the training of large models. And so it was clear that this is something open source could do. Because there is, at least at the level of training base models, and then also at the level of like now, we also see beyond base models, like actual instruction tuned models. Because there is at this point, almost a playbook for how to do it. And so if you have the resources, and if you have the people who know the basic techniques,

• 52:56Speaker 01

  which are for the large part in the literature, like you can read papers and learn how to do it, then you can train very good models. So we made that bet early on. We don't train from scratch. We do a lot of post-training on models for retrieval and generation. And so we definitely benefit from open source models coming out. And we generally try to keep up with basing our models on the best available open source model that's out there. Yeah.

• 53:23Nathan Labenz?

  So this, I think this particularly caught my interest, this reinforcement learning from developer behaviors, because I've been kind of looking for something like this to emerge for a while. My sense is like, the compute requirements for the reinforcement learning aren't like so crazy. And the data sets don't even have to be so huge. It seems like a lot of sort of, you know, product user bases or just communities potentially in general, if they're like passionate about a certain subject or whatever, could kind of gather enough feedback or behaviors from people to power this sort of thing. I haven't seen too much of it. And I was wondering like, why it wasn't happening? And now it seems like it is happening. I wondered, though, if you could shed any light on where you think it's going.

• 54:07Nathan Labenz?

  In particular, one might think that the fact that there's not like a true, you know, absolutely canonically right answer, as there isn't like a, you know, math problem with a numerical answer, would suggest that maybe this process would sort of top out at like human level and might not go past human level, because how would it go past human level if it's learning from humans? The flip side of that also would be that it would seem like this approach would be very extensible to, you know, reinforcement learning from lawyer behaviors or reinforcement learning from doctor behaviors and other, you know, any, basically anything where you can gather enough data that's like pretty trusted, even if it's not absolute bedrock ground truth.

• 54:53Nathan Labenz?

  So what do you think? Like, is there a top out that we should be thinking about? And is there any sort of limit on the sort of, you know, breadth of how far these sort of approaches could generalize?

• 55:06Speaker 01

  Yeah, so I guess, well, a few thoughts. So first, if we think about the trends we've seen in the beginning, you know, start with GPT-2 and then GPT-3 and the scaling loss trend, you know, what was the trend? It was, there's a whole lot of data out there on the internet. Let's get as much of it as we can, you know. Process it properly, clean it up, filter it, because there's also a lot of garbage out there. But basically, that was the first resource that large language model training reached for. I think at this point, that data resource is more or less exhausted. And so what can we do? There are two things we can do, I think roughly, or three things. Let's say three things we can do. One is synthetic data.

• 55:50Speaker 01

  I mean, we know that we can generate more data out of these models to train new models. So that's certainly one approach. Another approach is pay contractors to give us the data that we need, right? And that's how most, I mean, RLHF works. That's how you train something like ChatGPT, essentially, if you don't, if you don't have any other data sources. And the third is user data. So if you have real user data, if you can figure out how to use data from users who are using your product to do real work, that's in some sense the holy grail. Because that is the, the closest you're ever going to get to the actual distribution of what users are trying to do, because it is what they're actually trying to do.

• 56:32Speaker 01

  So there's no distribution gap in that case, if you can do it well, between the data you're training on and the data you're going to encounter in the wild at test time. Now, I think the reason we haven't seen more of that is that there aren't that many products that are amenable to that. So I think if you mentioned beyond coding, doctors, lawyers, what do they do where you can actually get the ground truth? If they're editing a document, you can get the ground truth, just like you can from, from an ID. If they're using a chat interface, it's a lot harder to get the ground truth. Maybe you can guess at the ground truth, because you know, maybe they tell you, no, no, no, that's not what I meant. Do this or that. And then you could, but that, that, there's a lot more work that you have to do to extract the ground truth from something like that.

• 57:17Speaker 01

  But I do believe that as we exhaust the available information on the internet, user data is just going to become a lot more valuable and people will pay more attention to it. In terms of tapping out, well, uh.

• 57:31Speaker 01

  I would just say that everything we've done so far is trained models based on human data. There's nothing really new here, like all the data from the internet is human-generated data. RLHF is human-generated data. We can automate some of it, so we can throw models in there and let humans supervise at a higher level. But human supervision is always there, so far. In the future, if we want to break away from that, we need some other source of signal, right? We need some other source of reward for these models. That's where I think code is probably the place where it's going to come first, because the thing that's special about code is that you can execute it and you can get feedback from that.

• 58:12Speaker 01

  And so I can see how in the future we'll be able to do, and probably some of it is still happening. Like, if you look at the way DeepSeek was trained, they don't say a lot on how they did it, but they do get feedback from code execution for RL purposes. It's a very natural fit. So for code specifically, I think we'll be able to do a lot more of that. For other domains, we'd have to find something else. And so, you know, if you're asking the model to write a story, or a poem, or an essay,

• 58:41Speaker 01

  how are we going to automatically assign a reward to that if we don't already have a better model that can judge what this model did, right? So that's where, I can't think of a way to go beyond human capabilities there. But when you do have a ground truth that's separate from humans, code execution, maybe for science, this could be experimental validation, things like that, then we'll be able to, yes, at some point, shift away from humans and rely on these other reward signals.

• 59:10Nathan Labenz?

  Have you seen any reward hacking in your reinforcement learning from developer behaviors?

• 59:19Speaker 01

  Oh, oh, in our reinforcement, um, no, no, I can't think of an example. No, there was nothing as spicy as that. The mistakes were... Yeah, just not understanding what the user wanted was just the, I think, the most common mistake. Yeah.

• 59:37Nathan Labenz?

  Okay, well, keep an eye out.

• 59:38Speaker 01

  Yes.

• 59:38Nathan Labenz?

  Uh, we're all looking for reward hacking. Uh, we should be these days, I think.

• 59:42Speaker 01

  Yeah, yeah, yes.

• 59:45Nathan Labenz?

  So let's see, just prioritizing, kind of triaging a little bit.

• 59:50Speaker 01

  I can give shorter answers also, if that helps.

• 59:52Nathan Labenz?

  No, you're doing great. Maybe a minute on just the economics of businesses like this. So, you know, it's public and again, you can go into as much depth as you want. It's public information that you guys have raised 250 ish million dollars. And I looked on LinkedIn, I saw 100 employees. Perhaps not everybody's listed there. Uh, but if I were to just do like, you know, kind of traditional SaaS app math and take a multiplier of employees times some, you know, Bay Area salary and then try to calculate a runway, I get to like a really long runway. And so I guess I'm wondering, kind of, and that's before any revenue, and it sounds like there's quite a bit of revenue. So what are you doing with the money? Like, are we burning a lot on

• 1:00:37Nathan Labenz?

  training models? It sounds like if you're not pre-training, that doesn't seem like it would be. Are you like subsidizing users? I mean, you are subsidizing open source. So, yeah, I guess, you know, maybe to the degree you can, I'd be really interested to hear about the economics and then, you know, a slight extension of that would be, is there a 10x more expensive version of the product that you could dream of or imagine what that might look like?

• 1:01:04Speaker 01

  Yeah, that's a great question. So, yeah, one thing we've learned is that AI is different than, yeah, we're all used to thinking of SaaS businesses and like you're developing your basement, you set up Google Cloud project, you start serving users and it's all very. It's it's pretty cheap, I would say, right? Your main cost is salaries and so on. But AI is more capital intensive. There is the training, but inference is also very expensive. So serving all those requests of every keystroke and then the chat request and then now the agents, it gets expensive quickly. Yeah, there's some amount of subsidizing users like in the free plan. And I think everyone in this space is kind of trying to figure out right now the economic model.

• 1:01:49Speaker 01

  Because I would say, you know, on the one hand, usage is exploding and we can talk more about, I think it's going to accelerate dramatically and we can talk about why that is. Models are getting cheaper, but they're not getting cheap. It's not matching the pace at which the usage is growing in this space. And also, yeah, they're they're getting cheap. I mean, a given model is getting cheaper, but people always want to be on the latest model. And that's not getting cheaper as as quickly. So all these factors combined, it's it's running an AI company. Both if you're training your own models and also if you're not, can get pretty capital intensive.

• 1:02:33Speaker 01

  So that that's the short answer. Now, on a more expensive product. So if you just look at the shift, so we're we're launching our agent feature. The cost of agents compared to chat, for example, is it's a substantial jump. Because with agents, you give it one instruction and then it goes and that will probably generate 10 or more language model calls, including large calls for like editing files and like running whole commands and then parsing their outputs and doing all those things, all from a single user instruction. And that's at a point where we are giving users a single agent in their ID. So it's just going from chat to one agent.

• 1:03:17Speaker 01

  On the other hand, the value is clearly there. So I can say personally, I think I have not written, I've been using our agent. I have not written a line of code in several months. I've, the agent has written a lot of code. I personally have not had to. The value is very obvious with these things and it's super early. I expect usage of agents to explode over the next year and with it the cost. So if you're talking about like a large jump in cost going from chat, let's say, to agents. There's going to be, I expect, a jump that's at least as high once we, once we're able to unlock the full value out of agents.

• 1:03:59Speaker 01

  And I don't think that cost decreases and models are going to keep up with that. And so cost, cost does become, does become a challenge. So the whole thing is just very capital intensive and cost is actually a major factor. Unlike traditional SaaS businesses, I think, where it's not as much of a factor.

• 1:04:16Nathan Labenz?

  Are you managing your own clusters? Are you, are you doing like the actual, you know, buy up all the GPUs and, and manage them in-house? You're not like leasing or renting from some, some other provider?

• 1:04:29Speaker 01

  No, no, we are leasing, we are leasing, yes. We're not in the business of managing data centers. We are leasing the GPUs, yes.

• 1:04:34Nathan Labenz?

  Yeah, gotcha. So does that translate to a higher price point at some point in the future? My rule of thumb has been, I think, companies should expect to spend a thousand dollars a month on AI to augment their employees, no pun intended, in the not-too-distant future. I personally am like probably halfway there just with stuff that I've signed up for. And, you know, then your prices are 30 and 60 bucks a month. And it feels like, if I'm right, you should be probably like five to 10 X-ing those prices. But, you know, I don't know, is that where you think it goes or not?

• 1:05:12Speaker 01

  Uh, I don't have definitive answers. To, to me, these are open questions. There's even a question of should it be a fixed subscription price or more of a consumption model? So, you know, we, we actually started, our current pricing model is a somewhat consumption-based model where we sell credits. And then if you use the product for a given month, if a developer uses the product, they consume a credit. But if they don't, then they don't consume a credit, which is different from the, I think, more common seat-based pricing. Where you just sell seats and you pay no matter if users use it or not. So we already took a step in this direction of consumption-based pricing, which was meant to really align our interests with those of the users. Like you use it, you pay, you don't use it, you don't pay.

• 1:05:55Speaker 01

  And I expect because of the cost, we're probably going to lean, my guess is, more heavily into that model, but I'm not sure yet. So this question of is it going to be a thousand dollars a month? Maybe we'll end up there or maybe we'll end up with a different model that's kind of more aligned with how users actually use it. What I can tell you is that there's a very wide distribution in how users use these things. There are absolutely users who will justify a thousand dollars a month's price point even today. And then there are users who don't, they just don't use it as much. And so I think we and everybody else are kind of trying to figure that out. Let me throw another thing in there. I think right now we're all just thinking about

• 1:06:35Speaker 01

  user-driven agents, let's call them, or interactive agents where the developer is kind of there. Maybe they go get a coffee and come back because it takes the agent a few minutes, but it's like, you know, it's a few minutes and the developer is kind of staying up to speed with what the, what the agent is doing. I think things are going to evolve rapidly over the next year. And it's not even clear to me if that will continue to be the dominant use case. I'm pretty sure we will have agents that run for hours or overnight or over days to accomplish tasks. I'm pretty sure we will have agents that work on non-user triggers. Maybe it's API calls or maybe it's an agent that goes and does code reviews for you and things like that, that are not just run automatically.

• 1:07:18Speaker 01

  If you're in that world, then you're not even talking about per developer pricing exactly anymore, right? You're, you're kind of putting intelligence into a lot of tasks that are not triggered by the user. Or maybe there's a wide variance in the cost of what the user triggered. I think the pricing model is going to kind of have to adjust to that, at least in the short term, until all this stuff becomes super cheap. So yeah, it's a, it's a complicated question and I don't, I don't have a good answer for it. All I can say is it's a good and complicated question that we're definitely thinking about.

• 1:07:47Nathan Labenz?

  Yeah, I think it feels to me like aligning interest with users is like, uh, it's hard to go wrong when you're generally, you know, keeping that in mind is a true north. The thing that I am always kind of allergic to is when I feel like the product is not performing as well as it could for me because I've got some like fixed price and they're trying to keep my, you know, cost to them under that price to maintain a margin. And it sounds like you're not doing that by just basically accepting the fact that you'll have some thousand dollar a month cost users and, you know, kind of figuring you'll, you'll figure that all out later. But yeah, it's definitely, I want to, I want to be able to be that thousand dollar a month user, even if I do have to pay for it.

• 1:08:27Nathan Labenz?

  You know, what frustrates me is when I can't be because I'm kind of locked into like a, you know, more conventional price point. So, okay, time is short. Maybe two more questions if we can fit them both in. One is, you guys have a blog post on why you think RAG, it's multiple predictions, but the one that jumped out most to me is, why you think RAG will trump fine tuning? And here, I wanted to just super quickly sketch an idea that I've been chewing on for what the drop-in knowledge worker of the future might look like and kind of just get your reaction to it. And then the last one is just future of the software industry, like, you know, what should junior developers do as well?

• 1:09:10Nathan Labenz?

  So, drop-in knowledge worker, we've covered the RAG stuff, like, what's hard for the models today? They don't have context. And I always am so, like, I feel bad for them in some ways, because like, when I'm searching through my Gmail or my Drive or my codebase, one huge advantage I have is I kind of know what's in there and I know when I've found it. And in contrast, like, the models today just like, get what they get and they sort of have to do the best with whatever is returned, right? And you can turn up hyperparameters and that helps, but they don't know in general, like, have I found the right thing? Should I, like, keep searching? And so I have this sense that continued pre-training is maybe one way to describe it on a company's proprietary data.

• 1:09:52Nathan Labenz?

  Basically, try to get to the point where the model knows the company from an inside perspective, as well as the models today kind of know the world at large. And then, you know, kind of continue with your post-training, your behavioral refinements. But try to get to a point where the model knows, yes, I actually found what I'm looking for, like, this is the ground truth that I, you know, that I needed to go on this task or not. And therefore, I'm going to kind of keep searching and, you know, maybe using different tools until I actually get there.

• 1:10:24Nathan Labenz?

  What do you think? Does that seem like too far-fetched? Or how would you just generally react to that, that vision of continued pre-training? So I know, yes, this, you know, yes, I found it, or... Or maybe that is not necessary for some other reason that I don't see.

• 1:10:40Speaker 01

  Yeah, so I can say there are a few challenges with that approach. So one, one challenge with the continued training is that even though, so training is very sample inefficient in that you need a lot of data for the model to learn something. And typical company knowledge bases, they're too big to put in the context window, but they are not large in the sense of training data sets. They're actually typically pretty small. So if you think of a typical code repository, it's not a lot of data to train on. And so if you want the model to pick up on what's there, you're going to need to do multiple epochs, probably, train on it multiple times. But then you quickly overfit, which you also don't want to do. You don't want it to just memorize what's in there. You want it to actually learn from it, sort of.

• 1:11:23Speaker 01

  So I'd say one challenge is the amount of data for this to be effective is typically too small. Another challenge is keeping it up to date. So RAG, you can work hard, but then make your RAG solution instantaneous, or you can reduce the delay as much as you want. Training models is, there's more friction there. If you want to do it for email, for example, every user has their own email store. So if you're going to be training a model for every user and keeping it up to date, that's, again, there's not that much data and the logistics of doing that separately for every user is tricky. I think it used, but I would say it, it used to be like, do you do RAG or do you do this? These days, I would honestly try to solve this problem with an agent

• 1:12:08Speaker 01

  who tries several approaches until it thinks it finds the answer. Like, it doesn't have to be just, we'll do retrieval and did it get it or not? You can do a lot more now. And so we've, we've actually built a lot of that into the product. So the more advanced versions are not just like a, a one-shot retriever. We do more to give you the best retrieval quality we can. I would, just in the sense of simplicity, models are so good now that I would reach for solutions like that and try them, definitely before doing fine tuning. Yeah.

• 1:12:40Nathan Labenz?

  Yeah, interesting. So you think you can basically just, in short, like, get good enough performance without ever having a model that really knows, in a confident, intuitive sense like I do, that it actually has found the right thing?

• 1:12:53Speaker 01

  Yes, I think so. And what we're seeing kind of, I'm seeing that, I'm seeing that kind of based on, based on the evidence that, at least how I interpret the evidence that we're seeing. Yeah.

• 1:13:03Nathan Labenz?

  Okay, cool. Um, okay, last one, and it's not a, not an easy one necessarily. You know, we've got people talking about superhuman coders, you know, within this calendar year and the next year, I'm referring to, obviously, Dario there who said that repeatedly recently. I guess I wonder, like, do you buy that? That soon, or, you know, even if you extend the timeline a little bit? And if so, what do you think that means, or what advice would you give to people that especially are like, early in their software career today? From what I see on the internet forums, it seems like people that are just coming out of school with a CS degree are sort of like, yikes, this is not what I thought I signed up for.

• 1:13:46Nathan Labenz?

  And, you know, not everybody can, like, just pivot into being a machine learning all-star. That's like, a great option if it's open to you. But for the rest who are like, I did this because I thought I was gonna have like a nice stable career with like a, you know, solid income and like, you know, never have to worry about my employment status. Where do you think we're going? And what advice would you give them for navigating the challenges that might be coming for them kind of soon?

• 1:14:11Speaker 01

  Yeah, so first, I think it's good to separate the short term from the longer term. So I think in terms of what Dario said, the way I understood it is, if you look at what actually, like, in the near term, you go to a new line of code and you ask, what, what actually generated this line of code I'm looking at? Then his statement very likely is going to be a model or an agent that did that rather than a developer. Yeah, I buy into that. Like, maybe it's not, he said something like three to six months. That's probably too quick from the adoption that we're seeing. But not, not like three years, I don't think. I think like the short, shorter than that. However, that, that doesn't mean that the model decided on its own what to do. It doesn't mean that it supervised its own output. It doesn't mean that it's fully autonomous.

• 1:14:56Speaker 01

  I expect for a long time, they're still going to be a developer there steering the model. And I expect that because this is how I work. This is how I see people who are picking up agents work. You look at their code, the code was almost, once they pick it up, it was almost entirely generated by an agent. But if you took away the human, nothing, nothing good would happen. Like you wouldn't get anything useful out of it. Because the models are, they're nowhere near that good. They're not even good enough to say, here's the product requirements, go build this. Like, we're definitely not there yet. So I think I buy into that statement, but that doesn't mean we don't need software developers. Like in the next year.

• 1:15:37Speaker 01

  Now, in terms of advice, so I have two kids, they're seven and 14. And we're having discussions with the older one on like what, what to go for. My advice is to go for a career that's more tied to the physical world. Could be mechanical engineering or robotics or something like that, where it feels like it will take longer to be disrupted. Because it's very hard for me to predict what software development is going to look like in, in, let's see. Three, four, five, six years, something like that. That's very hard with the rate things are changing. I, I don't know where it's going to land. I think we'll still need developers who understand the system.

• 1:16:20Speaker 01

  Because if you're just vibe coding your enterprise software, you will run into trouble. I mean, I, I can already see it happening with the code that I'm writing. It will get better, but I don't think it's going to get better at that scale. But then still the question is, well, how many developers do we need? And how much software do we need to write? And I don't know. So those are the discussions we're having with my older one. With my younger one, we have a bit more time to figure this out. Maybe at that point we have AGI and everybody can just do art. I don't know, but I'm glad we have a bit more time to figure it out with him. Yeah.

• 1:16:51Nathan Labenz?

  Yeah, well, the local artisanal economy, yeah, could be a beautiful future as long as everybody has their basic needs met. Are you guys like, how does this get operationalized for you in terms of your hiring? Like, are you hiring junior developers at all? You know, is there any on-ramp for somebody out of, you know, a CS program to get into a frontier company like yours?

• 1:17:16Speaker 01

  Yes, there is. Yeah, for sure. We look for excellence. We hire junior and senior developers. I think this is still a time where not, there's gonna be a learning curve in knowing how to extract the value out of these models. And so it's not that everyone immediately becomes, even if you're using agents, getting value out of them takes time, especially in an enterprise environment or in a codebase. Like, we're not, we're, we're small. I wouldn't call us an enterprise, but even in our codebase, which is, let's say, small to medium size, using agents to navigate that codebase requires some skill. So I think for a while, there's going to be ramping up time where it seems that the people who, you know, as always,

• 1:18:00Speaker 01

  people with less experience are also like, quick to, to jump on new technologies. I think we're gonna see a lot of that in the near future. But yeah, short answer is we do, we certainly still hire junior developers.

• 1:18:12Nathan Labenz?

  Great. Anything else you want to leave folks with before we break?

• 1:18:18Speaker 01

  So Augment is, is out. It's really good at understanding your codebase. So I encourage you to download it, give it a try and really feel the power of an excellent AI assistant that, that fits into how you work.

• 1:18:32Nathan Labenz?

  And I would definitely encourage people also to check out the blog for a bunch of deep dives. We didn't even get into the inference optimization work and all the detailed analysis of batch sizes, which I did think was super interesting. And there's a great write-up of the next edit feature as well. So there's plenty more to be unpacked from the Augment team than we've had time for today. But nevertheless, this has been a great conversation. I really appreciate it and look forward to continuing to play with the product. For now, Guy Gourary, co-founder and chief scientist at Augment. Thank you for being part of the cognitive revolution. Thank you so much. This was a lot of fun. Thank you. It is both energizing and enlightening to hear why people listen and learn what they

• 1:19:13Nathan Labenz?

  value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.