Flow Engineering and Code Integrity at Scale with Itamar Friedman, CEO of Codium AI

In this episode, Nathan sits down with Itamar Friedman, CEO of Codium AI, the company on the mission to create the code integrity paradigm. They discuss harnessing LLMs for code integrity, task decomposition, workflow engineering, and much more. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api X/SOCIAL: @labenz (Nathan) @itamar_mar (Itamar) @CodiumAI (Sodium) LINKS: Codium: https://www.codium.ai/ Cognitive Revolution Feed: https://cognitiverevolution.ai/ SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api ODF is where top founders get their start. Apply to join the next cohort and go from idea to conviction-fast. ODF has helped over 1000 companies like Traba, Levels and Finch get their start. Is it your turn? Go to http://beondeck.com/revolution to learn more. TIMESTAMPS: (00:00:00) Episode Preview (00:06:32) The evolution of AI in software development: beyond code generation (00:09:51) The shift towards code integrity and its impact (00:14:23) Unveiling Alpha Codium: A leap in AI-assisted code integrity (00:16:53) The broader impact of AI on software development processes (00:17:11) Sponsor break: Oraclel | Omneky (00:18:52) The future of AI in coding: enhancements, testing, and more (00:24:41) AI's role in enhancing code quality and developer efficiency (00:43:38) Exploring the challenges of large codebases (00:44:03) The impact of LLM context growth on code understanding (00:45:19) Innovative approaches to data retention and context computation (00:46:08) Deep dive into repo analysis and future directions (00:42:31) Personal experiences and the quest for efficient code management (00:48:26) Dynamic graph strategies and future features in IDEs (00:49:10) From code testing to advanced flow engineering: a new paradigm (00:52:37) Unveiling Alpha Codium: A leap in coding challenge solutions (01:07:31) Reflecting on workflows (01:23:30) Final thoughts and recommendations for AI enthusiasts The Cognitive Revolution is produced by Turpentine: a media network covering technology, business, and culture. Producer: Vivian Meng Editor: Graham Bessellieu For sponsor or guest inquiries, email: [email protected]

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I say that very soon, even if for sophisticated programs, we will see like higher quality coming from AI. Even in the case we're in a hurry, you don't think a problem, here is my code. Usually like, here's the problem, let's think a bit about how I want to design it. Even if it's not too much, you do a few steps. It doesn't make sense that we will ask a model like prompt, generate for me, even if it's super intelligent, let's let it sink like the same way we do.

• 0:22Nathan Labenz?

  It doesn't seem intuitive to me that you would ask an AI to do something, then immediately take its solution and ask it to critique its own solution. And yet it works.

• 0:30Speaker 00

  One day, we would want the AI to be a team member, a virtual team member, cannot just write code without testing it, without understanding the deeper context. A team member needs to be on the entire three parts of pre-build, build and ship.

• 0:45Nathan Labenz?

  Hello and welcome to Turpentine AI. What's that you say, Turpentine AI? Well, we've got some news for you. After a year and 100 episodes of The Cognitive Revolution, Turpentine is doubling down on AI. We're creating a new dedicated channel for The Cognitive Revolution. Turpentine will be spinning up new AI-focused feeds as well. And this feed, soon to be known as Turpentine AI, will become a shared platform featuring best of content from multiple shows and voices.

• 1:20Nathan Labenz?

  We'll be making this transition gradually over the next month or so, and I'll be posting original content to both the new Cognitive Revolution feed and here on the Turpentine AI feed. So if you want to keep up with all of the latest, definitely make sure to subscribe to the new Cognitive Revolution feed, where I've just posted part two of my recent appearance on the 80,000 Hours podcast as the first full episode. You can find the new feed at the same URL, cognitiverevolution.ai. And there you'll find links to the new YouTube channel, Apple Podcasts and Spotify feeds. And for those of you like me who use an old school podcast app, the Bear RSS feed as well. Please do take a second to make sure you are subscribed to both feeds and stay tuned

• 2:04Nathan Labenz?

  to both channels for lots more original AI content coming soon.

• 2:10Nathan Labenz?

  Today, my guest is Idamar Friedman, co-founder and CEO of Codium AI, a company that's on a mission to make code integrity simple.

• 2:21Nathan Labenz?

  Now, for context, at this point, it's broadly understood that large language models make useful coding assistance. Microsoft GitHub's Copilot was the first commercially successful implementation of a large language model for developers. And it's already been four months since Microsoft announced that it had become a $100 million business all on its own.

• 2:42Nathan Labenz?

  However, code generation and code completion, while they make for amazing and highly inspirational demos, support just one part of the software development lifecycle. Before developers can write code, they must work with teammates to figure out what to build in the first place. And after writing code, they have to test it and integrate it into broader systems and production environments. It turns out, especially as products and teams grow, that the coding itself is often a minority of the total work. And that's where Codium AI comes in.

• 3:16Nathan Labenz?

  Focused on code integrity, Codium is meant to support the rest of the software development process with focus on explaining, testing, improving, and integrating code. These are areas where just about every team could stand to improve. And though the company is only 18 months old, it already serves hundreds of thousands of developers.

• 3:37Nathan Labenz?

  In the first half of this conversation, we discuss Codium's product, starting with the value that I found in it, even as a solo prototype developer who doesn't really have to worry about production issues. And then broadening out to discuss how Codium supports larger teams and their correspondingly larger products. Then, in the second half, we dive into their recent publication, Alpha Codium, which shows how careful task decomposition, that is, the process of breaking down larger projects that language models might struggle with into much smaller tasks that they can do far more reliably, ultimately enables successful workflow automation. The Codium team calls this "flow engineering," short for workflow. And while they do use some techniques that are specific to software development,

• 4:20Nathan Labenz?

  what stuck out to me most about this research is how readily generalizable and applicable it is to many other domains as well. The project is open source, and I definitely encourage anyone building AI-powered workflows to take a close look at their framework, prompts, and insights. Codium's work, as well as that of Google DeepMind, show that large language models are already competitive with human participants in coding competitions. It seems very likely to me, and as you'll hear to Idamar as well, that they will continue to improve and soon achieve elite performance levels. But even as that happens, and in some ways perhaps even more so then than now, it will be critical to ensure that AI-generated code aligns to requirements and performs reliably.

• 5:03Nathan Labenz?

  And with that in mind, I think Codium is extremely well positioned for success and definitely encourage you to check it out. As always, if you're finding value in the show, we appreciate it when you take a moment to share it with friends. This one naturally should go to the software developers in your life. Please don't hesitate to send us any feedback, guest suggestions, or questions. My DMs are open on all of the social media platforms. And remember to visit cognitiverevolution.ai to subscribe to the new Cognitive Revolution feed, where you will find exclusive episodes starting this week. Now, I hope you enjoyed this conversation about harnessing large language models for code integrity with Idamar Friedman of Codium AI.

• 5:44Nathan Labenz?

  Idamar Friedman, founder and CEO of Codium AI. Welcome to the Cognitive Revolution.

• 5:50Speaker 00

  A pleasure to be here. Thank you for inviting me.

• 5:52Nathan Labenz?

  We've got a lot of ground to cover. You guys have built a very interesting product, which I've been playing around with over the last week, and also have some cool new research out, which is pushing the frontiers of what we can get out of language models. And excited to dive into all of that. I guess for starters, you've started this company that focuses on code integrity as opposed to the somewhat larger space of just coding assistance or code generation. I'd love to hear how you understand the impact of AI and large language models on the software development profession as it has already happened, and how you came to focus in on this question of code integrity

• 6:37Nathan Labenz?

  as the area that you wanted to build in.

• 6:40Speaker 00

  As I see it, I think that and as I saw it also in two zero two two, which I call B C before G P T. That code completion and code generation, like per se code generation, like give me, you know, code according to my request or something like a prompt. There's like a first use case, very useful use case, by the way, very fitting for the technology of L L M. It's not, you know, by chance that this is the, I'll call it the M D P P, minimal desirable viable product that was like introduced to us. Uh, but it's the first of many of what we're gonna see in the intelligence software development landscape. When let's say three years, I don't think it will even take take ten years.

• 7:21Speaker 00

  Yeah, and and when I thought about like in two zero two two, like let's say August, like I saw the first signs of Copilot actually working. I thought that like what we're actually getting with code completion is like getting your search inside your I D and even a very smart search instead of Googling it five different like times and each time diving five different times and and stack overflow. Uh, then I get a like transformed result into into my code. And that's why probably we see a decline and continuing decline of stack overflow, et cetera. But but eventually, this is just one of the tasks that we kind of hate doing, like starting to Google.

• 8:02Speaker 00

  But what about testing? What about code reviewing? By the way, each one of the tasks is also like nice. I like sometime to see what people write in stack overflow comments, right? It's just also like a matter of like having a community. But in many cases, I I hate testing or I hate like verifying that the code fits a spec. Checking that I didn't miss miss any of the flows or features that exist in my Jira ticket or whatever, Monday ticket, et cetera. So so basically, I think like. We just seen the early birds, the first sign of how AI is going to impact our software development, life cycle and tools, et cetera. And then coming back to the like end of twenty twenty two, again, BC before ChatGPT.

• 8:47Speaker 00

  I thought that code completion and generation is going to be commoditized. We actually, myself and my team that we started at the end of twenty twenty two, we used ChatGPT three point five API before ChatGPT and we knew it's coming. All of this and we thought it's going to be commoditized. You probably know there are like fifteen tools for code code generation, completion. Copilot is the leading one, but there are others. Some some days soon, very soon, you will even run a model like, you know, locally and it will be good enough, et cetera. But again, when I look on my one hundred percent like time of development, this usually takes like thirty percent the most. And it's not the most painful part, to be frank. Again, so that's why we we saw like we need to focus on integrity.

• 9:29Speaker 00

  And like to just wrap it up, why did we call it integrity and not like testing? Testing is the old way to verify your code intent. There are other ways to do that. For example, if I automatically create a specification for your code, then you did write a specification somewhere else. We can try to match it. Here we go, we check if your code is working as expected without going per se via testing. And and by the way, when the long term and you see signs in our product already. When I check your code, I also want to check best practice and you know, even culture stuff like. I'm talking like not only cultural, like best practices of development, but even culture of the company and we might get to AI alignment. Later on this call, I I can elaborate. So that's why we call it called integrity and not testing.

• 10:13Speaker 00

  That's what I thought of like the most impactful um points.

• 10:17Nathan Labenz?

  Thirty percent, you said, of software developers' time is spent on the actual development of the code, like writing the the sort of happy path. Is that what that? Essentially means?

• 10:30Speaker 00

  Yes, I actually think it's a good number. You know, I'm not talking about the individual freelancer, you know, working by herself on some cool project. I'm talking about like a team of fifty developers, five hundred, et cetera, like, you know, in a real company. Then even a hyper gross company doesn't like in four years old, doesn't have to be like twenty years old or one hundred years old company. There are so many other things you're doing like, uh, by the way, some of them are in the software development regime like our rum, like architecture. Uh, you know, like thinking about how to design your software, et cetera. But actually writing code, I think there is a lot of evidence that it shows that it's much less than fifty percent. And even I think a few researchers researched, I don't remember to quote them, that it is less than thirty, even like, I mean, like much less than thirty.

• 11:16Nathan Labenz?

  It definitely does highlight a big distinction between, and this is like a theme in AI application development in general, right? Like it's easy to create a demo, it's hard to get things to production. That is, you know, when people typically say that, or when I hear it, I typically sort of think of the, just the, you know, the last mile of the AI optimization of trying to get the, you know, the prompts to be reliable and, you know, the edge cases and whatever. There's, there definitely are challenges just in the AI part of the application. But now you're also kind of bringing to the fore the fact that real software that serves like large, you know, customer bases is typically developed by bigger teams with whole, you know, cascading processes, whether it's waterfall or agile.

• 11:59Nathan Labenz?

  And, you know, there's obviously a whole other podcasts that can, you know, dive into the, the fine grain distinctions between all the different ways that people try to structure their software development. But it's a highly structured process and people put a lot into figuring out what exactly are they going to build? How are they going to prioritize it? How are they going to define it? Finally, we get to this, what turns out to be, you know, thirty percent of the work that is what people actually, you know, tend to think about the most, of course, is, you know, that you're sitting there writing code. And then once you've got that done, now you've got all these other steps to. Finally, you know, get that through approval and finally integrate it into a production environment. So that's interesting. I'd never heard that that number is as low as it is.

• 12:41Speaker 00

  But by the way, we even, we even didn't talk about the fact that a lot of the code that you write is more like maintaining code, like changing, refactoring code. It's also part of it. Like a lot of the cool AI demos, especially coding agents, they're all about, yeah, let me create a new amazing website, which is basically five pages of register and some dashboard. But it's usually like, anyway, you could download some like, to be frank, like a GitHub repo that does all that. And, and in like real software, a lot of times after you wrote something, after a week, it's legacy code. Right. And so it's a lot about like fixing that, refactoring that, elaborating on it.

• 13:20Speaker 00

  So like, yeah, that's why like, I, I think like, there's a lot of opportunities here for more important use cases and for, for AI to help. And in addition to code completion.

• 13:31Nathan Labenz?

  Understanding where this sits, these are more that you wanted to get to on kind of the, the criticality of, if I were to break it into three phases, it's like figuring out what to build, doing the initial build and then making sure that that build works and fits with everything else that we've already got going on. But you, you had maybe other points to expand on there.

• 13:51Speaker 00

  I'll make another point really quickly is that one of the reasons we focused on coding AI and code integrity because we believe, and we kind of tried to prove that, and I think we did with the Alpha Codium that we're going to talk about later. It's a groundbreaking, I would say practically, it's like beats like Alpha Code by DeepMind, which we admire. And that's why we call our work Alpha Codium and beats OpenAI on their submission with GPT-4 like by, by tenfold on, on their submission on a code contest. Like it's a benchmark derived from real code competition from code forces. And the entire idea in that work is that we put a lot of our know-how and expertise into the code integrity part.

• 14:33Speaker 00

  That's what makes the code generation part very like reliable and, and does better than the majority of professional competitors there. But, and by the way, it's open sourced and you can reproduce the only, by the way, work out there. From all the three companies that submitted, there are only three companies like actually trying this challenging benchmarks. DeepMind, Google and Open, OpenAI and us, our own is, is only like one submitted, but like. Closing the brackets, what I wanted to say is that the reason we we focus on code integrity because we think it's not only directly related to a few obvious use cases like helping you to test, but we think it's a key factor to enable most, if not all, of the other use cases in the intelligence

• 15:18Speaker 00

  software development life landscape, even in code generation, which is, you know, the same that they in their game like in DeepMind and then mostly OpenAI and GitHub game, even there like AlphaCoing did better because of the integrity. And if you're talking about use cases like ResFactor, that you obviously when you factor something, you need to test it, etc. In many other use cases, the integrity of the code, like testing of the code, is a key factor. That's why we were so focused on that. That's our unique. Yeah, sorry if it's stupid, too bitchy, but I really believe in it. We're looking on the future and we want to like make techniques like impact with AI in addition to code completion.

• 15:58Nathan Labenz?

  It certainly relates to a lot of the things that I think about around just, you know, how are we going to take full advantage of AI while still, you know, making sure that we have it under control. And I mean, under control in kind of every respect, you know, of course, like there's the big picture runaway AI scenarios, which I don't dismiss, but there's also a lot, you know, of kind of smaller scenarios where AI's impacts on the world are not immediately obvious, but could turn out to be quite important. And so I think that that that angle makes a lot of sense. Let's just talk for a minute then about kind of some of the upstream stuff. I use Copilot. I take my code snippets and queries to ChatGPT.

• 16:40Nathan Labenz?

  New startups coming into the space like Cursor that's creating an AI-first IDE. Those, you know, I think are all pretty powerful. How would you describe what you see as the impact of those tools in that? You know, again, I'm shocked by the the smallness of the 30% number. Maybe I shouldn't be, but I but what would you say? How would you describe what you see as changing in, you know, that kind of center activity as these tools are coming online and becoming extremely popular?

• 17:08Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:13Speaker 00

  By the way, last time about the thirty percent, I think like one of the reasons that enterprise, you see them by code completion by mass is because first of all, this thirty percent does matter. And it's also one of the first cases that that actually works really well. It's the most, again, I really like love this use case, but it's like the most trivial that one to apply with this new L L N technology, like L L N were trained to complete the next lines of codes according to a context. And this is exactly what code completion and even generation does. They're less trained and well designed per se purely by themselves to deal with this code. Does this code actually work? And that's like Alpha Codium, we presented more like an agent like more like a flow engineering

• 17:58Speaker 00

  than problem engineering approach. That's why, by the way, the enterprise are really interested in the next use cases. So don't be so surprised if you're already by the end of twenty twenty four, you will use A I in many other cases. We'll probably talk about it later. For example, we have a Git plugin that helps you review pull requests, et cetera. It's like the huge pain and I really, I don't think like you're about to try it, but solves a major pain and things that we hate to do. And the pull request that leaves you to the same things that you love to do. Copilot is amazing. Also, other some code completion tools. I think Chat G P T is amazing a lot because of the interface of the of the chat. And by the way, that's why we introduced chat that is more chat Chat G P T style in our I D plugin.

• 18:43Speaker 00

  And we call it, by the way, by the way, Code You Mate. So not only we have many commands there that are specific for code, which Chat G P T doesn't have, but also we enable like pure free text like Chat G P T. But but we do enable you to mark context more easily than Chat G P T. You need to copy it. So also recommend you to try. And then I fully agree with you that are amazing startup. I think you mentioned Cursor, we really love them. By the way, it's the first time I'm mentioning it. We we are experimenting with coming up with our own I D as well. And I see why it's like such a big promise because you can then choose how you want. It's much more native to integrate like some of the L L N features. Having said that, there is a reason that we didn't do it because again.

• 19:28Speaker 00

  The thing that you were like, if I could say like slightly shocked about is that coding is thirty percent. And and that most of these solutions are focused on code generation. And we're focused on code integrity, which is through the entire software life cycle. You want to check your test your code while you're coding. You want to test your code while you're merging. You want to test your code in your pipeline, like in the I C D. And and and et cetera, and there is more point, by the way. And if you build a new I D, you really need to be focused on that. And we didn't want to put that in there as their main focus, more on the entire software life cycle. So I definitely think that we will see like these tools, they will not disappear in your I D while you're coding.

• 20:13Speaker 00

  They're amazing. And for example, coding man is not coming to to replace that. So that's like how how I see it like we, by the way, like Sequoia did like a nice overview. You mentioned at the beginning that's really like that pre build, build and ship. They did something similar and and I think like one of the things that many startups are focused on one part. Like cursors and they give an environment that enables you to code. But one day we would want the A I to be a team member. Right? And with all our respect, a team member, a virtual team member cannot just write code without testing it, without understanding the deeper context of even going and reading your Jira ticket or whatever, et cetera.

• 20:55Speaker 00

  So that's why like a team member needs to be on the entire three parts of of like the pre build, build and ship in order to really be a virtual team member for you that can do some of some of your work.

• 21:06Nathan Labenz?

  I'll describe my experience with the product in a second, and then you can expand on that. But just one more kind of question on the impact of A I on the software development process and profession today.

• 21:20Nathan Labenz?

  I feel like we hear widely varying reports. I personally am not the world's greatest coder and I would say I probably get a multiple X efficiency boost in many cases, even from just going to chat GPT. Regular listeners will have heard me before talk about coding by analogy where I'll basically take a class or whatever I have, it could even be documentation, but often if it's if it's out of my repository, you know, the way that I implement caching is there and you know, a couple of other things, patterns that I wanted to follow are there. And I say, you know, do a new version for me that does a different thing, but kind of follow these patterns. I have pretty good luck with that.

• 22:01Nathan Labenz?

  I genuinely do think I accomplish a lot of things a multiple X faster. Then you see these more like rigorous studies that come out and say, oh, 30% faster here, very meaningful numbers there, but mostly the head, like those kind of more sober studies would be more incremental than like multiple. And then more recently too, I've seen some of these reports that say that code quality is declining. And again, I don't think that's the case in for me because probably my, you know, bar maybe wasn't so high to begin with, but you know, the, the sort of Gary Marcus position, which I do take seriously is that at least as a possibility, or at least in some contexts is that, yeah, maybe you're able to get stuff done faster, but perhaps you're doing it

• 22:45Nathan Labenz?

  at lower quality, you're building up technical debt and you know, this is perhaps more of a mirage than, you know, it seems. So do you have a perspective on that? I mean, maybe it's everywhere, you know, everything, maybe it's everything in different contexts, but yeah, how do you, how would you break that down?

• 23:02Speaker 00

  Okay, cool. So first I think both philosophies, both opinions are right. I think there are some cases that it's like magical and it does really productive and without reducing the quality. And in some cases it does, like, I don't think it's a conflict. It's a matter, there are so many setups. For example, when you wanna start, and I have a feeling that this is in many of your use cases, start a project from scratch. I think like in most, and by the way, you're not trying to build a product that reaches millions of people and need to be robust. You probably get a really pretty good code. Good, I'm talking about good, not like reasonable. For this case, for these cases, much faster. If you're a company with millions of users or hundreds of millions of users and bugs are really critical and performance is really critical and other things.

• 23:48Speaker 00

  And you're not trying to build something from scratch, but rather like add a feature, etc. You need a lot of context, uh, then maybe the quality actually very likely will be reduced. And that's why the way eventually like people in companies will use like tools that are more chat GPT in the ID that integrated with the context like we offer in Kode Me AI, but that's put that aside. And now I'll tell you what I think more specifically. Uh, yeah, I saw those reports like lately for reduction in code, in code quality. And I try to explain like, I think why there is a difference. For example, in GitHub report, they say like it's twice better quality or something like that. I don't, the matter, the number doesn't matter.

• 24:30Speaker 00

  They tried, like it was a controlled, uh, report, controlled experiments that where they tried to isolate it from scratch, build something, you know? And that thing was actually almost an average request that you could easily like L L N can extract from, from, from the internet. But again, when you have complicated code, uh, and in this case, like I mentioned, yes, I do expect even higher quality uh, than a regular developer. When we're talking sophisticated software, it's not. Having said that, I say that very soon, even for sophisticated programs, we will see like higher quality coming from AI. And now I want to explain a bit about that. So I think the way we designed code completion and code generation.

• 25:13Speaker 00

  Until today, I'm talking, we like, I'm talking about the industry. By the way, I love a lot of our competitors, actually, I see ourself occurring as like a complementary, but you know, that mind-share competition to some extent. I think like most of the system were designed, system one-like thinking. You know what I'm talking about, I'm referencing to Dan Kahnman, a book, system one, system two, like sinking fast and sinking slow. Most of the cases are sinking fast, system one. Hey, here's a bit of a context. Here is a prompt, give me the solution, that's system one. I think we will see, and that's by the way, Alpha Codium, that what we did in Alpha Codium, we will see more of system two. Hey, let's think about it step by step, slowly, and not just of a chain of thought.

• 25:57Speaker 00

  Okay, but actually using various tools, it's like an agent-like. Okay, by the way, open, opening a brackets for a second, you can do agents that are more system one and system two, though. So doing an agent doesn't necessarily mean, I would say that system two is like the properties of that. It's like, like we see on Alpha Codium, like not only reflection and steps, but also including additional information that is relevant in the process. Like when you're thinking, you want to think something deeply, usually you go read a bit something, you go fetch the right information and things like that. And once like tools, even code generators, by the way, not code completion, it's gonna be hard to implement it, code completion, you have to be fast. But when code generation, like, will do system two, I think we, we're in a very good chance

• 26:40Speaker 00

  that even in very complicated code, we'll see higher quality. So even if it's worrying right now, I think it will be solved or like even greatly mitigated by the end of the year, even the end of this year.

• 26:52Nathan Labenz?

  Yeah, things are certainly moving unbelievably quickly. One of the things I am saying over and over again to people right now is, we now have AIs that can reason, plan, and use tools. And then, of course, people are always like, well, but not that well. And I'm like, yeah, but two years ago, they couldn't do it at all. Right. So we should not assume that they're stopping here. So it's a good reminder. You are right about kind of the sort of work that I do. And that I think that's a good diagnosis. You know, I typically do R and D type projects or prototype type projects. I, you know, don't have to worry about all the edge cases most of the time. And I don't have to worry about, you know, uptime or SLAs or, you know,

• 27:34Nathan Labenz?

  all of the, the demands of a, you know, a high volume production environment. Nevertheless, though, you know, there is this gap that I also have between the kind of stuff that I spin up and, you know, what my partners on the development team, you know, for example, at Waymark will be happy to, you know, trust and run with. So I do see a lot of potential, even just for me with the Codium plugin. So here's what I did. I, you know, downloaded it, installed it onto my VS Code as an extension. You've got a quarter million active users, which is not a small number when you're talking about developers.

• 28:10Speaker 00

  And VS Code, by the way, there's also in Jetbrains and the Git plugin.

• 28:13Nathan Labenz?

  I assume that VS Code is like the biggest population, but maybe not.

• 28:17Speaker 00

  Yeah, you're right, but Jetbrains is like, I have to say surprisingly like growing faster for us. I guess I can try to estimate, but maybe less interesting. And try to estimate why, but the Jetbrains is really growing fast for us. Almost like, I think it will suppress like VS Code like in a couple of months.

• 28:34Nathan Labenz?

  Wow, interesting. I guess that would just be that people who are going to use an IDE that isn't like the most standard, industry standard vanilla IDE are more inclined to just embrace new tools generally?

• 28:46Speaker 00

  I think maybe we just invest in it like more than others. I have a few assumptions, like we care about enterprise, right? And VS Code is definitely like also trendy and enterprise, but Jetbrains is also very dominant there in enterprise and we care about that. So we invested in it. Actually, we are going to do some few even more. Bigger releases like in the next couple of months, even one month because we're focused on that. So I guess generally getting high scores in Jetbrains is hard. I think the average scoring in Jetbrains is lower and we have like 4.8 stars there or something like that.

• 29:21Nathan Labenz?

  It's a big world out there. When I see these numbers are always a reminder of just how big the world is. We were talking, we both had a tie to Ann Arbor, Michigan. And for me, the visual that I always keep in mind is the unit of Michigan Stadium, which seats 110,000 people. And I always just imagine that view of being in that stadium when I think of 100,000 people. So in my mental picture, you know, you've got over two Michigan stadiums full of developers just on VS Code. And for me, okay, I'm again, not really subject to the, you know, rigors of, you know, real at scale enterprise software development. But I could definitely see how this is going to add a lot of value to a lot of people because I installed the thing and now I'm

• 30:05Nathan Labenz?

  just kind of poking around, right? I can ask it to explain my code. That's, you know, super useful just in general. I had a lot of success with just clicking generate tests. I thought that was a pretty interesting workflow where it first popped up 20 different things that I might want to test, different behaviors, as well as kind of the first half a dozen tests to actually test those behaviors. And then you've got the ability to run. And I was also interested in run and fix. Run, I was like, okay, yeah, totally. Run and fix. I was like, how do we ensure the integrity of that?

• 30:41Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 30:46Speaker 00

  So I'm very, very happy that these are the commands that we tried because we do have a plenty. By the way, like there is an additional set of commands. There are two sets of commands for us. One that we like frame it as a code assistant, like everything that you're doing right now while you're coding. And another, like it's framed as like a PR assistant, that everything related to the changes. Like if, if I want to analyze my changes only, like, or let's say focusing, like the entrance point is the changes. Of course, then we need to like have a deep context, etc. But, but you can like recap your changes, describe your changes. Rest in peace, description, fix bug 111. You can just call the slash describe and get a really comprehensive description for your pull request, etc.

• 31:29Speaker 00

  So but but I think like if I need to choose the top two, I would choose exactly the the test, the generate test and and and they explain now, I'll I'll share why. But I also want to point out that we also have quick test. You might enjoy that because one of the things we learned is that our generate test, it's it's an advanced feature. Like it even even opened you an advanced panel, right? And it didn't like present you it inside of your ID like code file or even in the chat, it opened you an advanced panel because it's relatively advanced. And if you're looking some more quick, quick and and fun, then then then then you can try to quick test the one, it's it's in the chat. And so you actually were right about the or main value position with the generate test.

• 32:12Speaker 00

  And by the way, the twenty behaviors that are were generated, uh notice that you can click and open sub-behaviors. So you basically have like eighty ways. And by the way, in some functions, classes, it could come fifty ways and some others could come one hundred ways. To to to test your code. So one of the main value position we hear from developers is that, hey, with Cody May, I learned how I want to test my code. Because actually, like I mentioned before, like generating the code itself. It's it's cool, it's fun, but it's and and and it does save me time. But thinking how to how to test my code, that I didn't forget edge cases, etc. This is where things we kind of try to sometimes hate to do, but we know it's important.

• 32:55Speaker 00

  And that's why we focused there on on on behavior analysis. By the way, like you you asked me a few times about the future, so I'll allow myself to say that this is where we're also going to focus on. Right now, the behavior analysis are pretty much like generated from two types of content, like your code and your comments. And soon, if we connect to your Jira or Google Docs, we can extract behaviors that are not necessarily yet in the code or not implemented correctly in your code. What I talked in the beginning, like spec spec matching. So it's definitely like an important area for for us, like. It's like automatically creating BDD like cucumber, et cetera, for you, like a specification.

• 33:36Speaker 00

  And and this is why also explain for us is a very important like a command. Because the way we see seeing is like you have code. We believe in the concept now, I'm gonna like open a break. It's really important for us, really, really. We call it dry by, you know, dry, don't repeat yourself. And until today, developers tried to be dry, like not repeating themselves in code. But what about repeating yourself? When you're writing your user story in Jira and then implementing it and then writing test? All of these three have a lot have a lot in common. So dry by AI means that AI will help you to complete all the three. And not needing to work on all of them from scratch.

• 34:19Speaker 00

  Now you you write everything from scratch on the specification. You write everything from scratch on the test. You write everything from scratch for code. No, you now you can write your code. And and like now in in twenty twenty four, I mean, you can write your code and get the relevant test and only work a bit to complete them. And get the spec or the other way around, write the spec and get the test in the code and and keep it dry and always have them in sync. So so that's why I mentioned like explain for us. If you look at it, we kind of trying to create a read me, right? Like a spec for your code and generate test. It's kind of like the test. And then you have the code and all of that we want to keep dry by AI. So right now it's with the push, you needed to click it, right?

• 35:01Speaker 00

  When the accuracy would be good enough for us and we are aiming there, then we also do that with tool. We will try to keep it aligned for you and even give you alerts if something is wrong. So that's how we see the future there.

• 35:13Nathan Labenz?

  The vision of a more kind of passive, you know, assistant that pops up to help you. As opposed to the one that you engage, which is kind of how it all mostly works today is definitely really interesting. Foreshadowing. I also wanted to flag too, that I got a lot of value from the enhance function. And it honestly was like software tutor kind of for me. You know, I've I've been largely self taught, largely entrepreneurial, you know, making stuff work. I used to go to Stack Overflow a lot. I go a lot less now, but I would go there, you know, a lot of times kind of, you know, with like pretty beginner questions in a lot of different domains. And so, you know, I've been writing Python code for years, but I just hit.

• 35:55Nathan Labenz?

  Enhance and you know, it gave me the enhance and it was like, wow, okay. It's like typing everything for me. These are all the things that I like, should be doing that, you know, I'm just not, you know, even bothering to. It definitely felt like I could learn from this, that my output could be better, that my handoff to the development team, you know, would be smoother. And so I do think this is likely to become a part of what I do, even though it's like, you know, I'm still in this kind of R and D mode and not really accountable for, you know, the code integrity in the way that, you know, the bigger teams are.

• 36:31Speaker 00

  Here, here is the thing, I saw that's interesting about the enhance. So first of all, thank you. I love, I love that you love it. By the way, I want to be clear here, there is no feature that we have that 100% of the developers love, by the way. Developers have a taste. And that actually leads me to, to the next point. Right now, I don't know if you noticed, when you do enhance, you can give it a bit of a prompt, like enhance in a certain way. But think about it, and by the way, in implementation as we speak, part of it is already in the alpha users, like design partners, et cetera. Think about it, if, if you could have a configuration file, by the way, maybe for each repo, saying, what are the principles of the enhancement that you like to get? Like in this repo, for me, enhancement means A, B, C, D. And another repo enhancement mean like E, F, G, or whatever.

• 37:15Speaker 00

  And, and then every time you call enhancement, not only like it gives you a general best practices enhancement, but yours. And think about tech leads being able to influence this for their junior developers, et cetera. And now, if you like to enhance, I also suggest you to improve. And think about like, you get a code improvement and think about like improve that you can, you can impact, you can configure and digest per, per repo. For example, one of our clients, what they do is they, they configured that they, they have like an old API and, and they have a new API, the old one they want to deprecate. And they configure that every time someone developer mistakenly like calls the old API, then it gives an improvement, gives an alert, hey, use this new API.

• 37:58Speaker 00

  For, for Dev Team, this is like almost like a game changer. So, so, so, so this is another, I want to like use this opportunity to share with you how I think the future is coming is that there's also options that AI will learn by itself your best practices in a team best practice. And we're thinking about it, but the first step is actually letting you adjust it by yourself. And that's how how we see it. So right now, what you liked about that enhance, I really loved it. You you loved it. I I could tell you that 50% of developers or so, they hate it. Because they think of enhance in a different way. And and that's why we want to let let them configure what enhance means for them.

• 38:35Nathan Labenz?

  Well, I'm, you know, again, my low bar is such that even the baseline enhances is a win for me.

• 38:41Speaker 00

  It's not, it's not, it's not low or high bar. Like, it's it's taste. It's like best practices. It's like, it's not like better or worse or or juniors or or advanced. It's it's like spacer tab, right?

• 38:54Nathan Labenz?

  Certainly, I don't deny the importance of taste among the, you know, those that have the the true professional chops. One big question I was wondering about, though, as I was using it, is the question of kind of the deep context, right? The what the things that I was doing were mostly like function level, class level, but pretty, you know, contained, right? And and most of the things that I got back were also pretty contained. Test this function, uh, enhance this function. When you get to the PR agent type workflow and, you know, just in general, to kind of continue to add more and more value, it seems like one of, if not the major big leap there, is going to be the challenge of the much bigger context of these bigger projects.

• 39:40Nathan Labenz?

  So in one sense, things are kind of coming to us as implementers of AI, because the context windows have grown dramatically over the last year. You know, trying to do this sort of thing with a 4,000 token window sounds, you know, extremely hard. And at 128,000 tokens, you know, you have like a lot of room to put stuff in there. Doesn't mean it's all going to work. It doesn't mean it's it's trivial, but at least you have a lot more room to play with. What kind of high level principles can you share about how you guys are thinking about the challenge of the much bigger projects and managing that context?

• 40:16Speaker 00

  This is amazing question, by the way. I think it's critical for the future of intelligence offered in life cycle, the entire idea of being able to, you know, understand. Big repos and multi repos, it sounds similar, like some for clients, they have like a huge monorepo of 10 million lines and how is that different than, for example, 10 repos of 1 million? There is a difference, but I don't think we that interesting. But the challenge of dealing with lots of liner code is critical. I do think that as LLM context grow, really efficiently grow, that will help, but not solve, but it definitely help. I do want to point out, there are some research talking about that the big windows, they have also like

• 41:02Speaker 00

  problem with focus, attention and all that. It's getting better, but it's still, I think it's a problem. Because even the way you order things, even if there's not, they will solve kind of the focus and attention. It's like you still need to flow the process. Again, it throws me to the alpha code. Like if I just give you like, as a human, here is all the code. No, you probably processes work on it and try to organize it. And the same way that you do it in steps, the LLM needs to do it in steps. Yes, there are like integral steps inside the LLM, there are stages, there are blocks. But like Harpati said a few times, like basically calling an iteration is basically adding more blocks of thinking.

• 41:45Speaker 00

  And I think it's critical, even if you like make a better context window. And from our point of view, like Cody and I, we do implement deep context, but not as some of other solutions right now. And here is why we did it differently. We decided that in the beginning, the inception of the product, it was like high level tactics, like I would say on the border of strategic. We want to be a zero data retention company. It means like even on our on prem, like cloud prem solution, even there, we wanted to start with zero data retention. It means that we compute the context on the fly and we do.

• 42:28Speaker 00

  Like we build all of the dependencies, we go until like, we have like a, like kind of a heuristic algorithmic stop where to stop the graph of dependencies and upwards and downwards. But we don't do that upfront and we don't try to look on the entire 10 million repo at runtime. That's impossible. That's what we wanted to do. So we do have like deepish context. And now, after we progressed quite a lot with these capabilities, now we are looking into like ways, and there are quite a few, and needs to be done right to do like digest repo and multi repo. And, and then exploit that background process that needs to happen all the time.

• 43:11Speaker 00

  And then on runtime to, to exploit that, we're, we're, we're working and then it will be even deeper context. By the way, it costs. It costs the environment and, and the company and et cetera. So it needs to be done right. So the cost is going down. Yeah, but, but it's still something that, that costs, but it's not, wasn't the main reason the cost why we didn't do it. Because it's more like we wanted to give the confidence to our customers that, you know, that this is like zero data retention product. And there were, some of them were really scared from, of, of their code being indexed and saved somewhere. Anywhere, even in their own, like, you know, their GitHub is strongly controlled, or GitHub, Gitlab, Bitbucket, et cetera. They're strongly controlled. It's a really mature product.

• 43:55Speaker 00

  And then suddenly you come with a vector database you save somewhere and didn't pass like a lot of times since the inception of it. So we decided strategically to, to wait a bit and now it's getting mature. Well, well, our product and this field of, of let's call it vector databases, et cetera, we're, we're getting into it as well.

• 44:15Nathan Labenz?

  So yeah, this, this context question, I mean, it's, it's one that I have been thinking about a lot personally, because I tried to create a GPT to help me with my, you know, relatively modest, you know, personal R and D repo for Waymark. And it really has not worked, you know? What I've kind of found is that at least so far, I just have to manage the context manually. Loading the whole repository in there, you know, tried a few different ways. It kind of sent me off on this path of like knowledge graphs because I realized that. What it seems to be doing is like matching on a chunk level, loading that stuff into context, but then it's missing all these things that I that I really needs to take advantage of. And I've like tried saying like, no, it's your job to go find the pattern in the,

• 44:58Nathan Labenz?

  you know, you're the one that's supposed to find the caching pattern and use it. Like, I don't want, if I have to do that, then I don't really benefit from the GPT relative to just using, you know, chat GPT and pasting in whatever context is relevant. Do you have any kind of findings or tips or whatever on the sort of graph side? And if I understand correctly, you're doing this all dynamically in the in the IDE at the moment. So you've got code in the plugin that's kind of spidering out, creating a graph. In the future, I guess you're also saying like, yes, this is going to go toward vector database as well, and we'll have chunking. But any thoughts about the, you know, what can we say about a good graph strategy? Because I need, I personally need some, some tips.

• 45:40Speaker 00

  So first I have to say that, man, you're revealing all our near future features. I'll explain what it's going to be. Actually, what I'm going to tell you is already exists, but it's a bit hidden. And then some developers notice it and we got really good feedback about it. So I'll share with you like, I did a spoiler and like, let me focus on that for a second. For one of our features, I think you didn't try it. It's called not, what you try to generate tests. It's like generate tests from scratch. We also have a feature called add me more test, or officially we call it extend test suite. Like if you already have extended, and we did an experiment and there we actually UI UX wise revealed the graph, revealed the, actually, we didn't reveal it as a graph, although it's a good idea to do that. We're thinking about it. It's like,

• 46:26Speaker 00

  sometimes like developers don't like too much UI in their IDE, right? So, um, so right now we did it like a list like of the computed, let's call it sub-context. Like there is the, the direct context, the code under test, the code under explain, the code under command, what you're, you asked about different types of commands, testing, explain, enhance. But then there's the sub-context that the dependencies and all that. This is being revealed to the users. And the interesting thing is that then the user, the developer can change it. You, you, you, you pulled this one, but I don't like it. But hey, what about this something you missed? By the way, even if the AI, like the algorithm were correct. And did bring all the right minimal graph, like a part of the graph.

• 47:11Speaker 00

  Then you might actually want to add more if you want subtly to get like a different test like that covers more areas or different areas. So it's interesting. That's what we're working on to to give it like provide as features in other cases. So one thing I want to say is that I think that AI assistants would benefit. The developer would benefit if the AI assistant will reveal what the underneath computing. By the way, just I think it's true for not only the context, but other things we're thinking about it. How to, you know, I don't want to overload. But if you do like to see what's going on behind the under the hood, here you go. That's like our philosophy looking forward. We did it in some places and not enough. Specifically about about the graph, one of the complicated thing about it is that it language dependent.

• 47:54Speaker 00

  It's like, it's almost like unbelievable for me that there's not like sufficient open source around it. Like there is this, the the tree sitter and many other, sorry, and a couple of other open source out there. But they're not really completing everything we would want to have to have a very terrific graph for each one of the leading thirty languages. Uh, I I can tell you like with relatively high confidence that jetbrains have that. They need it for for the IDEs and GitHub have that for whatever like search, et cetera. And what they do in their in in in VS code, usually what you do is like you you in you you download an extension that does it for you, right? And each language, et cetera.

• 48:34Speaker 00

  But it's not like open sourced enough, like in some one complete framework that gives you all that. So it's not easy to do. That's why you're you're having a hard time. If if it's okay for you not to do that and in in the backend side, then what one of the things you can do is actually use the IDEs APIs and and exploit their graph building. That leads me to one thing I wanted to tell you how we do it. You you said like we're doing an ID. Basically, we're doing it in two two steps. We're doing part of the preprocessing in ID and your and and like a consumer side or sorry, in game client side. And and then a part of it, another part of it on the on the backend.

• 49:12Speaker 00

  It's it's for efficiency, like to to try to like minimize what we send, but also like deeper algorithms could run on the back backend.

• 49:20Nathan Labenz?

  Well, let's get to the research itself then for a little bit. So, you know, for all this talk of context, like this is a very, we're kind of back to a a toy environment, right? With the with this thing, but I I understand it as essentially you and your team. Flexing on your ability to maximize the performance of language models. Is that like a fair characterization? Like you wanted to just show, hey, we are awesome at this. And here's, we're going to go head to head with, you know, some, some well-known publications on, you know, an established benchmark to demonstrate and kind of teach the community, demonstrate that we're awesome at this and, and teach the community some of what we know. Is that basically the, the mindset behind it?

• 50:00Speaker 00

  The word toy, I'm not sure if you related it to the Alpha Codium or the benchmark, but I wanted to relate to both. Like the, the reason we chose a code contest, which is a benchmark designed and, and gathered by DeepMind from mostly code force competition. We chose that because we think this is the most serious coding challenge there is out there. There is like some lead code stuff and human eval includes like coding stuff. They're either saturated, simple, even the lead code hard, et cetera, is like simpler than the code forces in most cases. And, and, and by the way, a lot of them are like shown to be presented in, in the training set. Like, I don't know if you noticed, like GPT four, like really advanced on human eval.

• 50:45Speaker 00

  And then somebody presented that it's, it's doing 100% accurate before some date and zero. And after it shows like kind of almost proof that a lot of the stuff there is in, in the training. And with code forces, the idea is that currently the code contests are like a benchmark is set, but we do see it as something that should always progress as more code forces. Like to add more problems, code contests also should progress with it. And then there is higher chance that the other good, that's not in the training set. But the other good thing is that code forces, like problems have like private tests. It's like, it's like almost like kuggle, you know, though.

• 51:25Speaker 00

  So you're not revealing, like it's hard to, to, to, to, to really find you on it. So that's, that's why we chose it. And I think this is one of the reasons that there's only two companies until today that, that competed on, on the code contest, which is DeepMind and OpenAI. And we're the third, all the rest of the, let's say LLM generators, they usually do on human eval, et cetera. And there is a lot of debate around it, but they're basically around the same numbers. They're getting the same numbers. Now about the Alpha Codium itself, if it's a toy or not, I can tell you that on one hand, it's doing really well on the code forces. And these are really hard problems. Try it out. And and at the same time, yes, these are isolated problems.

• 52:09Speaker 00

  They're not like connected to many other moving parts and and companies, huge software stack, etc. So we are in 2024, step by step, taking many of the concepts that we had in Alpha Codium and integrating it into our Git plugin called PeerAgent and ID plugin called Codimate and by Codeme AI. And and and we are integrating it. And you will see not only like new capabilities, but new interfaces that are more flow engineering. If we talk about Alpha Codium. And so I I do think that that that we will see the impact also in real world and software.

• 52:51Nathan Labenz?

  Yeah, a better word than toy would have been self-contained, I think, because. And another way that it's interesting is that it is, I think, a framework that generalizes probably extremely well beyond code too. So let me just set it up a little bit. I give a little bit more kind of, you know, foundation for folks. And then we can, you can, you know, give me more detail. So first of all, this code contest, as you said, is a standard that humans compete on. So I always like to ask the question like, how good are humans at this task? You know, that's a, that's a great grounding. And it is true that, you know, these, it's, it's pretty easy to create devilishly tricky little problems and people like to compete on these.

• 53:33Nathan Labenz?

  So the kind of prior state of the art before your work came out was DeepMind's Alpha Code. And they published a result, I believe this was maybe even in Nature, that said, hey, we have achieved median human performance among coding contest participants, which is really critical, right? Because I mean, people are always like, well, how do we compare to humans? And it's also which humans are we comparing to? So we are comparing to people who. Participate in coding contests. And with AlphaCode, I think this is about a year ago, they got to the 46th percentile or roughly, you know, the median participant in these coding contests.

• 54:17Nathan Labenz?

  And that's just by successfully executing or succeeding on 25% of the challenges. So these are some, these are some pretty hard problems. If the sort of person who finds it fun to go spend their time doing coding contests is only getting one in four of them correct. Now, to just unpack a little bit more of the structure, you have essentially a problem statement, right? That's kind of like a story problem from like a high school algebra textbook, except, you know, for a coding, expecting a coding solution. And then if I understand correctly, you have both the public tests that you can see, right? That are given to you, like you must pass these tests. And then also the private test, I wasn't clear on, are

• 55:02Nathan Labenz?

  those like, are they so private that they're not published anywhere and you have to like just ping somebody's servers? And so those are on the code forces servers where those tests and they're, and they're literally nowhere on the internet. Is that true?

• 55:15Speaker 00

  That's why it makes it like a really good like Kaggle like competition. So about what you said, I think perfectly correct. I want to elaborate a bit. Okay. So first of all, there was AlphaCode. I'm not even, I think of even more than one year. And then there was AlphaCode 2, which was just recently. One important thing, the first thing I want to relate is that both of them were fine tuned on this specific problem. Like fuck. And, and this, by the way, makes them less general. Like you talked about, if it's like, by the way, I'm fine with calling it a toy because I do agree. It's currently not fitting for real world cases, which is going to be like in, in our products and this year. But as opposed to AlphaCode 1 and 2, that they find you in their model on this competition specifically like fuck, our solution does not.

• 56:00Speaker 00

  It gives like a huge hint that our solution, AlphaCodium, is more generic for other problems. And thi- this is critical. Second, Alpha Code one specifically, like especially, but also Alpha Code two, makes millions of phone line calls to solve one problem. So when they said they reached twenty five, I think it's twenty eight, if I remember correctly. What you're referring to, like in Alpha Code two, they kind of reshuffle the entire competition. They don't say which one they did, they don't share. Like with all respect, I think like they did an Alpha Code one amazing report. Alpha Code two, it looked to me like they've been in a rush. Now, they don't know exactly. They test Alpha Code one on Alpha Code two's like stuff and like shuffle of code contest,

• 56:45Speaker 00

  which is public, but they had like a private now something. And there they twenty five, but the original one at twenty eight. You to get it to twenty eight percent, you call one million calls for one problem. This is impractical. So they fine tuned and one million calls, forget about it. This is not even a toy. This is like, uh, it's like a kind of proof of concept, which is amazing. They inspired us. Yeah, okay. With Alpha Code two, which they fine tuned a new model, which is Jimmy pro like they still fine tune, but they do much better on how many calls they make. They, it's like we did, while we were fourfold better than than Alpha Code one, we were just like twenty, thirty percent less calls than Alpha Code two.

• 57:27Speaker 00

  Having said that, still they fine tune their, their model. It means that now for every competition or every new software or whatever, you need to find your new model. That's like almost impractical. Uh, and while our, our solution is a generic. And you, you usually like, but when, when, one of the things I, I liked about the project, that what we thought is the main contribution here was also recognized by others as the main contribution. It's not always like that. Like you always wait for what's gonna like, how the community is gonna look like and what, what you, what you did. And Kartati, when he reviewed our paper and tweeted about it, he actually related to the fact. That that how well we designed the the flow.

• 58:10Speaker 00

  Now, you can call it chain of thought, you can call it an agent. We decided to call it flow engineering. Because chain of thought, like you can read in the paper React. Like the chain of thought is like one type of flow. And we designed a different type of flow, so we didn't want to confuse. And we don't call it like agent per se, because we didn't want to fall in that buzzword. And so it's basically what we did is like, we did a code oriented. Iteratives iterative fact checking flow engineer, flow engineering. Okay, so we start with the problem. Sometimes the problem is is defined like in such a humanly way. And we wanted to represent it in intermediate way, which human can read.

• 58:55Speaker 00

  But it's much better for L L M to digest. It's like like taking a product description from a product dude from a P M. And like refactoring it to a technical document. And this requires us a few steps. But the interesting thing is that each one of the steps are relatively reliable. Sometimes when you try to do like the best prompt from the problem to the code. Practically, the L L M needs to do so many things in the process that it makes this unreliable. And even with the best prompt, it still will be unreliable. But if you design a flow, like I just described the first part of our flow, which is a few steps to redefine the problem.

• 59:41Speaker 00

  Then each step is more reliable. You get a better a better result. Now, yes, I know what you might be thinking. If you take like from like the original to the end, maybe it's like forty percent accurate. But if you take, even if it's ninety percent accurate, each one of your steps. But if you take zero point nine, multiply zero point nine, multiply zero point nine, you might get less than forty. But but but practically, we have a few techniques that we show in Alpha Codium. How we avoid like, you know, this this situation with what we we call like gathering and injection through the process. In the old days, we might call it self supervision or something like that. So basically, the flow starts with a few steps where we redefine the problem.

• 1:00:25Speaker 00

  And then we offer a few solutions. For this problem and then we then Alpha Codium sorry reflects on it. And then start to generating one or two solution, running the public test if exists. By the way, some of the problem come with zero, if I remember correctly, I hope not. From zero, some with one, two, some with many public tests. And then it's try starts to reflect on it and and and run the code. So it's like specific, it's not like a general chain of thought. It's like it's a flow specific for for for programming. And I want to relate to that and then if you want, like if you take it to to the healthcare or you take it to to, I don't know, like legal. You can anchor your results in different ways. In this case, in software anchoring via testing.

• 1:01:10Speaker 00

  And not only that, the coding, the Alpha Codium then tries to generate additional tests that are edge cases. Because you know, in machine learning, when you wanna like have like a good solution, you need to have the edge cases to to to draw the boundary. And this is what what you also see in coding, I what we talked in an hour ago with the behavior analysis and giving you the edge cases. And then although it's AI generated code, we know how to smartly, naively, it's not so complicated. But smartly to exploit these edge cases, even if they might be wrong. Uh, by choosing like a curriculum learning, which one is accurate enough to conclude and anchor it to use it and anchor. And this is how eventually, uh, like, like after a dozen of of steps, you you reach at a final solution.

• 1:01:54Speaker 00

  So so that's Alpha Codium, well designed for for for solving problems from from a spec, from a definition to you with a few data points to to a solution. And and it is could be generalized to real world product because you do have some data point you can extract. Uh, I can talk about it. Yeah, that that's like a overview. So the name of the paper is From Prompt Engineering to Flow Engineering. Like the the time we needed to work on prompt engineering really reduced to almost five percent. Because once you move step by step and every time you ask for something relatively clear, then the sensitivity of the prompts is being reduced.

• 1:02:35Nathan Labenz?

  I think this notion of relative simplicity of prompts, I went into the GitHub repo and read through the prompts. And I was pretty impressed by just how relatively straightforward they are, pretty brief, pretty easy to read, crisply written. I thought, you know, clearly some care has gone into articulating very clearly what each task is, but the real work. Has been breaking down this macro task into the subtasks in a way that really embodies the best practices or a successful workflow to get from beginning to end. So I thought maybe I'll just read the steps. And I think, again, this is something that you could bring to almost any other area where you might want to apply AI.

• 1:03:20Nathan Labenz?

  I might think of it as, I mean, you call it flow engineering. I might call it workflow engineering. The first challenge that a lot of times they have is that they haven't documented their workflows or they haven't even really thought at all about like, what did they just do the work? But they haven't really taken the step back to think like, okay, what are the steps that I do? And, you know, maybe I'm kind of blurring them together, but, you know, could I really break them down into discrete and kind of clearly distinguishable subtasks? So that is the lesson here, I think. I mean, you can, you can borrow from the Alpha Codium pattern, but you can also do this for your own type of work, whatever you're trying to do. And I think you'll, you'll find pretty good results from it.

• 1:04:02Nathan Labenz?

  Elicit, which is a former guest on the podcast, and I'm a very small, proud investor, also is like a pioneer of doing this kind of thing, you know, just rigorous task decomposition in their case for research. So here's the flow. You've, you've kind of divided into two buckets, but I mean, really it's, it's, it's a pretty linear flow with a couple nodes that you can kind of iterate on. So first you, you know, provide the, the coding problem. The kind of first core step is problem reflection. That's your classic like unpack this, you know, think step by step, but you're not even doing anything yet. You're just kind of unpacking the problem itself, really just, you know, literally thinking about the problem, you know, LLM style. Then in this context, you have the public test.

• 1:04:46Nathan Labenz?

  So then do the same thing about the public tests. Then generate possible solutions. I might also say, describe that as generate possible strategies. Because if I understand correctly, there's no code generated yet at this point. But it's like possible approaches, strategies, you know, ways to think about solving the problem. So you got a few of those. Now rank those solutions. So again, each, each of these is a distinct L N call with a pretty basic, you know, straightforward prompt, but really dialing into doing that narrow part of this overall workflow. Once you've ranked those solutions, now generate additional A I tests.

• 1:05:29Nathan Labenz?

  I think this is, you know, obviously more specific to a programming environment than other things. And, you know, we'll leave it to the, an exercise for the audience to think about what, you know, the, the version of that might be in, in their context. But this, I thought was really interesting too, because the way that this, the, the thing is ultimately going to be evaluated on is, does it pass the private test, right? Like if it passes the private test, it's deemed to be successful. And yet you only have the, the public ones. So, you know, I assume these things are generally kind of designed in a way where, you know, there will be some failure modes that the private tests are supposed to capture that the public tests, you know, would not capture. So you're taking that extra step of saying, okay, well, what other things, you know, even if we were to pass all these tests, what other ways might we fail?

• 1:06:14Nathan Labenz?

  So generating additional A I tests to try to cover those bases. You've got an interesting diagram that kind of shows the, the problem and solution space and how you expand the, the coverage of tests by generating these additional tests. And all that has happened before we get to any code, right? So this is literally just all in the pre-processing, you know, bucket as it's described in the paper. Now we're finally though, having reflected on the problem and, you know, come up with a few different ways to think about solving it and generated new test ideas and, you know, chosen and ranked those, those various candidate strategies. Now we're finally ready to generate some code. So generate some code, see if it passes the test. Actually, there's even a there's a self improvement.

• 1:06:58Nathan Labenz?

  That's another thing I think is pretty interesting is just multiple rounds of improvement even before testing. So generate the code that then feed the generated code back in. You know, can this be improved? Then go on to actually run it. This is the only thing, as far as I understand, that that gets outside of just fully programmatic. You know, or to kind of fully deterministic language of model calls is that we're now actually going to run the code. So we do need access to a runtime for this portion. Right now you either succeed or fail. You can feed in the error messages, work on that again, again, again, until you finally pass all the tests. Then do the same thing on the AI tests. There's an interesting nuance there around like the fact that the tests might be wrong.

• 1:07:42Nathan Labenz?

  Can you give a little bit more color on how you think about that conceptually?

• 1:07:47Speaker 00

  Priscilla, I want to say that you're spot on in some of the insights. Like, think about it as a human, probably this process as a developer is what we do. You don't take a problem like even in the case we're in a hurry, you don't take a problem. Here is my code usually like, like, I'm not saying that we don't 10% of the time or etc. But usually like, here's the problem. Let's think a bit about how I want to design it, even if it's not too much. You do a few steps. It doesn't make sense that we will ask a model like prompt, generate for me, even if it's super intelligent. You know, okay, let's, let's, let's let it sink like the same way we do it. Like that's, that's, that's the core idea here. So yeah, let's think about it. Yeah, what could be the edge case? What could be like, what I'm trying to achieve? How do I break it into points? You know, okay, what could be like a relevant algorithm here? Like, think about it.

• 1:08:30Speaker 00

  You've been asked to do something, you will think, yeah, what's the principal like technical engineering aspect that I need here? Before you start writing the code, at least that you do that. So why wouldn't AI do that? Like, when you get to the tests, basically, the AI might be wrong. And like similar to us, how many times if you do testing, like the world divides into two, the developer world divides into two. Basically, either you don't do testing because you hate it, or you do testing because and you hate it. Right? But basically, like, when you do the testing, hey, how many times it happened to you? You made a test and it was red, it didn't pass. And then you're thinking, is it a problem in my test? Or is it a problem in my code? It happens a lot, right? So why wouldn't it happen with with AI as well? So how how do you do? How do you deal with it?

• 1:09:14Speaker 00

  You kind of reflect about it, right? But by the way, we all also like a bit could be wrongly biased towards accepting green test that pass. And it could be test pass, the test is wrong and the code is wrong. But biased like we would accept that a green test, right? So and then we will put most of our time on the red test. Or is it the problem in the test, problem in the code? And and and basically what you see here is that that what this bias of developers we put into Alpha Codium. When a test is like passing, maybe with a bit of reflection or not, we anchor it. Okay, this this should not only pass now, but also in next revision. But if the test didn't pass, let's reflect if we think we need to fix it.

• 1:09:57Speaker 00

  Or or, you know, generate another one or actually fix the code and try to reflect on it and act accordingly. So basically, you build a growing anchor group of passing tests. And that's concept you can take to other like other fields. Let's say I have some, I don't know, like on the legal part, I have different items I need to comply to. And I write something and I check and somewhere or another, if I comply to three, okay, from now on, I want to comply to them. And and keep that safe and move on to the next one. I mean, maybe not, I didn't give the the best example, but but there there could be like other fields that could definitely enjoy this this. By the way, it's it's a field, sorry, like I'm so excited about this stuff.

• 1:10:39Speaker 00

  So maybe I'm saying too much, but but it's it's relatively like a researched field. We we didn't quote other papers, etc. We don't want to. Make it such an academic work, but curriculum or learning in general like it's a very adjacent or exact feel or so much work there. Do you start with the hard cases? Or you start with the easy cases? Do you start like, you know, so you can apply, there's a lot of form in alpha coding for innovation or a trial and error like to see which other curriculums could work. We chose one according to our like our intuition best practice.

• 1:11:15Nathan Labenz?

  Yeah, okay, cool. Interesting. So I will just summarize then the results. And maybe also give a just at least my understanding of kind of the economic, you know, comparison between the human performance and the AI. So you try this with a few different models at the time of the research, GPT-4, clearly the best model available for coding. I would be very interested to hear what what Code Llama recently released might look like in this context. But the punchline is with a total number of API calls per run through this entire workflow that averages in the sort of 15 to 20 range. And then also allowing the AI to run this

• 1:11:58Nathan Labenz?

  whole process up to five times, you get up to half of the code contest problems solved correctly. That's basically double what Alpha Code reported some time ago, and it's essentially on par with what Alpha Code 2, also out of DeepMind, has reported more recently. And that gets us up to close to, if not all the way to, the 90th percentile of humans. Again, and these are humans that are spending their time competing in code contests.

• 1:12:31Speaker 00

  All other competitors, they mostly did past 10. What past 10 means, they submit 10 times on the code competition, and we submitted five. Why did we choose five? Because we thought about your XUI. Think about like if you are a product and you want to suggest a few options, then we thought a 10 options is a lot. But one is people are will be willing to get like here is top three. What do you think? Which one would you like? Or top five or 10 like? Practically also mean less running time and easier for us to research when we didn't do ten. And by the way, it favors AlphaCode, DeepMind and GPT with their report because they did ten submission, we five. So if we do ten, by the way, we get better results.

• 1:13:13Speaker 00

  So and and these five attempts, in each one of them, there is roughly like three solutions that AlphaCode automatically tries. And in each solution, there is like fifteen calls. So roughly speaking, there is like one hundred calls, which is equivalent to AlphaCode two. When they reach similar results to us, which is thirty percent solving. Okay, so not fifty, it's we with with one hundred calls. We reach to thirty percent of the of the task are being solved. Solved means that all all the private tests are passed, which put it put us like in the fifty one, fifty two percentile. Like the we we beat the majority of, but not the ten, ten percentile.

• 1:13:56Speaker 00

  To to reach the ten percentile, uh we needed to run like more than one thousand calls. And AlphaCode, by the way, it's more towards the million. And and we didn't want to do that because it's not practical. It's like when you want one hundred calls, it's like three minutes and it costs you, let's say, a few bucks. When you need to run five thousand time, for example, to to beat it and the ten percentile of competitors. Then basically it takes you tens of minutes and more more bucks. And we practically wanted to show a practical result that people can reproduce. So we aimed on the, hey, let's give you a tool that lets you beat the majority, but not necessarily the top ten. If you scale the parameters, you will do much, much better.

• 1:14:39Nathan Labenz?

  I think the difference is that I was taking that number off the validation portion as opposed to the test.

• 1:14:45Speaker 00

  Oh, yeah, yeah, you're correct. You you know why I'm relating to the test? Oh, yeah, sorry. So what you said is correct about the validation, but let me tell you why I'm relating to the test. I think it's much more correct or real world or like it's better to use the test. And by the way, noticed that when we, because it it's not available out there and and it's like it's more pure a result. So I related to a test and you're you're right and you related to validation. And but what I said is like everything I said, it's about a test. But why do I relate to a test? Because because I think you can see interesting insights. For example, GPT four results on the validation is like much opens much a bigger gap than other models. And and then then on the test, actually, the gap shrinks.

• 1:15:30Speaker 00

  This is really interesting, in my opinion. Like, it shows the power of the flow and how maybe GPT is trained on so much data that like somehow includes some of the somehow the interpolation of what's going on there, but not the other private one. And by the way, you you mentioned like we we decided to keep things simple and we we presented a result on Deepsec code and GPT three point five and four. Uh, we didn't at that time manage to do an code llama on the version three or or whatever it's called, because it was just released a few days and we released like ten week, uh, ten days ago. Um, but we did test on on more APIs.

• 1:16:10Speaker 00

  And I have to tell you, surprisingly or not, Deepsec is doing relatively well, especially on the on the tests. And so side like it's very generalizable. And so we didn't want to complicate it too too much and we presented that. All but but we might later on in Alpha Code two, when we want to even do more rigorous and not a eight page report, but like sometimes OpenAI DeepMind, they do like seventy pages. I'm not sure we want to go there. Then we'll show much many more, but Deepsec is doing relatively well. There are other cool models. Some other models are really good for chat, but not so good for code. So you know, some of the bedrock, I'm AWS, I don't want to call names. Uh, they're really good for chat, but less for code.

• 1:16:51Speaker 00

  Uh, I'll always have to say that it was a pleasure and we can chat again if you if you want to host me again later on there. Maybe I'll have some exciting things like hinting a new model or things like that.

• 1:17:01Nathan Labenz?

  Sounds great. Well, I'll just let me just run down really quickly a couple of other quick takeaways that jumped out to me. One, the value of YAML as a a format for output. That's definitely one I'm going to be using because I have been using XML and I think I've given that tip plenty of times to people. And I like XML better than I like JSON because I can at least like read, you know, as I'm just manually getting started my XML. But the YAML is like token, fewer tokens, also easy to read, you know, and apparently works really well. So that's an update for me. Couple other just general. Patterns that people can follow from your work, starting with easy, moving toward hard.

• 1:17:42Nathan Labenz?

  You use the phrase knowledge accumulation, also kind of deferring the big decisions for as long in the process as you can, right? So you're doing that reflection, analysis, et cetera, et cetera. And finally, then choosing your strategy and getting into the code. That I definitely think is something that, you know, whether you're drafting legal letters or, you know, any number of things, you can kind of follow, I think, a very similar logic. Asking for small modular functions, that's a code specific thing, but again, makes a ton of sense for a lot of different scenarios, right? Can you both break the task down and, you know, have it break the task down in its response to you? And then finally, immediate self review, just it's, it's not intuitive necessarily. I think we have, maybe should be more intuitive, you know, this is maybe an area where like

• 1:18:26Nathan Labenz?

  more anthropomorphizing would even be better, you know, but like we, for, for whatever reason, it doesn't seem intuitive to me that you would ask an AI to do something, then immediately take its solution and ask it to critique its own solution. Like it just, for whatever reason, seems like that wouldn't, you know, be a way to get a better answer because it's a, you know, whatever, it's an AI. And yet it works. So, you know, ask literally just immediately after generating the solution, have the AI critique its own work with a different prompt. Super interesting stuff there.

• 1:18:55Speaker 00

  100, like a really, really good summary. I would say the, the point to think about a bit, the prompt, because we simplify the steps, like the prompt is less sensitive, but on the reflection part is where to read a bit about in our paper, also in the blog, but the paper talks about like do and don'ts, like kind of how to do a reflection. It could, could be done wrongly. And about the knowledge accumulation, one thing like I said it for, but the knowledge accumulation is not only like how the data is like progressing from one step to another, but somehow, somehow, sometimes it's a skip connection. Like sometimes you take a output that was generated in step number four and inject it again in step, step number seven, because you want to like, have like the, the knowledge

• 1:19:36Speaker 00

  being given almost purely and not after all the digestion that happened on the way. That's how like, kind of trying to prevent the accumulation of the, the probability of mistakes.

• 1:19:47Nathan Labenz?

  Yeah. Knowledge accumulation, hopefully without error accumulation. Well, I definitely would refer people also to the GitHub repository where you can check out all the prompts. Everything is there with, you know, down to the temperature, all the settings, all that good stuff. And I do think this is something that. Is a very practical, easy to digest project that that a lot of people could take inspiration from. So this has been great. I really appreciate it. Any final thoughts or comments before we break?

• 1:20:15Speaker 00

  No, it was really, really, really enjoyable and love your podcast. I think it's a good time to tell me which one do you think is the, if I may ask, the one I must listen to, which which one of the episodes for someone like me that like really likes the detail, the code, software development.

• 1:20:33Nathan Labenz?

  One that I think has been really good for a lot of people is the Tiny Stories episode with two guys from, in fact, one is also Israeli, Ronan Eldon. They are from Microsoft Research and they created this kind of like three-year-old reading level, you know, if that's a thing, but like very kind of basic language, short stories, simple concepts. And then they trained language models on that. And they showed some super interesting results and some really interesting and for me, kind of foundational intuition building results for how language models learn and why. And you get even to micro reasoning skills at like very small models, like just, you know, a few million or a few tens of

• 1:21:17Nathan Labenz?

  millions of parameters. You're already starting to see reasoning skills developing. And why is that? Well, we'll leave it to the episode, but I thought that was a, I learned a lot from that one. This has been a lot of fun. Itamar Friedman, founder and CEO of Codium AI. Thank you for being part of the cognitive revolution.

• 1:21:35Speaker 00

  Thank you. Co-founder. It's always important and my team is the reason for for our success now and for the future. People is everything. Thank you.

• 1:21:44Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.