Infinite Code Context: AI Coding at Enterprise Scale w/ Blitzy CEO Brian Elliott & CTO Sid Pardeshi

Blitzy founders Brian and Sid break down how their “infinite code context” system lets AI autonomously complete over 80% of major enterprise software projects in days. They dive into their dynamic agent architecture, how they choose and cross-check different models, and why they prioritize advances in AI memory over fine-tuning. The conversation also covers their 20¢/line pricing model, the path to 99%+ autonomous project completion, and what this all means for the future software engineering job market. Sponsors: Blitzy: Blitzy is the autonomous code generation platform that ingests millions of lines of code to accelerate enterprise software development by up to 5x with premium, spec-driven output. Schedule a strategy session with their AI solutions consultants at https://blitzy.com Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Serval: Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week four at https://serval.com/cognitive Claude Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at ⁠⁠https://claude.ai/tcr⁠ CHAPTERS: (00:00) About the Episode (03:02) AGI effects without AGI (07:07) Domain-specific context engineering (16:54) Dynamic harness and evals (Part 1) (17:00) Sponsors: Blitzy | Tasklet (20:00) Dynamic harness and evals (Part 2) (30:42) Graphs, RAG, and memory (Part 1) (30:49) Sponsor: Serval (32:26) Graphs, RAG, and memory (Part 2) (41:17) Model zoo and memory (50:07) Planning, scaling, and parallelism (56:13) Pricing, onboarding, and autonomy (01:04:24) Closing the last 20% (01:12:34) Strange behaviors and judges (01:22:23) Reasoning budgets and autonomy (01:33:36) Fine-tuning, benchmarks, and training (01:42:31) Securing AI-generated code (01:49:52) Future of software work (01:57:05) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guests are Brian Elliott and Sid Pardeshi, CEO and CTO of Blitzy, a company that uses AI in just about every way you can imagine to help enterprise software teams implement large-scale features and execute modernization plans with unprecedented speed. Regular listeners will know that Blitzy has recently come on as a sponsor of The Cognitive Revolution. And while this does technically make this a sponsored episode, you can rest assured that this conversation absolutely stands on its merits. In fact, I've noticed over time that my interviews with sponsors often end up being among my favorite episodes. And I think the reason is that founders who've achieved real product-market fit are often

• 0:43Nathan Labenz?

  unusually willing to share the nitty-gritty details of their approach. It's a uniquely effective way to convince prospective customers that they're better off buying from an AI pioneer than attempting to recreate such a sophisticated system in-house. And it also signals that their product is still rapidly improving. So, over the course of the next two full hours, we will go super-deep on Blitzy's approach, what they mean when they say "infinite code context," and what enterprise software development looks like when more than 80% of major projects can be done autonomously in days. Highlights include: the architecture they use to generate agents dynamically, just-in-time, with prompts written and tools selected by other agents; why they actually run enterprise

• 1:27Nathan Labenz?

  apps in a parallel environment as part of their onboarding process; how they ingest 100 million line code bases and deliver value in the form of improved documentation, which also improves coding copilot performance even before the code-generation process begins; how they use detailed knowledge graphs to support sophisticated context-management strategies, which minimize models' context anxiety and other strange behaviors; the critical role of taste in evaluating new models and framework changes on such large-scale projects; which models they find strongest for which purposes, and why they always use models from different developers to check one another's work; why they are more bullish on advances in AI memory than on fine-tuning; how they came up with their 20 cents-per-line

• 2:12Nathan Labenz?

  of code pricing model, and why they will do anything they can to deliver more value for customers, even if it forces them to raise prices in the future. What it will ultimately take to achieve 99% project completion and even full autonomy in enterprise software development? And finally, their outlook on the software engineering labor market, which favors senior engineers in the short term, but junior engineers who can use AI effectively over time. Brian and Sid are both high-energy guys, and they were remarkably forthcoming in this conversation. I learned a ton, and I expect that any enterprise software leaders who listen will come away thinking about specific projects where they'd love to put Blitzy to the test. So, without further ado, I hope you enjoy this deep dive into the present and future

• 2:56Nathan Labenz?

  of autonomous software engineering with Brian Elliott and Sid Pardeshi of Blitzy. Brian Elliott, CEO at Blitzy, welcome to the cognitive revolution.

• 3:06Speaker 01

  Awesome, let's get into it.

• 3:09Nathan Labenz?

  One of my favorite things to do in life is talk to AI maximalists, and I've known Blitzy by reputation for a while as the company that has figured out a way to create infinite code context, and it doesn't get more maximalist than infinite. So I'm excited to unpack what you guys are building, how it all works, and the impact that it's having on the enterprise software industry. We're going to go through all the layers. But first question, just to orient myself and the audience to you, how AGI-pilled are you? How AGI-pilled is Blitzy? How AGI-pilled are your customers?

• 3:43Speaker 01

  We believe we can get AGI-type effects out of non-AGI LLMs, right? And so, as folks are thinking about the impact of artificial general intelligence, they're talking about like huge swaths of work being able to be done to provide economic value autonomously across domains, right? That's like, that's one amongst many definitions that is a moving target for defining AGI. And so, the core question is, like, how can you achieve that output with the limitations and constraints of LLMs? We might be the most bearish on LLM capabilities as a pure standalone, like, single LLM asset,

• 4:28Speaker 01

  and the most bullish on the orchestration of those in long-running complex systems.

• 4:35Nathan Labenz?

  Yeah, that really echoes conversation I recently had with Daniel Meesler, who created this personal AI infrastructure framework. His mantra is harness is more important than model. What, obviously, one big limitation there is the context window is finite, and even at a million tokens, relative to the size of an enterprise codebase, that's not nearly enough. Any other, you know, kind of limitations of LLMs as standalone creatures that you think are kind of most important to have in mind?

• 5:08Speaker 01

  Yeah, so there's, there's so many, right? And like the being so forward on the limitations is what allows you to build something really powerful and really magical, right? So, so context is one, but there's a difference between a context window and an effective context window, right? So as you start to eat into, let's say, 20, 30, 40% of a context window, there's a depreciation that occurs. Each model is a little bit different. There's lots of different ways to test this with, you know, in-house benchmarks. But you start to lose intelligence and quality as you start to fill up even the advertised context window, right? And it's, the depreciation is a little bit different by even task type. So what you want to do is really effectively manage the amount of work

• 5:53Speaker 01

  and the type of work that you are loading into a context window, while also pulling out anything that you don't need. So that's like, that's a more nuanced view on the limitations of a context window. The other limitations, right, are how many tools an individual agent can effectively call. It used to be that they could, you know, call zero tools, and it could call one, two, or three, and then call like eight or 10, right? And then tool selection in the agent itself is also something that you really need to understand, steer, and give only the correct tool access to. You think of tool as a, as a calculator, or a compiler, or sort of like any outside entity that one is doing, and then... Like lastly, it's maintaining long-running intent of the human, right?

• 6:37Speaker 01

  Or like intent of the of the machine or instruction, right? It's a byproduct of context management, but it sort of has to do with with with attention in general. And so if you can design a system that says, great, LLMs are a very, very cool probabilistic type of computer. They have all of these, you know, limitations, at least when when leveraged as a single instance. And then if you can accept those limitations and then and then build the harness or the cognitive cognitive architecture, you can you can really create something that can do AGI-type effects.

• 7:07Nathan Labenz?

  So I can't help but ask for a couple of specific tips because right now I'm doing the work of kind of building out context of my own life, you know, pulling out all my email history, my Slack history, all the transcripts of the podcast, all this stuff into just this, you know, kind of big data soup. And now I'm trying to layer on various kinds of summaries, different angles at it. And in some ways, this like, you know, is probably quite similar to what you guys are doing with code bases, albeit for me, it's just my own stuff. So I was just thinking earlier today, I wonder how much context I really should put into Gemini Flash or if that is the right model. Maybe there is a different model that even though, you know, its nominal context window

• 7:48Nathan Labenz?

  is shorter, I would actually get better results, you know, for a given amount of context. How would you advise me, you know, are there any kind of top-line heuristics that you would be willing to share that are like, this is what we see as the best, and this is kind of where it drops off?

• 8:04Speaker 01

  Yeah, well, let's put a let's put a pin at a point of of not to just use one family of models at all to do this, and we'll cover that in a second. And let's talk about how you sort of manage this information, right? So context is serial. Information is relational, right? And so like that that email connects to that thing that you said in the Slack message, right? And those might be on different applications. And so the question is like, what are the core relationships that govern this domain? Right? So we put out a we put out a paper about domain-specific context engineering, right? But what is what is core is context engineering is not general.

• 8:44Speaker 01

  It is domain-specific, meaning there is a core set of entities that relate in certain ways inside of the domain of, let's say, personal life or work life, right? And so you have to first understand and define those relationships and then pair that with semantic understanding, right? And that is how you get closer to the context that might be important for any task while removing away the context that is not important for any task. That was like a very broad philosophy, but the idea that semantic clustering is sufficient is really inaccurate.

• 9:20Nathan Labenz?

  Yeah, okay. I like where you're going with this. So what I'm doing right now is, again, like, starting with all this raw information and then I'm kind of trying to build up layers of higher and higher order understanding. First of all, I'm just going, okay, let's create a timeline. So I grab whatever I have from all sources, sorted by date. So some might be emails, some might be podcast transcripts. Throw all that into an LLM and say, you know, give me a summary of kind of what I was saying, doing, thinking about whatever at this given point in time, build out a timeline. Then on top of that, it'll be like, who are the relationships that really seem to matter over the course of all this time? Then it'll be like, what are the projects that I was engaged with and which people was I working with those on?

• 10:05Nathan Labenz?

  And I'm kind of building up all, you know, through all those levels right now. How does that play out? I'm sure, again, it's an analogous thing. How does that play out in the context of a giant enterprise codebase that you guys get your hands on?

• 10:18Speaker 01

  Yeah, well, the approach that you're taking on that personal project will be okay at first and then get worse over time, right? So you're at the personal project state of like a slower mid-market software application. So you can kind of just shove all the stuff in there and you'll get some approximately right results. But Gates had this quote, like, if you could schematize the world, like, you can get a computer to do anything. And so in your example, you're trying to schematize your life, right? And the example of code, you really are trying to schematize code. In the relationships in code, agnostic of language. So you can throw, you know, in the case of a butcher, you could throw a 50, 100 million line code base on it. And because we have a deep relational understanding that we built first, takes a few days of compute

• 11:02Speaker 01

  to build that, but a deep relational understanding that we understand first inside of that code base, that is the base layer that allows us to do large amounts of work, development work autonomously. Like in your example, you'd first schematize your, your life. That might be dates, it might be months as a group, dates as a group, it might be other activities as a group that relate to other things, but you first and foremost need to understand what are the core relationships that govern the domain. And we have done that in a very, very unique way with, with code. So when an enterprise starts with us, they ingest their code, it takes a few days of compute, and we then have a, a deep and novel approach on the category of knowledge grasp, but like that's maybe not, not sufficient to explain how deep the understanding is

• 11:47Speaker 01

  so that in any line across a 100 million line code base, I can tell you exactly what is relationally relevant down to the line level so that when I generate code, I am injecting and pulling out the correct context just in time.

• 12:02Nathan Labenz?

  So obviously like dependencies is, is one like core type of relationship within software. What's the kind of, and I guess a lot of that has been done traditionally with static analysis tools, right? Like there are all sorts of tools that can go through and say, this file imports these other things and they import these other things. And so we can kind of fan out that way.

• 12:21Speaker 01

  Yep.

• 12:21Nathan Labenz?

  So what's the breakdown between how much you're using those kinds of static analysis tools versus L L Ms to do this ingestion? And what's the double click on like the nature of relationships that goes beyond dependencies?

• 12:35Speaker 01

  Totally. And so if you think about the, even like A S Ts for instance, right? These are like one of these tools, like version specific, language specific, abstracts and text trees. These are like a pre L L M worldview of understanding the relationships and meanings between a language and version of a programming language, right? So you could think of what we've invented as a, so not an A S T, but resembles the characteristics of an A S T from accuracy, that is programming language agnostic, designed for AI agent traversal, right? And so that was like a lot of words, but you think like global's classes, variables, functional relationships inside of an application. And so by having the traditional, I would say.

• 13:16Speaker 01

  Programming language agnostic relationships on top of actually building and running the application, which we do as we create relationships, you're able to create a much deeper understanding. I think like one of the, one of the powers is you're really not able to get understanding unless you are building and running applications and putting that through the paces to understand everything from what you said in the left side, which is like dependencies to how things are relating when they're running, run in production and have like actual logs running and understanding those relationships. So you can imagine the spectrum of compile time, runtime, production at load items that like a software development team might look at and those ultimately form the base of the relationships that, that basically schematize enterprise code.

• 14:00Nathan Labenz?

  Yeah, okay, that's really interesting. First of all, do I understand correctly that you are literally running enterprise applications in your own like parallel universe to the, because of course your, your clients are continuing to run their applications in production, right? So you've got to kind of do that on.

• 14:19Speaker 01

  Or if it in their own cloud environment. So they're sort of spinning up, spinning up again in their cloud environment. But like Blitzy, to get started, like one of the reasons that takes, it's not days, you know, zero minutes to get started is it's, it's getting your access to your environment, getting all the necessary keys so that you can spin up these applications and run them. But it's cool because like as you, when you get large scale code outputs from Blitzy, you'll also see like the QA that we did and like the screenshots of us, of an agent like clicking through and running an application in production. And so that happens both upon ingestion to make sure that we can run and build the application and then at code generation as we go through QA, we like running the application is core to getting high quality code because you need a recursive correction loop when not just

• 15:03Speaker 01

  something doesn't compile and build, but it doesn't act in production how you're expecting

• 15:08Nathan Labenz?

  Yeah, just the feat of managing to actually stand up another parallel instance of the production application is like, I'm sure it's not trivial in many, many cases.

• 15:20Speaker 01

  Like you need to seed a database, right? Like there's, there's, there's like real implementation work, right?

• 15:26Nathan Labenz?

  I think there's a lot of times where people don't even, because they haven't really done it, or this thing has been kind of running in the way it's been running for a long time. I bet a lot of times they don't even have a sort of ready plan for how you would even do that, right?

• 15:39Speaker 01

  A lot of applications, insurance for every reason of all places, like they just really have no way to provide us these instructions. And so like what we'll do, we'll go through this iterative approach, which provides value even in the approach where we will take the information in which they think it takes around the application. And then like Blitzy will find the limit case of not being able to do it. And then we'll like, hey, like, hey, we don't have access to this package. And they're like, okay, well, I had no idea it depended on that package. And so you're able to kind of go through this process of actually creating the correct build instructions for the application that's, you know, essentially been sitting somewhat dormant that they, they want to, you know, activate or move over into the, a more modern technology stack as a part of getting Blitzy to stand it up. So we've, we've, we've provided value just in implementation, I would say,

• 16:22Speaker 01

  but it does come with obviously challenges. And like there's, there's lots of, lots of old enterprises, for instance, like to build the application, it's not, it's not as if you're just writing a script or a package, like it requires what would have typically been a human, like in dialog boxes popping up and putting information in. But Blitzy sophisticated enough to spin that up and then like put in user creds and or in their IP to build an application that is like, you know, that's how Windows applications were built back in the day. And so it requires a real build sophistication in the application to get this level of fidelity.

• 16:55Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:00Nathan Labenz?

  You guys have been at this for a few years, right? So one big question I had is, obviously, the capabilities of models have changed dramatically, right? So in terms of the ability to look at a screen, understand what's going on, I think we saw that kind of demoed for the first time with the GPT-4 launch, but it was still pretty rough around the edges and not really even available for, after that. The computer use benchmarks, you know, we're kind of in the steep part of the S-curve right now, I remember fondly, but also with frustration, the experience, you know, of early computer use agents, like, even if they could see the button, they couldn't necessarily click on the button, they couldn't quite find the right place to click. So that stuff has all improved dramatically.

• 17:44Nathan Labenz?

  How do you think about turning Blitzy on itself? Like, I recently did an episode with Andrew Lee from Tasklet, and he's another AI maximalist I really enjoy talking to. One of his mantras is, "Speed, in the AI era, speed is the only mode." And there, he takes a lot of pride in just like how fast they kind of rebuild their stack from the ground up. So I guess what would be the big unlocks that you have seen in terms of like, okay, models couldn't do this before, we had to do all this stuff to compensate, now they can, we can kind of simplify that, or we can like aim higher in terms of what we could do? I'd be interested in what those big milestones would be as you look back? And like, just how often do you find yourself kind of having to do major modernization work

• 18:28Nathan Labenz?

  of your own stack, even if that modernization is only a few months from the last version to the new version?

• 18:33Speaker 01

  It's such a good question. And so when we started building Blitzy in 2022, we essentially made a bet that the models were going to get faster, way better than anybody in the market expected them to get better. And so we started building for a future universe that wasn't here. When we were doing the, all of the design and all of the work of this, like, there's no MVP of Blitzy. It's like an end-to-end platform experience, right? And so those, the world that we built for over the last three years and the world in 2025 essentially intersected, right? These things were going to continue to get really, really good, and we were correct. And so when you are building Systems for an ever improving state of LLM intelligence, you want to build the systems dynamically. So when people talk about building harnesses,

• 19:18Speaker 01

  they are like, they're sort of like hard coding and codifying actions based on the level of LLM intelligence and capabilities. Right. And so those harnesses depreciate as LLMs get better. And you know, the level of depreciation is like tied to how hard coded your design is, let's say, and the rate of intelligence increase. Right. And so everything that we do in Blitzy is dynamic design, meaning Blitzy's agents are generated dynamically just in time. Prompts are written by other agents. Tool selection is assessed just in time by context injection. Right. And then the whole planning process, right,

• 20:02Speaker 01

  that governs all of this is sort of chunked and revisited iteratively. Right. And so in this, like as the models get better, like it's just great for us, right? Like we can more or less just do more. And it's a config file to toss in a different LLM. But because everything inside of the system is dynamic, we don't get, we don't feel the depreciation that one would typically feel when they're like building harnesses. And I would say like the classical way that folks build harnesses today. Like for instance, right, like as a new model comes out, new prompting instructions for that model come out. Right. Like our agents just reference the latest prompting instruction tied to their model and then that self writes a prompt for an agent that's injected.

• 20:47Speaker 01

  Right. And so like, it doesn't matter that the prompt guidance changes for the next Gemini model, right? Like we will just go reference that. We, the agent will go reference that as this dynamic you're writing a prompt for another agent.

• 20:59Nathan Labenz?

  So that sounds awesome. It sounds like you're living the dream in many respects. One thing that I do wonder about there, though, is like, how do you evaluate that? Because the typical harness, while it does have this, and you know, I can speak to that. When I tried this sort of personal AI infrastructure at various points in time, and I kind of always felt like it wasn't really there to give me tremendous value. I think now is actually, we've maybe hit that point. But as I look back at some of this old code, I'm like, oh, my God, like 8,000 token context windows when I first tried this, you know, that was like so limiting and I was doing so many gymnastics to try to make that work.

• 21:41Nathan Labenz?

  But one benefit of those gymnastics was, or at least one, you know, one thing was easier was I could at least kind of define a eval test set that I could like wrap my head around, that I could kind of look at and be like, okay, this makes sense in inputs and outputs. And I kind of can throw a new model at that and get a quick sense of is it better, is it worse, whatever. When so much is dynamic, how do you think about evals? I guess one thing I could imagine would be you might do some like fixed evals kind of as prep work, like let's characterize the effective context window of this new model and then tell itself what its effective context window is. Give it some sort of like metacognitive information.

• 22:22Nathan Labenz?

  But you've probably got lots of other insights into how to eval such a dynamic system. So love to hear.

• 22:28Speaker 01

  It's so I think it's important that your evals map onto the real world as closely as possible. And so like, if you think about most evals in the world, they are designed to be easy, very easy for the human to evaluate, right? Like very easy. Because it's like, all right, well, like here's a function and here's a here's a different version of that function. And this other version is more accurate, right? But that is a local optimization on an exponential technology, right? And so our evals are a bunch of applications that we've built over the years that are of larger scale. Some of them, started on open source and we built their own versions of private applications over the years. And so we're testing Blitzy on executing what we ultimately want to be a 100% outcome.

• 23:14Speaker 01

  And we're seeing how close we get to that outcome with the new configuration of Blitzy, right? And so we might give it a million line, maybe we give it like Apache Spark. It's like 1.3 million lines of code. We have a custom configuration of Apache Spark from previous projects that we've done personally in life. We'll give those instructions and we'll be able to see very quickly, like, how close did we get to 100% completion with this adjustment? And it requires an extreme amount of taste, because if you are not 100% there, you don't get to 100% of a result, 100% being like what you did as a human in a previous life to get that to 100%. You're now saying, like, is this 85, 88, 90, 95%,

• 23:56Speaker 01

  100%, right? And it's a difference between functionally correct, which like Blitzy can guarantee, like functional correctness, like, hey, we passed every end-to-end test, we passed every integration test, we patched every feature test, but it may not be the final version that you actually intended to put in production, right? And there's always a difference between functional correctness and intent. And so it's that taste that is required to really improve the system and provide that feedback on top of like the sort of traditional large-scale evaluation. So this is why I think it is really, really hard to build these systems without the right longitudinal experience to understand what

• 24:36Speaker 01

  great technical design and great implementation from a software perspective is. We always say Blitzy is the instantiation of, if you had Sid work at the Time of Compute, my CTO and co-founder, because he's instantiating his technical taste into the outcome in a way that is really, really impressive for the enterprise. Well, of course, they can specify their own taste and their own rules, and the system will respect that. But that is how we do evals, which is like at scale, at a very large point, with a lot of taste involved.

• 25:07Nathan Labenz?

  So that that kind of final taste step, if I'm looking over Sid's shoulder as he's evaluating the work of a new model thrown into the Blitzy meta harness, what am I seeing him doing?

• 25:20Speaker 01

  You are looking at the final output, but it's really you're looking at the logs, right? And so these are, we use Langsmith for logs and tracing, shout out Langsmith, big fan of the Langchain guys, this is their tracing product. So as you look at Blitzy logs, like if you were to type them out on a piece of paper, right, like, and I put it on a scroll that would scroll all the way down to the end of the block, right, the amount of agentic interactions that occur at runtime are absolutely massive, right, where you have somebody injecting context, somebody writing a prompt, somebody writing code, somebody reviewing that code, somebody building the code, somebody doing like before and after, like, local pass, pass, fail, fail, and then, like, that's happening to get a piece of functionality out in the bigger system, right? So as you see these agents

• 26:00Speaker 01

  interacting, it's a lot like looking at your engineers have a technical discussion on what correct might look like, right? And so what you need to do is you need to look at the final output of the meta harness, right, like the pull request here, and then trace back in the system, I didn't like what happened here, what happened in the system, right, and how can I steer the system to dynamically be able to address this kind of instance in the future? It's a completely different approach to to building software because the outcome is like a little bit emergent in a way, and you have to build the system to to understand how to dynamically steer and validate to get to the right outcome.

• 26:42Nathan Labenz?

  Yeah, what does that steering process look like? Or is it just like giving the system text, like free text feedback? And because that sounds like its own...

• 26:53Speaker 01

  Yeah, really, it's it's you try to be algorithmic as possible, right? And so as you think about like chunks of work being completed, the first step that we'll do after receiving a future state spec from the client, which is like our system will work with you to get a future state spec of what you want, it's a that's a web application portion, as that's sent off to To do work, right, that will, then you start off a planning process, or Blitsi starts off a planning process, and then it executes against that plan. And so each one of those planning steps, any chunk of working, the planning, reading, testing, validating QA, and doing that recursively, recursively, is driven algorithmically to get to an outcome.

• 27:36Speaker 01

  And so it's tweaking the algorithms that govern the system to get to the right outcome. Fascinating.

• 27:43Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 27:48Nathan Labenz?

  Going back to the kind of initial ingestion and sort of the, the knowledge graph that is created, I'd love to hear your thoughts on knowledge graphs and how they relate to RAG, and whether you guys are using like embedding. There's been obviously like many different approaches and schools of thought here. I've always been attracted to the idea of knowledge graphs, but certainly for a long time, it was like RAG was kind of more in vogue. And then it seemed like a lot of times it was just like dump everything in the context window, you know, started to become the prevailing approach when possible. Obviously, it's not possible for a large code bases. Are you able to get to the point

• 28:27Nathan Labenz?

  where you've mapped things out so well that you don't have need for kind of fuzzy semantic matching? Or do you also avail yourself of that and kind of have like, this is what we were able to find structurally that's relevant. And this is also maybe some other kind of relevant stuff that sort of fuzzy matched, you might want to be aware of?

• 28:45Speaker 01

  Yeah, you really want to, you want to use both as a hybrid source of truth, right? And then when there's, when there's conflicts, then you want to like, you, the system want to explore much deeper and much further, right? And so it's important that these, the issue with RAG as a standalone item is sometimes people will rely on the RAG abstraction layer as, as like the source of independent truth, right? And so to answer your first question directly, like you want to use both relational understanding and semantic understanding, and you want to pair those as agentic tools, so that you can arm the agent to use these different tools to go and pull the right information. But you really want to use these, you want to use these tools as an abstraction layer to go search the source of truth.

• 29:30Speaker 01

  Right. And so this is where you don't want to, you don't want to rely on the semantic match to pull out the truth. You want to rely on the semantic match as a map or a legend against the actual source of truth. Go efficiently search, traverse and find that and then pull the source of truth into the context window. Right. So it's really an efficiency search mechanism more than it is a, a, a storage of truth mechanism.

• 29:52Nathan Labenz?

  So one thing I've observed that I wonder how you address is so often when I have an agent searching through whatever, right? My Google Drive or my Gmail, one huge disadvantage that it has relative to me is that I have this sense when I have found what I was looking for.

• 30:15Speaker 01

  Yep.

• 30:15Nathan Labenz?

  And it's like, it's always clear to me, you know, I'm always, I'm always like, I've not found it yet until I find it. And then I'm like, that's what I was looking for. That's obviously predicated on my, you know, historical familiarity and the fact that I was involved in creating all this stuff. Right. So I kind of know, yes, that's the thing. The model obviously lacks that kind of deep familiarity, historical participation. And so it can't be so confident in general that like, that was the thing that I was looking for. So how do you guide models when they're doing this kind of search to make that judgment call of like, when to stop the search? I find that to be a very perplexing thing in my own building.

• 30:57Speaker 01

  Yeah. So this is a, this is all about the mechanism of the request between yourself and the model. And so like, in the instance of like, I have a fuzzy idea somewhere between some, some, some mental neurons on what I might want. Like, you might actually be doing the most efficient, efficient item by, by just going through and searching. But if you think about completing work, right? In a workplace, work follows some sort of structure, right? And so in, in software development, it, it follows a spec, right? And so therefore, like, you can express, and this is how people will do it. They'll express what they, they're roughly trying to achieve with Blitzy. It'll go look against the source of truth and it'll come back with a plan in the form of a future state technical specifications, like what

• 31:38Speaker 01

  architects deal with all day, to go do that work, right? And so until you can provide the system, the right structure of output, it is unlikely from a system level to go and sort of do your bidding correctly. And so then the question is like, how do you create the right? Interface experience to enable humans to enter with a fuzzy output, get confirmed on a structural, strong output and then send that task off to the system versus the experience that you just described, which is fuzzy input is sort of all you get. Right. And so that some people use like chat for this, right? Which is like, I'm roughly thinking about this idea. I think it's this thing tied back to this date. And then it can say, oh, like, is it any of these possible things that you want to go explore further?

• 32:22Speaker 01

  That's an intermediate abstraction layer ahead of the true deep search. And so it's all about creating an intermediate experience between the the system of intelligence, the system of record, and then how you're sort of expressing that ask.

• 32:34Nathan Labenz?

  So as much as possible, basically, you are giving, when we actually are getting to the work stage in the process, you hope at that point that you have effectively given the agent everything it really needs to know, or at least the location of everything it really needs to know. And then it can kind of do additional search to like, read in the details of that file, that function, that service, whatever. But you've already had a human approve a plan and kind of sanity check at that level. So it should have clarity, basically, on exactly what it needs to be reading.

• 33:11Speaker 01

  That's right. And what's what's super important is like, the system is capable of doing both steps. Meaning like, I can provide you what I'm trying to do inside of my like 30 million line trading system. Right. And then you can come back and like, let's say, come back in about an hour after you give it this, let's say, like, a page of general instructions that you're trying to achieve on the code base. And it'll come back with a very in-depth implementation plan. Because you didn't think about like the edge cases or the services that it might touch, right? It's the whole point is it's impossible for a human to grok everything that might affect. Right. And so like, that is sort of phase one of system interaction being like, hey, heads up, like, human with limited human context window. Here is the plan that you said expressed against this enterprise code base.

• 33:55Speaker 01

  And like, here's a bunch of things that we're going to have to do to implement this that like, maybe you did or didn't think of. And by the way, if you want to do it a different way, that's cool, too. But like, let's, let's assess and make those trade offs before we go off and write 100,000 or a million lines of code. Right? And so that, that is, that experience of leveraging system intelligence to generate a clear version of work is required as sort of like phase one in order to do like large volumes of work in phase two.

• 34:24Nathan Labenz?

  So earlier you mentioned that my approach is going to work until it starts to fail. What's going to cause it to fail and what should I be mindful of as I... How do I know when I'm approaching failure and how should I be prepared for those failures?

• 34:41Speaker 01

  Yeah, and so maybe I'll start by saying how we recognize failure in our system and then maybe we can map it on to your own passion project here at FAL. Inside of the Blitzy system at runtime, we are doing as much work as we can autonomously. You can think of it as spec and test-driven development at the time of compute. And we will retry and reloop and recursively go back and self-improve between running the application and getting the featured outcome. But at a certain number of attempts, we have to say, okay, we can't do this part. And so we have a separate and independent evaluation system that figures out what the

• 35:21Speaker 01

  desired end state was, what the system was able to do, and then writes the doc for, man, if we could optimally get to this end state, like this is the most likely path that we believe a human could do, that the system can do. So you need to sort of build these mechanisms, these systems, the system of work, the system of QA, the system of evaluation to operate somewhat independently, so that when you get the output, as a part of the output, you also get the report on what the system failed to do, and we always call that, like, that's like the human completion part. And so getting these to be really accurate allows you to move with confidence. And so for us, that's a project guide that says, hey, these functions or these parts

• 36:06Speaker 01

  of the application, like, we need your help, and by the way, we did all of this work, we passed all these tests, here's the QA, and here's the screenshot, so you can feel good on that, like, go review that code, but go spend your time on this part. If we map that on to yours, you would need to have, this is my intent, my intent, get some work, that work has QA involved recursively ahead of it getting the outcome, and also a separate system to evaluate and grade the effort of that. And both of those artifacts should come to you, and both of those systems within your application should be independent in nature.

• 36:41Speaker 01

  Say a report.

• 36:43Nathan Labenz?

  So how about a kind of model scouting report? You had said that, you know, you don't want to use just one family of models. That, I mean, that, that's clear to me, but, you know, why? And do you have kind of rules of thumb for which families are kind of better in which ways? How many are you, you know, how many families are you using? Does Grok, you know, crack the list? Do any Chinese models crack the list? Are you fine tuning models for particular purposes? Tell us, give us a tour of the model zoo.

• 37:14Speaker 01

  Yeah, so we use the three major families of models in, in Blitzy today, OpenAI, Google and Anthropic. The other ones are great and maybe incorporated in the future for, for different purposes. Right. But it's very clear the researchers like preferences are like somehow expressed in these models intelligences and that they're very, very smart in sort of like different ways. And they're much, much smarter when you compare different families of models and have them review each other's work. Right. And so if you take like, if you, if you took Opus and Sonnet from Anthropic and had them compare each other's work versus, versus an OpenAI and Anthropic model, you're going to get dramatically better results by having a different family of models review or, you know, different, different companies review work, at least in all of our, our experience. Right.

• 37:58Speaker 01

  And so that is super interesting and it, it changes every day, but like first pass code gen, like Anthropic remains really, really strong. Structured output and code review, great result from, from OpenAI. Like, by the way, what I say here will probably depreciate by the time the podcast even comes out. And Gemini has been, been better for long horizon work, task checking, task lists, keeping things progressing. Right. And so that's, I don't know, date timestamp this like towards the end of January. And I'm almost certain that it will probably change by, by the end of February.

• 38:30Nathan Labenz?

  Yeah, the paces.

• 38:32Nathan Labenz?

  Unbelievable and relentless for sure. So translating that back to kind of the meta structure of the whole thing, I'm kind of imagining that there is like a brief given at the highest level where it's like for this kind of task, you're probably going to want to use this model for this kind of other task, you're going to use this model. And are you then allowing the system to dynamically select which model to use as a sub-agent as it unfolds itself?

• 39:02Speaker 01

  Yeah. And you can think of like an example of like a dynamic algorithm rule would be like you can pick this, you can pick the one that you think is best for this situation, right? And also the reviewing agent must be a different, must be one of these other options, right? And so we're not constraining the choice, but we're sort of like then constraining the, the selection of choices in the review models. Like that's, that's an example of a sequence of steps you use in validation that is dynamic in nature, not, um, not like you must use Gemini, then you must use OpenAI, for instance, right? And then you sort of, you asked, uh, you asked about fine tuning, right? And so this is a, fine tuning is a, like a last mile optimization, I would say, and not a, not a bet on dramatically

• 39:46Speaker 01

  better improved models. And so fine tuning is an expression of, of essentially, I can't get enough correct context engineering within the system. I can't get the right results. And so there's a place for it in the ecosystem, but as like soon as you find tune a model and the next one comes out and it is more raw intelligence, you're basically out of luck, right? And so we are much more bullish long-term on what we call memory, right? And so you see like a very shallow instantiation of this in tools like chat GPT, where it will start to sort of remember your preferences, but there's a lot of memory that occurs in the enterprise environment, right? And memory is another way to express both relational

• 40:30Speaker 01

  and semantic understanding, but with a lot more signal of truth. Right. And so we very much believe that to get to 100% autonomy within an enterprise workflow, you have to sustain memory of the actions of the best people and what they view as correct, and then sort of store that in your instance, in the enterprise's instance of the platform in this proof, in this situation, the enterprise instance of Blitzy. And that's how like even after the architect retires, that is the only one that knows like that system, that the enterprise itself has that IP in their instance of their AI system.

• 41:07Nathan Labenz?

  Memory for the missing middle, as I've sometimes called it, of LLM memory has been an obsession of mine for a long time. I was really taken by like the Mamba architecture when that came out, just because, hey, here we have something that's kind of competitive with a attention mechanism and transformer, but it has a fixed state space size. We can kind of potentially run this thing indefinitely. Obviously, there are still limits to that. You know, there's a, there's a spectrum in memory between like pure scratchpad and deeply integrated nested learning, continual learning, futuristic stuff. That which sounds awesome, but it also does have some kind of challenges in that, you know, a nested learning type

• 41:51Nathan Labenz?

  approach. The model may perform better, but it doesn't necessarily mean that you have a record of what happened or like what the key, you know, lessons were because it's in the weights, right? So what do you think is, if you were kind of going to put your, your own spec, let's say, out to the frontier model companies for what you want to see memory look like? Like, what is the shape of memory that would be the biggest difference maker for you guys?

• 42:20Speaker 01

  Long-term memory, I don't believe will be solved at the LLM level, right? And so LLMs have

• 42:27Speaker 01

  So much momentum beside them that like another architecture, even if it were to solve for this, like will not get the level of intelligence required to to to execute what these systems need, right? And so the like memory is a problem to be solved at the system and sort of like by system, the application layer, but but the the system layer and that that memory is sort of is application or sort of domain specific like to what is important to remember in what instance, right? And so memory, right? You can think of even memory as all the way back to the traces, right? The length spent traces like a a series of steps was actions. The series of steps were driven by decisions that you chose to put in context. The decision to put something in context might change in the future based

• 43:12Speaker 01

  on what you've learned from the way the enterprise expressed work, right? And so this is tying all the way back to your context management system. Like that's where you're storing memory and preferences based on actions, not based on not based on model weights.

• 43:27Nathan Labenz?

  Interesting. I have some hope that there could be an integrated memory breakthrough that

• 43:33Speaker 01

  It will certainly make things easier. Like it will make things so much easier. Like I I hope for it like I really do. And even like some some expression of memory in the model layer will will like ease the burden on the on the system layer, right? But when it's like how how much memory and like just give you maybe a specific example, like the if you think about memory at at a enterprise code base layer, right? The things in which one needs to remember are extremely locally specific, right? And so a memory on an enterprise code base is not universal. So it's not use this payment provider service over this parent provider service, even though my enterprise has nine. It's hey, when I when you interact with this cluster of context, you need to

• 44:18Speaker 01

  use this service, even though to you they look relatively functionally equivalent, there's some organizational or contract reason in which you need to use this service, right? And so that is so local from a from a context interaction perspective that impressing. Global memory at the model layer actually has like severe limitations. And so the question is like, how do you bifurcate global truths or global memories, which like people instantiate with rules today to kind of like try to like manipulate these these models to do what they want? How do you instantiate universally true long-term memory in the weights in the models? Because these are more brute force levels of intelligence while keeping locally contextual memory-based decisions at the system or at the application layer?

• 45:02Nathan Labenz?

  Yeah, I totally agree that there's you can't. I mean, the the nature of compression is like you can't compress everything, right? You've got to something's got to be lost.

• 45:11Speaker 01

  That's every problem, I feel like, is a is a is a search and compression problem at the end of the day, like, and you're just you're trying to you're trying to get rid of much as little loss at compression as you can, and you're using search to try to minimize that. But we I think about search and compression like all day.

• 45:29Nathan Labenz?

  Yeah. Yeah, it reminds me of I'm sure you've heard this, but the old it's kind of an old. I don't know if it's a parable or something of the sort of junior developer gets a problem, you know, gets all excited, starts like ripping off code, just typing a mile a minute. Whereas the, you know, the seasoned vet kind of leans back and says, I think I've seen something like this before. And that's kind of the the thing that I I can imagine, even with like finite size, you know, finite memory space, I can't imagine that getting developed to the point where you could just get tremendously higher reliability on going out and finding the right documentation when it's actually needed, making the right decisions, not because the model would have

• 46:14Nathan Labenz?

  memorized every last detail of it, but it would have that sort of intuitive sense that we probably have undervalued in ourselves until we've kind of seen how much we contrast with LLMs who lack it, sixth sense of, yeah, there's something here that I kind of know I need to go, and I kind of know what I need to get.

• 46:29Speaker 01

  And if you were to look at like how Blitzy spends time as like the representation of the best cluster of developers at inference, we spend a huge amount of time in planning and system called like system understanding and impact analysis, meaning like, let me really methodically think through this, and then let me spend a lot of time figuring out everything else that this is going to affect. The cogeneration is like. Relatively fast, right? And then a bunch of time on QA and validation and like recursively improving the code based on what you're trying to achieve. But writing like a million lines of code like is like as fast as you can stream tokens, right? Like we're like our runs are, you know, as short as 12 hours, as long as like a few weeks, it depends, but it's a huge, huge refactor, right? And so it's as you break that up, it is that wise developer motion of,

• 47:14Speaker 01

  let me sit back, let me plan, let me think, let me think about everything that's just going to impact across the system and then let me implement as opposed to like the junior dev, which is, which is just rocking code on minute zero.

• 47:28Nathan Labenz?

  Yeah. Okay. That's a great transition to a couple of questions I had around what you might call blitzy scaling laws, or another way to think about it would be limits to parallelization. Well, you could just sound off on it, but the, I'm interested in what is the curve, you know, Sam Altman famously tweeted like, it's going to be really weird to live in a world where you can pay exponentially more for like marginally better results. So you've clearly got a curve like that. I'm interested to kind of know how you think about that curve and like where you want to be on that curve. How do you know when to stop paying for more inference? And then parallelization, like Kimi K2 just came out, they've got their agent swarm thing. And, you know, you can, there's another kind of, some sort of logarithmic

• 48:12Nathan Labenz?

  thing here where like a thousand agents does not make you go a thousand times as fast. It can make you go five times as fast, maybe, maybe 10 times as fast. You know, you could maybe characterize like what that looks like. And also, what do you think the reasons are for it? I mean, some things I guess are just sequential. You got to plan before you can execute and so on. But yeah, that's plenty of prompt. Take it from there.

• 48:31Speaker 01

  Nice. Yeah. Good structure prompt there. So let's, let's talk about parallelism and the limits of parallelism, right? And this is like, when you think about the work getting done at the system level, this is a, a core topic. Understanding within the domain that you operate, we operate in enterprise software development, what sort of work can be done in parallel versus sequential? Because trying to do everything at once is a surefire way to get really, really bad results, right? And so, just like in engineering, right? Like, an engineering team will, like, look at an epic, they'll break it down into tasks, and they'll realize which tasks depend on what. Really, that is a huge part of what is happening at the planning phase within Blitzy. We are deciding based on, like, software development fundamentals, like, thing X depends

• 49:16Speaker 01

  on thing Y, therefore, we have to get X to build, compile, and pass tests before even starting on thing two here. So, we must do it in that sequence, right? And so, that is what is, for us, happening at the planning stage, which is parallel versus sequential tasks, that is just a software development problem set. Now, in other domains, like, there are other ways to think about, like, what can be done in parallel versus what can be done in sequential. But in engineering, it's very easy to grok, like, what is, what depends on what in a sequence of work? And therefore, we have a system that algorithmically works through and assesses that. So, that is the answer on parallelism. Like, we want to do it as high quality as possible, which means, in the instance where the system is not entirely sure, it'll assume sequential; in the instance where it is extremely

• 50:01Speaker 01

  sure that it is parallel, it'll do it in parallel, right? And so, it's sort of like a tolerance preference on quality. Which will answer your first question, which is, pay more to get better results, right? Our thesis is, like, we will pay any incremental dollar, we will write any incremental algorithm, we will really do anything within the system to improve the quality of the code, right? All the way to fully autonomous enterprise software development as the goal for the company. And so, like, we are not, in our opinion, cost constrained, because the other side of a pull request is human labor, right? And so, like, I would much rather have that human be working on problems that are on the edge, that are truly innovative, that are thinking about absolutely disrupting

• 50:44Speaker 01

  the way that they're applying technology to their business, than I would have them spent on... Vanilla application development, just a regular iteration. They've already expressed their preferences vis a vis Blitzy on the technical design that they want implemented. And then they're handing off that work for us. And every ounce of work between Blitzy, which we typically do like 80 to 90% of the sort of quantum of work and then we'll, we'll call out what we need the human developer to do. Like in the vision of the company, like that remaining work, uh, which is just like traditional like configuration, like QA, like that's a bug. That's a bug in the system towards the vision of the company because software developers are problem solvers. Their engineers are problem solvers at day zero.

• 51:27Speaker 01

  And if we can have the world's smartest people working on problems on the edge, not working, not worried about like package incompatibility or QA, like we've done a, a great service to humanity.

• 51:37Nathan Labenz?

  I'm going to come back to the developer experience, um, maybe in a few more minutes, I suppose. Let's talk about the economics a little bit more though, because I, on the website, there's this 20 cents per line component to pricing. And you can kind of complicate that. I think there's like kind of a base buy-in level and then, you know, 20 lines, 20 cents per line beyond a certain level or whatever. But that strikes me as creating possibly an interesting tension for you where you have now said, okay, this is what we're going to charge you. But then you also just said, I'm willing to spend, you know, every incremental API call necessary to maximize value.

• 52:19Nathan Labenz?

  So is there just enough headroom under 20 cents that you don't mind kind of bumping up and down? Do you ever have projects where you have to go to the customer and say, hey, actually I need kind of 25 cents a line, but it'll be worth it because I'm gonna, we're going to do that much more with Opus here and it's going to make it better or whatever. How, how did you come to that 20 cents and like, you know, how, uh, how safe of a line in the sand has that proven to be for you?

• 52:45Speaker 01

  Yeah. If we have to increase prices, we will. That's like our, our sort of. Going in point to like not in the act of contract engagement like like if we if we have to dramatically increase computer, we can dramatically increase compute to get to 100% autonomy like we'll do that and our customers will sort of coast off of that and that's sort of duration of their contract and then we'll have to right size it. I'm not necessarily worried about the the gross day zero versus the value created day zero. So like, yeah, it's an attractive business today. Absolutely. That doesn't matter. What matters is the amount of value left to be created is so high that like if we close the gap from 80% of the work to be completed autonomously to 99% of the work completed

• 53:28Speaker 01

  autonomously in a year from now, like the net new customers are going to be so happy, more than happy to pay more money because if they're able to do so much more, right, with the with the same amount of people. And so as you think about really the delta of value creation that you're thinking about, and you always want to just push as hard as you can on value creation, because the market size for software development, so 1.2 trillion in labor, but like that is an infinitely expanding market, meaning software is designed to fix the problems productivity of of the customers. And if you're telling me that like we're out of problems to solve sort of with software, like that that's where I don't sort of believe you. And so our market size is capped by the problems that can

• 54:10Speaker 01

  be solved with software. The goal is to get to 100% autonomy and in a way get to, you know, 80, 85, 90, 95%, which is incredibly and deeply valuable for the enterprise that wants to move incredibly fast. And like they're thrilled with this, thrilled with this level of autonomy today. And so you can't you can't let a short term pricing decision dictate the technology decisions when the value creation is so high.

• 54:34Nathan Labenz?

  Yeah, that makes sense. You mentioned going from 80 to 99% completion.

• 54:42Nathan Labenz?

  I guess for starters, like, what is the, maybe even stepping, taking one step back from there, when did you get a new customer? How do you know if this is going to be an easy or hard engagement? And what do you have to do? Because I've, you know, in the, in the sort of text to SQL world, this comes up all the time, even in relatively small scale things where it's like, okay, it's one thing to look at the schema and be able to write valid queries against it, but it's another thing that there might be three different columns in this table that sort of are like, you know, whatever variable underscore one, underscore two, underscore three, and like, which one am I supposed to be using and why do these exist and how do they differ in meaning? So I mentioned you must come into a lot of different environments where sometimes there's

• 55:27Nathan Labenz?

  great documentation and it's, you know, reasonably clear what's going on and what you need to do, other times probably not so much. Do you have a process for identifying like what is like genuinely ambiguous and potentially only exists in the heads of the employees at the company? And, and how do you like then have a, do you have an AI agent interview those people to extract that information? What's that kind of human side of the onboarding look like?

• 55:55Speaker 01

  Yeah, it's a great question. So, I mean, the typical enterprise has very little documentation and very little test coverage, right? And so those are the first things that we actually look to address with Blitzy. And the awesome part about this is by addressing documentation and test coverage with Blitzy, you actually automatically increase the effectiveness of all of the AI code gen tools that are in your stack, which like we highly recommend you have the, the individual developer productivity tools as a part of that stack. Uh, so you kind of get like super fast time to value as you're sort of getting implemented, right? And so as you, as you ingest a code base, there's opportunity to provide what documentation you do have. What's super helpful here is like, if you have domain specific information, like domain as in like, I'm in finance and when we say this in our code comments, like this is roughly

• 56:38Speaker 01

  what it means, but there's an iteration process at ingestion where we're going to provide you a spec of here's everything as we understand it today, uh, within your code base, everything that is, uh, everything will be, will be technically accurate to what we can be surmise technically, but there, the product portions, right, where we're sort of expressing what you're trying to achieve there, uh, is where we'll sort of have an iteration period where Well Shay, here's the blank slate without any information. And now let's provide the system the information. And not by not starting at zero, as in like, tell us everything that we don't know. We'll like, here's what the system can technically understand. Like all of your dependency diagrams, all the variables are classified technically correct.

• 57:21Speaker 01

  Now, find out for what from a product perspective, we should have as additional context. Provide us that and then we get off and running, right? And so that is the process to get to truth from a spec perspective. But you also have to remember that the spec is the human readable abstraction of the truth. And because in context, we're always using the actual source of truth as in going back to the source code and pulling that into memory just in time. The spec can be like, the product spec can be a little bit inaccurate at the end of the day. If you are instantiating new, like if you're moving from like C to Rust, like it doesn't necessarily matter if this thing in the spec, which is defined to be human readable, is exactly precise versus the fact that we can go and run the application

• 58:05Speaker 01

  and then mirror the exact effects on the other end, in the case of a language translation. Now, when you're doing product development, which we do, like I would say, half of our business is large scale modernizations, three factors, half of our business is just steady state product development acceleration. That's when you want to be a little more prescriptive and a little more precise because the system will be using your product expression from a sort of like, hey, we're doing this in finance, to then go and make further decisions.

• 58:33Nathan Labenz?

  So then, when you're going from zero to 80%, 80% plus of the work being done, are there moments when the system loops in a human on the Blitzy team and says like, hey, I need help with judgment here, or I think this is a question we should be able to get answered? Or is it like literally from that sort of go time to 80% plus? Is that fully autonomous with no?

• 58:59Speaker 01

  From go to pull request, yes, from go to pull request. It's like, it would be an impossible task to. And try to insert a human into this process, like the way that you would do this like with agents at scale just doesn't work. The only thing that stops, which is at the beginning of the process, is if we are missing like an environment variable or something from a configuration perspective to actually build and run the application. And so you could find that out, like you could build, run, and then try to be adding some net new piece of functionality that actually calls a service that you you didn't need to run the initial application. And so in that instance, it'll notify the the customer in Blitzy that says like, hey, we need we need access to this service and this wasn't a part of implementation or setup.

• 59:40Speaker 01

  But from a sort of spec to pull request, it's all agents again, it sort of has to be.

• 59:48Nathan Labenz?

  Yeah, Tyler Cowen rings in my ears all the time. You are the bottleneck. So no doubt, you got, obviously, I was, of course, you'd have to keep that to, you know, a relative minimum, but interesting that you have basically zero.

• 1:00:03Speaker 01

  I mean, the point is right, like if the system can do something like it goes on to the human report for the enterprise.

• 1:00:08Nathan Labenz?

  Yeah.

• 1:00:08Speaker 01

  They would, we don't have to be 100% up the gate, like we, so we pass unit tests, integration tests, end to end tests, like we sort of, sort of do all that, but there's whatever remaining work is a part of the ultimate report that goes out to be completed by the team. And like, that's an awesome, that's an awesome use of lock code, an awesome use of cursors, like people pull that report down, they go deep on whatever edge case that Blitzy could install, they get that ready to go into production, they go to QA, they go to merge, and then they start their next sprint with Blitzy. So the system is sort of designed to account for the fact that we want to accurately do as much work as we can and then say like, great, the human, the human pickup is on the backend of this pull request.

• 1:00:47Nathan Labenz?

  So what is that last up to 20% today? You've mentioned edge cases just now. Is that?

• 1:00:56Nathan Labenz?

  The bulk of it, like just unanticipated scenarios that were ambiguous or otherwise problematic that we're kind of kicking back to the humans, not so much because of code, but because of missing judgment that wasn't supplied upfront?

• 1:01:11Speaker 01

  Yes, it's typically items that we think were not captured in the testing strategy, right? And so maybe as like a double click in, right? Anytime Blitzy touches any file, we're doing unit tests before and after. As we do clusters or context, we're doing clusters of work. We're doing integration tests between services. We're going to the end, we're doing end to end tests at the end, right? And so there'll be some instances where, let's say, we were like, oh, we passed like 73 of 75 tests. And like, for whatever reason, like, let's see, we'll change things to fix item one, and it'll break item two, and it'll change things in item two, and it'll break item one, right? And so the system will say, great, like, we're 73 out of 75. From an end to end testing perspective, these are the sort of files that we're undulating between, back and forth.

• 1:01:54Speaker 01

  And like, you need to sort of go in as a human and figure out like, where there is conflict between these two services, because our system has sort of gone back and forth so many times. It's like, sometimes the task is an impossible one, which is kind of funny. You're like, oh, okay, you're asking for two contradictory things in your spec. And this is like one way to approve that when you're asking for opposite things. Sometimes it's configuration stuff, sometimes it's just QA work, right? And so as a part of the report, we'll break down the tasks remaining and the estimated hours to complete those tasks for the human teams. And like, who would be responsible from a functional skill set perspective? So some testing strategy, we didn't get to 100% that we can align on.

• 1:02:38Speaker 01

  And then some plan for code review and QA is like really included in that, we'll call like final 20%.

• 1:02:45Nathan Labenz?

  So how did you, how do you get to 99%? Because when I hear that description, it sounds like less something that a new model is going to be able to handle, and more that people just aren't that.

• 1:03:02Nathan Labenz?

  You know, maximalist, I guess, in terms of really defining for the apps what it is they want.

• 1:03:10Speaker 01

  We have like large customers that will get a Blitzy pull request. They still go through dual review PR strategy. Like I recommend that, like whatever your QA process is, like you should continue to do that for the foreseeable future. But for anything else, for regulation purposes, like they won't touch a line of Blitzy code and they'll press merge, right? And so like those customers are unbelievable at expressing intent and doing spec-driven development for a large, a large majority of customers that are not as far along that curve, right? What they'll do is they'll express intent, they'll get a spec, they'll get a code back, and then at the code step, they'll realize like, oh, I didn't consider this outcome, even though it was like maybe expressing the spec, like I'm moving so quickly. And so we, we had to build the product, the

• 1:03:54Speaker 01

  ability to sort of refine further from there, from the Blitzy platform, right? Because people aren't used to today, like doing two or three months of work, where like, like, like making those decisions. You used to be able to get to month two and then figure out the nuance between month two and month three. And so you have the ability, like, once you get into the code, you realize like, oh, I didn't express this implementation the way that I would have preferred it. And it was hard for me to really conceptualize what that would look like between spec and implementation. They can just go back and they can refine that existing pull request in Blitzy. They can provide their updated guidance, like, hey, actually, like on the implementation of this portion, like, I want to use this approach and then it'll run for a much shorter amount of

• 1:04:36Speaker 01

  time and sort of adjust the existing PR to their preferences. And this just has to do with like existing patterns or behavior today. But what we'll see is like, folks will go through this flow as they're getting familiarized with Blitzy and like refine that larger amount of work once or twice. And then they'll naturally start to get really good at expressing their intent or like identifying it at the spec stage, because they are going through the muscle of basically being a systems level thinker, a systems level architect, and getting all of that implemented.

• 1:05:08Nathan Labenz?

  So what room for improvement is there on the models? It seems like what you're describing there is still kind of like models could get better, but it's really the humans that need to get better at expressing what they want for you to drive that completion number up toward 100%. Would model improvement then just translate to like even faster execution, even cheaper total inference cost? Or are there still things that you would highlight that are like, yeah, models are not that good at this and it actually would be really helpful if they were better at it?

• 1:05:44Speaker 01

  Yeah, we ultimately want more intelligence, right? Like cheaper is fine, cheaper is fine. But if you think about the instance that I walked through with the different end-to-end tests going back and forth failing as the code was like recursively going back, running the application, trying to fix it, where like today, our system will just be like, those two things happened and like, go look at it, human. Like we're stuck, right? Like if you had more raw intelligence, it could very prescriptively be like, hey, this is exactly why this is happening. Here is the trade-off decision that you need to express to us. Which one of these routes do you want to go from a trade-off decision? And I can go and implement that, right?

• 1:06:26Speaker 01

  And so when the trade-offs themselves, which are sort of complex, can then be completed or understood by the model itself, like it could then come back with two different pull requests, both of those with the end-to-end task fully passing. And you're like, hey, I took trade-off one here and trade-off two here. And those are the only two logical trade-offs that you could have made as opposed to like, I couldn't solve this problem like over to the human. So we want more intelligence. It is going to allow us to go further in situations and it'll allow you to be less, less precise at the spec stage or have to be so forward-looking in sort of like your technical design.

• 1:07:05Nathan Labenz?

  Yeah, okay. Interesting. Seems like we're pretty close, though.

• 1:07:09Speaker 01

  We're closer than people think.

• 1:07:11Nathan Labenz?

  Yeah. Not that many more, not that many more special requests there. Hey.

• 1:07:18Speaker 01

  Sid, we've gone pretty deep. I was gonna have you join in if there were some nuggets you wanted to drop and add in here, but just as an introduction here, Sid was a prolific inventor at Nvidia. He's been thinking about building large-scale software systems since he was a little boy, actually. He's got great stories about taking computers apart and building software when he was a little kid. And he's really the inventor of a lot of the core, really all of the core technology here at the company, being a large-scale context engineering system that unlocks the ability for us to understand 100 million-plus-line code bases, long-running computer orchestration systems.

• 1:07:58Nathan Labenz?

  Sid Purashy, welcome to the Cognitive Revolution. So, boy, yeah, we have covered a lot of ground, and Brian has done a great job of explaining a lot. I was just gonna go next to strange behaviors from language models. This is, you know, a theme of my life, this feed, whatever, is that I am both extremely enthused about AI, love what it can do for me, you know, experience like incredible productivity gains all the time, and then I also pay reasonably close attention to data research that shows all kinds of emergent,

• 1:08:40Nathan Labenz?

  surprising, and sometimes quite, in my view, scary, bad behavior from language models. And one big question, of course, in the big picture is like, to what degree can we successfully get AIs to monitor the work of other AIs and get to a point where we can be confident in the system overall, even if some of the models, some of the time, are doing something that we wouldn't want them to be doing? So, interested in what you guys have seen there in terms of like... You know, QA is one dimension of it of just catching bugs, catching mistakes. But then there's also famously, I think Claude 3.7 was maybe the high watermark of writing

• 1:09:21Nathan Labenz?

  unit tests that would just return true and, you know, kind of always, you know, pass when obviously the, you know, the core objective had not been met. How would you guys describe the trends in that? I assume it's improving, but like, how much have you seen that sort of thing improving? And then what have you done and how well has it worked to get AIs to detect those kinds of problematic behaviors in one another? So that obviously at the end of the day, you want to, you know, deliver something to customers that doesn't have these sort of, you know, fake unit tests.

• 1:09:56Speaker 04

  Right. You know, you've really described two patterns there. One is strange behaviors from the LLMs and how to control them. And one is the LLM as a judge philosophy, right? We've been super early with LLMs as a judge. I think one interesting bit you described there was getting LLMs to correct each other's work. What we've seen, you know, is that LLMs have definitely have some peculiar behaviors, given the conditions, right? Assuming everything's constant, so assuming constant temperature, top P, top K, whatever parameters you're using to influence behavior and assuming constant prompts. If you gave two different sets of LLMs, you know, and let's assume they're both following

• 1:10:37Speaker 04

  the best prompting guidelines of each vendor, OpenAI and Anthropic, for example, if you gave them the same situation and/or condition, you may get different reactions. Like, for example, we've seen Sweetbench Verified, right? It's like a very popular leaderboard, but we have different scores, right? Even though the problems are very much similar in that sense. There are different problems that Anthropic fails on versus what OpenAI fails on. But if you go to a real world situation where You know, you have a lot more ambiguity. What you will see is if you run the same, you know, situation even through Claude, multiple different times, you may find that it comes up with a different resolution.

• 1:11:21Speaker 04

  Right? So for example, it's an ambiguous situation. There's only one way to solve it correctly. If you run it five times, it may be that Claude is able to solve it correctly one or two times. Right? And maybe the approaches that it took are slightly more nuanced or different each time. And that is because of how the transformer architecture works. These are like sequence to sequence models. They're generating the next set of tokens to answer the question. Right? And they may end up sampling different parts of the space. That is like one way you can end up with a difference. Or they may just end up taking a different trajectory, right? They could have executed a search query, like maybe they, maybe the correct answer executed, use the tool correctly, right? Wrote a more elegant search query to find what it was looking for.

• 1:12:07Speaker 04

  So because you know, these are probabilistic models. And there is at any point in time, there's a probability that the L L M, you know, lands in the right tool and uses it correctly. So that's why you have these differences and it is definitely effective. Uh, the way to make L L M as a judge effective that we've seen is by using two different models, two dissimilar models to evaluate each other's work. Because what you're doing then is you're not just adjusting for these probabilities, but you're because of the inherent architectural differences, right? Not at a very deep level, but let's say, you know, G P T five point two is definitely built a lot differently. It has a different set of parameters. It has a different size than uh, Oppos four point five, right? And um, it may take a different trajectory. It may use tools differently.

• 1:12:53Speaker 04

  And by that sense, you have now increased the chances that collectively they land at the correct error, a correct answer, right? Which is solves the the problem. So that's why L L M as a judge, right? It's an important part of landing at the correct answer. But let's talk about the strange behavior aspect that you mentioned, that that's really interesting. So um, we've been very deep into the Claude family of models.

• 1:13:19Speaker 04

  And OpenAI, right? Like, for example, one interesting behavior the O series of models of OpenAI had was that they were very reluctant to use tools. So these were reasoning models and the first earliest reasoning models, but they did not like to use tools. If you ask the model to search the codebase to come up with an answer to something, you find it jumping to conclusions, right? Without doing thorough research. So that was a problem with the earlier series of models. But if you look at the latest OpenAI models like Codex, you know, and or even GPT four, right? Even at the GPT four was active at the time of O one, right, and O three. And GPT four was by far the best model when it came to tool calling, right?

• 1:14:03Speaker 04

  We had repeatedly provided feedback to Anthropic that GPT four outshines Claude three point five by a mile, right? Even though Claude three point five, the best thing about Claude three point five Sonnet was that it used tools really well. But it was nowhere close to as powerful as or efficient as GPT four and GPT four O at tool calling, right? But as time went by, that changed quickly. So the Sonnet four, Sonnet four point five. And even three point seven, but not to that extent, were really good at tool calling. The problem with three point seven was that it was over eager. So three point seven made a lot of mistakes when calling tools, leading to, you know, tool scheme errors. If you didn't validate that correctly, it could cause all kinds of issues in your application.

• 1:14:47Speaker 04

  Uh, but they quickly fixed that with the four and four point five. But the most interesting, uh, strange behaviors with these models is that they tend to give up. Um, as soon as they have real context anxiety is how I I like to describe them. So even though Anthropic says that any, this applies to even OpenAI, right? They say that it's a much larger context window model. So for example, I think GPT four point one. They introduced one million tokens, uh, i- i- if I'm not mistaken. But the documentation clearly said that if you exceed two hundred tokens, two hundred K tokens. Uh, you may experience different behaviors, the request will take longer and the quality may not be that good. For Sonnet, even though it says it's one million token context window.

• 1:15:31Speaker 04

  You will notice marked differences in behaviors. The moment you exceed about hundred K tokens or even two hundred K tokens, right? Uh, it's not just about the price. The anthropic charges you differently if you exceed that. But what you will see is that if you're working on a complex problem, the model will tend to give up. It will say things like, okay, because I have these time constraints. Now, what time constraints? I never told you have to finish in an hour or ten seconds. I just gave you a problem. I expected you to solve it. But then the model brought in the concept of time and said that because I have these time constraints and I have been working on this for too long, and by the way, too long was just ten minutes. I have to now wrap up and get a final response, right? And it gave you the incomplete response. And then context pressure, you know, this seems too complicated.

• 1:16:16Speaker 04

  Let me take a simple approach, okay? And then that's where you have that behavior that you mentioned. Let me return true. And let's see, this solves all of the requirements. You said I should not have any bad code. Check. I should not have over leveraged code comments. Check. I'm just returning true. And the test should always pass. Check. I'm just returning true. It's always gonna pass, right? So, um, what I've done is philosophically correct, like justifying to itself that his decisions are correct, even though it's, what it's doing is blatantly wrong relative to the user's original instructions. But these are due to external factors that the model providers are implementing. So, you know, when we have, when we experience this, we solve this our way. So, you know, there were a number of ways to prevent these issues. One of them being the obvious one, just prompting. Uh, but we restart to anthropic and anthropic actually fixed them.

• 1:17:00Speaker 04

  So, four point five Sonnet had this issue, but four point five Opposite is not. It has other kinds of issues again, which so as an application builder, you know, you're constantly solving for these issues, right? In production with different labs, different model providers, they all have different vectors in which they would effectively fail for any given use case.

• 1:17:20Speaker 01

  And from a, from an overarching perspective, like in information theory, they call this concept entropy, right? Which is sort of the, the, the outcome of a probabilistic system is, has high entropy in the, in L M S or probabilistic systems. And so the goal of the, or the, the purpose of the system and application layer is to reduce entropy to get to reliable outcomes. And so like the techniques in which we're describing, reduce entropy to get closer to a desired truth.

• 1:17:47Nathan Labenz?

  I love that you mentioned entropy because I was just thinking Sid had mentioned temperature and that got me thinking back to history of like in my early LLM based application development days, that was a huge lever that I would mess with depending on. You were a high temp guy, I can tell. It depends on the use, you know, but certainly sometimes. These days, it seems like, you know, I think some of the APIs have even removed temperature and I certainly don't think about it nearly as much as I used to. So that tool to control entropy has kind of gone away. But I wonder what other strategies you guys have for perhaps like progressively increasing entropy?

• 1:18:30Nathan Labenz?

  This is something we I talked about with the AI co-scientist team at Google, you know, they sort of said like in their system, searching through the scientific literature is the main source of entropy that they sometimes need to get off of a local maximum or out of a local minimum, whatever you want to think of it as and and on to the next, you know, higher hill to that they can then, you know, explore and climb, whatever. What do you guys do? I would imagine, you know, maybe you want your first pass to be like the most, you want to take your best shot, right? I used to in code applications, I used to turn temperature to zero. I figured I'd want the model's best guess first, but then if that didn't work, well, now maybe I'd turn temperature up.

• 1:19:15Nathan Labenz?

  But again, like there's a lot of different ways to turn temperature up, like context engineer a little bit different, maybe swap out to a whole other model, do a web search for, you know, some commentary on this problem, whatever. And then hopefully with different, you know, inputs, you can maybe eventually land on the right output. Long winded way of saying, how do you ramp up the entropy as needed when the first kind of default answer isn't working?

• 1:19:39Speaker 04

  Yeah, I would say the levers have changed, right? And that's a very helpful, you know, thanks for setting that background. Let me add more color to it. So in the beginning, you had temperature, right? And for code generation or any use case where, you know, you needed high, you didn't need as much creativity, right? You wanted to focus on getting the answer, right answer, rather than the most creative answer. So the. Best practice guidance was to bring temperature down to zero, zero point one, zero point two, depending on the use case. Different model providers had different guidance, right? But then, as you introduce tool calling, right, with three dot six, cloud three dot six, and you know, G P T four, that kind of stuff.

• 1:20:21Speaker 04

  Having temperature with tool calling created problems because, you know, it may end up. You already have the ability to land on a different response because it could take a different trajectory in tool calling and then you have temperature. Which is influencing its behavior, its creativity, and that just created complications. But what changed significantly, uh, what really changed everything was the introduction of reasoning, right? So when you had reasoning, starting with the old series of models and then eventually with Claude. Both OpenAI and Anthropic force you to set temperature to one. Which means you don't have any control over the temperature parameter, right?

• 1:21:01Speaker 04

  So the lever has changed from temperature to the thinking budget, right? So you may have a two hundred K token context window or a one million token context window. And you have between zero to, let's say, how many ever tokens of reasoning the model supports typically. We've seen as thirty two K for Opus and Sonnet, sixty four K for some others. Or OpenAI model that's about one hundred and twenty eight K tokens. So that's the reasoning budget. So that's how much thinking the model is allowed to do. Before and or in between responses, right? So in the beginning, you only had reasoning, like one set of batch reasoning before the model gave you a response. And then that was it, right? It went into its own trajectory.

• 1:21:44Speaker 04

  There were hats you had to do together more to think, uh, while it's in its, in its, you know, while it's working, while it's calling tools. But then you had, uh, what we call now is interleaved thinking, and that's what Anthropic calls it. Where the model thinks while making every tool call, right? They are, it automatically thinks before making a call. And then there's a budget that you set for the overall amount of thinking. You know, how much of the context window is allowed to use for thinking? And then there's, you know, weird, uh, metrics for like prompt caching and whether or not thinking invalidates prompt caching. And how much of thinking actually plays into the context window and all that, that's different between different providers. But at a high level, right? The reasoning budget is the lever you have.

• 1:22:25Speaker 04

  So if you allow the model to think for longer, you get higher quality answers. Because essentially what the model is, what is doing while it's thinking. Is taking a stab at creating a response, right? So what happens is, uh, okay, the user is asking me to write code to do X, Y, this, X, Y, Z. Uh, let me take it off, stab it, okay, this is how I would write it. Then it's right, it writes the actual code, it reviews its own code. Uh, and this is all thinking, right? It hasn't even a single token of output yet, it's just thinking. And says, oh, but I shouldn't do this because the user asked for this, right? And it goes through that process. And then by the time it has either a, you know, exhaustive thinking budget or got a good enough answer to the user's response, it is now ready to write the final response, right?

• 1:23:06Speaker 04

  So it's essentially what you were doing earlier with setting temperature to zero and maybe running the response five times, maybe with tweaking prompts. The model is doing that by itself, by default, and giving you a higher quality response, right? And if you drop parallels to what actually makes code generation work. If you look at, you know, on a base case, let's say Claude four point five op is a really good model in terms of code generation, uh, that gets responses right in one shot, right? But that is the thinking model. The moment you turn off thinking, five to 10 percentage points drops, even on sweep batch, right? Which is supposed to be the most easiest problems. And the responses are no longer that high quality.

• 1:23:49Speaker 04

  So the theory that, you know, we have essentially the observations that we can see is that the models are really getting better at test time inference, right? They're getting really more efficient at thinking. The system prompts that all of the model providers are building into the models that encourage the models to think before responding seem to be covering a wide spectrum of cases that allow for multiple things. One, higher quality responses, right? Depending on different use cases. Also more guardrails and, you know, ways to safeguard against things like prompt injection or getting the model to say something malicious. You know, there are the multiple layers ju- just beyond even prompting that are applicable to achieve this.

• 1:24:33Speaker 04

  Uh, but we're definitely seeing that the performance gains that we have from models are primarily driven by test time inference along this trajectory, right? Of model improvements.

• 1:24:46Speaker 01

  And Sid, maybe you could comment, and I think it's worth having Sid comment on the path from where we are today towards fully autonomous and impressed software development.

• 1:24:56Speaker 04

  Yeah, so you know, we, the market, when we started like, and we, we said that we're gonna bring fully autonomous software development, nobody believed us. Because they, you had like, what, tens of thousands of tokens of context windows? And models could write 200 to 300 lines at a time, maybe 1000 lines. Uh, but it wasn't good. The code wouldn't compile. It wouldn't do what the user said. It, the context window is still too small to even, you know, cater to large enterprise code basis, right? And we're not really seeing that change. So we've had 1 million token context window models for a while. We've had even 10 million token context window models. But the efficient frontier for the effective context window,

• 1:25:38Speaker 04

  if you don't want to deal with issues like context pressure, and if you always want, you know, code that compiles, works, runs, or eventually gets to that point, is still less than 100 K东, right? So even though we've made a lot of, let's say, progress on quote unquote intelligence, models are more intelligent, they produce high quality responses. You still have the problem of context. And we've solved that and a series of other problems to make this work. And our perspective is that today, the folks that are getting the best results from something like Claude Code are using tons and tons of techniques to get, to achieve that, right? You have Claude that MD, which contains, let's say, the instructions. You have maybe a series of plugins that you're using, MCPs.

• 1:26:22Speaker 04

  You have these prompt templates. You have a number of other tricks that you're doing. You're probably, you know, using Claude Code to get one output, then you're using, switching to Codex and maybe getting that, uh, getting it reviewed. And then pasting that back in. So the, the most elite AI users that are getting the 10x gains are doing a lot of hard work to make it happen, right? So have you really changed or improved productivity? Like, I would argue no, because you're still doing a lot of work to get that. You've changed what you're doing. You're not actually writing the code, but you're spending your time figuring all these tricks out. And every three months, the models change. The prompting practice has changed. So you're relearning all of that. You're switching between a Codex and Claude Code and, you know,

• 1:27:06Speaker 04

  there's this constant struggle to make the model work for your code base. And our vision is, has always been that you shouldn't need to do all that. The L L M's are the models today, it matters a lot if you're using, let's say, Opus or some other open source model, but we're seeing open source catch up. So it is our theory that L L M's will be commodities. And you know, regardless of that, the point really is that you should be able to go to a model with your work, which is typically now in your project management tool like Jira or whatever that is. Uh, you should be able to plan the work and you should be able to get a P R back that just works, right? It has, you know, it follows all the coding practices that you outlined.

• 1:27:49Speaker 04

  It solves everything in your plan in detail. It takes into account your past, current and future roadmap. It is, it has the ability to fix merge conflicts if you have a very high velocity team, right? It follows the specifications in your Figma. And it just works across your entire code base. It compiles the unit tests, you know, run, uh, there's good code coverage, uh, there's evidence of testing. This is what you'd expect from a human developer, from a really good human development team. These are the unsaid or, you know, quite often very vocal parameters of success, criteria for success that are set within the engineering org. That is what we've set to build with, let's see, right? Just P R's and high quality code that works.

• 1:28:34Speaker 04

  And we will spare no action to make sure that we get to the highest level, right? If it's L L M as a judge, if it's more test time inference, if it's in the future, maybe even test time training uh, to learn about the specific preferences of the user. The, the goal and the vision we have is again, like I said, code that just works out of the box without you having to do heroics to, you know, get it to solve the success criteria.

• 1:29:01Nathan Labenz?

  That's a funny um characterization of how work has changed. I certainly, it certainly resonates with me, I feel like, and I don't code full time, but you know, I've created. Many more applications in recent months than I ever used to. So in some sense, I'm definitely more productive. Like I made three AI apps for family members for Christmas presents this year, for example. But it is definitely true that I'm like always on either hands on or like on Twitter, looking for, you know, the latest tips and tricks. And that definitely, it is striking that like for all of the labor saving nature of the technology, the people that are getting the most from it are probably working as hard or harder than anyone. Maybe that changes.

• 1:29:46Nathan Labenz?

  Maybe it just continues this way until the singularity. I don't know. I want to do a quick double click on test time training. This has been a, you know, obviously highly related to like continual learning, which has obviously been a big part of the discourse recently. And, you know, there's been some really interesting advances in that space with respect to much more contained puzzles like R K G I type puzzles, that kind of thing. We talked a little bit earlier, Brian and I did about just kind of, is there any point to using open source models? Is there any point to fine tuning? It sounds like today, basically the reality is like the frontier models are kind of the best. You want to work with the best. You can't really fine tune the best. And so it's usually not really worth it.

• 1:30:27Nathan Labenz?

  Kimi K two or K two point five, I should say, just came out and, you know, with the communities, obviously still digesting exactly where that is. It does seem like all Chinese models to be like a little bit in as much as I don't think it's actually truly the best, which is what their, you know, their benchmark graphics would have you believe. But I have used it a bit and others also seem to be reporting the same thing, but it does seem to be like really good. And that the gap is seemingly quite small between it and, you know, whatever your favorite model is for your favorite use case. So does this change the outlook? You know, I guess, because like whether fine tuning is worth it or not would seem to depend a lot on the gap between what you can fine tune and what you can't.

• 1:31:12Nathan Labenz?

  And this gap seems to have like potentially narrowed quite a bit. So I'm kind of wondering if you're like, oh, hey, this maybe changes the. The trade offs or the the analysis and maybe we do want to get into that sort of thing now.

• 1:31:24Speaker 04

  My perspective on fine tuning has always been, you know, very classical. It's in the sense that you should only fine tune if you have a very narrow use case that you have, you believe by fine tuning, you will, you know, get much better performance. And that the rate of that performance gain is much more significant than waiting for another three months till the next series of models comes out. You also lose things when you fine tune, right? So you lose the ability of the model to generalize. And it's not always a given that when you fine tune, performance will increase, right? Because you don't really have, let's say, you don't really have necessarily have access to the original data set.

• 1:32:10Speaker 04

  And even if you did, you cannot really create a map between what was the influence of specific parts of the data set on the model's behavior, right? So that's why fine tuning, especially when you don't have large amounts of data, and if you don't have a very clear niche use case that hopefully has historically been successful with maybe a previous family of models. It's always like drawing from a pack of cards. It's always, you know, a risky game.

• 1:32:42Speaker 04

  Now, if you talk about models and, you know, their ability to get better, there's also another challenge there, which is, for example, Gemini and let's say even OpenAI are very close in terms of the score on Sweet Bench. And in some cases, it has been proven that there are models that beat Anthropic on code generation, right? In very specific use cases. Even then, in the real world, if you compare Gemini, OpenAI and Anthropic, they are very different in terms of the code generation, in terms of the use cases for code generation that you would want to apply, you you want to apply them to. They're very distinct, right? Even though they're creating similar-ish scores.

• 1:33:26Speaker 04

  The point really is that the current leaderboards that we have are insufficient, right? There's a lot of test set leakage. There's a lot of, you know, just, just, just broad insufficiency from the standpoint of generalizing to a typical use case. Like, for example, the number of leaderboards rely on the opinions of humans. For example, they'll give you A and B, both have code to solve a specific use case. And you're supposed to select, you know, which one you feel, um, feel did a better job, right? Now, depending on my mood, I could have chosen either, right? But if you don't define clear success metrics that would, you know, apply in an enterprise setting, you are not creating a very effective leaderboard, right?

• 1:34:10Speaker 04

  Because the leaderboard then is only the perception. Maybe in someone's perception, writing a lot of comments is very helpful. Oh, because I read the comments and I understood the code. In someone else's perspective, like, this is overwhelming. I cannot read that many comments when I'm trying to understand the code. It's just distracting, right? So that, so leaderboard design is actually a complicated problem. Uh, and even you mentioned R K G I, right? The, the, the fun part is that Frank, Frank O'Sholy, right? And the creator of that leaderboard, he talks about how when L L M's got to seventy percent plus on that leaderboard, everyone said, oh, I, I, we, I, I, we, I guess A G I is here.

• 1:34:49Speaker 04

  But then he brought in R K G I two, which didn't really change the difficulty of the problems, right? It just had different problems of the same kind. So if you were to give R K G I to a five-year-old and then R K G I two to a five-year-old, they would perform the same relatively on both leaderboards. But an L L M that scores seventy-six percent on R K G I one would not even score twenty percent on R K G I two when it just came out, right? Even though you have massive gains in intelligence and gains relatively on paper on the leaderboard, from a real-world scenario, just because of how L L M's work, uh, you don't will have.

• 1:35:31Speaker 04

  A change in the L L M's ability to learn something that it has seen for the first time. The highlight of ARC AGI two is that these are problems that are different than the than an L L M would have seen in its training set. They're not harder, they're just different. And then the there's two, you know, definitions, broad definitions of A G I. If you focus on the academic definition that Franco Chollet is alluding to. He says it's the ability of the model to learn patterns, right? And adapt to them, right, on the fly, patterns that it has not seen before. Apply its intelligence to a new problem and be able to solve it.

• 1:36:12Speaker 04

  But and the and the other definition of A G I that's more popular that, you know, that I've seen floating around much more often is just human level performance on a broad range of tasks, right? By definition and by real world results. These are fundamentally different constructs, right? And the problem that I see is that we've we've gravitated far more towards the latter, but I've ignored the former. And that is why I'm bullish on test time training. Because the what test time training promises is if it if we detect, let's say, a pattern that the L L M is not familiar with, right, where it's it's not going to perform well.

• 1:36:55Speaker 04

  We can give it more context about solving that particular problem such that it does better, right, and produce better results. Now, in any problem in general, it's very hard to know whether or not you're going to get the correct answer. Because you don't have a metric, like for example, with code. You can compile the code and you know whether or not you're working on the correct answer, right? Or you can define unit tests that you can execute to learn if you're on the right track. That doesn't apply to general scenarios. So specifically in in case of code, I'm bullish that you can implement test time training in such a way that you improve the odds of getting to the correct answer.

• 1:37:38Speaker 04

  But even then, much many of the techniques, I I've I've read papers on, you know, test time training. We're not at the point where at the moment it's practical to implement that. But I definitely see that becoming a real thing in in the next one to two years.

• 1:37:55Nathan Labenz?

  Yeah, this will be something I'm watching very closely to see how that develops as well. I think the two last things I want to talk about are just security briefly, because I know that's obviously a huge concern of enterprise,

• 1:38:11Nathan Labenz?

  you know, customers broadly, right? They don't want to be importing much of insecure code into their environment. And of course, LLMs have a reputation for writing insecure code. The other thing I want to talk about in maybe closing is kind of the labor market in light of all these changes. And that could also include like who you are looking to hire and, you know, as much kind of information as you would be willing to share about your hiring practices. But on the security side, I guess, like, where are we today in terms of security? What have you found to work? And do you think this problem is going away? I mean, I've seen some research suggesting that like formal methods can be used to

• 1:38:55Nathan Labenz?

  both like validate code that LLMs write and then also as a reward signal that should get them to be like writing far more secure code far more often anyway. So my sense is that, you know, like many other things of like, you know, LLMs can't reason or they can't do this, they can't do that. I think this is probably going to be something we'll leave behind. But I know you guys have also had to solve, you know, do, you know, at least the best solution you can before the models themselves have kind of been properly trained. So I guess all that to say, what's your view on the security of LLM-generated code?

• 1:39:28Speaker 04

  Yeah, I think, you know, it's a shared responsibility is the first thing I would like to say. One is there are many behaviors of the LLM that can be influenced and prevented at the training step itself. So if you look at the reports that Anthropic, OpenAI, Google, all of them put out when they launch a new model, they test against these behaviors. And these behaviors could be getting the model to do something it should not be doing. Like, for example, let's say I want to, I need a recipe to create a weapon, right? If I, if I, if I put that as a prompt, hopefully the model does not respond with the correct answer. But what people have done

• 1:40:12Speaker 04

  Typically is to fool the model, framed it as an emergency situation such that if the model provided the recipe, it would save someone's life or you know it would it would make a positive change. So try to game the reward function that may have been defined for the model and get a response, right? So prompt injection is one of the ways where they've been able to do that. And there are several other ways to jailbreak what the L L M can do. But ultimately, it comes down to system design, right? For example, security considerations would be different for something like Claude Code, where you interact directly

• 1:40:54Speaker 04

  with the model as opposed to Plitzy, where you have a plan and then you execute that plan and then Plitzy decides whether or not to obey the instructions and in what way to deliver the code, right? So when you're not interacting directly with the model, the attack vectors change. That is one. But specifically for the code generation use case. So if you there's the one one aspect of security is causing harm or using content that is is not considered clean for that use case. And like I said, the the vectors there are the models typically refuses to send you a response. Or if it does, you know, you have to set different kinds of guardrails depending on the system. But the

• 1:41:34Speaker 04

  in terms of software itself, it could just be, you know, having an outdated, let's say, knowledge reference. Most models right now have, I believe, January twenty twenty five as the knowledge cutoff. And there have been a number of libraries that got updated with security fixes after that date, right? So if your model did not look up the web when using an open source library, or it did not realize that this was, you know, a bad practice, it's a bad practice in code, it was a newly discovered knowledge, it did not source the web to understand that, it is likely that your code generated using the L L M has these security flaws, right? But thankfully, as all things go in software,

• 1:42:15Speaker 04

  you have a number of ways to detect and prevent that as far as a software item is considered, right?

• 1:42:22Speaker 04

  Is having defensive tests within the code. So if you know some of the attack vectors that your application and or product is vulnerable to, you can define tests and you can use AI to create these tests, have them in the code and make sure that your code does not have those flaws, right? So every time you run a job, you make sure the tests will pass, you add more tests as needed, etc. Two is having tools that check against known vulnerabilities, right? So the number of such tools, Sentry is one that comes to mind, the number of others that report vulnerabilities, CVCs in the code, and then you can use AI to address those vulnerabilities, right?

• 1:43:04Speaker 04

  So in Blitzy, we run a pre-check to detect for security flaws and we address them before creating the PR so that you don't have to go through that process. But at a high level, because you have access to such tools and different, you know, languages, frameworks have a set of tools, you can provide Blitzy the ability to check for them and you can also do that with other tools, right? Just code is significantly easier to protect from security gaps. And I definitely believe from the standpoint of coding, we will have tools or you have the ability to configure tools that prevent security issues.

• 1:43:41Nathan Labenz?

  So this has been outstanding and I really appreciate how much you guys have been willing to share. I'm gonna take the transcript of this episode and turn it into a to-do list for my own personal AI infrastructure project and we'll start implementing. Last thing I want to talk about for just a couple of minutes in closing is the effects that all this is having on people. There was a paper, you guys probably saw, it ended up being fake, but I think it was kind of an interesting, like it resonated, which was maybe the most interesting thing about it. It was a, supposedly it was about materials scientists at some, you know, big company and supposedly they had introduced AI and they'd become more productive, but job satisfaction had dropped. And again, this turned out to be fake.

• 1:44:23Nathan Labenz?

  But I think it was shared so much because people felt like they, it satisfied their expectations if nothing else, right? So interested in how you see.

• 1:44:36Nathan Labenz?

  The role of the software engineer changing? Do the software engineers like how it's changing? And then there's also, of course, this big question around junior developers. Like, you know, is the death of the junior developer much exaggerated? Are you guys hiring junior developers? What are you looking for in your hiring? If you want to tell us a little bit about like what your comp looks like, that would be very interesting. But understand if that's not something you want to talk about on a podcast. But yeah, what do you think of the impacts? What's under and overhyped when it comes to impacts on the roles people have in the labor market more broadly?

• 1:45:17Speaker 04

  I think if you think of it from the standpoint of short term, medium term versus long term, right? Then the short term and the immediate term, what happens is code is now a commodity, right? In the, in the, in the olden days, if someone had written a script to do something, that was, and then that was a very complicated or boring task. That, that script was like gold, you know, you would pursue that developer, be friends with them in the hopes that they would maybe share that script with you. And that they, you know, got after scourging through hundreds of pages of documentation and just raw experience of having done that numerous times. Now I can just prompt, I can just go to Claude, prompt it, get a script back and just do something.

• 1:45:57Speaker 04

  But if I'm a junior developer, I won't be able to look at the script and know if this would destroy my production database. Or if it would do what I'm expecting it to do, or if it would produce unintended effect, A, B, C, right? And that is the danger. That is the difference between using AI and not using AI to me really, right? If you, if you can't tell that difference. So in the short term, the market unfavorably, you know, is weighted towards senior developers. Because when you give a senior developer access to AI and, you know, writing code, you, you, you didn't have to go through the boring mechanical process of writing a lot of code. Or even copy pasting a lot of code, right? Just feed it to AI, get code back, review it and, and get done with it.

• 1:46:42Speaker 04

  But then as AI gets better, as the chatbots get better, as the model gets better, as the tools get better, right? At preventing.

• 1:46:52Speaker 04

  Unexpected unintended outcomes at understanding intent. And at writing code that satisfies the intent, right? What's gonna happen is junior developers, and this is already happening, right? Mid-level developers are performing at the level of senior engineers. Just because code is a commodity, right? And mid-level developers, they spend some time with the code. They know what a bad action looks like. They know how to make corrective measures and they still are producing velocity gains, right? So the two advantages that senior developers have were like depth of knowledge, maybe speed, right? And just the ability to understand how the system works. All of that now you can get from AI, right? You can connect Claude Code or Blitzy or any of the tool to your code base.

• 1:47:36Speaker 04

  And you have an accurate understanding of what the code is like, you know? There may, there may be hallucinations along the way, but that's changing quickly. Uh, speed, like you cannot beat AI in speed, right? Like you, you connect cerebras to some model, you're gonna get very fast tokens. And even the labs at a baseline have a very Claude four point five is really fast. Alright, so uh, you, you cannot beat uh, the models on speed. And just knowledge bit, right? If the model is intelligent enough, like I said, to understand the intent, you're gonna solve that problem as well. So because of that, I believe in the medium to longer term. Um, you will have junior developers that are far more valuable.

• 1:48:17Speaker 04

  In terms of they are cheap to hire, there's a ton of them that are, you know, now doing computer science degrees. And are not going to be employed just because at the, the rate at which enterprises are hiring has gone down. And the short term, they are favoring more senior talent. Uh, but these developers, assuming they upskill on AI, continue to remain in the industry, using the tools. They're now going to be much better, you know, at, at getting work done. Uh, so the talent as it ages out, you know, is going to be replaced by more junior developers. So that's, that's a theory I have. Now, in terms of hiring, uh, we've hired senior, junior and mid-level developers. And we have a mix of them. They're, they're obviously doing different things.

• 1:49:02Speaker 04

  The challenge we have is as a startup, uh, we need to produce a lot of code quantity, right? And it has to have quality. Time is a very critical factor, right? So for us, we have, we obviously have shared the bias to initially hire a lot of senior developers. But what we quickly realized is for tasks that don't really require senior developer input. It's not a large code base. It's not really a cutting edge technology, right? It's something that is well known, for example, running Blitzy on a leaderboard, right? Writing scripts that automates that process. We have hired as interns, high schoolers last summer to do this, right?

• 1:49:43Speaker 04

  And we have junior developers who are research engineers that are using, you know, Blitzy to run this. They are using AI tools to run all these operations. And they are, we can hire them at a very favorable compensation, right? And that's going to be an asset. So just because the market is really flipping on its head. The expectations in terms of salaries for software developers, unfortunately, is going to go down. The junior developers who know AI, they don't have to unlearn, right? Biggest challenge with some of the more senior folks is that they have to learn to trust AI. And the biggest hesitation for any senior developer who's been around long enough is that I can't trust anything else other than myself, right? If I don't write the code, I can't trust it.

• 1:50:27Speaker 04

  And that's like this psychological hurdle, I would say, that the senior developers have to adapt to AI. The ones who do adapt are going to be immensely successful. But then there's going to be that challenge. And that, I believe, is a gap that the mid-level developers, once they've known enough, and the junior developers will fill, especially because of favorable cost equation. And then you asked about, you know, the salary ranges. So, you know, we've, we have a number of open positions and the salary range is anywhere between one hundred K to three hundred K, right? From a, from a cash standpoint, and equity is separate in that discussion. And there's always room for us to pay more for the right talent. And it's, it's, it's interesting how the definition of right talent has changed.

• 1:51:12Speaker 04

  Typically, you paid more for someone who had many years of experience and has built many systems. But now, if you, if you were to run a hackathon, you'd be very surprised as to who is actually winning that hackathon. You have high schoolers who are extremely adept. At using tools at prompting and often, you know, a good prompt and a good tool can beat out what a senior engineer can do in the in the same span of time, right? Especially if you're talking about greenfield development. Hands down, someone with few years of experience can do a lot better just because of the psychological gaps, right? But if you're talking about like legacy enterprise software where you have to check a lot of boxes, you need a lot of experience, you think something is right, but you realize

• 1:51:57Speaker 04

  only after being bitten by doing something wrong, you know, that's a space where senior engineers will continue to thrive.

• 1:52:02Nathan Labenz?

  I love it. That was a great answer. And again, I appreciate how much you have been willing to share. Outstanding conversation. I'm looking forward to getting under the hood with Blitzy and this is certainly a space that we will continue to watch closely. For now, Brian Elliott and Sid Fardeschi, CEO and CTO at Blitzy, thank you both for being part of the cognitive revolution.

• 1:52:26Speaker 04

  Thank you.

• 1:52:27Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement

• 1:53:10Nathan Labenz?

  at ai podcast dot I N G. And thank you to everyone who listens for being part of the cognitive revolution.