How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp)

Zach Lloyd is the co-founder and CEO of Warp, an AI-powered terminal and software factory platform used by tens of thousands of engineers. Before Warp, he spent nearly a decade at Google, including time as a principal engineer on Google Sheets. He built Warp from the ground up as a modern, AI-native alternative to legacy terminals, and the team has since expanded into software factories: a full cloud-based system that takes an idea in Slack all the way through to a merged PR. In this episode: Why a software factory is more than a coding agentThe public Slack → Linear → GitHub → QA workflowHuman interactions per PR as a signal of automation and throughputWhy human review is still the bottleneckScoring agent runs, finding failure modes, and self-improving agent workflowsReplaying real tasks to choose model cost and quality tradeoffsCEO workflows with Figma MCP, Granola, and research agents— Brought to you by: DX https://getdx.com/howiai—Engineering intelligence for the AI era OpenArt https://openart.ai/arena?utm_source=online&utm_medium=influencer&utm_campaign=infl-howiai-arena-na-acq-web—An all-in-one AI creation platform for images, videos, music, audio, and more — In this episode, we cover: (00:00) Intro (02:35) Warp’s AI software factory, Wilson (09:23) Automatic factory triggers (11:12) The engineering leader dashboard Zach wishes he’d had (15:18) How code review is changing in an AI factory (17:08) Tracking cost per PR across model configs (18:47) Using LLM-as-a-judge to score every agent run (20:02) Catching redundant tests (22:19) How the factory self-improves from failed runs (26:03) Quick recap (28:33) Building a cost-quality Pareto chart for model selection (31:35) How Zach uses AI for non-technical CEO work (32:10) Figma MCP demo (35:43) Granola MCP demo (36:41) GOG CLI demo (38:20) Thinking in parallel tasks instead of sequential ones (40:42) Zach’s prompting strategy for factory tasks (44:48) Where to find Zach — Tools referenced: • Warp (AI terminal and software factories): https://warp.dev https://warp.dev/ • Warp Factories:https://warp.dev/factories • Linear (project and issue tracking): https://linear.app https://linear.app/ • GitHub (version control and PR management): https://github.com https://github.com/ • Slack (team communication and factory input layer): https://slack.com https://slack.com/ • Sentry (crash reporting and automated issue triggers): https://sentry.io https://sentry.io/ • Figma (design, used via Figma MCP): https://figma.com https://figma.com/ • Granola (AI meeting notes and MCP integration): https://granola.so https://granola.so/ • Grok Bot (fast inference, cost/quality trade-off): https://x.ai/bot/guides/grok-bot-101 — Where to find Zach: X: https://x.com/ZachLloydTweets — Where to find Claire: ChatPRD: https://www.chatprd.ai/ Website: https://clairevo.com/ LinkedIn: https://www.linkedin.com/in/clairevo/ X: https://x.com/clairevo — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Claire Vo?

  Can I give you a hard time that humans really are the bottleneck? Because if you look at kickoff to PR time, it's 35 minutes. But if you look at PR to first human review, it's three and a half hours. And when you're doing, I think it was like over 2,000 PRs in the last month. Like, how do you keep things in the team from going, as I say, like chaos reigns?

• 0:20Speaker 01

  What is a software factory? For us, at least, it's an actual noun. It's like a product concept where it consists of a bunch of repos, a bunch of like MCP servers, like a bunch of configuration, and then a bunch of agents, essentially. So like a code review agent is not different agents, different automations. And it's all defined in code. Scoring happens across all runs. But then there's a second loop, popular thing on Twitter right now called self-improvement, where you have like an observer agent and it can then create updates to your factory that will try to prevent the particular failure mode. One of the things that's most helpful is like you could have these factory agents do computer use verification. So in this case, it made a video.

• 1:05Speaker 01

  I'm just like talking to this thing that is doing this job that I've done for the last 20 years and it's now it's like doing it kind of better than me.

• 1:11Claire Vo?

  Spoiler alert, the humans are the problem.

• 1:17Claire Vo?

  Welcome back to How I AI. I'm Claire Voe, product leader and AI obsessive here on a mission to help you build better with these new tools. Today I have Zach Lloyd, CEO of Warp, and he's going to show us exactly what he means by the software factory. He's going to show us how you can kick off tasks in Slack, what it means to go beyond the software development lifecycle, and how a technical CEO uses non-technical tools with AI. Let's get to it. This episode is brought to you by DX. In a recent study across more than 500 engineering organizations, DX found that spend on AI tools has grown 28x over the last year. The share of AI authored code is climbing, but overall innovation has remained flat.

• 2:02Claire Vo?

  As teams generate code faster, new friction in code review and validation is offsetting those early velocity gains. DX tracks speed, quality, and cost together across the software development lifecycle, giving engineering leaders clear visibility into how AI impacts delivery and whether those investments are translating into real value. Download the full report at getdx.com/HowIAI. That's G E T D X dot com slash How I AI. Zach, thanks for joining How I AI.

• 2:38Speaker 01

  Thanks for having me. Excited to be here.

• 2:40Claire Vo?

  There are a lot of trends on the timeline right now, but one that I think is going to be big for 2026, or seems to be big for 2026, is the factory. The factory, this like fantastical idea that we have that we're going to, you know, input tokens and output enterprise values, something like that. At least we'll output code. So I, you know, I'm excited for you to show us as CEO, as engineer, as builder, you know, what, what to you is the factory and how it looks inside Warp?

• 3:12Speaker 01

  The factory is definitely trending. I actually don't love the term factory, but it is the sort of thing everyone is standardized on. It feels a little bit sort of like dehumanizing to me. But, um, but yeah, the, the idea is exactly what you said. It's like, how do you, in a world where we have these sort of magical agents, kind of harness their power in a more organized way to have them, um, you know, basically build, build software for you from your ideas. And so, yeah, excited, excited to, to show you how I, how I use factories and like what it means to me.

• 3:48Claire Vo?

  I was at Lenny's Summit recently, and I think Marty Kagan also said he hated the word factory. I wonder if it's more like Santa's workshop, right? Like, we have magical elves that craft wonderful things that spark joy and are delivered gift wrapped for us. So I'm going to say the AI, the AI magical, magical workshop. No, show us, you know, what does, what does this look like for you? Yeah, you did, you just say, you know, you put in your ideas, you output product. Is that real? How, how does that really work?

• 4:19Speaker 01

  Yeah, so it kind of works like that. The, the way that I think of, of factories is like there's, there's basically two types of personas who are using them. So there's, I'll, I'll show you like basically how we do this in Warp. There's like the, the builder persona. So the builder persona is someone who is contributing ideas, wants to build stuff. And from that persona, it doesn't look like, at least the way that we have it set, it doesn't look that different from like how one of these engineers or designers or product people on our team might build with a local coding agent. The big difference you can see here, so this is like, this is our Slack, uh, is that most work is starting in a public place.

• 5:03Speaker 01

  And so the way we think of factories is like, you, um, you basically can say in a public Slack channel, you can tag, we named our factory Wilson. You can tag Wilson and be like, okay, I want you to, to build something for us. And when you do that, uh, what Wilson will do is like essentially do not just like the direct build, which is what you might get if you're using like Claude Code or Codex, but will also like do a few other steps. So what, what, what it will do is it'll kind of go through all the steps of like, first it will triage whatever this is. So it'll treat it as like an input to the factory, not just like a direct thing to

• 5:43Speaker 01

  build, meaning it will, um, you know, for this one, Harry's saying he wants to change the way this feature looks. He gives like pretty detailed instructions. He, um, you know, attaches an image. He says use computer use. And this kicks off a flow where the, the first thing that the factory does is actually kind of open an issue. So you can see over here, it's like we use linear in addition to using Slack so that everything gets tracked. So the factory integrates not just with Slack, it opens an issue. Then it does the actual implementation. For this one, it's like, it was simple enough and ambiguous enough that, um, we could

• 6:28Speaker 01

  actually implement it. Then it does, um, it will create the, the PR, so it integrates also with like GitHub. Then it does the QA, which I think is like really important these days. Like if you want to not spend all your time doing code review, one of the things that's most helpful is like you could have these factory agents do like computer use verification. So in this case, it like made a video of the completed feature. So it's like showing the thing, showing all the keystrokes. And so what you get here is, is not just like, and then it gets merged, right? Is not just like the single step of build, build the thing. Like if you were doing this with like the coding agent in like the pre-factory world,

• 7:13Speaker 01

  what you would typically do is like pull up your local coding agent, do the change there, test it locally, and then like push it up to GitHub. But you get the whole thing of like the ticket, the, the code review, PR, the video, all, all done together. And then the other like really, um, kind of magical thing is that it's all done in public. So if you are, you know, someone else wanted to come in and like look at this and see how the task was done, or even contribute to it. And so we all have people, like multiple people on these Slack threads. You get to a world where as like a builder, you're no longer sort of working in this local silo. Instead, you're working in a public space where everyone can see what you're doing.

• 7:57Speaker 01

  And so that's like, um, from the builder's point of view, how they like use the factory? Does that make sense so far?

• 8:04Claire Vo?

  Yeah. And, you know, I think what I'm reflecting for folks who are maybe still trying to grapple with, okay, like what's a coding agent? What's a factory? What's the difference? I think what I'm hearing from you is the factory, quote unquote, is comprehensive, comprehensively designed to reflect your version of what the software development lifecycle should be. And so it's not just like idea to code to PR to push. It is, okay, an idea needs to go through several steps. It needs to go down the conveyor belt of product and into like the funnel of issue tracking, and then we need to code it, and then we need to QA it, and then we need to have these very specific verification loops where we can take our sticker and say quality

• 8:47Claire Vo?

  controlled by, you know, number one, two, three. And so to, to you and the way you're describing it, where it goes beyond a coding agent or a co-pilot is that it is actually designed to take, take the very specific end-to-end product steps, um, not just the engineering kind of like input-output.

• 9:09Speaker 01

  That's, that's definitely part of it, for sure. It's like, it does, it does more of the, the software lifecycle. So it does more of those steps. The, there's a, a whole other, bigger part of it to me, which, um, is, so I showed you this from like the perspective of the, the individual builder. If you were to go into our Slack, you would see like this is happening over and over, like all day long, the people on our team are building in this way. And sometimes, by the way, it's not just like the engineers who are kicking off work like this, it could be work that's being kicked off, um, automatically. So for instance, we have, I don't know if you know Sentry, but it's like a crash reporting system.

• 9:53Speaker 01

  And so we have signals when there are crash reports coming from like our terminal app that we try to automatically fix those, and those go into the factory too. So it could either be human initiated, it can be initiated by like an external system, and it can be initiated actually in any of these tools. So it's like you can initiate it, like we could have started this whole thing in linear. We could have started this whole thing in GitHub. And so it's, it's integrated into all the tools. But the bigger piece of the factory approach, in my opinion, is not like, it's not so much how it, uh, the individual builders work, because it's not all that different from

• 10:37Speaker 01

  individual builders who might be using like, like, who might be going into the terminal, for instance, and using a coding agent to do something. It's different, it's in public. But what's really different in the factory approach is that everything is like centralized in the cloud. And there's a whole other aspect to it, which is for, not the builders, but for the people who are like the managers, who are like trying to scale software development on their team.

• 11:04Claire Vo?

  Finally, something for the manager.

• 11:08Speaker 01

  What everyone's been wanting, right? Too much attention on the builders. No, but in all seriousness, if you're like, like my concern is someone who's running a company is like, I want, I want soft, like it's a super duper competitive market. I want to see how quickly we're moving. I want confidence that the way that we're building software is actually improving over time. I want to make sure that we're not like wasting too much money. And in the world of like all, like every builder has their own individual local setup. That's very, very hard to get. And so this is like an engineering manager's dream.

• 11:49Speaker 01

  It's like, it's like a CTO's dream, like, not that this is the coolest thing in the world, but it's like, now I have the ability to sort of centralize and see what everyone's doing. So for instance, on this screen here, it's like, these are, this is actual data from our team in terms of like, how we're using the factory. It's like, there's a sort of sense of like, how automated is the factory? What's our velocity? How long does it take us to like ship stuff? And so there's just all these measurements.

• 12:20Claire Vo?

  Can we pause real quick on your productivity? Because I actually haven't seen this measure before. And we've seen a lot of measures, which is human interactions per PR. Just talk us through why that, because honestly I haven't heard that one before and I like it.

• 12:36Speaker 01

  You know, this sort of instinct underlying this is that we are gonna be, not, not to put this the wrong way, but it's like humans are a little bit of the bottleneck in terms of production of software. They're also the, the creative force. But in general, what companies want, and we're, you know, we're trying to build for companies that warp largely is like, how do you automate more? How do you make things that can truly just be done agentically, be done in an almost fully automated way? And so, you know, the, the general instinct is like, the more times you have to prompt, the more times you have to steer or like, like cajole your agent. That's like, that's going to be a limiter on throughput.

• 13:21Speaker 01

  So this is where you start to get into like factory world. Like this is like, imagine you're running like a Tesla plant or something. It's like, how many times do you have to stop the line? And so, you know, we're, we're trying to give engineering leaders a view of like, okay, how are you, are you, are you able to sort of make things more efficient over time?

• 13:43Claire Vo?

  And quick question, are those, just because my mind, I mean, I, you know, as a once, one time CTO, I'm like, yeah, this is exactly what I want. Are these interactions, do you think of these interactions as like the prompts, the sort of like upfront steering? Are you thinking about like, how many comments on PRs? How many loops? Like, how inclusive is this like interaction per PR score?

• 14:04Speaker 01

  It's, it's all of those. So because the, and again, I think this can definitely be refined over time. And bear in mind, we're like trying to figure out, so what is the right set of metrics? But this actual metric, because the factory is like integrated into all of your knowledge work tools, it includes all those things, like how many re-prompts in Slack? How many comments on linear? How many times did you have to correct the thing in code review? And so it's, it's like a kind of like, yeah, it's a proxy for how much work did you have to do? In order to, you know, get the thing to do the job.

• 14:41Claire Vo?

  Okay, wait, I want to, I want to give you a hard time, one more thing.

• 14:44Speaker 01

  Yes, please do.

• 14:45Claire Vo?

  You know, but before we started recording, you said like, how technical? And I was like, okay, well, we're going to put on the CTO, at least the CTO hat right now. Um, can I give you a hard time that humans really are the bottleneck? Because if you look at kickoff to PR time, it's 35 minutes. But if you look at PR to first human review, it's three and a half hours, three and a half hours. And so it's like so funny that still, cycle time, man, like it's, it's the thing you really have to focus on. It's that, it's that number.

• 15:18Speaker 01

  Yeah, there's still the bottleneck. And we're, I mean, we're still doing human code review.

• 15:26Claire Vo?

  Do you do all, all your PRs get human code review?

• 15:29Speaker 01

  Currently, all of our PRs get human code review. And so now this is like, I think of a team choice, an organizational choice. We have, the one thing that we have changed around this is not, like, we used to require, like the workflow used to be like person A on our team would build something with an agent, then person B would review the agent's code. We no longer require that, like the person who prompts the agent can also review its code.

• 15:58Claire Vo?

  Okay, you're making me feel better.

• 16:01Speaker 01

  That's a better thing, but it's still like, we, you know, we, we don't have yet complete trust. I think the way this will evolve is like some percentage of, some percentage of stuff will, will eventually, we'll feel confident enough that we can skip that.

• 16:18Claire Vo?

  Yeah, I did an episode recently, I built an Eve agent called Merge Mommy. And this is, this is the flow that I often see in even mature engineering organizations, is what you do is you basically like risk score every PR automatically. So go through, risk score it. And then anything that is low or extra low risk gets a stamped approval from Merge Mommy. And then a human's allowed to just smash the button and merge. Anything that's like medium, high or whatever, or has some like outlier on risk requires human review. And that just like lets you get that bottom tranche of the queue out. Also, just so you have good capacity for high quality human review on the things that really matter.

• 17:02Speaker 01

  A hundred percent. I think code review becomes an exercise in risk management. I think that's right. I want to show you some other stuff in the factory, just to show you like how this is different than just standard interactive agents. So you get, cost is on everyone's mind right now. And so you get like, this is agent cost, it doesn't factor in that sort of cost of the people at the moment. But it's like, you know, you can see like, we, we were really expensive a few weeks ago. We made some changes to our model configuration and we've driven this cost down. And we want to continue to drive it down. But just like, having the centralized view for this across your whole team.

• 17:45Claire Vo?

  Can I ask you a question on cost? Quick question on cost. So, I mean, this is a pretty, you can, you can see that the drop here. I, I see this a lot when I'm talking to engineering organizations. They often see like a rise in cost of PR as their, per PR as they're adopting AI, which you all already have. And then we see this drop as you're doing optimization. Do you feel like the biggest lever here is model right now? It's like that, is that the lever?

• 18:11Speaker 01

  Yeah, it's, it's model. I think model is the biggest, I think like context, like the way.

• 18:21Claire Vo?

  Context management.

• 18:23Speaker 01

  Like context management matters as a secondary thing. I would say model is the biggest.

• 18:30Claire Vo?

  Yeah.

• 18:31Speaker 01

  And the, the, the way to really figure this out actually is to test, which is something I want to show as well. But basically, I think, I think model is, is the biggest one. Any other questions on the, just like the spend view?

• 18:46Claire Vo?

  No, that's great.

• 18:48Speaker 01

  So the, the other thing, like this is where it really gets kind of factory oriented is like the, the way that we're changing development and this is, I think this is most relevant for folks who are watching. We're trying to manage like these teams and scale coding agents. It's like, we now, we're like measuring everything. And so this last thing in here, which we call scoring, basically gives you a view across all your agent runs of like how they're doing on different dimensions. And so we, we basically, because the factory is like a closed, kind of like closed loop system,

• 19:31Speaker 01

  like every, every time an agent does a task in the factory, that task gets recorded. And what that opens is the possibility for you to go and like, like go back retroactively and look at how well the task was done. And so you could do this as, as a person, you could literally just go back and look at, at tasks. But the other thing that you can do is you can have agents do this, which is, is what, is what we do. And so for instance, let me see if I can find an interesting one. Like here's an interesting one, like redundant tests. So if you're, if you're working with, you know what I'm talking about here?

• 20:09Claire Vo?

  Oh, I know what you're talking about. Every PR I push is like, I have run and set up 135 tests here.

• 20:19Speaker 01

  It's, it's the, it's like, it's, it's like, and the reason it does this is because, you know, they're writing tests not to like prevent regressions necessarily, but also just to like test behavior along the way. And so you end up with a bunch of tests that you don't need. And so we, you know, if you take the factory approach, you kind of know this might be a failure mode. And what you can do is you can write essentially a, like a scorer that uses LLM as a judge to be like, okay, I want to go and look at, you know, all of the agent runs. And I want to see how often where, you know, an agent thinks, a different agent than the one that did the task thinks that there's redundant tests. Does this make sense?

• 21:01Claire Vo?

  Yeah, totally.

• 21:02Speaker 01

  And so you basically are like, okay, I want, I'll, I'll pick a judge model. This is like a kind of medium smart judge model. You don't want it to be too expensive, otherwise you end up spending a lot of money on scoring. You have it classify in terms of like, for this task, like, how did it look? You pick like a sort of sampling rate on, in terms of how you want to do this. And then you get over time, a set of runs where you can see that like, sometimes this agent thinks that there were some like surplus tests. And so this is what I mean by like the real measurement. Then what you can do is like, you can go and you can actually, you know, as a human, you can go and kind of look at what's happening here.

• 21:45Speaker 01

  But I'll show you, there's also a better way to do this in the factory world where it's like, you can actually.

• 21:52Speaker 01

  Use agents to sort of identify what's gone wrong in these runs and try to improve the factory. Does this make sense?

• 21:59Claire Vo?

  Yeah, I'm curious, do you do this on a per-run basis or on an aggregate basis across a set of PRs?

• 22:07Speaker 01

  Aggregate, great question. So, yeah, so what we do is like the scoring, and this I call like this thing, we call this like the score, the scoring happens across all runs, but then there's a second loop, which is another like popular thing on Twitter right now called self-improvement, where you take an agent and you basically say, okay, for all of the failed runs, and you need like, you need like a real sample, like maybe 20, 25 failed runs, you need some significant sample size, otherwise it starts to over-correct based on single, based on single things. You have it look, you have like an observer agent look, and it can then create updates

• 22:52Speaker 01

  to your factory that will try to prevent the particular failure mode. And so, let's see if I can find one that's like, I don't know, this, I'm just literally picking a random one here, but this is finding some, some issue with our skills that are driving the factory, it's presenting evidence, and it's saying, okay, we should change the definition of one of our factory agents in a particular way, and if you go under the hood and look at this, it's like, it's changing this like step 10 of what our factory agent should do. Does this make sense?

• 23:27Claire Vo?

  Yep.

• 23:28Speaker 01

  And so, this is, to me, this is really exciting, like the way, the thing that's like enabling this whole thing to work is that you define the factory in code.

• 23:42Claire Vo?

  Code, yep.

• 23:42Speaker 01

  And so, you know, what I mean by that is like, if, if you were to look at this sort of definition, you would say, okay, the factory, what, like, what is a software factory? It's like, it's like, for us, at least, it's an actual noun, it's like a product concept, where it consists of a bunch of repos, a bunch of like MCP servers, like a bunch of configuration, and then a bunch of like agents, essentially, so like a code review agent, design, like different agents, different automations, and it's all defined in code, and the, the value of doing it that way, which like in warp factories, is that you

• 24:24Speaker 01

  get the ability to actually, you can like test different configurations and know, like, you're basically freezing the state of the factory at a given point, so you can be like, okay, if we were to, to change the factory definition, then like run all these tasks again, we could see if things were better. It also makes it so that like, an agent can actually, because these are all, it's all code, like coding agents can actually update the factory to make it better. Does this make sense?

• 24:54Claire Vo?

  Yeah. This episode is brought to you by OpenArt Arena, the global leaderboard for creative intelligence. Every week, new AI models launch, and everyone claims to be the best, but best at what? OpenArt Arena is built to answer the question that actually matters: which model is best for your specific job? Instead of one overall winner, OpenArt Arena ranks models across real creative tasks, from advertising and film to animation, product, graphic design, editing, and lip sync, covering both image and video. And these rankings aren't based on hype, they're judged by professionals, industry

• 25:37Claire Vo?

  leaders, and working creators through blind evaluations, so judges never know which model produced which output. That means you can see how models actually perform when it comes to the creative work you're doing. So stop guessing which model to use. Explore rankings based on real creative work and find the right model for your project and save time and cost. See the rankings at OpenArt Arena. Yeah, I just want to like, you know, kind of sum up where we are so far, because I think there's so much rich stuff in here, especially for engineering leaders and builders, and then I know we're going to get to some workflows that are not engineering focused, which I'm excited about, but the, you know, a couple trends, themes that I've seen as you talk through this, one, work happens in public, so I'm seeing, you know, this move

• 26:22Claire Vo?

  towards, instead of work being assigned in tickets, and then tickets being worked on, on laptops, work is happening in public, whether it's Slack or some other channel, and then, and then executed in the cloud, so sort of anybody can interact with that. The factory does have this software development lifecycle, kind of like definition to it, but on top of that, it has this aggregate meta-analysis that you're doing across all the behaviors, which is giving, you know, as we said, giving the people, the managers, what they want, which is, are we getting more efficient over time? Is this factory actually effective? How many humans? How many agents? What's the interbalance between the two? Spoiler alert, the humans are the problem right now.

• 27:03Claire Vo?

  And then, and then what you're doing is you're also doing these evals against key behaviors in your factory that you want to correct. And this is something that, when I talk to engineering organizations, I tell them all the time, which is such a challenge when you're using something locally, like, for example, Claude Code, is I say, you need session-level telemetry, because you need to be able to aggregate up the failures within sessions across your engineering team. And so, some teams do this truly by like sucking up every local coding session to S3 and running their evals, kind of like on, on their own, in their own platform, but

• 27:45Claire Vo?

  I do believe that if you do not have session, tool call, MCP call, test failure, computer use level, observability into every single, single coding session, you're missing a lot of opportunity to optimize efficiency, cost, model, just like how your team uses the tools. And so, I think it's super important that people do this, and then what you've added is this extra layer of, great, then take those insights and make the factory better, however you define better.

• 28:17Speaker 01

  This is a great summary.

• 28:19Claire Vo?

  Great, I did it. Professional podcast.

• 28:22Speaker 01

  There's, there's, let's just think of a sense of how you can continue to like, you can go even further, once you have this data. I think your summary was awesome. The other thing that you can do, and like, I think Norgs are going to do this, because this is, this is how, like, there's going to be nothing more important than optimizing the way that you like build and ship software in the future. And so, one other thing that is really powerful, if you set up everything you just said, where it's like, you have the, you have the tracking, you have the evals, there's one further way that you can make it even more powerful, which is adding the ability to basically take your own data and replay those sessions with different configurations to actually

• 29:07Speaker 01

  measure, like to your question earlier, like, well, how would it have done from a cost perspective or a quality perspective, if you had done like different models? And so, for instance, again, this is what we have built into warp factories, but there's lots of ways you could do this, like, the, the idea is, you can, just like, there are these public benchmarks, like Sweetbench and Terminalbench, that are on like generic data, you can recreate on your own data, like from real past factory tasks, how would things have gone had you used a different model configuration, for instance?

• 29:48Speaker 01

  So, this is like a kind of Pareto chart, which, again, these things are on Twitter all the time, but it's like, what's cool here is like, if you take the factory approach and are like really scientific around, okay, I want to know, like, like, I want to, let's say you wanted to curate a bunch of front-end tasks, and then see like, can I be using GLM-53 on those instead of using, you know, Opus? Yes, definitely. Should I be using Gemini 3.7 Flash? You're going to take a quality hit, and so like, you get to choose your own sort of trade-offs here, and then you can feed this back into a model routing strategy, where

• 30:29Speaker 01

  you actually have evidence that, okay, on your own tasks, for these types of, like, the best model configuration for cost and quality is, is like whatever you find.

• 30:39Claire Vo?

  Are you comparing that to what actually shipped? Like, how are you evaluating quality? You know, there's like seven ways to skin a CSS, like, how do you actually decide, you know, this is better quality or, or not on front-end tasks, for example?

• 30:55Speaker 01

  Yeah, awesome question. So, the, the way that we do it in warp factories, and you're going to probably, you can do different things here, is we use the exact same like scoring infrastructure that I showed you earlier. So, for instance, you know, we, over here, we have all these different scoring dimensions, so you can, if you have confidence in this scoring infrastructure, basically, you're replaying past tasks and trying to see if there was like, how it affected these scores. So, it's, it's primarily LLM as a judge, but you can also, it's like, you, you could do this with human judges, you could do it algorithmically, but the thing that we have built is LLM as a judge.

• 31:33Claire Vo?

  Got it. Super interesting. Okay, so, you've given me so many ideas on how to take the factory idea further. I'm curious, just like, take off our builder hat, our CTO hat, let's, let's put on, you know, I was talking to you, again, before we came on, I was like, people love to see how non-technical people can do technical things, and how technical people can do non-technical things. So, show us some of your other like AI use cases that are maybe less about evals and benchmarks and, and MCPs and more about, you know, being a CEO.

• 32:09Speaker 01

  Yeah, so we'll get like, I think like, zooming out, this is cool with all these charts, I still have a hard time making changes to Figma.

• 32:19Speaker 01

  And like, I don't know if you're the same way, but when I go to try and change like a Figma thing, I'm like, it's like I have three thumbs or something. Like, I just, I cannot figure out how to use it. But I do know that it's like, if you, like, this is, this is how we like do our slide decks. For instance, it's like, we, we are using Figma slides. You can make things that look very nice. So one thing that I do now is like, if I need to change a slide deck, I do it through the Figma MCP and the coding agent. And so like, just to kind of show what that looks like,

• 32:53Speaker 01

  we have this kind of semi-boring, like, slide here on cloud execution. So I'll show you how I would do this, just so you get a sense. So I'm gonna do this. I use this in Warp. I use Warp as a coding agent. This would also work in Cloud Code or Codex, anything that's Figma MCP. And so I'll just paste this in, and then I'm gonna talk to it, which is another thing. I assume people are on the voice train at this point?

• 33:21Claire Vo?

  People are on the voice chain. Yeah, we, we, we give all credit to Hillary Gridley. She calls it the yappers API, highest big with way to talk to in LLM.

• 33:31Speaker 01

  Yeah. So I'm gonna, I'm gonna say like, I'd like to make a new version of this slide, use the Figma MCP to get the context, duplicate the existing slide rather than making changes directly to it. Let's have it be so that the host box contains the runtime box. Let's, um, let me see what else. I gotta, I gotta switch back here. So we're gonna have host contain, sorry, host should contain sandbox.

• 34:07Speaker 01

  I'll go back and edit this. Let's make the context system something that's like, kind of like, you know, cloud around these things that feeds into them.

• 34:19Claire Vo?

  True CEO, put a cloud on the slide.

• 34:21Speaker 01

  Uh, you're gonna see some bad design here.

• 34:25Speaker 01

  Let's make the launchpad have a sort of like rocket type theme. This is my, this is not what my sales team wants, by the way.

• 34:34Speaker 01

  And let's make tracking, and why don't you come up with some ideas for how to make tracking better? The overall idea here is to make this slide more visually appealing than the simple, you know, five boxes and semantically show the relation of the boxes to each other.

• 34:57Speaker 01

  So I'll do this.

• 34:59Claire Vo?

  Do you know what word I say to Figma?

• 35:03Claire Vo?

  I say semantically.

• 35:05Speaker 01

  Now what?

• 35:05Claire Vo?

  To Figma MCP all the time.

• 35:07Speaker 01

  Really?

• 35:07Claire Vo?

  Oh, my God, all the time. Semantic color, semantic, semantic. It is, it is my orthogonal.

• 35:16Speaker 01

  That's really funny. Yeah, I mean, um, uh, we'll see. So and I'm using, I'm using Grok a lot recently.

• 35:28Claire Vo?

  Same.

• 35:28Speaker 01

  I don't know what your model of choice is. I think Grok is pretty good from like the cost and speed and quality, quality trade-off. I could show you how I could do some other, you know.

• 35:39Claire Vo?

  Yeah, why don't, while that's loading, why don't you show us something else?

• 35:43Speaker 01

  Yeah, I'll show you another one. So I'm a big granola user as well. And so, again, if I were really trying to improve this sales deck, another thing that I would want to make sure is that the sales deck is speaking to the things that customers are actually talking about. So I'm gonna start a second task here using the granola MCP where I'm like, can you use the granola MCP to look back over my last four weeks of sales meetings? And try to build up a list of the top 10 frequently most asked questions in these meetings as they pertain to Warp software factories?

• 36:24Speaker 01

  Don't list any specific customer info in the summary, anonymize it, uh, because I'm doing a podcast.

• 36:35Speaker 01

  So we'll, we'll get this one cooking as well. I can do one more if you want.

• 36:40Claire Vo?

  Yeah, let's, I mean, let's queue them up. Again, I love to see a AI-pilled CEO just open tab after tab after tab and kick stuff off.

• 36:50Speaker 01

  I mean, this is stuff that, this is stuff, these are all, by the way, real things that I am constantly using AI for. So the last one is like, uh, a task that I use AI for is trying to rediscover, uh, potentially like cold leads that I might have talked to in the last, um, like, you know, six months or so. You know, our product has changed a crazy amount. I might want to reproach them. And so the thing that I, I use for this, or had in the past been using for this, I actually just started using Instinct. Do you use Instinct?

• 37:25Claire Vo?

  Oh, no, I had a bad experience with Instinct.

• 37:28Speaker 01

  You had a bad experience with Instinct. So the, the, the normal way I would do everything is still through my, is still through coding agent, but I would, um, let's, let's, I'll do one more prompt here. Can you use the GOG CLI to look for emails and calendar events I've had in the past six months with potential enterprise leads? Who might be, um, useful for another outreach for Warp factories? You can learn about Warp factories at warp.dev/factories. And make me a Google Sheet with the info on them and share the sheet link,

• 38:12Speaker 01

  but don't print out any specific customer email or name in this thread.

• 38:19Claire Vo?

  What I like about this at the meta level is like, is this how your brain works? Where you're just like, I need to do the slide, kick off the slide. I need to like, update some of our sales position, kick that off. I need to like, follow up with cold leads. Like, is this three panel? Is this a reflection of your CEO brain? Because it's certainly a reflection of my brain.

• 38:40Speaker 01

  Yeah, I mean, if I were, if I were in like, sales mode, go-to-market mode, and like, I am in that mode quite a bit now, unfortunately, more, more so than like, straight up builder mode or engine manager mode, although I do all these different things. I'm constantly trying to find like, what is the right positioning? Are we, are we communicating, um, in these meetings in a way that lands? And like, it's so amazing to have, A, this ability, like, for me, who can't design or draw at all, or like, to have this now superpower, and we can go, let's, let's see how this Figma is doing over here. Has it started to do it yet?

• 39:20Claire Vo?

  Oh, it's starting.

• 39:22Speaker 01

  Oh, it's got some work to do. But it, it will, it will do better. But yeah, if I was in sales mode, like, I'm trying to figure out, are we positioning it right? What are people asking about? Um, yeah, and this is, this is right. Like, this is from our actual sales calls. And like, this is the, what I would say is the number one thing for the factory, like, the product category we're in is like, buy versus build. And it's backed up by evidence, and it makes sense to me. Security comes up a lot. Um, workflow, what's the workflow? What's the measurement? How do we manage costs? It's so cool. And so it's like, what I would do is like, you know, I want to then go make sure that our,

• 40:04Speaker 01

  our sales deck, our website, all this stuff is speaking to the questions people have. I, I might turn this into like a literal FAQ. Like, I think we, we have, you know, probably ability to improve that. Or I would just try and work this into our positioning. But, uh, I'm constantly doing this and trying to understand if, if what we're building is the right thing and we're positioning it the right way.

• 40:26Claire Vo?

  Oh, you got a rocket ship. Sorry, we're back at Figma for those that are not watching. We got a rocket ship. We got a cloud of some sort. It's trying.

• 40:36Speaker 01

  It's not very good yet. It will get better.

• 40:38Claire Vo?

  The rocket ship's not terrible. It's not bad. Zach, this is, this is so fun. I want to go to quick lightning round questions, and then we'll get you back to.

• 40:50Speaker 01

  Cool.

• 40:50Claire Vo?

  Back on the factory floor. All right, first lightning round question. Everybody wants the factory. Everybody wants the go-to-market intern that will happily go through your call transcripts, and the design intern that will happily make your ugly cloud slides as a CEO. But when you have the factory humming, and when you're doing, I think it was like over 2,000 PRs in the last month, like, how do you keep things in the team from going, as I say, like, chaos reigns? Like, how do you keep your arms around all that activity, all that work, all that context, all those tasks? And know at the highest level, you're doing the things that matter?

• 41:27Speaker 01

  It's a great question. Um, like, the kind of boring answer is like, a lot of the stuff that we've always done as software engineers still applies, which is like, you know, we're, we're dog fooding. So we're just like huge, constant users of the thing that we're building and making sure that like, the quality of the thing still works well. There's like cool knowledge transfer that's happening, uh, where it's, I don't know if you've seen this too, but like, the, this, there's like a kind of power law for like, how these tools are used in terms of like, we have some people on our team who are super duper power users and other people are like, kind of more at the average. And so there's this, um, when you, when you're working in this,

• 42:12Speaker 01

  what seems like a very chaotic way in public with everyone slacking these factories all day, you do get a chance to sort of see how like, the, the really expert people are using it. And that kind of uplevels other people on the team. And then, like I said earlier, we're still doing.

• 42:29Speaker 01

  You know, it's not just like chaos, like everyone like, you know, slop stuff into the machine all the time. It's like, we still have a product development process that, uh, you know, is somewhat traditional in a bunch of ways. So it's like, we do a whole bunch of user interviews. We watch people use the product. Um, we do design jams before we start building to not, not so much like figure out exactly like how the UI should look, but just to make sure that we're tackling the right user stories. Uh, like, cause at the end of the day, like, doesn't matter how fast you build software. It's like, it still has to solve some user problem and solve it in a good way. And so we, we're, we're trying to figure out how to like, not lose the key parts of like the human input here, but also, you know, use, use the fact that

• 43:13Speaker 01

  there's this like magic technology to, that can make you just go much faster.

• 43:19Claire Vo?

  I love it. I, amen. I could not say it better. Um, okay. Last question, and then we will get you out of here. When your AI is not doing what you want? And, and also I am curious if this question is, if you answer this question differently when you're working with AI privately versus working in, with it in Slack. With it's not doing what you want, what's your prompting strategy? Do you yell? I am lately, I will admit this to the audience. I am doing a lot of like, why are, why are you like this? Why? Why? A lot of questioning.

• 43:56Speaker 01

  That's funny. Um, I'm like a pretty level guy. I'd like, if, I think if you ask the people on my team, I don't, I've never like really yelling out my style. Um, I get like passive aggressive. That's kind of what I would say. I, I get, I get like eye roll-y, like really? Like this is what we're doing now? Like you built this whole thing that no one asked you to build? So I'll get like annoyed, like in like subtle undertones, but I, I'm not a, I'm not a yeller. And so, uh, I don't know. And I also try to just keep the perspective that this is just like a bonkers thing that I'm just like talking to this, uh, talking to this thing that is doing this job

• 44:40Speaker 01

  that I've done for the last 20 years. And it's now it's like doing it kind of better than me. And so, I don't know. I'm still like, I'm in like a little bit of like a wonder phase with it. Um, I don't know if you've ever seen that, um, Louis CK skit where he, he talks about, uh, how when internet, like wifi first got onto airplanes, stop me if you know this, but it's like, it's a skit. He's like, he's riding on an airplane. It's like, it's like 20 years ago and there's wifi for the first time. And he's just like, this is incredible. Everyone, like, this is like amazing and nobody's happy. And he tells a story about how like the guy next to him is like trying to watch a YouTube video and like throwing his hands up like, screw this,

• 45:25Speaker 01

  like I can't get any signal. And, and Louis CK is like, like this is bonkers. Like we're on an airplane. This thing is like going up to space. Give it a second. And so I think it's, it's so easy to lose perspective of like how wild this technology is. Like I'm, I'm like, I'm, I'm generally like pretty, probably like kind of more patient with it than most, I would say.

• 45:45Claire Vo?

  I love it. That's such a great perspective. Well, Zach, this has been super fun. Where can we find you and how can we be helpful to you?

• 45:52Speaker 01

  Um, so you can find me personally, like I'm on, I'm on Twitter. I have Zach Lloyd Tweets is my, is my Twitter. You should obviously come check out Warp at warp.dev. Um, if you want to build software factories, you know, I hardly even talked about it, but we have an extremely popular, uh, agentic terminal, uh, that's open source that you should also come check out and use. It's a great place to work with interactive coding agents. Um, but yeah, this was awesome. I really appreciate you having me on.

• 46:20Claire Vo?

  Yeah. Thanks for joining How I AI. Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube, or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiaipod.com. See you next time.