The AI Coding Factory

We are joined by Eno Reyes and Matan Grinberg, the co-founders of Factory.ai. They are building droids for autonomous software engineering, handling everything from code generation to incident response for production outages. After raising a $15M Series A from Sequoia, they just released their product in GA! https://factory.ai/ https://x.com/latentspacepod Full Video Episode Timestamps 00:00 https://www.youtube.com/watch?v=74Du4Ej_-yM Introductions 00:35 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=35s Meeting at Langchain Hackathon 04:02 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=242s Building Factory despite early model limitations 06:56 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=416s What is Factory AI? 08:55 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=535s Delegation vs Collaboration in AI Development Tools 10:06 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=606s Naming Origins of 'Factory' and 'Droids' 12:17 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=737s Defining Droids: Agent vs Workflow 14:34 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=874s Live Demo17:37 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=1057s Enterprise Context and Tool Integration in Droids 20:26 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=1226s Prompting, Clarification, and Agent Communication 22:28 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=1348s Project Understanding and Proactive Context Gathering 24:10 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=1450s Why SWE-Bench Is Dead 28:47 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=1727s Model Fine-tuning and Generalization Challenges 31:07 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=1867s Why Factory is Browser-Based, Not IDE-Based 33:51 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=2031s Test-Driven Development and Agent Verification 36:17 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=2177s Retrieval vs Large Context Windows for Cost Efficiency 38:02 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=2282s Enterprise Metrics: Code Churn and ROI 40:48 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=2448s Executing Large Refactors and Migrations with Droids 45:25 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=2725s Model Speed, Parallelism, and Delegation Bottlenecks 50:11 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=3011s Observability Challenges and Semantic Telemetry 53:44 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=3224s Hiring55:19 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=3319s Factory's design and branding approach 58:34 https://www.youtube.com/watch?v=74Du4Ej_-yM&t=3514s Closing Thoughts and Future of AI-Native Development This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:04Alessio Fanelli?

  Hey everyone, welcome to the Legends Space Podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-hosts, Wix, founder of SmallAI.

• 0:12Speaker 01

  Hey, and today we're very blessed to have both founders of Factory AI. Welcome.

• 0:16Speaker 02

  Thank you for having us.

• 0:17Alessio Fanelli?

  Yeah, thank you.

• 0:18Speaker 01

  Matan and Iino. My favorite story about the founding of Factory is that you met at the Langchain Hackathon. And I'm very annoyed because I wasn't at that hackathon and I didn't start a company.

• 0:29Speaker 01

  I didn't meet my co-founder. You know, maybe you want to quickly sort of retell that little anecdote because I think it's always very fun.

• 0:35Speaker 02

  Yeah, yeah, both Iino and myself went to Princeton for undergrad. And what's really funny is retrospectively, we had like 150 mutual friends, but somehow never had a one-on-one conversation. If you pulled us aside and asked us about the other, we probably knew like vaguely like what they did, what they were up to, but never had a one-on-one conversation. And then at this Langchain Hackathon, you know, we're walking around and catch a glimpse of each other out of the corner of our eye, you know, go up, have a conversation and very quickly just gets into code generation. And this was like back in 2023 when code generation was all about baby AGI and AutoGPT. Like that was like the big focus point there. And both were speaking about it, both were very obsessed with it.

• 1:18Speaker 02

  And I like to say it was intellectual love at first sight because basically every day since then, we've been obsessively talking to each other about AI for software development.

• 1:28Speaker 01

  If I recall, that Langchain Hackathon wasn't about code generation. How do you sort of get, find the idea maze to Factory?

• 1:34Speaker 03

  Yeah, basically, I think that we both came at it from slightly different angles. I was at Hugging Face, working primarily on advising like CTOs and AI leaders at Hugging Face's customers, guiding them towards how to think about research strategy, how to think about what models might pop up. And in particular, we had a lot of people asking about code and code models in the context of we all want to build like a fine-tuned version on our codebase. In parallel, I had started to explore building, at the time, the concept of agent wasn't really like clearly fleshed out. But imagine basically a while loop that wrote Python code and executed on it for a different domain for finance.

• 2:17Speaker 03

  On my mind was how not very helpful it felt for finance and how incredibly interesting it felt for software. And then when I met Matan, I believe that he was exploring as well.

• 2:30Speaker 02

  Yeah, that's right. So I was, at the time, I was still doing a PhD at Berkeley, technically in theoretical physics. Although for a year at that point, I had really switched over into AI research. And I think the thing that pulled me away from string theory, which I'd been doing for like 10 years, into AI was really the string theory and physics and mathematics really makes you appreciate fundamentalness or things that are very general. And the fact that capability in code is really core to performance on any LLM. And loosely, the better any LLM is at code, the better it is at any downstream task, even that's like writing poetry.

• 3:09Speaker 02

  And that fundamental beauty of like how code is just core to the way that machines develop intelligence really kind of nerd sniped me and got me to leave what I was pursuing for 10 years. And that mixed also with the fact that code is one of the very few things, especially at the time, that you could actually validate. And so you could have that agentic loop where the LM is generating the output and you're actually verifying in ground truth the quality of that output. It just made it extremely exciting to pursue.

• 3:40Alessio Fanelli?

  How did you guys decide that it was time to do it? Because I think maybe if you go back, the technology is like, it's cool at a hackathon. But then as you start to build a company, it's maybe like there's a lot of limitations. How did you maybe phase out the start of the company of like, okay, the models are not great today, so let's maybe build a harness around it to then now the models are getting a lot better. So it's time to like go GA as you're doing now and all of that.

• 4:03Speaker 02

  There's kind of a more quantitative answer and then a more qualitative answer. So the qualitative answer, kind of building off of what I said before of, you know, it was intellectual love at first sight. I think it was also one of those things that was kind of just like, if you know, you know. We met and we got along so well and basically like the next 72 hours, we didn't sleep. We were just like building together on initial versions of what would become Factory and When something like that happens, I think it's good to just like lean in and not really question it and overanalyze. Yet at the same time, if you do actually go and analyze, I think there are exactly the considerations that you're talking about, which is, yeah, the models at the time, which I think at the time it was just 3.5, which was out. Certainly, that's not enough to have a fully autonomous engineering agent. But very clearly,

• 4:48Speaker 02

  if you build that harness or you build that scaffolding around it and bring in the relevant integrations or the sources of information that a human engineer would have, it's very clear how that trajectory would get to the point where more and more of the tasks that a developer would do actually come under that line where you can automate it.

• 5:04Speaker 03

  Yeah, and I think that at the time, as you mentioned, there was like baby AGI and a couple of these other concepts that had come out, which involved putting a while loop around the LLM, feeding back some context. And on the other hand, there were papers coming out, the chain of thought, self-reflection. Of course, the scaling law papers at this point had been somewhat established. And so there was kind of this clear direction where models were going to get better at reasoning. They were going to get better at having larger context windows. They were going to get cheaper at, or at least the Pareto frontier of the model capabilities was going to be expanding so that good models would get cheap. The best models might stay the same price, but they start to get really smart. And this was, I

• 5:48Speaker 03

  wouldn't say super obvious, but if you spent a lot of time just reading through these papers and working through, there was definitely a rumbling amongst most of the people in the community that that was going to continue to extend. And so you blend all of these together with kind of meeting somebody who has this kind of energy that clearly they want to build. And I think that it became really obvious that the opportunity was available. I also think that we made a lot of very solid progress on the initial demo, enough to convince ourselves this was actually going to be possible.

• 6:18Speaker 02

  To be clear, though, it was eight days from us first meeting to me dropping out of my PhD and Eno quitting his job. So there was analysis, but it was also just like, yeah, let's do it.

• 6:29Speaker 03

  Yeah, pretty crazy, like eight days for sure. Yeah.

• 6:33Alessio Fanelli?

  My first company was a hackathon project, and I dropped out of school to actually found the company with one of my best friends. So the story resonates.

• 6:39Speaker 01

  I think I'm doing hackathons wrong.

• 6:42Alessio Fanelli?

  Maybe.

• 6:42Speaker 01

  Like I've had, I met one girlfriend out of it. No, that was about it.

• 6:46Alessio Fanelli?

  Hey, that's, you know, some people might say that's how you are.

• 6:49Speaker 02

  Is it still ongoing?

• 6:50Alessio Fanelli?

  No.

• 6:52Speaker 01

  I mean, it's a part of the funnel.

• 6:56Alessio Fanelli?

  So yeah, maybe CodeGen was not the topic of the hackathon back then, but I would say today, every other event that I go to, CodeGen is part of it. There's a lot of CodeGen products. Do you guys want to just talk about what Factory is? And maybe just give a quick comparison on the different products that people might have heard about and that we can kind of dive deeper?

• 7:14Speaker 02

  Our focus is on building autonomous systems for the full end-to-end software development lifecycle, and in particular for enterprises. I think, especially given the like, so okay, obviously code generation is very exciting. A lot of, you know, the best engineers coming out of any of the, you know, the popular schools. It's like, they want to work on like RL, they want to do cool things with like GPUs, you know, training models. Code is one of the like, most obvious things because it's very easy to resonate with if you're an engineer. And so that's led to a lot of the players in the space really focusing on coding in particular, and on like solo developers or building a quick like zero-to-one project. And really like that use case that appeals to that profile.

• 7:55Speaker 02

  And I think something that we're focused on is the relatively underserved enterprise perspective, which is there are hundreds of thousands of developers who work on codebases that are 30-plus years old. It's really ugly, it's really hairy, it's really messy. And if you made a demo video, like doing some like COBOL migration, that's not very sexy. You wouldn't go viral, you wouldn't get a lot of views because it's like, it's just not that visually appealing. But the value that you can provide and how much you can improve those developers' lives is very, very dramatic. And so seeing that underserved group is kind of why we've focused our sights there.

• 8:32Speaker 03

  Yeah, and I would add that there are a lot of really interesting constraints that people take for granted in the broader market as being like fundamental to the coding assistant, the SDLC assistant kind of market. In particular, a lot of the players look at a platform that has been the dominant tool for software developers, the IDE, right? And you think, this is a tool that was designed 20-plus years ago or has been iterated on for 20-plus years, primarily for a human being to write every line of code. When you take a tool like that and you start to introduce AI, you start to introduce additional constraints that exist just out of the nature of where you're interacting with these systems and where those systems live.

• 9:17Speaker 03

  So for example, latency matters a lot when you're inside of an IDE. The cost, when you are local first and your typical consumer is on a free plan or a like $20 a month paid plan, limits the amount of high quality inference you can do and the scale or volume of inference you can do per like outcome. When you are freed of a lot of these constraints, you can start to more fundamentally reimagine what a platform needs to look like in order to shift from a very collaborative workflow, which is what I think we see with most tools today, and a more delegative workflow, where you're actually managing and delegating your tasks to AI systems. So I think that the the product experience of delegation is really, really immature right now.

• 10:00Speaker 03

  And most enterprises, though, see that as the holy grail, not like going 15% or 20% faster.

• 10:06Speaker 01

  And you call them droids.

• 10:08Speaker 03

  We call them droids.

• 10:09Speaker 01

  Is there a just story behind the naming of either Factory or Droids?

• 10:13Speaker 02

  Yeah, so we were initially incorporated as the San Francisco Droid Company.

• 10:19Speaker 01

  Really?

• 10:20Speaker 02

  We were.

• 10:20Speaker 03

  Is this before?

• 10:21Speaker 02

  You have to bleep that out in the in the live podcast. Uh, sorry?

• 10:24Speaker 01

  Oh, you had to beep that out?

• 10:25Speaker 02

  No, no, no, no. Our our legal team advised us that Lucasfilm is particularly litigious and that we should change our name. And at the time, so at the time, while we were thinking of renaming, I was still in my PhD because we incorporated like two days after we met, which was also ridiculous. But we're thinking of renaming and I was still in some ML class at Berkeley and I was reading a paper on actor critic. In there, there was some equation that was like, you know, some function of the actor, we're just calling that Y. And so it was like F of A equals Y, A is actor. Put the actor in there and then it's, you know.

• 11:01Speaker 01

  Factory.

• 11:02Speaker 02

  Yeah. And so that's, uh, that's how it originally came about. It actually works quite well also, you know, automation, that sort of thing, but.

• 11:08Speaker 01

  Yeah, yeah, yeah.

• 11:09Speaker 03

  Yeah, and also the factory method, I think, was we, at some point, we had that written up. And then I think that inspired this line of thinking. And droids kind of remained because we felt that there was a lot of hype at the time around the concept of agent. But it referred to such a specific thing that everybody saw, which was this like endless while loop unreliable system that just kind of went on and on and took a bunch of actions without guidance. And our thought process was, well, that's not really what our system looks like at all. So even though underneath it is an agentic system, do we need to say that we're an agent company? Doesn't really make sense.

• 11:48Speaker 01

  I kind of like that, like, yeah, actually, last year, we pushed, even though I put you guys, you spoke last year at the, well, it's fair. I put you on the agent's track, but I almost didn't have an agent's track because I was like, this is so well ill-defined. And I think that instinct is good, but now, now the agent's wave has kind of come, come back the other way. And now everyone's an agent. I think like defining your own term for it and just getting out of that debate, I think is, is a positive. Would you like, is it closer to a workflow, which is like, I guess, the more commonly industry accepted term now?

• 12:17Speaker 03

  Yeah, no, it's a great question. I think that the original version of the droids were a lot closer to what we called workflows. So they were asynchronous and like event-based. They would trigger and each one had a specific purpose.

• 12:29Speaker 01

  Kind of deterministic, top to bottom.

• 12:30Speaker 03

  Exactly, like semi-deterministic. That was the original version. I think what we've grown to as the models have evolved and as our ability to build out guardrails and the system has just improved. We've gotten to the point where when you interact with droids inside the factory platform, they are relatively unbounded in the path that they take. And they're in general guided mainly by the concept of planning, decision-making and environmental grounding. So they can stay loosely goal-oriented over a long duration without needing very hardcoded guardrails. But they still tend to hit their goal according to their original plan. So I think now agent actually is probably like the proper way to describe them.

• 13:12Speaker 01

  Sure, but you know, I think, I think droids have a nice ring to it.

• 13:16Speaker 02

  It's also funny, our customers really, really love droids as a name.

• 13:20Speaker 01

  These are the droids you're looking for.

• 13:22Speaker 02

  I cannot tell you how many times, like if, you know, with an enterprise customer, we'll do a POC and like, you know, a day later, you know, they're excited and like things go well, they'll share a screenshot and be like, these are the droids we're looking for. And honestly, every time, it's just, it's so funny.

• 13:36Speaker 01

  I know, and everyone thinks they're the first to make a joke, but it really is better than like agents or intern or like, you know, autonomous.

• 13:45Speaker 02

  Another human name, like the number of human name AI products.

• 13:48Speaker 01

  Yeah, yeah, yeah. It's actually a pretty good insight.

• 13:50Speaker 03

  A hundred percent. And, and actually, I think we, we to a certain extent, take a bit of an objection to the idea that these things are a replacement for a human being. I think that very much as we work through harder and harder problems with agents, it's become more clear that the outer loop of software development and what a software developer does, planning, talking with other human beings, interacting around what needs to get done, is something that's going to continue to be very human driven, while the inner loop, the actual execution of writing lines of code or writing down the doc, is probably going to get fully delegated to agents very soon.

• 14:25Speaker 01

  You just need to put Roger Roger once they ask a question.

• 14:29Speaker 03

  When it finishes the task.

• 14:31Speaker 02

  We, we have that emoji in our Slack and use it very frequently.

• 14:34Speaker 01

  Roger Roger. Do we want to do a quick demo?

• 14:37Speaker 03

  Yeah, happy to jump in. When you land on the platform, you're presented with the opening dashboard. We try to make it really obvious that there are different droids available for key use cases that people tend to have. Of course, you can always go and speak with a default droid that can do a lot of things pretty well. But what we've learned is that there's three major use cases that people keep coming back to the platform for. The first is knowledge and technical writing. So that's going in and more of like a deep research style system that will go and do some research. It will use tools available to its search, et cetera, and come back with either a high quality document or answers, and then you can back and forth.

• 15:20Speaker 03

  The code droid is really the one that's the daily driver for a lot of folks, and this system allows you to actually I'll jump into that in a second. We can actually go through a full walkthrough. And then the reliability droid. This was pretty surprising to us, the degree to which people love doing incident response, the kind of like SRE style work inside of the platform. I guess in retrospect, it's nice because no one loves to to be on call at 3am waking up, being like, what's happening? And being able to just pass an incident description or say, hey, something's going wrong. And have a system really compile all the evidence, write up an RCA, provide that for you is super high leverage. And so that's actually one of the more popular droids that people use. But I can start by just going into the code droid.

• 16:06Speaker 03

  And when you start a session with a droid, you're presented with this interface. It's a little different from typical where we see a lot of tools really want to focus you in on the code. Our perspective is that code is important to review when it's completed. But as the agent is working, what matters most is seeing what the agent is doing. And having a bit of like an x-ray into its brain. So we have an activity log on the left and a context panel on the right. So you'll notice as we go through this task, that context panel starts to get updated. And I'm going to start by just doing something that's pretty common entry point. I'm going to paste a ticket into our platform. We have integrations with a bunch of different stuff, linear, Jira, Slack, GitHub, Sentry, PagerDuty, you name it.

• 16:50Speaker 03

  We have a bunch of these integrations that our enterprise clients have wanted over time, such that you can easily pull this info in. And if I were to say something like, hey, can you help me with this ticket? And then I'm going to use my at command, which lets me easily reference code or code bases. Hey, can you help me with this ticket in Factory Mono? I like to be nice to them. I'd love for your help. And so you'll note that right off the bat, the droid starts working. It's doing a semantic search on part of my query in that code base. And actually, the system has access to a bunch of different tools here. Memory, project management tools, GitHub, web search. Right now, the code droid only has search enabled by default. But you'll note that as the system starts working, it may actually want those additional tools added.

• 17:35Speaker 03

  So that it can, they can do its job.

• 17:37Speaker 02

  Maybe an important note there is, you know, as we deploy these droids in the enterprise, I think something that we're pretty like ideological about is everyone expects these, you know, agentic systems to perform at the level of a human, right? Because that's what they're always going to compare them to. But in a lot of cases, they'll have these agents just in the IDE. And that's like the equivalent of onboarding a human engineer, just like throwing them into your code base and being like, all right, like go. But like the reality is when you onboard a human engineer, what do you actually onboard them to? Slack, Notion, Linear, like Datadog, Sentry, Pedro. They have all of these other information sources that they need to actually be a productive engineer. And yes, in theory, if you're like really, really good and you don't need contextual information,

• 18:19Speaker 02

  you could just work based on code, but like that would be a lot harder and it would probably take a lot more time.

• 18:24Speaker 03

  A hundred percent. And having those connections ends up being super important as it works through harder problems. Like in particular, you can see that the first thing it did after that search was reference some of the information that it found in saying, hey, this is what I found so far. It gives an initial crack at a plan, right? Presents that really clearly to you and then goes to ask clarifying questions. So a lot of users, we believe, should not need to prompt engineer agents, right? If your time is being spent hyper optimizing every line and question that you pass to one of these systems, you're going to have a bad time. And so a lot of what we do is to be able to format. If I say, help me with this ticket, right? There's clearly going to be some ambiguities.

• 19:06Speaker 03

  The system knows when you give a very detailed answer or request to follow your instructions. When you give more ambiguous requests to ask for clarification. This is actually a really tricky thing to get right in the model, but we spend a lot of time thinking about it. And, and so I'm just going to answer some of these questions. Are there any UI mockups? No, try to imitate the other examples. Two, only preview when the button is clicked. Three, no, it's actually not implemented. Four,

• 19:40Speaker 02

  Just answering back and forth.

• 19:41Speaker 03

  Which specific fields must be displayed? Your choice. And five. Your choice. So now I'm basically saying to it, you decide for some, giving my preferences on others.

• 19:52Speaker 02

  And this, this is really the balance of like even, you know, delegation that will happen to non-AI is like, you know, as a good manager, you give autonomy to people that work with you when needed, but also if you're like, hey, I'm a little worried about this, or I'm going to be really strict about what I expect here. You want to extract that behavior as well, because a lot of times, like you know, mentioned, if you give a really poor prompt and you just say, hey, go do it. It's going to go do it, but it'll probably make assumptions. And then at the end, you might not be happy, but that's just because there were some constraints in your head that you didn't actually explicitly mention when you're communicating. So.

• 20:25Speaker 03

  Yep. A hundred percent.

• 20:26Alessio Fanelli?

  Do you guys have like a template that you've seen work? When I'm boarded to Devon, for example, they have the fixed prompt button. And then it refills it in their template, which is like, give the agent instruction on how to debug, give agent instruction on how to do this, and they ask you to fill out these things. Do you guys have the, a similar thing where like you think for each project, these are like the questions that matter, or is that more dynamic?

• 20:51Speaker 03

  No, that's a great question. And it's something that we talk about a lot internally is, it's surprising how many people are building products that have reactive like information requests like, you know, please fill out this form to explain how to do this thing, or you need to set up this dev environment yourself manually in order for this to work. We think of trying to be proactive with a lot of this stuff. So you'll notice in the right hand corner, there's this project overview, right? And the system started to code after doing some search. So that's going to pop up while we do this, but when I click into this project overview, what you're going to see is basically a, a, and I'm hiding it because I'm realizing this is actually semi-sensitive. It's probably fine for them.

• 21:35Alessio Fanelli?

  We can hide that in the video.

• 21:36Speaker 03

  Yeah, no worries. It's totally fine for, for folks to see that it's a mono repo. If I scroll down, that's when we'd get in a little bit of trouble. But inside that project overview, we're actually synthesizing a bunch of what we call synthetic insights on top of the codebase. And that is looking at things like how to set up your dev environment. What is the structure of the codebase? How to import? And modules connect to each other. And as we index code bases, we're actually generating these insights at a much more granular level across the entire code base. We think that in general, systems should be proactive in finding that information. However, with features like memory and we have like a .droid .yamal, you can set a bit of your guidelines. However, we also feel that it's like that XKCD about standards.

• 22:21Speaker 03

  Right, like everyone's got like a .blank rules file. And so we ingest those automatically from all of the popular providers as well. Wow, okay.

• 22:29Alessio Fanelli?

  Does something like a cursor rules is complementary? Because people might take this and then work on it in cursor separately.

• 22:36Speaker 03

  Yeah, what we found is that there are sometimes extraneous advice in those because people need to give a lot more guidance to those types of tools than they do to ours. So our system parses through and only picks the things that we don't already know.

• 22:52Speaker 02

  Another thing that kind of comes to mind related to your question, and this is something we've been thinking about a lot as well, is as we have more and more enterprise customers and a lot of the developers in the enterprise are not going to be as up to date on every new model and how it changes its behavior. Something that's interesting that we're thinking about is these developers are getting familiar with factory and how to get the most out of it. And then we, let's say, when we upgraded from Sonnet 3.5 to 3.7, we suddenly had a lot of developers being like, hey, wait, it now does this less or it does this more. What's happening? Or when they go to Gemini, let's say, and they want longer context. And so something that I think is interesting is how much of the like behavior difference from the models should we act as like a shock absorber for?

• 23:34Speaker 02

  So that they can basically, as a user, use it exactly how they've been using it before and get the same sort of output. But then also, how much of that do we actually want to translate to the user? Because presumably, over the next three years, the way you interact with models will change. And it's not just going to be up to behavior, but it's rather like, I guess it's alpha versus like beta in the model. Like there's some models have like different personalities and it's just the way you prompt it to get the same out of it. And then there are others where it's like, I mean, for example, like the reasoning models, they just work in a fundamentally different way. And so you, as the user, should know. How to interact differently. So that's something that's kind of fun to wrestle with.

• 24:10Speaker 03

  How do you evaluate the new models? We listened a lot to how the model providers actually think about building out their eval suites. And in particular, trying to look at things like desired behavior versus actual behavior. And in a way that's sustainable for a small team, we don't have like, you know, a hundred million dollars to pay data providers. And so a lot of the evaluation ends up being a combination of point, like task based evals. So like the ADR has an awesome benchmark that we built on top of internally for code editing and file generation. Versus we also for the top level agent loop have like our own behavioral spec.

• 24:49Speaker 03

  Where we set a bunch of high level principles, we break those down into tasks, those tasks then have grades and rubrics. And then we try to run those in order to determine is the behavior suite that we like. For example, asking questions when it's ambiguous versus not asking questions, does that match up? And we also use that to optimize the prompts as well.

• 25:09Speaker 01

  Just a quick question on these types of things. I think every company should have their own internal evals, right?

• 25:13Speaker 03

  Yeah.

• 25:14Speaker 01

  That is not in question. And obviously that is your IP, so we can't know too much about it. But like, what is the right amount to spend on something like this?

• 25:23Speaker 03

  Yeah.

• 25:23Speaker 01

  Because let's say we talked about Sweet Bench before recording, Sweet Bench costs like eight thousand to run. I've heard varying numbers between eight to fifteen thousand to run.

• 25:31Speaker 03

  Yeah.

• 25:31Speaker 01

  That's high, but you should be able to spend some amount to ensure that your system as a whole works and doesn't regress. So like, what's a rule of thumb for like, what is the right amount to spend on this?

• 25:42Speaker 03

  Yeah, I mean, I think it's important to separate out the two purposes of benchmarks, which one is marketing. And like, there are so many customers that we have that was purely because they saw the charts.

• 25:54Speaker 01

  Yep.

• 25:54Speaker 03

  And they saw big bar versus little bar and they were like, okay, we want to go with big bar. Which is funny, but that's just, I mean, that's just the way things go. And so I think that motivates. I think that's actually a good thing because that motivates more resources to be put on, you know, benchmarking and evaluation. On the other hand, you know, there definitely is a risk of going like too far in that direction or like even go.

• 26:15Speaker 02

  Getting to the point where you're fine tuning just to satisfy some benchmark there. And so

• 26:20Speaker 01

  Like we were saying before the taping, like you guys don't bother competing on Sweetbench anymore because that's not that relevant.

• 26:26Speaker 02

  Yeah, that and also like just in the enterprise, the use cases are pretty different than those represented in something like Sweetbench. So we do have pretty rigorous internal benchmarks as well, but I think also there's a certain extent to which like the vibe based or the sentiment based internally actually matters a lot because who has like a more intimate understanding of the behaviors of these models than the people who work on it every single day, like working with them and building with them. Because I mean, we use factory internally every single day. And so when we switch a model, we very quickly get a sense of how things are changing.

• 27:00Speaker 03

  Definitely. And I think that those task based evals tend to be the ones where it's most critical that we hill climb continuously on versus the the top level evals, they change so much with the new model providers that we try to make sure that they have some degree of consistent behavior, that the feel is smart. But the top level agent is actually not that responsible for what most people call quality. That ends up being, is it fast, accurate and high quality code edits? Does it call tools with the right parameters? Is the tool design such that that model can easily fit into it? And we have noticed a lot of really interesting behaviors with as the new models that have a lot heavier RL

• 27:43Speaker 03

  on post-training related to their own internal like agentic tools. So for example, Sonnet 3.7 clearly has, it smells like Claude code, right? Same with Codex, it very much impacted the way that those models want to write and edit code such that they seem to have a personality that wants to be in a CLI based tool. What's interesting is how do we combat the preferences that RL brings into the product? For example, search with CLI is like grep and glob. But what if you gave it a search tool that was way better than grep or glob at finding precisely what you wanted? But the model just really loves to use grep.

• 28:25Speaker 03

  They're gonna fight each other. And so our evals have to figure out, how do we make sure that as we build tools that are better than what maybe the model providers have in their slightly more toy examples? That the models use those with their full extent. And that's actually been a very interesting novel challenge to us that only started happening in the last three to six months as these new models have come out.

• 28:48Alessio Fanelli?

  Does that make you want to do more reinforcement fine tuning on these models? Like, kind of take more of that matter into your own hands, or?

• 28:55Speaker 03

  I definitely think that it's an interesting idea, but our take in general is that freezing the model at a specific quality level and freezing the model at a specific data set just feels like it's lower leverage than continuing to iterate on all these external systems. And it also feels like this is a bit of a bug. Like, we spoke with a bunch of the research labs, and I don't think that they actually want this type of behavior. What it is ultimately is it's a reduction in generalization.

• 29:24Speaker 01

  Cool. Anything else to see on the demos?

• 29:25Speaker 03

  Oh, yeah. I mean, we can, uh.

• 29:27Speaker 01

  It's still coding.

• 29:28Speaker 03

  Yeah, yeah. So.

• 29:30Speaker 01

  Let me go on him.

• 29:31Speaker 03

  Yeah, so you can see here that we're running.

• 29:33Speaker 01

  Oh, because you gave it like a whole bunch of things.

• 29:35Speaker 03

  Yeah, so I actually gave it like quite a large project to do, to execute live in front of us. Gotta earn its keep.

• 29:41Speaker 01

  Yeah.

• 29:41Speaker 03

  Yeah. This is why this delegation style flow we see is like really different, where in general, we expect the answer or output of this to just be correct, right? It's running code, it's iterating on code, it's making edits to a bunch of different files, it's gonna have to run pre-commit hooks and test all this stuff. I think that this is a big difference in workflow, right? Where we've just had a podcast conversation. Meanwhile, the agent is working on my behalf. This is probably gonna be mergeable at the end of this. It's ideally gonna create a pull request and we can check in on it at the end. But I think that this difference is like, what would I be doing right now? I think today, a lot of people just like open up their phone maybe and start browsing, or they go context switch to a different task.

• 30:27Speaker 03

  But the real power is unlocked when you start to realize this is the main thing that I'm gonna be doing is only delegating these types of tasks. And so you start jumping to, okay, while this is happening, let me go and kick off another task and another one and another one. And so being cloud native, being able to parallelize these like, I'm only sharing one tab, but if I just open another one and started right now, we support that natively. I think that this feels a little bit more like how people are going to work, where you maybe start the day setting off a bunch of tasks in motion, and then you spend the rest of it on maybe harder intellectual labor, like thinking about which of these is actually highest priority to execute on.

• 31:07Speaker 02

  And this actually goes into something that Eno was mentioning a little bit before, but also like a question that I'm sure everyone, when they see this, is going to ask, which is, why is this browser-based? Why is this not in the IDE? Like, I'm used to coding in the IDE. And the kind of higher level answer here is that, and Eno was alluding to this before, like, the last 20 years, the IDE was built for this world where developers are writing every single line of code. And something I think everyone can agree on is that over the next few years, what it means to be a software developer is going to change dramatically. Now, some people disagree, and some people say there will be no more software engineers. Some people say there will be, everyone's going to be a software engineer and everywhere in between. But the reality is, very clearly in the next few years, the amount of lines of code written by a human will go down.

• 31:51Speaker 02

  Like, the percentage of code written by humans will go down. And our take is that it is very unlikely that the optimal UI or the optimal interaction pattern for this new software development, where humans spend much less time writing code, I think it's very unlikely that that optimal interaction pattern will be found by iterating from the optimal pattern when you wrote 100% of your code, which was the IDE. Internally, we talk a lot about the Henry Ford quote, which is, you know, if you ask people what they want, they would say faster horses. And for us, the analogy here is like, can you iterate your way from a horse to a car? And there's like this very grotesque, like ship of Theseus, you can imagine of like trying to turn a horse into a car. It doesn't really look pretty. And our take is, you know, even though the world was built for horses at a certain point in time,

• 32:36Speaker 02

  right, like there were stables everywhere throughout a city, you were used to feeding. This thing and you're taking it with you everywhere. And it is kind of a higher barrier to entry to start introducing this new means of transportation. In this analogy, we are taking that more ambitious angle of like, everything is going to change about software development. And in order to find that optimal way of doing it, you do need to think from scratch, think from first principles about what does that new way to develop look like? And to give some early answers that we are pretty clear about is the time developers spend writing code is going to go way down. But in turn, the time that they spend understanding and planning is going to go way up. And then also the time that they spend testing so that they can verify that these agents that they delegated to

• 33:19Speaker 02

  did indeed do the task correctly, that's going to go way up. The promise of test-driven development is going to finally be delivered with this world of AI agents that are working on software development, because now if you do want to delegate something like this while you're doing a podcast and come back later, ideally, you don't even need to check their work and you just merge the PR. But how do you do that with confidence? You need to be really sure that the tests that you put up and said, hey, you know, Droid, you're not going to be done until you pass all of these tests. If you wrote those tests well, then you can, all right, great, pass the test, let's merge it. I don't even need to go in and see how it did everything.

• 33:54Speaker 01

  I mean, sometimes you do have to break the tests because you're changing functionality.

• 33:57Speaker 02

  Yeah.

• 33:58Speaker 01

  Yeah, there's a whole bunch of hard problems, but I just wanted to tap off the sort of visual component of the thing. There's one thing you haven't shown, which like there's like a built-in browser. So like I have a next year's project here that I'm running, the conference website. It spun it up itself. When I tried it out in ChatGPT Codex, it didn't work out of the box and it didn't have a browser built in. So it's nice that you have that kind of stuff.

• 34:18Speaker 02

  No, for sure. Like, like being able to view like HTML SVG, et cetera, on demand is is super nice. And I think it's pretty much wrapped up. It actually finished these changes. I think it's like roughly 12 files that it edited, created. And so, you know, right after this, because of the GitHub tool, I would just say, go ahead and create a pull request.

• 34:37Speaker 01

  Okay.

• 34:37Speaker 02

  And then it'd be good.

• 34:38Speaker 01

  Amazing.

• 34:39Speaker 02

  Yeah, good stuff.

• 34:40Speaker 01

  And you even show like a little 43% of context size used. That's actually not that much, given that this is factory's own code base.

• 34:46Speaker 02

  Yeah, and this is actually a large monorepo. I think the

• 34:50Speaker 03

  The big thing that I'd love for people to try out is like, look at how efficient it is. It's able to really execute on precisely what it needs to edit with a relatively lower token usage than other agentic tools. Obviously, if you're just getting autocomplete, that's going to be a little bit more expensive. But compared to other agents where you get like five credits and it takes a while to execute on anything, I think they'll see a better experience with Factory.

• 35:15Speaker 01

  Yeah. When you started saying things like, oh, we can pull in from Notion, we can pull it from Slack, like that sounded like a lot of context. You're gonna have to do pretty efficient rag to do this, right? Like, I guess it's not even rag, it's just retrieval.

• 35:28Speaker 03

  Yeah.

• 35:29Speaker 02

  Yeah. But there is the temptation and I remember maybe a year ago, there was really a lot of hype on large context. Because right, it's the dream of you can be super lazy and just throw in your whole code base, throw in everything at it.

• 35:40Speaker 01

  Which is what Claude Code does, right?

• 35:41Speaker 02

  Right, yeah, exactly. But I think the one downside of that is, okay, great, like if you do have billion token context window model, you throw it all in there, still gonna be more expensive. The reason why retrieval is so important for us is because even if there is a model that's gonna have these larger context windows, and certainly over time, we're gonna get larger context windows, you still want to be very cost efficient. And this is something that our customers care a lot about and they see a lot of the value in the effort that we put in on retrieval, because they'll see like, wait, this was a huge mono repo and I gave it all this information, but then I see for each actual call, you're really good at figuring out what do I actually need, as opposed to just like throw the whole repo in and pray that it works.

• 36:19Speaker 01

  You mentioned the credits. What's the pricing model of the product?

• 36:23Speaker 02

  We're fully usage based, so the tokens, I think for us, it's really important to like, respect the users and their ability to understand what this stuff means. And so I think like all the stuff around like credits and all, it just kind of obfuscates what's actually happening under the hood. And I actually think that we get better users, the more they understand what tokens are and how they're used, you know, in each back and forth.

• 36:46Speaker 03

  Yeah, so it's a direct bill through to the, we call them standard tokens and it's benchmarked off of the standard models that we have. So like right now, when you get access to the platform, your team would pay like a small Fixed price for just access. Every additional user is another like very small fixed price. And then the vast majority of the spend would be on usage of of the system. And so I think that this is just nicely aligned where you get a sense of how efficient it is about the token usage. This is a big reason why we've tried really hard to make it more token efficient. And then you can track, of course, in the platform how you're using it. And I think that a lot of people like to not only see just like raw usage.

• 37:29Speaker 03

  And this kind of gets into like tracking success, something that a lot of people do by maybe like number of tabs that you accepted or chat sessions that ended with code. For us, we try to look a little bit further and say, look, you use this many tokens. But, you know, here are the deliverables that you got. Here are the pull requests created. Here's merged code. We help enterprise users look at things like code churn, which it turns out the more AI generated code you have. If the, you know, if the platform isn't telling you the code churn, there's a reason for that.

• 37:58Speaker 01

  Code churn, meaning amount of code deleted versus added?

• 38:01Speaker 03

  Yeah, it's basically a metric that tracks when you, uh, it it's kind of like a variability in a given line of code.

• 38:07Speaker 02

  It's very imperfect because like some people, some people will say like the difference between code churn and like refactored code is like somewhat arbitrary because it's a time period at which, okay, if I merged some line and then I changed that line. If you change that line in a shorter period, it'll churn versus a longer period, it'll count as like a refactoring, which is like a little.

• 38:29Speaker 03

  So generally in enterprise code bases, if you merge a line of code and then change that code within three weeks, it's because something was wrong with that code generally. It's not always true, but it's a useful metric.

• 38:40Speaker 02

  It averages out because sometimes it's like, wait, what if you just had like an improvement or, you know, some, some change that wasn't about quality? But.

• 38:46Speaker 01

  But like, is code churn up bad?

• 38:49Speaker 03

  Yes, because what it tends to be is that in a, in very high quality code bases, you'll see 3%, 4% code churn when they're at scale, right? This is like millions of lines of code. In poor code bases or poorly maintained code bases or early stage companies that are just changing a lot at once, you'll see numbers like 10 or 20%. Now, if you're at Lassian and you have 10% code churn, that's a huge, huge problem because that means that you're just wasting so much time. If you're an early stage startup, code churn is less important. This is why we don't really like report that to every team, just enterprises.

• 39:27Speaker 01

  Any other like measurements are popular that I mean, you know, this is nice that I'm hearing about Cochurn, but like, what else are enterprise VPs of N, CTOs, what do they look at?

• 39:38Speaker 02

  For the enterprise, I think the biggest thing, because there's so many tricks and different dances you can do to like justify ROI.

• 39:44Speaker 01

  Number of commits.

• 39:45Speaker 02

  Like number of like.

• 39:46Speaker 01

  Lines of code.

• 39:46Speaker 03

  Dora metrics are usually popular.

• 39:48Speaker 02

  And at the end of the day, like what we, we initially went really hard in all the metric stuff.

• 39:52Speaker 01

  Yeah.

• 39:53Speaker 02

  What we found is that oftentimes if they liked it, they wouldn't care. And if they didn't like it, they wouldn't care. And so in the, at the reality, like at the end of the day, no one really cares about the metrics.

• 40:02Speaker 01

  Hmm.

• 40:02Speaker 02

  What people really care about is like developer sentiment. When, when you're kind of playing that game, at the end of the day, if you want to do a metric, talk to developers and ask if they feel more productive. Or if you're a large enterprise and you want to justify ROI, the biggest thing that we've seen and like, that's allowed us to deploy very quickly in enterprises is pulling in timelines on things. So there's this one very large public company that we work with and pulling in just a large migration task from taking four months to taking like three and a half days. That is the best ROI that you don't need to measure this or that like trim. Like we had something that was going to be delivered in the next quarter and we got it done this week with no downtime. Like that is, you know, music to a VP of Engineering's ears.

• 40:44Speaker 01

  Yeah.

• 40:44Speaker 02

  And so that's what we tend to focus on is like pulling in deliverables or increasing the scope of what you can get done in a quarter.

• 40:50Speaker 01

  In order to achieve a very large refactor like you just described, do we use that same process you just saw or does that more setup?

• 40:57Speaker 03

  I think that the workflow for let's say a migration is probably one of the most common. I can even give a very concrete example. Let's say you are the administrative service of a large European nation, right? Like Germany or Italy. And you have a hospital system that runs on like a 20 year old Java code base.

• 41:17Speaker 01

  Yep.

• 41:17Speaker 03

  Now a company wants to come in and big four consulting firm or something like that says, we would like to transform this entire code base to Java 21. It's going to take X amount of time, couple of months. And, you know, by the end you'll be on a relational database, you'll be into the future, right? On Java 21. When that typically happens, you kind of have to almost break. Break down what that means from a human perspective first to then map it to how it works on our platform. You'll have a team of anywhere from four to 10 people come in. And you have a project manager who is going to work with engineers to analyze the code bases, figure out all the dependencies, map that out into docs, right? First, analysis and overview of the code base. The next is a migration strategy and plan.

• 42:00Speaker 03

  The third is like timelines and you're going to scope all this out. What do you do next? Will you go to a project management tool like Jira? And so you take those documents and a human being translates that out. We've got two epics over the next two months. This epic will have these tickets, that epic will have these tickets. And then you find out the dependencies and you map those out to humans. Now, each of these humans are now operating such that one after the other, they're knocking out their work mainly in parallel, but occasionally pieces have to connect, right? One person misses and now the whole project gets delayed about a week. And so this interplay of understanding, planning, executing on the migration incrementally and then ultimately completing.

• 42:45Speaker 03

  Now, there's a handoff period. There's docs of the new artifacts that we've created. There's all this information. You map that over to a system like ours. One human being can say, please analyze this entire code base and generate documentation, right? And that's one pass, one session in our platform. Analyze each of the modules. We already do a lot of this behind the scenes, which makes this a lot easier. And actually generate an overview of what the current state is. You can now pull those docs in with real code and then say, what's the migration plan, right? If there's some specific system, you can pull in docs. And then when you have this, our system connects with linear jury, it can create tickets.

• 43:26Speaker 03

  Just create the epic, ticket this whole process out and figure out which are dependencies and which can be executed in parallel. Now you just open up an engineer in every browser tab, right? And you execute all of those tasks at the same time. And you as a human being, just review the code changes, right? This looks good, merge, did it pass CI? Okay, great. Onto the next one. This looks

• 43:47Speaker 02

  Good merge. So a process that typically gets ultimately bottlenecked, not by like skilled humans writing lines of code, but by bureaucracy and technical complexity and understanding now gets condensed into basically how fast can a human being delegate the tasks appropriately? So it it happens outside of like one session, like what we just saw, which would be one of those tasks. But the planning phase, that's really where we see enormous condensation of time.

• 44:14Alessio Fanelli?

  We just talked about your pricing, your your just usage base, but you tempted to have four deployed engineers that just like the the current meme right now to execute these large things.

• 44:24Speaker 02

  Yeah. So I think this is something we definitely do a little bit of for our larger customers, just because, you know, like we said at the beginning, this is the way we think software development will look, and it's an entirely new behavior pattern. And I think it would be a little naive to just be like, hey, we have this new way of doing things. Go figure it out yourselves. Right? So we definitely go in and, you know, help show them how to do. So like in this migration example that I was mentioning before, we worked with them like side by side, but just us and like two of their engineers showed them how to do it. They were like, they saw the light, if you will. And then they ended up being the internal influencers within their org, teaching everyone else how to do it. But if you want to change behavior, you can't just assume that the product is going to be so good that everyone's going to immediately get it.

• 45:09Speaker 02

  Because developers, you know, we need to know who we're we're selling to, and developers have very efficient ways of working that they've built out over the last 20 years. And we want to make sure that we accommodate that and earn their trust and slowly bring them into this new way of building. And to do that, we need to extend that olive branch and, you know, come meet them where they are, show them how they can do new things.

• 45:29Alessio Fanelli?

  We did an episode with Together AI maybe a year ago or so, and we were talking about what inference speed actually we needed. And they always argued, we need to get to like 5,000 tokens a second. And we're chatting whether or not that makes sense because people cannot really read it. As you think about factory, how much do you think you're like bound by the speed of these models? Do you know, like, if the models were like a lot faster, would you just complete things quicker? Would you like maybe? And out more in parallel like what are like the limits of the models today?

• 46:01Speaker 02

  I want to let you know answer this, but immediately every time this thing comes up, I always just think about the memory that Chrome tabs take, which is like it's never enough and you always want more, but then it's also it lets you be lazier.

• 46:13Alessio Fanelli?

  Yeah.

• 46:13Speaker 02

  But anyway, I want to.

• 46:14Speaker 03

  Yeah, no, for sure. And and I think that the this is kind of a funny question. It it kind of has two directions. It's like practically, would this make a big difference on someone who knows and loves and uses our platform on a daily basis? I think you would probably improve the quality of life such that, yeah, definitely faster tokens would be awesome. I think that actually where this impacts is for those who haven't yet made the jump from collaboration to delegation. So if you are used to very high latency, high feedback experiences, then that speed difference and seeing like most of that delegation happen very quickly. And then being able to immediately jump in, I think feels very nice.

• 46:54Speaker 03

  And so for the larger enterprise deployments where they start to familiarize themselves with how this works and the migrations, I don't think this actually makes a big difference because most of the bottleneck ends up being, like I mentioned, almost like bureaucratic in nature, but for the average developer, I think this improves the user experience to the point where it would feel very magical. So I'm I think we could get a lot faster.

• 47:18Speaker 02

  It probably wouldn't change what's possible, but it would really, really change like ease of adoption for people who maybe aren't as in the weeds on AI tools.

• 47:26Speaker 03

  And and if you combine though, the latency with a cost reduction as well, I do think that cost is one of the reasons why we haven't so greatly scaled out. Like originally, we had a lot of techniques that would generate a lot of stuff in parallel. And we still know how to do that and we're very excited to to bring that. But right now we don't do it because it's cost prohibitive and the quality delta is not enough to justify the cost increase.

• 47:51Speaker 01

  I have a kind of a closing question, if you don't mind. This is more or less asking you on like a limiting factor. It's basically four questions in one. So what do you see as your limiting factor right now in terms of models? Basically, like what capabilities would really help you? Hiring, what skills are really hard to hire? Customers, like what, you know, do you really want to unlock that you're like? It's like weirdly not working, but like you have an ICP, which is a more enterprise doing well. But like, you know, what's like the next one? And then like finally dev tooling, like what do you wish existed that you had to build for yourself? Or you just feel like could be a lot better?

• 48:27Speaker 03

  Yeah, maybe I'll do models and dev tool and you can take hiring and customers. I mean, I'm thinking right off the bat, probably the biggest thing is models that have been post-trained on more general agentic trajectories over very long time spans. That feels like something that is, there's an effort for that right now. But what I mean is an hour or two hours or three hours of seriously working on a hard problem, such that the model knows how to keep that long-term goal-directed behavior the whole time. That is something that I assume we'll get soon.

• 49:07Speaker 01

  OpenAI has put out that like operator benchmark.

• 49:10Speaker 03

  Yeah.

• 49:11Speaker 01

  That was that he had a human testers like actually try for two hours and give up.

• 49:15Speaker 03

  Yeah.

• 49:15Speaker 01

  Did you see that one?

• 49:16Speaker 03

  Yeah, yeah, yeah. I mean, I think that that's exactly the type of work that we want to see taken further, because that I would argue is probably one of the bigger blockers.

• 49:25Speaker 01

  I would say like, would you ever do that yourself? I don't see you guys as customizing your own models a lot, but you work with the frontier labs, right? But like, is there a point where you would just be like, all right, screw it, like we'll do it?

• 49:37Speaker 03

  We are currently building benchmarks with a lot of the post-training techniques very much in mind right now. And so I don't know what exactly at this point in time, how much we're going to commit to that. But for sure, we will be using those benchmarks for our own internal goals. And if we need to use them later on for post-training, I think that there's a lot of compatibility. And then maybe for dev tools. Um, since I'll just, I'll just jump that one.

• 50:03Speaker 01

  Yeah.

• 50:03Speaker 03

  It is still surprising to me that uh observability, it remains like very challenging.

• 50:11Speaker 01

  Really? There are like 80 tools out there.

• 50:12Speaker 03

  I know, for sure. And Langsmith is actually fantastic. Like, we use.

• 50:16Speaker 01

  Are you guys a Langchain shop?

• 50:17Speaker 03

  Uh, we don't use Langchain, but we just use Langsmith.

• 50:19Speaker 01

  Oh, okay.

• 50:20Speaker 03

  And, and Langsmith is awesome.

• 50:22Speaker 01

  See, the hackathon is our wife or Harrison.

• 50:24Speaker 03

  I know, yeah. No, and like, like, they, they've been fantastic and that's been cool. But I think that there is, it's really tricky to deal with enterprise customers where you can't see their code data at all. But you're trying to build a product where you can improve the experience. And a lot of it is actually subjective. It's like, I don't like the way this code looks. That remains something very unclear to us is how do you build almost like semantic observability into your product? Uh, I think Amplitude and uh, and Statsig and a lot of the feature flag companies actually are closer to this than.

• 50:56Speaker 01

  Like product analytics, actually, really. It's more about like, observe if they still, if they're on the platform. Like basically anything other than up and down uh thumbs, right?

• 51:06Speaker 03

  Yeah, and like, and like, what was the user's intent when they entered into this session, right? It's the type of thing where you almost need LLMs in the observability itself.

• 51:13Speaker 01

  Are you saying they've actually done it or Amplitude could do it?

• 51:16Speaker 03

  Uh, it, that's where I would like to see it.

• 51:18Speaker 01

  Yeah, as far as I understand, it hasn't really, you know.

• 51:21Alessio Fanelli?

  Yeah, because mostly everything is kind of like span based. They're like, oh, the observability products are.

• 51:26Speaker 01

  We started trace.

• 51:27Alessio Fanelli?

  But not at the.

• 51:28Speaker 01

  Yeah.

• 51:28Speaker 03

  Yeah, exactly. Not, not in the like semantic direction.

• 51:31Alessio Fanelli?

  Yeah, yeah. So for you, that's kind of solved in a way. It's like the actual traces, it's like you can get that information anywhere.

• 51:38Speaker 03

  Yeah, our, our team comes from like Uber and all these amazing places where they, they know how to do that part. I think the more tricky thing is when human beings have like messy intents that are natural language. How do you really classify and understand when users are having a good time versus when they're having a bad time?

• 51:55Speaker 01

  Okay.

• 51:55Speaker 03

  That's hard.

• 51:57Speaker 02

  Great, hiring and customers. Yeah, so maybe I'll start with customers. So we've been at it for just over two years now. I think the first like year and a half was really, really focusing in on the product and what is the, the interaction pattern that works for the enterprise. And over the last 90 days, the deployments that we've had with like the large enterprise and Fortune 500 has been exploding. It's been going really well. It's very exciting. How are they mostly finding you? So this is a good point. This is part of why we're, you know, doing more podcasts is because so far we've really just relied on word of mouth, like working well with one enterprise and then, you know, they're at some CEO dinner, they mention it to someone else. And that's great, by the way, like that's why we have the conference to like, we put all the C, the VPs in one room.

• 52:40Speaker 02

  Totally. And so like, it's worked really well, but I think when every one of those conversations ends up leading to a happy customer, that means you need to increase top of funnel. And so accordingly, we're really putting fuel on the fire for our go to market for Fortune 500 large enterprises, which is obviously, it's a very exciting thing to do where the team has been pumped. I mean, there was a particular day in January of this year where one of those large enterprises basically had the magic moment of like, if I was the only one at my company using this, I would still tell them to like, have me use this instead of hiring like three engineers for myself. One of the biggest moments for us where it was like, people in the enterprise are really getting dramatic value out of factory and that kind of kicked off this, really like the last 90 days

• 53:23Speaker 02

  has just been a whirlwind. So getting to more of these Fortune 500 companies is kind of top of mind for us right now. To that end, you know, as you serve Fortune 500 customers, it becomes important to have a larger go to market team, both on the sales side, the customer success side, and then also of course, on the engineering side. So we are very much hiring. Yeah, I think like everyone's hiring. It's just like, what are you finding that's hard to hire? Oh, I see. Like the particular roles. Yeah. Yeah. Like what's the rate limiter here? I think a big rate limiter is like, for us going to these Fortune 500 companies, one of the most important things is having both that ability to, you know, talk to the CIO, VP of engineering and have that sales presence, but then also the

• 54:06Speaker 02

  Ability to be sitting side by side with some of their developers and jumping into the platform, jumping into their use cases and having that.

• 54:13Speaker 01

  So you need like a hundred Eno's.

• 54:15Speaker 02

  Basically, yeah. Literally, our profile, actually, our profile, when we're like looking for this role is like, is this a junior Eno or not? Like, that's, that's basically the, uh, the template there.

• 54:26Speaker 03

  Yeah, I don't know about that, but I definitely think that if you are highly, highly technical, but you want to be a founder, you want to move into a role where you are interfacing with CIOs, CTOs, like this, we have like three maybe of these roles that are probably going to be the most important like roles in our go-to-market team. And so I think that's a huge opportunity for anyone interested in what we've talked about.

• 54:52Speaker 02

  We joke that this person would basically be my best friend because any trip we go to, to fly to a customer, they'd be there with me, you know, talking to whoever is the, the buyer, as well as, you know, going in, working with the engineers. So I'm also, I guess, hiring a best friend.

• 55:07Alessio Fanelli?

  I thought that's what AI was gonna be for.

• 55:09Speaker 02

  Yeah.

• 55:10Alessio Fanelli?

  I guess not. Just to wrap, I think we're all fans of your guys' design and kind of like brand via, which is very.

• 55:16Speaker 01

  Speaking of best friends.

• 55:17Alessio Fanelli?

  Yeah.

• 55:17Speaker 01

  You know.

• 55:18Alessio Fanelli?

  Who does your design?

• 55:19Speaker 02

  Yeah, so, uh, a huge privilege of, you know, Factory has been working with my older brother, Cal, who, uh, who joined us. He, you know, moved from, he was in New York for five years. He moved out to San Francisco. He actually, even before he moved, he's the one who designed our logo way back when. And, you know, he's been a part of Factory from the very beginning, and it's been an absolute pleasure working with him from the brand design, the marketing design, and then, of course, the, the product and the platform itself. I cannot recommend enough working with a sibling.

• 55:50Speaker 01

  Sure. I mean, you know, not, not, not all of us are lucky to have that. What do you learn from working with a designer like that? I think a lot of technical people listen to us, want to build a startup, they don't have the polish that you have. They don't have the hype.

• 56:03Speaker 02

  Yeah, I think, I think a big part of this is like, one of our core operating principles is like embracing perspectives. And so Cal is not an engineer. And I think what's great is a majority of our team are engineers. And having that ability to come in. With that design perspective and then also the engineering perspective and like bash those two things together until we get something perfect out of it. That has been really, really important. I think it's a lot of times it's kind of easy to fall victim to. Oh, I'm building like I'm the profile who I'm building for. So I know what's best. And that obviously works a lot of the time, but sometimes there are some like core design tenants that you just might not think of. If you're building for yourself. So I think that's been, that's been pretty important there.

• 56:43Speaker 03

  Yeah, and we do live in a very AI native company or operate in a very AI native company. So being able to have someone set principles that are then consumable by our own agents, right? Design systems and consistency. I think it's pretty surprising the degree to which. Even like droids can actually imitate a brand voice and style that Cal created for us. And so a lot of that comes from not just the droids doing that, from our entire team of product engineers. Are all incredibly thoughtful about what they're putting in front of users. And I think that they're able to bring a lot of that into it in a way that feels like safe and on brand.

• 57:22Speaker 02

  And also have fun. Like I think factory, like our like semi tongue in cheek slogan is like the machine that builds the machine. Like it's fun. Does it transmit exactly what it is that we do in the most clear way?

• 57:33Speaker 01

  No, the factory doesn't build factory.

• 57:34Speaker 02

  Yeah, like we don't, but it, but to a certain extent, it's like, you know, software that writes software, right? The machine that builds the machine.

• 57:41Speaker 01

  Yeah.

• 57:42Speaker 02

  It's fun.

• 57:42Speaker 01

  When you say fun, it's more, actually I see you guys hosting a lot of events at your office. And like to me, that's like, oh, like these guys are actually social, you know?

• 57:49Speaker 02

  Yeah, I think it's important for us because not only is this incredibly transformational, but it's also like. These are people that we spend all of our time with and we want to make sure that while we're doing it.

• 57:58Speaker 01

  It's like right next to the Caltrain, you can like advertise out of your window.

• 58:02Speaker 02

  Yeah, no one, no one peek in though. There are a lot of, a lot of secrets in there.

• 58:07Speaker 01

  It's pretty sweet. Cool. I mean, you know, I'm very excited for your talk, you know, like we, we touched on a few things I'm interested in, right? Like we're seeing tiny teams is a topic that I'm seeing like, you know, one person can do a lot more. So like the average team size is really shrinking. The interaction of AI design and engineers, that's also another thing I'm exploring. Like I think we're really trying to like push the frontier and then obviously there's always like the suite and stuff, which is always ongoing. So like, yeah, there's a lot of interesting work going on.

• 58:34Speaker 02

  And one interesting addendum there, there are sometimes. Individuals who weren't even really developers, who will use Factory and have more usage than in a hundred-person enterprise.

• 58:45Speaker 01

  Yeah.

• 58:45Speaker 02

  Which is crazy to see. There's some really interesting dynamics that we've seen play out, just in how people use these tools, whether it's like for that design or for that like small, small team use case.

• 58:54Speaker 01

  Yeah, there's an AI-native attitude that like is going to set people apart if they're just open to it, but then also like they're maybe not, they don't drink too much Kool-Aid. I think there's a, there's a medium there.

• 59:05Speaker 02

  Yeah.

• 59:06Alessio Fanelli?

  Thank you guys for coming on. This was fun.

• 59:08Speaker 01

  Thanks for having us.

• 59:08Speaker 02

  Thank you guys for having us. This was awesome.