Automating Software Engineering: Genie Tops SWE-Bench, w/ Alistair Pullen, from Latent.Space podcast

In this special crossover episode of The Cognitive Revolution, Nathan shares an insightful conversation from the Latent. Space podcast. Swyx and Alessio interview Alistair Pullen of Cosine, creators of Genie, showcasing the cutting edge of AI automation in software engineering. Learn how Cosine achieves state-of-the-art results on the SWE-bench benchmark by implementing advanced AI techniques. This episode complements Nathan's recent discussion on AI Automation, demonstrating how far these practices can be pushed in real-world applications. Don't miss this opportunity to explore the future of AI-driven software development and its implications for businesses across industries. Check out the Latent. Space podcast here: https://www.latent.space Apply to join over 400 Founders and Execs in the Turpentine Network: https://www.turpentinenetwork.co/ SPONSORS: WorkOS: Building an enterprise-ready SaaS app? WorkOS has got you covered with easy-to-integrate APIs for SAML, SCIM, and more. Join top startups like Vercel, Perplexity, Jasper & Webflow in powering your app with WorkOS. Enjoy a free tier for up to 1M users! Start now at https://bit.ly/WorkOS-Turpentine-Network Weights & Biases Weave: Weights & Biases Weave is a lightweight AI developer toolkit designed to simplify your LLM app development. With Weave, you can trace and debug input, metadata and output with just 2 lines of code. Make real progress on your LLM development and visit the following link to get started with Weave today: https://wandb.me/cr 80,000 Hours: 80,000 Hours offers free one-on-one career advising for Cognitive Revolution listeners aiming to tackle global challenges, especially in AI. They connect high-potential individuals with experts, opportunities, and personalized career plans to maximize positive impact. Apply for a free call at https://80000hours.org/cognitiverevolution to accelerate your career and contribute to solving pressing AI-related issues. Omneky: Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ CHAPTERS: (00:00:00) About the Show (00:00:22) Sponsors: WorkOS (00:01:22) About the Episode (00:04:29) Alistair and Cosine intro (00:13:50) Building the Code Retrieval Tool (00:17:36) Sponsors: Weights & Biases Weave | 80,000 Hours (00:20:15) Developing Genie and Fine-tuning Process (00:27:41) Working with Customer Data (00:30:53) Code Retrieval Challenges and Solutions (00:36:39) Sponsors: Omneky (00:37:02) Planning and Reasoning in AI Models (00:45:55) Language Support and Generalization (00:49:46) Fine-tuning Experience with OpenAI (00:52:56) Synthetic Data and Self-improvement Loop (00:55:57) Benchmarking and SWE-bench Results (01:01:47) Future Plans for Genie (01:03:02) Industry Trends and Cursor's Success (01:05:23) Calls to Action and Ideal Customers (01:08:43) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share a special crossover episode from the Late in Space podcast, co-hosted by Swix and Alessio. For any that don't already know, Late in Space is the number one podcast for practical insights for AI engineers. And while longtime listeners may remember that The Cognitive Revolution was voted number three in the same poll,

• 0:45Nathan Labenz?

  and I can at times be a bit competitive about these things, in this case, I have to agree with the wisdom of the crowd. The Late in Space podcast is a regular must-listen for me, and I am grateful to Swix for allowing me to share this episode here. I wanted to cross-post Swix and Alessio's conversation with Alastair Pullan of Cosign, makers of Genie, in particular, because it serves as such an excellent compliment to my recent episode on AI automation. In that episode, I covered the core best practices needed to automate routine business processes, which, by the way, are pretty much exactly the same as you'd use to evolve a software application from DIY to done-for-you-by-AI. And this conversation really demonstrates just how far that paradigm can be pushed. Alastair and his team are implementing

• 1:30Nathan Labenz?

  almost exactly what I outlined. They are very thoughtfully breaking complex work into smaller AI-manageable parts, curating gold standard examples, using synthetic data generation to fill in critical reasoning chains, and fine-tuning intensively to achieve the desired AI behavior. They just happen to be doing it with an unusually high level of ambition, a significant budget for data curation, creation, and fine-tuning, and one major additional step of weaving all of the individual optimized workflows together into an AI coding agent. The key takeaway here is clear. If Cosign can achieve a new state-of-the-art on the popular software engineering benchmark Sui Bench with this approach, there is no reason to doubt that you can apply these techniques in your own business

• 2:14Nathan Labenz?

  if you buckle down and do the work. With that in mind, as I recently mentioned, I'm exploring working with a few companies as an AI automation advisor. I've recently started one project, helping a company evolve their application from DIY to done for you by AI. And I have bandwidth for maybe one or two more projects right now. So if you're interested, please reach out and we'll see if it makes sense to work together. As always, if you're finding value in the show, we appreciate it when folks take a moment to share it with friends. And we always invite your feedback and suggestions, either via our website, cognitive revolution.ai, or by DMing me on your favorite social network. And I really do encourage you to subscribe to the latent space podcast if you haven't already. Another recent episode with Michelle Pocras of OpenAI,

• 2:57Nathan Labenz?

  which includes a deep dive on the structured outputs API and other underappreciated features of OpenAI's APIs was particularly valuable for me. I've been using some of the techniques that they discussed in just the last two weeks since that episode aired. And I have no doubt that latent space will once again provide some of the best and most comprehensive coverage of this week's OpenAI Dev Day, which will again help me raise my game. Now, I hope you enjoy this glimpse into the cutting edge of AI automation with Alistair Pullan of Cosign from the latent space podcasts.

• 3:34Speaker 01

  Hey, everyone. Welcome to the latent space podcast. This is Alessio, partner and CTO and resident at Decibel Partners, and I'm joined by my co-host, Swix, founder of SmallAI.

• 3:42Speaker 02

  Hey, and today we're back in the studio in person.

• 3:46Speaker 01

  We are.

• 3:46Speaker 02

  After about three to four months in visa jail and travels and all other fun stuff that we talked about in the previous episode. But today with special guest, Ali Pullan from Cosign. Welcome.

• 3:57Speaker 03

  Hi, thanks for having me.

• 3:59Speaker 02

  Very lucky to have you because you're on a two day trip to San Francisco.

• 4:03Speaker 03

  I would not recommend it. Don't fly from London to San Francisco for two days.

• 4:06Speaker 02

  And you launched Genie on a plane, on plane wifi, claiming state of the art in Sweetbench, which we're all going to talk about. I'm excited to dive in into your whole journey because it has been a journey. I've been lucky to be a small angel in part of that journey. And it's exciting to see that you're launching to such a claim. Such results. So I'll go over your brief background and then you can sort of fill in the blanks on what else people should know about you. You did your bachelor's in computer science in Exeter.

• 4:36Speaker 03

  Yep.

• 4:37Speaker 02

  And then you worked at a startup that got acquired into GoPuff. And around about 2022, you started working on a stealth startup that became a Y C startup. What's that overall story?

• 4:46Speaker 03

  So basically, when I left university, I I met my now co-founder Sam. At the time, we were both mobile devs. He was an Android developer. I was an iOS developer. And whilst at university, we built this sort of small consultancy, sort of we'd be approached to build projects for people. And we would just take them up and start with they were student projects. They weren't. They weren't anything crazy or anything big. We started with those and over time, we started doing larger and larger projects, more interesting things. And actually, when we left university, we just kept doing that. We didn't really get jobs, traditional jobs. It was also like in the middle of COVID, middle of lockdown. So we were like, this is a pretty good gig. We'll just keep like writing code in our bedrooms. And we did that for a while.

• 5:30Speaker 03

  And then a friend of ours that we went to Exeter with started a Y C startup during COVID. And it was one of these fast grocery delivery companies. At the time, I was living in the deepest, darkest countryside in England, where fast grocery companies are still not a thing. So he he sort of pitched me this idea and was like, listen, like, I need an iOS dev. Do you fancy coming along? And I thought, absolutely. It was a chance to get out my parents' house, a chance to move to London, you know, do interesting things. And at the time, truthfully, I had no idea what Y C was. I had no idea. I wasn't in the startup space. I knew I liked coding and building apps and stuff, but I'd never, never really done anything in that area. So I said, yes, absolutely. I moved to London just sort of as COVID was ending.

• 6:13Speaker 03

  And yeah, worked at what was Fancy for about a year and a half. Then we brought Sam along as well. So we, Sam and I were the two engineers at Fancy for basically his entire life. And we built literally everything. It's like the front, the client mobile apps, the the backends, the internal like stock management system, the driver routing algorithms, all those things, literally like everything. It was my first, you know, both of us were super inexperienced. We didn't have like proper engineering experience. There were definitely decisions we'd do differently now. We'd definitely buy a lot of stuff off the shelf, stuff like that, but. It was the initial dip of the toe into like the world of startups and we were both like hooked immediately. We're like, this is so cool. This sounds so much better than all our friends who are like consultants and doing like normal jobs, right?

• 6:58Speaker 03

  We did that and it ran its course and after I want to say eighteen months or so, GoPuff came and acquired us. And there was obviously a transitionary period, an integration period, like with all acquisitions. And we did that and as soon as we'd vested what we wanted to vest and as soon as we thought, okay, this chapter is sort of done. In about twenty twenty two, we left and we knew that we wanted to go it alone and try something like we'd had this taste. Now we knew, we'd seen how like a Y C startup was managed like up close. And we knew that we wanted to do something similar ourselves. We had no idea what it was at the time. We just knew we wanted to do something. So we tried a small, um, some small projects in various different areas. But then Sam talked to me about G P T three. He'd seen it on Reddit and I'm his. The source of our knowledge.

• 7:43Speaker 03

  Sam loves Reddit. I'd actually heard of G P T two and obviously had like loosely followed what OpenAI had done with. What was the game they trained a model to play? Dota. Was it Dota? Yeah. So I'd followed that and I knew loosely what G P T two was. I knew what Bert was. So I was like, okay, this G P T three thing sounds interesting. And he just mentioned it to me on a walk. And I then went home and like Googled G P T three and there was the playground. It was the, and the model was Da Vinci two at the time. And it was just the, the old school playground, completions, nothing crazy, no chat, no nothing. I miss completion still. Yeah, oh completion. Honestly, I had this conversation in OpenAI yesterday. I was like, I just, I know. But yeah, so we, um, I, I started playing around with the playground.

• 8:27Speaker 03

  And the first thing I ever wrote into it was like, hello, world. And it gave me some sort of like fairly generic response back. I was like, okay, that looks pretty cool. The next thing was I looked through the docs, um, or so they had a lot of example prompts because I had no idea. I didn't know if the, if you could put anything in. I didn't know if you had to structure in a certain way or whatever. And I, and I saw that it could start writing like tables and JSON and stuff like that. So I was like, okay, can you write me something in JSON? And it did. And I was like, oh, wow, this is, this is pretty cool. Um, can it, can it just write arbitrary? Jason for me. And immediately, as soon as I realized that my mind was racing and I like got Sam in and we just started messing around in the playground, like fairly innocently to start with. And then, of course, both being mobile devs

• 9:11Speaker 03

  and also seeing, at that point, we'd learned about what the codex model was. It was like, this thing's trained to write code. It sounds awesome. And Copilot was start, I think, I can't actually remember if Copilot had come out later. Yeah, it might have done. It's around about the same time. Around about the same time, yeah. And we were like, okay, as mobile devs, let's see what we can do. So the initial thing was like, okay, let's see if we can get this AI to build us a mobile app from scratch. We eventually built the world's most flimsy system, which was back in the day, we're like 4,000 token context windows, like chaining prompts, trying to keep as much context from one to the other, all these different things where essentially you'd put in an app idea in a box. And then

• 9:50Speaker 03

  we do like very high level stuff, figuring out what the stack should be, figuring out um what the front end should be written and back end should be written and all these different things. And then we'd go through like for each thing, more and more levels of detail until the point that you actually got codex to write the code for each thing. And we didn't do any templating or anything. We were like, no, we're going to write all the code from scratch every time, which is basically why it barely worked. But there were like occasions where you could put in something and it would build something that did actually run, the backend would run, the database would work. And we were like, oh my God, this is insane. This is so cool. And that's what we showed to our co-founder Yang. I met my co-founder Yang through through fancy because his wife was their first employee.

• 10:31Speaker 03

  And we showed him, he was like, you've discovered five. What is this? This is insane. He has a lot more startup experience. Historically, he's had a few exits in the past and has been through all different industries. He's like our dad. He's a bit older. He hates me saying that. He's your COO now? He's our COO, yeah. And we showed him and he was like, this is absolutely amazing. Let's just do something. Cause he, he at the time um was just about to have a child, so he didn't have anything going on either. So we applied to Y C, got an interview. The interview was, as most Y C interviews are, shortcut and pretty brutal. They told us they hated the idea and they didn't think it would work. And That's when we started brainstorming. It was almost like the interview was like an office hours kind of thing. And we were like, okay, given what you know about the space now and how to build things with these L L M's.

• 11:16Speaker 03

  Like, what can you bring out of what you've learned in building that thing into something that might be a bit more useful to people on the daily? And also Y C obviously likes B two B startups a little bit more, at least at the time they did back then. So we were like, okay, maybe we could build something that helps you with existing code bases, like can sort of automate development stuff with existing code bases. Not knowing at all what that would look like or how you would build it or any of these things. And they were like, yeah, that sounds interesting. You should probably go ahead and do that. You're in. You've got two weeks to build us an M V P. And we were like, okay, okay. We did our best. The M V P was absolutely horrendous. It was a C L I tool. It sucked. And at the time, we were like, we don't even know how to build what we want to build.

• 11:58Speaker 03

  And we didn't really know what we wanted to build, to be honest. Like, we knew we wanted to try to help automate dev work. But back then, we just didn't know enough about how L L M apps were built, the intricacies and all those things. And also like the L L M's themselves, like four thousand tokens, you're not going very far. They're extremely expensive. So we ended up building a code-based retrieval tool originally. Our thought process originally was, we want to build something that can do our jobs for us. That is like the gold star. We know that. We've seen like there are glimpses of it happening with our initial demo that we did. But we don't see the path of how to do that at the moment. Like, the tech just wasn't there. So we were like, well, there are going to be some things that you need to build this when the tech does catch up. So retrieval being one of the most important things, like the model's going to have to be able to like pull code out of a code base somehow.

• 12:42Speaker 03

  So we were like, well, let's just build a tooling around it. And eventually when the tech comes, then we'll be able to just like plug it into our our tooling. And then it should work, basically. And to be fair, that's basically what we've done. And that's basically what's happened, which is very fortunate. But in the meantime, whilst we were waiting for everything to sort of become available, we built this code-based retrieval tool. That was the first thing we ever launched when we were in Y C, like, and it didn't work. It was really frustrating for us because it was just me and Sam, like, working like all hours, trying to get this thing to work. It was quite a big task in of itself, trying to get like a good semantic search engine working that could run locally on your machine. We were trying to avoid sending code to the cloud as much as possible. And then for very large code bases, you're like, you know, millions of lines of code, you're trying to do some sort of like local H N S W thing

• 13:27Speaker 03

  that runs inside your V S code instance that like eats all your RAM as you've. Seen in the past, all those different things.

• 13:32Speaker 02

  Yep.

• 13:32Speaker 03

  Yeah.

• 13:33Speaker 02

  My first call with you.

• 13:35Speaker 03

  Yeah, it sucks, man. I'm like, yeah, I know, I know, I know it sucks. I'm sorry. Um, but building all that stuff was essentially the first six to eight months of, of what at the time was built.

• 13:48Speaker 02

  Which, by the way, build.

• 13:49Speaker 03

  Build, yeah, the terrible, terrible.

• 13:52Speaker 02

  That was the worst part of trying to think about whether I would invest is whether or not people could pronounce.

• 13:58Speaker 03

  Put out a name. No, and when we, so when we went on our first ever Y C like retreat, no one got the name right. They were like, build, build, build, well. Um, and then we actually changed the names cosign, like although some people spell it as in like, as if you're cosigning for an apartment or something like, it's like, can't win. Yeah, that was what built was back then. But the ambition, and I, I did a talk on this back in the end of 2022, the ambition to like build something that essentially automated our jobs was still very much like core to what we were doing. But for a very long time, it was just never apparent to us, like, how would you go about doing these things? Even when like you had 3.5 16K, 16K suddenly felt huge.

• 14:37Speaker 03

  Cause you've gone from four to 16, but even then 16K is like a lot of Python files are longer than 16K. So you can't, you know, before you even start doing a completion. Even then we were like, eh, yeah, it looks like we're still waiting. And then like towards the end of last year, you then start, you see 32K, 32K was really smart. It was really expensive, but also like you could fit a decent amount of stuff in it. 32K felt enormous. And then finally 128K came along. We were like, right, this is like, this is what we can actually deal with because fundamentally to build a product like this, you need to get as much information in front of the model as possible and make sure that everything it ever writes in output can be traced back to something in the context window. So it's not hallucinating it. As soon as that model existed,

• 15:21Speaker 03

  I was like, okay, I know that this is now going to be feasible in some way. We'd done early sort of dev work on genie using 3.5 16K and that was a very, very like crude way of proving that this loop that we were after and the way we were generating the data actually had signal. And worked and could do something, but the model itself was not useful because you couldn't ever fit enough information into it. For it to be able to do the task competently and also the base intelligence of the model, I mean, three point five, anyone who's used three point five knows the base intelligence of the model is. Is lacking, especially when you're asking it to like do software engineering is quite quite involved. So.

• 16:01Speaker 03

  We saw the hundred and twenty eight K context model and um at that point we'd been in touch with OpenAI about our ambitions and like how we wanted to build it. We essentially are, I just took a punt. I was like, I'm just going to ask to see, can we like train this thing? Cause at the time four turbo had just come out. And back then there was still a decent amount of lag time between like OpenAI releasing a model. And then allowing you to fine tune it in some way. They've gotten much better about that recently, like four oh fine tuning came out either, I think a day for a mini fine tuning came out like a day after the model did. And I know that's something they're definitely like optimizing for super heavily inside, which is great to see.

• 16:37Speaker 04

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 16:42Speaker 02

  Which is a little bit, you know, for a year or so, Y C companies had like a direct Slack channel to OpenAI.

• 16:48Speaker 03

  We still do.

• 16:49Speaker 02

  Yeah.

• 16:49Speaker 03

  Yeah.

• 16:50Speaker 02

  So it's a little bit of the diminishing of the Y C advantage there.

• 16:53Speaker 03

  Yeah.

• 16:53Speaker 02

  If they're releasing this fine tuning ability like a day after.

• 16:56Speaker 03

  Yeah, no, absolutely. But like, you can't build a startup on the Y C advantage. It's obviously nice. It makes you feel warm and fuzzy inside, but like at the end of the day, it's not that that's going to make you win.

• 17:05Speaker 02

  Yep.

• 17:06Speaker 03

  But yeah, no, so like we'd spoken to Shamil, their Devrel guy, I'm sure you know.

• 17:11Speaker 02

  He's solution, head of solutions or something.

• 17:12Speaker 03

  He is in their applied team. Yeah. We'd been talking to him from the very beginning when we got into Y C and he's been absolutely fantastic throughout. I basically had pitched him this idea back when we were doing it on three point five, sixteen K. And I was like, this is my, this is my crazy thesis. I want to see if this can work. And as soon as like that hundred and twenty eight K model came out, I was, I started like laying the groundwork. I was like, I know this definitely isn't possible because he released it like yesterday, but know that I want it. And in the interim, like G P T four, like eight K fine tuning came out. We tried that. It's obviously even fewer tokens, but the intelligence helped. And I was like, if we can marry the intelligence and the context window length, then we're going to have something special. And eventually we were able. To get on the experimental access program and we got access to for turbo fine tuning.

• 17:56Speaker 03

  As soon as we did that, because in the entire ramp to that, we'd built the data pipeline. We already had all that set up. So we were like, right, we have the data. Now we have the model. Let's put it through and iterate essentially. And that's, that's where like genie as we know it today really was born. I won't pretend like the first version of gene that we trained was good. It was disaster. That's where you realize all the implicit biases in your data set and you realize that, oh, actually this decision you made that was fairly arbitrary was the wrong one. You have to do it a different way. Other subtle things like, you know, how you write Git diffs in, you know, using L L M's and how you can best optimize that to make sure they actually apply and work and loads of different little edge cases. But as soon as we had access to the underlying tool, we were like, right, we can actually do this.

• 18:39Speaker 03

  And I was, I breathed a sigh of relief because I didn't know it was like, it wasn't a done deal, but I knew that we could build something useful. I mean, I knew that we could build something that would be measurably good on whatever eval at the time that you wanted to use. Like at the time, back then, we weren't actually that familiar with. But once Devin came out and they announced their sweep bench score, I like, that's when my life took a turn.

• 19:04Speaker 02

  Challenge accepted.

• 19:05Speaker 03

  Yeah, challenge accepted. And that's where like, yes, that's where my, my friendships have gone, my sleep has gone, my weight, everything gone into sweep bench. And yeah, we, it was actually a very useful tool in building genie because beforehand it was like, yes, vibe check this thing and see if it's useful. And then all of a sudden you have an actual measure to, to see like, couldn't it do software engineering? Not, not the best measure, obviously, but like it's, it's the best that we've got now. We were just iterated and, and built and eventually we got it to the point that where it is now and a little bit beyond since we actually like did, we, we actually got that score a couple of weeks ago. And yeah, it's been a hell of a journey from the beginning all the way now. That was a very rambling answer to your question about how we got here, but that's essentially a potted, yeah, answer how we got here.

• 19:47Speaker 02

  Got the full origin story.

• 19:48Speaker 01

  Yeah, no, totally. You mentioned bias in the data and some of these things. In your announcement video, you called genie the worst verse AI software engineering colleague. And you kind of highlighted how the data needed. To train it needs to show how a human engineer works. I think maybe you're contrasting that to just putting code in it. There's kind of like a lot more than code that goes into software engineering. How do you think about the data mixture? You know, and like, uh, there's this kind of known truth that code makes models better when you put in the pre-training data. But since we put so much in the pre-training data, what else do you add when you turn to Genie?

• 20:25Speaker 03

  Yeah, I think, well, I think that sort of boils down fundamentally to the difference between a model writing code and a model doing software engineering. Because the, the, the software engineering sort of discipline goes wider. Because if you look at something like a PR, that is obviously a artifact of some thought and some work that has happened and has eventually been squashed into, you know, some diffs, right? What the, very crudely, what the pre-trained models are, are reading is they're reading those final diffs and they're emulating that and they're being able to output it, right? But of course, it's a super lossy thing, a PR. You have no idea why or how for the most part, unless there are some comments, which, you know, anyone who's worked in a company realizes

• 21:05Speaker 03

  PR reviews can be a bit dodgy at times, but you see that you lose so much information at the end. And, and that's perfectly fine because PRs aren't designed to be something that perfectly preserves everything that happened. But what we realized was if you want something that's a software engineer, and very crudely, we started with like something that can do PRs for you essentially. You need to be able to figure out why those things happened. Otherwise, you're just going to rely, essentially you just have a code writing model. You have something that's good at human eval, but, but not very good at Sweetbench essentially. That realization was, was part of the, the kernel of the idea of, of the approach that we took to design the agent that, that is Genie.

• 21:44Speaker 03

  The way that we decided we want to try to extract what happened in the past, like as forensically as possible, has been and is currently like one of the, the main things that we focus all our time on. Because doing that is, getting as much signal out as possible, doing that as well as possible is the biggest thing that, that we've seen that determines how well we do on that benchmark at the end of the day. Once you've sorted things out, like, like output structure, how Consistently writing diffs and all the stuff that is sort of ancillary to the model actually figuring out how to solve a problem. The core bit of solving the problem is how did the human solve this problem and how can we best come up with how the humans solve these problems? So all the effort went in on that, on that pipeline.

• 22:28Speaker 03

  And the mix that we ended up with was, as you've probably seen in the technical report and so on, all of those different languages and different combinations of different task types. All of that has run through that pipeline and we've extracted all that information out.

• 22:40Speaker 01

  How does that differ when you work with customers that are private workflows? Like, do you think, is there usually a big delta between what you get in open source and maybe public data versus like?

• 22:50Speaker 03

  Yeah, when you scrape enough of it, most of open source is updating readme's and docs. It's hilarious. Like, we had to filter out so much of that stuff because when we first did the sixteen, three point five sixteen K model, like, the amount of readme updating that went in, we did like no data cleaning, no real like, we just sort of threw it in and saw what happened. And it was just like, it was really good at updating readme's, really good at writing some comments, really good at complaining in Git reviews and in PR reviews rather. And it would, again, like, we didn't clean the data, so you'd like give it some feedback and it would just like reply and like, it would just be quite insubordinate when it was getting back to you like, no, I don't think you're right. And they would just sort of argue with you.

• 23:30Speaker 03

  So the process of doing all that was super interesting because we realized from the beginning, okay, there's a huge amount of work that needs to go into like cleaning this, getting it aligned with what we want the model to do, to be able to get the model to be useful in some way.

• 23:43Speaker 01

  I'm curious, like, how do you think about the customer willingness to share all of this historical data? I've done a lot of developer tools investing in my career and getting access to the code base is always one of the hard things. Are people getting more cautious about sharing this information? In the past, it was maybe like, you know, you're using static analysis tool, like whatever else you need to plug into the code base, fine. Now you're building a model based on it, like, what's the discussion going into these companies? Are most people comfortable with like letting you see how to work and sharing everything or?

• 24:16Speaker 03

  It depends on the sector mostly. We've actually seen, I'd say, people becoming more amenable to the idea over time, actually rather more skeptical. Because I think they can see the the upside. If this thing does what they say it does, it's going to be more help to us than it is a risk to our infosec. And of course, like, companies building in this space, we're all going to end up, you know, complying with the same rules and there are going to be new rules that come out to make sure that we're looking at your code, that everything is safe and so on. So from what we've seen so far, we've spoken to some very large companies that you've definitely heard of. And all of them obviously have stipulations and many of them want it to be sandboxed to start with and all the like very obvious things that I, you know, I would say as well.

• 24:56Speaker 03

  But they're all super keen to have a go and see because like, despite all those things, if we can genuinely make them go faster, allow them to build more in a given time period and stuff, it's it's super worth it to them.

• 25:08Speaker 02

  Okay, I'm going to dive in a little bit on the process that you have created. You showed the demo on on your video.

• 25:15Speaker 03

  Yes.

• 25:15Speaker 02

  And by the time that we release this, you should be taking people off the waitlist.

• 25:18Speaker 03

  Correct, yes.

• 25:19Speaker 02

  Launching people so people can see this themselves. There's four main parts of the workflow, which is finding files, planning action, writing code and running tests. And controversially, you have set yourself apart from the devins of the world by saying that things like having access to a browser is not that important for you. Is that an accurate reading of what you wrote?

• 25:41Speaker 03

  I don't remember saying that, but at least with what we've seen, the browser is helpful, but it's not as helpful as like ragging the correct files. If that if that makes sense, like, it is still helpful, but obviously there are there are more fundamental things you have to get right before you get to like, oh, yeah, you can read some docs or you can read a Stack Overflow article and stuff like that.

• 26:01Speaker 02

  Yeah, the phrase I was indexing on was the other software tools are wrappers around foundational models with a few additional tools such as a web browser or code interpreter.

• 26:09Speaker 03

  Oh, I see. No, I mean, no, I'm not, I'm not, I'm not, I'm deriding the, the approach there, not the, not the tools.

• 26:14Speaker 02

  Yeah, exactly. So like, I would say in my standard model of what a code agent should look like, Devin has been very influential, obviously.

• 26:21Speaker 03

  Yeah.

• 26:22Speaker 02

  Because you could just add the docs of something and, you know, now I have, now when I'm installing a new library, I can just add docs.

• 26:29Speaker 03

  Yeah.

• 26:30Speaker 02

  Cursor also does this, right? And then obviously having a code interpreter does help. I guess you have that in the form of running tests.

• 26:36Speaker 03

  I mean, uh, the genie has both of those tools available to it as well. So, yeah, yeah. So we have a tool where you can like put in URLs and it will just read the URLs and it also uses Perplexity's API under the hood as well to be able to actually ask questions if it wants to.

• 26:49Speaker 02

  Okay.

• 26:49Speaker 03

  So no, we use both of those tools as well. Like those tools are super important and super key. I, I think obviously the most important tools to these agents are like being able to retrieve code from a code base, being able to read Stack Overflow articles and what have you, and just be able to essentially be able to Google like we do, is definitely super useful.

• 27:09Speaker 02

  Yeah. I thought maybe we could just kind of dive into each of those actions. Reach code retrieval, one of the core problems, you had an indexer that

• 27:16Speaker 03

  Yes.

• 27:17Speaker 02

  you've worked on, uh, even as, as built. What makes it hard? What approach you thought would work, didn't work, anything like that?

• 27:24Speaker 03

  It's funny, I had a similar conversation to this when I was chatting to the guys from OpenAI yesterday. The thing is that searching for code, specifically semantically at least to start with, I mean, like keyword search and stuff like that is, is a soul problem, has been around for ages, but at least being able to, the phrase we always used back in the day was searching for what code does rather than what code is, like searching for functionality is really hard, really hard. The way that we approached that problem was that obviously like a, a very basic and easy approach is right, let's just embed the code base, we'll chunk it up in some arbitrary way, maybe using an AST, maybe using a number of lines, maybe using whatever, like some overlapping, just chunk it up and embed it.

• 28:06Speaker 03

  And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the code similarity and get the top K right. That doesn't work, and I wish it did work, don't get me wrong. It doesn't work well at all because Is fundamentally, if you think about like semantically how code looks is very different to how English looks. And there's like not a huge amount of signal that's carried between the two. So what we ended, the first approach we we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take in English code queries and then produce a hypothetical code snippet that might look like the answer. Embed that and then do the code similarly.

• 28:50Speaker 03

  And that process, although very simple, gets you so much more performance out of the retrieval accuracy. And that was kind of like the start of our of our engine, as we called it, which is essentially like the aggregation of all these different heuristics, like semantic, keyword, LSP and so on. And then we essentially had like a model that would, given an input, choose which ones it thought were most appropriate, given the type of request you had. So the whole code search thing was a really hard problem. And actually, what we ended up doing with Genie is we let the model through self-play figure out how to retrieve code. So actually, we don't use our engine for Genie.

• 29:31Speaker 03

  So instead of like a request coming in and then like say GPT four with some JSON output being like, well, I think here we should use a keyword with these inputs and then we should use semantic and then we should like pick these results. It's actually like a question comes in and Genie has self-played in its training data to be able to be like, okay, this is how I'm going to approach finding this information, much more akin to how a developer would do it. Because if I was like, Sean, go into this new code base you've never seen before and find me the code that does this. You're gonna probably, you might do some keywords, you're gonna look over the file system, you're gonna try to figure out from the directories and the file names where it might be. You're gonna like jump in one and then once you're in there, probably gonna be doing the, you know, go to definition stuff

• 30:13Speaker 03

  to like jump from file to file and try to use the graph to like get closer and closer. And that is exactly what Genie does. Starts on the file system, looks at the file system, picks some candidate files. Is this what I'm looking for? Yes or no? If there's something that's interesting, like an import or something, it can it can command click on that. Then go to definition, go to references and so on. And it can traverse the code base that way.

• 30:33Speaker 02

  Are you using the VS Code LSP or?

• 30:36Speaker 03

  No, that's no, that we're not doing this in VS Code. We're just using the language servers running. But we really wanted to try to mimic the way we do it as best as possible. And we did that during the self-play process when we were generating the data set. So although we did all that work originally and although like Genie still has access to these tools, so it can do keyword searches and it can do, you know, basic semantic searches and it can use the graph, it uses them through this process and figures out, okay, I've learnt from data how to find stuff in code bases. And I think in our technical report, I can't remember the exact number, but I think it was around sixty five or sixty six percent retrieval accuracy overall. Measured on, we know what lines we need for these tasks to find, for the task to actually be able to be completed.

• 31:19Speaker 03

  And we found about sixty six percent of all those lines, which is one of the biggest areas of free performance that we can get hold of because when we were building Genie, truthfully, like a lot more focus went on assuming you found the right information, you've been able to reproduce the issue. Assuming that's true, how do you then go about solving it? And the bulk of the work we did was on the solving. But when you go higher up the funnel, obviously like the funnel looks like, have you found everything you need for the task? Are you able to reproduce the problem that's seen in the issue? Are you then able to solve it? And the funnel gets narrower as you go down. And at the top of the funnel, of course, is RAG. So I'm actually quite happy with that score. I think it's still pretty impressive considering the size of some of the code bases we're doing, we're using for this.

• 32:02Speaker 03

  But as soon as that, if that number becomes eighty, you think how many more tasks we get right? That's one of the key areas we're going to focus on when we continue working on Genie.

• 32:09Speaker 02

  Be interesting to break out a benchmark just for that.

• 32:12Speaker 03

  Yeah, I mean, it's super easy.

• 32:13Speaker 02

  Cause I don't know what state of the art is.

• 32:15Speaker 03

  Yeah, I mean, like for a, it's super easy because like for a given PR, you know what lines were edited.

• 32:20Speaker 02

  Oh, okay.

• 32:21Speaker 03

  Yeah.

• 32:21Speaker 02

  So you can just, you can source it from Subbench, actually.

• 32:23Speaker 03

  Yeah, you could do it. You could do it. It's super easily. And that's how we got that figure out at the other end. For us being able to see it against our historic models was super useful. So we could see if we were, you know, actually helping ourselves or not. And initially, one of the biggest performance gains that we saw when we were, when we did work on the RAG a bit was Giving it the ability to use the LSP to like go to definition and really try to get it to emulate how we do that. Because I'm sure when you go into an editor with that, where like the LSP is not working or whatever, you suddenly feel really like disarmed and naked. You're like, oh my God, I didn't realize how much I actually use this to get about rather than just find stuff. So we really tried to get it to do that and that gave us a big jump in performance. So we went from like 54% up to like the 60s, but just by adding focusing on that.

• 33:05Speaker 04

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 33:10Speaker 02

  One weird trick. Yes, I'll briefly comment here. So this is the standard approach, I would say, most code tooling startups are pursuing. The one company that's not doing this is magic.dev.

• 33:23Speaker 03

  Yes.

• 33:23Speaker 02

  So would you do things differently if you have a 10 million token context window?

• 33:28Speaker 03

  If I had a 10 million context window and hundreds of millions of dollars, I wouldn't have gone and built. It's an LTM. It's not a transformer, right? That they're using, right? If I'm not mistaken, I believe it's not a transformer.

• 33:41Speaker 02

  Yeah, Eric's gonna come on at some point.

• 33:43Speaker 03

  I'm sure, listen, they obviously know a lot more about their product than I do. I don't know a great deal about how magic works. Yeah, I don't know. I'm not gonna, I'm so I'm not gonna, I'm not gonna speculate. Would I do it the same way as them? I like the way we've done it because fundamentally like we focus on the act of software engineering and what that looks like and showing models how to do that. Fundamentally, the underlying model that we use is kind of null to us, like so long as it's the best one, I don't mind. And the context windows, we've already seen, you can get transformers to have like million, one and a half million token context windows. And that works perfectly well. So like, as soon as you can fine tune Gemini 1.5, then you best be sure that genie will have, will work, will run Gemini 1.5.

• 34:28Speaker 03

  And like, we'll probably get very good performance out of that. I like our approach because we can be super agile and be like, oh, well, anthropic have just released whatever, you know, and it might have half a million tokens and it might be really smart. And I can just immediately take my JSON L file and just dump it in there and suddenly genie works on there and it can do all the new things.

• 34:44Speaker 02

  Anthropic have the same fine tuning support as OpenAI? I actually haven't heard.

• 34:48Speaker 03

  They are working on it. They are partnered, they are partnered with AWS and it's going to be in bedrock. As far as, as far as I know, I think, I think, I think that's true.

• 34:56Speaker 02

  Cool. We have to keep moving on to the other segments. Planning, the second piece of your four step grand master plan. That is the frontier right now. You know, a lot of people are talking about strawberry, Q Star, whatever that is, Monte Carlo tree search. Is current state of the art planning good enough? What prompts have worked? I don't even know what questions to ask. Like, what is the state of planning?

• 35:17Speaker 03

  I think it's fairly obvious that with the foundational models, like you can ask them to think by step by step and ask them to plan and stuff, but that isn't enough because if you look at how those models score on these benchmarks and then they're not, they're not even close to state of

• 35:28Speaker 02

  Which ones are you referencing?

• 35:29Speaker 03

  So like just like Sweet Bench and so on, right? And like even the things that get really good scores on human evaluation agents as well because they have these loops, right? Obviously, these things can reason, quote unquote, but the reasoning is the model, like it's constrained by the model's intelligence, I'd say, very crudely. And what we essentially wanted to do was we still thought, obviously, reasoning is super important. We need it to get the performance we have. But we wanted the reasoning to emulate how we think about problems when we're solving them as opposed to how a model thinks about a problem when we're solving it. And that was, that's obviously part of like the derivation pipeline that we have when we, when we, when we design our data. But the reasoning that the models do right now, and who knows what Q Star, whatever it ends up being called, looks like. But

• 36:12Speaker 03

  certainly what I'm excited, on a small tangent to that, like what I'm really excited about is when models like that come out, obviously, the signal in my data when I regenerate it goes up. And then I can then train that model that's already better at reasoning with improved reasoning data. And just like I can keep bootstrapping and keep leapfrogging every single time. And that is like super exciting to me because I don't, I welcome like new models so much because immediately it just floats me up without having to do much work, which is always nice. But at the state of reasoning, generally, I don't see it going away anytime soon. I mean, that's like an auto regressive model doesn't think per se. And in the absence of having any thought, maybe an energy based model or something like that, maybe that's what Q Star is, who knows, some sort of like high level abstract

• 36:55Speaker 03

  space where thought happens before tokens get produced. In the absence of that for the moment, I think it's, it's all we have and it's going to have to be the way it works for what happens in the future. We'll have to see. But I think certainly it's never going to hinder performance to do it. And certainly, the reasoning that we see Genie do, when you compare it to like, if you ask GPT-4 to break down step by step an approach for the same problem, at least just on a vibe check alone, looks far better.

• 37:22Speaker 02

  Two elements that I like that I didn't see in your initial video, we'll see when, you know, this genie launches, is a planner chat, which is I can modify the plan while it's executing. And then the other thing is playbook switch also from Devon. Where here's how I like to do a thing and I'll use Markdown to specify how I do it. I'm just curious if like, you know, those things help.

• 37:45Speaker 03

  Yeah, no, absolutely. We're a hundred percent. We want everything to be editable, not least because it's really frustrating when it's not. Like if you're ever, if you're ever in a situation where like, this is the one thing I just wish I could, and you'd be right if that one thing was right. And you can't change it. So we're going to make everything as well, including the code it writes. Like you can, if it makes a small error in a patch, you can just change it yourself and let it continue and it will be fine.

• 38:04Speaker 02

  Yeah.

• 38:05Speaker 03

  So yeah, like those things are super important. We'll be doing those two.

• 38:08Speaker 01

  I'm curious, once you get to writing code, is most of the job done? I feel like the models are so good at writing code when they're like in small chunks that are like very well instructed. What's kind of the drop off in the funnel? Like, once you get to like, you got the right files and you got the right plan?

• 38:23Speaker 03

  That's a great question because by the time this is out, there'll be another blog, there'll be another blog post. There'll be another blog post, which contains all the information, all the learnings that I delivered to OpenAI's fine tuning team when we finally got the score.

• 38:35Speaker 02

  Oh, it's good. Go for it. It's already out.

• 38:37Speaker 03

  And yeah, I don't have it on my phone, but basically I broke down the log probes. I basically got the average log prob for a token at every token position in the context window. So imagine an X axis from zero to one hundred and twenty eight K, and then the average log prob for each index in there. As we discussed, like the way genie works normally is, you know, at the beginning you do your rag and then you do your planning and then you do your coding and that sort of cycle continues. The certainty of code writing is so much more certain than every other aspect of genie's loop. So whatever's going on under the hood, the model is really comfortable with writing code. There is no doubt and it's like in the, in the token probabilities. One slightly different thing, I think, to how most of these models work is, at least for the most part,

• 39:21Speaker 03

  if you ask G P T four in chat G P T to edit some code for you, it's going to rewrite the entire snippet for you with the changes in place. We trained genie to write diffs and, you know, essentially patches, right? Because it's more token efficient and that is also fundamentally, we don't write patches as humans, but it's like what the result of what we do is a patch, right? When Genie writes code, I don't know how much it's leaning on the pre-training like code writing corpus, because obviously it's just read code files there. It's obviously probably read a lot of patches, but I would wager it's probably read more code files than it has patches. So it's probably leaning on a different part of its brain is my speculation. I have no proof for this. So I think the discipline of writing code is slightly different, but certainly is its most comfortable state when it's writing code.

• 40:05Speaker 03

  So once you get to that point, so long as you're not too deep into the context window. Another thing that I'll bring up in that, in that blog post is performance of Genie over the length of the context window degrades fairly linearly. So actually, I actually broke it down by probability of solving a sweep bench issue, given the number of tokens of the context window. At 60K, it's basically 0.5. So if you go over 60K in context length, you are more likely to fail than you are to succeed, just based on the amount of tokens you have on the context window. And when I presented that to the fine tuning team at OpenAI, that was super interesting to them as well. And that is more of a foundational model attribute than it is an us attribute.

• 40:46Speaker 03

  However, the attention mechanism works in GPT-4, however, you know, they deal with the context window at that point is, you know, influencing how Genie is able to form. Even though obviously all our, all our training data is perfect, right? So even if like stuff is being solved in 110,000 tokens, sort of that area, the training data still shows it being solved there, but it's just in practice, the model is finding it much harder to solve stuff down that end of the context window.

• 41:10Speaker 01

  That's the scale with the context. So for a 200K context size is 100K tokens, like the 0.5 or?

• 41:18Speaker 03

  I don't know. Yeah, but I, I, I hope not. I hope you don't just take the context length and halve it and then say, oh, this is the usable context length. But what's been interesting is knowing that, actually really digging into the data, looking at the log probes, looking at how it performs over, over the entire window, it's influenced the short-term improvements we've made to Genie since we did the, that got that score. So we actually made some small optimizations to try to make sure as best we can without like overdoing it, trying to make sure that we can artificially make sort of stuff sits within that sort of range, because we know that's our sort of balance zone. And if we go outside of that, we're starting to push the limits and we're more likely to fail. So just doing that sort of analysis has been super useful without actually messing with

• 42:01Speaker 03

  anything like more structural and getting more performance out of it.

• 42:05Speaker 01

  What about different languages? So in your technical report, the data mix is 21% JavaScript, 21% Python, 14% TypeScript, 14% TSX.

• 42:16Speaker 03

  Which is JavaScript, JavaScript.

• 42:18Speaker 01

  Yeah.

• 42:18Speaker 03

  Yes, yeah. It's like 49% JavaScript. Sure, but TypeScript is so much superior, but anyway.

• 42:23Speaker 01

  Do you see, how good is it at just like generalizing, you know, if you're running Rust or C++ or whatever else, it's quite different?

• 42:31Speaker 03

  It's pretty good at generalizing. Obviously, I think there's 15 languages in that technical report, I think, that we've covered. The ones that we picked in the highest mix were the ones that selfishly, we internally use the most and also that are, I'd argue, some of the most popular ones. When we have more resource as a company and more time and, you know, once all the craziness that has just happened, it sort of dies down a bit, we are going to, you know, work on that mix. I'd love to see everything ideally be represented in a similar level as it is. If you took GitHub as a data set, if you took like how are the languages broken down in terms of popularity, that would be my ideal data mix to start. It's just that it's not cheap doing all this. Yeah, trying to have an equal amount of Ruby

• 43:16Speaker 03

  and Rust and all these different things is just at our current state is not really what we're looking for.

• 43:22Speaker 01

  There's a lot of good Ruby in my GitHub profile. You can have it all.

• 43:25Speaker 03

  Well, okay, we're just training that.

• 43:27Speaker 01

  For running tests, it sounds easy, but it isn't, especially when you're working in enterprise code bases that are kind of like very hard to spin up.

• 43:35Speaker 03

  Yes.

• 43:35Speaker 01

  How do you set that up? It's like, how do you make a model actually understand how to run a codebase, which is different than writing code for the codebase?

• 43:43Speaker 03

  The model itself is not in charge of like setting up the codebase and running it. So Genie sits on top of GitHub. And if you have CI running GitHub, you have GitHub actions and stuff like that, then Genie essentially makes a call out to that, runs your CI, sees the outputs and then like moves on. Making a model itself set up a repo wasn't scoped in what we wanted Genie to be able to do because for the most part, like at least most enterprises have some sort of CI pipeline running and like a lot of, if you're doing some even like a lot of hobbyist software development has some sort of like basic CI running as well. And that was like the lowest hanging fruit approach that we took. So when, when Genie ships, like the way it will run its own code is it will basically run your CI and it will like take the, I'm not in charge of writing this, the rest of

• 44:27Speaker 03

  the team is, but I think it's the checks API on GitHub allows you to like grab that information and throw it in the context window.

• 44:33Speaker 01

  What's the handoff like with the person? So Ginny, you give it a task. And then how long are you supposed to supervise it for? Or are you just waiting for like the checks to eventually run? And then you see how it goes like, what does it feel like?

• 44:47Speaker 03

  There are a couple of modes that it can run in. Essentially, it can run in like fully headless autonomous mode. So say you assign it a ticket in linear or something. Then it won't ask you for anything. It will just go ahead and try. Or if you're in like the GUI on the website and you're using it, then you can give it a task and it it might choose to ask you a clarifying question. So like, if you ask it something super broad, it might just come back to you and say, hmm, what does that actually mean? Or can you point me in the right direction for this? Because like, our decision internally was it's gonna piss people off way more. If it just goes off and has and makes a completely like ruined attempt at it. Because it just like from day one got the wrong idea. So it it can ask you for a lot of questions.

• 45:27Speaker 03

  And once it's going, much like a regular PR, you can leave review comments, issue comments, all these different things. And it, because you know, he's been trained to be a software engineering colleague, responds in actually a better way than a real colleague works. It's less snarky and less high and mighty. And also the amount of filtering it has to do for LGTM.

• 45:48Speaker 03

  And when you train a model to like be a software engineer, essentially it's like you can just do anything. It's like, yeah, it looks good to me, bro.

• 45:55Speaker 02

  I just wanted to dive in a little bit more on your experience with the fine tuning team. John Ellard was publicly sort of very commentary supportive and you know, was was part of it. Like, what's it like working with them? I also picked up that you initially started to fine tune what was publicly available, the sixteen to thirty two K range. You got access to do more than that. You've also trained on billions of tokens instead of the usual millions range. Just like, take us through that fine tuning journey and any advice that you may have.

• 46:23Speaker 03

  It's been so cool. And this will be public by the time this goes out. Like, OpenAI themselves have said, we are pushing the boundaries of what is possible with fine tuning. Like, we are right on the edge and like, we are working, genuinely working with them in figuring out how stuff works, what works, what doesn't work. Because no one's doing, no one else is doing what we're doing. They have found what we've been working on super interesting, which is why they they've allowed us to do so much like interesting stuff. Working with John, I mean, I had a really good conversation with John yesterday. We had a little brainstorm after the video we shot. And one of the things, you mentioned the billions of tokens. One of the things we've noticed, and it's actually a very interesting problem for them as well. When you're building like a self serve fine tuning API, they have to decide how big your PFT adapter, your LoR adapter is going to be in some way.

• 47:08Speaker 03

  And like figuring that out is actually a really interesting problem because if you make it too big, and because they support data sets that are so small, you can put like 20 examples through or something like that. Like if you had a really sparse, large adapter, you're not going to get any signal in that at all. So they have to dynamically size these things and there is an upper bound. And naturally we use models that are larger than what's publicly available. It's not publicly available yet, but when this goes out, it will be. But we have larger LoR adapters available to us just because the amount of data that we're pumping through it. And at that point, you start seeing really interesting other things like you have to change your learning rate schedule and do all these different things that you don't have to do when you're on the smaller end of things.

• 47:48Speaker 03

  So working with that team is such a privilege because obviously they're like at the top of their field in, you know, in the fine tuning space. So we, as we learn stuff, they're learning stuff. And one of the things that I think really catalyzed this relationship is when we first started working on Genie, like I delivered them a presentation, which will eventually become the blog post that you'll love to read soon. The information I gave them there, I think, is what showed them like, oh, wow, okay. These guys are really like pushing the boundaries of what we can do here. And truthfully, our data set, we view our data set right now as very small. It's like the minimum that we're able to afford, literally afford right now to be able to produce a product like this. And it's only going to get bigger. So yesterday, while I was in their offices, I was basically, so we were planning, we were like, okay, how? This is where we're going in the next six to 12 months.

• 48:33Speaker 03

  Like we're putting our foot on the gas here because this clearly works. Like I've demonstrated, this is a good, you know, the best approach so far. And I want to see where it can go and see what the scaling law is like for the data. And at the moment, like it's hard to figure that out because you don't know when you're running into like saturating a PFT adapter as opposed to actually like, is this the model's limit? Like, where is that? So finding all that stuff out is the work we're actively doing with them. And yeah, it's, it's going to get more and more collaborative over the next few weeks as we, as we explore like larger adapters, pre-training extension, different things like that.

• 49:04Speaker 02

  Awesome. I also wanted to talk briefly about the synthetic data process. One of your core insights was that the vast majority of the time, the code that is polished by a human is is in a working state. And actually, you need to fine tune on non-working code.

• 49:17Speaker 03

  Yes.

• 49:18Speaker 02

  So just, yeah, take us through that inspiration. How many rounds did you do?

• 49:23Speaker 03

  Yeah, I mean, it might be generous to say that the vast majority of code is in a working state. I don't know if I can.

• 49:28Speaker 02

  I was like, that's very nice. A few to say that my code works?

• 49:31Speaker 03

  Certainly, it's not true for me. No, I think that, so, yeah, no, but it was, you're right. It's an interesting problem. And what we saw was, when we didn't do that, obviously, we'll just hope, you have to basically like one-shot the answer. Because after that, it's like, well, I've never seen iteration before. How am I supposed to figure out how this works? So what the, what you're alluding to there is the self-improvement loop that we started working on. And that was in sort of two parts. We synthetically generated runtime errors, where we would intentionally mess with the AST to make stuff not work, or index out of bounds, or refer to a variable that doesn't exist, or errors that the foundational models just make sometimes, that you can't really avoid. You can't expect it to be perfect.

• 50:15Speaker 03

  So we threw some of those in with a probability of happening. And on the self-improvement side, I spoke about this in the blog post, essentially, the idea is that you generate your data in sort of batches. First batch is like perfect, like one example, like here's the problem, here's the answer, go train the model on it. And then for the second batch, you then take the model that you trained before, that can look like one commit into the future, and then you let it have the first attempt at solving the problem. And hopefully, it gets it wrong. And if it gets it wrong, then you have like, okay, now the codebase is in this incorrect state, but I know what the correct state is. So I can do some diphing, essentially, to figure out how do I get the state that it's in now to the state that I want it in?

• 50:57Speaker 03

  And then you can train the model to then produce that diff next, and so on, and so on, and so on. So the model can then learn and also reason as to why it needs to make these changes, to be able to learn how to like learn, like solve problems iteratively and learn from its mistakes and stuff like that.

• 51:11Speaker 01

  And you picked the size of the dataset just based on how much money you could spend generating it? Maybe you think you could just make more and get better results.

• 51:19Speaker 03

  What multiple of my monthly burn do I want to spend doing this? Yeah, basically, it was very much related to, yeah, just like capital. And yes, with any luck, that will be alleviated soon.

• 51:29Speaker 02

  Very soon. Yeah, I like drawing references to other things that are happening in the wild. So because we only get to release this podcast once a week. The Lama three paper also had some really interesting thoughts on synthetic data for code. I don't know if you have reviewed that. But I'll highlight the back translation section because one of your dataset focuses is updating documentation. I think that translation between natural language, English versus code and back and forth. I think it's actually a really ripe source of synthetic data. And Lama three specifically called out that they trained on that.

• 52:02Speaker 03

  Yeah.

• 52:02Speaker 02

  We should have gone more into that in our podcast with them, but we didn't, we didn't know. But there's a lot of interesting work on synthetic data stuff. We do have to wrap up soon, but I'm going to briefly touch on the submission process for Sweetbench. So you have a 30% state of the art Sweetbench result, but it's not on the leaderboard because of submission issues. I don't know if you want to comment on like that stuff versus, you know, we also have like, we also want to talk about Sweetbench verified. Yeah, just anything on the benchmarking side.

• 52:30Speaker 03

  The potted history of this is quite simple, actually. Sweetbench up until I want to say two weeks ago, but it might be less than that or more than that. But I think two weeks ago, I suddenly started mandating what they call trajectories when you submit. So but prior to this, essentially, when you run Sweetbench, you run it through their harness. And out the other end, you get a report dot JSON, which is like, here's how many I resolved. Here's how many I didn't resolve. These are the IDs, the ones I did, these ones, the IDs I didn't. And it gives you only ones that like might have erred or something like that. And what you would submit would be all of your model patches that you outputted and that report. And then you would like PR that into the Sweetbench repo and that would be it. That was the, still the case when we made our submission on whatever day it was. They look at them every Monday.

• 53:15Speaker 03

  We submitted it at some point during the week. I want to say it was four, four days before that. And I sort of like. Sat back and waited. I assumed it would be fine. When it came to Monday, they then said, actually, no, we want model trajectories. And I was like, okay, let me see what this is and so on. I sort of dug into it. And like model trajectories are essentially the context window or like the reasoning process of like, show your working. How did you get here? If you do a math exam, show me your working. Whereas before they were like, just give me the final answer. Now they want to see the working, which I completely understand why they want to see that, like. Sweetbench fundamentally is an academic research project and they want all the stuff to be open source and public. So people can learn from each other and improve and so on and that's very good. I completely agree.

• 53:58Speaker 03

  However, at least for us, and the reason that we're not on the leaderboard is that obviously the model outputs that we generate are sort of a mirror of our training data set, right? Like you train the model to do a certain thing and output a certain way. Whatever you output looks like your training data. For the moment, as a closed source company, like fighting for an edge, we've decided not to publish that information for that exact reason. I don't want someone basically taking my trajes and then taking a model that's suing the MGA and just distilling it immediately. And then having genie for themselves. And you know, as a business owner, that's the decision I've had to make. The patches are still public. So like the, dare I say, traditional Sweetbench submission. You can go to a GitHub repo and see it and run them for yourself and verify that the numbers come out correctly.

• 54:43Speaker 03

  Like that is all, that is the potted reason.

• 54:45Speaker 02

  That's the story.

• 54:45Speaker 03

  That's the story.

• 54:46Speaker 02

  Uh, Sweetbench verified? You have a score?

• 54:48Speaker 03

  I do have a score. I do have a score. Forty three point eight percent. It's one of those things where like, there aren't that many people on the leaderboard yet. So you don't know how good or bad that is.

• 54:56Speaker 02

  And it's a, it's a smaller data set, right?

• 54:59Speaker 03

  Oh, it's, it's great. So on a tangent, Sweetbench, original Sweetbench was two thousand two hundred and ninety four.

• 55:05Speaker 02

  Which is expensive. It's like eight thousand dollars to run.

• 55:08Speaker 03

  Oh, that's, that's cheap.

• 55:10Speaker 02

  They're cheap.

• 55:11Speaker 03

  I don't, I don't, I know, at least for us, I don't, I don't even want to say publicly how much it costs us to run that thing. Expensive, slow, really like crap for iteration because like, you know, you make a change to your model. How does it do on Sweetbench? I guess that's why Sweetbench Lite existed, but Sweetbench Lite was not a, it was, there was easy stuff, right? It wasn't a. Comprehensive measure of the overall thing. So we actually had the idea a month ago to what we were going to call sweep bench small, where we were going to try to map out across sweep bench, like what is the distribution of like problem difficulty and all these different things? And try to come up with like three hundred examples that sort of map that where given a score on sweep bench more, you could then predict your sweep bench large score and sort of go from there.

• 55:53Speaker 03

  Fortunately, OpenAI did that for us and probably much better than we would have done. They use some human labelers and as obviously we're working with with OpenAI quite closely, they talked to us about it and they, you know, were able to let us know what the instance ID were IDs were that were in the the new sweep bench version. And then as soon as I had that, I could just just take the report from the one that I run and just diff them. And I was like, oh, we got two hundred and nineteen out of five hundred, which is forty three point eight percent. Which is, to my knowledge, at least right now, state of the art also, which makes sense. But also GPT four O gets, I believe, thirty three percent, which is like, I double check that, but I The August one, the the new one.

• 56:35Speaker 03

  Yeah, it's in their blog post. I I can't remember which one it was. I don't know what the model version was, but GPT four, I believe, gets thirty three percent, which is obviously like significantly better than what it got on the original, like sweep, but sweep, but Two percent. Yeah, exactly, exactly. Something ridiculous, Lilo. But no, sweep bench verified, like, it's so good. It's like, it's smaller. We know that the problems are solvable. It's not going to cost me a lot of money to run it. It keeps my iteration time, you know, lower. And there are also some things that we're going to start to do internally when we run sweep bench to have more of an idea of how right our model is. So one of the things I was talking to John about yesterday was sweep bench is a pass or fail, right? Like you you either have solved the problem or you haven't.

• 57:21Speaker 03

  That is quite sparse, like it doesn't give you a huge amount of information because your model could have got a lot of it right, like looking through when you do a math paper, you could have got the reason, you know, you're you're working right until like the penultimate step and then you get it wrong. So we're gonna look into ways of measuring, okay, well, your model got it right up to this line and then it diverged. Um, and that's super easy to do because obviously you know the Correct state of all those questions. So I think one of the ways we're going to keep improving genie is by going more in depth and saying, okay, for the ones that failed, was it right at any point? Where did it go wrong? How did it go wrong? And then sort of trying to triage those sorts of issues.

• 57:56Speaker 02

  So future plans, you have mentioned context understanding and open source model. But basically, I think, you know, what the genie is, is basically this like proprietary fine tuned data set and process and software that you can add on to any model. Is that the plan? That's the next year is going to just be doing that?

• 58:10Speaker 03

  We're gonna, we're gonna get really, we're gonna be the best in the world at doing that. And continue being the best in the world at doing that and throwing it as many models as we can. Seeing what the performance is like and seeing what things improve performance in what places. And also making the data set larger is like one of the biggest things that we're going to be working on.

• 58:28Speaker 02

  I think one of the decisions before you as a CEO is how much you have like the house model be like the one true thing. And then how much you spend time working on customer models?

• 58:39Speaker 03

  That's the thing that really gets me so excited genuinely, like we have a version of genie that we named after one of our employees.

• 58:49Speaker 03

  It's called the John.

• 58:52Speaker 03

  We have a version of genie that is fine tuned on our code base. So we basically, it's the base genie. And then we run the same data pipeline that we run on, like all the stuff that we did to generate the main data set on our repo. And then all of a sudden you have like something that is both very good at software engineering, but is also extremely good at your repo. And that is phenomenal to use. Like it's really cool.

• 59:13Speaker 02

  More broadly, outside of Cosign, what are you seeing? What trends are you seeing that you're really excited by who's doing great work that you want to call out?

• 59:21Speaker 03

  The one of the, one of the ones that, I mean, it's not an original choice, but Cursor are absolutely killing it. All the employees at Cosign love using it. And it's a really, really good example of like just getting like UX right, basically. Like the, putting the LLM in the right place and letting it allow you and getting out of the way when you don't want it there. And making it familiar because it's still VS Code and all these things. They've, yeah, they've done an amazing job and I think they just raised a round. So congrats on that to them. So like they're doing amazing work.

• 59:50Speaker 02

  The decision to fork VS Code, I think was controversial. You guys started as a VS Code extension.

• 59:54Speaker 03

  We did, yeah.

• 59:55Speaker 02

  Many, many, many people did that and they did the one thing that no one wanted.

• 59:58Speaker 03

  I commend the bravery, honestly. Like I commend the bravery because like. In hindsight, obviously it's paid off, but at least for me, in the moment, I was one of those people being like, is that gonna, are people gonna do that? Are people gonna download that? And yes, obviously they are like, sure. Doing the hard thing, which is having worked on Genie recently, you know, for the past eight months or whatever, as taxing as it's been on us, like, one of the main things I have learnt from this is like, no matter how small you are, how much resource you have, just like trying to do the hard thing, because I think it has the biggest payoff.

• 1:00:31Speaker 02

  More broadly, just like, uh, lessons that you've learned running your company. Oh, it's been a two, it's been a two year journey.

• 1:00:38Speaker 03

  Two year journey. Um, I mean, it's better than any real job you can ever get. Like, I feel so lucky to be working in this area, like, especially, you know, it was so validating to hear it from the guys at OpenAI as well, telling us like, we're on the cutting edge, on the bound, we're pushing the boundaries of what's possible with what we're doing. Because like, I get to do, I get to be paid to do this, you know, I, I have briefly, as you heard at the beginning, done real jobs and, and normal stuff. And like, just being able to do this on the daily is so interesting and so cool. It's like, I pinch myself a lot genuinely about the fact that I can do this. And also that not only I can do this, but fortunately, being a co-founder of the company, I have a huge amount of say as to where we go next.

• 1:01:19Speaker 03

  And that is a big responsibility, but is also so exciting to me. Cause I'm like, you know, steering the ship is, has been really interesting so far. And I like to think that we've got it right, you know, in the last, in the last sort of eight months or so. And that this is like really the starting point of something massive to come.

• 1:01:35Speaker 02

  Awesome. Calls to action. Uh, I assume you're hiring. I assume you're also looking for customers. What's the ideal customer, ideal employee?

• 1:01:43Speaker 03

  On the customer side, honestly, people who are just willing to try something new, like the Genie UX is, is different to a conventional IDE. Give it a chance, like that what we, we really do believe in this whole idea of like, developers' work is going to be abstracted, you know, levels higher than just the code. We still let you touch the code. We still want you to dive into the code if you need to. But fundamentally, we think that if you're trying to offload the coding to a model, the model should do the coding and you should be in charge of guiding the model. So people who are willing to give something. You a chance size of company and honestly, well, preferably the languages that are the most represented in our in our train day. So like anyway, if you're like doing typescript, JavaScript, Python, Java, that sort of thing. And in terms of size of company, like so long as you're willing to try it.

• 1:02:28Speaker 03

  Um. And there aren't any massive like infra sec things that get in the way like it doesn't really matter like code base size can be arbitrary for us. We can deal with any code base size and essentially any language, but your mileage may vary. But for the most part, like anyone who's willing to give it a try is the ideal customer. And on the employee, honestly, we just want people who um we're going to be hiring both on like what we call like traditional like the traditional tech side. So like building the product essentially, and also hiring really heavily on the AI machine learning um data set side as well. And in both cases, essentially what we just wanted, like really passionate people who are obsessed with something and are really passionate about something.

• 1:03:09Speaker 03

  And are willing to, it sounds so corny, but like join us in what we're trying to do. Like we have a very big ambition and we're biting off a very large problem here. And people who can look at what we've done so far and been like, wow, that's really impressive. I want to do that kind of work. I want to be pushing the boundaries. I want to be dealing with experimental stuff all the time. And but at the same time, be putting it in people's hands and shipping it to people and so on. So if that sounds, you know, amenable to anyone, that's the kind of person we're looking to apply. Excellent. Any last words, any Trump impressions that you did you like the Trump impression? Yeah, everyone loved the Trump impression. Yeah, I mean, it's funny because like I I I have some blooper. I'll show you the bloopers after we finish recording. I'll probably tweet them at some point.

• 1:03:53Speaker 03

  The initial cut of that video had me doing a Trump impression. I sort of sat down into the chair and been like, cosine is the most tremendous air lab in the world. Unbelievable. I walked in here and I said, wow, this is an amazing lab. And like we sent it to some of our friends and they were like, nah, you can't cold open with Trump, man. You just can't like, no one knows who you are. You can end with it. But you can end with it. Now that that has gone out, we can now um we can now post the rest of the bloopers, which are essentially me just like fluffing my lines. Entire time screaming at my co-founder out frustration. So.

• 1:04:25Speaker 02

  Well, it was very well executed. Actually, very few people do the launch video that you did. I'm as a sort of developer relations person, I'm actually excited by that stuff. But well, thank you for coming on very, very short notice. I hope you have a safe flight back and excited to see the full launch. I think this is a super fruitful area and congrats on your launch.

• 1:04:43Speaker 03

  Thank you so much for having me. Cheers.

• 1:04:55Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.