⚡ [AIE CODE Preview] Inside Google Labs: Building The Gemini Coding Agent — Jed Borovik, Jules

Jed Borovik, Product Lead at Google Labs, joins Latent Space to unpack how Google is building the future of AI-powered software development with Jules. From his journey discovering GenAI through Stable Diffusion to leading one of the most ambitious coding agent projects in tech, Borovik shares behind-the-scenes insights into how Google Labs operates at the intersection of DeepMind’s model development and product innovation. We explore Jules’ approach to autonomous coding agents and why they run on their own infrastructure, how Google simplified their agent scaffolding as models improved, and why embeddings-based RAG is giving way to attention-based search. Borovik reveals how developers are using Jules for hours or even days at a time, the challenges of managing context windows that push 2 million tokens, and why coding agents represent both the most important AI application and the clearest path to AGI. This conversation reveals Google’s positioning in the coding agent race, the evolution from internal tools to public products, and what founders, developers, and AI engineers should understand about building for a future where AI becomes the new brush for software engineering. Full Video Episode Timestamps 00:00:00 https://www.youtube.com/watch?v=emWgP_fr04k Introduction and GitHub Universe Recap00:00:57 https://www.youtube.com/watch?v=emWgP_fr04k&t=57s New York Tech Scene and East Coast Hackathons00:02:19 https://www.youtube.com/watch?v=emWgP_fr04k&t=139s From Google Search to AI Coding: Jed's Journey00:04:19 https://www.youtube.com/watch?v=emWgP_fr04k&t=259s Google Labs Mission and DeepMind Collaboration00:06:41 https://www.youtube.com/watch?v=emWgP_fr04k&t=401s Jules: Autonomous Coding Agents Explained00:09:39 https://www.youtube.com/watch?v=emWgP_fr04k&t=579s The Evolution of Agent Scaffolding and Model Quality00:11:30 https://www.youtube.com/watch?v=emWgP_fr04k&t=690s RAG vs Attention: The Shift in Code Understanding00:13:49 https://www.youtube.com/watch?v=emWgP_fr04k&t=829s Jules' Journey from Preview to Production00:15:05 https://www.youtube.com/watch?v=emWgP_fr04k&t=905s AI Engineer Summit: Community Building and Networking00:25:06 https://www.youtube.com/watch?v=emWgP_fr04k&t=1506s Context Management in Long-Running Agents00:29:02 https://www.youtube.com/watch?v=emWgP_fr04k&t=1742s The Future of Software Engineering with AI00:36:26 https://www.youtube.com/watch?v=emWgP_fr04k&t=2186s Beyond Vibe Coding: Spec Development and Verification00:40:20 https://www.youtube.com/watch?v=emWgP_fr04k&t=2420s Multimodal Input and Computer Use for Coding Agents This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:04Speaker 00

  Okay, Jed Borovik, welcome to Lane Space.

• 0:07Speaker 01

  Yeah, thanks for having me.

• 0:08Speaker 00

  So we're sitting here at F. Inks beautiful podcast studios and we're actually meeting at GitHub Universe. How's it been so far?

• 0:15Speaker 01

  It's been great. I mean, yeah, the keynote today was was awesome. It was fun to see Jules up there a little bit. You know, but we have a lot of folks from our team here. Jules is partnering with GitHub for the new agent HQ step, which we're excited about. And also, this is this is incredible podcast space. So yeah, I'm excited to do this here.

• 0:32Speaker 00

  I'm glad for them to loan us this space. You're also an emcee for AI Engineer Code. That's exciting in New York where you you went to college, but you don't live there anymore.

• 0:42Speaker 01

  Yeah, no, I spent a bunch of time in New York. You know, it's funny. Being part of the New York tech scene, I actually think it's great having big, major conferences there. Like, I think so. There's a lot, obviously, that happens on the West Coast. But being someone in tech on the East Coast, it's, yeah, it's just awesome to have have stuff there. So.

• 0:57Speaker 00

  Yeah, you mentioned you fly over to SF a lot and like, what's it? What's the scene like in the East Coast? Like, like, obviously, we are pretty new. We're like, this is our first year coming to New York. What else happens in the New York? Like, what are the highlights for you in the New York tech scene?

• 1:13Speaker 01

  Yeah, I mean, there's so much. There's, you know, obviously, a ton of great companies. I think the thing that's interesting about New York is it's such a big city with so much going on. Right? And so there's like, your tech is a huge part of it, but there's also, you know, so many major issues there, whether it's.

• 1:26Speaker 00

  Fashion.

• 1:26Speaker 01

  Media, fashion, finance, like, it's a. And so it's like, I think that helps push the tech and do do all kinds of stuff. But yeah, the East Coast is a great city, great, you know, the all the schools, there's, you know, all across the East Coast. Ton of great schools and great students doing all kinds of stuff. So, yeah, you know, I went to school there, hackathon scene there was amazing. Really fell in love with tech and and programming there. So.

• 1:49Speaker 00

  Is there a big NYU hackathon like like in Stanford with like Cal hacks and stuff?

• 1:55Speaker 01

  Yeah, so there's.

• 1:56Speaker 00

  A tree hacks?

• 1:57Speaker 01

  Yeah, there's one that was put on by, you know, this was a while ago, but we put on by NYU in Columbia. We do to get hack and why. So there's, there's, there's a bunch of events kind of that we did together. It would bring, you know, people across New York. Cities, students across from your city, and those are super fun. Yes, it'd be the Columbia one, one time, then NYU the next, and it was cycle back and forth. So yeah, a lot of cool stuff was made there.

• 2:18Speaker 00

  Nice. So you've been at Google for a while, nine years. You worked on a bunch of things, including with Malta, which, which is also another guest that I'm interviewing today. How'd you get into Jules? Like, what's the, what's the AI journey?

• 2:30Speaker 01

  Yeah. So, you know, so this is gonna sound really cheesy, but I've told a story a couple of times to folks when they're like, oh, how'd you end up, you know, doing this? But it is actually very true. So I worked on search for a long time and specifically kind of like news and freshness. And then, well, you know, when stable diffusion came out, that to me was the first like Gen AI moment. I know some people talk about like ChatGPT as like the first thing, but for me, stable diffusion, you know, it was a couple of months before ChatGPT came out. It was a huge thing. I was, I was falling in a ton online. And there were two groups of creators having reactions to it. You know, there was one group that was, you know, this is stealing my art. This is stealing everything that's near and dear to me. I hate this. This is ruining my life.

• 3:11Speaker 01

  And there was another group of artists and creators who were like, oh, this is a tool to create better art. And so I was watching.

• 3:16Speaker 00

  It's a new brush.

• 3:17Speaker 01

  Yeah, exactly. And right around then, I was having conversations with a couple of people who would say things like, you know, if I had a kid in college, I wouldn't recommend they study computer science. I was like, wow, why? And this was long before like Jensen Huang and people like had been saying this kind of stuff. I was like, whoa, what's, well, I, and it was like, oh, AI, like software engineers are gonna change, it's gonna be so, you know, who knows if there are gonna be jobs. And I was like, I love being a software engineer. I love programming. Like, and I was like, wait, this is my stable diffusion moment. This is either it's gonna take my, my art, my craft, this is a tool to create better art. And I was like, I definitely know which path I'm taking. So I got, you know, very into, to building coding. So I was still working on search. But I spent a bunch of time, you know, making stuff for my own time and playing with things and, and you ultimately tried to find a role that would, you know.

• 4:03Speaker 01

  The most exciting role I could find to do this stuff and that was to join Google Labs and Jules where we were, you know, right around then, we're starting to build these kind of coding agents at Google. And yeah, the timing worked out well and I joined and yeah, it's been, it's been awesome.

• 4:17Speaker 00

  Can you, since we're talking about Google Labs, I am actually unclear about where Google Labs starts and the rest of, and then DeepMind and the rest of Google, like what is the org chart layer?

• 4:28Speaker 01

  Yeah, yeah, yeah. That's a great question. So Labs' mission is to build kind of new, like, innovative products that the rest of Google isn't well positioned for.

• 4:38Speaker 00

  Yeah, which we've had like a riser from OPALM.

• 4:41Speaker 01

  Exactly, exactly. So OPALM is maybe the most wide.

• 4:44Speaker 00

  The most, yeah. And then it's called Nano Banana. I don't know if it's.

• 4:47Speaker 01

  Yeah, so some of the, so the thing about, that's really exciting about Labs is we work incredibly closely with DeepMind.

• 4:53Speaker 00

  Yeah.

• 4:54Speaker 01

  Right? So all the stuff in terms of the, you know, the, we're building a product, but we work so closely for the model and, you know, one of the nice things about being at Google is you have this opportunity to really build an end-to-end AI product, right? From like pixels on a page, through the infrastructure, through, you know, the model and the training and all of that loop. So Labs is here to build new products and we're really like a product org, but a true AI product org where we work incredibly closely with, with, you know, DeepMind, but also, you know, other parts of Google, you know, as it makes sense.

• 5:22Speaker 00

  Yeah. Just on the history of AI coding, I had heard that actually Google had an internal version of Copilot or something like that, that was never released. Is that, is that true? What can we say about it?

• 5:35Speaker 01

  Yeah. So, you know, I think there are, there are, you know, Google's published papers in this space for a while. And so, yeah, we have, you know, being Google, we built a lot of our own tools and, you know, Cider, which, you know, folks maybe have heard of is our, you know, our internal IDE. And we've had all kinds of, you know, capabilities and tools there for a while. So yes, you know, we certainly have had pretty good tools for a while, but they were for internal use.

• 5:58Speaker 00

  Yeah. Yeah. I think, I think it was interesting because like, I think the, one of the hype moments when Google started getting into the sort of like LM game, like basically when everything rebranded to become Gemini and like starting to start Gemini, where people were like, oh, like, did you know that Google, probably like Google's entire repo is probably around the same size as GitHub?

• 6:22Speaker 00

  And like, you know, there must be some interesting data in there.

• 6:25Speaker 01

  Oh, yeah. I mean, and that's one of the things, you know, in building a lot of these internal systems, you know, the, the, the data is incredible.

• 6:31Speaker 00

  Yeah.

• 6:32Speaker 01

  Especially when it's, you know, not only is the model and the training in-house, but all the data on the usage and whatever. So, you know, we could build really kind of sophisticated, sophisticated things there.

• 6:41Speaker 00

  Yeah. Okay. So let's, let's introduce people to Jules.

• 6:43Speaker 01

  Yeah.

• 6:44Speaker 00

  On the, your, your website says Jules Autonomous Coding Agents. We've seen lots of these.

• 6:49Speaker 01

  Yeah.

• 6:49Speaker 00

  They, they're not octopuses. They're not purple. So you got that, you got that going for you. But, but like, what, what really is like the core thing you're trying to nail in a very crowded coding agents landscape?

• 6:58Speaker 01

  Yeah. So what we think about and what we set out to do, you know, back when I joined, it was like, where is, where coding agents going to go? And as these models get more and more powerful and sophisticated and what is that experience going to be? And let's build for that future. All right. And so when you think of a really powerful agent that can run for a really long time, doing really complicated things, that's when like the products start to take shape for us. So for example, autonomous, you know, means like it has its own computer, right? So for Jules, it's

• 7:25Speaker 00

  The V N.

• 7:26Speaker 01

  Yeah, exactly. So tons of agents that run, you know, locally or in your workspace with you while you're coding. But if you want something that's going to run for hours or let's say days, you know, you might want it to have its own environment where it can, can do its own work. So that's just one of the pieces that's important for kind of this autonomous coding agent. But it's really like, you know, think about this, this future where they're incredibly powerful. You can spin up tons of them, right? They're, they're autonomous, but also, you know, we've, we're thinking about what is, what does it mean for it to be ambient, right? Like it's, it's kind of, when it has its own infrastructure, its own computer, its own ways to interact with it. How does that start to change what it can do? For example, like we have an API. So people are using it for all kinds of things, triggering it from, you know, when something happens and we saw an example where someone

• 8:11Speaker 01

  has, they're triggering Jules for, to do kind of all kinds of updates to their site. And then they have a GitHub action that is going to automatically merge Jules' board of questions. So it's just like, all kinds of stuff is flowing, really kind of changing how people are able to do stuff.

• 8:25Speaker 00

  And is that CLI related just to close that loop?

• 8:27Speaker 01

  Yeah. CLI says, we also have a CLI, which is, you know, we want to meet developers where they are.

• 8:31Speaker 00

  Yeah.

• 8:31Speaker 01

  Part of the, you know, an API is like you can trigger it from anywhere, but also, you know, when you're working locally, like you want to be able to trigger stuff. So we, yep, so we have, we have the, the Jules CLI we launched a couple of weeks ago, which lets you interact with it. By the time this podcast comes out, we'll be integrated with the Gemini CLI.

• 8:50Speaker 00

  That's what I was thinking, like, you have a number of CLIs, I'm not sure.

• 8:53Speaker 01

  Exactly. So Gemini CLI, all kinds of places where we're going to kind of mix and, and be able to harness this power, right? Cause developers work in all kinds of spots. And so making it easy, easy to, to, you know, have this autonomous ambient agent that can really do all kinds of work for you.

• 9:12Speaker 00

  Yeah. What is your journey? Like, when, when you started out, like, did you find any assumptions that were quickly challenged? You know, when working with Jenny I and coding agents in, in general, like, I guess you're maybe not too unfamiliar with it because search uses a lot of like machine learned, like black boxy type things. Including BERT, which, you know, was, was a major update a few years ago. Yeah. So I mean, just, just fill us in, like, what is your AI engineering journey?

• 9:39Speaker 01

  Yeah, totally. So I think one of the things that keeps coming up is like, the, the model makes such a difference. I mean, maybe it sounds obvious, but it's like the, the quality of the model really changes like what you're able to do and how, how you engineer around it. So for example, when we started, this was, you know, with relatively early models of Gemini, we had the agent scaffolding around it was incredibly complex. I think one of the things we've seen is scaffolds get simpler and simpler over time as the models get better. And in some ways, this, the scaffolding is almost a crutch for, for things the, the model struggles with. For example, like, you know, really complicated sub agent systems, you know, we, we've played with that, we've experimented with that.

• 10:22Speaker 00

  Can you give an example of like a kind of sub agent that you had to abandon?

• 10:25Speaker 01

  Yeah, we're just basically like you have, you know, you give you give Jules a coding task to do. And it's going to have different agents for whether it's, you know, making a code edit or handling a sub problem or, you know, doing any kind of action with an integration or, you know, and having like full sub agents for for different parts of like a reviewer agent or even like people, sometimes people do these like different personas where you're like, you know, one of the things that cracked me up is like, you know, you're the product manager agent and then you have the code reviewer agent, and that lead agents, we didn't go, you know, that far. But I think a lot of these things aren't as in favor. I mean, certainly people do, you know, they like, I would say the agent harness isn't sophisticated. It

• 11:08Speaker 01

  certainly is. But, you know, as the models get better, like less is more, especially as it comes to like being able to improve through whether it's machine learning or just, you know, regular maintenance. I think certainly we found that, you know, we're finding that less is more. I think that, you know, that we were talking about a little bit before we started recording, like the like RAG, right? And like, you know,

• 11:32Speaker 00

  Koby's and next thing

• 11:33Speaker 01

  and all that stuff. And, you know, it seems like, you know, not just for Jules, but kind of across across the industry, that like agent based search, right? Like it's, you know, maintaining embeddings is hard, but getting the chunking right is hard. Like in terms of like the the black box aspect you mentioned, a lot of that is is hard to improve upon and

• 11:55Speaker 00

  Yeah, I would even say it's maybe not even hard so much as it will never be good.

• 12:01Speaker 01

  Yeah. Well, tell me why, why do you say that never?

• 12:05Speaker 00

  Because like a chunk that happens to capture the thing you're looking for, you know, will will fail to capture something else. And so if you only retrieve based on like your embeddings of a chunk, like it's it uses very arbitrary boundaries that are drawn like with like some hope of like semantics being captured, but you could just throw attention at it.

• 12:28Speaker 01

  Totally. Yeah.

• 12:30Speaker 00

  And you can scale probably much better using grep, like.

• 12:35Speaker 01

  Totally. So I think that's, you know, that's an example of, of, you know, in these, these harnesses, how like they're simplifying, you know?

• 12:41Speaker 00

  Yeah. Well, I haven't abandoned it completely because one of the things that we were doing, I don't know if you saw the cognition sweep work, was basically using semantic search and chunks and with embeddings as a tool. But on the same level as the other tools like Grep and File Access and GloP and whatever else other variants you have. So I think like that, yeah, I mean, that makes sense. Like, don't abandon it. Just, just don't reify it into like the only way to do things.

• 13:11Speaker 01

  Exactly. And to be clear, like, you know, this is an area of research we're doing tons of work on. And, you know, I actually expect, you know, in the coming months, we'll, we'll be talking about some stuff we're doing here too. But it's, it's, yeah, it's, it's not the, I feel like when we started, it was like RAG. It was like embedding based RAG.

• 13:28Speaker 00

  Yeah.

• 13:29Speaker 01

  It was like the thing everyone did. And it's interesting to see how it's changed.

• 13:32Speaker 00

  People asked me for like, where are the code embedding models? And, you know, I pointed them to like a few like Chinese ones. There are some like Nomic was working on one. And then like we found that we didn't need them.

• 13:43Speaker 01

  Yeah, exactly.

• 13:44Speaker 00

  Exactly. Very bitter lesson. So, so, so, you know, I, I think like that these, these are good things. I think like when Jules came out, it was kind of a preview. I, I mean, like the trusted testers group. So I got to see a little bit. But now it feels like more of a real product.

• 13:57Speaker 01

  Yeah.

• 13:58Speaker 00

  What's that transition like? Is there a process in Google Labs to promote things when you feel like there's some traction?

• 14:04Speaker 01

  Yeah, absolutely. So I think the Google Labs is not, you know, about just experiments, right? So like, you know, notebook, as we talked about, it was like, you know,

• 14:13Speaker 00

  It's not very serious. This is incredibly successful.

• 14:15Speaker 01

  We save money.

• 14:15Speaker 00

  It's really, yeah. And for us, I O was kind of a little bit of a turning point. So in, in May, when we, we announced Jules, you know, it was like great reception following

• 14:28Speaker 01

  I O. And that was, that was a real moment of us to like, turn this into, you know, a very much a real thing. I mean, it is something that we were, you know, we always intended to. It wasn't ever intended. You know, I didn't, you know, talk about my journey. Like, I didn't, it was always the goal to build a real product here. And, but for us, that was kind of a very key moment, very key milestone for us. And so, yeah, now is, you know, it's very much a real thing. You know, as mentioned in talking before, like, you know, Jules and the, you know, being talked about on the, in the GitHub keynote. It's, yeah, it's certainly here to stay. We're, we're, we're excited to kind of keep building and expanding.

• 15:04Speaker 00

  Awesome. Let's talk about just like coding and just in general, you're you're coming to MCD, AIE Code Summit. It's gonna be your first time at AIE and and the MCE. What do you want to know?

• 15:15Speaker 01

  Yeah, yeah, yeah. Well, tell me, why? Why would someone want to?

• 15:18Speaker 00

  Oh, guys.

• 15:19Speaker 01

  Yeah, this is, yeah, let's turn it on.

• 15:22Speaker 00

  Oh, boy, this is embarrassing. So, I mean, you know, fortunately, we're in our third year, fourth year now, and we have a bunch of, you know, prior art. We can just point people to it and say, look at our YouTube. You like that? You'll like this.

• 15:36Speaker 01

  There's some great talks. You know, I haven't been before, but I've watched the talk.

• 15:39Speaker 00

  Yeah, yeah.

• 15:39Speaker 01

  There's a lot of good stuff.

• 15:40Speaker 00

  Yeah, and I'm proud that it features content from all labs. And basically, we are like the, this is a pattern I've seen across my career in terms of like every industry needs, it's like focal gathering points to just like trade tips and stuff. So I've seen that in JavaScript. I've seen that in cloud native. I've seen that in data engineering. And I was like, probably AI engineering will need something like this. And then also, the concurrent thread to this was, I went to a bunch of the academic ML conferences, NeurIPS, ICML, ICL. And a lot of them, like NeurIPS is 40 years old, and it hasn't really changed, and is very focused on academics and PhD students. Whereas, I think, really, you know, the the transition in AI going from research to industry is that if you

• 16:25Speaker 00

  you gradually see a shift, unfortunately, less open source, less papers, and more products, and more startups and and closed models and what have you. But these people still want to share. People still want to hire. They want to promote their work. So they need a place to to do that. You can always do that at your company conferences, obviously, I/O, and like GitHub is GitHub, and Microsoft is Build, and Ignite. But like, there usually is one place where it's like the industry neutral thing, where everyone is on the same playing field, and me the best person with. And like, honestly, some people like that. You know, it's not like, you're not going to be treated as like the VIP, and like, you know, you kind of have to like, earn your spot. But like,

• 17:06Speaker 00

  when you earn your spot, I think like people give that that requisite level of attention better, because you had to.

• 17:13Speaker 01

  Yeah, of course. You know, let's say I've watched, I've watched the videos online. I kind of get a sense of this, but what's happening between that, for someone who hasn't been before? Like, what goes on other than the talks?

• 17:23Speaker 00

  Yeah, a lot of, well, just logistical stuff of like invoicing and like vendor selection and venue selection and like, did you know we have like five different pieces of software to like coordinate speaker logistics and booth logistics and

• 17:40Speaker 01

  BORs and

• 17:41Speaker 00

  AV

• 17:42Speaker 01

  And attendees, so I'm gonna go, I'm gonna sit.

• 17:43Speaker 00

  Oh, yes, sir.

• 17:44Speaker 01

  Well, yeah, what am I gonna, what am I gonna?

• 17:47Speaker 00

  Yeah, yeah, so actually, it's really weird because like, I'm the content guy for AIE, right? I curate the speakers, I invite them. Right, and, but I actually know that the content is like the least important part.

• 17:57Speaker 01

  Yeah.

• 17:57Speaker 00

  Because all of it's filmed and we're gonna edit it and post it for free on YouTube anyway. But the reason you come is because one, you can talk to the speakers, but also you can talk to each other. And so like the, you know, I always say like the hallway track is the most important track.

• 18:09Speaker 01

  Yeah. And how do you get the most out of the hallway track? What's your guide? Beginning hallway track.

• 18:15Speaker 00

  I don't have as collected a thought as I should. One, I think if you have some prior history of like what you're interested in and work on. So basically, like, the best intro to somebody is if they've seen you online before. So they can skip the whole like, who the hell are you part? And just get into like, oh yeah, I saw you wrote that thing, like, let me talk to you in person about this, they're both here. That's way better than like, who are you, what do you do? And, and that's, and that's like a very cold interaction. Ideally, people come warm or they can come with some clarity of like, here's, here's why I'm here. Here's, here's what I'm looking to get out of this. Because if, I think if you show up with like no real intention, or if you're like in and out for, for your thing and nothing else,

• 18:58Speaker 00

  then you don't have the space and the mental energy for the unstructured, serendipitous connections. And the thing about AIE, at least in, at least in our scale, our size right now, especially for the summits, which is the one that you're going to, everyone had to apply to get in.

• 19:14Speaker 01

  Yeah.

• 19:14Speaker 00

  So usually, you know, our first summit, we had like something like a ten to one applicant to invite ratio, invited spots ratio. This one's gonna be, it went up to like ten to sixteen to twenty something, this one's gonna be twenty-three. One out of twenty-three people who Yeah. Yeah. So, so like, uh, yeah, it's, it's, it's a lot. Um, I think like, um, and, and really was trying to filter for people who would be speakers at any other conference. So, but like, they, they are, they are the top of the, the field. They are either founders or honestly enterprise buyers of the, the best companies you can find in New York. Uh, which, you know, and that's another reason for this, the, our New York conference, which is we're bringing kind of the best of San Francisco or, or tech

• 19:59Speaker 00

  to the, the, the finance sector, really. Yeah. Uh, there, there is a little bit of media, but mostly finance. Yeah. And like, yeah, that's, that's great. Like, I mean, I, I think, so what I'm trying to say, I guess, is you're there to meet the other people. So make time to meet them, have a calling card, like, who are you? Like, like a quick, like, what, who are you? What do you do? What, what can you help with? What are you looking for help for? That kind of intro stuff is really good. Going with friends is really good. Obviously, like, we, we actually offer, we, uh, for the Walter, we offer bundle discounts. This one, I don't think we do, uh, but just reach out if you, if you need something. But yeah, I mean, like, uh, I, I think like the idea of getting immersed in the code agent community is really important.

• 20:42Speaker 00

  Uh, we, and I think maybe the, the last part I'll bring up is that we themed it for the first time, right? So you used to be, these are just generalists. Here's the state of AI, the best because we can get at any, any point in time. But now we're, we're really trying to push ourselves to theme everything. So we have the best people in code, the best people in data sets, the best people in RL. I want to do a mech and terp one. That'll be fun. Cool. Uh, that one, that one, I, I'm thinking it will be in London because, um, the, the people I want to target are in London. But yeah, I, I think like, when you do a summit, it should be focused. Everyone there should have an agenda of like, trying to learn what's the, what's the state of the art? Trying to have off the record conversations with their peers, doing the same thing at the other companies. And who knows what could happen? Like, that's the, that's the weirdest thing.

• 21:26Speaker 00

  Like, I organize the thing and I, I don't even know half the things that go on just because my job is to provide the nexus of people to just connect. Last time we were in New York, there were thirteen, maybe fifteen side events organized by people, just like dinners, meetups, whatever, around this, around the summit. And we encourage it. We, we posted, uh, and we just want people to meet up.

• 21:47Speaker 01

  Yeah, I was gonna ask, is there a whole like off-menu set of events happening? Like, how do people?

• 21:51Speaker 00

  Yeah, they organize it. Honestly, if you're not scared of strangers, you should organize your own. Like, a little dinner. We leave all the evenings open.

• 22:01Speaker 01

  Okay.

• 22:02Speaker 00

  So, just like, organize a dinner or a meetup, focus on your thing. Like, we have people doing only voice. So, if you wanna do voice, great. If you wanna do like code review agents, as a small subset of general as coding agents, do that. And I think you'll find it, right? Like, or you can do like AI in finance, AI in bio, whatever the particular sector might be. And I think like, that is honestly the highest signal way to get a bunch of people who really resonate with your thing to meet and have high-bandwidth conversations.

• 22:36Speaker 01

  Yeah, yeah. Are you gonna do the autonomous coding agent dinner?

• 22:40Speaker 00

  Uh, well, no. My job is to float. Yeah, yeah. My job is to handshake, ask how everyone's doing, fight fires. So, I tend to just leave myself open until, you know, the end. But yeah, it'll be a sprint. It's always a mad rush. Because then I have to do my own talk. And...

• 23:02Speaker 00

  I don't know yet. I think so far, so like, the last time I did this summit, I was talking about how this year had to like develop as the year of agents. And like, it's really played out a lot. Obviously, now, you know, the trendy thing is to say it's no, it's not just the year, it's a decade of agents. But like, this year, I think agents really took off and most people got it right. Like, the consensus was correct. You don't have to be too spicy or counter-consensus to say, like, if you worked on an agent, you're probably a lot better off, you probably made a lot of progress this year. And maybe you can tell me how it feels on the Jules point of things. I didn't see myself at the start as you're joining an agent company. And I ended up doing that. And but like, I've gone so agent-pilled to the point where like, people come to me with startup ideas for infra companies.

• 23:47Speaker 00

  They're like, what if we made an agent framework so that other people couldn't build agents? And I'm like, well, don't you just build agents yourself, bro? Like...

• 23:55Speaker 01

  There are a lot of frameworks, yeah.

• 23:57Speaker 00

  Frameworks and infra companies. And all of these guys are just like, they're good developers with no conviction whatsoever in what they wanna build. They don't know what the customer they want, they're just like... We want to build developer tools, so that's where we feel comfortable. But honestly, it's not that hard to actually take a stand and be full stack and verticalize in some particular agent field that you want. Because guess what? The business and the economics are aligned that way. And I'm not saying that you cannot make it as an infra company. There's some fantastic infra companies that are sponsors and that I admire and I would invest in myself. It's just that comparatively, those are a lot harder.

• 24:38Speaker 00

  And agent companies seem like they're shooting fish in a barrel. They seem like they're ramping up in ARR a lot faster. And it seems like their margins are better, so why not?

• 24:47Speaker 01

  Yeah, so I think for us, it's really been, you're the agent. Like, as the models, you know, what was you talking about? What is, let's build Jules for where things are going. And as the models get better, I think it just becomes clear and clear that agents are super powerful. You know, like, we have, you were talking about, like, before, high context and management, all that stuff's important. Like, we have people, we had, this is a very funny story, we store some data for a session, but it only lasts, we only store it for 30 days. And so after 30 days, your session becomes locked. And when the first user first started hitting that, they were upset. We're like, there's no way anyone's going to be using a single session for 30 days. Like, nobody would do a single track of work for 30 days. But you're just like, how powerful that could be.

• 25:29Speaker 00

  How do you compress context when you run into it?

• 25:33Speaker 01

  Yeah, so we have, I mean, I can't talk too much about it, but we do a lot of the standard things and it's also, you know, we're developing a bunch of stuff. It's an active area of research for us.

• 25:42Speaker 00

  Yeah, I think, like, you know, just, I'm not asking you for how exactly Jules does it. It's just a number of approaches, right? And you just have to pick one, because you can't just use up your 2 million token context window. Is it 2 million? It is up to 2 million.

• 25:57Speaker 01

  Especially for coding agents, because like, you're reading files, like, it's so, you're running commands with huge outputs. Like, I think coding agents are a really interesting area, both product-wise and the impact they're having, but also for research. They really push the limits of, you know. What other domains are you running an agent for 30 days? What other domains are you accumulating so much context and so many turns? And so it's, yeah, it's coding agents are, I think, kind of a special spot of like super interesting product, impact, research.

• 26:24Speaker 00

  Yeah, I see the M folks drop the auto compaction for a handoff mechanic, which was pioneered by the agent SDK, which is basically the subagents pattern where like you spin up a subagent and do your thing. You don't need all their context that the subagent is doing. And then you can sort of come back to the main thread.

• 26:43Speaker 01

  Totally. Yep. Yeah, it's a good pattern. And also has its challenges, like how do you make sure enough stuff, information is going back and forth? But that's the part of, you know, the summarization is a pattern. You know, like kind of externalizing some of that context, where it's like writing it to, you know, like a note kind of thing is a common pattern. So, yeah, there's, there's tons of things, tons of things to try and do.

• 27:04Speaker 00

  Yeah, yeah. I mean, and one thing I do want to get more consensus about is what is the best? Because I don't think I've read any papers about which methods compare better.

• 27:15Speaker 01

  Yeah, it's also like as models change, like the answers change a little bit too.

• 27:19Speaker 00

  Yeah, Claude, you probably know Claude externalizes too much.

• 27:23Speaker 01

  Yeah.

• 27:25Speaker 01

  Yeah.

• 27:26Speaker 00

  How much does your work actually like, I feel like I switch back to Jules' mode.

• 27:30Speaker 01

  Yeah, we're flowing here.

• 27:32Speaker 00

  Well, I mean, like, you know, how much does your work inform the model creation, right? Like, at the end of the day, like, you obviously are a very big consumer of Gemini models, but also you are not the only consumer and they have other priorities than you.

• 27:45Speaker 01

  Yeah, totally, totally. I mean, I think we're lucky in kind of how we're positioned. We have very close relationships with DeepMind. So we have, and you know, coding agents are an important area, let's be honest, right? Like for any kind of company building models, like you can see it in all the labs, like coding agents are important, coding capabilities are really important.

• 28:04Speaker 00

  Yeah, my OG image of the AI code, I wrote something obnoxious, like code is the first spark of AGI.

• 28:12Speaker 01

  Yeah.

• 28:13Speaker 00

  Which is like probably true.

• 28:14Speaker 01

  Totally. Yeah, it's important from a kind of AGI perspective. It's important from a dollars perspective, it's important for all of it. So it's, I think we're really. Lucky position.

• 28:25Speaker 00

  Yeah.

• 28:26Speaker 01

  We have, we're able to have a lot of kind of good collaboration.

• 28:29Speaker 00

  Yeah.

• 28:29Speaker 01

  But both ways, you know, like all kinds of capabilities that are being developed.

• 28:33Speaker 00

  Yeah.

• 28:33Speaker 01

  And you know, it's interesting, it's, it's a whole host of things, right? Cause you know, in terms of like AGI and the capability of things, it's also like computer use models and browser use models. And so it's, it's, you know, models that output code, but it's also the whole suite of, you know, things that you want an intelligent agent to be able to do. So it's, you know, multimodal, you know, it's all kinds of stuff that goes into it. So it's, yeah.

• 28:57Speaker 00

  What would you want to find out from your peers at other coding agent companies? Cause you're going to meet all of them, basically.

• 29:03Speaker 01

  Yeah. So I think one thing, and you know, I don't think of this as a zero something. I think this is like, really like, uh, there's this tide that's going to lift all of our boats. And it's, we're inventing a new way to do our art, right? You know, um, and how to create good art as a software engineer. And so what does that look like? And how does that feel? What is that, you know, what is the experience we want to create? I think as, as people working AI, sometimes we don't do a good enough job describing this beautiful future we're creating. I mean, I know like, you know, like the CEOs and heads of these labs have started like, you know, writing their, their think pieces on this. But right, um, you know, for software engineers, like what is this beautiful future we're creating? And like, you know, I think that's like, one, it's, it's inspiring.

• 29:47Speaker 01

  It makes it, you know, maybe less scary for, for people who are, who are thinking about these tools. But also like, you know, if we can't articulate it and think about it, it's less likely we'll get there. Right? So like, what is this, you know, great place we want to create? Like writing software is so hard. Like it's so many companies, it's such a, especially like big companies, it becomes so challenging to manage a code base and create. And, and what can we do to make, you know, being a software engineer, absolutely incredible experience? What are these, you know, how do you want to interact with your model? How do you, how are you doing things locally versus in the cloud? And how does that interop? And, um, so I think like, as, as an industry, we're trying to like, you know, which is changing. Like we're inviting, in some ways inventing, and there's this movement to, you know, change how we do our art. And yeah, the more, you know, the better we can create this experience.

• 30:31Speaker 01

  Like we all, we all win to some degree. So, uh, yeah, I think that'd be one thing where it's like. Yeah.

• 30:38Speaker 00

  Yeah, the local to cloud sync is the most contentious or important, I guess, topic for a lot of people. I wonder if we'll ever get like some kind of interrupt thing, probably not, but a man can dream.

• 30:51Speaker 01

  Tell me more about what was your dream, what's your dream flow here?

• 30:54Speaker 00

  I don't know. Start with J U C L I, end up in Devon. I don't know.

• 30:57Speaker 01

  Oh, you're not between ages, you should name this guy.

• 31:00Speaker 00

  It's probably, it's probably meaningless, but no, but like, I'm not actually serious about it, but like. Traffic to me all centric. Yeah, I think, well, so I think Codex, or is it Cloud Code, Cloud Code Web can do this teleport.

• 31:13Speaker 01

  Yep.

• 31:14Speaker 00

  Where they just basically dump like the entire history and you can pick it up in Cloud Code on your desktop. And probably that's the right move.

• 31:22Speaker 01

  Yeah.

• 31:23Speaker 00

  Maybe there's, there's some more sort of elegant things, but they were first, so why not? And like, and actually, maybe the, maybe the real thing is, maybe it's not the conversation. Maybe you don't need to teleport if the unit of, if the artifact that you pass back and forth is the linear ticket or the GitHub PR. Right? So you don't need the full JSON, you don't need the full chat history. You just need to pick up where other people left off because that's how humans do it.

• 31:50Speaker 01

  Right, right, right.

• 31:51Speaker 00

  Like, I don't, I don't transfer my brain state to you, I just tell you what I did.

• 31:54Speaker 01

  Yeah.

• 31:54Speaker 00

  And then, you know, if I didn't, if I forgot to say something, you find out eventually.

• 31:57Speaker 01

  Right, right. You say like the cloud agent like to dump some kind of summary onto the ticket or whatever kind of it needs to pass on to the next.

• 32:03Speaker 00

  In Slack or linear and whatever.

• 32:06Speaker 01

  Yeah, yeah, that's interesting. I think we have, there, there are some patterns emerging, the like IDE, CLI, Cloud. Right? Like these, these are the pieces.

• 32:15Speaker 00

  VS Code extension.

• 32:16Speaker 01

  Yeah, VS Code, yeah, like where there's.

• 32:18Speaker 00

  Which you guys don't have yet.

• 32:20Speaker 01

  So there's a, like the surface area is like standardizing it feels a little bit. And yeah, how these things interrupt, how you can kind of make this like great experience with all of those. Yeah, I think it's really interesting.

• 32:33Speaker 00

  Yeah. Yeah, I think like, and then the other point, I just want to backtrack a little bit to something else you said, which is like what the the thick pieces that the CEOs and stuff do. I I think there's a lot of question about the impacts that coding has on the software engineer industry in general, the humans. Do we end up, do we stop hiring juniors altogether? Do we, is it actually increasing productivity? Or do you just feel like you're increasing productivity? I don't know if you have any take on that and stuff.

• 33:02Speaker 01

  Yeah, it's only so, I mean, I, something we spent a lot, I spent a lot of time talking and thinking about with folks. And I also, I also spend time talking to people at companies. And, you know, I think sometimes working on these tools, it's interesting to see. It's not as like diffused, this technology isn't as diffused across software engineers as I sometimes expect, right? There's plenty of places that I think are are not really using AI a ton. A lot of companies, a lot of software engineers aren't. That being said, I'm very kind of excited about what this, what the future of software engineers are like. Could you imagine going back to not having these tools?

• 33:37Speaker 00

  No.

• 33:37Speaker 01

  That sounds horrible, right? Like that. And so that's one aspect of it. I also think, you know, I don't really buy this like, you know, that we're not going to hire more software engineers story. I think like for for a few reasons. I mean, this is an example that that often comes up, but is it like kind of the elasticity of the demand for software?

• 33:59Speaker 00

  Yeah, Jevon's paradox.

• 34:00Speaker 01

  Exactly. And, you know, like a lot of the cases sometimes come up as you look at like farming, right? And so, you know, there was a time in America where like the vast, vast majority of Americans were farmers, right? And then technology happens and today it's like less than 1%.

• 34:13Speaker 00

  Yeah.

• 34:14Speaker 01

  And that's one example. But the flip side of that is you have electricity, which like as that gets cheaper and cheaper, people just consume more and more and more electricity. And, you know, with food, there's only so much food we're going to eat, right? There's there's a kind of a, there's an elastic demand for that, whereas just a very elastic demand. It seems like software, you know, software keeps getting better and better. Like the ability, like we're creating more and more software from like, you know, obviously like punch cards due to where we are today is like remarkably different in terms of how you're able to create software. So much more software is being made and software just keeps becoming more and more of our GDP, right? Like it's it's a, so I'm I'm bullish on kind of the the amount of software we'll be able to create, how it'll be created.

• 34:54Speaker 01

  I think there's also something here about, you know, as as an engineer, being able to be more productive. Like encourages more investment in people building software, right? If it's, you know, the job of a software engineer can now, you know, they can do 50% more, 100% more, 10x more. Like justifying investment dollars into projects like dramatically changes, right? And so, um, uh, yeah, I'm, I'm bullish on this idea that, that is actually going to be great for software, just both for, you know, our ability to kind of do our craft or art, but also just what it means for the number of companies and the amount that's made and the quality of it and what we're able to do with it. So, uh, yeah.

• 35:37Speaker 00

  Yeah.

• 35:38Speaker 01

  Rose colored glasses, take.

• 35:39Speaker 00

  Rose colored glasses, indeed. Yeah, I have this take on the different kinds of work, like we're spreading out the different kinds of software work. And there's a lot of commoditized work that we used to spend a lot of time on, and now we can basically entirely delegate to agents. And then that leaves us ideally for more strategic, important, novel, high risk, whatever, uh, work, deep focus work that, uh, you know, is, is, is something I, I posted here on the semi async value of death, uh, where basically you kind of need to, uh, on the, in the extreme end, you can delegate to async agents, which Jules, uh, you know, uh, quote code, whatever. But then over here, you kind of need the sort of deep involvement in understanding the

• 36:21Speaker 00

  code base and like, feel like not vibe coding.

• 36:25Speaker 01

  Go.

• 36:25Speaker 00

  Whatever the opposite of it is. Actually, that's my talk, which is, I've been thinking about this.

• 36:30Speaker 01

  Okay.

• 36:30Speaker 00

  So I tweeted out like, uh, this phrase, cause I, I think, I feel it was in the air that like the term vibe coding was obviously coined by Andre and he's super influential in February and like people have just come to kind of use it as a blank check to just YOLO on prompts and stuff.

• 36:47Speaker 01

  Yeah.

• 36:47Speaker 00

  And then create the worst code imaginable and like leave other people to clean it up.

• 36:51Speaker 01

  Yeah.

• 36:52Speaker 00

  Uh, so I think like people are kind of at their limits with this. Like it was probably maxed out in terms of popularity. They would, and but we don't have yet what's next.

• 37:01Speaker 01

  Right.

• 37:01Speaker 00

  So my talk is really challenging every attendee, every speaker to come up with like, what is the aspirational good version of I coding that we can actually trust?

• 37:12Speaker 01

  Yeah, what is it? Well, the punchline right now, what is it?

• 37:15Speaker 00

  I mean, the current leading candidate is agenticoding, which is what Dharmesh Shah was like, I don't know if you know what Dharmesh is, he's he's pretty good track history when he's naming things. Uh, it's just too many syllables. I don't think it just has the, it doesn't have the joy that that five coding invokes, which I think people want, but then people also want care and craft and like reliability and all that stuff that.

• 37:42Speaker 01

  But if we don't have the term to describe, but maybe we don't have the catch this phrase for it, but what is, what is, what does it look like, even if we don't have the phrase?

• 37:48Speaker 00

  Yeah, it's a great question. Um, I, well, we have some speakers who are going to be pitching spectrum and development that you have to really be thoughtful and effectively write a PRD. I think the, and I think like that is obviously very correct in terms of like, basically it's just a glorified prompt, but a very, very, very good one. And uh, models are tuned to follow your prompts for good and for worse. If you prompt sloppily, you're gonna get slop. So a spec sounds good, I think. Uh, I don't know how often it'll be followed in practice. Because effectively what that transitions us to is a waterfall development approach where you spend three days writing a writing a 50-page document and then you kick off the agent.

• 38:31Speaker 00

  That doesn't seem right. Uh, so like, you know, obviously I, I have some bias here because uh, cognition has from the start believed in interactive planning where like you kick off a thing, you get some feedback, then you're like, you're like, oh, that's not what I meant. Let me correct myself. Cause I don't know what I wanted when I, when I started. So you, you work with the, the machine to discover what you wanted and the machine works with you to either get you what you wanted or show you the errors of your ways. And then you correct it from there.

• 38:59Speaker 01

  Yeah. And one thing we talked about was very aligned is what you were thinking is like, they're kind of like. Two problems as these things get better. One is like, how do you specify what you want? And the other one is how do you verify that what you got is what you're doing?

• 39:10Speaker 00

  Yeah.

• 39:11Speaker 01

  Yeah. And so, um, yeah, whether it's specifying through a spec or this like, you know, interactive plan or whatever it is, but then, yeah. And then on the flip side, with the vibe coding thing is you might specify, but you never come back and verify, right? You're like, it's more hands off the wheel. Like, maybe I'll click around the app a little bit and see how it works. But I'm not really engaged with the code. So how do you, yeah, how are you verifying and making sure that it's, you know?

• 39:33Speaker 00

  To my knowledge, you guys don't emphasize tests that much, right? It's not like you volunteer to write my tests.

• 39:39Speaker 01

  Yeah, I mean, it depends. Like, if we, um, if there are tests in your codebase.

• 39:44Speaker 00

  It's right. It's right out of the picture here. Jules will run your test feed.

• 39:47Speaker 01

  Exactly, exactly. So.

• 39:49Speaker 00

  But it's not like, it's not like, you know, after everything, everything must have a matching test to the prompts that was mentioned, you know?

• 39:55Speaker 01

  No.

• 39:56Speaker 00

  That would be the extreme of what we mentioned.

• 39:58Speaker 01

  People always want that. I mean, maybe it'd be helpful to do that to kind of show that it was right. But let's say I don't write tests in my codebase. Like, I want to merge that pull request that is introducing tests just for this one thing. Like, you know, I think in some ways the, the engineer should be able to control what, what kind of outputs they want.

• 40:14Speaker 00

  Yeah.

• 40:15Speaker 01

  If it, if it helps and they want it, you know, absolutely.

• 40:18Speaker 00

  And then do you think there's other innovations on specifying apart from just chat?

• 40:22Speaker 01

  Oh, totally, totally. Um, I mean.

• 40:24Speaker 00

  Agents M D.

• 40:26Speaker 01

  Yeah, I mean, uh, the spectrum of development, I think, is in this, this category. I think, um, one of the reasons is like multimodal, right? Like, you know, if I'm going to show you a bug on our website, like, do I want to come and like type it with words to describe it? Or am I going to point?

• 40:41Speaker 00

  Yeah.

• 40:42Speaker 01

  The picture.

• 40:43Speaker 00

  Yeah.

• 40:43Speaker 01

  Um, and so, you know, with Jules, you can upload images now, um, but, you know, kind of more, you know, we have certain ways we communicate as humans that are easier in certain situations.

• 40:53Speaker 00

  Yeah.

• 40:54Speaker 01

  But just bringing that to, to our engagement with, with the agents.

• 40:56Speaker 00

  So of all people, I expect you guys to be best at this because Gemini has video understanding. I want to submit a video because some things like do cannot be screenshots.

• 41:06Speaker 01

  Yep.

• 41:07Speaker 00

  Uh, it's more about the behavior of, of things appearing and disappear. Uh, yeah, I mean, I would love that if you, if you guys did it. Because no one has it yet.

• 41:15Speaker 01

  I know, I would love it too. Well, I'll tag you over there and also.

• 41:19Speaker 00

  On my side, the vision, the version of that that we're exploring is computer use.

• 41:23Speaker 01

  Yeah.

• 41:24Speaker 00

  Computer use was kind of introduced by Anthropic and then OpenAI did their toe in with Operator and now Agent Mode in Atlas. I don't know if you guys have done anything super splashy on computer use, but anyway, it's coming back. I can feel it.

• 41:38Speaker 01

  Yeah, I think, yeah, definitely. And it ties into, you know, it ties into coding agents, it ties into just, you know, using AI systems in general.

• 41:47Speaker 00

  But basically, your VM now needs to render a UI or a browser and then you need to let the agent click around in it.

• 41:53Speaker 01

  Absolutely.

• 41:53Speaker 00

  And you need to have precision and speed and cost and like, you know, affordable cost.

• 41:59Speaker 01

  Yep.

• 41:59Speaker 00

  It's a lot.

• 42:01Speaker 01

  Yeah, I know. What kind of, punch is so fun. There's just so much to build. There's so much. You know, I think also as a software engineer working in this space, like I think one of the reasons we, you know, you see so many companies in this space is partly like, it's just so fun. There's so many things to build. There's so many tools that seem like, you know, fun sci fi. Like there's, it brings up a demo of what I've worked on. It's clicking around and I can see a video of it or I can even take over and use it like, so. Yeah.

• 42:25Speaker 00

  It's, yeah. Awesome. Okay. So just moving towards wrapping up, if people run into you at AIE, they've, you know, they heard your pitch on Jules.

• 42:34Speaker 01

  Yeah.

• 42:35Speaker 00

  What else should they also talk to you about? Like, you know, what, what, what do you, what can you help with versus what are you looking for?

• 42:42Speaker 01

  Anyone should feel free to come and talk to me at any point. I, you're obviously very interested in anyone who's doing stuff with coding agents or someone who's using coding agents in interesting ways. I'm always curious about, you know, workflows people have with their data agents, whereas, you know, whether it's, you know, hey, I'm using this tool in this way and I've, you know, configured this crazy thing. Like, I always love hearing how people are using it. I also love hearing people who are having bad times with it, where it's like, actually, I don't, you know, maybe they're not coming to this conference, but, you know, I. Jeff Charlie's tools, and I don't like them, and I don't use them, and here's why.

• 43:12Speaker 00

  Yeah.

• 43:12Speaker 01

  So, you know, I'm totally open for any side of the, all the way from your full AI pill and coding, coding AI lovers to people who hate it. As far as what I'm looking for, you know, I think, you know, really just going to kind of connect and meet people. I think, you know, we are always hiring. So like, you know, I'm, anyone who's, you know, interested in working on this stuff, I'm always happy to talk. But yeah, really just kind of, you know, meeting people, spending time, geeking out on this stuff.

• 43:38Speaker 00

  Yeah, there'll be lots of geeking out, too. All right, thanks for your time, looking forward.

• 43:41Speaker 01

  Yeah, same.