[One Million AI Engineer Special] How Stanford Teaches AI-Native Engineering — Mihail Eric, Monaco

What do you assign when a coding agent can finish last year’s homework in one prompt? In the second run of Stanford’s CS146S, Mihail Eric brought open-source maintainers into the course. Students have to learn an unfamiliar repo, propose a change, and ...

• 0:00Speaker 00

  Okay, this is a pretty special episode because we're celebrating 1 million subscribers on the AI Engineer YouTube channel. So go over there and subscribe if you haven't heard about it. Or the guest is Mihail Eric, who is not only teaching the modern software developer at Stanford, which is going viral right now for basically covering all the AI-native development best practices, but also he is head of AI at Monaco. So he's not only a teacher, but he's a practitioner. And those of you coming to AI Code Summit next week, it's going to be the celebration of all the extreme progress that you've seen in coding in the last year. I'm excited about all the themes of fighting slob, of system one models, and of increasing your ambition for major rewrites of basically everything that you've

• 0:42Speaker 00

  seen in code. There's never been a more exciting time to be an AI engineer. And with 1 million AI engineers, I think it's just getting started. So let's get into the interview. Okay, we're in the studio with Mihail Eric, adjunct professor at Stanford and head of AI at Monaco. Welcome.

• 0:56Speaker 01

  Thank you. I'm glad to be here.

• 0:58Speaker 00

  You caught our attention because of your course that you are refreshing for the third time.

• 1:03Speaker 01

  Second time.

• 1:04Speaker 00

  Second time?

• 1:04Speaker 01

  Yep.

• 1:05Speaker 00

  The CS146, or 146S, the modern software developer at Stanford.

• 1:08Speaker 01

  This particular generation of junior developers, of junior engineers, the people that are now entering the workforce, will I think be the first kind of generation of that new shift.

• 1:18Speaker 00

  And I looked at the syllabus and I was like, this is everything we cover. So every now and then, I like to invite people like yourself, who like, you cover a broad range of topics. And then we can sort of go topic by topic and give people the state of the art on each of these things, which I assume you're doing with all your guests anyway.

• 1:34Speaker 01

  For sure.

• 1:35Speaker 02

  Tell us the history of the course. What made you want to spring out CS146?

• 1:40Speaker 01

  Yes, this is the second time it's being taught. And I taught it for the first time last fall, so fall 2025. This was pre Opus 4.5, 4.6 days. But even before fall 2025, the course had been about like a year in the making. The genesis of the class was like November of 2024. And I had sort of had this idea that we're seeing all this stuff happening in coding agents. And yes, they were an earlier version of the coding agents, but already it felt like there was something drastically changing in terms of how developers are doing their work, how quickly we were getting work done, all that fun stuff. And so at the time, it was all just scattered blog posts, scattered like things you could find online if you were trying to really get into the methodology of how to do these things. And so I had this idea like, hey, well, is there a way to kind of structure a curriculum around this? It's a little bit more formal. I had the idea, and you know, I, my home matter is Stanford,

• 2:25Speaker 01

  and I always had this dream to go back and teach something and kind of pay it forward. I've always been very deep into education. I talked to Chip, who also taught a class, one of my good friends.

• 2:35Speaker 00

  Chip.

• 2:35Speaker 01

  Yeah, Chip, she exactly. I spoke to her, who she had taught her, I guess, designing machine learning systems. And so like, asked her what that process was like, and sort of, you know, took a lot of inspiration from how she designed the curriculum and thought about, you know, getting some through the school, put it together, spent about nine to 10 months. This was now fall of 2024 through to like, you know, mid to late of 2025, got it through the committee, made a case for the class to exist, you know, had to put together proposal and everything. And that took me to fall 2025, when I had it for the first time, which was very well received. You know, students were really interested in it. There was even, I guess, like a Business Insider article that was written about the class last year. And then, yeah, this year, a lot of things changed. I would say really

• 3:17Speaker 01

  coding agents, autonomous coding agents really have picked up in a way that we had not seen before. So it felt like it was time to do this again, refresh things. And then here we are.

• 3:30Speaker 00

  Meanwhile, you're also head of AI at Monaco, which is also a very buzzy startup.

• 3:34Speaker 01

  Yes, yes. We are, we are, we are in the airwaves right now and having a fun time doing it, you know?

• 3:39Speaker 00

  Yeah. So like, I do like that you're also, I guess, a practitioner. You guys are obviously implementing all these stuff in Monaco.

• 3:44Speaker 01

  Yeah.

• 3:45Speaker 00

  Any, I guess, principles or observations about just what you have changed from last year to this year at a high level?

• 3:54Speaker 01

  Yeah, yeah. I mean, a ton. I think that when I reflect on what last year's version of the class was, I mean, coding agents had been around, but we were still in kind of that phase where companies were emerging that were tackling every single part of like the ISDLC. There was like a coding agent for CoderView. There was a coding agent for, you know, spec design. There was a coding agent for SRE. And, you know, some of those, of course, some of those companies still exist. But it felt like last year, it was a lot more about a broad tour of here's everywhere you can use coding agents. And in a way, a lot of the curriculum last year was geared toward exposure and this kind of survey, this kind of like odyssey of sorts through, hey, everything that you knew about the ISDLC is now changing because of AI. This year, I think there's some principles that are

• 4:37Speaker 01

  solidifying, right? The idea of like, very careful context engineering, the idea of like really thinking through sort of like design and architecture as a way of handling really complex tasks. And so the nature of my classes, it's not intended for the beginner programmer. It is not, it is not a replacement for like traditional computer science in the sense of like algorithms. And in fact, a lot of those classes are prereqs to what I'm doing here. But I do know that if you have a good foundation, going now toward applying that and sort of using those principles in production environments in like real world, that's something that I think is now at a place where, you know, at Monaco, we deal with massive codebases. You know, I help a lot of companies also with thinking through their massive codebase designs and things like that. And so really, it's more of a practitioner's course for

• 5:20Speaker 01

  once the students have graduated or now entering the workforce, how to be the most productive members of the workforce, you know?

• 5:26Speaker 02

  You said you're passionate about education. I feel like students in that phase of being in college are in a pretty interesting spot. Any advice and any take? Should people study computer science right now? Is Cloud Code and Codex all we need? Like, I want to, you know, hear your thoughts.

• 5:44Speaker 01

  Yeah. I, you know, there's a lot of anxiety from students, as you can probably imagine. And, and I think a lot of what I'm trying to communicate this time around is that the anxiety, a lot of it still is here. And there was anxiety last year as well. But I want to make the case that a well-trained AI-native engineer, and I know that's like a buzzy word, but let's, we'll, we'll, we'll dive into maybe what assets that are most interesting. Like, well-trained and native engineer can be some of the most, you know, the most useful assets like any company around. And so, and, you know, this is the first week of class, actually. And like week one, you know, first lecture, like, hey, I really believe that every company needs you. But the floor is much higher, right? So you're entering as

• 6:28Speaker 01

  like your first job out of school, but it's not sufficient to just be like, you know, an L3 and like a Google term, if you're going to use like that, that kind of leveling system there. It's like, you have to generally be better than, you know, almost at the senior level going into this. So like, what can we, how do we get there more quickly? Right? And so I'm trying to say that there is still a core bit of like foundational software engineering that is important to have, some of these core values of sort of how software systems are built and the trade-offs of those systems. But then beyond that, we go and, you know, that's where I think you as the humans should still be valuable rather than, and the rest of the implementation details, you can kind of defer more often.

• 7:03Speaker 00

  Yeah, this is something I've said before, which is almost like the controversy. It's like the kind of thing you say to be spicy, but I almost, I do genuinely kind of believe it, which is, most students are cooked. Some of them are going to be super valuable. I don't think it's a very popular thing. Stanford's, some of its kids are going to be fine, right? They're from Stanford. But like, what happens to like the rest of America?

• 7:27Speaker 01

  Right, right.

• 7:28Speaker 00

  That is an open question.

• 7:29Speaker 02

  It's a weird situation, because I feel like a lot of education is slow to move. Like, I'm even curious on how, how do you determine where do you put the end date to curriculum in a course on AI?

• 7:39Speaker 00

  Oh, my God.

• 7:40Speaker 02

  You're teaching a Stanford modern software. Like, that's changing everything.

• 7:43Speaker 00

  You cheat. You just get the people from the coding agents to come and speak, right?

• 7:48Speaker 01

  Oh, yeah. Like, yeah, have them be the, yeah. That's a funny.

• 7:51Speaker 02

  You all guest lecturers.

• 7:52Speaker 01

  It's a funny, well, yeah. I mean, that was, okay. So on the guest lecturer's point, I think that was, I mean, that was also last year. That was also the case. So that was by design, because I, look, I came in, right? And I'm saying, I'm a practitioner, as you pointed out, so it's like, I'm a practitioner. I do this thing on my, you know, my day-to-day, I design these systems, you know, we build these things in Monaco, and have seen that many times before. And so, that being said, I don't want to claim that I have all the answers today. And that would be, that would be like, very, you know, a lot of hubris to make that claim. So the structure of the class was always, here's one practitioner's sort of aggregated set of insights from working in the field, from like talking to a lot of people in the field. But then there's people that are like literally on the frontier, you

• 8:37Speaker 01

  know, seeing things even beyond that, and invite them to sort of put their, their proposals forward, their view of the world, the world forward. And that's why the guest lectures are there.

• 8:45Speaker 02

  I think it's really impactful for students to see big names, people actually doing it, like hear from them directly, be able to like ask them, of course, you know, they want to intern, they want to get jobs. It's, it's good to have guest lecturers.

• 8:58Speaker 00

  No, I think what he's saying, so, okay, I'll sort of rephrase. And, no, so, okay, there, you know, we've also interviewed Anj, who teaches the Coachella course, right? Right? Which is like, here's all the names, like, it's only the names, right? Whereas like, he's actually also producing, you know, here's Cloud Code and 200 lines of code, right? And which is, no one's telling you to do that. No one's, no one's going to come and speak to you, do that. But like, yeah, that, when you want to teach fundamentals in a framework that you can use and working understanding.

• 9:24Speaker 01

  That's right.

• 9:24Speaker 00

  This is the kind of thing you do.

• 9:25Speaker 01

  That's right. That's right. Yeah, yeah, exactly. And I, but I do think there's a balance for this kind of material to, to your point, is that there is a, you can't, it's not 100% solidified. I mean, by, by design, by design, it's a feature and maybe a bug of some, some kind, but. A big portion of the curriculum did change from last year, right? And so there's like a dynamism to the whole development of the curriculum. But I'm trying to at least put forward that there are some things that I believe are kind of sticking around. And yes, prompting coding agents, they've gotten smarter. What does that mean in terms of the new ways of dealing with them? But then still Boris is also going to notice something that I don't know. Right? So let's have, let's have him also come to the table and share his learnings.

• 10:01Speaker 02

  One of the fun things I noticed this time around is a good bit of the portion is on open source contributions.

• 10:08Speaker 01

  Yeah.

• 10:10Speaker 02

  What's the thinking behind that? You have people.

• 10:12Speaker 01

  Yeah, yeah. This is, um, this is one of the things that I'm honestly kind of most proud of and most excited about. Uh, I mean, this about to be week two of the class. Yeah. So this is, this is the very gracious open source partners that I spent the greater part of this summer just kind of talking to and getting them kind of excited about and interested in sort of this offering. Basically, the idea was, so last year when I taught the class, we had like nine assignments. And then we had like a capstone open ended project, right? The nine assignments were basically me sort of like coming up with these pedagogical exercises. Like, oh, here's a repo, like use an AI code review tool to, to, you know, post a code review and then see what, why is it good, why is it not good? So relatively pedagogical, honestly, pretty easy. I think for a lot of folks, they could have just like one shot it. And I thought, okay, this is not that helpful.

• 10:57Speaker 01

  At this point, agents are around. I mean, they, they have every student that I surveyed, you know, on Tuesday, not a single person really types by, by hand anymore. They're like, everyone's using coding agents. There's no, there's, and this is, you know, like all sophomores, juniors at Stanford.

• 11:10Speaker 00

  Do you, do you survey them with the market share?

• 11:12Speaker 01

  Was that like what, what tools they use? Okay, I haven't yet, but that's.

• 11:16Speaker 00

  You should, you should do it.

• 11:16Speaker 01

  Yeah, yeah. I will do that.

• 11:19Speaker 02

  It can be spicy. Stanford prefers.

• 11:21Speaker 01

  No, no. Well, I think through like the university, I think they have some, like most of the providers have like an educational program, right?

• 11:27Speaker 02

  So, yeah, they're, they're very generous on plans for students.

• 11:32Speaker 01

  Right, right. So I think by default, even before the class, there's like, you know, Claude license. I imagine cursor license, there is like probably a cognition license as well. I don't know, you might know that better than I do. But what I was able to also just accrue on top of that was, hey, like if I don't want people to be bottlenecked by like the $20 a month subscription. Like I want you guys to go, if you want to like token max something just to see what's possible, then, then you should. And so, and I'll come back to the, the partners, but the, the additional thing that I was able to accrue through, instead of talking to folks, was like additional credits for students. So like factories also are getting involved, the cognition folks are getting involved, OpenAI as well. But anyway, so, so for the, the open source partners, I sort of like was like, okay. One of the things I'm most concerned about when it comes to educate the next kind of generation of, of students

• 12:13Speaker 01

  is that if everyone is saying, kind of to Spex's point, like they're cooked, like, you know, what's going to happen to them? There, there, there's almost like this sense of like them being left behind. And I think that mentorship is like a really important kind of piece of just upleveling everyone and making sure that there isn't this like massive gap that folds, that, that occurs between, you know, the senior people that have already been in the industry before coding agents and know what's up. And they adopt coding agents and they become amazing. And then the juniors that are just like, okay, now what do we do? So in a sense, one of the, the kind of goals was, hey, coding agents are really good. I want to put you guys in situations where you're dealing with complex code bases, large code bases at scale. We're going to work with someone like what I think are among the best, like AI native repos, you know,

• 12:53Speaker 01

  that all do AI focused things or either coding agents themselves or building tools in the service of agents, you know, in the modern world. I want you guys to get involved with these projects. I want part of your assignments to be actually contributing to these projects. In the course of being in all those projects, you will actually be talking to the maintainers. They will be reviewing your code. They will be helping you build mental models of like good software. In that way, there's effectively this like passing on of like the guard of true good software design in a way that, you know, I don't think most students get the opportunity to do. And then there's the benefit, of course, that like, I'm sure some of them will get, you know, poached and picked up as part of the doing well in those assignments.

• 13:28Speaker 00

  Yeah, one of the interesting things, you know, for, for what it's worth, I'm going to get a bunch of hate if I don't say this. I think you say cooked and versus cracked for dramatic effect. But, you know, like it's just nice to state extremes.

• 13:42Speaker 01

  Sure, sure.

• 13:42Speaker 00

  Yeah. So, so maybe we should go into the sort of detail of the curriculum? Or is there anything else you want to explore at the top level?

• 13:49Speaker 01

  Oh, I mean, like that, that's very helpful. Yeah, all that's there.

• 13:52Speaker 02

  I have high level questions about students using tools, best practices. Are you pro it? Like individual tutors? Um, I'm assuming this will also be picked up by students. So, you know, before we dive straight into the course, just anything you have guidance on there for.

• 14:11Speaker 01

  Like in terms of what?

• 14:11Speaker 02

  Yeah, you said you're passionate about education, right? Like, there's also a stigma of, one side is how, how do teachers, professors use it? And then how could students best use these tools? That's not just throw your assignments in, it can kind of one shot it. How do you still learn, like, if you have anything there?

• 14:28Speaker 01

  Yeah, yeah. And I think that there's, I still have some assignments thrown in there, which I think are not intended to be the hardest things in the world, but are meant to be more about like concept, concepts, like more conceptual things. Like, you know, what does a skill mean? It's sort of like the, maybe more of the, the tactical of some of these tools, skills, like, and, you know, agent ready code bases, like agents, MDs, things like that. So there's a certain amount of just, hey, if you haven't heard these terms, kind of get up to speed on them. I do think most of the learning though, in terms of the kind of putting the tools to the test, will happen both in the OSS and then also in the, like the final project where it's like very open ended. And they'll have to just kind of explore something interesting.

• 15:03Speaker 00

  Cool.

• 15:04Speaker 01

  Yeah.

• 15:05Speaker 00

  So, so maybe, yeah, just on the top level, let's go through one, two, three, four, uh, two, 10.

• 15:09Speaker 02

  Yeah.

• 15:10Speaker 00

  Uh, and then we'll dive in deeper. So like breath first, basically.

• 15:13Speaker 01

  Sure, sure. Yeah.

• 15:15Speaker 00

  So we start with internals of coding agents. You teach people how to write their own Claude clone. It's very helpful that it's now open source.

• 15:23Speaker 00

  And then?

• 15:23Speaker 01

  Yeah, yeah. So, uh, week one, it's all internals of harnesses, you know, kind of build a coding agent from scratch. Uh, week two is more, okay, now assume that there's, how do we start thinking about like context engineering at scale? Uh, so beyond just vanilla prompting. Uh, and then I think coupled with that is just, um, in, in doing that, thinking about like tool design, MCP. You know, people, I think for a while, MCP was getting a lot of hate. I'm still pro MCP. I see it all the time. And so I think that there's, at this point, it's a standard that's existed and I think it will continue to exist. So, you know, talking about that, effective tool design, that's week two. Uh, week three, I would say is more about just like, uh, skills and like, what does it mean to like extend beyond simple, single purpose tools to more like procedures? And that's where we'll talk about skills.

• 16:06Speaker 01

  And, you know, that kind of starts leading into like the automation of agents. Week four is, is a lot of just more of like the foundational pieces of how can you customize an agent? Uh, this is things like agent config files, Allah, agents, MB, Claude, MD, hooks, um, you know, kind of sub agents. Like there's a lot of mechanisms that exist in modern harnesses, uh, which you can use for different purposes. So I want to expose students to that. Agent ready code bases, this is probably one of my favorite topics. It's like, and this is especially relevant for folks that are going into like industry and thinking about code at scale is, you know, essentially what does it mean to have a repo that's ready to be ingested, consumed effectively by an agent? And so thinking a lot about, you know, the trade offs of designing of kind of service level boundaries,

• 16:49Speaker 01

  but also docs and, and verifiability and things of that nature. So that's week five. Uh, yeah. Week six, sort of agenda code review. Um, and here I think what is interesting is thinking also kind of extending from week five, week five around code base, like agent ready code bases. Going to week six is more about like, okay, in order to, like code review is now becoming a bottleneck, which I don't know that students, unless they've, you know, have done a lot of, some of them are master students. They've been in the industry, but a lot of them have not seen this. But for most people that are using code agents today, we're all experiencing something of a new bottleneck in code review. So how do we unblock that or sort of unbottle the bottleneck a little bit? What mechanisms are there there? And so verifiability and, you know, things of that nature and how we can extend that into the code review process.

• 17:33Speaker 01

  Week seven, uh, security. I mean, obviously I think there was a point, so this was in last year's class as well. And this, this I wanted to reinforce. I mean, with any autonomous system taking a lot of actions, we're seeing a lot of, this year especially, a lot of things getting hacked. Um, you know, and not least of which are just some of the, the core pieces of software from, from Vercel to, you know, uh, gateways, LM gateways entirely. So I want to spend some time just like talking about what new attack vectors exist, exist in security and, and, you know, how we can mitigate some of them.

• 18:05Speaker 01

  Uh, week eight is background agents. So this is, uh, what I want to start touching on here is, you know, how do you like, basically run agents in many different like product service areas, not just in what was traditionally an IDE, but also having things that are like based on triggers in your code base and in your repo at, you know, during C I C D, uh, things that get triggered even when, you know, based on events, things like SRE style bots. How do you even build a system like, uh, I think this past year was when a lot of companies started releasing their posts like Ramp Inspect. There was Spotify has their own version of this, you know, I think Stripe minions. So all these folks were doing really cool write ups of like their internal infra they were building to do this. I want to touch on that, uh, get people exposed to what that is.

• 18:50Speaker 01

  Yeah, this is, um, yeah, this one is more on like what happens when you're in a team in the modern world and sort of like the core pieces of team level infrastructure. So everything from LM Gateways to, you know, MCP portals, which is like MCP at scale. And then just also how to think about the evolving roles, I would say. So this is maybe a little bit like a soft thing. It's like, what is now, you know, and the vanilla, maybe like not super nuanced thing to say is like, everyone's a builder. But I want to like go a little bit deeper beyond that and like, what thing is actually getting subsumed? What roles are actually either evolving or emerging that didn't once exist? What should students think about in terms of defensible skill set that's maybe wasn't true before? And so a lot more

• 19:32Speaker 01

  like team dynamic type considerations. And then, yeah, this is, uh, it all kind of builds up toward the factory, which I think is top of mind for now, for a lot of folks today. Once we're building factories, where do humans still exist and all that? What is the, you know, where are the new choke points that are intentionally there around, around human involvement? Things like intent, things like setting up a verifiable systems. And what happens in the limit when maybe 89% of code is fully driven by just agents? Where we kind of filled into all that. So, yeah.

• 20:04Speaker 00

  Nice, solid 10 weeks. I was gonna say, like, you know, some of these are more established patterns than others. And I think people need to know that not all of this is settled science.

• 20:18Speaker 01

  Sure.

• 20:18Speaker 00

  You know, but it is everything, I'll endorse everything as like things that people talk about and you should know. You just, we know the open problems, we may not know the answers.

• 20:27Speaker 01

  That's absolutely right. Yeah. Yeah. I mean, even, I mean, you probably have your own version of this. I'd be curious to hear what you think is still not as defensible, but or not defensible, but not as set in stone. But, you know, even like the software factory, are we talking about like dark factories? Are we talking about these sort of half lights on factories? Like, I think even there's a spectrum there. Yeah. And there, you know, there's the people that are on the very frontier, like the warps and the factories that are, that are like building factories and selling them as services. And, you know, even they only claim like 30 some percent of even that of like fully, you know, agentially driven contributions. So

• 20:58Speaker 00

  Yeah.

• 20:59Speaker 01

  What does that mean in the limit? It is hard to say.

• 21:01Speaker 00

  And there's people who went very, very far dark factory and then came out saying, I was wrong, guys.

• 21:06Speaker 01

  Yeah.

• 21:06Speaker 00

  Right? So like, we covered that on AIE. Just be aware that, you know, not everyone, they probably believe the things they're saying at the time.

• 21:15Speaker 01

  Yeah.

• 21:15Speaker 00

  But they have the right to change their mind and they do. And even Boris recently tweeted that, yeah, I am often wrong.

• 21:21Speaker 01

  Yeah.

• 21:22Speaker 00

  Like, and like, everyone is like, because he created Claude Code, it's like treating his word as gospel.

• 21:26Speaker 01

  Totally.

• 21:27Speaker 00

  And I'm like, no, well, sometimes he disagrees with himself.

• 21:29Speaker 01

  Yeah. And that's okay. I mean, that's like, this isn't politics, right? This is where we're still scientists, we're engineers, we're subject to, I don't claim to be dogmatic, but you know, I just want people to experiment. I think that's like a principle as well. You asked about principles earlier, just like experimentation. I mean, like, you just have to be ready to, and oftentimes when I sort of like give sort of these lectures or talk to people about this stuff, they're like, how do I, there's so much going on, how do I stay on top of this? And it's like, you know, there's something similar to just like experiment for 10 minutes a day, like just spending some 10 minutes doing something that you didn't do before. If that's a traditional way that you were solving a problem, there's now a new tool that you want to try or a new like framework or something. And just spending, maybe even doing the exact same problem, just with that new kind of thing is alone will cover most of the gap.

• 22:08Speaker 00

  Yeah. I will call out, so some of your, for example, you have cognition on the week that you're talking about code review.

• 22:15Speaker 01

  Yeah.

• 22:16Speaker 00

  I actually did work with Silas on Devon Review.

• 22:18Speaker 01

  Yeah, yeah.

• 22:19Speaker 00

  Pretty good, I would say. Like, you know, it's hard to benchmark these things.

• 22:22Speaker 01

  Sure.

• 22:22Speaker 00

  There is a review bench, no one uses it. But like qualitatively, Devon Review is pretty good.

• 22:28Speaker 01

  Well, we use you guys. That's also

• 22:30Speaker 00

  Oh, really?

• 22:30Speaker 01

  Yeah. So Monica does use Devon.

• 22:32Speaker 00

  Oh.

• 22:32Speaker 01

  And that's, that's, I mean, and Silas actually spoke last year. And so, I mean, Silas is great. He's just super, super smart. But, um, yeah, we use you guys. And so I was like, okay, among the providers of kind of code review tools that we, and we sampled a lot of them. We actually did kind of converge on Devon for

• 22:46Speaker 00

  Yeah.

• 22:46Speaker 01

  A lot of reuse cases.

• 22:47Speaker 00

  Silas is actually more on the model side these days. And then actually, it's Walden that you want.

• 22:51Speaker 01

  Yeah, yeah, yeah. I spoke to Walden as well.

• 22:53Speaker 00

  I will also shout out Eno from Factory because, you know, talking about agent readiness, they've been all about that. They build signals, tooling on that. Do you guys use that?

• 23:02Speaker 01

  Yeah, yeah. Yeah. So we do use agent readiness of our own, of our own kind. I mean, Eno is awesome as well. Like, I think they were super early on this, on this train. And I think when he qualified that in his talk, I guess at AIE, what was it, like last, I forget when it was, but one of his

• 23:15Speaker 00

  Europe, yeah.

• 23:16Speaker 01

  Yeah, Europe, or just fantastic talk. And I think it was kind of in a way ahead of its time. And so I was like, there's only one person that I could see being like the most appropriate person to talk about this, where I like really wanted to give him the spotlight and

• 23:26Speaker 00

  They're super excited. Yeah, that's funny. I was at Cog, we were talking about this stuff and we're like, okay, we need to form some opinions about it. And then like Factory like shipped a full product on it that week that we were talking about it. I was like, okay, you want?

• 23:37Speaker 01

  Yeah, yeah.

• 23:38Speaker 00

  I mean, like, everyone kind of agrees on the same things.

• 23:41Speaker 01

  Sure, sure.

• 23:41Speaker 00

  And it's just a question of like, do you want to turn it into a dashboard?

• 23:44Speaker 01

  Yeah, yeah, yeah, yeah, yeah.

• 23:45Speaker 00

  Because it's a dashboard.

• 23:46Speaker 01

  Right, right, right, right, right, right. But you know, there's, there's value. And like, I think they did well from a marketing perspective, jumping on like the readiness and report. And I've definitely spoken, you know, I will regularly kind of pitch that to other people and say like, yeah, you should run readiness reports of your own kind. And, you know, of course, now people have open source versions of it, but, but I think the principles are super important.

• 24:03Speaker 00

  Yeah. Do you want to go piece by piece? Or do you have any other things? I can go piece by piece. Yeah. Okay. So the first one, right? You focus on Claude Code, but then you also have Pi as an open source sponsor.

• 24:14Speaker 01

  Yeah.

• 24:15Speaker 00

  What's up?

• 24:15Speaker 01

  Okay. So, so, okay, this one is maybe a little bit more marketingy, but, but okay. The one thing I do want to highlight, and I've made that, I also made that clear last year, but I'm not a, I'm not endorsing any tool. Okay. Like, I don't, I think that is the wrong way to approach something like this. There are tools that come and go. And when, you know, transparently, when, when the class was taught last year, Claude Code was not as popular. It wasn't as like really accepted. Boris was only at 4,000 Twitter followers. And now he's at, you know, whatever million, I don't even know if he is, but you know, like, he's at a lot of followers. So Claude Code has really like emerged on the scene. I think

• 24:50Speaker 00

  Yeah, I think we also underestimated Claude Code when we had Boris and Kat on the podcast.

• 24:54Speaker 01

  Right, right.

• 24:54Speaker 00

  It was like two months old. I hadn't really, I used it once. A query cost me 50 cents. I was like, who's gonna use this shit?

• 25:00Speaker 02

  It was still in beta for a while, but I think it's good on them for how they built it before the model was ready.

• 25:05Speaker 01

  Yes.

• 25:06Speaker 02

  Like, you hear so much build for where the model will go.

• 25:08Speaker 01

  Totally.

• 25:09Speaker 02

  Super ambitious. Like, models were not that good. You couldn't feel it. And then we see, we see that's where it came from.

• 25:14Speaker 01

  It's, yeah, it's funny, because when he gave his talk last year, I remember, you know, it was like a very high level sort of set of slides, but I remember one of the slides was literally built for where the model is going to be six months from now. And this was like October of last year. And I can see why he would have made that point if he knew within six months where like Opus was going to be. Because that was also what I think was a huge unlock where I would argue was one of the reasons why the curriculum changed as much as it did, right? Where there was starting this huge arms race around true like autonomy in these agents, like long running agentic coding agent tasks. But yeah, so, so, you know, back to your points, like less on, I'm not endorsing Claude, I mean, you know, Claude Code, of course, I use it and use other things as well. But, but here I was more about like the, what does a modern coding harness look like? I wrote a blog

• 25:55Speaker 01

  post that actually, yeah, this one, where this was, yeah, earlier this year, where I sort of broke down some of, some similarish principles. The TLDR was that, you know, you could sort of, through like pretty much introductory algorithms level understanding, you could just find like essentially an agentic loop that does the bulk of what you would call a coding agent. And of course, there's like bells and whistles. There's no subagents, there's no MCP and none of that stuff. But if you want to see something that you talk to, and then it literally will edit your file, and you can do pretty sophisticated things with it, it doesn't require more than 200 lines. So I, you know, a lot of what I try to do in education is to try like peel away the stuff that's like the secret sauce and show that like really at its core. I mean, these are still students and you want to teach conceptual principles. There's nothing more to this than, in Stanford, there's this

• 26:40Speaker 01

  class called 106B. It's like the introductory, introductory algorithms. You don't need more than 106B to build something like this. And so, there you go. And there it is.

• 26:47Speaker 00

  Given that you already taught this class, any pushback, any questions, any weirdness that you observed from the students?

• 26:53Speaker 01

  On

• 26:54Speaker 00

  What do they not get? What do they ask, seem to cluster questions around?

• 26:57Speaker 01

  Hmm, you mean like now that this, like this particular concept or just in general?

• 27:01Speaker 00

  Uh, this concept.

• 27:04Speaker 01

  Of like

• 27:05Speaker 00

  Of like, of like building it from scratch and then like doing, doing what you did. Because this is basically what you needed for Pi.

• 27:10Speaker 01

  Right, right, right. Exactly. Right. Yeah, yeah, yeah. Right. So, yeah, it's, uh, it, not really. I think they're, I haven't got received any like pushback on like this idea at all. I think they're actually more just, most of them are consumers, right? Most of, most of the students do just like end up using the tools. I don't think that they get a lot of exposure, if any, certainly in the coursework around peeling that onion back. Right? And I think, so this was the week of the harness. So, so Tuesday, first lecture, you know, logistics, blah, blah, blah, all that stuff. But then first half is, you know, the first true like bit of curriculum is the 200 lines of like a harness. And then yesterday, when I had sort of the follow up to that is like, we actually took Claude Code and then, you know, you have these like proxy things, these like proxy tools where you can intercept the

• 27:53Speaker 00

  MIT in proxy?

• 27:53Speaker 01

  Yes, exactly. Exactly. Right. So use that, use that tool, intercept the requests and responses.

• 27:59Speaker 00

  Nice.

• 27:59Speaker 01

  Look under the hood, see what's happening there. And then, then it's like, okay, I did say you have Claude Code and 200 lines, but really, okay, here's like actual Claude Code. And so what I literally did is I took Opus 5.5. So this is like, you know, model of two days old. And it was like, this is, I intercepted those requests. Here's exactly what's going on. Here's the prompt. Here's the system prompt. Here's the plan prompt. Here's the compaction prompt. And then we just like went through all of that and sort of showed it to the students. So now they really, really have like a state of the art harness that they can understand.

• 28:24Speaker 00

  Yeah.

• 28:25Speaker 02

  It's nice to get those three levels. Like, students are very much consumers. Here's the baseline agent in a loop. And then the real one. I like how you have a video walkthrough of your blog post as well. Do you see students are on like extremes of, some people are power users, they consume a lot. For some people, you could assume that this is kind of very straightforward if you know how agents work.

• 28:49Speaker 01

  Right.

• 28:49Speaker 02

  How do you kind of balance that in the classroom setting? Like, do you tailor towards the average?

• 28:55Speaker 01

  Yeah, I think it's a bit self-selecting because the curriculum is all here, right? So the curriculum has been available since, I mean, you know, at least like a few weeks. And I guess I announced it a few weeks ago, but it had kind of been in the works for like probably at least one to two months. So if you looked for it, you could probably find it. And I feel like the reality is if students are not interested in this, then they won't like, they'll, they just won't show up. And they will, they will choose not to show up. What I'm finding that is interesting is there are definitely students that show up, but then there's also just a lot of folks from, you know, broader Stanford, postdocs at other departments. There's a lot of like interdisciplinary sort of interests where folks that are in, you know, somewhere in pathology or like, you know, like medicinal sort of fields or even like random other engineering fields that aren't pure CS that are coming in and

• 29:39Speaker 01

  also interested in kind of getting exposed to this. Also a lot of GSB students, MBAs that also want to, I guess, learn how to not have a technical co-founder or whatever the motivation there is, but.

• 29:52Speaker 00

  Yeah, so I think, by the way, if anyone hasn't done the MITM proxy exercise, you should do it.

• 29:58Speaker 01

  Yes.

• 29:58Speaker 00

  You'll be shocked. Cloud Code has the longest system prompt of all the coding agents, something like 20,000 tokens.

• 30:05Speaker 01

  Yeah, yeah, although, yeah, that is true, that is true. Yeah, yeah, yeah. Although they've also reduced the size.

• 30:09Speaker 02

  They cut it by 80%.

• 30:10Speaker 01

  Yeah.

• 30:10Speaker 00

  Yeah, I know, but then they grow again and then they cut and they grow again.

• 30:14Speaker 01

  It's a seesaw.

• 30:15Speaker 00

  Yeah, yeah, no, I think it's interesting. You'll be surprised like what goes in there that actually sometimes you don't use. Sometimes it actually hints at the internal workings of some of the tools. Anything that you've found there that could be cool to talk about?

• 30:28Speaker 01

  Yeah, no, I mean, in fact, yesterday, when I was talking about this, it was a little bit of this like, one of the things that I thought was most interesting about Cloud Code back in the day, and I think part of why it really blew up in terms of popularity was their focus on simplicity, right? So in the early days, they were among the first to really lean into this. Let's fewer tools do more, right? Lean into SQL, lean into Bash, lean into like the things that are powerful, like Swiss Army knife tools and have that be a lot of the driver of capability. I think they did a fantastic job leaning to that. I think that's why they were as popular as they did. I think that when I presented something like the system prompt, it was like, well, there was a different system prompt about two months ago. It is now, they have reduced in complexity. And it turns out that now what was once part of the system prompt, things like how to use even Git, right?

• 31:13Speaker 01

  Or how to like set up tasks or things of that nature are no longer part of the system prompt. It's actually expected to be more kind of captured in the models, like the model weights itself. So they've kind of baked in a lot of those learnings more at the model level. And that just raises interesting questions around, you know, the model versus the harness, right? And how do you kind of think through that trade off? What do companies invest in? What should you invest in to really push the frontier of your own capabilities? And so I think we still, even as a community, we seesaw between the two a lot.

• 31:39Speaker 02

  I think you don't want to fault them for growing a system prompt and then cutting it. At some level, it's also how the model was, right? Weaker, older models, you need to give them examples of this is good, this is bad. And then obviously you cut that away at some point, right? And that's just part of learning what has been done, right? You can apply the same towards if you're using a local coding agent with not the strongest model, you might need to add these examples, right?

• 32:05Speaker 01

  That's right. That's accurate. Yeah, yeah. And then, and then of course, there's interesting AI learnings to fall out of that, which is why, you know, as the models get better, when are you over-constraining it by adding those examples, right? And then how do you pull away the over-constraining? And that's, I think, part of the learning as well.

• 32:20Speaker 00

  Okay, week two, you're going to talk about context engineering. Repit, I haven't seen, but there's many versions of speculative development. What would you, would you highlight to a broader audience about what you're putting in here that people should know?

• 32:35Speaker 02

  Yeah, this is like the, you know, one hour version of the course.

• 32:39Speaker 00

  Yeah, exactly. Please condense everything.

• 32:42Speaker 02

  I think that's where this is going.

• 32:44Speaker 01

  That's fair, yeah, that's.

• 32:45Speaker 00

  Well, no, it's also a trailer, right? It's like, well, if you want more, go to the source.

• 32:49Speaker 02

  If you're signing up next year or next fall for courses, you know, this is your one hour.

• 32:53Speaker 01

  This is like, this is like, almost like the quibi of the class, you know what I mean? Where it's, it's too long to be a trailer, it's too short to be the class. Is this like, what, I guess people will double click.

• 33:01Speaker 02

  You know, the best part, if it's a trailer, by the next time you teach it, it'll change again.

• 33:05Speaker 01

  That's true, that's true, that's true, that's true. Yeah, well, yeah, so, so, okay, so a good call out on India, it's funny. The, I think that the main thing that I just want to reinforce here is, and I see this even when I'm like interviewing folks. Very few people in industry even progress beyond like just vanilla prompting, right? There's like a form like, hey, do me the thing, right? Like do, do the thing. And, and that certainly as the models get good, that can carry you a certain distance. But inevitably, you know, any folks I speak to that are, come out of like a Google or someplace where there's a massive like codebase, the reality is you have to be a lot more thoughtful about how you're like ingesting context, right? And so thinking through what are some of those principles and those techniques around how to like point, you know, and I sort of like, you know, the rapid framework that I'm sort of describing here

• 33:49Speaker 01

  is functionally similar to like Dex's RPI, right? Or something of that nature where you have this like research kind of proposal and sort of spec. And then, and then you do like your implementation and then you kind of tie it together with some testing and whatnot. So what I just wanted to reinforce here is that like you should have this step, especially when you're entering on familiar codebases, when you're trying to sort of infuse into the context, something that an agent needs to understand about a very large codebase. There's value in something that is a research phase. And then in the case of this particular sort of framework that I'm putting forward, it's like also then suggests some proposal where there's a few alternative ways of solving a problem once you've done the research. And so this is where I sort of also invite the human in the loop to the sort of be the decision maker on one, one approach versus the other.

• 34:33Speaker 01

  And then you enter the plan, which is kind of the spec. All to say that the TLDR is like, there is a way to do this that actually does scale out to codebases that are, you know, millions of lines long. And it's not just the vanilla prompt. And I think that's what I want to highlight. If you're going to go work in these big codebases, you should use something a little bit more, a little bit more involved.

• 34:51Speaker 02

  Anything to add on context engineering?

• 34:54Speaker 00

  I think that probably the weirdest thing about how I think about context right now is when you split out to a sub-agent, because sub-agents can benefit from inheriting your entire context of the main thread or not. Sometimes there's a regular disagreement on that. And then sometimes you want to condense what the sub-agent does and researchers put it back onto the main thread. And it's very hacky, I don't like anything I'm doing. So I don't have any answers. All I know is like, when you want to fan out work, it is not just about you have a big spec and you, you, you know, you do RPI, you do R, Repit, whatever. It is not one agent anymore.

• 35:39Speaker 00

  It is about fanning out work and making sure they all have the right context and they all have the right guidance. If you have a review agent reviewing all the other works, you have to make that guy have everything. And you always go through this phase of expanding contracts like, oh my God, these guys don't have enough context, I'm going to give them more context. Okay, great. And then they're going to overture into some random one-off comment that you made a while ago and then make some really stupid mistakes because you didn't mean it that way. But like, they took it as the law, something like that. And so I don't know what the word is, I'm saying, right? It's just like multi-agent management for context.

• 36:12Speaker 01

  Yeah, yeah, yeah, yeah, that, that's absolutely right. There, there's like some aspect of, this is maybe the part that's more maybe among the most art than science, right? It's like some combination, I think to your point, yes, the fan out for the exploration, the fan out for like the subtasks and then, and then the consolidation, compaction, whatever, you know, you might want to call it. I think being thoughtful about refreshing of the context, understanding when things are getting a little bit too bloated. I think, you know, Dex had this concept of like the dumb zone and sort of like being a little bit more thoughtful about the dumb zone.

• 36:38Speaker 00

  He's good at these equating things.

• 36:40Speaker 01

  Oh, yeah, yeah.

• 36:40Speaker 02

  Little diagrams.

• 36:42Speaker 01

  Yeah, oh, yeah.

• 36:42Speaker 00

  Why didn't you have him on?

• 36:44Speaker 01

  Yeah, well, I think he's, I mean, he helped, definitely informed a lot of the curriculum here. And so he's just fantastic, I think, thinker in all this. No, I think the diagrams are huge. That's useful.

• 36:53Speaker 02

  I would add breadcrumbs for people that are trying to look into this. It's good to see best context engineering preferences, like on your sub-agent inheriting. Cloud-managed agents has a version where sessions are separate and it's not inherited. This used to be a big topic before reasoning models. When agents didn't run as long, we had shorter context windows. Everyone knows very, very managing that precious context window, right? Linkchain used to write a lot of blogs about this and good context engineering. All this stuff is good for people to know. I think in the student context as well, you can expect the average student to take your class because it's a great class and they'll know all these things.

• 37:35Speaker 02

  So you should know them, know them better as well. This is a fun space for research as well. One of the papers, research papers presented at AIE, they looked at the delta in the Cloud Code system prompt, what got changed? And they like, they, they plotted a whole bunch of changes in performance. They tried to reverse engineer it. It's just good research that you can do. There's another one that also looked at context window management and how it affects performance downstream. They weren't the best papers, but like it's quite accessible and it puts out good blog posts. So people that are trying to work on it, you know, I just recommend it's a, it's a good area you can easily poke into.

• 38:14Speaker 01

  Yeah, good shout out.

• 38:16Speaker 02

  Agent skills, CLI?

• 38:17Speaker 01

  Yeah, yeah, I think all this is, so a little bit of an extension from a single, single kind of quote function tool calling, I like MCP, right? Using, interacting with like third-party external services and then extending into like what you would call like procedures, right? Like, well, how do you automate some set of actions over and over again that you then formalize in the form of like a skill MD or something like that? So I think just making that a little bit more, giving people exercise in codifying some of those procedures. I think what I'll also sort of like include here is a few, few kind of fun skills that I think are good to use. Like I think Matt, Matt Peacock's like GrilleNeat is like a good one. And then I, there's ones that I also like from more from the educational perspective, like Jeffrey Litz, he has like Explain Diff, which I think becomes especially interesting

• 39:01Speaker 01

  once you start talking about code review because, you know, the whole.

• 39:03Speaker 00

  Understand the code.

• 39:04Speaker 01

  Understand the code, exactly. Yeah, you got to have understand the code. And, and I don't know if you guys have used that, but effectively like you run, you know, you have your, your kind of, the diff you've put in your code and then you run the skill. And at the end of it, you have this, not only like kind of a visual representation of what's happened. So you can, you know, kind of skim it and understand what's, what's actually at the core of it. But then what I really liked was this almost like quiz at the end of, at the end of it, where you go through and sort of like really reinforce whether you understand what's happening. And I think that that level of interactive sort of back and forth, I think is, is quite nice.

• 39:34Speaker 00

  Yeah, there's, there's a couple of blog posts in this domain that I really like to bring up. This one, I think recently, which is like kind of, there's no situation in which turning your brain off is a good idea. Like, you know, and, and I think these skills actually force you to turn your brain on. And when everyone wants to use coding agents, turn their brain off. They're just like, make me an app and get, read my mind and get, get everything correct without me telling you about it.

• 39:57Speaker 01

  That's right.

• 39:58Speaker 00

  But like, a lot of these skills are mostly in the genre of like, no, no, like ask me more questions, please, like.

• 40:04Speaker 01

  Yeah.

• 40:05Speaker 02

  I think there's a lot more output than people are used to. Like I can type in a one-line prompt and get tens of thousands of lines of code and a whole structure and.

• 40:13Speaker 00

  It's like, look at my productivity, guys.

• 40:16Speaker 02

  But you know, more effectively, you should know what you're working on. You can really be put out of touch, out of the loop. You should spend more time thinking about what you're actually doing. Doing the work and getting the code written is much easier than it was.

• 40:33Speaker 01

  Yeah, yeah, the understanding there's no excuse for, for sure.

• 40:36Speaker 00

  What is a web skill?

• 40:38Speaker 01

  Yeah, so I mean effectively like the skills that interact with like the web itself.

• 40:42Speaker 00

  Computer use.

• 40:43Speaker 01

  Computer use, yeah, things like this.

• 40:44Speaker 00

  Anything else?

• 40:45Speaker 01

  I mean mainly things that involve like pulling pulling information from the web.

• 40:48Speaker 02

  Are there best practices on skills that will interact with the web versus?

• 40:53Speaker 01

  I think this is more of like a bit of a survey or kind of a demo of stuff that I found that was kind of interesting. I think it was, it might have been out of, was it like a versel? I'm trying to remember now. I'd have to look back at the full like reading materials, but there was like one site that I found where I think they, maybe it was, it might have been browser based, actually. They might have like done, do you remember this?

• 41:12Speaker 00

  Well, it would be them that been coined this thing. It's just I was like, did I miss something? This was like a big part of the skills community.

• 41:19Speaker 02

  This is good. He's being educated.

• 41:22Speaker 01

  Yeah, I think they like made some sort of, because I guess they were, yeah, I mean browsers obviously like.

• 41:27Speaker 00

  They like browsers.

• 41:27Speaker 01

  They like browsers and, but yeah, they had like a good, I think website where they showed some of these interesting capabilities.

• 41:33Speaker 02

  Seems legit. CLI is a fun one. Do you have a lot of students or this is their first time in a terminal in a CLI or is it kind of the norm now?

• 41:41Speaker 01

  Uh, no, I don't, I don't think so. I think that given that most of them, again, last year, I would say that some of them had probably still remember the days of finger on keyboard typing. This is, most, I think, students have like interacted with Claude code at this point.

• 41:57Speaker 02

  I mean, I'm curious the split of people that are now just using the apps, right? Claude code also has a desktop app. So does Codex. Codex is primarily an app now. You're kind of not in the CLI, not in the terminal. It's a thing that puts off a lot of average people, but just random note.

• 42:16Speaker 00

  Yeah, I would say, you know, I think there's also another thing about skill that are skills that are bundles of skills where they sort of cross reference each other.

• 42:25Speaker 01

  Yeah, like the plugins, right?

• 42:26Speaker 00

  Like plugins. And then there's also, I guess, the art of writing a skill, which is a thing that I've also focused on for AIE. I think a lot of these are focused on skill consuming. And you have all these influencers like Matt Polkog and what have you. They're like, this skill will save, it will solve all your problems. But no, you should write your skills yourself. And actually, when you try to write them naively, they're usually bad. And actually, it takes a little bit of skill to write a skill.

• 42:55Speaker 01

  That's absolutely right. That's absolutely right.

• 42:56Speaker 02

  What are Srix's best tips on writing?

• 42:58Speaker 00

  I have a skill, I have a public skills people. I went through definitely a phase of psychosis of like skill, everything.

• 43:03Speaker 02

  No, no, no, don't give us skills. How do you write good?

• 43:05Speaker 00

  Yeah, exactly. So I have a skill for cutting skills. So I, I like there, so every like Claude code and Codex, they all have skill creation skills, right? So then there's, I have a skill cutter skill that cuts it down because they're all, it just documents every way in which those default skills that are created are bad. So you always run this after any skill is created to cut it down, right?

• 43:30Speaker 01

  So like this.

• 43:32Speaker 00

  Yeah, it's really annoying because sometimes there is slop and like, but there's good in the slop. And then you have to like sort of whittle down what they're, what they're.

• 43:41Speaker 01

  Right, right. Oh, interesting. As in you're saying like even the slop is, it's still in the language of the agent. So the slop means something. It's like, it's like a, it's a, yeah, okay, okay, got it, got it.

• 43:49Speaker 02

  It's fun. Everyone has their own version. I used, first, there's very easy to like vibe check a skill, right? You just call it, run it a few times. And then I distill these preferences into a skill that's like, this is what I like, this is what he doesn't like. And it adjusts over time and then, you know.

• 44:05Speaker 00

  Well, so the problem is that, yeah, you're calling it, but the actual, the worst part is when a skill triggers when you're not expecting it to. Like you wrote it one time to fix a thing and then like now you're working on a completely different thing and it triggers. Why? Because it's badly written, because the trigger says like, oh, you call me whenever blah, blah, blah happens. And you're like, okay, read's good. And then you do a different project and it, and it triggers and you're like, oh, goddammit.

• 44:28Speaker 01

  What the hell should happen?

• 44:29Speaker 02

  Yeah, there's a, there's like a few people that are just like, okay, I have a bunch of skills. I keep adding on, sometimes run this up. And I know some people like Eugene Yan, they are pretty particular about how many skills they have. They prune them. If not used, they clean them every few weeks.

• 44:45Speaker 00

  Yeah, yes. So, so yeah, this is like a huge problem if you're, if you're sort of in that skills world. I think this is actually why I suspect they're good, they're very smart people at OpenAI and Anthropic that want skills to die. Which is not a public statement. No, now they won't say this publicly, but it just feels sloppy. This feels sloppy. I know it's, I can see my own skills people being sloppy. It doesn't feel AI pilled.

• 45:14Speaker 01

  Right.

• 45:14Speaker 00

  Do you know what I mean?

• 45:14Speaker 01

  Yeah, yeah.

• 45:15Speaker 00

  It's not, it's not what AGI was supposed to look like. It's like everyone having a giant bunch of markdown files.

• 45:21Speaker 02

  I mean, I think part of it is like.

• 45:23Speaker 01

  This is a gardening problem. Yeah, in some sense.

• 45:25Speaker 02

  A lot of people actually don't look at their markdown files, right? Like the UX of it, of just chatting what you want, creating a skill in chat, using it, seeing output. There's a lot you can, there's a lot of value you can get without ever looking at your markdown. Don't, don't overcomplicate it at some level for yourself. You can get a lot of value. If you do stuff repeatedly, just be like, look at the last five times we did this, make me a skill and you can iterate it with chat. There's a good side of it as well, right?

• 45:54Speaker 01

  There's another thing, there's like, I guess, another way to look at it as well, because there's, there's the use of the skill as just procedure that the agent kind of runs whenever to, you know, do like a code review or whatever. But then in the case of something like Devin, right, where we've qualified skills as a way of, of sort of, of representing our best practices, right? And then making, and exposing that to the agent or the code review system to then, you know, institute the APIs the way that we like to define them, institute kind of the service level boundaries, the way we define them. That's like an alternative way of sort of representing like org level knowledge. And maybe not so much this like arbitrary thing you want the agent to do, so you don't have to write the prompt each time.

• 46:33Speaker 01

  And so I like, I like that. I feel like that positioning might, that, that purpose might exist longer than just, you know, these kinds of random procedures.

• 46:42Speaker 00

  I just Googled Monaco AI skills and it's giving me the, the country.

• 46:49Speaker 01

  One day, well, we're gonna beat them on AEO, that's for sure.

• 46:52Speaker 00

  Can you just buy the country?

• 46:54Speaker 01

  Well, here's the thing, okay, on this, we, we, we're about to buy, no, we're not about to buy the country. But, but, but, um, you know, we, we did, we did get into a little bit of flak when we bought the monaco.com domain. I will say that, we did get sued by them, so.

• 47:08Speaker 00

  What?

• 47:09Speaker 01

  Yeah, there's, there's like a whole public, there's a whole public thing about this where, yeah, so they didn't like that, but, but here we are. And we won, we won, so.

• 47:18Speaker 02

  Okay, then we're keeping it in. I was about to say, this is spicy, but you won.

• 47:22Speaker 01

  Yeah, yeah, no, no, no, it's.

• 47:24Speaker 02

  Is that why we see the flags flying around?

• 47:27Speaker 01

  Because it's like a victory. It's because it's a victory. It's our victory. That's funny.

• 47:31Speaker 02

  For those that are in SF, just look up and you know.

• 47:35Speaker 00

  Yeah, yeah. So for those who are on audio, we're showing this on YouTube. It's, you fly a plane around. I think you're still one of the few that actually do this on a regular basis. Sam Blonde came up with it, $6,000 a day. I don't necessarily think it directly causes me to Google what Monaco is, but all of Monaco's ads are just Monaco dollar sign.

• 47:58Speaker 01

  Right.

• 47:59Speaker 02

  We stand for money. Like dollar sign in the next numbers.

• 48:02Speaker 00

  It starts the conversation.

• 48:03Speaker 01

  Right.

• 48:04Speaker 00

  All, the whole point of the ad is what the hell is Monaco?

• 48:07Speaker 01

  Yeah, and, and you know, you could, you could, like, that's a debate to be had, but, but in some sense, like, what is the purpose of marketing if not to like get attention, right? And, and yes, maybe there's one level beyond this, which is if somebody, so, you know, I'll go to events and people like, oh, you guys are the ones with the planes. You guys are the ones with like the big number on the billboard, blah, blah, blah, 193, you know, million, whatever it is. And, and that, but they're like, but what do you do? And then, but that's, but that's, but maybe that's the whole point, right? Is that I just needed someone to come and use it as an icebreaker.

• 48:36Speaker 02

  I will say, it's what I said as you walked in as well. I was like, ah, the planes.

• 48:41Speaker 01

  And now, now, what do you guys do? Well, well, let me tell you about what we do.

• 48:44Speaker 00

  As someone who's been in those meetings, deciding what we do on the Devin billboards, we spend so much time on the words and maybe we should just be like, Devin.

• 48:53Speaker 02

  I think you do.

• 48:54Speaker 00

  This is so fucking dumb, but like. Okay.

• 48:58Speaker 01

  You know, look, I think Sam had another post this past week where he was just talking about like, you know, I mean, Sam is effectively like our, you know, marketing sort of mastermind here. And yeah, I think there's a little bit of history behind the planes as well. But, but the TLDR is that his whole shtick is like, I don't have all the answers. You just got to try things. And I think a lot of this is we just try things and, and if it sticks, great. If it like, did we know this was going to blow up as much as it did? I mean, not really, but, but now that it did.

• 49:23Speaker 00

  An idea.

• 49:24Speaker 01

  Yeah, exactly.

• 49:24Speaker 00

  And minimalist, start small, go high instead of start big, go small.

• 49:29Speaker 01

  Right.

• 49:29Speaker 00

  Way easier to start small, go, go big.

• 49:31Speaker 01

  Yeah, exactly.

• 49:32Speaker 02

  Also, you know, it's six K a day, like it's, it's very much just try things. I think, I think people.

• 49:38Speaker 01

  Cheaper than billboard, cheaper than a billboard for sure.

• 49:40Speaker 02

  Yeah, and it's fun.

• 49:42Speaker 01

  Yeah.

• 49:42Speaker 02

  Not, not much harm done.

• 49:43Speaker 01

  But you know, it does, it's, um, but you know, the kind of simplicity here that that's worth mentioning is then when we're trying to like one up the plane, we're like, you know what's better than one plane? Two planes, two planes.

• 49:53Speaker 00

  What?

• 49:54Speaker 01

  My, my way to rocket.

• 49:56Speaker 00

  Well, okay.

• 49:58Speaker 01

  I had that idea. Uh, I would, I would like for that to happen as well at some point, but, uh, we'll see.

• 50:03Speaker 02

  This is the next SpaceX launch, we just see Monaco. For the ship of the stars.

• 50:09Speaker 01

  There you go, there you go. Got to warm up to our, our SpaceX friends.

• 50:12Speaker 00

  Okay, I'm happy.

• 50:14Speaker 02

  You don't want to.

• 50:15Speaker 00

  Well, I mean, we're, we're on Monaco. Is it worth talking a little bit about Monaco? Do you leave it today?

• 50:18Speaker 02

  Yeah, yeah, yeah.

• 50:20Speaker 00

  Because this is like an intermission between the, the topics.

• 50:23Speaker 02

  You decide.

• 50:24Speaker 00

  Okay, we, we mentioned Monaco, right? Like, let's, let's give, I, I, I think it's, I think it's a little bit annoying to like mention what the hell is Monaco and then not answer it.

• 50:32Speaker 01

  Sure.

• 50:33Speaker 00

  Can you answer it?

• 50:33Speaker 01

  Sure, well, I hope so, yeah, so.

• 50:37Speaker 01

  This is the country, so it's a small little, I guess.

• 50:42Speaker 01

  There you go, there you go, okay. That, this is Monaco, this is the Monaco that, that I know and love. Yeah, so we are, uh, you know, we're, we're a series B company, uh, here at Basin SF. We are, are building effectively like an all in one sales platform. Fully AI native agents doing all the fun stuff, uh, or at least the stuff that humans don't like to do, but, but that agents are, do like to do, because they don't really have opinions, but do in a less air prone fashion. So the salespeople and sales teams can focus on doing what humans are good at, which is, you know, closing deals.

• 51:12Speaker 00

  So presumably Mark Benioff doesn't like you guys.

• 51:16Speaker 01

  Probably not.

• 51:17Speaker 00

  But he's like, also honestly, a very AI native person, very aggressive and agent force and everything.

• 51:24Speaker 01

  Yeah.

• 51:24Speaker 00

  What are you doing that he doesn't do or he cannot do?

• 51:26Speaker 01

  Well, okay, so, so right now we are, uh, we started the, the company in a sort of like, the, the market that we went after was not enterprise from the get go, right? We sort of said like, we're going to start by, like, if our goal is to build the all in one, then we need to kind of build this in an iterative fashion with, with a group of customers. And we need to like grow with them into this. It's much harder to pull someone away from, from Salesforce than to say like, hey, if we meet you at the point when you are still just a Google spreadsheet tracking pipeline. And we sort of show you a, you know, more AI pilled sort of answer to the problem, then we can grow with you. And in a way, it kind of follows the same, the same, uh, model of like, you know, Brex are all like YC companies where you kind of sell to your batchmates, you kind of grow with them and, and you, then you go up, up, up market.

• 52:07Speaker 01

  And so we are, I would argue are the, you know, the, the, I would, I would claim that among the best, if not the best, sort of like a startup focused, uh, sales platform. And so all of our customers are like early stage startups at this point, probably through like, series A is, is kind of where, where we've primarily been focused, kind of seed to series A. And then now we're going more up market and we've kind of learned a lot about the all in one in the last year and a half of doing that.

• 52:30Speaker 02

  And you are the head of AI?

• 52:32Speaker 01

  Yeah, that's right.

• 52:33Speaker 02

  What's some cool AI stuff you guys have built?

• 52:35Speaker 01

  Yeah, I mean, uh, we have built our own agentic harness for, for sales use cases. So all this, I mean, to call back to the, to the coding harnesses, uh, taking a lot of inspiration from, from the systems like the codexes in the, in the clouds of the world and sort of building bespoke tools that are very sales focused, bespoke workflows that are all kind of geared toward GTM use cases. So we built our own version of that, that powers pretty much every single thing that, that powers the Monaco platform. So from, you know, uh, scoring of, scoring of TAMS, uh, scoring of ICPs and, and develop, define of ICPs, uh, through to pipeline evolution, through to like all the mundane kind of things that maybe a salesperson doesn't want to do, which is like, oh, update my meeting, you know, pre-meeting notes, update my summaries, like all these things.

• 53:19Speaker 01

  You know, we have agents powered, powering all those workflows. And so we, we've done that. It's, it's kind of comes, comes out of the box.

• 53:28Speaker 01

  And, uh, yeah, so long, long agents, long horizon agents for, long horizon agents and short horizon agents in some cases for, for, for GTM use cases.

• 53:36Speaker 02

  And it's pretty domain specific, built on foundation models? Would you do any of your own models for this?

• 53:42Speaker 01

  Yeah, it's a good call out, uh, because so far, I mean, I think this is generally true of a lot of like vertical AI companies. It's like, you kind of start where you sort of use the frontier models, like as third party services to see what's possible, because they are like the very frontier. And then I think we've seen this, I guess, in the past week with the whole like Harvey, sort of where they like the plus 50% to minus 50% margin kind of question. I mean, I think in, irrespective of the numbers, I think what is interesting is like, at a certain point, things do become interesting from a cost perspective. You, you start to wonder, can you do it faster on, you know, open weight models? And then that's a lot of that, then a lot of what I think the, the purpose of, of the AI pieces are is just to posturing those models, build environments for, you know, kind of to posture and using RL and things of that nature. So, um, we're kind of in that phase right now.

• 54:25Speaker 02

  Ooh, fun.

• 54:27Speaker 00

  Yeah, I've been calling this playbook to the agent lab playbook, but that is kind of my coining of like, you know, like agents start from the user first and then they, uh, they, they use whatever models you need to, to build products. But then you can backfill the, the model side once you have the user base. And then the model labs could kind of go the other way. They, they start the big P trains and then they try to find users. And it's like, very, you know, it's hopefully working as a, as a, as a split and Monaco would be on the sort of revenue GTM side.

• 54:55Speaker 01

  That's right, yeah.

• 54:57Speaker 00

  And I think it's interesting, uh, they would want to serve you while also try to compete with you. As, as monolabs.

• 55:04Speaker 01

  That is correct.

• 55:05Speaker 00

  One of the things that I've been, I was, I've been, I, I dabble in, in some of this. I think one of the hard problems is actually enrichment.

• 55:14Speaker 00

  I would say I spent a lot of my time doing that for, for basically SEO and like, uh, research and what have you. And I imagine for you, it's actually like a core part of that.

• 55:25Speaker 01

  That's right.

• 55:26Speaker 00

  Do you find that models are good at it? Do you have any vendors that like, you like working with, you want to give them a shout out?

• 55:32Speaker 01

  Yeah, yeah, absolutely, yeah. So we, we have used, we were early adopters of parallel and then I think they're very strong.

• 55:38Speaker 00

  Okay, so just your general search.

• 55:40Speaker 01

  Yeah, I mean.

• 55:40Speaker 00

  Not like zoom info or some one of those guys.

• 55:42Speaker 01

  No, no, no, no. But yeah, we use parallel, we also use Exa, um, so we'll give it, give a shout to both of those folks.

• 55:47Speaker 00

  Can you compare them?

• 55:50Speaker 01

  They're both friends. They're both friends, they're both friends, they're both folks that, that we like.

• 55:54Speaker 00

  They'd be very interested if you like one of them.

• 55:57Speaker 01

  Yeah, we will. I'll keep it, I'll keep it neutral. I'll stay, I'll stay Switzerland in this today. But yeah, no, so, so, and then of course we do our own version of enrichment as well. So we, it's, it's like a, most of our data systems, I mean, part of like, one of, one of I think the most valuable pieces of software that we built or most valuable systems is kind of like our world database, like the database of companies, enriched companies, enriched accounts, uh, enriched contacts. So like that whenever we are constructing ICPs for folks, we have the most up to date information about who you should reach out to, at which company at this point in time. Like, I would say that one of our, like, I think where we are like world class, hands down, is just demand. Like, we are a very good demand company. Like, we help companies increase their meetings booked in a way that actually.

• 56:36Speaker 00

  So find buyer at.

• 56:37Speaker 01

  Find buyer right at the right time, right? And it's kind of like the combination.

• 56:40Speaker 00

  What do you mean at the right time?

• 56:41Speaker 01

  Like, there's, there's times when, like, here's an example, like, if you are, um, if you are targeting someone that just, you are selling to, um, VPs of sale, let's say, okay? Like, the, the right time to, to reach out to them is not when they are like sort of in the middle of some, like, they've been at the same place for the last like five years. If they are now in the middle of like a job, like a, a good signal is like once, if they've gone from one job to a new job, they're now like entering a new place where they're more likely to be to purchase or be open to a purchase of a new sales platform because they're in the middle of that job transition. So going from company A to company B for like V sales, a good signal of like, hey, should you reach out to them at this point in time is did you detect that their LinkedIn changed or something like that?

• 57:19Speaker 02

  How much of this is your expertise put into model and harness versus models are good, there's generic best practices? Like, is there, there's obviously some value in both.

• 57:31Speaker 01

  Yeah.

• 57:32Speaker 02

  Is it just the glue, getting it reliable, doing it all in one platform, doing it well? It seems like there's also stuff that we wouldn't know that you guys are also domain experts in that's baked in?

• 57:43Speaker 01

  Yeah, I think without the former, it would, it would be like a fully commoditized. I mean, because there's still a lot of value in the all in one. I think like that is something that we definitely pioneered, like most companies in the sales space started as just like point solutions. And I think they're, they tried more of like the YC approach of you kind of like land and expand, you kind of pick a niche and then you expand from there. Uh, and we from the get go, I mean, that's why we stayed in stealth for the greater part of a year and some months was just building, because we knew we want to be the all in one from the start. But you know, that alone is, is powerful, it's meaningful, but I think the real power is the fact that today Monaco as Monaco has like a world class go to market team, like a truly like world class go to market team, selling Monaco, of course.

• 58:19Speaker 01

  But we also have the same go to market people that sell Monaco are the ones that are like what we call forward deploy account executives, they're FDAEs, they're effectively like, well, it's like human, human salespeople that are like the consultants of sorts that help onboard all of our customers, get them like building agentic workflows. And are effectively there to like help guide them through every part of their own sales go to market lifecycle. And so to your question, it's the all in one, yes, but it's also the fact that we have these like, you know, Sam and, and we, we have brought on a lot of like former Sam, uh, Sam's team members and stuff like that, who were like the best people in GTM at Brex and other places. So we're codifying all of that in the platform.

• 58:56Speaker 00

  Yeah, it's interesting to have, you talk very differently than a normal head of AI that I might also talk to, because you're also a former founder, right? And like, you're, you're also think, used to thinking about these things in this way. And it's like, interesting to, it's important to have that in your role. Yeah, cool. Uh, I, I, I think you've, you've introduced Monaco enough there and then obviously I think as the Monaco, the story develops, we can revisit.

• 59:19Speaker 01

  Sure, sure.

• 59:21Speaker 00

  Maybe the bridge back is also what is coding or agent native coding or modern software development like inside of Monaco? You know, I think you're, when you're teaching, you're maybe trying to be, here's a lay of the land, but maybe inside of Monaco, you guys are more opinionated on one thing versus the other?

• 59:40Speaker 01

  Yeah, yeah, no, I mean, we have, so we made investments in coding agents from basically like day one and we've like learned a lot. We have our own like scars that we sort of like picked up and learnings and I think we've now been getting much deeper into sort of sharing some of our learnings. So like in our blog, which might be worth sharing here, we've been increasingly posting some of what maybe not that, so that one is more like a product for the one at the very top of the stuff, I would say like, yeah, when AI gets desperate and before like scaling contributions, builder buy, SRE. So, you know, we spent a lot of time looking at different providers and vendors and ended up building a lot of things in-house. So for us, something like a background agent platform, we have our own version of like a ramp inspect that we call Monocoder, which is built on the Coder platform, but it is

• 1:00:23Speaker 01

  what we use to essentially scale up our engineering productivity.

• 1:00:26Speaker 00

  Coder like coder.com?

• 1:00:27Speaker 01

  Coder.com, yeah.

• 1:00:28Speaker 00

  Oh, okay.

• 1:00:29Speaker 01

  We actually, yeah, have sort of adapted it to some of our use cases and yeah, it's where a lot of our agenty development happens. It's just this really well-designed piece of software for essentially isolating of sort of development environments, scaling out to multi-agents, you know, being able to run these like long, long horizon agentic coding agent tasks. So Monocoder, shout out to John Tracy and Co. for, for building that out. And, and then yeah, like from there, we've also built our own versions of like a, you know, like a Slack bot, which is called Hal, but effectively Hal is like this part AISRE, part like customer success, part like kind of QA engineer, because it lives in our Slack,

• 1:01:11Speaker 01

  but it serves all the purposes of helping us debug things and put up PRs when things are broken, but also used by our own salespeople to find out how the product works, find out like how to get to the bottom of something. And so there's like interesting questions we've had to answer around the different personalities that Hal has, depending on whether it's talking like a salesperson versus like a developer. And so systems like this that we built out over time, a lot in the last like year or so, shout out to Yunja for, for Hal and John Tracy as well.

• 1:01:40Speaker 00

  So you're building a lot in-house?

• 1:01:43Speaker 01

  Yeah.

• 1:01:43Speaker 00

  You also use other tools?

• 1:01:45Speaker 01

  We do use other tools, yeah.

• 1:01:46Speaker 00

  So what is the line?

• 1:01:47Speaker 01

  Yeah, so.

• 1:01:48Speaker 02

  So versus buy.

• 1:01:49Speaker 01

  Was that, yeah, the builder's buy. It's a good question and we, I think there's this, like a thing, a general principle that I generally, which is like you build to learn and you buy to keep, or sorry, you buy to learn and you build to keep. Yeah, the other way around, I guess. So you, you buy to learn and you build to keep. And I think that's generally true here where in the early days, when we were still trying to figure out like, you know, get our feet under us a little bit, we were, you know, CoderView's an example, one of them where we spent a lot of time looking at different CoderView platforms. It turns out today, we still use Devon because we've actually still find a lot of value in how the system's been built. So, you know, shout out to the cognition team. But then along the way, we did try out some like SRE tools and, you know, we tried some sort of like off the shelf, maybe more like background agent platforms, decided that

• 1:02:32Speaker 01

  it was just given our infrastructure, given how we had set up our own cloud platform, it just made more sense to kind of build this out internally. Of course, we're not dogmatic, we are open to revisiting them if, if something comes out, but here I felt, I think usually we ended up building in-house when we felt that there was no like mature offering out there. So CoderView is maybe like an interesting example because we did do a whole survey of CoderView tools like maybe over a year ago and at the time.

• 1:02:57Speaker 00

  And I'll shout out like CodeRabbit, right?

• 1:02:59Speaker 01

  Yeah, CodeRabbit and them and you know, there's also like CoderRiot, Graphite, Graphite for a while before they got acquired, like Ellipsis and which like a YSC company before the fact, but there's a bunch, right? I mean, so there were a lot of folks that we did try and they were okay, but I don't think they had like crossed the threshold of like they're immediately useful yet. And so then we kind of like, we sort of like put it aside a little bit. This was, you know, I'm saying like well over a year ago. Then we kind of went back to this and sort of organically, I mean, again, shout out to the, to the Devon team and cognition folks where we tried it again. I think I had sort of like gotten that somebody told me like, hey, you should try this and you know, it seems like it's gotten a lot better. And I've been actually during Silas's talk during the class last year, and it was just good and it was picked up organically like, like.

• 1:03:40Speaker 00

  Purely vibes, right? Like this is, there's no way that you know this.

• 1:03:43Speaker 01

  There was, yeah, this, this was like, I mean, okay, okay, here's what I'll say. When we did that first evaluation, the first time around, where we didn't really pick any vendor and we were like, there's no vendor that we're super happy with, I did do like an analysis where I looked at our PRs, basically did sort of like a, you know, set of pie diagrams, like here's what this is solving, are we, are we happy with the kind of the trade-off of verbocity and how often it's posting with utility? And so there was, it was more statistically driven than just vibes. The second time around, I wasn't going to push hard on it. The first time I was kind of like, hey, we should adopt an AI code tool. Second time I was like, no, I'm not going to do that again. But then organically people were just like, hey, Devon is like good, like can you like add me to this? And so today, people are sort of just like getting organically added. So this is a vendor where, you know, one of the cases where we're still leaning to the vendor, you know, we'll see in the long term what that looks like, but everything

• 1:04:28Speaker 01

  outside of that, we just didn't feel like maturity was there.

• 1:04:30Speaker 00

  Yeah. And it's not painful enough that you would like, we're about to have it right you on?

• 1:04:34Speaker 01

  That's right.

• 1:04:35Speaker 00

  That's interesting.

• 1:04:35Speaker 01

  Yeah.

• 1:04:36Speaker 02

  It's a good, good talk for Devon.

• 1:04:39Speaker 00

  Yeah, I didn't, I didn't even know, you know, for the record, I'm only an advisor, I'm not that involved anymore, but like it's interesting to, to hear about it externally because internally there is a certain amount of, let's call it, the bubble where like on Slack, you're, you're like, you only see your positive things because people only want to circulate positive things, but you have no dog in the fight, like you just choose the best tool.

• 1:05:03Speaker 01

  I mean, if you want to give us like free credit, that's totally, that'd be cool too. Like that, wouldn't, I would say, no one said no to that.

• 1:05:08Speaker 00

  I think Silas, I think Silas will hook you up. Okay, so, so there's that, right? And then, and that informs all your teaching work?

• 1:05:17Speaker 01

  Yes, that's right.

• 1:05:17Speaker 00

  And you know, I think that's, that's really good.

• 1:05:19Speaker 01

  That's right.

• 1:05:19Speaker 00

  And we can bring in more of this as you like, because I think this is, this lived experience is very good.

• 1:05:24Speaker 01

  Yeah.

• 1:05:24Speaker 00

  I'm happy to go back to agent-ready code bases or more like, so week four and five kind of, you know, matching.

• 1:05:33Speaker 01

  Yeah, they blend a little bit, yeah.

• 1:05:34Speaker 00

  Blend a little bit.

• 1:05:35Speaker 01

  That's right.

• 1:05:36Speaker 00

  Whatever you want to, whatever strong beliefs you have introduced in Monaco?

• 1:05:40Speaker 01

  Yeah.

• 1:05:40Speaker 00

  And recommend people to start with.

• 1:05:42Speaker 01

  Yeah, no, for sure. Yeah, week four and five, I would say they, they are a little bit, four is maybe a little bit more like conceptual foundations and ways to do this, because there are a lot of ways to customize agents. And then I would say week five is where I really kind of like lean into, all right, let's think about this and apply this in practice on like a true code base. And so here I will definitely, you know, give a shout out to, to the factory team and, and want to just highlight the value of something like a readiness report. I think that the main piece that I want to reinforce, something like agent-ready code bases is to me like a precursor and a bit of a building block toward everything to come afterwards. When I say everything to come afterwards, I mean, like people always got, you know, for a while, and I think people still are like super hyped on like multi-agents, like paralyzing agents, like tens of agents doing all kinds of things.

• 1:06:27Speaker 01

  It's like you actually need to set a foundation in order to do that well. And the foundation is how reliable is your code base, how like well-structured is your code base, how understandable is it by an agent? And so I think like agent readiness to me is like almost, almost like piece one of the next four to five weeks where maybe until like week eight or nine, where, where I spend more time talking about like broadly, you know, AI software teams and like kind of org level structure, but definitely building up the, the building blocks for, for background agents, the software factory. I think this is where it all kind of gets interesting from, from that perspective.

• 1:06:58Speaker 00

  Yeah. One thing that I observed you talking about from Monaco, but it's not here, is the SRE side. When you close the loop, you actually want to have the agent read your logs for builds and then ideally read your analytics for live production.

• 1:07:15Speaker 01

  That's right, that's right, yeah. And not explicitly, but it's a really good shout out. So I think that, I'm going to weave that in, in like two places. One is in the background agents week and then also in the software factory week. So absolutely right. I do, it's a really good call out, this idea of like trigger slash event-based closing of the loop, which, you know, now is one of the, one of the probably two or three entry points when people talk about agents kicking off tasks in the software factory. That's absolutely right, that'll come in there.

• 1:07:39Speaker 00

  Yeah. It's interesting that people call it trigger or event-based, which I guess that is the actual technical implementation of it.

• 1:07:47Speaker 01

  Yeah.

• 1:07:47Speaker 00

  It just seems too low level almost. But so the trigger is usually a cron job, right? Like let's, let's be real.

• 1:07:55Speaker 01

  Sure, sure, sure.

• 1:07:56Speaker 00

  Event-based is usually a cron job that like surfaces an event and then it's event-based, but it's, it's all crons.

• 1:08:02Speaker 01

  Yeah, yeah, that's right, that's right. Yeah, like webhooks or something like that, but yeah, that's absolutely right. Yes, and, and you know, the technical term, of course, because at the end of the day, yeah, being sort of like a CS person by, by, by trade and yeah, we, we call it those things and so I guess that's a, I fall back on that terminology.

• 1:08:19Speaker 00

  Yeah. And have any AISRE companies made an impression on you?

• 1:08:24Speaker 01

  So, okay, I mean, like I think the Resolve folks are doing well there. The Resolve is doing well. I would say Night Shift is also doing quite, quite a good job there in terms of broadly like background agents. I'm a big fan of kind of like their Greylock-backed, I think they're doing, doing well in this area as well.

• 1:08:41Speaker 01

  Yeah, I would say that where I think SRE is an interesting use case. And this might back to the build versus buy question. I think what's interesting about it is like, where do, when does it become? Like, where does, where does an SRE, an AI SRE really see the most value? It's typically when you have like a tremendous number of like service dependencies and these like massive sort of like systems and tons of Kubernetes and Helm files, like, you know, just like tremendously large Kubernetes infrastructure Helm that's like files kind of strewn everywhere around. I think that's why Resolve has done really well in like the enterprise, right? Where you go and you sell to Ford, you go and sell to like a Walmart and you kind of say, hey, you have, your service all dependencies are crazy. And you see these, these like, you know, these diagrams of how things are set up. And then they kind of go in and they're like, well, we'll try and make sense of all of this.

• 1:09:26Speaker 01

  I think that like at smaller scales, it's usually still manageable. You can kind of roll your own. But, you know, I will say that for the enterprise, I think they've done a good job.

• 1:09:33Speaker 00

  Yeah, for, for people who I have never worked at this scale, because it took me a while to, to, to, to see this. It is on the order of, let's, let's call it tens of thousands of repos, more repos than developers. These are the situations where like you don't know what code is running. Anywhere in the company, because probably the guy who wrote it is no longer here. Right? And like, by the way, it's still low bearing.

• 1:09:56Speaker 01

  Yeah, that's, that's the scariest thing. That's absolutely right. It's like, it's, and then you're, you're the person, you know, you're the person who's on call and you're interacting with a system that goes down where that person still isn't there. And there's like old out of date run books and you're like, what the hell is going on here? It's a scary, scary thing. And yes, you're at the, at the enterprise level, it's like, it's a whole, whole different beast.

• 1:10:16Speaker 00

  Yeah, so like the, the kind of like college hacker, they, you know, one person per repo, they've never, totally, no appreciation. But like, I think, I think if you just give that idea, like you drop them in like a simulation of like, okay, like day one on your job, this happens, what now?

• 1:10:32Speaker 01

  Yeah. Yeah. So, so another shout out I'll give here. So the Resolve folks, actually last year, they, they made an assignment for the class where they effectively created like a sandbox for students to go and then the sandbox was like, I mean, the, within the sandbox, there was like many, many services that were doing their various things. And there was even like bugs in the services within that sandbox. And then as part of the assignment, the students had to like go and actually like use the Resolve agent and kind of like probe and play the, the, the human on call person probing the system, discovering what's happening.

• 1:11:06Speaker 02

  They've got an interesting.

• 1:11:07Speaker 01

  Yeah, it's an interesting website.

• 1:11:09Speaker 00

  I do not.

• 1:11:10Speaker 01

  It's been a while since I've seen this. Yeah, this is very interactive.

• 1:11:14Speaker 00

  Okay, they're, they're being a page of duty now.

• 1:11:16Speaker 01

  Yeah, yeah, that checks out. Yeah, but also this like this, this video I was.

• 1:11:21Speaker 02

  You know, you were shouting out, interesting, interesting, but okay.

• 1:11:26Speaker 00

  Yeah, so, okay, well, you know, we never get a chance to talk about this side of the world except for conversations like this. Any other segments? Because I think AI SRE is maybe too broad or even too narrow, depending on how you look at it, right? Like too broad, meaning while there's several subdivisions of AI SRE, that's interesting. Or too narrow being like, well, actually that's not even the biggest problem that we have. So take it however you like.

• 1:11:48Speaker 01

  Yeah, you mean like specifically for like Monaco or just like broadly for like organizations thinking about this?

• 1:11:53Speaker 00

  Broadly, because that is our audience.

• 1:11:56Speaker 01

  Yeah, I, I think that it, again, it's a, it's a function of how we just scale, right? And I think that for, for the enterprise, they, there are a number of issues that aren't always, I almost feel like the answer is not even always just more agents. And, and since we're still on the week five agent ready code bases, like when I talk to companies of like a certain scale, the, the things that are holding them back are really more just like low level like DevOps, C I C D stuff, right? They're, they're like, their, their merge queues are getting backed up. They, it takes too long to like stand up an app. There's just like actually more foundational things that have almost nothing to do with, that like throwing in one more SRE bot or Allah, you know, a bot wouldn't be the answer to the problem because there's like more systemic things that they need to figure out. And so.

• 1:12:37Speaker 00

  So that brings us to week six.

• 1:12:39Speaker 01

  Yeah, that's exactly right. So that's where we get into, yeah, easy, exactly right. Yeah, there we go. To the merge queue and beyond, um, yeah, this is a, yeah, a love letter of sorts to, to modern Gentik code review. Yeah, I think that the, the biggest thing that maybe happened between last year and this year was just, it suddenly became very clear that code review is a bottleneck, especially once you, people got really AI pilled and, and people, there's the velocity on generation became so high that, you know, you could in theory like, put up 30 plus PRs a day and then what? Like what's, at what point do you decide that you either have to institute yourself and put yourself in the middle as some way to, to gate and review things in the traditional way? But then you're, you're literally the constraint, you're the choke point of the velocity. And that's not good, right?

• 1:13:24Speaker 01

  And then what was the point of generating at scale? Or you say, well, how do we maybe get the best of both worlds? Where we're instituting ourselves enough that we have trust in what's getting merged, but we're also able to kind of like merge at scale, but we're also not kind of like free, freewheeling it here and anything goes. So a lot of, I think what I want to touch on here is just that, that middle ground, right? Which is like, there are, there is a class of problems that if you've put in thoughtful design into your agent readiness, you've put in thoughtful design around your verifiability, there's a class of problems that you can feel comfortable, you know, some percentage of the time, essentially like auto approving. And what has to be true from a kind of a design perspective, from like an architecture perspective, to just auto approve? And when do you have to insert yourself and say, actually, here, I still want to review this carefully?

• 1:14:08Speaker 01

  And even if you do review it carefully, are you still doing line by line review or are you doing something else, right? So, yeah, some of this.

• 1:14:14Speaker 00

  We, one of our hottest topics of this year, we have a guest writer program for Lean Space. This is from Ankit from Aviator. And he has a whole thing on killing the code review because I, you know, that, that is like maybe the final boss of, of, that's a bottleneck. He has a few gates by which he does this. I don't know if you've talked to Ankit or.

• 1:14:34Speaker 01

  Yeah, no, I've, I've read this post, yeah. I haven't spoken to him personally, but I've, I've definitely.

• 1:14:38Speaker 00

  Mostly right? Any criticisms?

• 1:14:40Speaker 01

  No, I think this is right. I mean, like broadly, right? I guess you mentioned one thing like the last boss, maybe not, I, I think there's two choke points. There's the review and there's probably like the intent or like kind of the thing at the top, right? Like this is.

• 1:14:50Speaker 00

  And there's some, there's some stats as well that, that I like to pull up. You know, everyone's stats is different.

• 1:14:54Speaker 01

  For sure, for sure.

• 1:14:54Speaker 00

  But like clearly more people are merging. Clearly the number of reviews are not increasing in line with, with merge.

• 1:15:02Speaker 01

  Yeah, that's right. That's right. And if you look at any modern sort of company that's, that's, that's really invested in this and thought about this carefully, and some people have already posted about this and, and you know, I'm sure we'll do posts about this as well. But I think recently like the Amplitudes and I think CloudFlare has, has done a good one. I think Augment has also done a good one where they talk about the design of their AI code review system. These are a bunch of really good, if you want to share some of those links, I think they're fantastically written posts. Yeah, so shout out to, I actually, shout out to CloudFlare because I think like very like, kind of like, you know, in a sense like legacy company or like, you know, bigger company. But I think they've done a fantastic job in terms of just their, their content around the AI, the AI S D L C. So I think they have a good post there. But T L D R, yes, I think this is true.

• 1:15:46Speaker 01

  I think that there's, we're still finding that sweet spot. What is auto approvable? What is not really auto approvable? And if it is not auto approvable, is even in that world of something that has to be somewhat human reviewed? What do we review? Are we still doing the line by line? Are we doing some, some combination of the spec? And, you know, I think this, yeah, again, it's a whole separate question there as well, but.

• 1:16:09Speaker 00

  Are people keeping, are you guys keeping the spec up to date at all times?

• 1:16:15Speaker 01

  The spec.

• 1:16:15Speaker 00

  Because no way, right?

• 1:16:16Speaker 01

  Yeah, no, well, I mean, okay, in a few different places, right? So like there's a spec that as it exists in like a linear issue. And there's a spec as it exists in like the context of a PR description, right? So you can like, you can have your PR description be the place for you to like describe more or less the entirety of the implementation in a more condensed format. And as long as that's the most, that's the most up to date thing representing this point in time of the, of the code base, I think that's sufficient actually. So we're thoughtful about like our PR descriptions. We make them be not just like, you know, one paragraph. It's more like, it is effectively like a spec of what has happened in the PR. And there's even like kind of line file level pointers to things that where things have changed. So you not only get the high level of what's happened, but even the kind of the, the construction of the files and sort of the service level boundaries that you're touching. So, yeah, I think that's where we, we've invested quite a bit.

• 1:17:00Speaker 00

  So I push back on that because to me, of course, it's, you know, bare minimum, it should be internally consistent with the PR spec.

• 1:17:07Speaker 01

  Sure.

• 1:17:08Speaker 00

  Yeah. The, no, what I meant is the spec for the feature, the code base, because that is the summation of all PRs. That is like, if your, if your PR spec got it wrong or missed something, it's going to be the delta between what the whole code base or the whole history of the code base is supposed to do versus what your PR is trying to do. So, so I, I mean, I don't expect you to have an answer. Because like I, I really stress about this myself. I try to instruct models to do it. They're bad at it. That's it. So the humans are the spec gods of like, well, five weeks ago, I did this other thing and now you're reversing it.

• 1:17:42Speaker 01

  Have you seen the post from Basis? Like, they had a tweet, tweet little thing, tweet article. I thought they did a great job talking about like different sort of file artifacts. They tracked via Git and some of them are like the specs, the intent of what is to be. And the other is like what they call like the canonical state of the code, which is like a little bit more maybe what you're describing, which is implicit knowledge, implicit intent that also has to be codified somewhere, right? So that part, that's all version controlled. And I think they found that to work for them. So that might be another thing to share as well.

• 1:18:12Speaker 00

  Yeah, Veebo was trying to pull that. This is Basis, the accounting company.

• 1:18:15Speaker 01

  Exactly right, yeah.

• 1:18:16Speaker 00

  We're having Mitchell at my New York conference soon. Talk to me about that. I like that they are coming out with this. My solution for this is, I don't know if like how ready I am to talk about it because I may never ship it, is something that is replacing Git in order to capture all of this. So I call it a program database, where it captures all traces and all, and then it reifies specs to something that we can ignore and keep. And it also controls the skills that are attached to it as well.

• 1:18:49Speaker 01

  That's cool.

• 1:18:49Speaker 00

  Because I think like basically what Git is only focused on code. And then you just need a source of truth that is not only everything else outside the code, including the implicit, what did you call it?

• 1:19:00Speaker 01

  Yeah, like implicit assumptions or like the constraints, the intent, the yeah.

• 1:19:03Speaker 00

  Other stuff.

• 1:19:03Speaker 01

  Yeah.

• 1:19:04Speaker 00

  But also multi, like highly concurrent.

• 1:19:07Speaker 01

  Right.

• 1:19:08Speaker 00

  Because Git by design introduces mutex everywhere, which sucks. So you need like super concurrency.

• 1:19:16Speaker 01

  Have you seen Git Butler, is it?

• 1:19:19Speaker 00

  Yes, yes.

• 1:19:20Speaker 01

  The guy who used to be CTO.

• 1:19:22Speaker 00

  Ian, I think is his name?

• 1:19:23Speaker 01

  I forget his name, but yeah.

• 1:19:24Speaker 00

  Scott, Scott. So yeah, so I think there's a lot of people going after this. And I think there's the question of like, really, what is the ideal representation of tasks?

• 1:19:36Speaker 01

  Yeah.

• 1:19:37Speaker 00

  And I think the winner, probably not me, Linear.

• 1:19:40Speaker 01

  Really?

• 1:19:41Speaker 00

  Because like that's where the tasks are.

• 1:19:44Speaker 01

  Yeah, yeah.

• 1:19:45Speaker 00

  And like clearly, they have decided to become a coding agent company.

• 1:19:48Speaker 01

  Right.

• 1:19:49Speaker 00

  And well, you know, that they have all the source of truth there as far as you, as long as you buy into Linear.

• 1:19:55Speaker 01

  Yeah, yeah, that is true. That is true. I wonder, I have like this somewhat like kind of held belief that I think a lot of companies are going to evolve into some version of a coding agent company, which is any company that has like an interaction with B2B software, where you're collecting something about the user or some like product insight, I think you can also close the loop. Because ultimately, like what a coding agent is, the purpose of coding agent is to act on some adventure thing that's happening, where users involved or customers involved, and then use that to like fix ideally some aspect of the product, right? I think what Linear is doing, I imagine the same thing like Amplitude could do as well, right? Because they're also getting insights about user analytics, user problems. I imagine even like you want to really stretch it, you could even have like a Decagon or Sierra do something similar where, you know, here's what's broken, go close the loop.

• 1:20:37Speaker 00

  Yep, all right. So set all those things up for agent-ready codebase and set up code review. Security is week seven, it's not week one?

• 1:20:45Speaker 02

  Students.

• 1:20:45Speaker 00

  What do you mean, security is not important? I make fun of it because I always say like, you know, at Amazon, job zero is security. And they always say this.

• 1:20:53Speaker 01

  Yes.

• 1:20:54Speaker 00

  Anyway, yeah, security review at the end.

• 1:20:56Speaker 01

  I was at Amazon and I don't remember that, but I believe you.

• 1:20:59Speaker 00

  This is a thing Andy Jesse says.

• 1:21:00Speaker 01

  Okay, okay, okay, I see, I see. And that was, I guess he, I was there partly during Jeff's time and then partly when this transitioning to Andy, so. But yes, I think this is a thing, a topic last year as well, and it's also won this year. And I think that there, Isaac also spoke last year, I find that he's an incredibly thoughtful thinker, thoughtful thinker, I was about to, you know, thoughtful, thoughtful leader in sort of this space. I think Semgrop is an interesting company because they've been around even before sort of agents, right? I mean, the OSS repo was, existed well before like kind of traditional SaaS kind of defense mechanisms. But I think they've done a good job of pivoting and, or not pivoting, augmenting their offering into this space. You know, I think it's, it's an interesting time, right?

• 1:21:45Speaker 01

  There's just a lot going on where these attack vectors are now kind of come to the surface. There's a lot more slop and AI-generated code. And I think maybe one of the core insights that I want to drive here is just the idea of like dependency, kind of dependency tracking and supply management, basically, like supply chain, right? You have to be careful about like your dependencies and pin things appropriately and don't just arbitrarily hit npm install and expect the chain to go.

• 1:22:10Speaker 00

  Yeah, so potentially, the most secure codebase is one with no dependencies.

• 1:22:14Speaker 01

  Yes, that's correct.

• 1:22:15Speaker 00

  You look at the dependency, you're like, cool implementation, bro, but I'm just going to make my own.

• 1:22:21Speaker 01

  That's absolutely right. That's absolutely right. And that's an interesting insight because it's sort of to the limit. And, you know, I think it's probably easier said than done, but if you believe that with code now being commoditized, code being easy and cheap, why not just build everything in-house, right? Like that, like, of course, and that would be the safest thing because you've seen everything, you control everything instead of, you know, depending on Axios or whatever. But, you know, we'll see how many people actually act on that. That is the safest thing, I agree with you.

• 1:22:46Speaker 00

  No, so that statement is dangerous because then you would just make your mistakes instead of making, relying on other people to discover mistakes. The whole point of open source is that, you know, what is it, like sunlight is the best disinfectant? It's like Linus Torvalds thing. It's like, well, yeah, you will be very slow to discover your own custom software dependencies because you only have your users and they won't report it to you. But if this thing has 100,000 users worldwide, someone will find something. And then you can depend on it more, right? Yeah, this is, this is, there's no, basically, I don't, I'm trying to be a voice of reason. And you cannot be absolute, listen both. And you, I kind of trapped you there. I talked, I talked you into the talk track of vendor everything.

• 1:23:31Speaker 01

  Right, right.

• 1:23:32Speaker 00

  And there is a bug there.

• 1:23:33Speaker 01

  Yeah, yeah, yeah, it's an interesting one. I wonder how much that, how much is that, I buy some aspect of that, but how much of that is impacted by the fact that everything, I have this belief that like most OSS repos, if not all of them, will just become software factories that are effectively agent-run in the future. What happens then where like, I mean, even, like even if you look at Warp, right? Like Warp, which is a modern coding agent, of course, they're also, you know, selling a software factory product, but, but they are running their Warp, open source Warp terminal as a software factory. And that almost suggests that there is a part where yes, teams are still involved, but increasingly less so. Certainly not every single person is reviewing every single thing that comes into maybe the same exact level that we did in the past. Does that make this codebase more or less vulnerable to security? It's a bit hard to say, right?

• 1:24:19Speaker 01

  I think that's just where there's a little bit of nuance that I think we're still, and even the folks that I know that work in OSS are trying to figure out exactly what that stable point is.

• 1:24:26Speaker 00

  Yeah, there's a couple of related arguments and then I'm happy to keep going. Like, security is like a non-stop, like huge, huge, huge surface area thing. There's one argument from Malta of Versell, who's like, okay, guys, let's take every single like high depend, highly dependent on software that's written in C, C++ port it to Rust. And it's like, we'll be faster, more secure. And like, it won't probably take like $6 million, but you get it done. And then there's another argument for, this is also linked to pacing the frontier from the open source, from the model labs, who are like, okay, let's slow down so that we, or like, we'll do daybreak, we'll do what's Glasswing, we'll make sure everyone's patched.

• 1:25:10Speaker 00

  And then we can let loose on rampant acceleration. So I think like there's an inherent belief that the models will fix all security. And I think it's just unproven to me because what if they introduce new bugs and they're just different bugs?

• 1:25:26Speaker 01

  That's right, that's right.

• 1:25:28Speaker 00

  So, I mean, well, I'm looking for a pushback. I'm looking for like, well, no, you're an idiot, like obviously a model's a way better at this.

• 1:25:37Speaker 01

  Yeah. I guess like, that's almost like in the limit, there's, the question is like, is a model, do we believe a model will create perfect software, right? If a model's creating perfect software, then you're done.

• 1:25:48Speaker 00

  Is there such a thing?

• 1:25:48Speaker 01

  Then that's done, right? Then we have, then we're done here at all. Like, then we're just going to go home and chill on the beach for the rest of our lives.

• 1:25:55Speaker 02

  I think there's also the question of, is it good enough to first give it for security? So let people patch, then just delay the release and do it after? Or is there a problem of, oh, no, it'll be much better at finding vulnerabilities? You can full send, test it for longer, even if you've patched or like, how long, right? You can give it for three months, someone can try to exploit it for six months, the problem's not solved, right?

• 1:26:19Speaker 01

  Right, right, yeah.

• 1:26:20Speaker 00

  Yeah, my whole problem with, and again, like this is a whole rant that we don't need to go into, but like, you know, you name 100 bless organizations that like, okay, these are like priority, they have access to a cyber model that you've just created 100 attack vectors for me to go compromise. And like, by the way, they're pretty easy to compromise.

• 1:26:40Speaker 01

  Yeah, yeah.

• 1:26:41Speaker 00

  I don't know. Anyway, above our pay grade.

• 1:26:46Speaker 02

  It's always fun to get juicy in, you know, security, like.

• 1:26:50Speaker 00

  I mean, security is right now very juicy, right? Like, I think actually the security folks love it because they usually don't get that much attention. And now they have like a ton of attention.

• 1:26:57Speaker 01

  Is that true? Like, I mean, that they don't get attention? I find that, um, I find that at least from like a VC perspective, you'd love, VCs love investing in security companies. And have like pretty good, like a Wiz or something like has done very, very well.

• 1:27:09Speaker 00

  Wiz is, Wiz, I'll give, Wiz and Socket, I literally, I had this conversation with VC two days ago.

• 1:27:14Speaker 01

  Yeah.

• 1:27:15Speaker 00

  Apparently Wiz and Socket are the only security companies in the last five years that have worked.

• 1:27:19Speaker 01

  What about like abnormal?

• 1:27:21Speaker 00

  I don't know. It worked for a VC point of view of.

• 1:27:24Speaker 01

  I see from like, from like an economics perspective.

• 1:27:26Speaker 00

  Yeah.

• 1:27:26Speaker 01

  Hmm, yeah. Interesting.

• 1:27:27Speaker 00

  Yeah. Not, not, not long for me to say.

• 1:27:30Speaker 01

  Right.

• 1:27:30Speaker 00

  Obviously many like great players in, in that field, but it's, it's a tough, tough industry, I think. And so basically like the, the best, I think it's not controversial. Best time for any security company is when a high profile incident just happened.

• 1:27:43Speaker 01

  Yeah.

• 1:27:44Speaker 00

  Right? Obviously, right? Then obviously you're gonna sell a bunch of security.

• 1:27:47Speaker 01

  Totally.

• 1:27:48Speaker 00

  Yeah. So, so I think, I think that's a fair thing. Like people, you, people should be aware, people should know the, the basic mitigations. I do think like code scanning is important, but like I, I just wanna give people the, the idea that like, oh, models introduce bugs that you wouldn't make as a human. And like, if you're not checking those bugs, then like.

• 1:28:08Speaker 02

  It's not even that they make bugs, it's that you have less awareness of what's being written, right? Like when.

• 1:28:14Speaker 00

  Yeah, cause you know, you're reviewing everything.

• 1:28:16Speaker 02

  Yeah, yeah, forget bugs. Models are pretty good. They'll have, on average, I would say, less bugs. The problem is also that you don't even know what you're putting out.

• 1:28:27Speaker 01

  Yeah.

• 1:28:27Speaker 02

  There's a lot of that.

• 1:28:28Speaker 01

  Yeah. And that's, yeah, I think we're to the top of mitigations and like the, and there's sort of a combination of security and, and security as part of code review as well. Right? I mean, I think this is also like doing those checks, I think is important.

• 1:28:38Speaker 00

  Yeah, so I think, um, maybe I would say one more thing here, uh, or is check your, check, check my intuition on, on this thing.

• 1:28:44Speaker 01

  Yeah.

• 1:28:45Speaker 00

  Is that, uh, not all parts of the code base are created equal. The parts of the code base where security is very important is data storage.

• 1:28:54Speaker 01

  Hmm, yeah.

• 1:28:54Speaker 00

  And, uh, billing. So let's call it like data billing, uh, off. And off is obviously related to all those things. And those are basically don't vibe code. And then everything else go, go nuts.

• 1:29:08Speaker 01

  Yeah.

• 1:29:08Speaker 00

  Right?

• 1:29:08Speaker 01

  I think that's absolutely right.

• 1:29:09Speaker 00

  Uh, and, and it's really funny how like, okay, well, if you don't know what the difference between off and an off Z is, please learn it. And then, you know, if you, if you don't know, if you have never looked at AWS IM, which you have, please learn it. Like, like this, like these are basic, like conceptual things to understand that you can prompt for. Now, once you have the language, you can prompt for it. You don't have the language, you're fucked.

• 1:29:32Speaker 01

  Totally, totally. Yeah. Yeah, to an extent, I would even extend that to like just general, like certain, certain like cloud platform of things, right? I mean, there was the, which company got, had that huge incident, huge incident where like, it was just like a DNS issue that brought down, was it? Do you remember this? Was it? Yeah.

• 1:29:46Speaker 00

  DNS issues, you're, you're gonna need to narrow it down, man.

• 1:29:48Speaker 01

  Yeah, I know, I know. Particularly prominent one, was it?

• 1:29:52Speaker 02

  There have also been many prominent DNS issues.

• 1:29:57Speaker 01

  We'll list them off.

• 1:29:59Speaker 00

  It's always DNS, it's always DNS. Yeah, so DNS is more of like a reliability uptime.

• 1:30:03Speaker 01

  Sure.

• 1:30:03Speaker 00

  Uh, you know, security, I think people are like, yeah, did you lose money? Did you leak customer information?

• 1:30:08Speaker 01

  Yeah, yeah, yeah, yeah, yeah. That, I would, yeah, probably put that above that for sure. Only DNS because of uptime, yeah, general uptime.

• 1:30:13Speaker 00

  Yeah.

• 1:30:14Speaker 01

  So, uh.

• 1:30:15Speaker 00

  Background agents. Uh, now you got your buddy from CloudFlare here. But, I mean, your guests aren't necessarily correlated with that week, right?

• 1:30:21Speaker 01

  Yeah, no, I mean, like, uh, the nine one was a little bit of a scheduling thing where I've thrown in, you know, Elad and, and I'm, I guess. I'm glad it's, it's definitely week nine, Elad, I was trying to get him at the end, but I think it was just timing wise, couldn't work it out.

• 1:30:32Speaker 00

  Yeah.

• 1:30:33Speaker 01

  Um, so yeah, it's, it's a bit of a weird one where I don't, I guess I'm not lecturing at all week nine, but, but we're coming back to those topics in week 10. So it's a little bit staggered in a, in a bizarre way there. But, yeah, week eight, um, Rajesh, who has been leading a lot of like the infra stuff, uh, at, uh, kind of cloud infra, uh, DevX and, and beyond at CloudFlare, which as I earlier said, was, I think they're doing a great job here. And definitely their adoption for a company of their size and a public company of their size is, is fantastic. And then their content, I think, is also really good. I mean, this is fundamentally about all the ways you can use like asynchronous background agents. Um, I think I'm also gonna spend quite a bit of time talking about, you know, the Stripe minions, the Spotify, I forget the name of their version of this. Um, the Ramp Inspects, the, you know, things of that nature that are, I think are all kind of showing what does it look like when you scale this out?

• 1:31:19Speaker 02

  Cloud tag.

• 1:31:19Speaker 00

  Instinct?

• 1:31:21Speaker 01

  Yeah, yes.

• 1:31:22Speaker 02

  Yeah, Muse.

• 1:31:24Speaker 01

  Yeah, I mean, I guess less on consumer, more on like, you know, software teams using them at scale.

• 1:31:31Speaker 00

  They're all coding agents under the hood.

• 1:31:32Speaker 02

  Everything is a coding.

• 1:31:34Speaker 00

  Yeah, it's like Grokbot is one of those interesting things where like it's very, very much presented as consumer E.

• 1:31:41Speaker 00

  Uh, it's, it's Grok, it's, you know, it's, it's cursor under the hood.

• 1:31:43Speaker 01

  Yeah, yeah, yeah.

• 1:31:44Speaker 00

  It's a cursor team, it's.

• 1:31:45Speaker 01

  No, no, for sure. Like, so I, I, I've seen the like, you know, cursor working in the, like the dot, dot, dots, but, but, but they are not, they don't have like a code base integration, right?

• 1:31:52Speaker 00

  They do.

• 1:31:53Speaker 01

  Okay, I haven't used that. I've only been using it for like open table reservations.

• 1:31:56Speaker 00

  Exactly.

• 1:31:57Speaker 01

  Okay.

• 1:31:58Speaker 00

  Uh, which, I think that's the point of what they're trying to do.

• 1:32:01Speaker 01

  I see.

• 1:32:02Speaker 00

  Yeah.

• 1:32:02Speaker 01

  As in like.

• 1:32:03Speaker 00

  They're, they're trying to be like, uh, well, you can use it for consumer stuff, but here's the extra power if you know how to use it well. Uh, because, because it's actually cursor under the hood.

• 1:32:12Speaker 02

  I think this goes to all coding agents, right? Like Claude code, Claude design, uh, people using Codex, not knowing they're using Codex, right? A random code interpreter window pops up and you're just asking GPT a question. At some level, there's coding agent under the hood, right?

• 1:32:29Speaker 00

  Yeah, so, so I, I'll give you another sort of practical example. Uh, so I, I run all my SEO and AEO inside of Codex and it has a code base. Even though like a traditional marketer would just check a address or something.

• 1:32:41Speaker 01

  Yeah.

• 1:32:41Speaker 00

  Uh, because I write all these scripts for doing all the, all sorts of crazy stuff.

• 1:32:44Speaker 01

  Sure.

• 1:32:44Speaker 00

  And, and also I need to store data and also I need to visualize data. And like pretty soon you go like, as long as you're doing anything on an ongoing basis that has like maybe a lot of steps or a lot of data, you need a code base.

• 1:32:59Speaker 01

  Hmm, that is an interesting take.

• 1:33:01Speaker 02

  I think models also like building code bases.

• 1:33:04Speaker 00

  Not, but that could be a temporary thing, right?

• 1:33:06Speaker 01

  I guess maybe my, my point is more on the interface, right? Like, will, are you going so, like, are you gonna go so far to say that a Grokbot will have a place where like the power user wheel will just see the code under the hood? Is that, or is it?

• 1:33:17Speaker 02

  Well, there is a computer.

• 1:33:18Speaker 00

  Maybe they'll just jump to cursorize, I guess, is that what you're saying?

• 1:33:21Speaker 01

  Well, exactly, because I'm like, otherwise then, then you basically cannibalize cursor. Like, if, if in a limit where you're saying is Grokbot is the cursor, then you don't even need like a cursor if all under the hood is just becoming.

• 1:33:30Speaker 00

  It's the same team, so they.

• 1:33:31Speaker 01

  Maybe.

• 1:33:31Speaker 00

  Right now they don't care.

• 1:33:32Speaker 02

  Right, it's all Grokput.

• 1:33:34Speaker 00

  Yeah, yeah, yeah.

• 1:33:35Speaker 01

  I, I, I buy that, I mean, I do accept that even things that seem outwardly just like non-technical use cases under the hood are technical, can be like written and designed and implemented in code. Like that, that much is absolutely true, right? The question is just more like, is a developer, is that their single point of entry?

• 1:33:52Speaker 02

  I think it's not even developer, right? Like you think of a regular person doing marketing that is using Claude code or just Claude in the app now is working in a code base that they would never know, never touch. But there's a code base manager.

• 1:34:05Speaker 01

  Hmm, yeah.

• 1:34:06Speaker 02

  I think Cloud Tag is a good one for, uh, background agents. It's a big way Anthropic works and, you know, they make it, you should see how they work.

• 1:34:15Speaker 01

  Yeah. To shout out.

• 1:34:17Speaker 00

  Uh, okay, AI native team, uh, we talked a lot about these topics of like gateway, uh, we just put a routing episode out today and MCP portal. I would say MCP portal is like the least established.

• 1:34:28Speaker 02

  On MCPs.

• 1:34:28Speaker 00

  And I'm bullish on MCP, uh, it's, it's not bullish or bearish, it's just is. It is a part of the landscape now, right?

• 1:34:35Speaker 01

  Yeah.

• 1:34:35Speaker 00

  Um.

• 1:34:36Speaker 02

  That's a, that's a statement. There was a whole MCP is so back, MCP is dead. People, people like extreme.

• 1:34:42Speaker 00

  MCP portals is an unproven business. Uh, I, I thought about starting one. Uh, I actually pitched a guy in topic on, on potentially starting one. And then I lost interest. I lose interest in a lot, a lot of things. But like, what, what's, what's in there?

• 1:34:55Speaker 01

  Uh, so, so what I'm thinking here is if this is, um, if this is the week, does, uh, more kind of geared toward scaling this out to like what happens when you take some principles? And a lot of this was informed by like the conversation I had with folks that were, you know, leading AI native teams, AI native teams at, uh, at respective companies. And so here, uh, I think that something like MCP portal really becomes interesting once you're at a certain scale, where you effectively have to have a single point of entry to do things like governance, security, auditing, cost optimization. Uh, and so, you know, I, I'd be curious to hear what you see as the future or maybe lack thereof of something like this. But it to me is, and if you believe in like an open router or, you know, someplace, like a single point of entry for, for many things to happen and connect to

• 1:35:38Speaker 01

  and be effectively, you know, governed, I feel like there's a world where this also can easily exist.

• 1:35:43Speaker 00

  Yeah. My, my only tweak on this is that this folds into the LM gateway. So they, they are the exact same platform.

• 1:35:49Speaker 01

  That I will give you, that I will give you.

• 1:35:50Speaker 00

  You know?

• 1:35:51Speaker 01

  I can, I can accept that. That actually can, can agree with, yeah.

• 1:35:54Speaker 00

  Uh, and, and then, by the way, it also folds into the other thing of, um, what did I say? Was, was, uh, off Z, you know, permissioning and all these things and billing. So like, uh, to the limit, all infra converges to the same exact problems where they have the box in the middle and then you have the box in two sets. Uh, that, you know, God box will solve everything, right? And it's true only if you take, uh, if you understand the job of the God box. And so separating MCP portals and LM gateways means you don't fully accept that they are, uh, you know, the, the same layer of problem. Uh, in a very similar way, I also find that people don't understand database querying, uh, to have a lot of the similar things where they will separate filtering from ranking

• 1:36:37Speaker 00

  from, uh, or, or just like general recommendation, they're the same problem.

• 1:36:41Speaker 01

  Hmm, hmm, hmm.

• 1:36:42Speaker 00

  Uh, and so like then you just come, because like when you take all these things, you combine them in the same layer, then you can sort of cross depend on, on things in useful ways. If there are different layers, then they cannot.

• 1:36:52Speaker 01

  Uh, yeah. I, I actually will not disagree with this. I think, um, maybe the, yeah, so it's slightly pedagogical. I think that the, this breaking up of like separate terms versus one, I think I, it is absolutely correct that the underlying architecture is the, the God box. Where one thing goes in and there's a bunch of other stuff that is some third party, some internal, some first party that all kind of exists, uh, on the other side of the God box. So yes, I agree with this, actually.

• 1:37:17Speaker 00

  I, I have a, I have a, I have a spicy, uh, spicy tweet, I, oh God, where, where is the God box tweet? Oh yeah, so this is the M, M, MXN to M plus N.

• 1:37:28Speaker 02

  Let me bring it up.

• 1:37:29Speaker 00

  Architecture. Uh, so anyone working in infra needs to understand, uh, congratulations, you now have a job in infrastructure or you want to be an infra engineer. You need to understand that your entire job is making God boxes and then taking them apart.

• 1:37:43Speaker 00

  So either your problem is either you don't have enough, you don't have a good gut box, or your problem is someone else's gut box you don't like, so now you have to take it apart and make your own.

• 1:37:51Speaker 01

  That's funny. That's very funny. I like that. That's very funny. Into the heat death of the universe. Wow. That's funny.

• 1:37:59Speaker 00

  Which is, this is a restatement of like, there's only two ways to make software, which is you bundle them, you unbundle them.

• 1:38:05Speaker 01

  Right, right.

• 1:38:06Speaker 00

  But like, what are you bundling? It is the things that go together, hopefully. And then like, maybe over time, people's opinions shift as what goes together, what should not go together.

• 1:38:15Speaker 01

  Right, right.

• 1:38:16Speaker 00

  So like, you know, there is that, that whole thing, you know, maybe Monaco should be more of a point solution. Maybe Monaco should own everything, right? And there's the question of, like, in some ways, you're bundling for the kind of startup segment that you serve. In some ways, you're unbundling from Salesforce, because Salesforce, and then that's how you win market share from Salesforce. You unbundle from Salesforce.

• 1:38:39Speaker 01

  That's interesting. Yeah, that's an even broader statement. I mean, connecting to broadly all of vertical AI. And now like bundling unbundling, I mean, that's like the whole Craigslist and Airbnb, you know, the Tinder and all those are separate.

• 1:38:52Speaker 00

  Tinder.

• 1:38:53Speaker 01

  Well, you know, like personals, basically.

• 1:38:55Speaker 00

  Yeah, yeah, yeah, it is. Yeah, I thought about it that way. But yes, you know, one of those things, so we just shipped the open router pod today, right? And open router, huge success. Lots of people tried to build gateways. None of them had success with open router. Open router only builds gateways. So there's an example of like this, they disagree with me that the router layer should also do ops. Because Arise is doing ops and observability and what have you. And they, and like the MCP gateway, they don't have that. And they just, they only handle model routing. And so like, I, like, I love being wrong like that, because they're like, we are the point solution. And we only do this one thing really, really well.

• 1:39:37Speaker 00

  And that's our, that's our engineering focus, which is good. It's also the marketing focus, which is nice.

• 1:39:43Speaker 01

  I think the, um, this becomes, I think the, what you've described of like the one, one stop shop for all the things is ideologically what makes sense. But I think what ends up happening in practice, and you know, part of it is probably also marketing, is that, especially when you look at the enterprise, is like, this is broadly under infra, like broadly under platform, but then there's separate teams owning different parts of the stack. And I think part of that consolidation at the, at the scale of an enterprise, if we're going to hit MCP portal and the gateway, someone has to come together and say like, now we're, there's a single endpoint that you're hitting for both of these things. And that fundamentally is like a hard thing to, you know, get everyone on board with.

• 1:40:17Speaker 00

  Yeah, yeah.

• 1:40:18Speaker 01

  So oftentimes, like more like political and even like people problem, rather than like a, from a software, pure software design perspective, this makes it out of sense.

• 1:40:25Speaker 00

  Yeah, yeah.

• 1:40:25Speaker 01

  But that's not it, right? Like, software systems sometimes are brought like by people problems more than, than pure software, right?

• 1:40:30Speaker 00

  The way, my framing of this is typically, you know how you ship your org chart? This is, well, I forget what the name of this principle is. Conway's law? You ship your org chart?

• 1:40:40Speaker 01

  Right.

• 1:40:40Speaker 00

  You buy your org chart. Like, like, who, like, when you start a startup or you make a tool, who in the company is your, are you best friends for? And if it's for everyone, it's for no one.

• 1:40:52Speaker 01

  That's interesting. That is interesting.

• 1:40:54Speaker 00

  So like, a conscious part of latent space in AIE is like, okay, there's this category of software engineer that really, really, really cares a lot about AI. Is it going to make it their day job? Let's only make it for them.

• 1:41:07Speaker 01

  Right, right, right, right, right. Not a bundling of engineering, never going to be the dev everyone.

• 1:41:13Speaker 00

  Yeah, look, we talk about databases, talk about security, but like, there's an AI angle to it. We'll see.

• 1:41:21Speaker 01

  That makes sense.

• 1:41:23Speaker 01

  One thing I'll just like shout out as well, related to the point is, I think, role of the future that is increasingly important, platform engineer, I think this is like a huge one.

• 1:41:33Speaker 00

  You know, okay, more lore. Temporal, my job before all this, I would have, yeah, so we wanted to own platform engineer. If I hadn't started AI, I would have done platform engineer. Because that is the role that decides, that decides all of this mission critical stuff. And then the kids over in applications can do whatever they want on top of our platform. But, but you know, the big boys do platform.

• 1:41:55Speaker 01

  Yeah, and I think in the modern day, they own, they own cloud, they own DevX, they own security, SecOps. So I think it's all the things.

• 1:42:02Speaker 00

  Like, everything where you can actually like lose the company or like, or make or break, like, yeah, platform guys. And then product guys can do, you know, adjust the CSS.

• 1:42:14Speaker 01

  That's saying, I mean, that's, you're starting to fight here. I mean, I'm not the CSS adjuster, but, but I, I appreciate everyone.

• 1:42:22Speaker 02

  I mean, it's a fun time. What's, what's the future? I feel like all the guys can adjust all the things.

• 1:42:27Speaker 01

  Yeah, yeah, yeah. So there's, there's a few aspects here. Okay, so there's a little bit of spillover from, from previous week. So yeah, coding agents, big teams, some of the concepts, but then, you know, we'll touch on the software factory. I want people to really start thinking about this, this world of, of this machine that builds the machine and where, if at all, are people still in the loop there? You know, today, I think the answer is some, some combination of intent and kind of review, or maybe the places, like, sort of intent where you're defining the specs and what needs to be built, and review where you're, you know, doing some amount of your own checking of the system. And of course, this also, this is, I think, that bottleneck is going to be loosened up before even the intent piece does. But for now, those are the two choke points, I think, in a fully automated system.

• 1:43:08Speaker 01

  So yeah, I just want to get people exposed to that. For one of the OSS partners, actually Warp, they're doing this thing as part of their contribution to the OSS program, where they are going to essentially make a version of Warp that is running the factory, and students will just be like building the factory, contributing to the factory, and keep it running.

• 1:43:24Speaker 00

  Like work in our factory.

• 1:43:25Speaker 01

  Work in our factory, exactly. And so that's going to be one of the ways. And then they're gonna, yeah, find things that they find that are interesting there and pull it back in.

• 1:43:30Speaker 00

  This is one of those things where like, I'm not sure, like, obviously, Factory itself is very, very keen on this.

• 1:43:35Speaker 01

  Yeah.

• 1:43:36Speaker 00

  I feel like everyone wants to build their own.

• 1:43:39Speaker 01

  Yeah, I think that's, that is, it is a good question. And I think they're, you know, of course entitled to kind of try and sell the products they do. I think it's a hard system to own in its entirety. It's a lot, it's a lot, right?

• 1:43:52Speaker 00

  Decide everything for me, please.

• 1:43:53Speaker 01

  Yeah, it is a lot. And I'm trying to, you know, I haven't like gone too deep in understanding like the, how they approach their GTM here. But I suspect you would have to do it piecemeal, right? You'd have to be like, hey, we first are your coding agent. Oh, then also, like, you basically land and expand, right? I mean, we talked about this bundling unbundling piece. You don't start by saying, here's your whole factory. There's no way you could get like a Ford on board with that, you know what I mean? Where you're going to go on and just plop the whole thing on their desk. That's at least how I would think about it, is you first be their, their coding agent of choice. You, you build up some background agent for them, you, you own some of their SRE pieces. And then you're like, well, now we can just weave it all together if you define everything in code or something like that. Or like, you know, configs.

• 1:44:32Speaker 02

  There was a fun blog post on this from OpenAI.

• 1:44:35Speaker 00

  Our buddy Ryan.

• 1:44:36Speaker 02

  We had Ryan on the podcast. Basically, they spent five months making an internal tool with zero lines of manually written code. They kind of walk through, one, it's called harness engineering, so you can learn harness engineering. But, you know, the learnings of, there was kind of a slump or slower at first, and then you offload and you had peer reviews. It's at this point, it's from February, but it's, you know, a vision of a software factory, right? Like you are full sending, letting agents fully write everything and how to do that well.

• 1:45:10Speaker 01

  Yeah, absolutely. Yeah, I mean, some version of this is going to exist. To Spix's point, I do wonder about the, the sales piece here, like the actual business of this. That's an interesting question. But from a principles perspective, I think this is the, this is a version of the future. And then beyond that, I think there's going to be some amount of dabbling into and speaking about like, okay, what does it mean to, yeah, what does it mean to be a developer in the future, right? Like, are we, people talk about taste, people talk about like, you know, understanding of systems and architecture and, and yeah, like software taste. I want to like go a little bit deeper here on maybe starting to like, at least put some answers to the question around this.

• 1:45:47Speaker 00

  Yeah. So this is your answer to where AI software engineering goes next?

• 1:45:51Speaker 01

  Yeah, so I mean, the factory is like the tactical, but I think beyond that, there's like, there's a question around like the role, right? Like, what does the role evolve into? If, if the factory is, is running and, you know, full steam ahead, doing its thing, the conveyor belt is churning out code, what, what do we do in all this, right? And I think we, I think you and I, we had this discussion with Jerry at the dinner once upon a time, a few weeks back, with Dex and Jerry.

• 1:46:11Speaker 00

  Yeah, yeah, yeah.

• 1:46:12Speaker 01

  Around the, around the factory. But yeah, I, I think that there was this part of, at least the discussion points there, which I am pretty much on board with, is like, increasing levels of abstraction, and then somehow, there's someone still, even a CEO of a company, has to have enough oversight and understanding of like different constituent parts to be able to make a judgment on direction, right? To be able to course correct, to be able to set metrics and things of that nature. I think there's a version that translates of that. There's, there's written that, of that analogy that translates, like, someone who oversees the technical. We call it a manager of agents. We call it a, you know, different terms, they're all buzz, buzzwordy. But in some, some respects, someone who understands product requirements, constraints, technical constraints, kind of loops through all of those and, and knows how to adjudicate.

• 1:46:53Speaker 00

  Yeah. I will say that in the chair before you, we just had Anthropic's head of products and head of engineering for the,

• 1:47:01Speaker 02

  Platform Engineering.

• 1:47:02Speaker 00

  Their platform. And I was like, when are, when are your jobs going to come by? And they're like, not yet, you know? And so if Anthropic's not doing, not there yet, you should not feel like it's imminent.

• 1:47:16Speaker 01

  Did they say how long or are they just not yet?

• 1:47:18Speaker 00

  Not yet, but who knows? And I definitely, there was definitely a lot of like, okay, yeah, we're, I'm doing her job, she's doing my job, like, yeah.

• 1:47:26Speaker 02

  Yeah, I think it's also an interesting perspective for students, right? You said there's a lot of sophomore undergrads, like, you've got two, three years to go to plan out what you want to be doing. There's different ways to look at it from that, right? You're not a straight in industry.

• 1:47:39Speaker 00

  The thing that I care, I'm thinking a lot about is, well, so one part of this discussion that resonates more with me is the loop stuff, where like, okay, you, you started writing prompts, now you no longer write prompts. You write prompts, they write prompts.

• 1:47:53Speaker 01

  That's right.

• 1:47:54Speaker 00

  Right? And then they just keep going up. And I think to me, the sort of recursive, concentric circles nature, going outward but also going inward, is something that builds the factory. I think you build up to the factory, not build down. But I think a lot of people will eventually start building down. Like, you start from the top level, like, what is my goal? And then you figure out the systems. That currently creates a ton of slop and you should not do that. But someday you will.

• 1:48:23Speaker 01

  Yeah, yeah.

• 1:48:24Speaker 00

  Yeah, so instead of, so, you know, that would be my question to prompt you is, do we, what is the sort of prime question that people exist to solve? Like, that is your, that is your thing, right? So, in, let's say, an engineer today, you are measured by your PR merge rate.

• 1:48:46Speaker 01

  Right.

• 1:48:46Speaker 00

  PR contribution rate. Well, two years from now, that's gonna be ridiculous. Okay, well, what's your main thing?

• 1:48:53Speaker 01

  Right.

• 1:48:54Speaker 00

  Yeah.

• 1:48:54Speaker 01

  Money?

• 1:48:55Speaker 00

  People always go to money too quickly. It's something in between money and.

• 1:48:58Speaker 01

  Right.

• 1:48:58Speaker 00

  PRs.

• 1:48:59Speaker 01

  Customer satisfaction, CSAT scores from your features.

• 1:49:02Speaker 00

  Even that's like a product decision, not really a code decision, right? Like, I don't know.

• 1:49:07Speaker 01

  But if, but if the engineering is evolving into like more of a PM engineer, like product engineers, you know, hybrid, then they, then I think that's, you know, that's what answers the question of like, what does the role evolve into? I think it goes without say that I would today recommend to students, spend more time thinking about holistic systems, not just your like single part of the engineering puzzle, but how that ties in with, and this is like in practices of happening, like students for their capstone projects, I encourage them to go out and like, in fact, what I give them as the seed of ideas for final projects is like the YC request for startups. I'm almost like, go build a company, you know what I mean? Like, go, go build something interesting.

• 1:49:43Speaker 00

  What do people build faster?

• 1:49:44Speaker 01

  Uh, I mean, a variety of things, like someone was like, someone came in the E department, was in the E school, was studying EE, and they were like, they built effectively a translator for like parsing of like manuals to that have like, like memory dresses in C and then converting that into like, like code that they could then like interpret and even like visualize. So, so something specific to their use case, someone was like a designer and is now actually at Vercel, and she like had, you know, like, take a picture of my like design notebook and generate me an app based on what I take a picture of, right? So literally, she was like, for her demo, I was like, here you go. Snaps a picture of like some random like scribbles that she was doing, and she got it fully fleshed out with her cell app and from that. And I think because of that, probably got a job at Vercel doing that, doing some original.

• 1:50:26Speaker 00

  Wow, nice. That's a high bar, actually. Vercel, very known for their design engineers.

• 1:50:31Speaker 01

  Yeah, yeah. And I think she's now working on Eve, if I remember.

• 1:50:35Speaker 00

  Ah, no Eve in your?

• 1:50:37Speaker 01

  Eve, Eve is an OSS, or like OSS partner. I mean, like, they, one of the ways Vercel is partnering with us is that they are having students build with Eve.

• 1:50:45Speaker 00

  Yeah, so this is the concept of meta harnesses, right?

• 1:50:48Speaker 01

  Right.

• 1:50:49Speaker 00

  Harnesses of harnesses. You can always go one level deeper. But I, I do think like it makes sense, at least for me, in the same way that model routers wrap model APIs, you, you need to wrap harnesses as well, if you're gonna swap between them.

• 1:51:02Speaker 01

  Agreed, agreed.

• 1:51:03Speaker 02

  I think at some level, students should also stay grounded and like, there's a lot of stuff happening at the edge. There's also a lot of companies that are still like, single call chat GPT in production, playing around, haven't tried it.

• 1:51:18Speaker 02

  You know, there, there is a lot of that, so there's, there's still time for what to work. But I think your point on taste is good. Have taste, try stuff, like, whether each company has their own version of a software factory, you're building tools to help companies do it, like, staying on top of it, actually using the tools, playing with it, like, you'll, you'll figure it out along the way, you know?

• 1:51:41Speaker 01

  Yeah, and I think the, a hundred percent agree with all that. I, I mean, first to your point that are a lot of companies still in the prompt chat GPT or some version of, you know, chat GPT, maybe they just got a cursor license and are just now getting people on boarded. That is way more true than we believe, certainly here in the valley. That is, I can, I can say a hundred percent confidence.

• 1:52:01Speaker 01

  But yes, I think for the students being exposed to this is just, at a certain point, this, the fact that they know this and are on the edge is what will eventually allow them to also be some of the more interesting and more valuable contributors when they go to maybe places where, that it maybe aren't up to snuff in terms of being on the frontier.

• 1:52:16Speaker 00

  Yeah, one of the, this, this kind of progression type talk, still very influential, about a year old, Steve Yegi, talking about the, the levels of psychosis. He's all the way through level 10 now, where he's, he's, he's given up and he's moved to a farm. But, uh, yeah, like, you want, you want, sorry, I'm making fun of him. He built Gastown.

• 1:52:35Speaker 01

  Yeah, I remember.

• 1:52:36Speaker 00

  All these things, moved to Gas City, now he's burned down the city and says, I'm building my game. I'm, I'm no more Gastown.

• 1:52:43Speaker 01

  I thought, I mean, I, I remember talking to him at, at the engineer fair early this year, and wasn't he, his whole thing, like, the only useful thing that was built with Gastown was Gastown?

• 1:52:53Speaker 00

  Right, that's a horrible failure, horrible.

• 1:52:56Speaker 01

  Totally, totally.

• 1:52:58Speaker 00

  And, and so like, people should be aware that like psychosis is real, and if you're not holding yourself accountable to delivering real products, you're just screwing along with LMS.

• 1:53:08Speaker 01

  Yeah, this is like a, maybe, I mean, relevant to, but I wonder how much of, I mean, I played a little bit with Gastown, very involved system, maybe a little ahead of its time. But I'd also wonder if they were like bottleneeted a little bit by the, by the marketing, in that it was a very difficult thing to grok from a terminology perspective, like there's a lot.

• 1:53:24Speaker 00

  It means thanks to him.

• 1:53:25Speaker 02

  I think the Gastown marketing also is one of the things that propelled it, right? Like, what was right before Gastown? There were like.

• 1:53:34Speaker 01

  The, like, yeah, his, his, yeah.

• 1:53:36Speaker 00

  Yeah. Unless you're talking about someone else's.

• 1:53:38Speaker 02

  No, no, there were like, there was the whole open claw, there was something else, and there was Gastown. And the whole manifesto also really helped spurred into the conceptual, the people that would never use Gastown, but learned about Gastown, because you can read the manifesto, like, there's a marketing angle to it.

• 1:53:55Speaker 00

  Yeah, I mean, so, so, I, but I do, so Steve is not necessarily someone to take literally, but directionally, like, you know, directionally correct, you know? And, and so I think he's directionally correct on this one. And so if you're at the IDE stage, it's time to move into a power user stage. If you're a single agent, you probably, it's time to move to multi-agent and blah, blah. And you probably should try to build your own orchestrator, right? And that's all that, arguably, this is not, this is that kind of out of date now. There, there is more to the software factory beyond this, but this, this article predates the software factory. I, I do think, you know, one of those things where like, I'm open to, as an employer, I have this problem in my company today, which is, I do not want to hire anyone who hasn't been through psychosis and come out the other end.

• 1:54:38Speaker 00

  Because I am currently paying for somebody to go through psychosis. And it's very expensive from his salary, very expensive on tokens, and I'm getting nothing out of it, and he doesn't get it, like, my, my feedback doesn't get through. And so like, it's almost like I would encourage your students to have psychosis and then come out and go like, okay, what was wrong with me? Right, because like, that's what Steve just went through, like, a very public two-year journey of this.

• 1:55:02Speaker 01

  Yeah.

• 1:55:03Speaker 02

  It's weird on the student side, there's like a lot of industry, you think you're being productive, you're not, is probably the student version, right? You're notion charting and making all these optimizations, personal essence, like, dang, I went through the psychosis, but nothing has changed, right?

• 1:55:18Speaker 01

  Right, right.

• 1:55:18Speaker 02

  Go through it, go through it.

• 1:55:19Speaker 01

  Right, yeah. I haven't seen this, I've seen a version of this from Boris, actually. Obviously more geared toward Claude use cases, but I remember he made like a Claude artifact, which showed, I think more at the team level, but I think he had four stages, not eight stages, but it kind of roughly followed the same progression of like, node option to through to, you know.

• 1:55:37Speaker 00

  Yeah, only four is good, this is, this is too many levels. Yeah, well, we'll see if Viva can pull it up. But other than that, you know, zooming out, have we, you know, discussed all the things?

• 1:55:47Speaker 01

  That's the one.

• 1:55:48Speaker 00

  Oh, that's the one? Okay, there we go.

• 1:55:50Speaker 01

  It is an artifact, okay.

• 1:55:52Speaker 00

  So it's gated, assisted, parallel, what's the last one? Supervised autonomy and AI native.

• 1:55:57Speaker 01

  So it's zero indexed, so it's five levels, right?

• 1:55:59Speaker 00

  All right.

• 1:55:59Speaker 01

  Started zero.

• 1:56:01Speaker 00

  Yeah, wow, actually, it's pretty good. I need to redo this in more, more depth.

• 1:56:08Speaker 01

  And use AI native.

• 1:56:08Speaker 00

  Yeah, yeah. I would give a shout out to dynamic workflows. This is one of those things that is a sub-agent pattern that the opening guys are also exploring, that is clearly fitting. And it's basically, you're writing a harness on demand for the thing that you're trying to do. And so this is maybe, like, a lot of sub-agent conversation is just you're copying the, the parent agent, but you actually should be writing way more code and then like running that as a, as a, as a sub-agent, and that is what the Anthropic people have chosen to call dynamic workflows. Okay, but, you know.

• 1:56:45Speaker 01

  Yeah, we, we built a version of this as well, yeah, it's, it's a good pattern.

• 1:56:48Speaker 00

  For Monaco? What does it do?

• 1:56:50Speaker 01

  Yeah, I mean, on the fly, sort of like harness sort of use cases for GTM, yeah.

• 1:56:54Speaker 00

  Ah, that's great.

• 1:56:56Speaker 02

  I will note that when Boris wrote this, it's for team adoption, for people that are listening. And also, he says, you know, Anthropic is on stage three, we're still trying to get to stage four.

• 1:57:07Speaker 01

  Wow.

• 1:57:08Speaker 02

  He's like, I am at stage four, I'm the guy. But, you know, it's, it's not as simple as Steve's, like, Steve is a very good personal, do this, do this, do this. This is a much more, if you're running a team and you wanna AI pill your team and not just be a user of one, you know, Anthropic is still at stage three, don't, don't be like, yeah, yeah.

• 1:57:30Speaker 00

  Okay, I mean, you should teach this, this is, this is great. This is, this has got all this stuff that, this is everything that we talked about.

• 1:57:35Speaker 01

  There you go.

• 1:57:35Speaker 00

  I, I can't believe I haven't come across this.

• 1:57:37Speaker 01

  Yeah.

• 1:57:38Speaker 00

  Take the research. Very, very good. Okay, that was a wild tour, one semester of Stanford in one hour, two hours. Thank you for doing this. Any party thoughts, any call section?

• 1:57:49Speaker 01

  No, this is, thanks for, I mean, ton of fun. It's, it's been a, you know, nine months in the making, I'm excited. This is end of week one, a lot to do, a lot to learn. I'm excited to see what happens with, especially the OSS partners and just getting students like deep in, in the code of some complex code bases, having them build some stuff. So, you know, it'll be fun, keep you guys posted.

• 1:58:08Speaker 02

  You're doing another version next year? You, you need credits for students, if OSS people wanna get involved, they can reach out.

• 1:58:15Speaker 01

  Sure, sure.

• 1:58:16Speaker 00

  Does this credits matter? Do they want it?

• 1:58:18Speaker 01

  What?

• 1:58:18Speaker 00

  Do they actually want the?

• 1:58:19Speaker 01

  The students, or?

• 1:58:21Speaker 02

  You, you have credit card, I don't think a lot of students wanna spend.

• 1:58:24Speaker 00

  I'm working on a AI credit card.

• 1:58:26Speaker 01

  Yeah, no, I, I think, look, I will, so like, I remember, you know, it was like, you know, there's a certain number of credits for sure, but then on top of that, he's like, look, if they wanna go hog wild with this, like, let me know, and we'll do, have these like one-off experiments where they can just like scale it to like a thousand plus agents.

• 1:58:39Speaker 02

  Great, this is email from Factory.

• 1:58:40Speaker 01

  Yeah, yeah.

• 1:58:41Speaker 00

  No, those kind of people are valuable. So this is Wilson from Cursor, right? Who is like, I'm gonna write a browser, right? Those people who are ambitious and like can actually execute, those are very valuable, and that's basically a recruiting call.

• 1:58:55Speaker 01

  That's right, yeah, that's a good point.

• 1:58:57Speaker 00

  Yeah, I, you know, but I do think like, students should, should like, understand how to be ambitious, because snapping a photo, I mean, respectfully, snapping a photo and turning into an app is not that ambitious in, in this day and age, writing a browser is, and there, and like, there actually is a lot more projects like that.

• 1:59:12Speaker 01

  Yeah, yeah, I think that's right, yeah. It's all about raising the floor. I mean, I think, yeah, last year was, it would, you know, you know, credits, credit to the students of, of last year, but, uh, yeah, this year, obviously, we're gonna expect more.

• 1:59:23Speaker 00

  That's a fail now.

• 1:59:24Speaker 01

  We're gonna expect more, I mean, yeah, but that's, but that's, that's the whole point, right? We're leveling up, right? Last year, it was like, yeah, hey, have you heard of prompting? Like, here's what it is. And I spent some time talking about like, you know, K-shot prompting is one of the lessons, and like, I don't even think you guys even think about that now, so.

• 1:59:38Speaker 02

  I think there's a, there's a version of thinking you stick out in your cohort of classes, and then there's also a version of, you're also in early engineering roles of a bunch of people in industry, people still trying to get jobs, and it's not, it's not, you know, not to call out one demo, but you can, you can do a lot to stick out.

• 1:59:56Speaker 00

  I'll, I'll give the students a freebie of a week 11, make benchmarks, make benchmarks and evals. This will get you a, a good one, it's gonna get you in the door of every frontier lab and every front, every top agent lab.

• 2:00:09Speaker 02

  Good, good engineering blog posts with some rigor on anything interesting. There's a lot of benchmarks, a lot of hanging fruit there, but the engineering blog posts do really well, right? You, you went deep on one thing.

• 2:00:21Speaker 01

  Yeah, and probably also, there's like big, a lot of money to be made in that as well. So, not just in the doors, but if you make that into businesses, you, you can, you can really, really cash in on that.

• 2:00:32Speaker 00

  Yeah, people don't know, like, Sweet Bench was like undergrad project.

• 2:00:35Speaker 01

  Yes, yes, that's right, from Princeton, right?

• 2:00:41Speaker 00

  Great, okay, well, that's it, thank you so much.

• 2:00:43Speaker 01

  Yeah, thank you, guys.