AI Can Write the Code. Can It Run the System? | Grepr CEO Jad Naous

a16z General Partner Martin Casado sits down with Grepr CEO Jad Naous to ask what comes after AI learns to write code: can it also debug and operate the systems that code creates? Jad shares how AI has transformed engineering inside his 10-person team, cutting some projects from six months to just 10 days. But operating production systems is a harder challenge, with unpredictable workloads, failures, and countless possible states. They discuss whether AI can learn to diagnose problems, test hypotheses, and debug systems like a human engineer. They also explore AI SREs, security, query optimization, and why the best engineers may be gaining the most leverage from coding agents.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Martin Casado?

  The world that the AI lives in is all digital and is the actual world. And like, those things are not necessarily the same.

• 0:05Speaker 01

  When we started doing AI for coding, it was autocomplete. Nobody really knew to what extent you can actually build complex software using AI. And then it basically broke through the expectations because we were able to model more and more complexity in the system.

• 0:19Martin Casado?

  I just don't know if operations is constrained in the same way. Where is even the data to learn how to like, build a system that understands like all of the operations?

• 0:30Speaker 01

  My assumption is that we have all the data. The question I think is maybe about complexity. And like, to what extent can you take the things that you've described that are operational and convert them into something that an AI can reason about?

• 0:44Martin Casado?

  The dirty secret for the longest time of running like a distributed cluster was just human beings managing it. Do you think we're at a spot where you could like, use AI to run one of these services?

• 0:54Speaker 01

  I think that there's enough to be able to...

• 0:58Erik Torenberg?

  AI can now write remarkable amounts of production code. But writing software and running it are two very different problems. In this episode, A16Z general partner Martín Casado sits down with Grepper CEO Jad Naus, a systems engineer Martín first met when they were both at Stanford, to ask what happens as AI moves from coding into operations. Jad explains how his 10-person team uses coding agents today, why he thinks the strongest engineers are getting even more leverage from AI, and how work that once took months can now take days. But the harder question is what happens after that code shifts? Production systems face unpredictable workloads, failures, and an enormous number of possible states. Martín and Jad debate whether AI can learn to troubleshoot those systems

• 1:43Erik Torenberg?

  by detecting what changed, forming hypotheses, and running experiments? Or whether operating in the real world is fundamentally harder than generating code? And that leads to the bigger question behind the conversation. How far can AI go when the answer isn't already campaigned in the world that was trained on?

• 2:03Martin Casado?

  Jad, thanks for joining me on the podcast. For those of you who don't know, Jad is the CEO of Grepper, but also we went to grad school in the same group. And when I first met you, we had a new student.

• 2:18Martin Casado?

  I walked into the office and there was a motorcycle helmet and a pack of Marlboros.

• 2:24Martin Casado?

  I'm like, this guy must be unbelievably cool. So my question is, what happened to all your coolness?

• 2:32Speaker 01

  Question number one.

• 2:36Martin Casado?

  Maybe just for context, maybe give a quick overview of your background. So just for those that are listening, Jad is one of the best systems thinkers I know. I love talking to him. We're gonna go kind of do a deep dive of engineering from a systems perspective and how AI is impacting us, especially when it comes to things like operations. But just to ground it, maybe a bit about your background?

• 2:54Speaker 01

  Sure. So you met me at one of the peaks in my coolness.

• 2:59Martin Casado?

  Yeah, you're very cool. Motorcycle and Marlboro cigarettes.

• 3:06Martin Casado?

  I remember thinking, I'm like, wow, are cigarettes even legal? It was like so discordant.

• 3:12Speaker 01

  I remember Nick seeing me one day smoking outside of Gates and looking at me and going like...

• 3:20Martin Casado?

  Nick Bakuna's our advisor, Gates was the building, yeah. Actually, one of my favorite things, I don't, do you remember Balaji Prabakar, the professor?

• 3:26Speaker 01

  Of course.

• 3:27Martin Casado?

  My favorite thing about him was that he smoked. I was like, the only professor. I just thought it was like, great.

• 3:32Speaker 01

  Just to be clear, I don't smoke anymore. No, too old, too middle-aged for that now. But let's see, so I went to Stanford with you. You were always the guy I had to look up to and be like, hey, this is the guy I want to be when I grow up.

• 3:47Martin Casado?

  Because I started four years before you did, so.

• 3:50Speaker 01

  And so you were four years ahead of me. You graduated a while ahead of me. You went and started NISERA. I stayed and stuck around, worked more on OpenFlow, helped build a bunch of stuff on the hardware side from a security perspective, actually. One of the most fun projects that I worked on there was I built the first set of SHA3 implementations that people were considering using. And I built them in hardware to kind of show whether they work or not and whatever for one of my projects for my thesis. That was fun. And then I decided after a while that I actually don't like networking because it's just about moving bits from one place and putting them in another place.

• 4:28Martin Casado?

  How hard can that be?

• 4:33Martin Casado?

  He has to either be philosophically aligned that is deep or not. Most people assume it's not. I think it's very deep.

• 4:41Speaker 01

  I mean, there's something interesting about it, but ultimately, to me, that's like, it's not...

• 4:47Martin Casado?

  Yeah, it doesn't do it for you.

• 4:48Speaker 01

  Yeah. And then I actually went and did a postdoc at MIT with Nikolai Zeldefitch, who's also amazingly deep and super smart.

• 4:57Martin Casado?

  Yeah, he's phenomenal.

• 4:58Speaker 01

  I did some application security work at the time, started a company based off some of the research I had worked on with Ramash Chandra over there, who was a PhD student at the time. And ultimately, the technology was too early. So what we did was, well, ultimately, a lot of the work was actually really Ramash's. And his PhD was on, well, hey, if you get attacked and you get exploited, can you recover from that by rolling back the state of the whole world that's related to that application? And then roll it over on top of a patched version of the application to remove the exploit?

• 5:30Martin Casado?

  Yeah.

• 5:32Speaker 01

  And so we started doing that in the real world.

• 5:34Martin Casado?

  And we're like, that sounds really hard.

• 5:38Martin Casado?

  It's actually kind of relevant to what we're talking about.

• 5:40Speaker 01

  Yeah.

• 5:40Martin Casado?

  To what extent can you actually bound the state for like operational systems? Which I would argue is very hard, but yeah.

• 5:48Speaker 01

  We almost got it working.

• 5:51Martin Casado?

  Almost.

• 5:53Martin Casado?

  I feel like for all long-tail disciplines, this is why people get into flytrap for smart people. It's like, you see a bit of progress, but your ass is toting. But anyways, I digress.

• 6:02Speaker 01

  Yes, yes. So then I joined AppDynamics. I was there for about three and a half years. They went from about 200 people to 2,000 people, got acquired by Cisco. I was a Cisco employee for one day.

• 6:14Speaker 01

  And then you're like, Jad! Come work with me! Come do this VC thing with me! I'm like, what the? What is VC? I'm an architect, I do engineering stuff. He's like, great. The less you know, the better. Literally what you said.

• 6:30Martin Casado?

  That's true.

• 6:31Speaker 01

  And then the first day I started here, you said, "Slow down."

• 6:36Martin Casado?

  It's actually people that move from like actually in operating world to VC. It requires a lot of reworking the brain.

• 6:42Speaker 01

  Yeah.

• 6:42Martin Casado?

  Just like going from a PhD to operating, by the way. You have to undo the brain damage.

• 6:49Speaker 01

  This is what real world is like.

• 6:52Speaker 01

  And then I stayed here for a couple of years, did a bunch of work with you. It was a really fun time, worked with a lot of great people like Jennifer, Mark, and Ben.

• 7:01Martin Casado?

  Satish?

• 7:01Speaker 01

  Satish. Matt was here when I first, what, almost towards the end when I was leaving?

• 7:07Martin Casado?

  Yeah.

• 7:07Speaker 01

  And then joined Apply as their chief product officer. Helped them build their engineering product teams, kind of scaled from about like 15 people to over 90 across product, engineering, infosec, and design. And after about three years, I kind of had a head for each one of these pieces that I owned and I kind of was sitting around just coordinating and doing meetings all day and wanted to go do something else. So left and then started talking to a lot of engineering leaders about the problems that they're facing and what they're trying to do. Everybody at the time, this was in '23, everybody was saying, hey, we're trying to really spend. And I asked them, what are you spending money on? Everybody would say, first it was infrastructure and then observability. And so I started looking at that problem, trying to understand like, why do people

• 7:52Speaker 01

  think it's too expensive? Turns out, there's a few things. First, people were spending somewhere between 5% and 20% of what they spend on infrastructure on observability. But ultimately, what they were really trying to spend it on was reliability. They wanted to buy reliability and...

• 8:09Martin Casado?

  Of spend, like predictability of spend?

• 8:11Speaker 01

  Reliability of their infrastructure and applications.

• 8:13Martin Casado?

  I see, I see, I see.

• 8:14Speaker 01

  Right? And so they were putting money into observability as a proxy for reliability. And for the longest time, the more money you spend, the more reliability you got out. But as architectures changed and turned into microservices that are much smaller, that are much more vertically integrated, the volume of data that's being collected at every layer expanded, there was much deeper market penetration for observability and kind of the volume of data that you're collecting and the cost of observability was actually starting to grow faster than the units of function that you're outputting. And the marginal rate of, or like the marginal increase in reliability for every additional dollar you were spending on observability was just flattening out. And so that's why people thought it was expensive. But people were still willing to pay 5% to 20% on reliability.

• 8:59Speaker 01

  And so the question was, can we actually solve that problem and bring back the, that increase in reliability, that marginal increase in reliability? Not necessarily by spending more money on observability, but some other way. And so the thinking was like, okay, great. If we can figure out what is the signal? What is the noise? Can we actually turn the observability data that you're collecting? That's mostly fluff and mostly insurance at this point. And do that kind of extraction of information upstream? Do whatever you need upstream and be able to ultimately massively reduce the cost of observability downstream. And if you can extract that signal in real time upstream, then you can, one, reduce the cost.

• 9:43Speaker 01

  But two, you can now feed it into other systems and use it for driving reliability operations afterwards. And the distance at the time, there was something that was called AI coming into the picture. And we're like, well, great. If we are in the data stream and we're extracting the signal, then we are in a strategic position to feed that to whatever is going to come in the future. Because that's going to be necessary. And so that's what we built. And so Grepr helps companies. We started with helping them reduce their spend on telemetry massively. And so you put us in the data stream, plug and play, pass the data through, reduce your cost by 90%. And now we're building out that second layer where we're feeding those signals into the AI layer.

• 10:24Speaker 01

  That's now going and troubleshooting and figuring out when things are going, going wrong and fixing them.

• 10:30Martin Casado?

  I'd love to just talk through kind of your experience and how AI is changing dev generally, which is like, a lot of people get asked this question. A lot of people answer, but I just feel like, you know, like the engineer's engineer, like you've been an architect, you've run product, you run engineering teams, like, you know, like, actually, how big is your company now? How many people?

• 10:50Speaker 01

  We're 10.

• 10:51Martin Casado?

  Yeah, so you know, you've got, I mean, you're building a lot of software with a relatively efficient team. So maybe just a high level sketch on like, you know, if you're a senior engineer like you, you know, that has architectural experience, like, like, what sort of gains do you think people are seeing? Maybe talk through like how you use AI, what gains you're actually seeing? What is the stack? What is your process? How you think about it?

• 11:17Speaker 01

  So maybe I can give you like a historical view of like what's been happening in terms of like running a team and so on. And so, you know, when we first started AR, we're still kind of autocomplete. All right, I remember like at the time we were using Copilot and Intellige and we're like, oh my God, look at autocomplete these tests. We're like, yeah, but it's just autocomplete. And very rapidly things started evolving. And what I've noticed is that really good engineers are able to somehow put it into their workflows in a way where they, the AI starts training them.

• 11:56Martin Casado?

  Oh, interesting.

• 11:57Speaker 01

  In terms of like how to talk to it, how to get the best out of it. And what I noticed-

• 12:02Martin Casado?

  As an aside, do you think that's model specific? Like, is it, do you learn how to use AI writ large or do you get efficient at a given model?

• 12:11Speaker 01

  I think there's like some nuances that different models bring.

• 12:17Speaker 01

  I'm sure you're looking at your tweets, you've-

• 12:23Speaker 01

  There's like nuances, but there is a way to work with AI. And like, I think that there's also a way to work with different harnesses. And I think like, you know, we tried, for example, using open code or like different harnesses or like using codecs versus Claude, like Claude code, like different ones have different ways of managing context or thinking through problems or when to spin up the right subagents and what to tell it, like the kind of prompts. Like, for example, with Claude code, I find that if I don't tell it, don't spin up, you know, 20 Fable agents to do basic research, it'll go and spin up 20 Fable agents to do research and burn all your tokens.

• 13:04Speaker 01

  And so like a lot of these things, kind of, you start learning about the harness and the, and the models and like how they interact with each other. And then how they can interact with the rest of the world. But I digress, like, I'm just gonna, in my mind, like, this is kind of what I started noticing is that like, some people were adapting and learning and kind of doing it that way. And others weren't as adaptive. And then as we started hiring more people into the team, that disparity between, you know, okay, engineers and really good engineers just multiplied 10x.

• 13:40Martin Casado?

  Can you predict who will, could you predict who would be good? Was it like good engineers got better and bad engineers didn't? Or was it really, okay?

• 13:49Speaker 01

  Yeah.

• 13:49Martin Casado?

  So like, if you're a good engineer, you're just a better good engineer. It's not like there's some like bad engineers all of a sudden became good engineers.

• 13:56Speaker 01

  No.

• 13:56Martin Casado?

  Okay.

• 13:57Speaker 01

  No, no. It always ends up being like, you have to really understand ultimately the setup. Because the more complexity that your brain can handle, the more you can break down the tasks and kind of assign them.

• 14:10Martin Casado?

  I think this might actually be like non-consensus opinion that like, I mean, I do hear it sometimes, but the idea that like, the better engineers like are much better.

• 14:24Martin Casado?

  You found this to be pretty universally true?

• 14:28Speaker 01

  I mean, look, my team is still small. We haven't done it across 100 people. In my experience, for the things that we do, that is the case because we build distributed systems and distributed algorithms. And like, your brain has to handle the complexity because like, AI doesn't do it.

• 14:42Martin Casado?

  Right, yeah.

• 14:44Speaker 01

  And so for the problems that we solve, none of the models can actually think through the things that we do. Like, there's a lot of, for example, we have a query translation layer. We were just talking about query optimization. And so you have to really understand the semantics of the query and how to get to the semantics of the query and then explain it to the model. And then have the model go and build the translation layer, let's say.

• 15:08Martin Casado?

  Yeah. So for the people that are listening that may not be so deep in computer science, what do you mean by semantics?

• 15:19Speaker 01

  So semantics are really the meanings of things. Like, the semantic of a dog, of the word dog is the animal that actually walks on four legs and barks. And the syntax is the fact that it's just three, a three-letter word. And it has three different letters. The shape of the word is different. And that shape can be different between different words.

• 15:43Speaker 01

  But different words can actually also mean the same thing. And so they might have the same semantics, but they might have a different syntax. So a dog and a puppy might be slightly different semantics. They refer to the same-

• 15:59Martin Casado?

  But they're so close. So the way, and just for like computer science, the way that I describe it very simply is syntax, you know, it's like the, the characters you type and semantics is what the machine actually does. Right? And so it's like, like the actual runtime properties of the, the system.

• 16:17Speaker 01

  So you did computer science, I did electrical engineering.

• 16:21Speaker 01

  You became a computer scientist and I became a VC.

• 16:27Martin Casado?

  That's the problem, I see.

• 16:33Martin Casado?

  Anyways, I just wanted, for people that are, that are listening, I just want to understand, like, there's the actual syntax, which is writing the code, you know? And then when you talk about syntax, you're just talking about the code itself. Then when you talk about semantics, you're literally talking about you have a machine that interprets that code, whether it's compiled or not. And that machine does something and semantics is what the machine actually does. And these things are, you know, clearly very different, right? Like, so for example, some semantics you can't even derive by looking at the syntax. Like, you don't know what the machine is going to do because whatever.

• 17:00Speaker 01

  You kind of have to test it.

• 17:01Martin Casado?

  You have to test it, you know, like, and it can go all sorts of different ways when you test it. And like a very classic version of this is, is SQL, for example. Like, you can write, you know, some SQL query, but like how it actually performs, how long it performs, depends entirely on like, let's say, what the schema looks like or what the data looks like in the database, et cetera.

• 17:23Speaker 01

  But that's, I would not call that semantics, though. I would call that performance characteristics.

• 17:31Martin Casado?

  Oh, sure. Yeah, I think that would be like-

• 17:32Speaker 01

  Like, I would say like, the semantics part of that would be like, depending on like what the shape of data is or like what the configuration, you might end up with different semantic. Because like you said, you want null to be treated as false. And you won't know that unless like you also know the configuration of the system and, and so on.

• 17:50Martin Casado?

  Okay, okay. So for the purposes of this conversation, I think we should just say that there's like the actual code you're writing and how the system behaves. And we'll call the second one semantics, just so we can have a conversation.

• 18:03Martin Casado?

  What do you want to call it?

• 18:04Speaker 01

  We need to dive into the semantics of this and make sure they're clear.

• 18:10Speaker 01

  Because there's a lot of-

• 18:11Martin Casado?

  I know, okay, okay. I know, let's keep it at a high level. Okay, so, so, so, sorry, so let's go back. So, you know, we're actually, before we started the podcast, you and I were just shooting the shit. And you were talking about like how you can do products, projects now that take a week where in the past would take a team and months. Can you maybe walk this through a little bit about like how you personally kind of evolved? Well, do projects and then kind of the gains that you see. So like a little bit about like your stack and your process.

• 18:41Speaker 01

  So I use Claude code.

• 18:48Speaker 01

  I'm gonna say something that's actually pretty funny, but I use Windows.

• 18:51Martin Casado?

  What?

• 18:55Martin Casado?

  Was that created by this company, Microsoft?

• 19:00Speaker 01

  And I use WSL. So I use Linux for everything, like development and so on. There's no like X or UI for it. I just use the command prompt, but.

• 19:10Martin Casado?

  How do you use Windows?

• 19:11Speaker 01

  Cause I just never really wanted to learn the keyboard shortcuts.

• 19:14Martin Casado?

  Are you a gamer? It's like you tab over to Counter Strike or whatever.

• 19:21Martin Casado?

  Like, why else? I'm not hitting on Windows. I mean, you know, like, well, I used it in the 90s.

• 19:27Speaker 01

  Really? Like, the only reason is I never wanted to learn the shortcuts. I hate the one button mouse. I hate the one button stuff. I hate the number of modifiers. You have to type for shortcuts and stuff like I just, I just never, I'm like, why, why, why? Why waste time?

• 19:45Martin Casado?

  Okay.

• 19:47Speaker 01

  So I'm like, yeah.

• 19:47Martin Casado?

  I'll use Windows, yeah.

• 19:50Speaker 01

  And so WSL, I use Claude Code.

• 19:52Martin Casado?

  What is WSL?

• 19:53Speaker 01

  That's Windows subsystem for Linux. So it's a, they actually run the Linux kernel within Windows in a hypervisor and provide you access to it. It's great. They've done a great job.

• 20:06Martin Casado?

  Great. Windows WSL.

• 20:08Speaker 01

  And Claude Code. And like my latest, the latest thing that I do is I use Opus 5.5. I tell it.

• 20:17Martin Casado?

  Which is a really good model.

• 20:18Speaker 01

  It's an amazing model.

• 20:19Martin Casado?

  Really good.

• 20:20Speaker 01

  And it's, I've shipped in the last two weeks, these massive 20 PR things that I just dump on my team to go and, and review and they're like, oh.

• 20:31Speaker 01

  I'm like, don't worry about it.

• 20:32Martin Casado?

  Horrible. It's like, it's like the new reporting structure, like the people below review the PRs for the CEO. Is that, is that where we've become like?

• 20:42Speaker 01

  No, everybody's like writing the same.

• 20:44Martin Casado?

  Oh, everybody reviews that. So you review other people's PRs too?

• 20:46Speaker 01

  Yeah, I review my people's PRs and like, everybody's like just, just PRs flying around.

• 20:52Martin Casado?

  Founding in PRs.

• 20:53Speaker 01

  Yes. And but like the thing that has changed a lot in the review cycle is that like, we started much more focused on, to focus much more on like the, the tests.

• 21:02Martin Casado?

  Sure.

• 21:03Speaker 01

  And like doing things like golden expectations and like, are they shifting in different ways and so on? But Opus 5 and like the, the trick that I've recently learned about is that you, instead of using subagents, you basically tell Claude to spin up command line subagents because those maintain their persistent session over time. And so you can have it spin up these agents separately. You can actually run them from other sessions and, and so on. And that allows you to be much more efficient from a context perspective.

• 21:34Martin Casado?

  How many agents do you run?

• 21:35Speaker 01

  How many?

• 21:36Martin Casado?

  How many?

• 21:36Speaker 01

  How many what?

• 21:37Martin Casado?

  Agents.

• 21:38Speaker 01

  Agents. Typically I work on like three or four session windows at the same time because I found that more than that, I just start losing context between the different things because like they, they start finishing faster and then I, I, then they wait.

• 21:56Martin Casado?

  And do they interact via the, do they interact via the PR? Is that how it works?

• 22:00Speaker 01

  The subagents like together?

• 22:02Martin Casado?

  Yeah.

• 22:02Speaker 01

  No, they interact directly through the command line.

• 22:05Martin Casado?

  Oh, wow. Okay.

• 22:06Speaker 01

  Yeah. So like I asked like the, the main code, like the main thing that does the implementation to have a tech lead reviewer and like they basically ask it questions. Does this make sense? Does this not make sense? Review my PR, et cetera. And so they iterate on these PRs together. And then they have a PR doc that basically they work on with a design doc and everything in it. Like the first thing that I do is I ask that agent to do it, ask the tech leads to review the PR doc. I go through the, the PRD, I go through the design doc, make sure it makes sense. Go through each piece in like detail, make sure that all the algorithms work and like all the pieces work, the division of labor works, the P, the vision between PRs and their relationships work. And then I tell it just go start, start doing it.

• 22:50Speaker 01

  Figure out if there's like parallelization that you can do and then spin up other agents to actually go do them.

• 22:58Speaker 01

  But ultimately you have to kind of put it all back in one stack and be able to go through it as a human one by one. Because otherwise it just gets messy and bushy and you can't really tackle it.

• 23:11Martin Casado?

  So how much time? So can you give a sense of like a project you do now versus, I don't know, six years ago, like time differences?

• 23:20Speaker 01

  So the latest project that I shipped was something like.

• 23:23Martin Casado?

  By, by the way, this is like, to be clear, this is production code.

• 23:26Speaker 01

  Yeah.

• 23:27Martin Casado?

  Like demo mockups, whatever. So like, the statement you're about to make is the gains that you get for like shipping product.

• 23:32Speaker 01

  Yeah. So I'll give you the example of the last, there, there, there's the last PR that I worked on or like the last set of PRs was what we call auto, auto promotion in a data lake.

• 23:42Martin Casado?

  Yeah.

• 23:42Speaker 01

  So in a data lake, you store all the data and typically the data is column oriented, parquet files and iceberg and so on.

• 23:53Speaker 01

  And when you get data from logs, they're semi structured, they're arbitrary columns and so on. In order for it to have the best layout of the data and to be able to query it efficiently and fast, you kind of, the ideal situation is that every field in the log event becomes a column that you can filter on independently of the rest of the rows. And so we call that auto promotion or shredding.

• 24:20Speaker 01

  And doing something like that is incredibly complicated, right? Like you have to go and figure out how does the, the metadata system work? How do the commits of the metadata system work? How do you coordinate across different writers and so on? And that took maybe about 10 days.

• 24:41Speaker 01

  In previous days, that would have taken maybe about six months.

• 24:46Speaker 01

  For a single person.

• 24:47Martin Casado?

  Wow. And was this just you or?

• 24:49Speaker 01

  That was just me. It is insane.

• 24:54Martin Casado?

  Yeah. So we're like factor of 10, factor of 20 improvement.

• 24:59Speaker 01

  Yeah.

• 24:59Martin Casado?

  For production code for new products.

• 25:02Speaker 01

  Yes.

• 25:03Martin Casado?

  You know, one, you know, one, um, question I always have is like, I can very much understand.

• 25:10Martin Casado?

  You know, the, these models are trained on all the code ever written, let's call it, right? So I can understand why they're able to generate code, which is the syntax piece.

• 25:21Martin Casado?

  But then like for, you know, you're doing core infrastructure. Let's say you're doing distributed algorithms, like, like, like the properties, like how it handles itself when it's running is very, very different. Like often you just can't even look at the, the syntax to know what's going to happen, right? Like a lot, let's say, let's say you're, I don't know, you're, you're, let's say you're writing some distributed system. And then like the way that it, like it will perform, the, the availability, the correctness even is, is going to be dependent on what the workload looks like. You know, like, like what are the access patterns? What is the traffic?

• 25:57Speaker 01

  Are you talking about like the performance aspects of it though? Or are you talking about like the?

• 26:02Martin Casado?

  Well, I'm thinking like availability.

• 26:03Speaker 01

  Yeah.

• 26:04Martin Casado?

  You may have to lock things so it's not available for a period of time. I'm talking about correctness. Like maybe you've decided to do eventual consistency. And so like, if you do a read, it's not going to be consistent because there isn't a lock. And then performance, like, you know, maybe you cache, maybe you don't cache. So these are all kind of trade offs in the design space, right?

• 26:20Speaker 01

  Yeah.

• 26:21Martin Casado?

  And like, unless you know, like the access patterns, you won't know how it's going to perform in these things, right? And so I just think to myself, okay, I've got an AI, I can understand how it will write, you know, like, like code for a generic distributed data store, for example. But I don't understand is how it will know to like make the trade offs needed because it doesn't even know what the access patterns are. And it's not like these things can like, like where's even the data to learn how to like build a system that understands like all of the operations, right? Like today we rely on human beings to like manage clusters and to like, you know, optimize databases, et cetera. And so can you maybe talk about how you think of, like, I get the writing the code, but like you have these deep operational systems.

• 27:06Martin Casado?

  Like, how do you apply AI to that problem? And like, hey, have we made any progress at all? Or is this still an outstanding problem?

• 27:12Speaker 01

  The question I think is maybe about complexity. And like, to what extent can you take the things that you've described that are operational? And like that happen in production and convert them into something that an AI can reason about? And.

• 27:28Martin Casado?

  Exactly. Like, how, how could an AI reason about this system? Well, A, first off, like, to what, to what extent do you see that it writing in the code that it can actually predict what's going to happen? Like, how good of a job does it do at that? And the second one is like, okay, now how do we make it reason about this? So let's ask the first one, like, like if I, if I use your method, the, the JAD method of writing code and I built some big distributed system, like.

• 27:52Speaker 01

  How often does it actually get it from performance correctness?

• 27:55Martin Casado?

  How, how often is it like actually operationally, like pretty sensible? And by the way, I, I work on this, this distributed Gaussian splat render and I would say it's shit. Like it's horrible. Like, because like, you know, you've got millions of Gaussian splats that you've got to page in and out and stream and like, it just totally depends on like the running system. And so like, none of the models do a good job.

• 28:12Speaker 01

  So in our world, what I've found is that we've defined basically really well, what performance benchmarks look like and how the system should run. And what the constraints of actually building within like our stream processor look like to get the right performance. But every once in a while, it'll do like, my latest project has been all about query optimization and query performance. And so I was like, hey.

• 28:40Martin Casado?

  The classic example of something that's like hard.

• 28:42Speaker 01

  We are basically generating code that corresponds to SQL and we want to run it in a pipeline. And we're like, okay, well, you're generating shit code. Fix it.

• 28:54Martin Casado?

  Yeah.

• 28:54Speaker 01

  And it started basically going through and what it didn't realize was that it wasn't actually doing initially any optimization on the code. It's like, oh, that's fine. It looks like, what's the problem? And I said, well, look, there's these, uh, these conjuncts over here, you're doing and then like this particular term is repeated and it should have been taken away. And then it's like, ah, okay, let me figure this out and so on. And then like, it basically went on a thread to try to figure out what are all, like it built a bunch of hypotheses around like, what are all the things that it can go and optimize to make this better?

• 29:27Martin Casado?

  Yeah.

• 29:29Speaker 01

  But ultimately it was able, because it was able to ground that in performance benchmarks. And I was like, look, here's, here's a data set of the queries and like what the users are asking and so on. Go work on it. And so it can basically do a relatively good job that way. But when it comes to saying something around like.

• 29:52Speaker 01

  How much memory should I give the system?

• 29:54Martin Casado?

  Yeah.

• 29:55Speaker 01

  It has no idea. And that has to come from actual experiments that run in production.

• 30:01Martin Casado?

  Yeah.

• 30:02Speaker 01

  And so what I think actually these systems seem to be doing well at this point in terms of operational stuff is that it knows that, hey, I probably won't be able to answer this question. Let me run an experiment somehow and come up with that information. But of course, when it comes to operations, like production systems don't behave the way that you expect, and ultimately that's why incidents happen.

• 30:26Martin Casado?

  So have you found, are you doing any of like the AISRE stuff? Where you actually have a running system that has logs and you feed it to an AI? And that kind of helps you like tweak it?

• 30:35Speaker 01

  Yes.

• 30:36Martin Casado?

  And how, how well does that work?

• 30:38Speaker 01

  So we're just.

• 30:38Martin Casado?

  Cause anecdotally, uh, my experience with other people doing this has not been that great. Like the reports I've been getting anecdotally have not been great.

• 30:46Speaker 01

  So a couple of things. The first one is we're doing that for both operational pieces and, uh, and for security. But the way that we're tackling it is not the way that other people are doing it from an AISRE perspective. A lot of those systems basically start from a lagging indicator like, hey, this metric spiked or like this alert happened. Go back through a very complex system and try to determine why that happened.

• 31:10Martin Casado?

  Right.

• 31:11Speaker 01

  Like.

• 31:12Martin Casado?

  Yeah, sure.

• 31:13Speaker 01

  It's, you know, that's why those don't work because like backtracking into a root cause is actually very problematic. And so instead, what we're doing is doing it the other way around, which is can we actually start by figuring out if there's something different in your environment that has changed? Like, can we do a large scale clustering, a large scale, uh, kind of figure out a needle in a haystack? Something that is different?

• 31:38Martin Casado?

  And then. So you're assuming that it was working fine before and now it's not. So something happened. But it could be just like a change in like normal traffic patterns, right?

• 31:49Speaker 01

  That's fine, too. That's fine, too. And so the, the, the thing that we're looking at is like, when things go wrong, usually the system will tell you something went wrong somehow.

• 31:59Martin Casado?

  Yeah.

• 31:59Speaker 01

  Right? If it's working fine and scaling as normal, nothing will happen. But sometimes you'll see some like message which says error connections ran out.

• 32:10Martin Casado?

  Yeah.

• 32:10Speaker 01

  Our memory exceeded or resource buffers emptied or like some kind of indication, like some kind of really early warning, like something happened. And that's the one message that we want to capture and then go from there and see is that an actual problem that we need to investigate and deal with?

• 32:27Martin Casado?

  So how does it do?

• 32:28Speaker 01

  So far it's doing pretty well. I mean, we are running it internally in our systems. We run it, uh, we ran it in a, in a couple of our customers' environments. And in the cases where, there was a couple of cases where we ignored it.

• 32:41Martin Casado?

  Yeah.

• 32:41Speaker 01

  And those turned into production incidents.

• 32:44Martin Casado?

  Really? Oh, wow. Yeah, it's kind of, just, just from an intuitive perspective, like I can understand how these models understand syntax. Because again, like I said, they're trained on all of the code ever written. And it just turns out that it covers a lot of useful cases. It's hard for me to actually even think of a bound on the actual operations of software. Right? I mean, it feel like, you know, this is a, it's a ridiculous statement, but it's actually kind of true. It's like every time you add a bit to a program, the state space of like, what it can do operationally increases by a factor of two. So you very quickly get to like, you know, like more, more, you know, states than atoms in the universe. And arguably, this is why,

• 33:27Martin Casado?

  this is why like product approaches to operations have never really worked, right? Like the chaos engineering stuff never really worked and we've always kind of relied. You know, actually the dirty secret for the longest time of running like a distributed cluster was just human beings managing it. And so I'm just wondering, and then, you know, and then in order to like, even train an AI to do it, you have to have like enough data and enough configuration. So like, I'm just kind of curious, like, do you think we're at a, at a, a spot where you could like, use AI to run a distributed cluster or run one of these services? Or do you think that's still on the come?

• 34:04Speaker 01

  I think that there's enough to be able to have it start that loop. Right? Like, I mean, ultimately, like, I think that, I agree with you, like there's infinite space. There's a lot of different things, but ultimately humans also operate on run books. And like, sure, there's a lot of things that happen in the, um, in production that you ultimately need some human to go and like, hey, this is too complex. There's like some state that's hidden that I know of because I talked to this other guy like at the water fountain the other day. And he told me about this thing he deployed. Like, there's a lot of stuff that potentially isn't visible in the state of the system.

• 34:45Martin Casado?

  Yeah.

• 34:45Speaker 01

  But I'll argue that like, the goal of what we need to be able to do is to like, get to have enough telemetry and enough understanding to like, make everything really visible. From a, hey, here's what the system looks like, here's how it's working, here's like whatever. And then I think ultimately, most of the issues that end up really showing up in operations, assuming you have the system previously running okay, is due to some kind of change. And if you're able to just focus on the goal of figuring out what the change was and like, what, and, and kind of going through.

• 35:22Martin Casado?

  Can I just ask a stupid question on this, which is, I, I, I never think of these generative models that good at predicting stuff. And it sounds like you're talking a lot about predictive modeling, like you predict a change or you baseline and you detect an anomaly. Like, am I thinking about this wrong?

• 35:35Speaker 01

  I, no, you're not.

• 35:38Martin Casado?

  Okay.

• 35:38Speaker 01

  Because the goal is not actually having the agent try to predict, but have it, uh, get a signal that something changed deterministically.

• 35:49Martin Casado?

  That's something, I see, I see.

• 35:51Speaker 01

  Yeah.

• 35:51Martin Casado?

  Wait, wait. So, so there, there, sorry. So there, so there is a separate model that does the predictive stuff and then that will alert the AI?

• 35:58Speaker 01

  Yes.

• 35:58Martin Casado?

  Okay.

• 35:59Speaker 01

  Yes.

• 35:59Martin Casado?

  That makes sense.

• 36:00Speaker 01

  Yeah. So like what we do, uh, like what we do is we basically, you give us your petabyte of logs per day.

• 36:05Martin Casado?

  Yeah.

• 36:06Speaker 01

  And we scan all of them and we've, we like, for every time we see a new kind of pattern emerge in your logs and use thing that we haven't seen before, we lift it up to the AI.

• 36:14Martin Casado?

  I see. So that's, that's a predictive model that actually does, like, does what predictive models are good at, which is determining out of norm things or whatever, whatever. And then it's like the, and it goes to the AI, the AI kind of uses that to figure out what to do.

• 36:26Speaker 01

  Exactly. So the AI basically can then reason and drive that.

• 36:30Martin Casado?

  I see.

• 36:31Speaker 01

  That down. So it can say, oh, this message, let me actually look at the code and figure out where it comes from and understand what the code does. Let me see if it's actually happened before. Let me see. So it basically runs all these hypotheses and heuristics to try to understand, does it make sense? Does it need meaningful in some kind of way? And then it can determine what to do with it.

• 36:50Martin Casado?

  So how far away do you think we are for, let's just go back to query optimization, not like logs or something like. So how far away do you think we are from having an AI basically do the job that a human used to do when it comes to query optimization?

• 37:01Speaker 01

  I think very close.

• 37:02Martin Casado?

  Really?

• 37:03Speaker 01

  Yes.

• 37:03Martin Casado?

  Wow.

• 37:04Speaker 01

  I think.

• 37:05Martin Casado?

  Running a distributed cluster.

• 37:08Martin Casado?

  I mean, I just think of like, okay, you've got like James Cowling, PhD from MIT, who runs a distributed cluster. And like, if the cluster's down, like the company doesn't work. And then if it's up, it's running. I don't know. Like, are we at the point where you would say that an AI can do that?

• 37:22Speaker 01

  No, not yet.

• 37:23Martin Casado?

  Okay.

• 37:24Speaker 01

  I think it's, I think, I mean, I think like, look, very, very simply, when we started doing like AI for coding, it was autocomplete.

• 37:34Martin Casado?

  Yeah.

• 37:34Speaker 01

  And nobody really knew to what extent you can actually build complex software using AI.

• 37:40Martin Casado?

  Yeah.

• 37:40Speaker 01

  And like, when we were looking at it, we're like, ah, like AI is never going to be able to build

• 37:44Martin Casado?

  Yeah.

• 37:45Speaker 01

  X or Y or Z or whatever. And then it basically broke through the expectations because we were able to model like more and more complexity in the systems.

• 37:53Martin Casado?

  I just, I just feel like, like I'm a very simple thinker on this stuff. And to me, these models are distribution in, distribution out. And it turns out like you can actually train on all the code ever created. And it's actually like that manifold is pretty manageable. And I just don't know if operations has, is constrained in the same way. If that feels to me more like, kind of like the universe, like heavy tailed and exception based and power laws and all the tough stuff for these models to learn. But I don't, I don't know. I mean, you're the one that's in the field, so maybe that's not the case. Maybe they're manageable. But, but I, I will, I will tell you, the history of systems, as you know, is, is you always want to build a smart algorithm that does the efficient thing, the smart virtual memory management, the smart congestion control,

• 38:38Martin Casado?

  the smart routing, and what do we do instead every single time?

• 38:41Speaker 01

  We build a.

• 38:42Martin Casado?

  We over provision, we know, we over provision every time, no, literally, we, every single time, like, like, remember like swap memory where you like swapped to disk? Dude, if you did that, you were fucked. Like that's it, your computer was dead. So like, we literally over provision RAM and like routing, you know how many like adaptive routing algorithms there are? Like, we will respond to traffic, but you have oscillations, like fuck it, we over provision, we just use ECMP. You know, so I just feel like the history of systems is you don't predict stuff, you over provision because you can't. And I feel like, I mean, you know, so I feel like in the operations side, so either we build this, you know, we will build the systems upfront that are over provisioned, so you don't have to like do kind of complex operational stuff. Or maybe AI can do it the way that a human being.

• 39:24Martin Casado?

  But I think it's very different than code because you're dealing with like exceptions, there's not a lot of data, et cetera. I just feel like a different domain, but I don't know. I'm a, like, I'm a VC. I'm just like.

• 39:34Speaker 01

  You know, I, I, I kind of like, the reason that I'm more, uh, optimistic here is because I live in the world where like what we do is actually deal with the observable world, like.

• 39:45Martin Casado?

  I was gonna say, you're the literal expert on this, so it's silly for me to even like ask you these questions.

• 39:48Speaker 01

  No, not at all, because like, in, in my world, we, we, like, my assumption is that we have all the data. Is that like, we know exactly when like a new thing happens or like an exception emerges or like a workload changes or like whatever. Like in my world, all of that has already been collected and streamed through and like.

• 40:11Martin Casado?

  Yeah, but, but the model that is not trained on a grounded source for that stuff, like.

• 40:16Speaker 01

  No, no, you're not.

• 40:17Martin Casado?

  That would be really hard.

• 40:18Speaker 01

  Right. But the, but the story here isn't like, hey, Mr. Claude, here's a bunch of logs, is something going on, right? And it has to be much more, uh, able to run experiments and kind of dive in and see like, what are the different shapes of possible causal chains that actually fit in the environment that exists?

• 40:41Martin Casado?

  I see. So yeah, this is what you said before. Now, now I see what you're saying. You're saying that like, the universe is heavy tailed, the, the number of bad things that can happen is, is probably an exception thing. But like, the set of steps used to debug it is a manageable thing. That's what you can, you can teach one of these models and that's what they have. And therefore, if you give it the right signals, it can go through a number of steps to actually constrain this problem.

• 41:03Speaker 01

  Exactly.

• 41:04Martin Casado?

  Interesting.

• 41:05Speaker 01

  Exactly.

• 41:06Martin Casado?

  And it is really is taking the role of the human. Because remember, remember the whole chaos engineering wave where like, and and the reason it didn't go anywhere is like you're very so quickly like trying to like, you know, reduce like a state space that was atoms in the universe like so like super quickly, like an act to make that manageable. And it was like the human being that would go through a steps that would do that on the actual debugging. And this is what you're saying the AI would do.

• 41:30Speaker 01

  Yes. Right. So the goal isn't to like have the model just be like, this is what happens. It's, hey, run through this workflow to determine what happened. And like, we're trying that also on the security side, for example, where like, hey, in security, you have a something different changed in your like statistics, like some behavioral difference in terms of what entities exist, the cardinalities of various things that exist. And so that becomes a perturbation and the whatever the universe. And so we take that signal and we feed it into an AI and we're like, hey, AI, come up with a bunch of possible causal chains that exist and then go through and verify

• 42:14Speaker 01

  if any of those fit the pattern of the data that we actually see. And so that becomes a much more like, hey, here's a, here's like a, I don't know, like fit this square into find the peg or like find the hole that fits this peg.

• 42:31Martin Casado?

  And that seems even harder to me because the operations one, the argument is the system can get in states into broken states that are kind of like very rare. And there's a lot of those rare states. And so it's just not something that you can, you know, train a model on just because there's so many rare states it can get into that are that are bad. And so maybe you can reduce that by doing the steps. The security one seems even harder because this isn't just like the natural order of rare states. You actually have an adversary that is trying to like intentionally find rare states and intentionally find stuff that you you can't do. And so it may be the case that like code becomes solved. It takes a little while, you know, longer to do operations and it take a lot longer.

• 43:16Speaker 01

  Yeah, but like also like think about, I mean, I don't know. To me, like the whole world is sort of quantized to a certain extent.

• 43:22Martin Casado?

  The plank length.

• 43:25Martin Casado?

  Is that what the physicists say? I don't know. I'm not a physicist. I think there's like, I actually, I literally don't know what I'm talking about right now. But I literally actually think that like, like, like spacetime is just good ties into a plank length.

• 43:36Speaker 01

  But, but yes, I mean, I agree. Like to a certain extent, what you just said, but for operations and systems in general, like there is a discretization of like the possible states and the pass through them that are possible.

• 43:53Martin Casado?

  Yeah.

• 43:53Speaker 01

  Right?

• 43:54Martin Casado?

  I just think when you're dealing with.

• 43:56Speaker 01

  There might be, there might be a ton of them. There might be a ton of them.

• 44:00Martin Casado?

  I'm gonna tell you, so remember Sundar, who was in our, in our grad school group?

• 44:04Speaker 01

  Yeah. Sundar Eyer?

• 44:07Martin Casado?

  Eyer, yeah, yeah, yeah. So I remember he came to me once and he, and he was, he was, he was looking very sad. He's like, Martin, I just had this really sad realization that, you know, if you take any MP3, remember MP3, he's like the music.

• 44:19Speaker 01

  Yes.

• 44:20Martin Casado?

  He said, he said MP3s have a finite, let's say, you know, four minutes. It's like four megabytes, you know? And that means there's a finite number of songs in the universe.

• 44:30Speaker 01

  That can.

• 44:31Martin Casado?

  No, because, you know, because it's, it's all made of, it's all made of bits, right? And it's four megabytes. And that means you can, you could just enumerate every one of those constants. I'm so, so, I'm so sad in this notion of like the, the finiteness of music, right? Literally, this is like, we're walking on the dish when he had this conversation. I'm like, motherfucker, dude, you know, you know how quickly you've like overcome like the atoms in the universe? You haven't even got to a K, right? It's like two to the two to the two to the two. Like, what are you talking about here, man? Be happy.

• 45:02Speaker 01

  Two to the two to the two. Four megabytes.

• 45:07Martin Casado?

  Like, there's so much state, like be happy. That's, that's effectively infinite, right? It's like, what are we talking about here?

• 45:14Martin Casado?

  So I do think at some level, like we do underestimate like the number of states that like, you know, systems can get into and like, maybe, may, I will tell you from the perspective of a VC that looks at these companies, the ones that are trying to tackle operations are having a tough go at it. It is, it seems much, much tougher than those that are tackling the code because I think it's a much tougher problem.

• 45:35Speaker 01

  I a hundred percent, I agree. It is a much tougher problem. And, and it's not just because like the systems themselves are, you know, there's like external loads that are random that kind of happen to you or that, you know, somebody just shipped a new thing and rolled it out into production, didn't tell anybody or like, there's some state that actually happens when like this thing happens and this thing happens and then you end up in a deadlock. It's also because it involves humans a lot more and not just like one human who's the coder. It actually also involves.

• 46:08Martin Casado?

  The pools of knowledge are distributed too.

• 46:10Speaker 01

  Exactly. And like the interactions between like what it takes to like connect the pieces and put them together like, but ultimately, I mean, I think about like where AI is and like, what are the possible futures and like the, the depth and the forward looking capacity of all the things. Like, I think code is just, to me, it's a microcosm of like, maybe, maybe I'm just too fatalistic. Maybe I'm just like too pessimistic about how the world works. And, and I think of everything as like following some kind of probabilistic path. And so to me, like, if we can make it work in code, we can continue expanding that level of complexity bit by bit

• 46:52Speaker 01

  and kind of adding layers and layers and layers until we can handle those problems of.

• 46:57Martin Casado?

  A lot of people say that's not fatalistic, it's the opposite. That's very optimistic is what people would say. For, for me, I don't think the universe gives up its secrets that easily, but like, we'll see. I mean, like, this is kind of the big question. And, and I would say to, as yet, it's not an answered question.

• 47:12Speaker 01

  No, yeah, no. But I don't know, a lot of things that as humans, we think of as being random or impossible to tackle. Yes. And that's what we saw with, with coding, like, you know, initially, like people thought like the, the possible world of all these things that you can implement in code are kind of limited and you can't really get a system to potentially understand the full complexity of a human brain that's required.

• 47:44Speaker 01

  And I think over time it'll all, it's so funny that I'm actually telling you this and you're like the VC who's looking into the future and being like.

• 47:52Martin Casado?

  No, well, I mean, I'll, I'll tell you my view. My view is like the, the, the study of these models are you train on data and either that, I mean, that the,

• 48:07Martin Casado?

  I'm trying to, so you, you train on data and that represents a distribution and that distribution is useful or not. And so like, when you train on language, the fact that it's useful, that to me says that humans are so remarkable because they've looked at the universe, which is a very high dimensional system. Then they've reduced it to a manifold that's actually useful for a model, right?

• 48:25Speaker 01

  Yes.

• 48:25Martin Casado?

  Right? But that's different than the universe. Like that's, there's a reduction from this high dimensionality to the language is much less dimensional. And I think that code is a reduction of all the operational states of, of a machine into like destruction that make it run. And so to me, there's still a disconnect between like the instructions or the, the description and the actual, like the actual running of the universe, the actual running of the system. I think we can make gains toward it, but I, what I will say is like, you're kind of converging on simulation. Like there are things that are computationally irreducible. Like you kind of have to run it to get the answer. You can't just like, you know, predict it. And so like, maybe we just are at an event horizon where like, like models will solve anything that you don't have to actually simulate to get the answer from.

• 49:11Speaker 01

  But why, why, why do you have to like, just think about it in terms of, hey, it'll have to answer the question like now as opposed to being able to simulate and, and, sorry, being able to plan its own execution to gain the information that it needs in order to be able to.

• 49:29Martin Casado?

  Yeah, maybe, no, no, maybe that's where we're going. Yeah.

• 49:31Speaker 01

  Yeah, that's kind of what I'm.

• 49:32Martin Casado?

  Fair enough.

• 49:33Speaker 01

  Yeah. And, and I think like.

• 49:35Martin Casado?

  There is a lower dimensional, there's lower dimensional space, which is like these steps that you would take that we can reduce a whole bunch with AI. And then whatever steps that you need that that can't do, it can create the code to do or it can simulate to do it.

• 49:47Speaker 01

  And the point that you're making though is that like, hey, look, the model can't actually go and figure out the additional dimensions of the world that we need to go and find to like model the universe into its world. Like, that's probably, I mean, that's probably true. But can it actually simulate a scientist sufficiently to be able to get to that point? Like.

• 50:11Martin Casado?

  Maybe.

• 50:11Speaker 01

  Maybe.

• 50:13Martin Casado?

  Cool.

• 50:13Speaker 01

  Can I ask you a question? How much of a, of a, the VC work do you think is something that an AI can drive?

• 50:25Martin Casado?

  So my model for VC is it's very relationship driven. It's, I think, so it's okay. So I just have a belief that markets are actually pretty efficient. You know, and so it's not about like finding the company that nobody else finds. I think people all find like the right company. I think they're efficient. And then I also have the model of VC that I'm, you know.

• 50:48Martin Casado?

  Tech, tech is positive and I'm kind of optimistic about all tech and I don't worry about TAM. And so I, I, I think the key is access, it's like raising money, it's access, it's ability to win deals. And there's ability to add value after that happens. And a lot of that is like, stuff that, that startups just can't naturally do like, you know, recruit, you know, big execs or help with that. And so the way that I think about venture, you kind of have to be a human being to do. It's like, you know, um, because, because that's the actual job. So for that, I don't think like an AI.

• 51:29Martin Casado?

  Like if you gave an AI simulated universe where it could like, you know, like whatever reward hack, like building trust relationships, fine. But I think that that would have to be, to be, that said, I will say, I have seen analysis done by AI that are as good or better than any, you know, genius I've ever worked with. Like so for the, the, the analysis piece, um, they're great. But I think this is more an indictment on VC's ability to like analyze than it is like how powerful, you know, the models are, I just think of all a Kabuki show and bullshit. I just think the right VCs are like, you know, you know, tech is, is, is net positive. You know, techways always add value, like let's find the best company and invest in it.

• 52:14Martin Casado?

  The markets are very, very efficient. And so I, I just feel like the answer to this question is almost kind of a technical, like my answer is almost a technicality.

• 52:22Speaker 01

  So how like, you know, one of the things, you know, I came here, I wanted to ask you this real, this question, which is like, you know, we started talking specifically about ops.

• 52:31Martin Casado?

  Yeah.

• 52:32Speaker 01

  But to me, like the, there's this world of AI writing code, which is kind of like this abstract simulation world that it lives in.

• 52:40Martin Casado?

  Yeah.

• 52:41Speaker 01

  And then there's the rest of the world.

• 52:42Martin Casado?

  Yeah, exactly. And the entire discussion is, is this, it's like, there's like this, there's like the AI, the world that the AI lives in, which is all digital and like whatever it is, the training data, the indistribution world, and this is the actual world. And like, those things are not necessarily the same.

• 52:55Speaker 01

  Right.

• 52:56Martin Casado?

  They're, they're quite often not.

• 52:57Speaker 01

  And so the question is like, to what extent do you need to train an AI to be able to survive in a world and like, what are the things that it actually, I, I don't want to answer that question. Like what do you need to, to train? But like, I'm curious about the open spaces. So we talked a little bit about operations. You said, oh, I don't know if it's going to be solved or not. I'm like, yeah, Martina, it's going to be solved.

• 53:19Martin Casado?

  Yeah.

• 53:20Speaker 01

  And, uh, in my mind, like there's so many other things, like for example, self-driving cars. Like for a long time, we thought that's, that's not going to happen, but it's clear that it's actually going to be done. Um, and it's going to happen, it's just taking time because of a lot of different things. And so what are like, what are the other next stages, the next areas that you think are kind of like lined up right after code?

• 53:49Martin Casado?

  I mean, my, my sense is, is, you know, AI is going to continue to take off in a wildfire for a constrained domain, where the, where the solution space is in a constrained domain. Right? Where the distribution of the solution space is the same as the distribution of the training set. The distribution of the syntax for code is the same as the distribution of the training set. The distribution of the generated images is the same as the distribution of the training set. In those, you know, areas, I think it's going to continue to take off as a wildfire. The domains where like the distribution of the solution is the natural world, I think it's gonna, you know, have, you know, it's gonna be much harder, take longer. And I think the economic equivalency is going to be much more difficult. Self-driving is a great example. I remember when Sebastian Thrun won the DARPA Grand Challenge when I was at Stanford. Remember that?

• 54:33Martin Casado?

  And we're like, hooray, you know, self-driving cars is around the corner. Seriously. And it took 20 years and $170 billion of investment to get to worse unit economics than Uber. Right? So that's, you know, so I think that like solving like, like solving real problems where you have to deal with the, the distribution of the real universe is just much, much harder. And so I view as AI as this general tool. It's a very effective tool. It's one where like you have, you know, a distribution of data and then hopefully you can apply that to like, you know, some problems. And we're going to apply it to a bunch of stuff. And I think economic equilibrium is going to determine what works and what doesn't work. Because I think it's a very, very general tool. It's like a universal function approximator.

• 55:18Martin Casado?

  And wherever that nets out, I mean, again.

• 55:20Speaker 01

  You know, one of the things that I said earlier was like how AI was training humans to like use it.

• 55:25Martin Casado?

  Yeah.

• 55:26Speaker 01

  And I very often think about like, you know, some of the problems that seem intractable was, is because the world is kind of that way.

• 55:34Martin Casado?

  Yeah.

• 55:34Speaker 01

  And then we're trying to fit AI or the tools that we're building to manage the world as it is.

• 55:40Martin Casado?

  Yeah.

• 55:40Speaker 01

  But in some situations, we actually tend to change the world to fit to what we can actually do.

• 55:45Martin Casado?

  Oh, a hundred percent, of course. I think this is the story of robotics.

• 55:47Speaker 01

  Yeah, exactly.

• 55:48Martin Casado?

  A hundred robotics, yeah.

• 55:49Speaker 01

  The, the automated data warehouse. And so the, the question, like that always comes to my mind, like when we were looking at these data warehouses and whatever. And like the only one that worked was the one where it was like, you know what? Just automate the whole fucking thing.

• 56:01Martin Casado?

  Yeah, yeah, yeah, yeah.

• 56:03Speaker 01

  And, um, and I wondered to like, what extent as we kind of improve or change how we, uh, or kind of expand the domain of what we want to solve with AI tools and so on, that we actually start modeling the world or changing the, the world and the things that can actually happen to fit what AI needs it to look like.

• 56:26Martin Casado?

  Yeah. I mean, listen, I, I, I think that right now, a lot of AI just operates in the digital world.

• 56:31Speaker 01

  Yeah.

• 56:32Martin Casado?

  And for that, it's just like literally the solution set is the same distribution as a training set.

• 56:35Speaker 01

  Yeah.

• 56:36Martin Casado?

  In a sense, and so that's fine. And I, I think that again, like in, in the case of equilibrium, like I think like when you're dealing with the real world, like you can solve any problem you want. It's just a matter of like computing money. And that the equilibrium point is going to be an economic one. And I think one way you can impact that function that you're doing equilibrium over is to constrain the environment, constrain the system. And so I, I think that we're going to like, we're in now in this meta economic equilibrium flywheel where we're kind of optimizing this super general, but very inefficient process with a solution set that people value. And I think, you know, we've got 30 years of this and a lot of value is going to be created.

• 57:20Speaker 01

  You know, one, you want to hear a funny story?

• 57:23Martin Casado?

  Yeah.

• 57:24Speaker 01

  So, you know, we built this AI thing and we're like talking to people, we're like reaching out to customers, we're like, hey, we built this AI thing, you know, we're helping customers become more reliable. Here's like what we're doing. They're like, great, we want to talk to you. And we go on the call and we're like, hey, here's what we do, you know, we built this system, helps you optimize your data, it helps you reduce your costs, and then it helps you become more reliable because now we can find these problems and solve them before they hit your incident. They're like, wait, wait, wait, wait, wait. Roll back, you said you can save 90% of our data dog bills?

• 57:55Martin Casado?

  Yeah.

• 57:55Speaker 01

  And like, that's what I want to buy. But, but, but the, but the AI thing, you know, like, no, no, no, no. That's what we want to buy right now.

• 58:02Martin Casado?

  Yeah.

• 58:02Speaker 01

  And it's, uh, it's interesting, like a lot of these, uh, like for code, you know, I always think about it as like, oh, for code, people just get it. They're like, oh shit, it just works.

• 58:12Martin Casado?

  Yeah, yeah, yeah.

• 58:13Speaker 01

  And on the ops side or like anything that's beyond that.

• 58:16Martin Casado?

  Yeah, it's very, it's still to understand the actual value and put a dollar value to it and be a payout to the darned 100%. So, and I also think from, from a coding standpoint, literally it's engineering productivity, which is one of the highest line items on these things. And so it's just very straightforward.

• 58:28Speaker 01

  Exactly. But you can also think about it, it's SRE productivity, or it's like your business going down, maybe you can save an incident and what the cost is. I don't know. It's, uh, it's weird. I think like how people think about it outside of the.

• 58:42Martin Casado?

  Yeah, I also, I also just think like, like this code was so magical because it was so good and nobody thought that could happen. And I think the operations stuff now, it's still like.

• 58:51Speaker 01

  Pretty early.

• 58:51Martin Casado?

  It's still pretty early, so.

• 58:52Speaker 01

  Yeah.

• 58:54Martin Casado?

  All right. Well, listen, it's great chat. Thanks, Jeff, for coming in.

• 58:57Speaker 01

  Hey, thanks.

• 59:00Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at A16Z.substack.com. Thanks again for listening and I'll see you in the next episode. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast.

• 59:45Erik Torenberg?

  For more details, including a link to our investments, please see A16Z.com/disclosures.