Latent Space: The AI Engineer Podcast
Hosted by Alessio Fanelli
Original podcast
Watch the full show on YouTube: https://www.youtube.com/watch?v=1N1QMJ98BeQ
9,449 words · 4 speakers
Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
Hey everyone, welcome to the Lit in Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Spix, founder of Small AI.
Hello, hello. We're delighted to be in the remote studio today with Anish and Raz of Traverso. Welcome.
Thank you for having us.
Yeah, thank you.
Yeah, so Anish, I've met you guys before. You were kind enough to show me a demo of Traverso before your launch, and then you spoke at AIE. Maybe you can kick us off with explaining your backgrounds and how you got into starting Traverso?
Yeah, absolutely. I'll start and I'll let Raz add to it. So my background in the world, I guess, like you, I'm from Singapore. Been in the US now for 15 years. Most of my time has been in technical institutions. I did my PhD at MIT in computer science about 10 years ago. I just fell in love with AI and ML and wanted to data center deeply. And most of my research at MIT was in a field called causal machine learning. So the idea that correlation isn't causation. So how do you get these AI systems to pick up cause and effect relationships from data? And also reinforcement learning, which is to me fundamentally about how do you search large spaces effectively? And I loved it. And my goal was to be in AI research for the rest of my life.
And I thought the best version of that is to be a professor and was lucky enough to get into Columbia as faculty. I'm a professor there of operations research and computer science. But I think with all the developments happening in LLMs and AI agents, you know, as I was making the transition from PhD life to faculty life, it just felt there's so much huge explosion of activity and of research activity in industry, especially in startups with research in their DNA, taking on a very technical problem. It felt like a good time to stretch myself in that way. And so that was kind of my philosophical reason for starting Traverso. And obviously Raz and Raj and Emmett, my other co-founders, they all moved to New York at the same time as me. I've known Raz and Raj for a long time from MIT. And so we just all saw the path, where the path was going with AI.
So started the company together in Jan of 2024, without really an idea, honestly. We just had an idea, we just basically said, We used to come to, we work every day at nine AM and be like, let's think until something comes to us. And but we're looking for something pretty specific. I think something at the intersection of causal machine learning, reinforcement learning, and how it intersected with AI agents. This is before school, like in Jan two thousand twenty four, not that many people are talking about it, but we felt like this is, has to be important because that's how you act in the world. And then our fourth co-founder, Ahmed, who was from Citadel Securities, before he was the one who really got us in this problem and think he had dealt, as you can imagine, uptime is of utmost importance at a, at a place like Citadel. And so troubleshooting and instant response is something they think about a lot.
As we looked into it, it kind of felt like a perfect problem for us to take on. And we can get into why, but it felt perfectly at this border of causal ML, reinforcement learning, and L L M's and AI agents that fit really nicely. So I can talk about why exactly, but I'll let Raz introduce himself.
Hey, I'm Raz. My background is, I mean, I I grew up in India. My background is also in AI research. I came to US actually exactly ten years ago on August fourteenth, twenty fifteen. And for my PhD at Berkeley, I started in foundations of machine learning more broadly. But then I made my way towards causal machine learning through, actually through a startup in observability for wireless systems. They're trying to do root cause analysis for when Wi-Fi goes down. Is it a, is it a device problem? Is it a LAN problem? So that kind of problem during an internship provoked my interest into this field. And I moved in that direction through the rest of my PhD.
So we say correlation isn't causation. And I joke, well, when I say it, then I'm allowed to say it because I have a degree. You've got to follow it. And then I went to MIT for my postdoc and I met Anish there. And we had very similar research interests. We vibed together and started dreaming of, you know, how do we convert theorems into companies? But that that really took off when I also got a job in New York as a faculty in Cornell Tech. And I was moving, again, similar department, operational research and computer science. And Anish and I were like, okay, we will be in New York. We have dreamt enough. Maybe it's time to do something about it. And we were like, what do we know? We said, we know AI, we know causal ML. What we don't? Have as a team, but slowly glued together and we don't have a problem.
Slowly we figured it out and here we are. I think it's a very exciting space of a very rich problem. No matter how you look at it, the pain point, the data richness from an AI's perspective, the product experience from a user experience. So it's a it it kind of feels like a blessing in disguise that we landed on this problem, although this is not the most natural problem that would have come to, you know, either of us during our research time. So, yeah.
Yeah, tell us more about that. I I think this was one of the first almost obvious ideas. When AI came out, I think I've seen dozens of AI SRE companies. It was almost kind of like a meme at some point, but then obviously you guys figure something out. So I'm curious how much of it is your unique approach versus timing with the model performance versus, you know, the market being ready, like maybe talk people through how you went through the idea maze to to get there.
Yeah, so funnily enough, I remember when we were flying to San Francisco to pitch, I was on the plane trying to understand what a trace is. I never heard the word SRE before. So we really came to this problem from like first principles, having no understanding of any of this. And so I think for the idea maze, we went through a few different ideas of what how causal machine learning and and L L M's fit together, like what A B testing for L L M's looks like, what dynamic prompt tuning for L L M's look like. Then we thought about what is something like amplitude look like, like product analytics for L L M's. So all in this general space of observability, but like hovering around it. And I think it was really when we saw this problem from the world of like core engineering that it kind of clicked. I think specifically because the market is really big, I think, and it's a longstanding problem
that it's not like you have to make a bet that L L M's will get bigger or like A B testing will be important for these systems. This is like a longstanding thirty, forty year old problem without really a great solution so far. So I think that's really what got us interested. And then I think in terms of the why we felt with a unique edge, because I think at the heart of it, if you if you, you know, just trim it down to the core of it, it's basically if some spike that happens, like latency shoots up or something like that. And you're trying to figure out why. The problem is there's thousand other spikes happening at the same time. And so, and you're trying to figure out as a given spike, a symptom of the issue, a spirits correlation because something else is wrong, what truly the root cause, right? And so it's about like finding this needle in a haystack with many fake needles everywhere. And that's exactly what causal ML and RL do.
It's about searching for this haystack and figuring out how to filter these fake needles. And so it fit aesthetically and also in terms of like our technical expertise really well. And I think we looked at the haystack is like logs and metrics and traces and code and config files and, you know, Slack messages. So, and it's exactly what these LLMs are good at understanding. And, you know, it's at any large enterprise, you're dealing with like petabytes of data. It's like a massive search problem. And so you cannot just put all of it in the context of an LLM and hope something great happens. You have to search the data sequentially and adaptively, right? And that's what these agentic systems are fundamentally about. And you're like query autonomously many different systems, whether it's Elastic, Grafana, ServiceNow, Datadog, Elastic. And so in terms of like the generational wave of technology and what we thought would be good at, it fit really nicely.
And I think that maybe the third thing is that everything we've seen at that point in the world of like AI suite, I guess, like all of these coding, AI powered coding assistants, like that was just starting to pick up. And so it's pretty clear to us that, you know, and we were trying to use it ourselves. And sometimes we didn't understand what the code was doing, but you know, we shipped it. And, and so it's like, well, if this is clearly, this is going to happen a lot more. And so if, and if it breaks, no one's has going to have any clue as to how it broke or why it broke. And so rather than, so we're going to spend all of our time firefighting and being on call and doing QA work, which I think would suck, right? If that's what engineering discipline became. And so we felt like something like traversal, which is thinking about software maintenance, not software development, has to come about to like fit with that wave.
And so that was in the sense of market timing, I guess. Yeah. So that was all the confluence of factors that came that made it work for us. And I mean, obviously other thing is that.
We got funded, so that helped. Uh, right, right. And I think the reason that happened was that, you know, like our first, our first VC back was Sequoia, and I think they had met, I think like, I think like 20 VC, 20 AI history companies before this, but unbeknownst to us. And that was the first question. Uh, like Bogomel, who's a board member from the office, like I've heard this pitch 20 times before. Why now should we go with you? And so we, we kind of explained, you know, like how we were thinking about it. And I think it was, I think through that conversation for a few days, it was clear to both of us that this is like really an AI problem, not observability problem at the heart of it.
And so then a team like us who doesn't know anything about observability, but knows a decent amount of AI are the right kind of people to solve it. I'd say that's honestly what last day was about was just being a research org, trying to figure out how to make this work.
It would be great if we can do maybe a demo in a little bit, so maybe keep that up. But can you talk through the, yeah, just talk about the inputs and then kind of like the runtime of traversal, so to speak, what it does. And then kind of the output of it. I think today, what I do, for example, is like I have a Sentry MCP server that gets my issues. I put in cursor, write a fix, figure out if it works, ship it, hope it fixes it. What, what's kind of like the future of that with, with traversal? Like, what does it look like to be a user?
Yeah. So I think a couple of things. One is I think that works really well, that kind of general flow you have when you have, whereas either the problem is really localized. And so the error log you're getting from say Sentry is very high signal and it connects to a small code base nicely. Now imagine you are a Fortune 50 company. Most of the people who are dealing with these kind of software operations, it's incredibly distributed, right? And building the fragmentation across. Fifty, hundred teams, you cannot do that essentially, there's just no way. For a variety of reasons, because you have to like be querying over petabytes of data. And it cannot, it's not like a simple MCP, high intent query. Like you have to be searching this space in a much more intelligent way. And you have to build a shared context across all of these different systems while limiting,
while respecting rate limits, because all of, any sort of serious observability system will put, you know, rate limits on you. And so how do you, and typically people want an answer in two minutes. Like that's what we've learned from just like working with customers. Two minutes is kind of the sweet spot. And so how do you search through petabytes of data in two minutes with many, many fragmented pieces of the systems, right? And so I think that in some ways is just as much an infrastructure problem as an AI problem. And I think, if I think about the last six months, we've been almost more of an infrastructure company than an AI company, like just figuring out how to search these things effectively, the right way to cache this data, so you can do it, you can search it in real time. So I think that's what the complexity is. I think for like very, very localized things, you know, it's, it's okay. But when you're dealing with like production grade scale, it gets hard quickly.
Yeah, let's jump into it with that, Mo. I think that's, you know, most people that listen also run into the same problem. So I think they'll, they'll understand it better.
Um, let me share my screen. There are two fundamental inputs. One is an approximate guess of when the incident began or it was first noticed. So that was in this case, evening on April 22nd. And then a brief description of what happened. And you can see it's like something was not working. People couldn't select some region while they were trying to create buckets. You can add more context and use images to kickstart an investigation as well.
And so when people are creating this, are they being notified of the issue somewhere else and then they come to you? Or are you automatically pre-filling this so that they can start their investigation?
Typically, I'd say it auto-triggers off an incident channel or an alert channel. And so that's typically the flow that happens. Sometimes people will come to our UI for, well, two reasons. One is, If you want to do a deep dive into an investigation, because on Slack you just get a small piece of information, basically like the leads. If you want to do a deep dive as to why we believe that is the answer, giving all of the associate evidence, doing follow-ups, all that kind of stuff can happen on our UI. The reason people would come to this page is a few reasons. One is sometimes they may not be an incident, but they just kind of want to get a sense of what's going on with their system.
So sometimes on, you know, let's say people will go to a, like Gruffa or Datedog or something, even when something isn't wrong, to just get like a health check of like, is everything okay? Is everything cool? And so people kind of do that too, it's like, oh, this, you know, this part of my system feels kind of slow. Can you just figure out what's going on for this, like this general health check type thing? So I'd say that's a typical use case for coming on the UI and beginning a manual investigation.
And then, so the next step is kind of like context creation from an engineering perspective. How do you guys think about building the context? And then maybe if you can, talk a bit about long context and whether or not that's something useful to you or, you know, I think there's always this question of like, if you could put all the data in, would it be better? Like I think here you have definitely more data than can fit in the context window. So I'm curious like where the sweet spot is.
So context, so I think there are a few ways people do add context, you know, which teams, which services, which alerts they are looking at, which dashboards they are looking at. So I think those nuggets are definitely helpful in making the investigation faster. One of the things that we have shied away from is asking too much from the user because then what is the value add of traversal, right? That if you have to find all the context and give it to the AI to go do the last mile delivery. So the onus is on us, so let me just run it and show you what is happening. So when, when the user asks this question, now traversal is going and figuring out what all context should be pulled in
for this particular issue that the user is describing. And in the background, it's leveraging who is asking the question, at what time the question is being asked, and its own understanding of the knowledge base or the system architecture in the background. Based on that, it's making queries to, you know, the, of course, the live data, the logs, the metrics, the traces, and it's building that context and it's doing this sequential hopping as it's building this context to hop from service to service, as you can see from index to index and finding. These three services impacted, okay, but they are also connected like this. So let me go hop there. So this kind of context building is being done by AI and that's why it takes a couple of minutes.
But the user can always warm start when they come to the UI or on Slack by sharing that, hey, begin the investigation, here is the three things I already know. And then it can accelerate the findings of the AI. But, uh, yeah, so that's how we're thinking about context.
And I'll add that, um, a lot of times the context is implicitly getting built by things like, what is growing in the Slack channel and like the L L M kind of parsing, all the relevant nuggets that are getting formed in that war room by itself and it's using that to guide the search as well implicitly through the process. But we found that the less you, especially like a highly stressful time, like an incident, the less you can ask of, uh, uh, you know, an on-call engineer, the, the better in some sense.
Yeah. But to the, to, I think your question of how much context is
being used by AI to come to the answer, you can see that, you know, it has looked at a few hundreds of millions of logs, tens of thousands of time series, a few hundred pull requests, deployments and thousands of dashboards. If you just take the token count of this much data, it's easily few million tokens. But then how do you do the processing of it so that the, whenever it hits an L L M or a reasoning or an orchestrator agent, the context is maintained? So that goes into, you know, the tool orchestration or the tool building and memory management that we do on our end so that at any given point, none of the L L M's or agents are overwhelmed.
So that's part of our agentic architecture, that raw number of context, raw context length is few million tokens, which cannot fit in a single L L M, but we still try to manage the right pieces to get to the answer quickly.
Yeah, so right now, if I were one of the VCs that have seen all the pitches, I would be like, well, aren't you just getting all this data and then asking an AI what to do? And I think maybe here is where you start putting some of the causal ML work background that you guys have. So can you maybe now show the forks? So there's kind of like the way that a lot of companies does it, which is like, hey, you you build the context. Now you ask the LLM to figure out why this was happening. What are like some of the things that you're doing differently that you can talk about?
So I'll note a few things, Allet, Raz, also, I guess. There's building the context itself is where a huge amount of the the the pain is, right? Because you're dealing with, let's say, petabytes of data, like, let's say, a billion possible symptoms, and you have to get it down to a small number of symptoms. And that entire, so it's this like massive data processing pipeline where you're reducing, we're getting more and more relevant context over time. And there's LLMs throughout that process. And each of those LLMs, there's basically AI agents throughout the process helping to to winnow down the the relevant amount of information. And the way it's winnowing down that information is by running these kinds of statistical tests that we have built in a proprietary way. Because most of the data you're looking at is like time streams data.
And these LLMs are really bad at at processing time streams data, right? And that's really where like good statistics comes in. So those are basically like statistical tests or the toolkit that the AI agent has access to. And like almost like a data scientist is deciding, you know, dynamically, what is the next best collection of statistical tests to run to filter down the search space? And also use at the same time the semantic information associated with the logs, the metrics, the traces, and so on and so forth. It's like a it's like a continuous cutting down of the data. And the way the data is being cut down is by an AI agent system dynamically deciding what's this whole test to run at that point in time. And Raz, you want to add to it?
Yeah, yeah, I think.
The question is, you build all this, as Anish said, you have to build this context in an intelligent fashion. You can use L L M for the semantic side of it and statistics for the time series aspects of it. So what we have built is like, you know, semantics meets statistics kind of framework. Let me show you a brief diagram that helps you navigate this huge amount of data. So if you look at this, at a high level, it is a reasoning agent that is trying to sift through your observability stack and you have so many different kinds of data and then process them and then go through, find all the signals in that data. So at any given time, I showed you there are 10 billion time series.
Somebody comes and tells me I am having these particular issues. Which of those time series should I look at? It's unclear and that reasoning has to be built into the agent. The agent is not good at looking at time series data. So that's, that is what statistics needs to take care of. But statistics doesn't understand what is the relationship between latency and memory usage and disk utilization. So that is the L L M part. So how do we intertwine these various tools to sift through the data and then come up with, you know, sifting through so much data to this one line root cause candidate that is also grounded in factual symptoms that are being observed in the data is where, you know, I would say
it's not just a simple L L M call or a simple M C P react agent that can do this easily.
Great. So now people come here and then what's the, what's the next step? Are people moving off to then fix the issue? So I know in the screen show you showed before, you suggested rolling back one of the commits. Are you guys mostly focused on fastest path to just stopping the bleeding, so to speak? Or are you also helping companies think about, you know, long term, this is how you can fix this?
Yeah, it's a, it's a good question. I'd say.
Typically, when you start with a company, we only ask for read-only access to your to your system because that's what security is typically comfortable with. And people also really hate it if you tell them that you're going to install like not an AI agent, but like an agent on their infrastructure host, right? Because that's even more of a computational overhead. So what people are generally comfortable with is read-only API access. And you have to kind of build trust there. Once you work with a company for a while, then obviously the next question that comes up is can you kind of self-heal the system for me? And there's a few different answers to that. One is simple things like pull back this revert this commit or something like that. If it's if it's very localized that way. And the other thing is that a lot of these companies, especially large companies, have have these automation scripts to heal their system and there are like thousands of them.
And the problem is they don't know which one to run, like restart this part, auto-scale this particular part of your infrastructure and so on and so forth. And so they don't know which one to run. And the thing is, if you find the root cause correctly, then these LLMs actually pretty good semantically understanding what is the what each of these different healing scripts are doing and they can then kind of connect it or match it to the right one to run. And and that's how we've been at least threading the needle of self-healing because people don't want to give you full write access where you can just do whatever to their system. So it has to be like a whitelisted set of commands. And typically that takes the form of pre-existing scripts that that they have found to be helpful.
That makes sense. I have another maybe slightly related question, which is the business model of these things? I think we talked about it on the podcast a little bit, but you had, you know, like I said, I use the Sentry MCP, right? But Sentry has a Sentry Seer product, but I don't pay for it because I use the MCP for free. Do the data sources, like, are they your friends? Like, are they happy that you're building this? You know, like, will the, you know, does Elastic want to build something here? Like, what are like the competitive dynamics in the market?
Yeah, it's a good question. I think at least what we have seen is that as you start hitting, you know, even midsize and large organizations, it's incredibly fragmented. Like they're using like seven or eight different tools. They're using, I mean, like one company I worked with using Datadog, Thousand Eyes, Splunk, Dynatrace, Datadog, ServiceNow. And I think, you know, and if you want to troubleshoot, you have to kind of traverse through all of these different systems, pun intended, right? And so that's the, and that's really the pain, right, is the fragmentation of the system and fragmentation of teams. Now, the problem is that all of these different companies, that typically the way they work is, is they price based on the amount of data they're storing, right? And so, you know, they have very little incentive for, for company A to provide you any insight on data being stored on, on company,
observability company B, right? Because that's just not, it doesn't fit with their pricing model as it stands. And so, but the problem is to troubleshoot well, you have to go talk to both of them in an iterative way almost. And that's why I think there's a chance for a company, a new, a new company such as ourselves to, to have value because we're not trying to sell you data storage. We're trying to sell you an outcome, which is like the investigation itself. And we're reasonably indifferent as to where the data is coming from and agnostic, right? And kind of the Switzerland of, we don't really care what the data is. So I'd say that's been at least our perspective on why a company such as ourselves has the right to win, even though, you know, there are these incredible incumbents in the space of observability.
Is this the pitch that you gave investors? How, how did you go into the, it's like you're switching the same thing and you're like, oh, we'll just be the Switzerland, we'll fix it, it's all good. What, what, what do you think? I guess, like, you know, if you do some self-reflection, it's like, what do you think is really unique about the approach you took that really resonated with people, especially when you don't really have a background that is like in this space, right? Like, I think like, you know, in a black box, you would imagine, okay, the founder of Traversal is probably somebody who ran DevOps or something at one of these big clouds or something like that.
Yeah, I think it was two things. As Sean said, fundamentally, this is a space of show, not tell. And I think we just had good references of customers actually using the product at scale and dealing with complex incidents. I think that the long term technical edge will be when you deal with complex incidents. And I'd say most of the companies that have been in the space haven't really put a product out that works and people are using it all the time. And also their use case has typically been like smaller, easy run book automation for alerts versus like actually dealing with incidents where no one has any answer. And so I think the biggest thing was just showing and having, you know, being able to point to customers that are actually using it.
And so I think that's probably the number one thing. And then I think the second part of it is explaining why this is such a hard AI problem, where it's not just like, oh, let's throw a chat GPT wrapper around your telemetry and something magical happens and why that's not possible. And so I think explaining that really carefully and we can talk about what those key AI challenges are. So I think, I think it was a mix of those two things, which is the, you know, actually having it work. And I don't think we've seen any other company in our space having something actually work at production. And then second, a very clear distillation of the key AI problems, which makes it an AI problem, not an observability problem.
Yeah, I mean, and to be clear, I think when, unless you was asking like, you know, the pitch, it was just, you know, you have very high profile investors, you know, I got connected to you by Bucky, who was at Kleiner at the time, and now it's at Lightspeed. And obviously it looks like you have Brian as well, who I know from Temporal in previously at Index Ventures. So basically like very, very top tier, you know, I know a little bit about your customer list, which you've talked about at the Duals Fair as well. And so like, yeah, I mean, it's just a really fascinating go to market that you've done. And then obviously that also, there's a, there's a lot of like the chicken and egg, right? Like you have to build the software and in design partnership with all these impressive names.
And everyone is trying to do it and you've somehow figured out how. So this is all we're trying to figure out.
And I mean, I can, I can talk through what some of the, the hard AI problems are here. I think one is, you know, if you think about AI agent company, like AI agent in some sense is a continuum, not a binary, right? It's like you, you can be more agentic, at least in my opinion. And that's fundamentally about how much of the workflow is deterministic versus not deterministic. Like at the simplest form, you have, you know, a fully predetermined workflow where in each of those steps, there's, there's some LLM being called, you know, in one of those steps or two of those steps. And people call that an agent as well, right? But that's really like writing traditional software with, and, you know, a machine learning routine in between. And then you have kind of the full extreme, which is the full control flow is decided by an AI agent, right? And that's really hard to do. And I'd say in our world of troubleshooting, you think about it, especially with complex incidents,
there's no, we, we have 50, 100 engineers on a call trying to figure out what happened. And the reason that's happening is because there is no playbook. And so we're just hoping that we throw more bodies at it somehow. You know, some hero engineer will figure out what happened, aha, and then the shared context will somehow crystallize, right? And that's kind of a crazy way of doing things, but that is the best you've done so far because of how complex the problem is. And so I think it's one of those situations where it is one of the most agentic problems, in my opinion, out there, troubleshooting, like incidents. And that's why I think it's such a hard problem to take on and why it's, it's a good AI problem, because you cannot like rely on a run book. And you have to kind of generate a lot of synthetic data. You have to think very carefully about statistics because these LLMs are really bad about thinking about time series data.
So these confluence of these few factors makes it, I think, a problem that's like prime for, you know, a team such as ourselves. This comes out of left field, not from central casting for DevOps. Was well suited to solve it.
And if, if I may add, I think one other thing is, I think that there is run books, right? People talk about, is it just a run book automation product? That's what happens, you know, for regular alerts. And that's what we call AI automation rather than AI intelligence. And then there is incidents where you're completely in the dark. Some people have some ideas, but then you need a lot of collaboration, coordination, knowledge sharing. So I think for us, when we began, the questions were, we don't understand observability as much. Don't understand these workflows as much, but that also forced us to build in a way that would generalize. Like, we couldn't cut corners because ironically, we didn't know what the corners are. So how do you cut them? So I think, so then when what we built runs and it actually finds value, we know that it is not hardcoded.
Because we didn't know what to hard code. So that's why, like, this, of course, accuracy is as a long-term battle in this space. Because the incidents are getting more and more complex, the scale is getting huge and huge. But when our AI one finds an answer, we know it's a good signal. Like, it is not by fluke, because there is no way we would have guessed it otherwise. So I think there was also some aspect of that in the excitement with the results, that they would not have just automated run books.
Yeah, awesome. One, one double click on just like the agentic control flow and reasoning models. Any commentary on just like what the best models have been so far for you? Do you use any open source models in your, in your stack? Is that useful or are you just frontier models all the way? And then like, obviously, if you, if you like G P T five, let's, you can talk about G P T five as well.
So I think we have, we keep trying all the models out there. A few things to note, I think, one of the reasons we, we have tried fine tuning and that does help in certain use cases. But for some of the enterprise customers, we cannot bring our own model because of all the AI regulations. The industry is still trying to figure out how conservative or liberal they have to be with the AI usage. So we built an architecture where the AI is rip and replace. As long as you tell me which endpoints to hit, we'll be able to point it. So we have typically found that reasoning models are critical for our use case. And actually, it is the use case where reasoning models are a necessity. You are trying to argue across so many symptoms and such a complex architecture of what the root cause is.
And a simple, you know, four O and these, the flagship models would not work. So reasoning works. Now, the next question is between Anthropic, you know, the three usual suspects, Anthropic, Gemini and OpenAI. We have typically found historically that OpenAI was better, the O three in general. But it was also because it was the most easy to use for us to. Build on top of, so we kept optimizing for it. So there is also this implicit advantage of if you have a well-supported infrastructure, people can move fast. Lately, we have seen Gemini start doing pretty well as well. But again, it's like the infra support is slightly lacking. So then we tend to default to OpenAI for many use cases. GPT five versus not, I think the verdict is still not clear. Like GPT five starts thinking when it shouldn't think.
So so that's like, that has been our feeling. But I would say we are still undecided about it. And we are running some final tests and tomorrow we'll make a decision. Should we switch to in some places where GPT five makes sense? Like, we use a collection in many places, but like for the heavy lifting, we I will decide tomorrow. So far, we have not seen overwhelming positive signal yet.
Okay, I mean, yeah, it's interesting. First of all, I think that that eval on GPT five, I think OpenAI will be very interested to hear that. They always want to know. I mean, they're going to fix bugs in the router, right? Like, I think this is one of the early days still. I think like for for you guys, actually, like, you know, I think there's one of these like more controversial views out there that actually you don't really need reasoning models, especially if you have a pretty deterministic workflow. That's like, you know, you're if you're fanning out a lot and you're you're sort of narrowing it down. You know, it's a pretty constrained domain. You don't have to reason for first principles a lot of the time. You can just kind of like literally spell out the steps to to to debug. So there's an argument to not use workflows, but not use reasoning models. But it sounds like you're pretty O three centric.
Yeah, and I think the reason is very simple. No amount of workflows will suffice for a big enterprise. Like, you have to link together some of the missing pieces, some of the poorly instrumented data. And that requires world knowledge and a few iteration with the world knowledge. So I think the way we think about O three versus four O, like just for reasoning versus not is, do I take my first thought seriously? Or do I rethink about it a few times, right? So if you compose four O ten times, maybe you can beat. But then you're aren't you essentially reinventing O three? So I think that's how we think about it is that no amount of workflows will ever be sufficient because of the missing pieces in system context and lack of good instrumentation in any real enterprise scale system.
And then, and then coming, yeah.
I was just gonna say one thing that sometimes people bring up is how the Claude models are much better at tool calling.
Yes.
Is that been your experience too?
Yeah, so I think for some of the agentic part of the stack, we are shifting towards Anthropic because their agentic and their tool calling, especially the unstuck part, you know, when you go down a wrong path and you, you build context that forces you to keep going down the wrong path. I think unstuck, there is signal that Anthropic and Claude are, are, are better. So I would say it's real news from our perspective as well. And it, it like we have seen it even like when you compare, for example, Claude code versus cursor. Like we are trying to leverage signals from every use case or internally in the company, we use cursor and Claude code. And you see that there is actually some real news in Claude code as well. So I agree, like it is slightly better right now.
Do, do you have some sort of like formal eval internally that you run? Or is it mostly vibes and then some tests?
So I mean, when we began, it was vibes. Uh, but uh, now we do have, you know, proper benchmarks and very detailed, detailed eval pipeline that informs our decision. And that's why I was saying that, you know, if you were to ask me at four, five months ago, is G P T O three better than O one? I was like, yeah, based on my vibes and a couple of incidents, but for G P T five, I said, the verdict is still pending because we are being more systematic about it. And I think for our problems, if you think about incident R C A, there's a lot of structure in the problem. That it's not only about you go from a symptom to root cause, did you get it right? Yes, that's the gold standard, but how do you leverage?
How do you break down this problem and see, are you making partial progress in that direction? Is important to, you know, help both the eval and the calibration of your architecture and potential fine tuning with synthetic data. I would say investment into eval at an earlier stage is highly recommended for that reason.
Yeah, I would add that in some ways I find, you know, the best AI companies will always have to be the edge of what the models can do, right? I think you you always want to be threading the line. If if everything works all the time, then you're not really pushing the limit and you're not innovating, right? So I think you always kind of want to be pushing. And so if you're always pushing, then sometimes your system will work, sometimes it won't work. And so and so then evaluation, I would say, has become sometimes a big bottleneck. You're spending the most engineering hours is to evaluating. And so the I think the best companies will be investing a lot of time in really good eval. And I think that's core IP. I tell that other company all the time is like one of the core piece of the IP of this company is how well we do eval. And so I think I think that is be true of of companies in our space and more broadly as as AI agent companies.
Yeah, and now we now that we've seen it, we have we just hired like probably four people from the the world of of PhDs in Europe's, you know, New York's people love benchmarking, right? And he also so so that's the type of person who have hired a bunch of now to help don that eval suite.
And and basically they were like, what will I do? And I was like, create the benchmark and beat the benchmark. They're like, oh, that sounds exciting.
But it's funny. On the more business side, said, oh, three is obviously very expensive, right? And you mentioned earlier about outcomes. How do you think about that? We had different, you know, we have Brett Taylor on the podcast, talk about outcome, kind of like based, resolution based. Darmesh from Offspot also talked about different kind of like, you know, results versus work. How do you guys charge for the product? I'm sure a lot of it in the early stages is like, hey, look, we'll figure it out, you know? But where do you see kind of like the market going? Like, do customers respond more to like base fixed fee that is kind of like tied to amount of work? Do they like the outcome based thing for things like this?
Yeah, it's it's a it's certainly we're in the phase of we're learning and the market is learning as well. I'd say it's a function of two things. One is the size of the search space, which is like how much, what is the fundamental like universe that we have to search through? And the best proxy of that is fundamentally the the the complexity of your of your infrastructure, like number of hosts, number of containers, whatever, that kind of thing. And then the second one is the number of investigations you're running. Though I'll say that it's different in customer support versus our world, because there you could actually sell a full outcome, like in a pretty binary sense, like was it done, was it not done? In our world, I'd say the maturity of the technology is such where you can basically give really good leads, right? And so it's not a full outcome where the system is healed every time.
And so then I think people have a bit more of an aversion to paying for a purely in terms of outcome based, because it's not there yet. So I think as the, as the, as our technology matures, as our market matures, then I think we'll move to more of a full outcome based pricing. But right now it's a mix of the size of your infrastructure and then the number of investigations running, maybe leaning more towards size of infrastructure than, than investigations. But for some of the settings where we are actually now, people do like full end to end self healing, then you know, we are trying to sell the full outcome itself.
How far away do you think we are as we kind of wrap for the self healing part? Do you think that's like one year, two years, three years? Or maybe like what are like, what percentage of tasks? Because there's obviously like different difficulties, you know, for like the easier ones, medium ones, hard ones, what are like the timelines in your guys' mind? Raj, you want to take this?
Yeah, sure. So I would say, I think as Anish said earlier, agents is a continuum. I think self healing is also a continuum, right? If you ask people who work in infrastructure, they are like Kubernetes auto scaling. That's a kind of a self healing. So, uh, so that analogy aside, our traversal is already doing self healing for a universe of issues where it's reliable to do a good root cause and then do self healing. So I would say those kind of issues are like, say, they occur 10 to 20% of the time, where end to end you're free and you can, it's like, think of if an L1 and L2 engineer can just do it, that is kind of where AI is today. Now for, then there is this level of 30 to 40% of the incidents or issues
where you need to involve, um, you know, a senior engineer for Santee checking. I think that healing will appear in, I don't know, six months to a year, that we will be comfortably, the technology will be there. It's implementation, the form it takes will continue to be a Collaboration with the enterprise. Now, if you go to the question that you had asked earlier in the podcast of like, how are you thinking about the long term, you know, re-orchestration or re-architecting of the code base? I think we are a tad bit away from that kind of a full agentic system that, you know, just reorganizes the whole code base. I would say, well, AI always keeps surprising you, but at least a couple of years away, if not more, where you can then rely on creating the code base.
And I would say that there is a difference between, you know, code-based debugging and re-orchestring the code and then it interacting with an external system. That is like, you can build a lot of unit tests and make your code perfect. That's what many good companies already do. But when rubber hits the road, then you start seeing the road bumps that you had never accounted for. So what I'm talking about a couple of years away is the issues where you have to see the real production system play out and run into issues that you cannot forecast just by unit testing. I think the things that can be easily taken care of by unit testing would be taken care of in the next year or two, a lot, the bulk of those things, but the more severe ones, I think there is still time.
Okay, I think that's a good overview of where we are. Just to end off, any things you want to get on a soapbox about, any predictions, any rants?
I think in terms of rants, I don't know of rants, but I'd say in the space of us and many other spaces, you know, I think more showing, less telling. I feel there's a lot of people saying the same thing. And so I think that, you know.
Okay, I think that would be the title of the episode.
So I think that would be good in general. And I think maybe, I don't know, I think that one interesting problem is that how do you create benchmarks, right? Because for us, the benchmarks, we create a such core IP for us because that's how we decide how we're doing. And then there's a world in which you want to go make that public, but if you make it public, that's you're also giving away so much of like, what is core to you? And so I think that is like a stalemate we find ourselves in in our world. And I'm sure it exists in other worlds as well that are adjacent, maybe the security or coding. That's, that's something to think about, I guess, is, is how do we collectively move everyone at the same time, retain core IP, which, which I said a bit part of it is benchmarking. So I think.
I'm actually curious about that. When your customers are evaluating it, how do you run that? Like, we have a company, Dropzone, that is a AI SOC agent. So they do something similar to you in the security space still.
One idea that I've heard is like, hey, you can create this kind of like simulation environments. We have a company called Verisai that does this for reinforcement learning. But there's also this application of like, you can use the same environment to see how an agent that I will buy from you will behave in it. Instead of using it as a data generator. Is this something that like, it's been on your mind, like, is there anybody doing interesting work there? Just like, how do I speed run the P O C? Because it's also best for you, right? Like, if you can put the traversal agent in a simulation and just parallelize all these different tests. That's much better than doing a P O C for like a month or two months. But it feels like that's not really happening yet.
Yeah, I think the problem is, you know, sometimes we've worked with customers where they say, okay, let's, let's, let's, you know, basically do chaos engineering and staging environment. And we'll see how you do, and then based on that, we'll decide. And then almost invariably, I promise you, you will crush it there. And then they'll be like, no, no, no, actually, they're like, to really know, we have to go in production. And you're like, what the hell, right? So, so I think that's kind of the issue. And then if you're trying to build a high fidelity enough simulator of your production environment, then it's just a production environment. So then I think that's the problem, I guess, is that we run into. And so the best thing I think we've found so far is if they give you like 10 historical incidents or something like that, that from production.
And assuming that their data retention of your production telemetry is there, that's probably the best way we've found to, to speed run it. Versus like a complexity environment and creating synthetic bugs there. Somehow that doesn't have that, you cannot get a big contract done that way, is what we found.
And quickly, I think in the spirit of, you know, the previous question and this question is like, everybody. The reason nobody gets convinced on staging environment is that everybody thinks that their environment is unique, right? Like, it's the most complex thing on earth. So they're like, can your AI actually figure out the complexities of my system? Although there are a lot of similarities across environments, but the belief that mine is unique. So I think unless you hit production and show value there, conviction cannot be built in a demo or a staging environment.
Cool, guys. Thanks for joining us. This was great. Any role that you are hiring for or any other call to action for people?
Yeah, I think we are certainly hiring aggressively. We're all in person in New York City. So we're building like a, at least for now, a very much like a New York company. And we're proud of that. So people who either live in New York or want to move to New York and want to work at a cool company, they're in section of the infrastructure. We are, we're hiring on all fronts. So, so please do reach out to us.
Cool. Thanks, guys.
Page updated October 24, 2026 · transcript and summary are machine-generated
