Grafana’s Approach to AI-Native Observability

Advanced software systems have long been more complex than any single engineer can fully understand. Observability is the established solution to this problem, but with AI agents now generating code, deploying changes, and operating autonomously, the challenge of understanding large software systems is entering a new dimension. Grafana is an open source observability platform, and one of the most widely used in the world. The company builds tools that help teams collect, visualize, and act on telemetry data across logs, metrics, and traces. They are now extending that capability into the agentic era with AI-powered investigation and monitoring tools. Anthony Woods https://www.linkedin.com/in/anthonywoods1/ is a co-founder of Grafana Labs https://grafana.com/. In this episode, he joins Matt Merrill to discuss how AI-generated code is straining software operations, why telemetry data volume has become as much a problem as a solution, how Grafana is adapting to a world where agents are the primary consumers of observability data, and what keeps him up at night about where the industry is headed. Sponsorship inquiries: [email protected] mailto:[email protected] The post Grafana’s Approach to AI-Native Observability https://softwareengineeringdaily.com/podcasts/grafanas-approach-to-ai-native-observability/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Advanced software systems have long been more complex than any single engineer can fully understand. Observability is the established solution to this problem. But with AI agents now generating code, deploying changes, and operating autonomously, the challenge of understanding large software systems is entering a new dimension. Grafana is an open-source observability platform and one of the most widely used in the world. The company builds tools that help teams collect, visualize, and act on telemetry data across logs, metrics, and traces. They are now extending that capability into the agentic era with AI-powered investigation and monitoring tools.

• 0:42Speaker 00

  Anthony Woods is a co-founder of Grafana Labs. In this episode, he joins Matt Merrill to discuss how AI-generated code is straining software operations, why telemetry data volume has become as much a problem as a solution, how Grafana is adapting to a world where agents are the primary consumers of observability data, and what keeps him up at night about where the industry is headed. Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in backend development, cloud architecture, and distributed systems design.

• 1:24Speaker 00

  He currently architects and delivers software products and leads a team of engineers at Deft Agency. You can learn more about his work at code.theothermattm.com.

• 1:47Speaker 01

  Hello, welcome to Software Engineering Daily. I am Matt Merrill, and I am here today with Anthony Woods, the founder of Grafana. Before we start, I am going to let him introduce himself.

• 1:59Speaker 02

  Thanks, Matt. Thank you for having me on the show. Yes, so I'm Anthony. I'm one of the co-founders. There were three of us at Grafana Labs. I'm based in sunny Perth in Western Australia. It's a beautiful city. It's just far away from everything, which is okay. I spent a lot of time on a plane traveling around. Yeah, I have a background in tech. I started off as a systems and network engineer, then, you know, kept building automation tooling until one day I found I was just writing code full time and become a software developer. And so that was a lot of fun, you know, being able to bring together all different parts of a system, you know, with tools we can build. And about 12 years ago, Raj, Tocal and I, we started a little company called Grafana Labs, right? Based on Grafana, the open source project that Tocal had created.

• 2:38Speaker 02

  And we took that love of open source and of helping people, you know, understand the data and telemetry that are getting out of systems and turned it into a business.

• 2:46Speaker 01

  That is awesome. I am super excited to talk to you because I am a backend nerd. I've done pretty much backend stuff my whole life. I've always been at the intersection of backend and DevOps and so super excited. So today I want to talk about software engineering operations in the age of AI. And I kind of want to go into how you're doing this at Grafana itself, because that's a company at scale and in this age. And also what you're seeing happening with customers, because I'm sure you're seeing a lot of it. And so I think where I'll start is when you look at how software teams operated five years ago versus today, what do you think are the most significant shifts you're seeing in software operations?

• 3:29Speaker 02

  Yeah, I mean, obviously things are shifting and changing very quickly. That's one of the the big changes is just the speed of change, right? Both from a capability and tooling of what's available, but also just on the demands on engineering teams, right? Like the pressure for them to continually deliver more and faster. You know, we've seen the big shift from building monoliths into, you know, microservices and the benefits that had for teams to have, you know, smaller teams with smaller scope so that they could move faster, right? And that that has really worked, but the consequence of that is now we've got these very complex distributed microservice architectures where no one actually knows how the whole thing works together. They just Might know their own little piece and we've certainly seen that both internally for how we build our software.

• 4:13Speaker 02

  You know, we find that that model works really well for velocity of of development, but also adds that complexity. And so, you know, the way we combat the complexity internally and the way we see our customers doing it is with, you know, better observability tools. It just means collecting more telemetry data to understand, you know, what's happening. We just, we look at all of these small little microservices and they're really just little black boxes that are going, you know, pushed into your production. And observability is that tool that turns that black box into a glass box, right? So when it inevitably breaks and and they always do, you've got visibility to kind of see inside, see what happened, what went wrong. And, you know, that's becoming increasingly important now with the the shift to AI where a lot of the code is not being written by a human. But it's still going to break. And so, you know, you need to have that visibility to know what's going on inside.

• 4:55Speaker 02

  And this is another thing I think that we've seen, you know, 10 years ago, I remember, you know, maybe the conferences, maybe Monitarama or something like that we would go to. You know, there was this big push around like measure all the things, right? Just because people wanted to capture more telemetry. And that really has come back to bite. People would just like the costs, right? Like we've got so much data now we're ingesting, it's expensive to ingest all that data and process it. But also, you know, we see so many of our customers who run into problems where now they just have too much data. And when things go wrong, they don't even know where to start looking, right? They're just overwhelmed by this ocean of data that they've got. So like trying to help people trim that down and focus on what's the important data that you need to be able to solve the problem, not being overwhelmed. I think it's really important. And so that's a good problem to have.

• 5:36Speaker 02

  And then I think the other big thing we're really seeing across the world is is, you know, open source, right? And the advantages, especially on the operational side. We're seeing, you know, technology changes fast and, you know, historically it was open source is a risk for a business, right? Because can you trust it? Do you know what's going to happen? You know, who's going to look after it? Whereas now businesses are looking at open sources as a must have, right? Just because that's where innovation is happening. You look at things like Kubernetes, right? You look at what we're doing with, you know, Prometheus ecosystem, you know, look at Open Telemetry is a great example, right? Organizations are realizing that just to future proof, you know, what they're doing from the operational side, they need to adopt open source technology, open source standards. You know, and these open ecosystems just to be able to get access to the innovation that's happening, but also move away from those kind of vendor lock-ins that people desperately want to avoid today.

• 6:22Speaker 01

  So I am immensely not familiar with open telemetry. So could you say a little bit about what that is?

• 6:27Speaker 02

  Yeah, definitely. So anytime you're building an application and you want to kind of instrument it, right? There's a lot of choices for how you go and do that. How do I, you know, when I'm writing my code, I want to emit logs, I want to collect traces or emit metrics. And so there's been a lot of different ways. A lot of vendors have, you know, provide proprietary kind of SDKs and things that you can use. And so open telemetry is the open standard, right? It's been around for a while, but now, you know, we're seeing where it's got the velocity, it's got the input and people are actually, you know, using it and getting a lot of value out of it. And so it's a growing ecosystem. And so it's really just a standard practice for how you instrument your applications and you collect that telemetry. So it has very good kind of semantic conventions, right? So things look the same. So that way, when you've got different teams working on different projects, building things slightly differently,

• 7:13Speaker 02

  the telemetry that's coming out of it should all look familiar, right? So if I'm not, if I haven't worked in that team before, but I can go and have a look at their logs or look at their metrics or the traces, it should look familiar. Like things should be named consistently where things are understandable. And just having that kind of consistent approach really helps. And obviously the huge advantage here is you don't want to have to be re-instrumenting your applications every time you change your observability vendor, right? It's this thing that you just want to do once and never have to think about again, but still have that ability to go and change your observability vendor over time or add additional tools or different capabilities. So having that open ecosystem, open standards and vendor neutral approach is very attractive for a lot of organizations.

• 7:50Speaker 01

  The HTTP of logging, that's the way, like, yeah, that's awesome. I can't believe I'm not familiar with that, but that is very interesting. And I'm assuming Grafana supports this. Are you supporting the project itself too?

• 8:03Speaker 02

  Yeah, definitely. Yeah. We're, I think, the third contributor by number of commits that we make to the project. We're obviously very heavily involved in it. You know, a lot of the products that we build, so within our Grafana Cloud platform, we have more opinionated solutions of how to do observability, how to do application observability or infrastructure observability. And a lot of that is built on the open telemetry ecosystem.

• 8:24Speaker 01

  Gotcha. Cool. So what you said about open source and open standards to me also intersects with AI quite a bit, which is where I want to kind of spend a lot of our time, right? And so one of the benefits, the many benefits that I see of open source is that it's well documented because it's out on the internet. These models can, for better or worse, scrape all this information and support it very easily without some sort of vendor lock-in or paywall or something like that. Are you seeing that happening already in this space? Or perhaps not? It's not caught up yet?

• 8:58Speaker 02

  No, definitely. I mean, we're really excited with certain Grafana Labs with some of the AI capabilities we've been able to build into the products, or things like our Grafana Assistant, where we had a team, we were worried for a while. We saw AI coming. If you'd asked me 18 months ago, how's AI going to impact operations? I was like, I'm not seeing anything useful. But then we did see that big kind of shift with some of the new models that came out beginning of last year, just like the end of 2024. And so we thought we were behind. We were like, how would it catch up? And then we had a team of just two people during one of our quarterly hackathons who put together the MVP of what is Grafana Assistant. And it was just amazing at how well it worked. It was able to understand what people wanted to do within Grafana and knew how to use Grafana. And so we were like,

• 9:42Speaker 02

  how did you guys do this? Why does it work? And they kind of shrugged and went, I don't know, it just does. But as we thought more about it, the thing that we realized was we've spent more than a decade building this great open source ecosystem, right? Of, we've got 25 million plus users around the world who love and use Grafana, but also they blog about it, right? They write tutorials, they write guides on how to use Grafana to solve certain things. Everyone's got public GitHub repos, which has got their Grafana dashboards in it, and they've got configuration for how they've employed it. They've got a whole bunch of public information. And it's this information that the foundation models from Anthropic or Google or OpenAI are trained on, right? So out of the box, the models know our technology. They know what our users are trying to do, and they know how to solve

• 10:25Speaker 02

  those problems for it, right? So we've been able to kind of leap ahead of our competition by not having to go and train expensive models, right? We can just use the foundation models. And then just build a tighter integration and build more tools and a more kind of integrated solution right into our product rather than having to do all the expensive part of it. So that's a huge advantage, right, that we've found and great, you know, minds we had twelve years ago when we decided to focus on open source. It was a great plan that we'd put in place, but it really has helped. And we see this a lot where even simple things like, you know, asking, you know, Claude Code to instrument, you know, with open telemetry, it knows how to do that, right? It can go and do that for you and, you know, add that to your code and make sure that it's collecting the telemetry that you need.

• 11:04Speaker 01

  That's amazing. It's, it almost seems like good karma, right? Like you've put out this good thing into the universe and now you're getting something back.

• 11:11Speaker 02

  Yeah, I mean, that's it. And it's a big kind of value for our organization, right? Where we just have this, you know, mandate that comes from, from RISE itself. It's like, we just do the right thing, right? We do the right thing for our employees. We do the right thing for our customers. We do the right thing for our vendors. We just think that when you do that, it comes around, right? And people will do the right thing for you. So, yeah, I'm a big believer in doing the right thing and, and, you know, it's worked out really well for us.

• 11:30Speaker 01

  Used to have a saying at a company I worked for is like, just do good work and the rest will follow. And we knew more of that. You don't have to answer this if you don't want, but are you afraid that somehow this might turn around and bite you in the butt and like somehow replace the tools? Like, I don't, I don't personally see how that might happen, but I'm curious if that's on your mind.

• 11:51Speaker 02

  I mean, I'm generally a very optimistic person. So I think there's certainly risk there, right? Like, you know, we can't just sit around and hope for the best, right? We are definitely seeing a change with observability. And I think the main thing, you know, especially for us, which is scary for Grafana Labs is, you know, we've built our reputation on being, you know, the company behind Grafana, the dashboarding and the visualization tool, right? And it's something that users interact with, right? And what we're finding more, both our internal use cases as well as, you know, our customers is that more and more it's not humans that are interacting with the data anymore, right? It's agents, right, that are going in, doing queries, looking at what's happening in your environment, trying to understand and kind of derive some kind of insights from it. And so, you know, we are seeing in the, you know, that shift, right?

• 12:36Speaker 02

  Where there's less about the dashboard itself. And so we're definitely making changes within our product to support that. We announced last week. We've got our new GCX project, which is a kind of unified command line tool, right? And it's designed for agents to be able to interact with our Grafana cloud service and all of your data and access it that way. But that said, I still believe that one of the things that's really important when it comes to AI giving you, you know, information is the trust but verify kind of piece of it, right? Where it's pretty smart, it's pretty clever at what it can do, but sometimes it gets it wrong. And so you need to be able to have that information that you can go and look at to say, hey, how did you come to this conclusion? You know, how did you make this decision, et cetera? And often the best way to remember that is still a graph, right?

• 13:20Speaker 02

  It's like, hey, and so we have that in Grafana Assistant when, you know, it'll do an investigation for you. It'll come back and explain to you, hey, I went and looked at this data, I saw this trend or I saw this spike. And so that's why I've then gone and, you know, down this rabbit hole to go and look for this problem. And so, you know, being able to just explain that with a graph is really easy, right? And that's something that that the user is still going to want to be able to consume.

• 13:40Speaker 01

  That makes sense. It makes a lot of sense that, you know, at the end of the day, mostly what you're looking for is the answer to a problem or a question that's causing a problem or a pattern or something like that. Are you doing things proactively? Is the tool looking at these patterns proactively at this point or?

• 14:00Speaker 02

  Yeah, I mean, so we have a few different things that we do. So like the age of AI, I think we're in it, but there's a lot to it. I mean, everyone kind of now just associates it with LLMs and kind of generative AI. But there's still a lot of other AI practices that we use internally to be able to get better quality data, right? Because the LLMs are great when you can give them a lot of context and kind of quality data. But it's the classic, you know, junk in, junk out. So you want to make sure that they are getting kind of clean data coming in so that they can make the right decisions. And so we use a lot of kind of AI tools internally to understand your telemetry data. So one, you know, a couple of things we do that for is you have something called a knowledge graph, right? Where we kind of look at all the telemetry and build a graph of all the different entities and how they relate to each other, right? So that you can say, hey, I've got this application. That's great, but I know that it's running on a pod, right?

• 14:45Speaker 02

  And that pod's running on a node and that node is in a certain cluster in a certain region. And so you can kind of know the relationships and that's great when we, you know, hand that off to the. L L M's because then suddenly when they do see a problem, hey, I'm seeing a latency spike for this application. They then can traverse the telemetry and understand, okay, well, I can find what node this is running on. And it's like, oh, oh, look, I can see that that's got a resource contention problem. Oh, look, your latency spike is because you've got a noisy neighbor who's consuming all the resources, right? And simple things like that, it sounds simple to us, but you know, being able to understand all of that telemetry becomes really important. And the speed at which things are changing, right? Like, you know, in the old days, you would just have like a C M D database and be like, oh, I can describe all my assets here and it's a static thing, right? And the reality is every time you have one of those, they're out of date by the time they get updated.

• 15:30Speaker 02

  And so we wanted something that was dynamic and just generated by the data that's actually being ingested in real time. So that way it's, it's always up to date and it's always a representation of what your environment actually looks like.

• 15:41Speaker 01

  So if I'm hearing you right, like this is different layers and different applications of different machine learning.

• 15:47Speaker 02

  Exactly.

• 15:48Speaker 01

  Gen like A I models that feed into reasoning models that can help you make, identify those patterns and things like that. That's awesome.

• 15:55Speaker 02

  Yeah. And I think that's really important. I think that's what we're seeing is just that the L L M's are very smart, but you have to give them the right data for them to be able to come to the correct conclusions. And so that's really what we, we, you know, put a lot of effort and a lot of focus on is how do we make sure that we're curating the data set properly and we can use a lot of machine learning kind of tools to do that. As well as leveraging things like the open standards, right? And open ecosystems, right? Leveraging things like open telemetry means that, you know, we know the naming conventions and the semantic conventions, so we know what the data is representing, right? And that's kind of public information. So the L L M's can understand it as well.

• 16:28Speaker 01

  That's awesome. So just to pivot slightly, I think one thing on my mind is observability has been about humans understanding what's happening in a system. But as we use L L M's more in the applications themselves, we start to lose a lot of control and understanding of what's happening there. So how is operations changing with that paradigm shift right now that you're seeing with your customers?

• 16:54Speaker 02

  Yeah, we're definitely, you know, from our perspective, we see the shift to this kind of like agentic model of software design, right? It's just a new design, you know, framework, you know, we had mainframes, right? And then, you know, we had monoliths and then we've moved to microservices. And now we just see this agentic is just a new way of building software, right? It's got some different problems that are introduces of how we understand what's happening. But the tools we have, for the most part, get you a long way there, right? Like, as we think about how do we understand what these agents are doing, it's typical kind of, you know, application observability or APM, right? Like, you want to be able to collect that telemetry, know what's happening. But then now we've got some new types of data and things that are coming out that we want to collect as well, right? So a lot of like the conversations, right, to understand. So we can go and write our evals and do interesting things. So, hey, are you actually giving correct responses?

• 17:39Speaker 02

  And then also when things go wrong, you know, trying to understand, well, how did it get there, right? What was the the input that kind of caused it to kind of go off the rails? And so we're seeing definitely demand, right, across the market where people want tools that can just do this, right, that can just build into their agent framework. And so that's something, you know, we've had to do internally as well, right? As we were building out, you know, our agents and capabilities, right, we needed to kind of understand this. And, you know, as we like to do at Grafana Labs, we just, you know, we're a company who just keeps scratching our own niche and solves our own problems. And then, you know, we share that with the world as products. And so we did the same thing, right? So, you know, we announced last week that we now have in Grafana Cloud, our AI observability capability, which does does this, right? So it builds on top of open telemetry to collect, you know, your raw telemetry and then collects additional things, right? Like, so the conversations and other data that that's really important.

• 18:24Speaker 02

  And then builds, you know, an experience where you can go and have a look at the things that are important to you when you're building agents, which right now is like reliability and quality as well as cost, right? How many tokens am I burning, right, for all these conversations, right? And which certain kind of workflows within my stack are the ones that are, you know, using the most tokens, right? And that could be a good thing, right? You know, often we we care about, you know, token usage because we want to see adoption, right? We want to see people using the tool. You know, we're in the kind of phase now, the growth phase, certainly for AI, but there's going to come a time in the near future where suddenly people are going to be like, wait, let's think about how we can spend less on our tokens. So, so we want to kind of give that visibility. Yeah, we want to give visibility, you know, to our users so they they can understand what these systems are doing, right?

• 19:06Speaker 02

  And be able to kind of dive in when things do not go as planned, be able to kind of have the the data they need to understand what went wrong so that we can make changes, right? So that we can improve things over time.

• 19:16Speaker 01

  Yeah. One of the questions I had was like, what you're seeing your customers, like, what does it mean to monitor an AI feature in production? And I think you explained a bunch of that, but what I'm wondering too is what are those hooks? What are those hooks into something like open telemetry? Are you know, like, if I'm sitting there thinking about how I might prompt a model to do something to me? Are people saying like, as part of what you're doing, log out your reasoning steps to open telemetry? Or something like that, where you have another agent that does that? Or is it more deterministic coding before and after calling? Like, what are you seeing in terms of those? Are they integrated with the APIs themselves in some way too?

• 19:58Speaker 02

  Yeah. So for a lot of the, there is like some good integration. You think about a lot of the tool calls, right? They're not really any different to any other kind of RPC call that you're making, right? Where you're going to want to measure, is it actually getting a response? Right? Am I getting valid responses coming back? You're going to want to look at latency, right? How long are these things taking to respond, right? Because that's going to impact the user experience. And you know, you talk about what does monitoring mean? Or what does observability mean? Why do we do it, right? The main reason for it is to make sure that, you know, we're meeting our customers' needs, right? Or our users' needs, you know, meeting their expectations. So one of the ways we do that with observability and the simplest approach we have is around the concept of SLOs, right? So service level objectives, right? So you, rather than saying, hey, I'm going to go and monitor my CPU usage, right?

• 20:41Speaker 02

  Because do I care if I'm using a hundred percent CPU? Not really, right? Because I'm paying for a CPU, shouldn't I be able to use a hundred percent of it? What I actually care about is what is the quality of the service I'm giving my user, right? Am I seeing a latency spike, right? That's something that I care about. And so being able to kind of pick what are those kind of core, you know, user experience things that I want to make sure my application is delivering on. So reliability is a simple one, but even like latency, depending on what your business is, right? You're going to have your own objectives. And so those are the things that you're going to want to measure over time. And then as they change, and that's when you know something's wrong, right? If your users are constantly getting 500 errors, well, you know, something's gone wrong, right? Or if you're, you know, maybe something simple where, you know, if you see traffic drop off, right? On your site, maybe that's an indication that something's wrong and you want to have someone go and look at it. Or you want to have an agent.

• 21:26Speaker 02

  Go and look at it and give you an idea of what it might be. But there's ways to kind of collect all this data today where you can, you know, a lot of logs, a lot of traces, et cetera, where you can bundle this data. One of the challenges with traces, for example, one of the challenges we have with traces with, you know, this agentic model is just the volume of data. So like those conversations that you want to have usually exceed what a trace can reasonably contain, right? Within a single span that it can send. So we're having to use kind of supplemental, kind of like data stores to store that kind of information. And this is where we see then, one of the nice things we like to be able to do in Grafana, right? Is, you know, we've built this ecosystem around integration and interoperability, right? We call our big tent philosophy, right? Being able to kind of bring data together from lots of different types of data sources and be able to kind of draw those correlations.

• 22:09Speaker 02

  And so we're able to do the same thing. We can go and build a new database. It could be, you know, a SQL style database that's storing all this information. But within Grafana, we can just stitch that together and be able to have you kind of, you know, correlate between my traces, right? Where I've got all of my traditional kind of APM data. But then be able to also tie that to all of the chat history or all of the other kind of context information that you want to go and store and be able to then visualize that in one place.

• 22:35Speaker 01

  Nice. Cool. So back to what you're doing inside of Grafana itself. So you mentioned that awesome hackathon assistant thing. In terms of the just day to day operations, where are you seeing AI have the most operational wins for you? Maybe it's even outside of observability of itself.

• 22:57Speaker 02

  Yeah, I mean, we're definitely very bullish on AI capabilities across the organization, right? Like we have a mandate of like everyone, you know, try and use AI wherever you can, right? So we can take advantage of it where possible. Obviously, we're seeing huge adoption and use within, you know, software development, right? Like it is great for building software, for getting code out there and just being able to innovate quickly. We're actually quite fortunate because we see kind of like the role of a software engineer is changing, right? It's less about the code and it's more about, you know, engineers are becoming more product managers, right? Where it's really about kind of understanding the user problem and then being able to explain that to your agent and we'll go and build the code for you. And that's something that we're again fortunate with at Grafana Lab. Where we've always had this kind of bottoms up kind of culture where our engineering teams have been responsible for the roadmap, right?

• 23:42Speaker 02

  Like we have a great product team, a great product managers, but they're there to facilitate. So they bring the information in, you know, from our customers, they bring information for what's happening, you know, in the industry. And we can see and then come and facilitate the conversations. But at the end of the day, we like our engineers to be responsible for the product roadmap, right? Cause we want them to own it. Part of that is because we are building products for engineers, right? So they do have good kind of, you know, intuition, insight is needed. Yeah. And so, and so we've always had engineering teams and had a big focus on giving them autonomy to go and make decisions about product. And to think about what are the needs of our customers and how can we go and solve that? And so that's now helping a lot with the tools that we've got where they are already experienced at being, you know, kind of mini product managers themselves. And so they're able to kind of leverage the tools quite effectively to go and build things and get new products shipped and delivered.

• 24:28Speaker 02

  The other area we see it is where it's interesting is obviously more on the operational side is, is investigations, right? Like understanding what has gone wrong. So again, you know, I talk about the Grafana assistant started as MVP. You know, one of the things that we like to do with our products, you know, we're scratching our own itch. And before we obviously release products to our customers, you're going to make sure there's good product market fit. And the way we do that is we just make it available to our internal teams. And if they start using it without anyone telling them to, then we know it's a good product. And we saw this with the assistant, right? Soon it was in there, people started using it, they get paid for something, they just send the assistant, hey, can you go and tell me what you think it is, right? And it's very like, it's remarkably good at doing those investigations and going through and finding the problems. So that's been really powerful. So we leverage that a lot to go and understand it's not always right, but often even if it's not right, it's going to kind of reduce

• 25:14Speaker 02

  the surface area that you need to go and look at, right? Cause it will come with some suggestions or it might exclude some certain problems, right? So it really does accelerate our team's ability to go and find that root cause and solve those problems. The other area that we're seeing more and more is, which is a little bit scarier, right? Is, you know, as we see AI shift from just the development side into more of the deployment side, right? Like, you know, when do we start giving our agents keys to, uh, go and just, you know, you've written the code, why don't you just go and deploy it?

• 25:40Speaker 01

  This is where I get uncomfortable. Yeah, yeah. I get uncomfortable, yeah.

• 25:44Speaker 02

  I mean, it's a scary prospect. But I don't think we're far away from it. I mean, obviously, we see a lot of things in the news cycles, right? With, you know, agents going deleting people's production databases, right? So people obviously a little bit scared about these kind of things. But me personally, I don't think we're that far away from it, but I'll caveat that a little bit. And I think one of the things that we see is that, you know, we don't inherently trust the AI to do the right thing. That's okay, right? We shouldn't, right? We should be kind of cautious because if there is an opportunity for it to do something wrong, it will find that opportunity eventually. But for me, I don't think that's any different to how we treat people, right? The number of times I've said to someone, oh, we don't need to worry about the normal kind of process. It's just a small change. It's not going to affect anything. I'll just deploy it in production, right?

• 26:28Speaker 02

  And then it's like, oh, oops, I didn't expect that was going to happen. And so we always, we need to protect ourselves from ourselves already today. And so like, you know, certainly at Final Labs, we have a whole bunch of tooling in place in our C I C D pipelines that put physical gates in place where it just prevents you from being able to do the wrong thing, right? You can't go and do a change where the blast radius is global, right? Like, it has to be small blast radius, limited impact. So if something does go wrong, it's not going to impact all of our customers all at once. You know, we have things around cost management, right? When we do a change, we have a C I check that runs and says, this is going to increase spend by ten X, right? And you can't do that, right? And so like, we have got all of these physical gates in place that prevent people from making mistakes, right? And that also works then for agents, right?

• 27:11Speaker 02

  Like, once you build these gates in place where they're hard blocks that stop the agent from accidentally shooting itself in the foot, you get closer to this reality where you can start to let it do some things in your environment. And I don't think we're far away from that. But I think that having those controls in place is an organization maturity thing, right? Like, it's taken us a long time to build that, right? Because we add new checks in place every time something goes wrong that we didn't expect would go and happen, right? And that's easy enough to do with people, right? Because we move much slower. The scary thing for an agent is that they can move very fast, right? So they can make a lot of mistakes very quickly. And so we're going to have to make sure we've got the right kind of controls in place. And so that's going to be the, I think, the thing that will allow us to give the agents a little bit more control to go and deploy things, you know?

• 27:56Speaker 02

  They can write code, they can go and deploy it, they can observe it and see is it having desired effect, is it working correctly? If it's not, make some more changes, go and deploy it, right? And just shorten that feedback loop so that they can base their changes based on direct feedback from users and what's happening inside the production environment. But it's a scary thing, but I also think it's exciting, right? I think it's an exciting challenge, right? Because we want to get to that place where we can free it up. Because right now, like as software engineers, you know, often our, our job is to balance the, how do I deliver faster? And balance that against how do I keep the system reliable and meeting my customers' expectations, right? And so we can write a lot of code right now, we can develop a lot of new features with the AI, but we're still kind of bottlenecked on how quickly can we get that in production and validate that it's actually working?

• 28:41Speaker 02

  And so I think once we, we can get past the like, where we can give agents a little bit more control, then that whole cycle then accelerates a little bit faster. And so we can then ship faster and we can get better insights and kind of keep iterating and going. And it's a scary prospect and I think there's going to be a lot of pain along the way, but I do think it is coming. We just have to find ways to do it in a controlled, safe as possible approach.

• 29:04Speaker 01

  Yeah, there's definitely a, a risk analysis that needs to happen there and how you step slowly into that. And if you have the right checks and balances, I can't wait for some of the stories that'll come out from this. It's going to be unbelievable.

• 29:17Speaker 02

  I'll give you an example of what we're doing in Grafana Lab. So one of the things we're automating is, is when we see, you know, we do a deployment, right? And be small scoped into a, you know, specific subset of users, right? And then we can go and look at that and make sure it works. And so we have some automated checks that we can leverage Grafana Assistant to go and say, hey, are there any problems or the new errors showing up, et cetera? And if it does find problems, we just have it automatically open a PR to rollback, right? And so those kind of actions we're seeing kind of creep in, you know, we still have a user look at that and approve that PR before it happens. But it's, it's a simple thing. It's like, oh yeah, this is just a simple rollback, can approve that. And we'll just roll back to the previous release and we can then go and fix it. And so we are seeing these things where we're slowly giving the agents a little bit of operational control,

• 29:59Speaker 02

  because they can do things much faster than we can, but we still need people in the loop right now.

• 30:03Speaker 01

  Yeah, self healing system. It also depends on where that change or what change it's making, what it's affecting. Like, I don't know how I'd feel about a agent deploying like a data migration on millions and millions of rows of data, but making a front and fix, something like that, something that could be easily rolled back. Hey, why not?

• 30:22Speaker 02

  Exactly. Yeah. And I think that's where we're going to start. It's those safer operations. We're going to be more comfortable, right? Giving the AI tools. But, you know, we'll definitely see there'll be a bunch of young startups where they're very risk tolerant. Like, I just care about moving fast, right? And they'll go nuts, right? They'll just let it do whatever it wants. And some databases will get deleted. But that's okay.

• 30:41Speaker 01

  Hopefully not important ones. So, so let's talk about the people using these tools, right? So as AI handles more of this operational work, how do you see the role of SRE, DevOps engineer, platforms engineer changing? Do you think they're getting bigger, smaller?

• 31:00Speaker 02

  I mean, I think right now they're bigger, right? Just simply because the volume of new software getting deployed is just growing so quickly, right? That said, I think they're getting left with, which is actually kind of, for me personally, right? Like, they're getting left with the harder problems, right? So a lot of the toil, the simpler work that the SREs used to do can be done by the AI now and be done by the agents now, which then leaves the SREs to focus on more high value kind of pieces of information to wrap the hard problems, right? How do I go and diagnose this? Or, you know, thinking more strategically around like, how do we build a more reliable way of doing things, right? More kind of process oriented or, you know, go and find, you know, how that AI is not supporting the needs of the business or, you know, helping you kind of drive things forward. So right now, I think there's just growing need for it, right?

• 31:42Speaker 02

  We're shipping a lot of codes. It's of questionable quality sometimes. So we just, we do need more of that kind of SRE kind of practices to, you know, make sure we are thinking about reliability, make sure we understand, can we measure it? Are we making sure that, you know, as we're building cool features and new tech, is it delivering the value that we want it to have, right? And meeting our customers' expectations? And most importantly, when things go wrong, do we have the data to be able to understand what happened, right? What went wrong? So we can go and fix it. So I think that's definitely the case. As that changes over time, I think, yeah, we're still going to need those practices. The thing that scares Me the most, both from an S R A perspective is also software development is just that because all of the easy things, right? Are being kind of taken care of. That's kind of the work that you give your junior people, right?

• 32:24Speaker 02

  Or you grads where you just have to go and do it and you kind of, as you're doing it, you learn, right? You learn how these systems work, you learn what to do. But as we take that opportunity away, it's like, how are we going to give people the opportunity to develop the skills that they need to become? These experienced kind of senior S R Es that we're still going to need, but they're not going to have that path, right? So that's the thing that scares me a little bit is what's going to happen in 10 years' time when we realize, oh my God, we just don't have anyone who has the skills that we need. It's all right. I don't know how we fix that. I think it's the same with software development, right? Like, we still need real experienced software engineers, but the new grads and kind of juniors aren't getting the opportunity to develop their skills where they can have that experience, right? Where they've learned things the hard way and have gained that experience, right?

• 33:07Speaker 02

  They've earned it by making a lot of mistakes along the way, but you know, we're taking that opportunity away from people. And I think that's the scariest thing for me when we think about the future.

• 33:16Speaker 01

  The high level thinking that's needed just keeps getting more and more and more. And you got, it goes from reactive and detail oriented things to proactive, high level thinking. Where, yeah, and yeah, I'm in the software engineering game. We're seeing the same thing. I saw an interesting, I wish I knew who the author was, but I saw an interesting thing of one potential way out of this is to go back to like a apprentice journeyman master model, which I am a huge fan of software as craft. And I thought that that was really interesting where it's like, the only way that you are going to get this experience is by sitting shotgun to an experienced person and feeling the pain with them and seeing their line of thinking.

• 33:58Speaker 01

  I think that's especially interesting with S R Es because you're reacting to these potentially nasty problems and trying to unravel that. And so I thought that was a really interesting one.

• 34:09Speaker 02

  I think it's a, it is an approach. I think the scaring challenge is like it's an expensive approach where you're not getting immediate value from, right? And I think a lot of businesses and organizations are going to be like, I'm sure, why don't we just let some other companies? Be the ones that train all the people and we'll just keep using the agents, right? So I think it's going to be very difficult to find that. I think one area where we do see this opportunity is still open source, right? Like we, you know, we'd love to kind of engage with our community. We do get a lot of young people who come and contribute and that's our opportunity to kind of coach them, right? And being open source, right? We're happy to invest that time, right? To nurture those people so that we can, you know, help them develop their skills over time. So I will see there will be more of those like community driven, right?

• 34:51Speaker 02

  Kind of be consistent because I do think this idea of training people up to be productive so that they can leave and go get a job at another company, right? It's not going to be an attractive option for a lot of organizations.

• 35:01Speaker 01

  Good point.

• 35:03Speaker 01

  Yeah.

• 35:03Speaker 02

  So I do think more of that, like as software engineers, as a community, I think it's our opportunity to find ways to kind of, you know, nurture those, those people coming through and give them opportunities.

• 35:11Speaker 01

  Yeah.

• 35:12Speaker 02

  I think open source is a great way to do that, right? There's big code bases, right? There's opportunity to write code. There's ways we can kind of communicate and share ideas and help kind of train and coach people a little bit to give them the experience that they're going to need.

• 35:22Speaker 01

  Well, also not all apprentices in the trades are paid either, right? And so we've been spoiled in this tech industry, but high paid internships that lead you to a high paying job. The playing field may be leveled, but yeah, good points.

• 35:38Speaker 02

  Yeah. But I think even like a lot of the people I talk to, you know, colleagues I've got, you know, even still people we hire today. Like I love to go to, we do our onboarding, you know, in person and get people out to go and talk to the new starters of what's happening. And a lot of them have similar stories backgrounds to me, right? Where they're very self taught, right? And I think that's true within this, this industry where a lot of people love the technology. They're excited about it. They love the fact that things are changing all the time and there's always new things to do. And so you have to have this mindset of wanting to, to go and, you know, teach yourself and learn new skills. And so I think that's still going to happen, right? But I think, you know, open source is a great avenue for that, where it's an opportunity for you to go and do something off your own bat to develop those skills that you're going to need in the future. Because the speed at which things are changing in this industry, right?

• 36:18Speaker 02

  You, you have to be committed to a life of learning because, you know, you blink and, you know, there's a new JavaScript framework, so.

• 36:26Speaker 01

  I hadn't planned on talking about this, but I feel like it could be fertile ground. So are you seeing open source projects struggle with the amount of code that agents are producing? Or what are you seeing in your area?

• 36:37Speaker 02

  A little bit, both from the volume as well as the quality. Yeah. So it's very easy to have an agent just go and build a feature, right? But just because it works doesn't mean it's code that you want to maintain over time. And so that's, I think, where we're seeing the challenges. I think this is still where there is, you know, that challenge with the AI assistant kind of code tools where we put a lot of thought into kind of like the architectural design and guiding principles for how we build code, right? And it's every organization is different, but you end up building your view of the world and how you want code to look, right? So it's got consistency. And so it's much easier for your teams to kind of like people to move around between your teams, right? Because the code bases are all similar kind of layout. You're using similar kind of

• 37:19Speaker 02

  practices for how you go and build code. And so things look a little bit more consistent. Whereas when you start getting the AI in, it doesn't care about those kinds of things, right? It's just going to find a solution that works and go and throw that code in. And so there's a lot more work that has to go on to think about and care about maintainability, right? So that certainly affects open source, right? Where you don't just want to accept any pull requests, right? That adds any new feature, right? You want to think about, is that feature right for this project, right? Is that something that we can maintain over time? How does that impact kind of the rest of the ecosystem? And so, so that's where we're seeing the challenge is that, you know, you'll get people that'll just submit a pull request, you know, that'll be 20,000 lines of code that, you know, the AI is generated for them. And you're like, well, no, right? We're not going to just to review this. It's too big. It's too much. You know, this is not how

• 38:04Speaker 02

  we build software. You need to kind of break this into smaller chunks and, and so that it can be reviewed. And so it can be maintained over time and understood. And so I think that's the, the challenge that we're seeing is code is a collaborative business, right? Especially for, you know, open source projects. And so the agents have to fit into that collaboration mindset, right? Of, of not trying to overwhelm people with too much change all at once, because it's just not going to get accepted, right? It's too hard to review. It's too hard to maintain over time.

• 38:30Speaker 01

  And to your earlier point, these things might be learning moments for people that are less senior too.

• 38:35Speaker 02

  Yeah, yeah, definitely.

• 38:37Speaker 01

  Yeah. So let's move towards the future a little bit. So what's the operational problem that you think AI and these tools are going to crack in the next two or three years that people are still doing manually right now, even right now with the existing tools that we have?

• 38:52Speaker 02

  Yeah, I mean, I think right now we're at the stage where AI can be that first responder, right? When something goes wrong in your environment, right? So an alert gets fired because you've got an SLO burn and it's like, hey, like, you know, error rates have spiked, something's wrong. You're going to page your on-call person, but at the same time, you can go and spin off, certainly in Grafana Cloud, you can go spin off in Investigation, say, hey, go and try and work out what's wrong, right? By the time your on-call engineer gets in front of their PC, right? There's probably already a response to say, hey, this is maybe what the problem is, right? They may have worked it out or it may have excluded a whole bunch of things. So having that, I think, is where we're going to see over the next few years, and that's just going to get more reliable, more accurate over time. We're going to build more playbooks, we're going to build more skills for it so that can do more interesting things.

• 39:35Speaker 02

  But I think that's where we're seeing it is that first responder, as soon as you notice something wrong in the environment, it will go in and it will try and troubleshoot that and find that root cause for you. So that way, instead of it taking an hour, right, to find that root cause and kind of solve the problem, we're now talking in 10 minutes, right? We can go and get things patched up and move on. We're still going to have the follow-up process, right? Which, you know, is so important for our SRA is to go and actually, don't just stem the bleeding, let's go and fix the problem. So that's still going to be important. But again, you can leverage to AI tools to kind of process more data and kind of do those kinds of things. But I think the near term, so the next couple of years, that will just be standard, right? Like you'll just, there'll just be an expectation that 60, 70% of the problems are kind of resolved before your on-call person even gets in front of a computer.

• 40:18Speaker 01

  Yeah, because right now for resiliency in systems, we rely on redundancy and things like that. But I think what you're saying is what we're starting to trend towards is actually, in a limited sense, but a self-healing system, a dynamic self-healing system, which is pretty amazing.

• 40:36Speaker 02

  Yeah, and I think that's going to be really important just because of the speed at which software is being shipped, right? If you want to move fast, right, you're going to inevitably break sooner, right? And more frequently, right? Because you're trading speed for accuracy, right? And so the need for, you know, better observability just becomes more apparent, right? You're going to want to make sure you've got the data that you need so it can go and fix. But also, we're just seeing that AIs are being able to use that observability data to understand, you know, what went wrong. And that feedback can be really fast. And that's just going to help us be able to ship things faster, but also maintain the level of reliability that, you know, our customers expect from us. And I think the other thing that we're seeing AI very good at is obviously processing huge volumes of data, right? Going and looking for those kind of trends, those hidden things that a user hasn't noticed.

• 41:21Speaker 02

  We do a lot of that internally where we can just say, hey, just go and query a huge volume of logs and find what's interesting. So we're seeing that use case change where, you know, we recently, you know, made changes to Loki. So our log aggregation system to, to just be able to do that smarter and better, right? To fit with that use case of what AI agents want to go and do. So we've, you know, made a whole bunch of architectural changes and performance improvements so that it can query 10 times faster. You know, more importantly, when you give it a query, it can reduce the volume of data that it needs to go and scan and look at to go and do that. So, so that's.

• 41:52Speaker 01

  I'm curious at a high level, what those architecture changes look like. That sounds like a very interesting problem.

• 41:59Speaker 02

  Yeah, I mean, so one of the things. I can, yeah, yeah, we announced it last week at Grafonicon. So it's in the open source, right? So, yeah, I can definitely talk about it. I mean, I'm allowed to talk about it, whether I understand it as well as the people who have written it, that's a different question. So what we really think about is, you know, Loki was designed for cost efficiency, right? So how do we store huge volumes of logs as cheaply as possible? And, you know, the way we did that is just like, just don't index them, right? The use case we designed for was the developer use case where you just want to kind of grep, right? Basically through your logs, right? grep, actually, I, you know, look at the log messages, I'm like, it's not that message. And you just keep excluding things or you might have a, oh, I'm looking for a certain, you know, string.

• 42:37Speaker 01

  Process of elimination.

• 42:39Speaker 02

  Right? And so, so that's what we kind of optimize it for. But more and more, we see people wanting to do more kind of analytic style queries or needle in a haystack queries more. Often where they're like, hey, I've got a transaction ID and I knew it happened sometime over the last four days. Go and find it everywhere in their logs. And so Loki was designed, you know, to be cheap to ingest data, right? And but we wear the cost on the query side, right? Where we just root for. So we'll just go and look at all the data and and look what what it's trying to do. And it can do that quite efficiently, but it still takes time, right? When you're talking about terabytes and terabytes of data that you want to go and process every second. So one of the big changes we've made is we kind of introduced kind of similar to like Bloom filters, but it's a slightly different kind of approach where we can then know which portions of the data to go and look at, which ones we don't need to worry about, right?

• 43:21Speaker 02

  So if you're doing that needle in the haystack query, we could very quickly say, hey, actually, you don't need to go and look at the petabyte of data we've got. You actually only need to look at these 10 terabytes that we've got here and it can do that very, very fast. So that's one of the ways. The other way is we've just put a lot of effort into like the query engine itself, just to optimize it and just make it faster. Being able to paralyze it as much as possible, because that's easy to do, right? With compute systems today, right? Where we, you know, we've got a huge volume of compute where we can just go and spread out the workload and get things done as as fast as possible. That works insanely well for us as a as a cloud provider, right? Where we've got the benefits of kind of statistical multiplexing, right? Where we can, you know, we've got thousands of users all using it, right? We can provision a very large infrastructure estate and, you know, not every user's querying it all at once, right?

• 44:05Speaker 02

  So we can say, hey, this one user's querying it this second, they can go and use a thousand cores. And then the next second, a different user uses a thousand cores. And so we can get really fast response times because we've got a bigger environment that we're ready to kind of build on top of.

• 44:17Speaker 01

  Efficiency of scale. Yeah.

• 44:19Speaker 02

  Exactly. Yeah.

• 44:20Speaker 01

  Man, I would love to just talk about that for an hour.

• 44:24Speaker 01

  All right. So a couple of wrap up questions. This one I'm really curious about. So what's an operational scenario that keeps you up at night, especially related to AI? Something that you think the industry isn't taking seriously enough right now?

• 44:37Speaker 02

  Well, that's a good one. I think one of the challenges for me is just, you know, a lot of the AI agents that are being built today are just black boxes, right? And we don't know what they're doing. Sometimes we don't want to know, you know? So I think that's the scary thing is the speed at which these systems can operate and make changes and do things, right? You know, because we've moved from the chat, right, to this agent model, right, where it's like, let the AI do things, right? And not just respond to questions. So that scares me, right? When you've got these black boxes that are just, no one knows what they're actually doing, right? We have a mandate, you know, internally where our engineering teams are leveraging AI wherever they can to build software, right? But at the end of the day, the person who merges that, they're responsible for that, right?

• 45:18Speaker 02

  Like, they're accountable for that. And so I get scared about when that starts to erode, right? And it's like, well, who's actually accountable for the things that are getting deployed into our production? I was having a conversation, someone actually just yesterday, right? Talking about kind of like this governance problem, right? We're seeing more and more agent-to-agent communication, right? What happens when you've got, you know, an agent in your organization, talking to an agent in another person's organization? And they're just doing things, right? And something goes wrong. Who's accountable for that, right? Whose fault is that? Who's responsible for kind of remediating that and take accountability? So that's the thing that scares me the most is around, well, I guess two things. One is the black box aspect of it of like, what are these things actually doing? And the other one is the accountability. Who's responsible when things go wrong?

• 46:01Speaker 02

  If we are giving all this autonomy to our AI agents?

• 46:05Speaker 01

  Some really interesting legal and contractual things that'll come out of that as well. Yeah, that's a really good point. It always comes back to humans anyway, right? You know, like, what's the throat to choke if something goes wrong?

• 46:16Speaker 02

  Yeah, that'll be the most lucrative career in the future, right? Where it's just AI scapegoat, right?

• 46:22Speaker 01

  And, and

• 46:25Speaker 02

  Yeah, I'll take responsibility.

• 46:28Speaker 01

  AI insurance. Ooh, okay. Hey, you want to make, want to make a new startup? No, I'm just kidding. So, all right, let's close with, I always like to leave the audience with something. So if you could give one piece of advice to our audience about how to evolve their operations for an AI-first world, what would it be?

• 46:45Speaker 02

  Yeah, definitely. My strong opinion about this is don't go and build an AI solution. Make AI part of the solution, right? So we've got so many great products out there. You don't need to go and write a new product, right? That's AI first or something. Really where we see the value in AIs is when it's behind the scenes, right? When it's not in your face, there's just, things just seem to magically work, right? And you don't think about it and you don't know why. You just know that it works, right? And you got the response that you want. And as we think about AI, I think that's the approach that we want to have is leverage its great capabilities. But it doesn't need to be the only thing, right? That you're doing, right? I think, you know, you think about when the internet came along, right? It was a new and shiny thing and everyone was like, great. It's now internet enabled, right?

• 47:28Speaker 02

  Now we don't care, right? It's just an expectation that things should just work and it happens to work because it's leveraging the internet and the technology that we have. I think AIs, we're in that same phase. Right now, we're in that hype cycle where everyone wants to label everything with, oh, it's AI, it's AI, it's AI. We're really, users are going to care less and less of that over time. What they actually just care about is, does it work, right? Is it doing the thing that I want it to do and does it make me happy? So like, I think that's the, the focus is just making sure that your AI kind of works with the existing kind of workflows. It's coming to meet people where they are and help kind of accelerate what they're doing and not trying to force them to to have to go and do things they don't want to do.

• 48:03Speaker 01

  Well said. Well said. Well, thank you very much. Really appreciate your time.

• 48:09Speaker 02

  No worries. Thank you.

• 48:10Speaker 01

  Learned some very interesting things. Thanks for being here.

• 48:13Speaker 02

  Thanks a lot, Matt. It's been a pleasure.