Agentic DevOps at AWS

AI agents have become capable of reasoning across large amounts of data, calling tools, and taking sequences of actions autonomously. These qualities make them well suited to some of the most persistent pain points in DevOps, including the on-call engineer woken at 3 AM to diagnose an incident, the build failure that takes hours to trace back to a root cause, and the operational toil of modern software delivery. Agentic DevOps is the emerging paradigm that applies these capabilities to the full software lifecycle, with the goal of matching the velocity of modern software delivery with an equally capable operational layer. Neha Gaswamy https://www.linkedin.com/in/neha-g-b3b4445 leads Agentic DevOps atAWS https://aws.amazon.com/ and has been at Amazon for over twenty years. In this episode, she joins Matt Merrill to discuss the AWS approach to agentic DevOps, how Amazon dogfoods its own DevOps tooling, how their DevOps agent works from alarm to root cause, why determinism still matters in an agentic world, creative MCP integrations customers are building, and what the future holds for SRE engineers as agents take on more of the operational work. Sponsorship inquiries: [email protected] mailto:[email protected] The post Agentic DevOps at AWS https://softwareengineeringdaily.com/podcasts/agentic-devops-at-aws/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  AI agents have become capable of reasoning across large amounts of data, calling tools, and taking sequences of actions autonomously. These qualities make them well-suited to some of the most persistent pain points in DevOps, including the on-call engineer woken at 3:00 a.m. to diagnose an incident, the build failure that takes hours to trace back to a root cause, and the operational toil of modern software delivery. Agentic DevOps is the emerging paradigm that applies these capabilities to the full software lifecycle with the goal of matching the velocity of modern software delivery with an equally capable operational layer. Neha Gaswamy leads Agentic DevOps at AWS and has been at Amazon for over 20 years.

• 0:45Speaker 00

  In this episode, she joins Matt Merrill to discuss the AWS approach to Agentic DevOps, how Amazon dog foods its own DevOps tooling, how their DevOps agent works from alarm to root cause, why determinism still matters in an agentic world, creative MCP integrations customers are building, and what the future holds for SRE engineers as agents take on more of the operational work. Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in backend development, cloud architecture, and distributed systems design. He currently architects and delivers

• 1:30Speaker 00

  software products and leads a team of engineers at Deft Agency. You can learn more about his work at code.theothermattm.com.

• 1:52Speaker 01

  All right. Hey, everybody. I am Matt Merrill, and I'm here today with Neha Gaswamy from Amazon Web Services. And before we dive into our topic today, which is Agentic DevOps, Neha, can you tell us a little bit about yourself and what you do at Amazon?

• 2:07Speaker 02

  Yeah, absolutely. Glad to be here, Matt. Let me start with what I do here at Amazon. I lead Agentic DevOps in Agentic AI organization in AWS, where my team is building products which bring agents into DevOps and application security aspects of development. I've been in Amazon for almost twenty two years at this point. And fun fact, I started my journey as an STE, stayed out of college in a team called Builder Tools. So I was building tools which was mainly focused on internal Amazon developers. And I was there in the team for three years and then moved around in different parts of the company

• 2:50Speaker 02

  and came back to building developer tools four years ago. And that has been like the most fascinating journey for me because I came back.

• 2:59Speaker 01

  Awesome.

• 2:59Speaker 02

  Yep, came back to build the same tools I was building but for external developers and in the world of AI. And it's just fascinating to see how this world has changed in all these years.

• 3:10Speaker 01

  Yeah, honestly, I've been really looking forward to this. I can't wait to talk about it. And what a great prep for dealing with outside developers. How many developers and engineers does Amazon have roughly? Because I mean, that's a heck of an audience to lead a set of tools for.

• 3:26Speaker 02

  Yeah, I don't know the latest numbers. They keep changing, but thousands is what I can say.

• 3:30Speaker 01

  Yeah.

• 3:31Speaker 02

  Yes, we have thousands of them. And to date, I I maintain this outlook that developers have been the hardest audience to please, whether internal or external.

• 3:41Speaker 01

  Very difficult. Yes, yes. You have my utmost respect. Yes. And to the audience out there, we wouldn't have it any other way, right? That's what the engineers do. So, okay, cool. So today we're going to talk about DevOps in general and Agentic DevOps. I can't wait to dive in to see what that progression has looked like, especially at Amazon. And I would love to take a view on this conversation of, you'll hear me ask some questions about, is Amazon dog fooding these things? They are drinking their own champagne, as they say, and how you release these features. So I can't wait to hear about this. So. Let's start, and I think agentic has become a really big blanket term for a lot of things.

• 4:24Speaker 01

  And I think like engineers have a very specific definition of agentic. But I think marketers have a much different one, right? And I'm curious, from your perspective in the DevOps space, what are you excited about in the DevOps space with agentic? And what do you think it really means in the DevOps space?

• 4:42Speaker 02

  Yeah, so roughly speaking, my definitions of agentic is that you have a model in the mix, which is doing the reasoning. But you're really using the model to drive a lot of reasoning decisions and deciding which tools are the right tools to use for the agent to achieve a goal that it's given. And we've now used agents a lot for coding. So we've had like so many coding tools available for many years now that are making it easy for people to generate a lot of code. Agentic for DevOps for me is more about, okay, now your code is ready and data is showing us that code is being produced at an unprecedented rate using these AI tools.

• 5:27Speaker 02

  What's the next step? And really DevOps, the way I see it changing with agentic development coming in is it's both a catalyst for change and the cure for change. It's a catalyst because with all these code changes that are now lying around for code reviews, the challenge is how do you take them to production safely? And you match the velocity at which the code changes are being produced in the first place. So I think that's the problem. And that's the catalyst of why DevOps had suddenly become much more important. And the cure is, and this is the exciting part for me, is agents are pretty good at doing reasoning on why our bills are failing, why our pipelines are stuck and what you should do to fix them.

• 6:11Speaker 02

  So agents will also help us solve some of these problems in the DevOps world. And that is like really, really exciting to me. And we've had things like I A C. We've always known those are really important parts of the DevOps cycle. Things like testing, I mean, this has existed for a long time. Basic practices for good software development. But I think agents are going to make a really big change by taking these elements and making them easier for developers to inject them into their DevOps workflows. So that's what I'm super excited about is we've seen agents do a lot of work for coding,

• 6:51Speaker 02

  but what can agents do to make your pipelines better, to make your build failures go down, to make your changes safer? And that's the exciting part.

• 7:01Speaker 01

  And so I think that you have kind of a flagship product right now. It's called DevOps Agent. Pretty straightforward name, which is good. I like that. I like that about this AWS service because I don't have to have this like, oh, what's that name that I need to remember? So anyway, but that, what I was really pleasantly surprised to read about is that the integration list is like a who's who of the DevOps realm. It's GitLab, Datadog, Splunk, ServiceNow. And I saw you say something in a previous interviews that AWS isn't building its own C I C D platform or code repo. And so for someone like me who has a mental model

• 7:42Speaker 01

  of once we're in AWS, we're going to stay within AWS because they have a service for everything, feels like a pretty big shift. So is it? And how are you thinking about that? And is that a pivot? Or tell us more about the thought behind that.

• 7:55Speaker 02

  Yeah, that's interesting that you say that because an insider view from AWS and Amazon in general for all these years, our mental model has not been about building a service just because we can. Our mental model has really been about what do customer needs are and what do they really need to run their operations to build the systems that they need to build for their customers? And we work backwards from that. And in working with so many customers over the years, what we found was the environments for software development, the tools that they use, whether it's the C I C D pipeline, or is it some policy related tools that they're using, or even the code repos.

• 8:37Speaker 02

  There is a lot of variety in these development environments. And the problem space was really on top of these environments. So the problem space was how do you connect the data or the process, which is maintained in these disparate tools and give solution at that level to the customer. So that's why we have not been obsessed with building the next code repo, so to say. But it's really been about people are using these code repos to do their work today. Then they're using a pipeline system to do the next step of that work. How can we help them at a higher level tie all these tools together to reduce their operational toil? So I'd say it's a much higher level goal that we are taking.

• 9:22Speaker 02

  And if it so happens that in order to achieve this higher level goal of reducing the ops pain for our customers, we need to build a code repo, we need to build pipelines, sure, we'll do that. But it's really about solving that, where is the customer problem?

• 9:37Speaker 01

  That's really nice. And by the way, just to clarify, that's my mental model. It doesn't make it right. So thank you for that.

• 9:42Speaker 02

  Quickly, I'll add one more thing to there, Matt. One of our tenets in AWS is also about choice. And you see that with things like Bedrock, where we give people model choice. And it's like you have open source models, you have the models from the large AI labs. And the same principle applied to our developer tools as well. Like we want to give people choice. Okay, you're using these tools which are not AWS tools, great, we will meet you where you are. So that's another tenet that has sort of guided our mental models and what services we have built.

• 10:16Speaker 01

  Okay, great. So I'm curious, this is morbid curiosity. Amazon Web Services at the scale it's at, how does Amazon Web Services do their own DevOps? And how has that evolved, especially over the course of the past year or two?

• 10:31Speaker 02

  Yeah, so Amazon and AWS in general, we have a very, very strong culture of end-to-end ownership for our developers. Our developers build services, they are also responsible for testing them, they're responsible for all the deployment processes and to actually run these services in production. So we don't have a separate ops team, we don't have a separate DevOps team or even a QA team for most of our organizations within Amazon. And I think that has been a big guiding factor to how the products, the AWS DevOps agent and even the security agent came about. But our model is very much around centrally owning all parts of operations by development teams.

• 11:15Speaker 02

  We feel like by doing this and by having this mindset, our services are just designed better for sustainability. So developers have vested interests and they have accountability to making sure that their services run well. And that has really benefited us as an organization and as a provider of these core services to other customers. Now, we do have a central team, which is responsible for doing a lot of centralized tooling and they provide platforms and tools which are used by a large majority of the team. For example, our pipeline system is owned centrally. We have policy management systems which are also owned centrally, our build and deployment

• 11:57Speaker 02

  systems are owned centrally and we have ticketing systems. So we do have a fair bit of machinery and tools which are owned centrally, but a lot of the ownership around how to configure these tools and run them, that is owned locally. So that's the way we've run DevOps throughout the years and I think it's really been a very strong culture within our organization. Now, coming to the side of how has it changed over the last few years? I would say for the central team, I have seen some pretty remarkable changes. One example that comes to my mind is around how we run campaigns.

• 12:37Speaker 02

  So campaigns are large-scale changes, they could be API upgrades, they could be migration from one framework to another. That impact a large number of teams. So it's not localized to a couple of teams here and there. At our scale, this really makes sense to run these campaigns centrally. And one of the big changes I'm seeing in the last couple of years is how these campaigns are running and how far we are taking these campaigns to the left in terms of the dev cycle. You might have heard the stat around how we were doing Java version upgrade centrally and saving millions and millions of dollars several.

• 13:16Speaker 01

  I haven't heard about this actually, but that's sounds interesting.

• 13:20Speaker 02

  Yeah, so this was the stat we published when it was many ST years of work that was saved in doing a Java upgrade. And this is almost like two years old that we had this stat. And that was done centrally and that was using the technology that we now have in our product called AWS Transform. And what was interesting about that was not just the time savings, it was just how far left we were going. Now you have a central team that is able to update so many software packages in one go. Produce a PR for them to review. They still own the last mile of making the code change happen. But those changes were also verified in the sense we would do a build of those changes to make sure that the bill doesn't break.

• 14:06Speaker 02

  And we could go that far. We could do some testing for those teams on behalf of those teams. So I felt like that is the change that Gen AI introduced in some of our central workflows. And it was not just a tool change, it was a mindset change too. There's a mindset change that we can actually go much further to make the life of developers and developers easy who now have to accept these campaigns and make some service changes and they have many of these campaigns happening at the same time. So I felt like that was an example of where it made a material change for individual development teams. Last two years have also been super interesting. A lot of teams are using Kiro to build.

• 14:49Speaker 02

  Bespoke agents that are doing things like they're basically putting together DevOps agent and some other agents that they have built into one consolidated workflow. That has been pretty remarkable to see. Many teams are of course using DevOps agent internally as well or managing their incident response, but they're also marrying it with their other custom homegrown agents that their teams have. And it's been pretty remarkable to see how they are doing that using tools like Kiro.

• 15:18Speaker 01

  Very cool. This was a few minutes ago, but you used the word toil. And there was another interview I did on this podcast. If somebody used that term as well, they called it the great toil shift. This grunt work now is like being pushed more to the right of how do you verify things with the speed of change? But what I like about what you're saying is like, no, this is actually pulling all that power and the change to the developers to be able to automate more of this with L L M's if I'm hearing you correctly.

• 15:44Speaker 02

  Yeah.

• 15:45Speaker 01

  That's awesome. So I was tempted to ask the question, what does the DevOps agent and security agent do? But I'm going to approach it from a different perspective of what is the origin story of these? And what I'm particularly curious about is was A W S or Amazon jog fooding or drinking their own champagne of these tools themselves before they were released to the general public? So yeah, what's its origin story? And that might lead us into what it does.

• 16:13Speaker 02

  So I would say the motivation to build these agents definitely came from our internal development processes. We've had many years of focusing on what we call pager pain. Now, how often are your S T's getting paged? How often is it pages at night?

• 16:29Speaker 01

  That is definitely toil.

• 16:31Speaker 02

  Yep.

• 16:31Speaker 01

  Yes.

• 16:32Speaker 02

  The other one which we have obsessed over is deployment velocity and deployment safety. So those have been the other two pillars. And every year we would have teams and orgs take goals on improving that compared to the prior year. And I would say they've been top of mind for us for many, many years. And that is where the overall motivation for these two agents came from. It's like, how do we speed up the development cycle and where do we see the most operational pain for our developers? So the operational pain, obviously, the pages was the top of the list. So that's where we said, okay, let's go attack that pain. And then it was, how do you get changes out faster? Application security reviews take a long time.

• 17:17Speaker 02

  And I would say rightfully so, because we want to be obsessed about security. But the fact of the matter is that they do delay releases. They introduce yet another part in the process that it takes for you to get your code out there. So the motivation came from looking at our workflows and the one which got us most excited was definitely the incident response. Of like, what problem do we want to solve in Ops is what we were looking at and we're like, oh, look at the number of times our engineers are woken up. I must have heard that in some story, three A M. in the morning. And something has gone wrong in the system. Let's start with that. How can agents help there? So that became sort of our hero use case to get the product out there.

• 18:03Speaker 02

  And before we launched this product, many of our internal teams were using DevOps agent. They were seeing success, which was, I'd say, between eighty five to ninety five percent, depending on what the team was doing and how they had configured the agent. And that gave us confidence. We have something that we can take now to the public because we had testimonials. We had enough evidence that this is something that can work for our customers as well. For Security Agent, our internal security team started to use that as well. And we got a lot of feedback from them around where the tool is doing well or how can we improve it? So we have, in a way, dog fooding is for me essential. That's the way for me to get active feedback.

• 18:47Speaker 02

  And we have some of the most complex systems in the world. So what's a better way to get feedback than to have your?

• 18:54Speaker 01

  And that is exactly why I was curious to talk about. Yeah, that's awesome.

• 18:58Speaker 02

  Yeah. So that's about how we build these agents. The other thing that. The reason why we wanted to dog food these was the more we learned internally, the better flywheel we have. We found so many defects just because internal teams were testing our product as we developed it. And with every iteration of a release, internal release, we would make a lot of improvements. We'll find things that we were not great at in terms of quality, we'll go and systematically address those things and build that into our product. So I feel like it's a beautiful flywheel that we were able to create as a result of just being part of Amazon.

• 19:36Speaker 01

  That's awesome. So this is something I actually ran into specifically today is just thinking about how to evaluate L L M based systems. How did that flywheel work? How are you evaluating that stuff? Was it humans voting? Was it? I'm just curious.

• 19:52Speaker 02

  We do a few things. We have created an internal benchmark, I think on both sides for both our agents. And in general, I would say methodology that we followed for some of our AI functionalities. We create internal benchmarks. They're not just internal representations. So you'll have an internal benchmark which is for in our internal system. So we would test on that. Then there is a parallel benchmark which is more representative of our current workflows internally plus externally. What do we learn from our customers that we can incorporate in our benchmark? And anywhere where you have public benchmarks, we would use that too. And I would say to start with, a lot of the evaluation was human. So we would have people actually verifying the output.

• 20:37Speaker 02

  We have cases where we also automated a large portion of that using L L M's. But I don't think we saw a case where we could entirely delegate that to L L M. I would say that, at least in my experience, you have to marry these benchmarks with what I call online data. Online data is like essentially things that are happening in production. So when we integrate these agents. Into our internal workflows, let's say the DevOps agent is integrated with our ticketing system for a lot of teams, we actually get to see their feedback. They provide us that feedback and we do have a person who's looking at that feedback, but we also have an automated system that's scanning through that feedback and it's identifying

• 21:20Speaker 02

  the things that we don't get right, our accuracy is not hundred percent. And very systematically, we go and then figure out which ones of those are improvements that are broad brush improvements that we need to get back into the product. Which ones are just misconfigurations, so that's a different kind of improvement which we need in our product, so that people don't misconfigure the product. So that fly, we again, you asked the question of how do we evaluate? It's a combination, it's a combination of automated evaluation with humans.

• 21:50Speaker 01

  And that feedback capture mechanism, is that like thumbs up, thumbs down, just plain text from people, is it something else?

• 21:56Speaker 02

  It could be thumbs up, thumbs down, it could be like a dropdown, things that we're giving them, sometimes we ask them much more specific questions. And if we have access to those teams directly, we just actually sit down and interview them. So that's very, yeah.

• 22:11Speaker 01

  Can't beat it.

• 22:12Speaker 02

  Yes.

• 22:13Speaker 01

  That you can't beat, yeah, okay, cool, that's nice. So, one of my questions was, are you using it in production? It sounds like you absolutely are. Do you have any particularly interesting wins and, you know, the interesting losses that you may have hit along the way, like any particularly juicy things that caught or missed?

• 22:32Speaker 02

  I would say the interesting things that we are catching, some examples have been, we found issues with lambda functions. There was a particular case when we were able to detect a port exhaustion on a particular host, we had a singleton running and just basically creating a bunch of connections. And then we ran out of them. That was detected by our agent, that's an example that comes to mind. With a few teams, they actually benchmarked against their homegrown agents and found that what they did with DevOps agent was they would get a base level accuracy of eighty five percent, but then they customized it. So I added their own runbooks as an example and they were getting into like high nineties with that. So that was pretty remarkable to see just in terms of accuracy.

• 23:18Speaker 02

  I would say that is sort of a larger theme of things we have learned internally. So if you're asking me for the challenges, I would say that we have seen as we have scaled internally are mostly around this customization bit, which teams have had to do some last mile configuration, but I call it like it is very easy to do that. It's not very heavyweight, but I feel like if you want to really ratchet up the accuracy as a team, you add your runbooks, you add some of these basic things which are very bespoke to your teams and the agent does much better. So that was one thing. The other thing which we learned and honestly, it was a challenge that

• 24:00Speaker 02

  we knew about, but I wish we had a better solution was we have so many bespoke systems in Amazon for even logging or metrics. We have teams that are using different version than Vanilla CloudWatch. And the learning was obviously the more information the agent has, the better it will be at giving you the root cause of what went wrong. And access to the information was one of the bigger challenges that we ran into. Because we said we support MCP integrations and we do support a lot of them. But internally, the number of MCP tools that teams have built and integrating with each one of them, that was a very insightful journey.

• 24:43Speaker 02

  We had to build a lot of what I call out of box integrations, working with our central platform team. So that every team did not have to go through that pain of, oh, now I have to integrate with my this system. We looked at what are the most popular MCP things that we should integrate with and we did that centrally and that took us time. But I feel like that is the way for big organizations to actually scale a DevOps agent is to build some of those things centrally. Don't leave that for your individual teams to configure. This is a much better way to scale.

• 25:18Speaker 01

  Yeah, the MCP sprawl has already begun. That's what I'm hearing. Yeah. So there's a line from the GA announcement that says on-call engineers wake up to a root cause instead of active incidents. And so I'm kind of curious what this product is like in experience. So somebody gets paged at 3am and we have the DevOps agent enabled. What does that look like?

• 25:41Speaker 02

  So the journey actually starts much before the human gets paged, the engineer gets paged. When a team configures DevOps agent, in at least the internal way we are doing it is we have a system for ticketing and we have a way such that we model our teams and which team owns what system. And for every system, there is what we call a resolver group. So that represents the team of people that owns that system. And anytime something goes wrong, those are the people that get paged. And when you configure your resolver group and your resources for DevOps agent, it doesn't work upfront to create a subsystem topology. And that is

• 26:24Speaker 02

  the intelligence that it has inherently. So when you do get paged, it will tap into that topology that it has already created for your system. And the interface for the user is really the ticket. So there is a ticket and it has different severity levels. For the higher severity levels, we page people. For the lower severity levels, we just cut a ticket and they can look at it during the regular working hours. So when as you, your system's already hooked up, your ticketing resolver group is configured to use DevOps agent. When you have an autocut, today, your engineer wakes up with the pager going off, the agent is kicked off as soon as the autocut is triggered. So your alarm basically

• 27:08Speaker 01

  What was the term you used? Autocut?

• 27:10Speaker 02

  Autocut. So as soon as an alarm reaches a certain threshold, you have an autocut ticket. And when that auto cut ticket, what it does is as soon as it reaches the threshold, it also pages the engineer. And that is the time when we start the agent to get in the agent to the agent will look at what happened. It'll look into the topology. It will give you a root cause of what it think went wrong. The engineer wakes up and looks at the ticket and the ticket now has information from the agent on this is the time and the alarm went off. This is the system involved. Here is the root cause. And I think what is really helpful and what for the on call engineer is that it gets to see the agent's work. Where all has it looked?

• 27:54Speaker 02

  How did it come to this conclusion? What was the hypothesis it had? So a lot of work is done for that engineer when the engineer comes in. So what they have to do after that is audit the work. The agent also suggests mitigation steps. So audit the mitigation steps as well, and then they take action on those.

• 28:14Speaker 01

  And so this interface plugs in with a customer's existing ticketing system?

• 28:18Speaker 02

  Yeah.

• 28:19Speaker 01

  Is that kind of what?

• 28:20Speaker 02

  Yes.

• 28:20Speaker 01

  Okay, I see. And there's like a link or something like that to the agent interface that shows this analysis? Or does it integrate it in with the ticket system as well?

• 28:29Speaker 02

  For us internally, we we show a lot of information within the ticket itself as much as we can. But we do have a separate app interface for doing more digging into the data if people need to go there.

• 28:40Speaker 01

  Yeah, so if my organization uses ServiceNow.

• 28:43Speaker 02

  Mm hmm.

• 28:43Speaker 01

  What would that integration look like? And what would the experience be? Would you hubby hopping between them? Would you be integrating them together?

• 28:50Speaker 02

  So we provide all the details back to ServiceNow. And they can then render all those details within the system itself for the most part. We still have our app available for those users. But I think the pattern we see is most users stay within the ticketing system.

• 29:06Speaker 01

  That makes a lot of sense. And so that setup step, I would imagine there's a lot it can introspect simply from the AWS setup of a customer. Right? Like, that's a lot. And you also have integrations with, you know, when I'm looking at these like Datadog and Splunk. So it can pull not only from something like CloudWatch, but also from other external logging systems as well. And that's, I would imagine, a one time configuration of this stuff. And it can then pull it all together. That's awesome.

• 29:39Speaker 02

  Yeah, so we will take any logging systems that you have and create the topology map, as I said, and that's at the application level. So we're trying to model the entire application for the agent. And as more of these incidents happen, the agent learns from that and gets better.

• 29:57Speaker 01

  What is the pricing model for this? Like, how do you charge for this? Is it per incident or something else?

• 30:02Speaker 02

  No, we charge by what we call active agent tasks. So we have pricing for task hours. And that's how we charge for it. And we don't charge for idle time. We only charge when the agent is doing its work.

• 30:15Speaker 01

  You really don't want a lot of potential incidents because otherwise you're going to get charged a lot, yeah, which is the goal, right?

• 30:21Speaker 02

  Yes, yeah.

• 30:22Speaker 01

  Just kidding.

• 30:23Speaker 02

  In fact, we do have something that we call a triage agent, which has been extremely useful for us. So it will do a dedoop of events for you. So if I have five tickets cut for the same event, the idea is triage agent is going to tell you that there's only one root cause, although you're seeing five tickets for it. The reason why we built this was what you just said, you don't

• 30:45Speaker 01

  Yes.

• 30:46Speaker 02

  want to pay. Yes.

• 30:47Speaker 01

  I've hit it so many times, you just get the same alarm over and over again. Yeah, that's the same thing. That's awesome. Cool. So this is a philosophical question, I guess, but if the agent is good enough to wake up to a root cause, why does the engineer we need to wake up at all? What's the line between the agent investigates and presents to the agent investigates and acts? And how do you see that line moving in the near future?

• 31:12Speaker 02

  Today, I would say we need the engineer in the loop. The engineer is still the one getting paged.

• 31:18Speaker 01

  That's good to hear.

• 31:19Speaker 02

  Yes, and they do have the accountability to make sure that the system is up and running. And I think as the agents get more sophisticated, the audit work of this engineer would become easier. What they do right now is basically they are an auditor. They examine and audit the work that the agent has done and say, okay, this makes sense. Or they will spot errors and say this does not make sense. And for us in DevOps agent, our job is to make, as the agent becomes Smarter, how do we make the life of this engineer easy? By giving them the right things to order the work. For example, can we show them that not just that the agent has done this, this is how the agent arrived at this.

• 32:02Speaker 02

  We help them test those changes that the agent is suggesting to make. So that's, those are the kind of problems we are looking to solve. And I feel like that is the line that will move. Is the engineers will be able to see not just how the agent arrived at this conclusion, but how what the agent is suggesting, how safe is it to even take that action?

• 32:22Speaker 01

  Yeah, increasing the signal to noise.

• 32:25Speaker 02

  Exactly, exactly.

• 32:27Speaker 01

  And maybe this already exists, but do you see that line moving into making suggestions for bug fixes that may have caught like actual application bug fixes?

• 32:36Speaker 02

  Yes, absolutely. I would say why not? We have, again, as I said, we picked the biggest pain point for our developers. The use case that we picked as our leading use case was this whole pager use case. But same thing applies to even your lower severity things, which can include bugs. You can have Java software agent also configured to look at your lower severity events, including your bugs too.

• 33:01Speaker 01

  So I saw you reported that ninety-four percent root cause accuracy in the preview. And so that leftover six percent, what does that failure look like? I don't know what's going on, a wrong recommendation. And who's accountable if that thing is misdiagnosed or there's some sort of worse incident as a result?

• 33:21Speaker 02

  It is still the engineer who's responsible. So as I said. So that hasn't changed. But the way it really shows up is the engineer sees the work of the agent and it'll see. The agent came to this conclusion, but it correlated this data incorrectly. So that's how the engineer would find out that there is, this doesn't look right. And that's why I keep saying that's the work that we will focus on the next. Months and even years is just to make that audit easier so that

• 33:56Speaker 01

  Yeah.

• 33:56Speaker 02

  The engineer can trust the work of the agent more and more. So I feel like that the trust boundary will start to shift. But yeah, the way it looks like today, you'll just you'll see the hypotheses and you will have to then say, okay, here is where I think the agent is going wrong.

• 34:11Speaker 01

  Ninety four percent right out of the gate is pretty good. You know, you can aim for a hundred percent. You'll probably never get there, but that's pretty good. So let's move on to determinism. So DevOps is built on determinism. So you want to run it twice, get the same results, but agents are probabilistic. So they might investigate the same incident in two different ways and provide two different hypotheses. And so how do you manage that in this system? And is that determinism going to go away? Or do you think it moves somewhere else in this whole problem space?

• 34:42Speaker 02

  I don't think determinism goes away at all. Actually, I think what is going to be fun and challenging is how do you inject determinism in the right way into an agent? Right? So that where do you inject it? What form does it take? Those are some of the more interesting problems that we are focused on. So today, the way we inject determinism is, I would say it's focused a lot around safety and security boundaries. So we give this agent really trimmed down permissions. So you have read only permission boundaries by default. You can't just arbitrarily expand them. It's a very intentional move if a user is giving the agent

• 35:26Speaker 02

  more permissions than what it comes packaged with. So I feel like that's one way we have introduced determinism. But more ways I can think of introducing determinism is doing some actual testing of your changes. That is very deterministic. You spin up a test environment, run your tests, something like that. I feel like the industry will start to move in that direction, even for DevOps. And the third thing, again, future looking for us is also automated reasoning, which it's a different technique. It's not the same as the L L M science that we have, but it's still a scientific. Big technique that allows you to model your system and you can do some pretty remarkable stuff with that.

• 36:06Speaker 02

  I think you were talking to Byron also here and he should have mentioned something more around how neuro symbolic AI is a different approach that's coming into the mix. So we are also looking at taking some of those techniques and putting them in DevOps agent.

• 36:22Speaker 01

  Can you briefly explain, I'm not familiar with automated reasoning, I could guess at it, but what is that? And how is it different from what an LLM does?

• 36:30Speaker 02

  Automated reasoning takes a more mathematical approach, so it will model your system in some sort of a mathematical representation. And you can do the before and after of a system. Let's say you're modeling a change, you can see what it is that it was before and what it is that it is after and then use that to see have you created a drift that was not intended. So it's it's super helpful for things like policy.

• 36:55Speaker 01

  And so that is built into the DevOps agent now or is that down the road?

• 36:59Speaker 02

  Some of it is built in and you'll see us doing more with that.

• 37:03Speaker 01

  Okay, so in Amazon, when something goes wrong and the agent analyzes it and you fix it, how do you do a postmortem? So can you actually like replay the decision tree? Some of it opaque and so for the agent itself, what does the observability look like?

• 37:19Speaker 02

  That's a really good question and something that comes up from our users quite a bit. And we obsess over transparency, actually, quite a lot. So for us, as long as we can show more of the agent's work, I think we help the auditor, the engineer in this case, really figure out where the agent took a wrong turn. And we do give the engineer the ability to give us that feedback. So that as in the product, they can give us that feedback and we get that signal. And then I said that the agent gets better over time and learns from its mistakes. This is one of the ways in which it learns. We take that negative feedback and then we

• 38:00Speaker 02

  record that in the agent's history and the agent actually learns that this was not the right conclusion. And oftentimes we also see that we know what the right conclusion was. Because ultimately, when the ticket gets resolved, at least internally, we we write down this was the root cause. So the agent will take that also into account and improve its performance over time.

• 38:22Speaker 01

  So I am morbidly curious about the architecture of this thing. And like, I'm not going to ask you to go into a dissertation about how it works. But I am very curious, just at a high level, how this thing is is operating. If you don't mind kind of indulging me there.

• 38:36Speaker 02

  Yeah, so it's a combination of the model and the tools that we have provided. The way we've constructed the agent, it's all about that. And we use state of the art models today in DevOps agent. But we also use the the models which are best suited for the job we are doing. So depending on the complexity of the job, depending on the latency requirements, we would pick the right model to solve the problem that the agent has. And really, a lot of our architecture has been about what are the right tools that we want to give this agent? How do we structure those? Problems have been around, even around those tools. What is the underlying data? How do we build the topology?

• 39:19Speaker 02

  That has been a very interesting problem that we have solved. And what data is useful and relevant? Because as you were saying in the beginning versus what is noise. So a lot of our architecture is about the right tool calls and picking the right set of data to actually take action. And we have modeled this agent in a lot of ways as our on calls actually work. So the way we have built this agent is modeling that pattern. When you do get paged, how do you go and investigate an event? So that has been the model for, yeah.

• 39:53Speaker 01

  How is that amount of data injected, you know, as context in the model? How is that managed?

• 39:59Speaker 02

  We do have a pretty large context windows for a lot of models today. So I would say that that takes care of a good amount of context. Then we, with the topology that we build, that's our intelligence too. We can compress it in ways that makes it more efficient for the context window for it to fit in there. And that has been, of course, one of the techniques that we focus on as well.

• 40:23Speaker 01

  I see. Very cool. Thanks. One thing that was really exciting for me to read about was that you can extend the DevOps agent by bringing MCP servers. You mentioned, you know, you had to manage and wrangle a lot of the MCP servers internally. So what's the most surprising or creative MCP integration you've seen one of your customers build?

• 40:42Speaker 02

  I think some of the more interesting ones for that come to mind are related to actually making changes. Now, these are not system changes, but they have integrated MCP servers, which go and update tickets, as an example. Or they have an internal system where they're keeping track of what incident has happened and what is the next step. So go and update that. So it's sort of a right operation. It's not a system, right? It's just still very much logging the work. But I thought that was an interesting thing in how they're using, bringing in some of the MCP servers, not just an input. It's actually like, okay, based on this agent's result, I'm going to update this other system.

• 41:18Speaker 01

  That is really cool. I know that one of the pain points I always used to have was, you know, just providing updates. It's just, just let me fix the problem. I don't want to make an update. If an agent can do that for me, that's wonderful. Great. So I'm going to shift the conversation a little bit away from DevOps agent. Let's just talk about the future of SRE or, you know, in a dev team supporting something on call. And so a lot of engineers, especially SREs, got senior by doing incident response. And developers, you know, feel that pain of, I want to make a solid system so I don't get woken up in the middle of the night. But if the agent's doing that work first, how is the next generation

• 42:01Speaker 01

  of SREs or on-call engineers going to get trained?

• 42:05Speaker 02

  That's a pretty deep question. I can share what I see happening. What I see happening is that I don't think developers or SREs Completely go away. I don't see that happening. The question is more about how will they train? In my view, the previous SREs were trained when systems, when you know, something would go wrong with the system. And they would see a problem over and over again. And there was a lot of pattern matching that starts to happen. Oh, I've seen this problem before. And with AI, what is happening is some of this pattern matching is being done by the agent itself. So fewer of these events will come to SREs, but they'll come. So I I anticipate

• 42:47Speaker 02

  that SREs will have to get much better at doing this pattern matching when the number of incidents that they are doing pattern matching over is fewer. The other thing I feel like will change for SREs or developers in general is they will deal with more complex scenarios where the agent just fails. And so the agent does half the work, maybe it's incomplete. Those are the places where the SREs and developers will get engaged. And I think that's actually a good thing because some of those trivial cases. No SRE engineer enjoyed working on those, but the real hard ones, I think that is how they will train and that is how they will learn.

• 43:27Speaker 01

  And also something you said earlier just came to mind too, which is like, you gotta get this thing off the ground. If it's a new system, there's going to be new problems and you're going to get trained along with the agent too. So the humans still have hope.

• 43:40Speaker 02

  Yes.

• 43:42Speaker 01

  So you lead a Agentic DevOps at AWS. So are you guys hiring more or fewer of the people like SRE types that you hired, say, three years ago for this work?

• 43:55Speaker 02

  I can't share very super specific numbers or even a yes or no for this question. What I can say is we do need a lot of developers and people with the SRE experience on our team. You know, as we build these products and we are going to be building more of them. So I would say the need for developers and the right folks with the SRE experience is still very much there. I don't see that going away.

• 44:22Speaker 01

  That is good to hear from Amazon. And so pivoting a little bit into the types of role, is there a particular role that's on your team that didn't exist a couple of years ago because of all these shifts?

• 44:33Speaker 02

  I think I would change the question a little bit. I wouldn't say that there's a role that didn't exist. But what I expect from the roles that do exist now and then that have existed for all these years is I think we are expecting a lot more and maybe different things from them now. For example, a product manager is now able to create prototypes using one of these coding agents themselves. They don't always have to rely on an engineering team for that. So that's a change in an existing role that I see. I also see more managers are coding than what I've seen in the past. It's just become really easy to do that. And then we have on the flip side, we have a lot of engineers who are much closer to the product.

• 45:16Speaker 02

  So they are, they have a lot of good product ideas that they can now put into implementation without involving the product manager. So it's it's same roles, but the expectation from each of those roles has changed because we are seeing them do a lot more beyond their, you know, traditional specialization.

• 45:36Speaker 01

  The lines are blurring.

• 45:37Speaker 02

  Mm hmm.

• 45:37Speaker 01

  That's what I'm seeing in my role too. Yeah. So a lot of engineers listening, if they're like me, we're worried about our jobs, not in a doomer way, but more of a like, okay, this is coming. What do I do on Monday morning? So if you're talking to like a mid-career DevOps or SRE engineer, what's your advice for them? What do the next five years look like? How can they best prepare themselves?

• 46:01Speaker 02

  My first point is if you haven't tried using any of these agentic tools and models yourself, start there. You have to, you absolutely need to know how powerful these tools are. And I would say use them to make your grunt work go away. That's precisely what we're doing with DevOps agent. We're making the toil go away. And the more you see these trivial cases handled by the agents and as the agents get more sophisticated, these cases may not be trivial. They will start to get more complex. But the wonderful thing that comes out of using these agents is you will find places where you are still needed to.

• 46:43Speaker 02

  Do more complex problem solving and the agent doesn't do it themselves. And what I have found is people who are continuously playing with these tools and using them for their daily workflows are getting way more creative about, you know, getting to the next level of complexity. Oh, now I can do this because this other thing is taken care of me for me by the agent. So that's my hope and wish for all the SREs and the DevOps folks is just move up the stack to hire more complex problems.

• 47:14Speaker 01

  Yeah, I couldn't agree more. And, you know, I mostly comes from a software engineering background. And that's what I tell people, just start playing and you'll be shocked at how useful this thing can be. And I tell them that the only way that you should be afraid is if you're afraid of the tools. Like just embrace them. Like it's going to be part of our, our lives. So for better or worse, hopefully for better. So let's end it on a positive note. What's the thing that you're most excited about in this whole space? Doesn't need to be DevOps, anything like that. What doesn't get enough airtime? There's something that you just want to tell the, the audience that you're excited about.

• 47:51Speaker 02

  I think if I were to pick what I am personally most excited about, it is the agents are doing what I call a lot of information synthesis work today. And I'm really excited about what actions they can take as they become more powerful. And of course it will come with challenges, there's safety, there's security and all of that. But I still remain very positively hopeful that these agents will be taking a lot of actions on our behalf. And some of those would be actions that are completely boring things that I don't want to invest my time in. So I'm excited about those and others would be things for which I don't have to wake up at 2am.

• 48:36Speaker 02

  I would be excited about those two. But I would truly be excited about, as I was just saying, because the agents are taking so many actions. I have more time to think of other complex, other creative things that I may not have had the time to do.

• 48:50Speaker 01

  Yeah, there's always going to be grunt work too, but hopefully less of it.

• 48:54Speaker 02

  Hopefully less, yes.

• 48:55Speaker 01

  Yeah. Well, thank you so much for being here. This was super interesting. I finally got to ask somebody at AWS about some of the inner workings of what goes on there. So I'm very, very happy and thank you so much for being here. I really appreciate it.

• 49:10Speaker 02

  Thank you, Matt. Happy to be here.