The CISO Playbook for AI Agents | Datadog

a16z's Joel De La Garza is joined by Emilio Escobar, Chief Information Security Officer at Datadog, to discuss what it takes to secure a company where nearly every employee is using AI and more than 4,000 engineers are working with coding agents. Rather than trying to block new tools, Emilio explains why Datadog chose to embrace AI early and build the security infrastructure needed to use it safely. They unpack how AI changes traditional assumptions around data permissions, credentials, developer access, and software supply chains. Emilio shares how Datadog uses role-based MCP servers and ephemeral credentials, as well as an AI "judge" built by his security team to evaluate the intent behind code and agent skills before they enter the environment. They also discuss why security teams can't afford to wait for commercial solutions to every new AI threat, how the relationship between developers and security teams needs to change, and why Emilio is less concerned about an AI "escaping" than he is about the sheer volume of vulnerabilities AI could uncover.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Joel de la Garza?

  The number one story on Bloomberg right now is that AI has gone wild. You seem remarkably calm.

• 0:05Speaker 01

  The way I see it is if it's not an AI model, it's gonna be somebody or something with actual malicious intent doing it. I do worry about what can the agents do? What tools can they call? What binaries can they pull? And also, how do they get access to credentials? If a code is meant to solve the bug, but it gets rewarded on that, but it doesn't care if it's actually doing something else.

• 0:25Joel de la Garza?

  The tree is sick, so to make it healthy, it cuts it down.

• 0:27Speaker 01

  Out of a necessity, my team built a judge that evaluates the intent behind a piece of code. Like, is this thing meant to do harm or not? And we actually find quite a bit of malicious skills in all these marketplaces.

• 0:41Joel de la Garza?

  Engineers now, security is very much front of mind for them.

• 0:43Speaker 01

  Developers have always cared about security. The problem has been that the version of security that we want them to do is just crappy.

• 0:49Joel de la Garza?

  My thesis 10 years ago was that security engineers have become real engineers, so maybe now is the time.

• 0:55Erik Torenberg?

  AI is changing the security landscape on both sides. Attackers have more capable tools, but security teams are also figuring out how to use those same capabilities to their advantage. In this episode, A16Z's Joel de la Garza sits down with Datadog CISO Emilio Escobar to talk about what he's seeing as AI adoption spreads across the enterprise, including thousands of engineers working with coding agents. They discuss how AI changes assumptions around permissions, credentials, and software supply chains, why simply blocking new tools doesn't work, and how security teams can adapt without becoming the Department of No. And as models get better at finding vulnerabilities,

• 1:36Erik Torenberg?

  Emilio explains why he's less worried about AI hackers themselves than a more practical problem. What happens when the number of vulnerabilities we can find suddenly explodes?

• 1:49Joel de la Garza?

  Thank you so much for joining us. I know you're not here to see us, you're here to go to Black Hat, and it's good to catch up. And it's been a really crazy week, so you are a CISO at a public company, tech company, one of the house of innovation, driven a lot of really cool technology out of it. And we've been talking to everyone about these models escaping, these models hacking. And I know you've been central to kind of the adoption of AI at Datadog. And would love to maybe talk a little bit about, how are you thinking through the risks? How are you deploying this stuff? And I know you're very close to the product team and you're kind of enabling it, so would maybe start off there and.

• 2:22Speaker 01

  Yeah, yeah, yeah. For us, it was like, we have to do it like any other software company. If we not, we're gonna be in the medieval ages real quick. And I like being a CISO, I don't like having my own goat farm. And we started real small. I remember our first install, we started with Cursor, and I remember getting like 50 licenses. And it was more of putting it out there and see who bit. And then, not surprising, it bloomed from there. So now we use every other coding agent that's out there. And some of the things that we're worried about as we deploy, well, first you start with the data. What are we worried about the data? And I never subscribed to the idea of, well, if I block these things, nobody's gonna use them.

• 3:00Joel de la Garza?

  That's never worked.

• 3:01Speaker 01

  So on the non-developer side, like, we just got ChatGPT licenses for everyone. And it was like, whoever wants it gets it, I don't care. And you sure use it to find the recipe for pot roast, I don't give a crap. And, cause then I avoided the all oops moment. And it was, this was back, ironically, this wasn't that long ago, this was two years ago. And it feels like things have moved 20 years since. But I remember having a lunch with a bunch of CIOs at RSA, where everyone was asking me like, how am I not blocking these things? Right? And it was like, well, A, we can, cause we're the type of company that we are. But B, luckily, we can just buy away out of that problem and zero data retention and all those things.

• 3:40Joel de la Garza?

  Well, and probably the inverse of Haley's block, it turned out to be correct, which is the people leaning into it the earliest and the most.

• 3:45Speaker 01

  Right.

• 3:46Joel de la Garza?

  Are the ones that you actually want to reward enough.

• 3:47Speaker 01

  Right, exactly. Yeah, yeah, yeah. So now we have over 4,000 engineers using them. I mean, actually, we have every employee in the company adopting some sort of AI, whether it's Coding Agent, or Gemini, Cloud Desktop, ChatGPT, or what have you. And so I have two, we have two, two verticals. We have the engineering side, which Alexi is deeply involved, our CTO is deeply involved in. And then I'm involved on the non-engineering side for how do we get marketing, sales, and everyone else to use AI? And the adoption is, I think we were like 98% adoption rate, something like that. It's one, one version of AI or the other, everyone is using. On that side, on the IT corporate side, the data becomes more the paramount issue. Permissioning and all those things.

• 4:29Speaker 01

  One thing that AI is gonna do is you think you have proper controls of who get access to what Google file or what have you. AI is gonna find a way to get it. All you have to do is prompt it.

• 4:41Joel de la Garza?

  It flattens the organization, right?

• 4:43Speaker 01

  It flattens the organization quite a bit. So we actually had a moment where internally, we built this business intelligence tool. It's really neat. And it was starting to show some data that I was like, it wasn't anything like a fire alarm, but it was like, should we?

• 4:59Joel de la Garza?

  And the data was probably always available to everyone, they just never took the time to look.

• 5:03Speaker 01

  So, so yes, so it's data that is in a data warehouse. And we thought we had the right table and row permissioning and all of that. But there was always a way to get access to the data if you knew the SQL well enough.

• 5:15Joel de la Garza?

  Yeah.

• 5:16Speaker 01

  Well, then what happened is that you went from only people who really know SQL are using this, to now having a sales rep telling, asking like, hey, what are, what is, I'm a commercial sales rep, I'm just coming up with an example. I'm a commercial sales rep, how is the enterprise deal, enterprise tier team doing?

• 5:31Joel de la Garza?

  Yeah.

• 5:32Speaker 01

  Which normally you wanna keep those things separate and what have you. And the agent just took out the SQL to do it.

• 5:37Joel de la Garza?

  Yeah, yeah, yeah. And it tells them what the comp plan is.

• 5:40Speaker 01

  Right. But it's not always find a way around the permissioning and all that. It's like, we had the right permissioning for the right time.

• 5:46Joel de la Garza?

  Yeah.

• 5:47Speaker 01

  And then the age, so we've been obviously improving all of that. And then luckily, my IT group is, what we've done is now we're at the point where we have role-based MCP servers as well. So we have an MCP server for the SDRs and things like that. So we control and govern that data. And then we're like, let them have it with whatever tool they wanna use. On the engineering side, obviously, the coding agent, like what my security team is worried about, similar to what you were just talking about, is on what can the agents do? What tools can they call? What binaries can they pull? What dependencies can they use? And also, how do they get access to credentials? So for that, we have a few things going on where we've contributed to this open source sandbox

• 6:28Speaker 01

  and expanding all that, where the agent doesn't actually get access to the, any sort of credential file. So even if you have a file in your home directory that has your RDO secret, your MPM published secret, or what have you, the agent on this sandbox can't touch it. The credentials get injected into the agent the moment it needs it. And the beauty of it is, we already have CLI tool that will give you these ephemeral tokens to these things. We just have the agent not know how to call it.

• 6:55Joel de la Garza?

  Yeah.

• 6:55Speaker 01

  So it's like, oh, I'm Emilio, I'm an engineer on this team. I wanna authenticate to GitHub, for example. I call this tool auth GitHub, and it gives me a credential that's valid for I don't know how long. And it gives me my access to GitHub, but now it's the agent getting my access to GitHub. But not, it's just a static token written in a file.

• 7:16Joel de la Garza?

  Yeah, yeah, yeah.

• 7:16Speaker 01

  The second thing is, we know, think that the threat has expanded. Developers are now the main target for attackers.

• 7:22Joel de la Garza?

  Absolutely.

• 7:23Speaker 01

  Because if I get one of these tokens, I can then build a worm that attacks.

• 7:28Joel de la Garza?

  Yeah, exactly.

• 7:29Speaker 01

  Or I can just escape, or do whatever, or access your production environment. The other ways that we see how these agents can do it is via skills. We track a few of these marketplaces.

• 7:41Joel de la Garza?

  Yeah, how do you control the skills? That's a really hard problem.

• 7:44Speaker 01

  So we actually have built hooks into the agents. And then, one of the things that, which we can talk about is, my team built out of a necessity, a judge that evaluates using AI LMS, that evaluates the intent behind a piece of code. I'm not talking CVEs, vulnerabilities, or anything like that. It's like, is this thing meant to do harm or not?

• 8:07Joel de la Garza?

  Yeah.

• 8:07Speaker 01

  Harm being a vague term on purpose. It actually does really well at scanning markdown files as well.

• 8:13Joel de la Garza?

  Nice.

• 8:13Speaker 01

  So we built it because for the longest time, Datadog, for those who don't know, we have a Datadog agent. We take third-party code contributions to it, things like integrations and what have you. So code written by other people. We had this process for the longest time that required a security engineer and an engineer on that agent team to reevaluate the code, and then approve it, and then merge it. Someone like, well, this thing can't scale. So then we built this judge, by we, I mean my team, I don't do anything, that got really good at it. And then the software supply chain hijack started happening, ID extensions started happening. And then we threw this thing at those packages, and it was able, actually able to identify the malicious piece of code that was injected or introduced during the hijack.

• 8:58Speaker 01

  So we're like, huh. I wonder how it will do with markdown files.

• 9:02Joel de la Garza?

  Yeah.

• 9:02Speaker 01

  And it actually does pretty well. So then internally, what that means is, we have hooks into the agents, we know what skills they're getting pulled. I've never believed in a security program that just restricts everything, cause it, the pain, innovation, and all that.

• 9:17Joel de la Garza?

  It's the empire of no.

• 9:18Speaker 01

  It's the empire of no. But also, it's like, we don't wanna be free willy either. So what we do is, now we put this judge in front of every skill that wants to get introduced. We actually find quite a bit of malicious skills in all these marketplaces. We're working with a few of them, partnering to see if they wanna use this judge, but also like, hey, giving a heads up of, hey, we found this one, we found that one, we found that one. And they do a pretty good job at taking those down. The other thing we worry about is what dependencies it can pull, what binaries it can pull. So I think it's plenty of it out there right now. I don't think we have to justify it, but the judge does a good job at it. And then evaluating the code output of the agents against malicious intent.

• 9:58Speaker 01

  I think intent, and it's funny how things, how this industry works, and this is maybe like where I can get a little cheeky here, if you allow me, is.

• 10:06Joel de la Garza?

  I have no control, you can say whatever.

• 10:08Speaker 01

  Yeah. Apparently, intent now is one of the must-have in any AI security type of solution.

• 10:17Speaker 01

  But the interesting thing is like it happened after we've had conversations about the intent anyways. But apparently that's a big deal, right? It's like the fact that these agents, obviously they're trained on existing data, on existing code.

• 10:31Joel de la Garza?

  And they have a reward structure.

• 10:32Speaker 01

  And they have a reward structure. So if a code is meant to solve the bug, but it gets rewarded on that, but it doesn't care if it's actually doing something else.

• 10:41Joel de la Garza?

  Yeah.

• 10:42Speaker 01

  Outside of that, so we were.

• 10:44Joel de la Garza?

  The tree is sick, so to make it healthy, it cuts it down.

• 10:46Speaker 01

  Right. So we were worried about like, you know, the typical thing that we read about of an agent producing code that either would have. Oh, you know, your database isn't scaling and it's paging people at 4:00 AM. So the best thing I can do is just turn it off.

• 11:06Joel de la Garza?

  Yeah.

• 11:07Speaker 01

  And therefore I solve the problem of paging people at 4:00 in the morning, right? So you have to be careful how you prompt these things, but also how it actually interprets your prompt and executes on that. So we have this judge now evaluating the code output of the agents to then make sure that we're doing this. So the cool thing is that we do this internally because we have to. And then the product team goes like, wait a minute, that might actually become a thing. Let's explore and then we evaluate it with customers and all of that. But I mean, I get to say I'm lucky that the team can do it. One of the things that surprises me, I started at a roundtable last week about agentic security. And the sense that I got from a bunch of the security leaders who were in that call was a sense of helplessness

• 11:52Speaker 01

  of just waiting for a commercial solution to come in and solve it all.

• 11:56Joel de la Garza?

  Well, you're, yes, I think that the profession is interesting in that there are a number of different CISO profiles. And I think, you know, I say this as someone who was relatively technical for a CISO. Like before, before, maybe just the generation above us, like there were no tech. I mean, my first CISO was Steve Katz, right? So that's like pretty recent.

• 12:19Speaker 01

  Yeah.

• 12:20Joel de la Garza?

  You know, and I think that there was just a lot of like, sort of like CFOs becoming CISOs or CTOs. I'm not really security people, right? So like, I think, and then, and then what happened, I think, is that security teams got quite big and you needed a manager, right? Not necessarily a leader or a thought leader. And so I think there's just a lot of that right now in the industry where there's very much a, you know, if you're, if you're a large industrial company and software is not your competitive advantage, you're probably going to buy most of your product.

• 12:49Speaker 01

  Right.

• 12:50Joel de la Garza?

  Versus like Data Dog.

• 12:51Speaker 01

  Yeah.

• 12:52Joel de la Garza?

  Which we're like, you better be the best of the best.

• 12:54Speaker 01

  He's totally fine, right? But I still believe even in those scenarios, you may have one or two people on that team that if you give them creative license.

• 13:05Joel de la Garza?

  Yeah.

• 13:05Speaker 01

  They could do something. But the fact that that wasn't even a thought process.

• 13:11Joel de la Garza?

  Yeah.

• 13:12Speaker 01

  Worries me a little bit because this thing is moving really, really fast. Naturally, like a lot of the companies that are solving this are new companies.

• 13:20Joel de la Garza?

  Yeah.

• 13:20Speaker 01

  You know?

• 13:21Joel de la Garza?

  I think it's really hard, and this is, this is, I mean, I think the talent shortage is still a problem. And I'm generally bullish on security jobs because of it. But like, I think, I think for some of the bigger programs, like it's really hard to get technologists in there because they are.

• 13:35Speaker 01

  But here's my kind of argument. Do you really need a security person to do it?

• 13:39Joel de la Garza?

  No, and I think some of the best security people didn't start as security people.

• 13:42Speaker 01

  Yeah.

• 13:42Joel de la Garza?

  Like, you know what I mean? It's like the developer that found their way into security.

• 13:45Speaker 01

  Right.

• 13:45Joel de la Garza?

  Yeah.

• 13:46Speaker 01

  But also in this, in this company is like, I'm sure there's a developer somewhere in there that they can be like, hey, how about we half you solve this problem for us in the meantime?

• 13:55Joel de la Garza?

  I think that a realignment is kind of happening. So I was talking to a family member at a big, big American corporation. And they were saying that sort of like with AI tool adoption, they don't need to write as much code manually anymore.

• 14:07Speaker 01

  Right.

• 14:08Joel de la Garza?

  They still need engineers and like that. But like they're seeing, they're basically saying like, hey, look, we don't need this many tier one engineers. And the security team needs them. And so I think maybe you're starting to see some of that cross pollination.

• 14:21Speaker 01

  Right.

• 14:21Joel de la Garza?

  Where you'll see that talent find their way into security. Cause it was always the case that if you're a developer and engineer, like you would just make more money on, on the software development side. And so like, I think those are equalizing and maybe, I mean, my thesis 10 years ago was that security engineers will become real engineers. So maybe, maybe now is the time.

• 14:40Speaker 01

  Yeah. Yeah, yeah. I mean, we pay them the same as software engineers.

• 14:44Joel de la Garza?

  Now we do, yeah, yeah. I mean, in the valley especially.

• 14:46Speaker 01

  Yeah. Like during the same.

• 14:47Joel de la Garza?

  I think it's, I think it's narrowing in large corporates. I think, I think it's still software engineers have the advantage there, but that, that'll probably narrow over time.

• 14:56Speaker 01

  Yeah. So that's how we've been thinking about agentic security is like, what are the things that we actually worry about? And then what do we, what can we build? What can we, what can we use?

• 15:05Speaker 01

  And then where do we find more synergies there? And I know that's a very buzzy term, but what that I mean is like our, our DevX team is also interested to understand what's happening in the agents. Cause they want to monitor developer experience. So why can't we just all together work on it? So that's, that's how we, that's how this thing has grown. Cause it's not just a security tool for the sake of security. It's also serving all the purposes.

• 15:28Joel de la Garza?

  And I've noticed with engineers, especially over the last decade, like engineers now, security is very much front of mind for them.

• 15:34Speaker 01

  Yeah.

• 15:36Joel de la Garza?

  If, if only because they don't want to have to keep fixing things. And so like, I've noticed that there is this, this front of mind security thing for engineers, which is great. It's like, I think, I think we've made a lot of progress in that regard.

• 15:47Speaker 01

  Yeah, and I, well, I actually have a little bit more of a, of a, maybe a fiery take on that. I think developers have always cared about security. I think the problem has been that the version of security that we want them to do is just crappy.

• 16:07Joel de la Garza?

  Yeah.

• 16:08Speaker 01

  Which is like, go fix this thousand of things that none of them are actually relevant to what you're building.

• 16:13Joel de la Garza?

  Yeah.

• 16:13Speaker 01

  But the scanner told me there's, they're critical, so you have to fix it.

• 16:16Joel de la Garza?

  Yeah, there is a lot of like the, you guys have sent me so much junk.

• 16:20Speaker 01

  Lost look at ability. So one of the things that I get to do at Data Dog is actually I get to meet a lot of the engineering side of the equation, talking to us about our security platform and products and how do we get their security team to use it? And all, and they told me the same stories that I'm sure you've heard of, like, I get a thousand tickets. None of them are relevant. Security doesn't know what we're doing. There's no intent to even understand the systems. And the number of times I hear a security person telling me that their developers write crappy code. It's just mind boggling because I'm like, well, then you do it.

• 16:57Joel de la Garza?

  Yeah, exactly.

• 16:57Speaker 01

  See how good you can do it.

• 16:58Joel de la Garza?

  Yeah. Probably no better.

• 17:00Speaker 01

  Yeah, exactly. Or maybe it's good code, but they're not fixing your noise vulnerabilities and therefore you think it's crappy code.

• 17:08Joel de la Garza?

  Yeah. I mean, oftentimes, right, with code bases, this was the learning experience for me was that like, you'll find these, in isolation, you'll find these issues in code, but then like up, there's, there's a framework, a layer up that's actually mitigating any potential fallout. And so it's like, whatever, it doesn't matter.

• 17:23Speaker 01

  Yeah.

• 17:24Joel de la Garza?

  And I just think, I think, yeah, security people tend to have an issue, a blind spot around some of that stuff.

• 17:28Speaker 01

  Yeah.

• 17:29Joel de la Garza?

  I guess, and I guess sort of like, as we sit here, you know, the, the number one story on Bloomberg right now is that AI has gone wild. And, and the world is over and everything is getting hacked. You seem remarkably calm. So maybe, maybe before we go, like, would love to get your take just on sort of like, you know, supposedly there's this great white shark cruising out there just eating people without them knowing. But in general, like I just haven't noticed a sense of panic from CISOs. Like, what's your take on all this?

• 18:02Speaker 01

  I mean, the way I see it is it's, it's, if it's not an AI model, it's going to be something with somebody or something with actual malicious intent doing it. So therefore I don't panic. I do, I do worry about our access to those same capabilities. I do worry about the lack of regulatory framework that's deciding who gets access to these capabilities.

• 18:32Joel de la Garza?

  Yeah, yeah. I feel you on that. It's sort of like, it's really interesting who's getting access to this stuff.

• 18:37Speaker 01

  Yeah.

• 18:38Joel de la Garza?

  And it's sort of like, hmm, what's at play here?

• 18:40Speaker 01

  But then, and then it's clear that is, or communicated in a way that makes it, or portrays it as clear that is outside of the frontier labs hands. And so like, we have to put our company name on the list and then we're supposed to wait. And then when I ask, what's the criteria for how these things are evaluated? I get jazz hands response.

• 19:02Joel de la Garza?

  But then you come to a conference and you talk to your buddies and they've had it for nine months.

• 19:05Speaker 01

  Yeah.

• 19:06Joel de la Garza?

  Yeah. And they're making stuff that's way less critical than what your product.

• 19:09Speaker 01

  Yes, our argument was like, hey,

• 19:13Speaker 01

  some of these companies are using our stuff.

• 19:15Joel de la Garza?

  Yeah.

• 19:15Speaker 01

  Wouldn't it make sense for us to get access so we can protect the stuff that you're using?

• 19:19Joel de la Garza?

  Exactly. Yeah, I'm sure there's a lot of government agencies running Data Dog software.

• 19:24Joel de la Garza?

  And vice versa, right?

• 19:25Speaker 01

  We have a lot of, also companies that provide services to the government.

• 19:29Joel de la Garza?

  Yeah, yeah, yeah, of course.

• 19:31Speaker 01

  So yeah, so I'm not worried necessarily about the models escaping sandbox and finding old days. Like I think, actually, I actually think inherently it makes things better for us.

• 19:40Joel de la Garza?

  Hundred percent.

• 19:41Speaker 01

  Because guess what? We're talking about it.

• 19:42Joel de la Garza?

  Yeah.

• 19:43Speaker 01

  Bloomberg is talking about it, you just said, right? So before we were like, well, nobody's talking about it, so who cares? But am I worried about one of those things hitting Data Dog and something happening? It's like, well, it's either a model or somebody else, right? Like if we have a gap, we have a gap.

• 19:56Joel de la Garza?

  Yeah.

• 19:56Speaker 01

  At some point it's going to get discovered.

• 19:59Speaker 01

  So I'm not too worried about that. What I worry about is the volume of things that are going to get discovered. And the things that are telling us how to handle that volume are not to par.

• 20:13Joel de la Garza?

  Hundred percent.

• 20:14Speaker 01

  So that you must fix every C V that's in your environment to now you're going to get a thousand X more C V's in your environment. Well, those things, two things don't necessarily align well. So I'm hoping that side also changes to the current day of things.

• 20:28Joel de la Garza?

  Totally.

• 20:28Speaker 01

  That's what I worry. And then the, sorry, and lastly, like the only thing that I worry is the hypersensitivity to the security of these findings because of the fact that it was found by a model rather than a human. And I'm already seeing some of that where like, oh, model X found this, so it must be extremely critical and true. And then you're looking at it and you're like, eh, it's actually not. But I, it's kind of hard to argue that it's us arguing against the, you know, this thing. Because just because a Greek god found it doesn't mean it's actually a critical thing.

• 21:06Speaker 01

  But it, that's, I think the, the third party risk management game is going to get really crappy, even more so than it is because of this thing. So I hope eventually those, so those things I do worry about. But a model escaping and hacking companies and all that is like, well, I mean, they were going to get hacked no matter.

• 21:21Joel de la Garza?

  Yeah. There's always been hackers.

• 21:23Speaker 01

  Right.

• 21:24Joel de la Garza?

  There's two more now and whatever.

• 21:25Speaker 01

  Yeah, yeah, yeah. There's two really good ones, but hopefully they make us better rather than like making us worse.

• 21:30Joel de la Garza?

  Well, and I think, I think the lesson that I have learned throughout my entire career is that any attempt to gatekeep security fails.

• 21:37Speaker 01

  Right.

• 21:37Joel de la Garza?

  And it only makes things worse and that an open and free discourse on these things is the only way to get back.

• 21:42Speaker 01

  Yeah, yeah, yeah, exactly.

• 21:43Joel de la Garza?

  Thank you for joining us and having that discourse, man. It's been awesome.

• 21:45Speaker 01

  Yeah, thanks.

• 21:46Joel de la Garza?

  Enjoy your time at the conference.

• 21:47Speaker 01

  Thank you.