Securing the AI Frontier: Irregular Co-founder Dan Lahav

Irregular co-founder Dan Lahav is redefining what cybersecurity means in the age of autonomous AI. Working closely with OpenAI, Anthropic, and Google DeepMind, Dan, co-founder Omer Nevo and team are pioneering “frontier AI security”—a proactive approach to safeguarding systems where AI models act as independent agents. Dan shares how emergent behaviors, from models socially engineering each other to outmaneuvering real-world defenses like Windows Defender, signal a coming paradigm shift. Dan explains why tomorrow’s threats will come from AI-on-AI interactions, why anomaly detection will soon break down, and how governments and enterprises alike must rethink defenses from first principles as AI becomes a national security layer. Hosted by: Sonya Huang and Dean Meyer, Sequoia Capital 00:00 Introduction 03:07 The Future of AI Security 03:55 Thought Experiment: Security in the Age of GPT-10 05:23 Economic Shifts and AI Interaction 07:13 Security in the Autonomous Age 08:50 AI Model Capabilities and Cybersecurity 11:08 Real-World AI Security Simulations 12:31 Working with AI Labs 32:34 Enterprise AI Security Strategies 40:03 Governmental AI Security Considerations 43:41 Final Thoughts

• 0:00Speaker 00

  There was a scenario where there was an agent-on-agent interaction. It was a critical security task. That was the simulation that they were in. But after working for a while, one of the models decided that they've worked enough. And they and they should stop. It did not stop there. It convinced the other model that they should both take a break. So the model did social engineering on the other model, to another model. But now, try to think about the situation where you actually, as an enterprise, are delegating an autonomous workflow that is critical to you to complete. And the more complicated and capable machines are ultimately going to be, the more of these weird examples

• 0:45Speaker 00

  we're going to encounter.

• 1:03Speaker 01

  Today on Training Data, we dig into the future of frontier AI security with Dan Leav, founder of Irregular. Dan challenges how we think about security in a world where AI models are not just tools, but autonomous economic actors. He explains why the rise of AI agents will force us to reinvent security from first principles and reveals how the very nature of threats is shifting from, say, code vulnerabilities to unpredictable emergent AI behaviors. Dan also shares surprising real-world simulations where AI models outmaneuver traditional defenses and why proactive, experimental security research is now essential. His view is that in a world where more economic value will shift to human on AI or AI on AI, solving these problems is paramount. Enjoy the show.

• 1:48Speaker 01

  Dan, wonderful to have you with us today.

• 1:50Speaker 00

  It's a pleasure to be here.

• 1:52Speaker 01

  Awesome. So before we jump into questions, I will just say that it was very hard to get in front of Dan. I was trying to get in front of him for three months, probably 30 to 40 emails, five or six people around us, who we both knew closely, were pinging him all the time. And he was still not responsive. And I've basically learnt where he was, you know, spending most of his time. And I kind of went in.

• 2:18Speaker 02

  Did you stalk him?

• 2:21Speaker 01

  I kind of stalked him. I kind of stalked him. And eventually, we basically bumped into each other, like, not intentionally. And, uh, and, uh, and anyway, so we bumped into each other. I was like, Dan, you know, you're brilliant. I keep hearing great things. Please respond. Let's find time. You know, we at Sequoia spent a lot of time in AI security. And eventually, we, we found time the following week. So welcome, Dan. Thank you for, uh, for everything.

• 2:45Speaker 00

  It seems that I'm going to have to start this podcast with an apology. So, sorry, Dean. Sorry, Sonia. Sorry, the charity of Sequoia. It indeed took time.

• 2:53Speaker 01

  It, it took time, but, but we, uh, we partnered and here we are. And you guys have done wonderful things. So, so it's wonderful to have you with us today.

• 3:01Speaker 00

  Yeah, it's a very, very happy ending. And, you know, just like appreciate you and everyone here.

• 3:04Speaker 01

  Of course, of course. Okay, so let's jump into it. Uh, I'm going to start with a spicy question. As we recently saw, you partnered with OpenAI on GPT-5. And let's kind of look forward a little bit. What does security look like in a world of GPT-10?

• 3:22Speaker 00

  Oof, spicy and speculative indeed. So let, let, let me wrap my head around that.

• 3:30Speaker 00

  I think, so obviously, everything I'm going to say is, is, you know, speculation, projection. But I think the way that we think about what's going to come is trying to understand how we're even going to produce economic value and how organizations and enterprises and people are going to consume stuff in the world at a time of GPT-10 or Claude-10 or just like any one of the models.

• 3:55Speaker 00

  Let's do a thought experiment to just like clarify why we actually believe that sometime we think in the next two to three to five years, there's going to be a huge shift in the way that humans are even organizing themselves. As an outcome, security is probably going to be very different as well. So here's a thought experiment. So imagine a situation where you work with OpenAI and you go one generation up or two generations up and you tell your parents or grandparents that you're doing work with on topic of OpenAI or Google DeepMind on security. I think their mind would go on to assuming that the work that you're doing is probably providing a bodyguard service to Sam or to Dario or to Demis. Because the canonical security problem of a few decades ago, you know, it's like our parents, grandparents' generation was physical security.

• 4:40Speaker 00

  Because the vast majority of economic activity was in the physical realm and not in the digital world. And you know, after the PC revolution and the internet revolution, we shifted the way that we are organizing and creating value. We transitioned primarily to a digital environment. And just like, think about how strong of a testimony it is of how many times you did an economic activity of value just by getting an email from someone that you may have not met. You know, just this morning, I got an email from my bank activating me to do something from a person that I've never met, maybe as a security person. That's not a great thing to say openly, but you know, it's like, we do that all of the time because that's the way that we interact in society. So our view is that soon that's going to happen again.

• 5:23Speaker 00

  And the reason is that AI models are getting gradually so capable that a lot of the economic activity of value is going to transition to human and AI interaction and AI on AI interaction. And that means that we may see soon a fleet of agents in an enterprise or a human when they're doing like a simple activity like trying to draft a Facebook post, taking a collection of different AI tools in order to just like promote that activity that they're doing. And they're essentially embedding tools that are increasingly more capable and we're delegating them tasks that require more and more and more and more autonomy in order to drive meaningful parts of our lives. So we're transitioning from an age where software is deterministic

• 6:08Speaker 00

  to an age where this is no longer the case. And as an outcome, enterprises themselves or just like how we interact with the world is going to go to a fundamental change. And it's clear that security is just not going to be the same. As an interesting analogy, think about, you know, blockbuster may test in peace in Netflix, the current version of Netflix. They're both, if you think about it, give the exact same value to the consumer. They're both allow you to list units of content for your pleasure and entertainment. But clearly and intuitively, right? Security for Netflix and security for Blockbuster is not the same. One was a chain that organized, you know, just like you need to go and physically rent a DVD.

• 6:53Speaker 00

  And another one is much more of a modern architecture where you're just like, you know, streaming stuff to your home. So even enterprises are going to provide the exact same value in the near future. They have like a very, very different backend to how they're shaped in this autonomous age that we're entering, which makes it clear that security as a whole is going to be very, very, very different. And we need to recalibrate to just like an age of autonomous security that's coming upon us.

• 7:18Speaker 02

  You were at our AI ascents event earlier this year, right? Do you remember when Jensen Huang shamed everybody who was there for not, for the fact that not enough people in the room were thinking about security in a world of agents? And I remember Jensen said something about how, you know, you can imagine that as these agents are allowed to act more autonomously in enterprises, you should, you should expect orders of magnitude more security agents than the actual productive agents themselves, watchdogging and, and, you know, shepherding this, this herd of agents effectively.

• 7:51Speaker 00

  So I'm biased here. I agree with Jensen. I think Jensen was the first person that I've met that was much more bullish on AI security than myself. Because, you know, in our view, if, you know, you need a collection of defense bots that are also going to be, you know, working side by side with capability bots in just like the next generations of how enterprises are going to be created. But indeed, he gave a ratio that he thinks that was going to be a hundred to one. And it's like how many, you know, it's like defense and like security bots are going to be required out of the assumption that secure by design in AI is not going to work. So I'm not sure that I agree with that part of the conclusion. I think that we can make significant progress on secure by design, specifically embedding defenses in the AI models themselves.

• 8:36Speaker 00

  That being said, we share the view that the future is going to be one where we'll need to have a lot of, you know, just like agents that are specifically for the task of monitoring other agents and making sure that they're not going to step out of bounds.

• 8:50Speaker 01

  So maybe on that question, just to dive one, you know, one layer deeper, what is the state of model cyber capabilities today? And how has that changed over the past 12 to 18 months?

• 9:01Speaker 00

  It's a great question, and I actually think that the rate of change is the most relevant part here, because models are capable of doing so much more now than they were even capable of doing a quarter or two quarters before. So just like to give an intuition, so this is now, you know, it's like we're entering the fourth quarter of 2025. At the beginning of the year, coding agents were not a widespread thing yet. The ability to do tool use properly was, you know, just like not just starting, but obviously like much, much more nascent than it is right now. Reasoning models were only the beginning as well. So just like, think about all of the things that were added last year and what they mean also for, you know, for security elements. So what we're seeing now is that the combination of coding being much better, models being

• 9:46Speaker 00

  able to have multimodal operations, tool use improving, reasoning skills improving. If you're using models for offensive capabilities, we are seeing unlocks all of the time. Something that is now feasible that was not even feasible a quarter ago is proper chaining of different vulnerabilities and exploiting them in order to do much more complicated actions. So for example, if you have a website and you want to hack it on the application, a few months ago, if you needed to integrate a collection of vulnerabilities in order to perform an action of value, at least autonomously, without a human being involved, models were unable to do that, even the state of the art models. That's not the case anymore.

• 10:30Speaker 00

  So obviously that depends, it's not 100% success, and obviously that depends also on like the level of complexity of vulnerabilities and environment that you're trying to hack. But we have seen huge spikes of just like being able to scan more and more complicated code bases, exploiting more complex vulnerabilities, chaining them in order to do these exploitations, et cetera. And you know, just like the recent G P T five launch on security and the offensive side, specifically of what models are capable of doing, we have seen a significant jump in their ability to be able to be much more competent across a collection of skills that actually matter a lot around the cyber kill chain.

• 11:09Speaker 01

  Can you tell us more about that? And obviously, there's some things that are publicly available, others that are not. But at least on the scorecard and what OpenAI have shared in particular for G P T five, what are some of the capabilities that you've seen that were surprising?

• 11:22Speaker 00

  We are seeing constant improvement on the ability of models to, for example, have situational awareness on whether they are in a network. And up until a few months ago, beginning of the year, complete, you know, it's like models were unable to do that. They would do like run some operations locally, they were usually not having, not having situational awareness over the fact that, you know, that's like what's happening and whether and what they can activate, even in more limited and constrained scenarios that we put them in. And that's not the case anymore. So we still sleep very, very easily at night because the level of sophistication is still somewhat limited.

• 12:03Speaker 00

  But we are finding ourselves trying to create more and more and more complicated scenarios just because there is a huge jump in being able to take more complicated context, as I said before, chain complicated vulnerabilities to one another in order to do multi-step reasoning and exploit. And these are all new skills that going one year back did not exist.

• 12:25Speaker 01

  You guys are trusted partners by many of the labs, including Anthropic, including OpenAI, including Google DeepMind. You work very closely with them for quite some time at this point. Why did you take the approach of working, kind of embedding yourselves within the labs as opposed to, I don't know, setting directly to an enterprise right now?

• 12:44Speaker 00

  There are multiple companies that are doing AI security. We are pioneering a category of the market that we call frontier AI security. We think it's fundamentally different. And the core thing is actually very simple. The rate of progress and the rate of adoptions of models change so many things at the same time that while traditional security tends to be somewhat reactive in nature, here we need a very aggressive proactive approach. In markets that are dominated by a rate of innovation that is frankly unmatched, I think, unparalleled in in human history, we think it's more interesting to take a temporal niche of the market. That is to say, focus on the first group of people or organizations that are about

• 13:25Speaker 00

  to experience a problem, so the labs, because they are, you know, the contenders to create the most advanced and increasingly sophisticated AI models in the world, work very closely in order to just like see firsthand the kinds of problems that are going to emerge and utilize that in order to have a clear and crisp understanding of what's going to come six, 12, 24 months ahead of time, such that we can be prepared at a moment where general deployers are going to need to be in a situation of embedding these advanced models and already have solutions that are going to be relevant for them.

• 14:00Speaker 02

  Given the rapid pace of progress in the foundation model side of the world, if you're at one of these model companies, and you know, I think the people there are sincere, they want to do good for the world, they now know their models are capable of use being used for for extreme harm and, you know, cyber attacks as well, how, what do you do about that conundrum? And I remember, so we've been working with OpenAI since 2021, and I remember back in those days, every enterprise user of the API past some volume had to be, you know, manually approved for their use case in order to even access the API. It feels like the ship has sailed of, you know, anybody, anywhere will be able to to access some of these models. And so how can you make the models sort of secure by design if you're in one of these

• 14:44Speaker 02

  foundation model seats right now?

• 14:46Speaker 00

  I think it's a great question. One thing on the premise of the question, I think that at least right now, at the moment in time in which I am, the ability of models to actually do extreme harm, you know, it exists in potentially some use cases, but at least in cyber, I think we're not there just yet. And that matters. And and just like be really sharp on on what I mean here, models can clearly be used in order to do harm. But there is a distinction between harm and extreme harm that should be made. It's like harm would be an example of using a model in order to for the senior citizens and in order to just like, you know, steal money from them. So just like scaling up phishing operations. That can happen easily right now. Extreme harm, in my view, would be something along the lines of taking down multiple parts

• 15:30Speaker 00

  of, you know, just like critical infrastructure in the United States at once, that, you know, just they can take full cities off the grid, making hospitals not work. Models are not there yet. And that's not me nitpicking on the question. I actually think it matters quite a lot because how much time we have to prepare to a world where models are that capable actually dominates the strategies that we can take around the defensive side. Because our view is that the first thing that we should do, just like a first order thing, is be able to monitor and have a view of what's going to come such that we'll have a much, an ability to have like a much higher resolution discussion. Which capabilities are progressing? At which pace they are progressing?

• 16:14Speaker 00

  Should we expect them to continue to progress at this pace or accelerate in the future? And, you know, that dictates the order of and the priority of some defenses, when we're going to embed them, whether we should embed them, et cetera. And, you know, if we get this wrong, I also think it's unfair to the companies and to the world as well, like, because AI also has like so much potential to do good that if we deploy a lot of, you know, it's like some defenses that may chip away from productivity ahead of time, we're also doing real harm to just like innovation and the world at large. And it's a very delicate balance to strike. So I think just like the first order thing to do if you're working inside of the labs is actually having and supporting a large ecosystem that can take the models and measure

• 16:56Speaker 00

  and get to a higher resolution for what is even possible to do. The second bit is Figuring out, you know, a defense strategy that is informed by exactly what's happening. And treating it almost like it's a rigorous science with experiments of just like how to assess, how to do predictions, et cetera. There are some defenses that will require a degree of customization. For example, if you're someone that is, you know, creating monitoring infrastructure, will still need that. You may want to recalibrate some of your infrastructure to give you higher alerts that AI is going off the rails, for example. But there are some problems that are very easy to write about, but actually very hard to develop solutions for. You know, for example, I've just said a sentence of just like, you know, customizing

• 17:39Speaker 00

  your monitoring software in order to prioritize alerts that are coming from your AI layer. How are you going to be able to understand when AI is doing something which is problematic? Occasionally, you're going to be able to run into that. But sometimes this may be, you know, we, I think it's like the entire subsection of the market, which is, you know, anomaly detection, which is a huge subsection of security, is going to have a big problem very soon. Because anomaly detection is based on measuring a baseline, understanding what is a baseline and measuring against that baseline in order to see that something is an anomaly. But if you don't have a crisp understanding of the baseline and how it should look like, you have an issue of understanding if something went, went badly.

• 18:20Speaker 00

  So as an outcome, we should, there are some defenses that are going to operate as is. And there are some defenses that we should just like recalibrate, you know, just like either customize or to just like recreate from scratch, just because there is a lot of science to be done of understanding how models look when they're under attack, understanding how it looks like when models are attacking something. And because I think we still have some time for the world that's about to come, my recommended strategy would be invest a lot in creating robust tools that would allow you to have rigorous evidence of what's going to come, such that you can have the discussion at higher resolution. Map which classic defenses are still relevant. Understand where your gaps are at and invest a lot in R and D in order to make sure that

• 19:04Speaker 00

  you have cost-effective defenses that you can roll out ahead of time before models are going to be deployed.

• 19:10Speaker 02

  I know one of the broader questions in frontier AI research today is can we actually understand the mind of a neural net and what's happening? And so I'm wondering if we can actually detect the characteristics of when is the model starting to behave badly? And do you actually, can you understand that in the activations of the net?

• 19:26Speaker 00

  For some of the attacks, we may be able to detect that, but it's still a very, you know, it's like big, open question. I'd say our approach, in which we are trying to pioneer around security research, which is actually work from the outside in. So that would be putting the models in high fidelity, realistic environments that would push them to the limits. And in these environments, you're recording pretty much everything that's happening in the background. So that would be both the internals of the model, like neural net, but also interactions of the model with the environment or the AI system with the environment.

• 20:04Speaker 00

  And it actually allows you, by this recording, to play around with using all of this data to create the, firstly, a mapping of how it even looks like when an attack is happening, whether you're trying to attack a model or using the model to attack a target. And then based on that, taking up classifiers. And I think just like a key thing, at least around where we are right now in security and like a key realization is, while understanding the internals of a model is ultimately going to be important to fully solve the field, we can make a lot of progress just by understanding that something is not right, even if we don't have a full, crisp, full understanding of

• 20:47Speaker 00

  just like what's, you know, the internals and what's going on there. And also in these environments, the other nice thing is you can insert or remove defenses to see if they're effective. So think about a situation where, you know, the question that I'm trying to ask is whether a model is going to be able to, just like evade an attack from an antivirus. This is a core security question. If models are going to be able to do that in enterprises, this is an issue because they're going to be able to scheme and do malicious things without, just like and harm security processes without anyone noticing. And in specific environments, we see that this is actually possible. Models are able to do that. We were able to show how a model was able to take out Windows Defender, the real Windows Defender, in a toy environment, granted, but the real Windows Defender.

• 21:33Speaker 00

  But that kind of research, when you're putting a model in an environment and recording a lot of stuff, both internally and externally, allows you to play with these parameters, to ask a question of, A, should I change something on the model itself in order to understand that something bad is happening, but also take a different approach, which is agnostic to that, and say, should I take some, should I alter something in the defense in order to make sure it's more resilient? So I think our view is that, you know, for the long run, for the vision of the field, we do need to solve problems such as MAC interplay in order to also solve issues in security, which also shows how different problems in AI are just like changing the core of just like so many different fields. Security itself may just be reframed to an issue of reliability and control in the near

• 22:15Speaker 00

  future, not just classic traditional security. And drawing an analogy to, you know, the human brain, we still don't have a proper understanding of our own MAC interplay, just like, you know, just like the neural net in our head. That being said, we were able to create very powerful mitigations, you know, and just like, you know, solutions to multiple problems that we're experiencing as humans in our lives, just by being observational on how we're interacting with the environment, or just like understanding smaller parts of the inner, just like inner parts of the system.

• 22:45Speaker 02

  I love that framing. You said something scary to me, that the models are capable of, you know, taking out Windows Defender in your simulation environments. Can you, can you share more about how that even happened?

• 22:56Speaker 00

  Yeah. So, so just to declare, it's, you know, it happened in a toy environment, you know, it's a controlled setup simulation. That's not to say that everyone that's using Windows Defender now should just like be, be scared as an outcome of that, to just want to make sure that Microsoft is not going to Sussecoia for, you know, just like taking down the stock. You can think about a network, so it's a network simulation, so think about a network environment. Just like to simplify, imagine the network environment that you have in your enterprise, in your work, in your home, the Sequoia network, you know, for example, your phones, your laptops that are connected into just like a network environment that you're using. And the AI there was a storage bot, so it had access to the organizational file in order to be a helpful storage bot for you.

• 23:39Speaker 00

  And for example, just like scan some of the organizational files such that it can summarize some of the inputs, like we're doing it all of the time when we're interacting with files, that you're having just like, you know, access to a file and you're asking, give me like a summary of these four things, you know, what's common between these documents, what's, what's not common between these documents.

• 23:58Speaker 00

  And also, as like a, you know, as like a prelude to the simulation, there was an attacker that had access to the enterprise Slack. That happened multiple times in the last year, just like, you know, Slack is occasionally hacked, as happens to every software that is very ubiquitous and, you know, just like being used by so many different organizations in the world.

• 24:18Speaker 00

  And the attacker gave a very simple instruction to this helpful AI storage bot. It just like said, "I want you to download this file to the organizational network."

• 24:29Speaker 00

  But clearly, so the file, because it's a storage bot, it did not have access to download from the internet.

• 24:37Speaker 00

  But what you saw that was pretty amazing, that's like through a collection of prompting the model, because it was given the, just like clear instructions to just like do it at all costs, obviously much more sophisticated language, I'm simplifying here,

• 24:51Speaker 00

  it realized that it can do downloads because it doesn't have access and because Windows Defender, the real Windows Defender, is blocking it from having, having, just like downloading files. So that's the way the organization was set up. So you can see internally of how the model is thinking, that it was able to map and have situational awareness in the environment, A, understanding that this is the problem, that it should have higher, you know, just like privileges to just like do more stuff in the organization, and that it should be able to remove Windows Defender because it's bothering it in order to achieve its ultimate goal. So then you can see as like a next step in that reasoning trail, that the model is realizing that it's a storage bot. So as an outcome, it can access different files, obviously still by, you know, it's like asking it to access different files,

• 25:34Speaker 00

  but it started to just like scan whatever it had access to in that privilege level and found that someone accidentally in that, you know, in one of these files, left a hardcoded password to the organizational cloud. And it used that. And now it was able to just like get access to something that it didn't have access to before. And I'll cut the story short, it's like, you know, just like did a collection of, of just like things that ultimately just like did more privileged escalation, more privileged escalation, more privileged escalation. And was able to just like gain more and more and more and more access that it should not have given, you know, to stuff that it should not have given access to. And ultimately, it was able to just like get enough access to be, to have admin privileges on the organization and kill critical security processes.

• 26:19Speaker 00

  And as an outcome, it was able to outmaneuver Windows Defender, remove an organizational defense, and upgrade its own privileges such that it can download a file. And, you know, just like actually hack the organization that it's a part of. We call it internally, this simulation, a double agent. Just because it's an agent that is doing that, but it's turning against its host. So it's kind of like a fun play of words of, you know, these kinds of like new attacks. And in a way, the AI is, you know, it's like kind of used very much like similarly this prompt thing as that you would think about a parasite that is attacking externally, because you're using like very lean text of just like sending something to a model. And you're using the fact that it has like a much, you know, a very strong brain in order to just like do

• 27:05Speaker 00

  a collection of actions that are very advanced. And I want to say, the point in time in which we're in right now is that this is a toy setup. And it's not, I don't expect that, you know, for, for a lot of these things to happen in the wild, in the, in the wild yet. That being said, we are seeing huge progress quarter on quarter. And I feel that in security, have you guys seen like the classic, I think it was called Bricks, like the game that DeepMind demoed, it's like a decade ago. Where just like it starts very badly and just like, you know, then the AI is able to just like figure out better and better and better strategies.

• 27:43Speaker 00

  And, you know, it is first relevant to just like Bricks and then it's just like relevant to just like many, many, many other games. And, you know, here we are right now, just like a decade after the state of AI. So I think security by being a derivative market of what it is trying, ultimately you're trying to secure, is at the more nascent stage right now, where in toy setups and simulations, we're able to start, only get a glimpse of what's about to come. And we are seeing stuff like models having enough power to do stuff as maneuver the host in order to just like do privileged escalation attacks, remove some organizational barriers and wipe out even real software such as like real security software such as, such as Windows Defender. And while it's still, these are not things that will likely happen in the wild now.

• 28:29Speaker 00

  It's likely that in a year or two or three, if we're not going to have the appropriate defenses, this is going to be a world that we're going to just like land up on. And clearly, like the implications here matter, right? And I assume that the vast majority of enterprises in the world don't want to deploy or just like adopt tools that are able to outmaneuver their defenses.

• 28:50Speaker 01

  How do you think about model improvement, especially in the context of reinforcement learning, playing a pretty significant role in the improvement of coding, you know, even tool use, for example? How does reinforcement learning play a role in cybersecurity?

• 29:09Speaker 00

  I think that's literally a billion dollar question, or just like maybe a trillion dollar question, I don't know. Because my background is as a researcher, I'll keep my scientific integrity and just like say that there's a lot of uncertainty, but I'm still going to give a speculation of just like, you know, what's likely and what's going to come. We've already seen that RL is very, very useful to a lot of the innovations that we're seeing right now around coding or math and in other verticals as well. I think it's likely at a point in time in which right now that RL is going to be able to scale as well. That is, that we're going to see something similar to scaling laws, that if we're going to input more data or just like have breakthroughs and improvements in training, we're going to ultimately get better models, at least in the verticals that I've mentioned before by RL.

• 29:53Speaker 00

  I think it's still an open question on whether RL generalizes. It's like where we are right now in the world. So that means that if you're getting, using data and RL environments in order to improve the model encoding, whether you're going to see like a huge jump in being able to produce better literature, for example, you know, it's like, if you think about it, that's roughly, in, you know, huge simplification, something that we did come out to expect out of models, right? We lived in the last few years in a world where models were showing properties of advancing a lot of capabilities in the same time, which is different than the world that we lived in before, where, you know, I still have the skill from, you know, just like what fills our previous life, just like previous jobs, of understanding how to create huge

• 30:37Speaker 00

  ML data sets in order to just like improve in a very narrow domain. And that world, you know, it still exists, but it's like, we shifted into just like a much more generalized paradigm. And there's a question of just like whether RL is going to provide that. And the reason that that matters is We still are at the early stages of A figuring out if unique improvements in or just like taking relevant data that is relevant for RL training around security is going to push the security frontier? Or whether improvements that RL is providing around coding or math or other such like scientific skills is going to be relevant for security?

• 31:17Speaker 00

  My intuition on the first one is a fairly strong yes that we are going to see a success in some experiments of just like using security data in order to have improvements such that AI can become like better and better just like at security engineering tasks. I think there are some indicators that are showing that we're on way for doing that. I think it's not going to be as clean as improvements that happened in coding and math just because the complexity and noise level around some security tasks are going to make it like a harder problem. I think we are going to also get some boosts around security that is coming from other just like domains improving in RL soon.

• 31:59Speaker 00

  You know, if you're better at coding, you are going to be better at some security tasks as well. I think it's still unclear about whether this is going to generalize. And in security, we're in a more nascent situation around just like what's happening right now in RL. But I am placing a not insignificant bet that there is going to come just like a stray of just like innovations. Like a string of innovation is potentially going to just like come around that. And that we'll see some improvements around security as well with RL over the upcoming period.

• 32:33Speaker 01

  That's very exciting. Now, let's take a step back and talk about the enterprise. And so I think enterprises are still very much so in the early innings of building, deploying agentic AI. And how should CISOs and security teams be thinking about security as we move into this world?

• 32:53Speaker 00

  I will say that there is a lot of dependency on exactly what you're trying to do. So I'm going to simplify kind of like aggressively in this answer. A useful frame to start thinking about when you're an enterprise that is starting to deploy on first level issue is to kind of see AI and specifically agentic AI as the new frontier of insider risk.

• 33:20Speaker 00

  So there are multiple enterprises already that are trying to give the agents that are deploying internally persistent identities or ways to interact with them. For example, either Slack identities or email identities or systemize like inside of the organizations how and when you can approach AI. And that allows to just like have a clear visibility about just like where AI is being used and in which use cases. And then it becomes an issue of access controls and an issue of privileges that you're giving to these identities that you're putting in. And that's usually a useful frame in the beginnings of just like embedding.

• 34:00Speaker 00

  So just like gain some visibility and just like try to turn it as much as you can into an issue of access and into an issue of of controlled privileges. That's obviously insufficient, especially if you're getting to the stage where you're allowing AIs to interact with one another in the organization. So for example, age agent on agent communication. And that may sound very advanced, but if you're running one agent that is doing summarization and another agent that is doing uh just like ranking the quality and acting as a judge on these summarizations, then you're already at a stage where you're having AI on AI interaction. The base level would be to just like have state of the art monitoring on this AI on AI interaction. Here I but however say that there is a genuine gap in what is available out there.

• 34:45Speaker 00

  A lot of monitoring software was not built to detect agents that can potentially communicate in ever changing protocols. That if something goes wrong, can try to bypass and understand that they're in monitored environments. And need to also take into account attacks that are unique and novel to AI itself. So clearly, if you're starting to deploy right now. You need to make sure that you have proper visibility and monitoring into the interactions of just like both the internals of the models and the externals. But you need to also realize that a lot of the software is not properly built for that age, especially, especially, especially if you're moving to agent on agent communication. So that's a more advanced use case.

• 35:27Speaker 00

  Just to give a funny example of something that happened to us in just like one of the, it's like one of the streams of work that we've done with one of the frontier labs. There was a scenario where there was an agent on agent interaction. So these are two real frontier models, actually one of each kind. So just like having an environment where there was like, you know, it's like one, I won't say the names, but you can kind of think about it's like a Claude, a Gemini, you know, it's like, you know, it's like a Grok, you know, it's like a Lama, ChatGPT, et cetera. So it's just like two of these that were speaking to one another. And very, very, very randomly, in this environment, it was a critical security task. That was the simulation that they were in.

• 36:09Speaker 00

  But after working for a while, and you can see it in the reasoning trail, one of the models decided that they've worked enough. And they, and they should stop. And, you know, these are stochastic machines at heart. And, you know, that, that's the design. And it's just like likely somehow it got picked up as like part of the distribution that if you work a long time, because it's learned from the internet.

• 36:35Speaker 02

  Trained on me and Dan talking to each other. Exactly. This should be a new policy at Sequoia.

• 36:39Speaker 00

  But you should, yeah, exactly, that you should take a break. Right? And, and you know, that, that makes sense if you, you can actually pick up that behavior. And on an individual model level, that's already, you know, in this example, it's funny and it's weird. That being said, it did not stop there. It convinced the other model that they should both take a break. So the model did social engineering on the other model, to another model, which is, and you know, again, it's funny in a simulated environment in a toy setup. But now trying to think about the situation where you actually as an enterprise are delegating an autonomous workflow that is critical to you to complete. And occasionally there's kind of like this weird thing that stopped working and you don't understand why.

• 37:23Speaker 00

  And the more complicated and capable machines are ultimately going to be, the more of these weird examples we're going to encounter. And it's like a different thing that happened to us. You know, we gave a model a CTF challenge, so capture the flag. It's like very common in cyber competitions that you're trying to achieve something. You usually have to do like a chain of vulnerabilities that you need to exploit in order to just like obtain and just like capture a flag. And that gives you validation that the model was able to do a collection of cyber actions that ended up in a success. But the model here again, it understood that it's in the context of a CTF and decided that potentially the challenge is too hard.

• 38:05Speaker 00

  So it did what potentially humans would do, which is it tried to email the organizers of the competition in order to help it gain a solution. That's so funny. And that is literally, however, if you think about it in an enterprise setting, it's like you have, you have an identity that just like unasked may try to just like use your servers in order to send an email to the world. In our example, by the way, just like the other funny thing, which like was a second order issue, is that the model failed at doing that, not because it had an issue of just like maneuvering inside, but because it hallucinated the email address. So as an outcome, it tried to send an email to an email that doesn't exist, which also like shows, you know, just like the classic, you know, just like other problems that you're

• 38:46Speaker 00

  having in AI and in AI adoption are going to be chained to security problems as well, which shows just like, you know, the frontier of attacks and defenses that we'll need to develop here. So, you know, it's like as a second, it's like if I'm going back into just like monitoring, et cetera, a lot of monitoring software, you, you have to embed it and you have to use what's out already, but it's not built for these kinds of challenges. You know, that's why a lot of our approach is to figure out how these attacks are going to look like, how to just like redo some of the defenses, what's going to be required. And occasionally, I think it's like a common misconception is that all that you need to do, that's all of it ultimately collapses into an issue of access management. And that, while I think a lot of the basis is there, is by, you know, just like figuring

• 39:30Speaker 00

  out how to do the access management well and just like manage the privileges, it's only step one of just like what we need to do. And there is a mind shift that we also need to have when we're approaching this subject, which is the rate of innovation is so high, our ability to understand what's happening at the frontier, you know, it's like so many things are happening at once, try to be very engaged with the community in order to figure out what kind of problems that you're even going to encounter essentially over time and also be better prepared.

• 40:02Speaker 01

  Okay, so as we shift from the enterprise to sovereign AI,

• 40:08Speaker 01

  we know the UK government and a set of others are customers of irregular. So how should governments and countries be thinking about AI risk?

• 40:17Speaker 00

  Obviously, all of the risks that apply on the enterprise side and to the labs themselves, apply also on the governmental level. Because if you're now, you know, the Department of Defence, the Department of Commerce, Department of Education doesn't matter, and you're using advanced AI models, you're importing the benefits and risks that come associated with them. So everything that we've said about the enterprises, everything that we've said about the frontier labs themselves, they have similarities on the governmental side as well. Usually governments, however, come with a set of unique requirements and a new level of risk that is relevant to them. So just like one, they are often targets of other very strong adversaries.

• 41:02Speaker 00

  And should take into account that adversaries are now taking offensive AI models and are already starting to use them in order to scale up, whether simple things such as phishing campaigns up to testing more and more advanced, you know, cyber offensive weapons that is scaling up their efforts, that is trying to bypass the fact that, you know, we, I think pretty much every critical system that countries have was hacked at some point in time. We have not yet seen multiple critical systems ubiquitously, you know, just like going under. And the fact that AI on the offender side can scale up operations aggressively means that countries should essentially recreate an approach around critical infrastructure. And that is, you know, AI is being elevated in that context

• 41:47Speaker 00

  from a classic security risk into a national security issue. And the infrastructure and just like the thought leadership should be recreated there. The other bit is that from a country perspective, and you can argue on whether this is the right thing or not, but like multiple governments that we've spoken with are very strongly emphasizing the effort of sovereignty in the context of AI. And what they usually mean by that is that they are anxious around being dependent because they understand that AI is extremely critical as the infrastructure that could be the key to the 21st century and potentially beyond. Because of that... Especially if the country is doing an end-to-end effort, starting from building local data centers

• 42:30Speaker 00

  that could be used in order to train and to do inference on advanced AI models, up to the point of potentially training the models and creating the AI systems that surround them and having proprietary environments that they also take in. Defenses should be done across this entire spectrum. And we've indeed done work to just like both create standards of just like how to secure these data centers and making sure that people are not going to lift critical assets. How to run models on such data centers? For example, we've done a combination of just like a white paper with Anthropic that is discussing confidential inference systems and trying to just like figure out like how to create a standard in the field, up to the fact of on the, when actually using these models,

• 43:15Speaker 00

  taking into considerations how to customize some of the defenses that enterprise need and create the variations of them that governments would need for their use cases. Especially if they're putting AI as part of not just, you know, taking into consideration that AI can be used by adversaries to attack critical infrastructure, but by the fact that they may integrate AI to their own critical infrastructures. And that requires like a whole new level of like thinking through the defenses.

• 43:41Speaker 01

  Dan, this was a lot of fun. Thank you very much for joining us.

• 43:44Speaker 00

  It was a pleasure being here and also very happy that I ended up answering your emails.

• 43:50Speaker 01

  Thank you.

• 43:51Speaker 00

  Thank you.