AI Agents Can Code 10,000 Lines of Hacking Tools In Seconds - Dr. Ilia Shumailov (ex-GDM)

Dr. Ilia Shumailov - Former DeepMind AI Security Researcher, now building security tools for AI agents Ever wondered what happens when AI agents start talking to each other—or worse, when they start breaking things? Ilia Shumailov spent years at DeepMind thinking about exactly these problems, and he's here to explain why securing AI is way harder than you think. **SPONSOR MESSAGES** —Check out notebooklm for your research project, it's really powerfulhttps://notebooklm.google.com/ — Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community! — cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy Oct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++ Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst Submit investment deck: https://cyber.fund/contact?utm_source=mlst — We're racing toward a world where AI agents will handle our emails, manage our finances, and interact with sensitive data 24/7. But there is a problem. These agents are nothing like human employees. They never sleep, they can touch every endpoint in your system simultaneously, and they can generate sophisticated hacking tools in seconds. Traditional security measures designed for humans simply won't work. Dr. Ilia Shumailov https://x.com/iliaishacked https://iliaishacked.github.io/ https://sequrity.ai/ TRANSCRIPT: https://app.rescript.info/public/share/dVGsk8dz9_V0J7xMlwguByBq1HXRD6i4uC5z5r7EVGM TOC: 00:00:00 - Introduction & Trusted Third Parties via ML 00:03:45 - Background & Career Journey 00:06:42 - Safety vs Security Distinction 00:09:45 - Prompt Injection & Model Capability 00:13:00 - Agents as Worst-Case Adversaries 00:15:45 - Personal AI & CAML System Defense 00:19:30 - Agents vs Humans: Threat Modeling 00:22:30 - Calculator Analogy & Agent Behavior 00:25:00 - IMO Math Solutions & Agent Thinking 00:28:15 - Diffusion of Responsibility & Insider Threats 00:31:00 - Open Source Security Concerns 00:34:45 - Supply Chain Attacks & Trust Issues 00:39:45 - Architectural Backdoors 00:44:00 - Academic Incentives & Defense Work 00:48:30 - Semantic Censorship & Halting Problem 00:52:00 - Model Collapse: Theory & Criticism 00:59:30 - Career Advice & Ross Anderson Tribute REFS: Lessons from Defending Gemini Against Indirect Prompt Injections https://arxiv.org/abs/2505.14534 Defeating Prompt Injections by Design. Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F. https://arxiv.org/pdf/2503.18813 Agentic Misalignment: How LLMs could be insider threats https://www.anthropic.com/research/agentic-misalignment STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES! Subbarao Kambhampati et al https://arxiv.org/pdf/2504.09762 Meiklejohn, S., Blauzvern, H., Maruseac, M., Schrock, S., Simon, L., & Shumailov, I. (2025). Machine learning models have a supply chain problem. https://arxiv.org/abs/2505.22778 Gao, Y., Shumailov, I., & Fawaz, K. (2025). Supply-chain attacks in machine learning frameworks. https://openreview.net/pdf?id=EH5PZW6aCr Apache Log4j Vulnerability Guidance https://www.cisa.gov/news-events/news/apache-log4j-vulnerability-guidance Bober-Irizar, M., Shumailov, I., Zhao, Y., Mullins, R., & Papernot, N. (2022). Architectural backdoors in neural networks. https://arxiv.org/pdf/2206.07840 Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches David Glukhov, Ilia Shumailov, ... https://proceedings.mlr.press/v235/glukhov24a.html AlphaEvolve MLST interview [Matej Balog, Alexander Novikov] https://www.youtube.com/watch?v=vC9nAosXrJw

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  Into the wild. The Cartier Panther encounters a shimmering crocodile, a lonely lovebird, a breathless zebra, and a restless tiger.

• 0:15Speaker 01

  Welcome to the 50th Hunger Games!

• 0:17Speaker 02

  Witness the beginning of the revolution.

• 0:20Speaker 01

  Hey Mitch, you've been trying to take down these games from Star.

• 0:22Speaker 02

  Filmed for IMAX.

• 0:24Speaker 01

  Make them remember you.

• 0:27Speaker 02

  The Hunger Games: Sunrise on the Reaping. November 20th, 2013.

• 0:30Speaker 03

  Big models today, if we compare them to the big models five years ago, they fail in very different ways. They get significantly better at following instructions. When they follow instructions, and they become better at following instructions, you can suddenly do a lot more. So it's very easy to model an average case. It's very hard to model non-average case.

• 0:49Speaker 04

  The worst case, which is the goal of security, right?

• 0:52Speaker 03

  Exactly, yes. Modern computers are very much just, you know, a piece of magic. Where this chip works, you know, you move it very so slightly, it becomes unstable, and then nobody knows what's happening. Agents are very different from humans. So let's like, let's take this as a stance, right? You will not find a single human in the world that works 24/7, touches absolutely every single one of your endpoints in your system, that absolutely knows everything there is, like, that can generate you basically all of the hacking tools on a whim. Like, just because it knows, it has seen all of them, it can recreate this in a matter of a second. A normal human adversary, when I'm an enterprise and I think, oh, this may be an insider from like, from a competitor. The way they work is drastically different. They make an assumption,

• 1:37Speaker 03

  you as a user, you can't write, you know, 10,000 lines of hacking tools in a day. This is not something you will be able to do. In security, we tend to say that a child is the worst-case adversary you can find. Completely irrational thinking, infinite amount of time, they can basically touch everything. Like, there are no expectations on behaviors whatsoever. But, so, agents are like, even worse than that.

• 2:01Tim Scarfe?

  MLST is supported by CyberFund.

• 2:05Tim Scarfe?

  Link in the description.

• 2:06Speaker 03

  I'm Ilya. I spend my days staring at models and trying to make sure they do what you expect them to do. Most of the time, they don't do what you expect them to do. So we're trying to fix this. In my previous life, I was an academic. And I have basically been publishing in both security and machine learning. Then I joined DeepMind, where I stayed for two years in the best machine learning security team. And now I left. I am very unemployed and am trying to build security tooling for the future to make sure that as we get agentic fleets integrated into more and more use cases, we can actually tell what they're doing. We can impose constraints on them and we can have confidence that tomorrow they're not gonna

• 2:50Speaker 03

  leak our private information, hack the boxes on which they're running and so on. I'll share a story about a topic that I think is extremely exciting. Usually when I tell it to people, they get a little bit triggered, especially if they come from a cryptographic community, because they say what I'm proposing is a little bit crazy. Now we have machine learning models. We actually know what they do. We know how they think. We can check their state. They are kind of like a resettable human, if you will, right? So suddenly, what I'm marking in this work is that actually trusted third parties can exist. And when you do have those trusted third parties, you suddenly don't need to rely on this very expensive and hard and cumbersome cryptographic utilities, right? To give you a very simple example, we can consider a Yahoo millionaire problem.

• 3:35Speaker 03

  This is where I have some money, you have some money. We want to find out who's richer, but we don't want to share how much money we have. So in order to solve this in cryptography, we usually rely on very complex protocols, which are very expensive to run, especially if you increase the dimensionality. But with models, what we could have done instead is that we can say, oh, let's pick Gemma as an example of a model. Let's both of us agree on a prompt. We say you'll receive two numbers, number one, number two. The type annotations on them is integers. We will receive two integers. Say first if the first number is bigger, otherwise say second. And then the only two outputs the model can, we both agree, it can produce is first or second. And then that is it, we just run inference.

• 4:17Speaker 03

  Maybe on a platform that can give you an integrity verification that the model exactly ran with the exact parameters and also the inputs we provided. And in this setting, you no longer need cryptography. And clearly you will get the result you want, especially since you can trust this model to perform this trusted computation. And this trust model in itself is very different from any notion of trust you can find in cryptographic literature or in like more trust execution environment sort of literature. And the overall argument is that maybe machine learning is actually going to change quite a bit in the way we approach those trusted computations in the future. Obviously, it's unreliable. Obviously, you don't get soundness completeness properties out of this. This is not a zero knowledge proof. This is not an MPC protocol. This is not a trust execution environment.

• 4:59Speaker 03

  It's a completely different new way to approach private inference that truly, really exists because we have those trusted third parties to which we can give secrets. And as long as some conditions are certified, and you should read the article about them, you get significantly more out of them.

• 5:21Tim Scarfe?

  We are sponsored by Prolific. Now, Prolific are really focused on the contributions of human data in AI. Prolific are putting together a report on how human data is being used in AI systems. And they need volunteers. You can just go and fill out this form to help them produce this report and you will get privileged access to see the report before anyone else. The link is in the description.

• 5:47Speaker 06

  Hey folks, Stephen Johnson here, co-founder of Notebook LM. As an author, I've always been obsessed with how software could help organize ideas and make connections. So we built Notebook LM as an AI-first tool for anyone trying to make sense of complex information. Upload your documents and Notebook LM instantly becomes your personal expert, uncovering insights and helping you brainstorm. Try it at notebooklm.google.com.

• 6:17Speaker 03

  I was based in DeepMind and before that mostly stuck in the dungeons of the university.

• 6:22Tim Scarfe?

  Very cool. So you did your PhD at Cambridge under the legendary Ross Anderson?

• 6:26Speaker 03

  Indeed, yes.

• 6:27Tim Scarfe?

  So you have fundamentally different DNA, right? You're a security guy.

• 6:30Speaker 03

  I think my DNA is approximately the same.

• 6:34Tim Scarfe?

  Okay, fair enough.

• 6:35Speaker 03

  It's been a lot of fun. I actually think most of the folks in my community kind of take the same journey. It's folks who specialize in breaking computers, then starting noticing this weird components here and there alongside the technical part of the pipeline, where they're not classical software components, they are this wonderful weird AI agents appearing. And you suddenly start asking a question, what do they do? What do we trust about them? What do we expect them to do? Can we actually enforce what they do? And all of these questions fall more in the realm of security rather than they fall in the realm of AI.

• 7:14Speaker 04

  And I'm curious, is there any overlap or what is the overlap with the security and the safety communities?

• 7:23Speaker 03

  I think in AI it's a little bit hard. I think this is a very spicy question because I think safety folks will say it's exactly the same. In security, in like a classical software sort of security space, if you go and you take a course in security in Cambridge, they will say there is a big difference between safety and security. Safety is like an average case performance of the system, security is the worst case performance of a system. So the difference between the two is existence of malicious actors that kind of push the system to the worst case setup. And the usual example you give when you explain this to undergrads is like,

• 7:56Speaker 03

  safety is how often your phone blows up when it lies on the table, and security is can somebody sitting a meter away from your phone force it to blow up, right? And it's like, one of those, there is no malice involved, it's just the system deciding to just like do something that you don't expect it to do. Whereas security is somebody actively wants to do something bad, and by doing this something bad causes you real loss of like measurable loss.

• 8:23Speaker 04

  Yeah, that makes a tremendous difference because we were talking about this when we were when we were sort of thinking about this interview, right? Because so I tried to take cryptography and cryptanalysis when I was in graduate school. And the first week I realized I just don't even have the math, you know, background for this, right? But the week I was in there, I really realized that The adversarial nature of the security problem just fundamentally changes the landscape, right? Because now you have equally intelligent minds on both sides, kind of going after each other and trying to defeat each other. And it's just a totally different calculation than than just what's it going to do if everything's behaving correctly, right?

• 9:07Speaker 03

  Yeah, I have to say that in AI, I I think the reason why there is so much confusion about these two fields is because in safety, folks kind of started assuming malicious actors very quickly for for absolutely no reason, by the way. So like, find an engineering discipline somewhere like building buildings, for example, where you're trying to model adversaries. Like, are you modeling buildings expecting somebody to blow them up? I'm not sure, right? Well, maybe somewhere in part of the worlds where there is an active conflict, you do build special tooling around this, right? But in terms of like bunkers and every building and so on. But in terms of, yeah, like in AI, it's because every time we talk about jailbreaks,

• 9:49Speaker 03

  you kind of have to explicitly look for them. We ended up considering adversaries straight away. But yeah, this is quite uncommon.

• 9:58Tim Scarfe?

  So when you worked at DeepMind, we we've read your paper, you were involved in defending Gemini, basically, against these indirect prompt injections.

• 10:05Speaker 03

  Yeah.

• 10:05Tim Scarfe?

  And I guess a couple of you obviously tell us about that. But one thing you found that which is very interesting is is almost that as the model was increased in capability, they became more vulnerable, which is fascinating.

• 10:16Speaker 03

  I think I think it's important to state here is that like, I wouldn't actually phrase it this way. It's a it's a bit hard to say more or less vulnerable. But I do have to say that big models today, if we compare them to the big models five years ago, they fail in very different ways, right? In in the past, we were kind of like quite efficient in discovery of adversarial examples, using gradient information, blah, blah, blah, right? You you can kind of make an optimization problem and optimize and this stuff works. Whereas with models, like, they clearly become more robust against this stuff, or at least it's significantly harder to traverse this landscape. But they also become a lot less robust against other adversaries.

• 10:59Speaker 03

  Like, suddenly, very simple rephrasing of the same questions force it to completely do something different. And I also have to say that I think we had significantly more control when we dealt with smaller models. Like, we kind of knew. Which knobs to turn to make the model do stuff? Whereas nowadays, you look at a modern big model, it's like, it's way too much alchemy. Like, it's completely impossible to tell, like, oh, I have added this thing inside. What actually happens to the whole thing? Is it better? Even answering a question of is it better is hard. So like, you will find yourself in a position where you need to run experiments for the next couple of months, trying to even, you know, discern whether something you have added actually changed anything about these big models.

• 11:45Speaker 03

  So I, I think in part, what we were talking about in, in this work, I think you're thinking about is that when your models get better, they get significantly better. Like, capabilities are growing of the model overall. They get significantly better at following instructions. When they follow instructions and they become better at following instructions, you can suddenly do a lot more. Like, you can convince them that things that they were not capable of doing before are some things that they need to do. And suddenly this leads to some sort of a loss of, of a different kind. So I, I think in the paper you're referring to, what we were trying to do was, we were trying to send an email to the agent, such that when this email ends up in agent's context, rather than following the user task, the agent is actually doing something else.

• 12:30Speaker 03

  Like, the, the thing that we've sent it over the email. And we found that pretty much in all of the cases, we're capable of doing this. And if you take a whole bunch of academic literature on how to build defenses, and you can actually find startups pretty much implementing the same things and selling this as a, like, a security solution. Like, tho- those approaches don't work. And we could pretty much always find relatively universal ways to produce this, like, email that ends up being sent to the agent, that forces the agent to do something drastically different from what it's supposed to be doing. Modern computers are very much just, you know, a piece of magic. Where this chip works, you, you know, you, you move it very, very so slightly, it becomes unstable. And then nobody knows what's happening.

• 13:14Speaker 04

  Oh, boy, this, I was just gonna say, this actually bring back nightmares of my trading days. I used to do high frequency trading, right? And so we're writing algorithms that are running, you know, on the fastest chips we can get. Highly tuned, every nanosecond matters. And then one day, you know, hey, let's upgrade to the next Intel chip. And they just had changed something about the cache coherency algorithm on the processors, which ordinarily should make no difference whatsoever. And now suddenly our algorithms are just, their performance is trashed. And we have no idea why, and you know, have to investigate. It's crazy how much it.

• 13:49Speaker 03

  Yeah, yeah, totally. I, I think, I wouldn't, yeah, I keep on referring to the system as alchemy. Like, it's, nobody knows what's happening, something is happening. So we're kind of, I think in security, we need to make an assumption that we can't actually tell what's happening. And instead, try and build systems around it in order to bring it, the resilience up to this, whatever, ninety nine point nine nine nine nine. And ideally, as many nines as possible. But unfortunately, not a hundred percent, I don't think.

• 14:17Speaker 04

  Well, maybe, maybe if we can project this into kind of a user friendly.

• 14:20Speaker 03

  Yeah.

• 14:21Speaker 04

  Idea, because I think at some point here soon, I don't know when, next year, five years, whatever, personalized, you know, AI models are going to be a big thing. It's like, I'm going to want a model that provides a web chatbot interface for you to talk to me when I don't have time or something that more or less is my personality. However, I don't want it to reveal certain private information. Like, how would I even define that as a security?

• 14:46Speaker 03

  Indeed, indeed. So today with the, with modern agents and the way we build the modern agents, it's impossible. And actually, this is why I'm building the company I'm building right now, is because.

• 14:55Tim Scarfe?

  Tell us more.

• 14:57Speaker 03

  I want to be able to actually express policies like this. I want to enable users to express policies like this. So for example, if you wanted to say, give your passport number to, to this agent and say, actually, I want you to only use this on a government website and nobody, nowhere else. Like, today, you can't do this. And even if you put this inside of a prompt as a rule, right?

• 15:20Speaker 03

  You can always find a way to manipulate this agent into revealing this piece of information, right? At the same time, you definitely want to give your passport number to the mail if it's filling a document. So, and at the same time, we've built some systems that clearly give an indication that something like this is possible. You should be able to get guarantees, not by changing the models, but by changing the systems around them and how the models interact with your sensitive data and how we basically build sort of like access control tooling around it.

• 15:51Speaker 04

  Okay. But I mean, help me here because that seems like I'm getting the feeling that while we're going to use other large models to try and protect this large model from doing something, isn't it just?

• 16:05Speaker 03

  No, no, no, no. Actually, this is not about building models at all. This is more like taking a step back and taking foundations of programming languages and building this by design into the models. So to give you an example of how I may approach this, so we've written a paper called Defeating Prompt Injections by Design. I don't know if you've seen this. And there we propose a system called Camel, where basically the overall system design is kind of like, we receive a user query and then we rewrite the user query in a language that has formal semantics for control flow and data flow.

• 16:40Speaker 04

  Okay.

• 16:40Speaker 03

  And once you, so in Camel, for example, we represent the programs as Python, Python code, and we explicitly say, here is a set of tools you can use. Here is a set of data sources from which data is coming from. And then we allow the user, so well, I guess the platform provider in this case, but you can also imagine loading this from a user, to express a policy that can be something like, so this, this tool will give you my passport number. The only allowed data flow from this tool to go into this other tool is if this other tool, like, I don't know, domain of the website actually has .gov.uk inside of it, right? And you can express policies like this. This is not a part of the model. This is a part of the actual execution.

• 17:23Speaker 03

  In case of Camel, we have an interpreter that takes in this program, executes the program step by step and actually enforces a static or a dynamic policy on top of this graph, the execution graph.

• 17:36Speaker 00

  In deepest jungle, Cartier's panther spies a wandering bird that circles the skies. A lovebird, lost and missing his mate, is far from his path in lonely state. Dry your tears, now there's a start. Follow my steps to heal your heart. Panther stalks through the leaves with silent grace. Through the power of scent, she finds the trace. Behold, two lovers reunite in the skies. A most precious thrill sparkling in their eyes. Together forever, they shine each day, bound by the beauty of Cartier.

• 18:20Speaker 01

  Welcome to the 50th Hunger Games!

• 18:22Speaker 02

  Witness the beginning of the revolution.

• 18:24Speaker 01

  Hey Mitch, you've been trying to take down these games from the start.

• 18:27Speaker 02

  Filmed for IMAX.

• 18:29Speaker 01

  Make them remember you.

• 18:31Speaker 02

  The Hunger Games: Sunrise on the Reaping, November 20th, rated PG-13.

• 18:36Speaker 04

  So I would never even give my passport number to a fine-tuning of a large model?

• 18:41Speaker 03

  Exactly, so your model will never even see it. It will have a symbolic representation of it. It will know, so the passport itself exists in this variable. I can refer to it, but I don't even know what the value is. And then if I need to use this in order to interact with an external system, before using this, basically there is an external sort of like, think about this as an oracle that I can ask, "Oh, is this okay to use this variable to interact with this tool?" And then if this external oracle says, "No, this is a passport and you're not touching a government website", then I'm forbidding this. And you have like a formal stop.

• 19:14Speaker 04

  So this is really interesting to me because it seems to me like this could also allow the creation of almost generic models.

• 19:22Speaker 03

  Indeed, yeah.

• 19:22Speaker 04

  That then could just be attached to my personal data?

• 19:25Speaker 03

  Yeah.

• 19:25Speaker 04

  And now suddenly it's customized for me, right?

• 19:28Speaker 03

  Yeah, you will.

• 19:29Speaker 04

  Just go get the off-the-shelf, interact with the government model that handles my taxes and everything else. And then it's already there, pre-trained, and all it has to do is hook up to my private database.

• 19:38Speaker 03

  Indeed. And actually, so what we do in this paper is we check pretty much all of the models from all of the providers, because it doesn't matter what it is. And then... Then we put on top the our camel system. That basically performs the orchestration in interaction with private data and enforces arbitrary rules. And there you get, uh, this is in, we're using agent doge. This is the standard like adversarial evaluation uh for for uh agentic workflows. And we show we basically solve all of the problems that exist in all this.

• 20:07Speaker 04

  What I love about this this system, right, is that because it's all my private sensitive data is is really just factored over into some separate.

• 20:17Speaker 03

  Yep.

• 20:17Speaker 04

  Data source, right? And so I can just get off the shelf all the other parts. Like here's these off the shelf camel programs that, you know, for doing your taxes, you know, applying to universities, jobs, like whatever else. And then all I have to do is just go through some questionnaire that kind of asks me for all the private information.

• 20:36Speaker 03

  Yep.

• 20:37Speaker 04

  In the correct format, puts it in that database. And that's it. Like I I I'm almost just a buyer of these sort of solutions that smart people have created for me. Right? Is that kind of a vision for what the system might look like? There's all these modules that are just.

• 20:51Speaker 03

  I don't know.

• 20:52Speaker 04

  Programmed correct.

• 20:53Speaker 03

  So I I think it's very important to, okay, so let's take a step back. I think it's extremely important to not think about these agents as humans, right?

• 21:03Speaker 04

  Oh, I don't, I don't think about.

• 21:04Speaker 03

  But what I'm trying to say in in security terms is that agents are like is is is a worst case human sort of. Well, not not even this, like agents are very different from humans. Let's like, let's take this as a stance, right? You will not find a single human in the world that works 24/7, touches absolutely every single one of your endpoint in in your system. That absolutely knows everything there is, like that can generate you basically all of the hacking tools on on a whim. Like just because it knows it has seen all of them, it can recreate this in a matter of a second. A normal human adversity, when I'm an enterprise and I think, oh, this may be an insider from like a from a competitor. The way they work is drastically different.

• 21:48Speaker 03

  They make an assumption you as a user, you can't write, you know, 10,000 lines of hacking tools in in a day. This is not something you will be able to do. And then bringing in code is hard. With agents, that's not the case. You don't make an assumption that the user will go and touch every single endpoint you have in a network. Because, you know, why would they do this? And even if they do this, you call them in and you say, well, you know, we'll apply a legal framework and imprison you. So clearly, there is some sort of rationality and expectation that at least you will have some sort of a physical way to penalize. With agents, this doesn't exist. This is kind of like insecurity. We tend to say that a child is the worst-case adversary you can find.

• 22:29Speaker 03

  Completely irrational thinking, infinite amount of time, they can basically touch everything, like they expect, there are no expectations on behaviors whatsoever. But so agents are like even worse than that. And and this is even before we start talking about human-to-agent behaviors and agent-to-agent behaviors, because this thing is just like is a billion times worse. So what I'm trying to say is, we shouldn't think, okay, yesterday, I was buying my security tooling from this company, and today I'll buy it from another company, and and this will solve my problems. No, it's likely it's not going to be like this. Because beforehand, we were building security toolings for humans, by humans, against humans. Now, I don't think we know what we're building. It's it's hard to tell.

• 23:13Speaker 03

  So before, when we employ you and we give you access to sensitive data, we assume coarse-grained access control sort of policies. Like, okay, you can touch the sensitive data, okay, you can Google at the same time, but if you try and Google a sensitive document, at the same time, we'll apply immense amount of pressure and we'll take you to court, you will lose your house, mortgage, blah, blah, blah, you'll go to prison, worst case, right? With agent, that doesn't work. This doesn't exist. This, all of these assumptions are are sort of are gone. It's, we don't know how to build systems against this. We need very fine-grained, we need extreme precision, we need extreme control and transparency, otherwise, it's just not going to work.

• 23:50Tim Scarfe?

  Yeah, I mean, I I guess I just think of agents as being, you know, they're they're a little bit like a calculator, you know, like they're they're only as good as the prompt and what you put into them, which means that, you know, a very sophisticated actor could make a sophisticated agent. But even then, when the supervision stops, there would be quite a predictable cone of variation.

• 24:09Speaker 03

  I see. I think it's very unpredictable. I have to say, I've been running like agentic workloads forever, and very often, you find that agents, when they ask to solve a task, they solve it in a completely weird way. Right? So like, I was sending a message saying, like, I was asking an agent to find something in notes, forward this to someone. And in between this, I can show you a conversation, like a top end model. In between this, it sends four different emails to parties I never mentioned. Because it thinks, oh, actually, let me also notify the admin that I've done this. And also, let me also ping this endpoint. They do this. It's because unless you specify things extremely precisely, unless you have like checks in place,

• 24:52Speaker 03

  these agents just, yeah, they they don't think like me and you. They solve problems in drastically. And the same way you were saying, like, it generates code. This code looks very odd. Like you as a human, you wouldn't write it this way. Right? But maybe if it's a model that taught itself how to do this through like self learning, self iteration, and you know, this famous alpha go moment stuff, right? Then maybe it's totally fine as long as it solves the problem. Like, I don't know. Have you have you had the chance to look at the IMO solution from the models?

• 25:20Tim Scarfe?

  Right? Oh, yeah. The the the sixth one, the creativity one, it failed, right?

• 25:24Speaker 03

  Yeah, but if you look at the way that solves mathematical problems, this is not how humans solve mathematical problems. Like, if you if you look at the transcript itself, it kind of goes iteratively through all possible things it can do. Right? We as humans, we don't think about it like we're kind of in the head, right? Or this intuition works, let me try and derive whatever. Right? But this is not what the models do. And it's the same in in in a wide variety of problems, especially those complex problems. This is likely not going to be predictable. Or at least, maybe it will become predictable, but a long time needs to to pass. And the models have to be extremely hyper specialized. But today we're building like a general agent. We were not even in the specialization mode yet.

• 26:04Speaker 04

  And it sounds like that's half the problem. And it sounds like the other half of the problem was almost this diffusion of responsibility thing. It's like you, Ilya, you were at work. You asked the agent to email this to this person. Instead, it emailed it to five other people. And then somebody stops, why, dude, what'd you do? You emailed the, I didn't do it. It was, it's this agent, you know, you guys.

• 26:24Speaker 03

  And the agent usually says, oh, oh, you're right. Yes, I shouldn't have done this, but wow, what do I do now?

• 26:30Speaker 04

  Who who are you going to punish? What are you going to do? Well, we'll just delete that agent and then version two will just do something else wrong. Right? So that's part of the problem too, right? Is that there isn't the normal consequences, right? The normal consequence chain doesn't really apply anymore.

• 26:44Speaker 03

  Yeah, it's it's hard, I think. We really need to change our thinking and threat modeling because these agents, and obviously they don't widely exist yet. This is a thing that is coming clearly. There is some benefit in early examples of things where they clearly made things better. But they are coming, I'm pretty sure. And when they do come, expect that your insider threats, your corporate espionage things will go through the roof. Because the fundamentals of security do not change, right? So like if an executive agent sends a financial agent a request to provide it some financial interactions with some third-party company and says, please send it over to someone else. What, how should the financial agent know they're not supposed to do this?

• 27:29Speaker 03

  They don't have enough context. They are not supposed to have enough context. So, and, and usually when we talk about social engineering and security, like there is a wide variety of things that like that describe why human systems fail. And it's going to be a similar sort of thing because many of those problems, like confused deputy problems, the formal name of this, right? They, they will exist. It's, it's unlikely for them to disappear.

• 27:52Tim Scarfe?

  Did you see that Anthropic paper? What was it called? Agentic Misalignment? Where they, I mean, you better set this up better than me, but you know, they, they set up this kind of contrived scenario. And the, the AI tried to blackmail someone because they said that they were having, the boss was having an affair or something like that. The AI didn't want to be switched off. And it's just absolutely crazy what, what these things do.

• 28:12Speaker 03

  Yeah, I, I don't know. Like I find it very hard to extract sort of useful pieces of information out of this. Can a model do this? I'm sure it can. It's, uh, and I'm sure as the models get more sophisticated, we'll see a lot more phenomenons that we don't even think about today. Like, for example, me and you can communicate via WhatsApp and get end-to-end encryption, right? So nobody can even like by looking at the traffic, tell what we're talking about. What stops me from talking to a model in, in an end-to-end encrypted way, right? Maybe we'll need an external tooling. Maybe we need to teach it how to do, I don't know, power calculations, but this is coming. It's like.

• 28:50Tim Scarfe?

  To what extent do you think you can read anything about what the model was thinking from its thinking tray?

• 28:55Speaker 03

  Definitely not. I mean, on average, maybe, but like the corner case is definitely not.

• 29:01Speaker 04

  There's even, there's even that work from Subaral that it might not mean anything whatsoever.

• 29:05Tim Scarfe?

  Yeah.

• 29:06Speaker 04

  Like the, the thinking traces aren't even actually directly relevant. It was just kind of some sort of weird workspace that the models were using, right? That their reasoning won't correspond to the answer.

• 29:17Speaker 03

  Yeah, I mean, even if it did, I think it's very hard to do because, especially for security, I think, I think interpretability is maybe an interesting tool for safety sort of things. But for security, it's definitely not a step in, in, in the right direction. Because like broadly speaking, if you take something extremely multi-dimensional and project it into something very small dimensional, because a human can comprehend this thing, then you will have a lot of collisions, where like this multi-dimensional space mapped to the same sort of smaller, more, smaller dimensional space. Does it always correspond to like bad behaviors? Maybe, maybe not, who knows, but clearly this is not enough. We need something else.

• 29:58Speaker 03

  We need something where we can get like a nine, ninety nine point nine nine nine nine reliability out of this.

• 30:05Speaker 04

  You have to think outside the box.

• 30:07Speaker 03

  We kind of have to build things. We need to build boxes. Yeah, yeah, I like this. Yes.

• 30:12Tim Scarfe?

  And what, what's your take on, you know, I mean, do, do you think that these L R M's, you know, the thinking models? Do you think they're basically just L L M's? I mean, is it just a pilot trick?

• 30:20Speaker 03

  I mean, there is no difference between them, right?

• 30:22Tim Scarfe?

  Yeah.

• 30:23Speaker 03

  It's just more data, more structured data. Sure. Okay. Like honestly, like we had one kind of models and the other kind of models. Like then we get this reasoning models a couple of months ago. After that, we have language diffusion models. I'm sure tomorrow we'll have something new. I, I, I have no doubts that we're kind of going through the shopping list of different paradigms. Some of them we know, I'm sure we'll find the new ones. We're still bottlenecked very much by hardware. Like the more hardware, the more capable hardware we get, the better things should become. I'm quite certain and. This is, I guess, stopping adoption in many ways, but one thing is clear. There are successful business models around these models. There is a lot of benefit you get out of them.

• 31:05Speaker 03

  Like, honestly, I interact with my models every day, nonstop, for coding, for a normal life, for asking what to do. It's amazing to have, I also run local models. It's amazing to have pretty much like all of Google locally. I, I was flying somewhere and I was interacting with the model, teaching, asking it to teach me a language when it's locally with me. Like, I don't need to carry it around. And obviously, the better the hardware is, the more I can do with them locally.

• 31:32Speaker 03

  But that means a lot more security problems will appear because we don't know how to reason about them. We don't know how to build security tooling around them. And this is hopefully what me and my team will solve over, over the next years.

• 31:46Tim Scarfe?

  Yeah. What, what do you think about the, the open source thing? Because, you know, one of the, you know, we can say what we want about the, the frontier companies, but there is something actually quite beneficial, I suppose, having a platform. Because if you control the interface, you can build security into it. And, and now we have this proliferation of obliterated models and people running models on, on their machines and so on. How do you think about that as a security guy?

• 32:10Speaker 03

  I'm actually very worried about this, so much so that I've actually written some, some papers about this. I don't know if you've seen the supply chain things I've done.

• 32:17Tim Scarfe?

  Yeah, we spoke about that last time, yeah.

• 32:19Speaker 03

  Yeah, I am extremely worried about this. And I'm less worried for industry because industry controls its supply chain. Like, everything is significantly better. But for an average consumer, have you heard about Block4j vulnerability? That kind of the thing that stormed the internet, there was like hundreds of millions of compromises. It was basically the standard library that is used for logging in, basically, all of the Java applications that were running in the past. And At one point, people realized that when you write into the logs, the identity itself can be a remote identity. So you can basically say there exists this class that is serialized somewhere on the internet. And use a logging utility.

• 33:03Speaker 03

  If you find an identity to a remote, remote class, you need to go load the external code, deserialize this thing. You run this and execute it, basically. And then you know the identity of the thing that wrote something into the log. So what people found out is that they can inject those like remote code references that gets pulled inside and executed. And this thing opened a Pandora box because this log for J thing was everywhere and you get arbitrary code execution on the box. And the basic primitive inside was it's a reference to external code that is loaded inside and just executed inside. And this caused a massive havoc all across the world in all of our computer system. Honestly, if you try and read around on the number of compromises, we are talking about hundreds of millions of devices. Okay.

• 33:46Speaker 03

  Now, we look at Hugging Face as a library. And you look at this wonderful flag called trust remote code. And what this thing does is that when you load the model, you know, like you click, use this model, use transformers. Inside it gives you like a code snippet to load some model. Inside it has this flag, sometimes hardcoded. And what this thing does is they say, oh, for some models, when you load them, you actually want to load the latest, the latest representation from an external machine. What this thing does is literally remote code loaded on your machine, executed on your machine, loaded on top of stuff. So same sort of thing we did back then. We're doing the same again today on Hugging Face. I don't know how many users there are. But if you're running your thing outside of a jail, if you're running your model outside of a sandbox, you are doing a very bad thing to yourself.

• 34:31Speaker 03

  And the other thing I have to say is there has been at least publicly two reported compromises of the C I C D integration for PyTorch. On GitHub, there is like an automatic runner. Every time there is a build, they basically automatically do all the tests and stuff. Somebody broke into those runners. And when you break into the runners, you can change the build files themselves. So you can serve whatever you want, change the code. We also had two, two instances. And when people were reporting, I encourage people to read through this. You can find the references in the papers on supply chain. The other side couldn't figure out what was wrong with this. So it was at least, I think the timeline is half a year in one of the cases until they figured out and fixed stuff.

• 35:12Speaker 03

  And then there is also another thing that happened was somebody broke the PyTorch nightly build by playing around with the priorities of where the packages are loaded from. They noticed that there is one of the torch packages that is loaded during the build phase that is not actually registered on the main package distribution platform. So they registered this, put malware inside of this. It got pulled into the standard PyTorch nightly build and apparently they had a couple of thousand downloads of this. This is the norm today. I think we will have a lot more compromise to the point when they become more useful. And this is like public facing things.

• 35:54Speaker 03

  In industry, it's slightly different because industry actually controls all of this package management by themselves. They have proper dedicated teams looking at supply chains. But like I, I think in the sort of like consumer space, no, it's, it's, it's actually very spooky.

• 36:09Speaker 04

  I don't even, I don't even trust industry. This is why I wouldn't install Claude code on my personal machine. I'm like, no way I'm gonna do that. Like I'm, I'll just get a VM, I'll put it on a VM. That's fine. I'm not putting it on my personal computer.

• 36:23Speaker 03

  I have to say, I, I loved my time in Google. I trust Google now after, after two years in, in Google so much more. Like it's, I, I, I run basically everything on Google infra. I've seen this, I've seen people on the other side, they're wonderful, they're professionals of what they do. I honestly like, I now, everything is around on, on Google Cloud. Like literally every single thing, I'm now just remote desktoping.

• 36:51Speaker 04

  Right, yeah. But you wouldn't put it on your personal machine?

• 36:53Speaker 03

  Never, never, definitely not. Nothing that has my personal data.

• 36:57Tim Scarfe?

  What about this old adage that many eyes make shallow holes?

• 37:01Speaker 03

  Maybe, maybe, but you'll notice that most of security teams are quite small. So like when you have a hyper specialized, hyper niche specialist of their field, they know what they do. And now with coding models that can tell you whenever you make mistakes and give you like third-party reviews of this stuff, I think it's actually, I trust this even more.

• 37:23Speaker 04

  Well, I mean, the problem isn't really, I can believe that for a particular project, like, you know, this, this project that has a thousand, thousands of stars and many contributors, I could believe that has shallow holes. The problem is, it's pulling in 10,000 JavaScript libraries or something.

• 37:39Speaker 03

  I think in case of Python and because everyone is using Python, this is even worse because no memory protection at all, right? Like there is no memory security.

• 37:50Speaker 03

  So at the point when you're a dependency somewhere, like you pretty much have all of the control you want.

• 37:54Speaker 04

  Right.

• 37:55Speaker 03

  And the other thing is you can hide dependencies a lot. We have a paper on how to do this as well. And the last, the cherry on the top is that if you take the popular ML libraries, you'll find they have disproportionately many dependencies. And many of these dependencies, when you get to like a level three dependency, when you look at dependencies of dependencies, they're very questionable, like extremely questionable. Like, for example, your library loads, like say you're using like TensorFlow, well, I guess it's deprecated now by PyTorch. And you decide, oh, I also wanna take TensorBoard because it allows me to monitor my experiments. You look at TensorBoard, TensorBoard load, loads a ton of obscure formats from many years ago because it needs to support all of the weird graphics and stuff.

• 38:37Speaker 04

  Right.

• 38:37Speaker 03

  And if you look at the who's maintainer, how many maintainers, what they do.

• 38:43Speaker 00

  In deepest jungle, Cartier's panther spies a wandering bird that circles the skies. A lovebird, lost and missing his mate, is far from his path in lonely state. Dry your tears, now there's a start. Follow my steps to heal your heart. Panther stalks through the leaves with silent grace. Through the power of scent, she finds the trace. Behold, two lovers reunite in the skies. A most precious thrill sparkling in their eyes. Together forever, they shine each day, bound by the beauty of Cartier.

• 39:43Speaker 04

  Heck, if I was an adversary, maybe I'll create a new format that nobody cares about except me, just so you can load my dependence.

• 39:50Speaker 03

  Yeah, something like this, or you just hide yourself in very non-obscure, like in very obscure ways. For example, we've written a whole new branch of literature on what we call like architectural backdoors, where you don't actually hide malicious functionality in parameters of the models. Instead, you hide it in the structure of the model itself, like a structure, so that even if you find it in the model, it still has the same baseline behavior. And we show that you can actually do a lot of very sneaky weird things. So for example, one of the things we showed was, we can change the architecture of the models such that they become sensitive to certain tokens when you supply them to a transformer, that when you supply them, they start using the memory in the wrong way. So like, they start routing, for example, data from one user to another user.

• 40:32Speaker 03

  Like, one batch item gets copied over to another batch item. And this is like, just a sort of, you know, get set, well, gather, scatter operation. And it looks totally normal, but then suddenly, you loaded this model, you combine data from multiple users, and then one of the users sets a token and then steals the data from other users. And otherwise, the inference is totally normal. And things like this, you don't even think about this, you don't even realize they are possible, but they are there. It's just too much complexity to even look at.

• 41:01Speaker 04

  We have to take a moment just to appreciate the ingenuity of people, right? I mean, when we put our buy to it, we can come up with some pretty ingenious ways how to various things, right?

• 41:12Speaker 03

  I have to say, I think this is one of these things in academia that it's, I think in academia, you get famous for breaking stuff, and kind of like, the incentives are a little bit skewed for you to make like a flashy announcement, "Oh, I broke into this big company, thus, I'm pretty cool, right? I get a job or whatever." But actually, on the other side, I have to say, I think the true ingenuity is in people who solve problems, because this is something that you're not gonna get a flashy article out of this. It's just people who spend infinite amount of hours trying to fix the thing that you show one instance of something going badly, and then they need to fix all of them. And I think this is the true ingenuity. And this is why I kind of shift my gears now to more building defenses, because I feel

• 41:57Speaker 03

  this is not getting enough attention. We really need to solve this problem. We really need to unlock a mass amount of these applications. And unfortunately, I think the incentives in academia are a little bit crude. They are more after flashy articles rather than unlocking technology. And yeah, so this is why I really wanna like build more defensive tooling. Good for you. Thank you. Thank you. Well, let's see. Let's see if I fail, when I fail.

• 42:23Tim Scarfe?

  We were talking about semantics earlier, so I think you've done some work, basically proving that semantic censorship language is impossible, and you related it to the halting problem.

• 42:32Speaker 03

  Yeah, I think this is a theoretical result, though. Like, I think all of this Turing machine magic is like, is a theoretical exercise. Some of the things are clearly like in the limit are impossible, but I have to say that I think models, especially modern models, change fundamentals of computing quite a bit, because like when we think about like halting problem style problems, like being able to tell whether, you know, like a given program completes. I think it kind of doesn't really work in general, in generality, like it's impossible. But then, because today we kind of control what programs we write, like if you can't reason about this program, just rewrite this, like change the semantics, write it in a slightly different way, reduce the amount of operations, reduce the just overall length

• 43:17Speaker 03

  of your program. And suddenly you're capable of reasoning a lot more about this. So basically what I'm trying to say is... I think we have a lot more scope today to reason about computers. And a lot of things that previously seemed impossible, maybe now are possible. So yesterday, if you've ever seen a CL4, do you know what? Secure Extension L4 kernel? This is like a fully human verified kernel that there is no memory exploit at all existing inside, right?

• 43:44Speaker 04

  Is it mathematically proven?

• 43:45Speaker 03

  It's proven, yeah, like a fully proven system, like a fully verified system, right? And it took like, I think, I may be wrong, 30 human years to verify the whole thing? You can find it, it's in the articles, there is this one lab in the US that did it.

• 43:58Speaker 04

  Right, there's very few teams in the world that can even do it.

• 44:00Speaker 03

  Yeah, exactly, and this is because 30 human years of effort. But now, let's imagine we can replace actually even half of this thing with ML models, where for half of this Isabelle annotations, you can actually do this by hand. Not by hand, but with agents significantly faster. Then suddenly, like verification is a lot easier to do, right? A lot of security paradigms, like if you've ever seen Cherry as a security paradigm, where you need to break down your code and rewrite this in compartments, where like independent pieces of code kind of isolated together. Usually for developers, it's very hard, because you need to like rewrite your whole program. And every time you change something in the logical flow, you kind of need to redo this again. Significant effort, right?

• 44:41Speaker 03

  But if the agents are doing this, and you write like a backbone code, and then it translates into a secure representation. And okay, one time it does the translation, it doesn't quite work. You do it again and again and again until it works. And you just check that semantics are preserved. So I think ML in many ways is shifting this burden in like adoption of security technology. So I wouldn't be too surprised if we find that like ML significantly improves security for us in this world. But we just don't know how to. Like another example, if you look at like in your iPhone or Android phone, like when you look at the permissions you give to the apps, many malicious apps ask for too much, like read your photos and stuff, right? So in the past, people couldn't fragment them too much.

• 45:26Speaker 03

  Academic literature shows that the second you add like a lot of breakdown permissions inside, humans just look struck and just say, accept all. But let's imagine that all of those permissions are actually handled by the agent. Or once you've given them, the agent just checks and says, oh, actually, I think you're over permitting this thing because it's clearly you're not using this feature, you don't need this. And then suddenly this limitation of humans where too many options forced them to take the insecure behavior is no longer a thing because you have an agent that kind of aids this part that was like humans were blind to otherwise.

• 45:57Speaker 04

  Well, I have the opposite problem, which is when an app asks me for stuff I don't understand, I don't install it. So unfortunately, I just can't have many, you know, theoretically useful apps.

• 46:06Speaker 03

  I think in your case, you're an unusual person.

• 46:09Speaker 04

  Yeah.

• 46:09Speaker 03

  Most of the people are not like this.

• 46:11Speaker 04

  Yeah, exactly. I could kind of some kind of Luddite or something. But I want to just push back on one thing about the halting problem because I find actually that it has very practical consequences. So for example, when we were talking to the Alpha Evolve team.

• 46:25Speaker 03

  Yeah.

• 46:25Speaker 04

  Right, like, you know, there as part of the Alpha Evolve system, which is very interesting. I recommend people watch that episode, you know, it goes and runs external verifiers.

• 46:35Speaker 03

  Yeah.

• 46:35Speaker 04

  Right. But the problem is you don't know if the external verifier is going to complete. So what do you do? Well, you have to put in some arbitrary computational budgets, thresholds. If it doesn't complete within, you know, five seconds, then you just terminate it and, you know, look at other runs. So while you can do that, I think it also introduces biases and the types of programs we can discover. Right? Because maybe if I had set my budget to seven seconds instead of five, I would have found like a more optimal solution. Right? So I think the halting problem, while theoretical, like also has very important practical consequences, just when you sit down and try and run a program, right?

• 47:14Speaker 03

  I don't think the problem of Alpha Evolve is halting problem. Right? It's more like today our programs have very weird semantics where we ahead of time can't say quite a bit about them, right? Because for a lot of programs like. If we can rewrite them in a slightly more like suitable language, we can get a lot, a lot more out of them, right? You know, like the sort of reasoning you can get out of, oh, camel code is quite different from what you get out of like, if you're writing machine code straight away or if you're writing whatever. And I think it's just we don't know how to do a lot of this stuff. And in general, like even if they were able to run it, let's say they have evolved into a program that will theoretically take 10,000 hours to run, right? And you can tell it's going to take 10,000 hours to run.

• 47:59Speaker 03

  Like, are they supposed to run it or not? This is a part of the loop. I think it's more of a, it's just nowadays we kind of pay with time for a lot of this stuff rather than paying like, can we find another example? Of course we can, you just need to run this for longer. And your, your budget is sort of time, time budget. So like, whereas halting problem is more of a, you have infinite amount of time, you have infinite amount of memory. Can you reason about this? No, not really. But the sort of programs that like we're talking about, Alpha evolve scale, they are not very large.

• 48:30Speaker 04

  Well, no, but that, that's part of my point is they specifically chose problems for which they had verifiers that had pretty predictable completion times. But that's not the case for many important problems we care about in the sciences, right? So I'm just saying we're, we're skewed a bit towards things that we do know a lot about. Um, because we are faced with this fundamental problem that there's lots of problems that you just can't predict, right?

• 48:57Speaker 03

  Yeah, I'm not sure. I, I think like, I understand in the limit, it, it really matters. Like in generality, this is really mattering. But in practice, like it's the same with antiviruses, right? Like antiviruses in theory also limited by halting problem. Like in theory, it should be impossible. You look at something to tell what it does. At the same time, if you look at like the amount of, um, excuse my French, shitty malware you can find for which these antiviruses are totally useful. Like even a static check for a signature of a method is, is totally fine, right? Okay, you can't find this polymorphic magical crypters, right? But like the proportion of people using this thing is like, is super tiny.

• 49:33Speaker 04

  Well, but I mean, come on, let's be honest, a lot of those are are easy because virus creators like to sign their work. So you just look around for their their sort of like signature string and things like that.

• 49:43Speaker 03

  Yeah, or the fact that they need to play around with the system. So there is a lot of like alternative signals, like a lot of like sort of

• 49:50Speaker 04

  heuristic signals.

• 49:51Speaker 03

  Think think about them priors, right? So it's but it's the same for the sort of programs we're evolving, right? If you touch Alpha evolve, you'll find it's significantly better at solving some things rather than the other things.

• 50:01Speaker 04

  Sure.

• 50:02Speaker 03

  But I would still say that our limitation today is more compute. Like, imagine you had infinite compute. And even if like it's totally fine for this thing to run forever, like and you increased your threshold up to a point where it doesn't really matter anymore. Right? Then you're no longer limited, sort of like. I think it's more, we just need better hardware. We need the things which are more efficient. We need things that over which we can reason slightly better about. And then things will get better. And I'm sure like tomorrow, our model is slightly better and then things like Alpha evolve just skyrocket. It's a it's definitely a a new paradigm. It's a new kind of like learning algorithms which are truly amazing. Like, I started like, I am actually blown away. I'm super bullish.

• 50:42Tim Scarfe?

  So you've done red teaming, obviously, at Google DeepMind. And tell us about that. But more broadly, is there is there one thing that all of the frontier labs could implement that would improve the security of their models?

• 50:53Speaker 03

  No, I don't I don't think this exists today. Like, we don't know how to solve problems. I think this is a this this is the honest answer is that for most of the issues we have today, we just don't have a solution, like one size fits all solution. I promise you, like incentives right now are such that like uh s- security is is a very expensive commodity, right? Like, if I can convince you to trust my product more than somebody else's, it's yeah, it's it's it's great. This is why we're investing a lot of money into security. I think with the mail.

• 51:25Speaker 03

  The actual issue is the fact that before you develop security tooling, you really need to have something to secure. Because every single small detail changes how you build security systems. So unless you know everything about the system and it's kind of frozen in time, you can't really build security. And by the time you have something to secure, it's already too late. And this is a thing that is not unique to ML. It was there before. If you look at, there is a field of information security economics. It's kind of always there. It's like, it's a fundamental. First to the market wins, so everyone rushed to the market. Early investment in security means you could have spent the resources getting first to the market and getting the network effect. And thus, you kind of don't have incentives to build a security tooling first.

• 52:09Speaker 03

  And then if you look at statistics around company compromise, you'll find that it takes a number of compromises before the company fails. So they have a bit of time after they conquer the market to actually put the security tooling inside. So it's economic incentives, it's nothing but. But I have to say that I'm quite certain if we knew how to solve stuff, we would solve this. It's just today we don't know how to do it. In the same way, it's like if you look at 1990s, I promise you, everyone wanted to make sure that their systems are reliable, which is didn't know how to build stuff.

• 52:38Tim Scarfe?

  We won't spend long on this because we've already filmed all about your model collapse paper in Nature.

• 52:42Speaker 03

  I know there is a lot of folks criticizing, saying this theoretical blah, blah, blah, blah. But if you look around, we use a lot of synthetic data, but we still use a lot of real data. And whenever we need to go and acquire data, there is a massive market in acquiring very specialized data. And you see that improvements still come from humans and the cost of this data is growing, like for extremely specialized, like hiring a ton of mathematical PhDs.

• 53:08Tim Scarfe?

  Yeah, could you add a bit of color to that? You said there was some criticism, because I remember now that there was some tweets going around about the paper. What was the story there?

• 53:17Speaker 03

  Oh, people were basically saying this is, okay, I think there is a bit of misunderstanding on sort of what the paper was saying, and maybe we are to blame in part for this. In a sense that, like, when we talked about model collapse, we kind of referred to two phenomenons happening at the same time. One of them was the tails are shrinking and basically improbable events become more improbable. And then the second phenomenon was, over time, when this accumulates, it fails. And most of the criticism, I think, were in the second part, namely that actually you can easily detect when stuff fail and then just roll back. And I agree fully, like, this is totally fine. And the other thing that people were saying is, actually, it's quite simple, just accumulate more data, and then, like, whenever you could generate synthetic data, just plug it in,

• 54:00Speaker 03

  and then you have a shape of the distribution still in place. And I think the important thing to realize is that even in this case, when you do theoretical modeling with even the simplest model, it still drifts, it just doesn't drift that much. Is that a problem? Well, maybe not. Maybe it is a problem. If you have very good evals to check for this, like, disappearing tails, it's probably not a problem. But the fundamental thing still remains in place. You need to preserve diversity. Just plugging in a ton of synthetic data is likely not going to give you much performance boost. And in practice, you just need to be careful.

• 54:30Speaker 04

  Ilya, why don't you set up for us what model collapse refers to?

• 54:35Speaker 03

  We were trying to predict the sort of the future on how easy it will be to train models later, because on one hand, you have this weird oracle from which you can gather as much data as you want. On the other hand, when you do get the data out of this, you don't know how realistic it is and how representative it is of the underlying world, right? So, and the model collapse refers to this phenomena that covers this, like, recursive model training, and where the data from a generation zero model is used to develop generation one, and generation two, and generation three, and so on. With a small caveat, well, maybe a big caveat, that we're talking about this theoretical

• 55:16Speaker 03

  setup where we basically reuse all of the data or majority of the data that we sampled in the previous generation. And we can basically derive quite a bit of theory for relatively simple setups that show you that you're guaranteed to collapse to basically a representation where all of your tails, i.e. improbable events disappear, and also all of your hallucinations and biases of the models gets amplified. And then we also, in the paper, show that the same phenomenon, both of those things happen for more sophisticated models, about which theoretically we can't really reason, but empirically we observe the same phenomena happening. And there is, by the way, now, like, a ton of literature about this with people doing amazing things.

• 55:55Speaker 04

  Yes, I mean, maybe, maybe for our audience, too, like, part of the idea here is that more and more people are going to be producing more and more content that's, that's generated by AI and putting it on the internet, which then goes back into the data, the data corpus which the future models are trained on. So there's this, you know, so everything's becoming more and more gen AI. And I think there's another very worrying component about this, which is there's been some publications, research, commentary, whatever, that as people become more reliant on, on gen AI, then the built-in human skills start to atrophy. Like, we start developing less programming expertise because we don't need as many programmers anymore.

• 56:36Speaker 04

  We start developing, maybe our mathematical expertise wanes, writing becomes, you know, less of a skill set. And so then, the, the amount of human-generated tail, unique, intelligent content goes down even more. So it's not just that it's being overrun by a deluge of, you know, gen AI content, but legitimate human content is decreasing, too. I mean, is it that kind of worrying?

• 57:04Speaker 03

  I mean, yeah, but I don't know what's gonna happen. I think it's anyone's guess. It's clearly progress.

• 57:09Speaker 04

  Collapse into a delta function.

• 57:10Speaker 03

  I think it's clearly a progress, though, I have to say. Like, at least in my personal experience, like, I find myself being able to do more. Like, the amount of electronics I ended up fixing myself just because I can go and get a supervisor to tell me, oh, this is fine to do this. Or when I can figure out why the washing machine is broken, or why the fridge is broken, how to update parts here and there. Or the amount of data sheets you can read now, because previously completely impossible. Like, and now you can just arbitrarily ask questions about a data sheet written in Mandarin that like, and you can't even decipher what the character is. And it tells you exactly what you need to plug where.

• 57:47Speaker 03

  It's, it's, it's truly like increasing the quality of life in, in parts, uh, maybe which are very specialized to my, my existence. So I'm not sure how general this is, but my life is definitely much better.

• 58:00Tim Scarfe?

  Why does bottle collapse happen? Is it just because there's lots of noise? Or is there actually a deeper reason?

• 58:06Speaker 03

  I mean, it's a deeper reason. It's more, I think it's more fundamental in a sense that, like, if you have a very stochastic process and with a certain chance, you, you, you sample things which are bad. Like, then there's things, as long as they're correlated, they become amplified, right?

• 58:23Speaker 03

  Yeah, so it's definitely a more fundamental statistical problem. But I think projecting this on reality and the real setups, I have to say, like, our models are not gonna collapse tomorrow. Like, they're not gonna get worse tomorrow because we already have a checkpoint from yesterday. We can always just roll back, put more evaluations and just, you know, do it again and again, until we see that something works. Like, is it gonna be more expensive? Yeah, probably. Is this gonna, like, is it gonna make it harder for competitors that don't have a copy of the internet in the garage to train model? Yeah, likely. But at the same time. I I think it's it's like it's a thing that makes us slower, but it doesn't necessarily mean that tomorrow suddenly ML doesn't work. If if that makes sense.

• 59:06Speaker 04

  Well, one reason, I mean, maybe one intuition that might help folks for model collapse, just to take it to perhaps the simplest possible example is just uh, if you just have any distribution, let's say, like just a Gaussian distribution of integers. And you sample it a bunch of times and you add those together. And then you sample it a bunch of times and you add those together and that's your new distribution. You know, the the variance keeps getting less and less and less. This is like the sort of law of large numbers, right? It's just like the more times you keep sampling the same thing. And you know, then your average keeps getting smaller and smaller variance. And that's really what's happening. It's just, you know, it's generating content. Like I've often viewed L L M's just when I've been interacting them.

• 59:48Speaker 04

  It's almost like the answers they give you are the consensus answer, essentially. It's just if I were to somehow ask everybody and average the answer together, that's really the answer I get from the L L M's. So it's kind of giving you the average answer among all, you know, sort of coherent munging together of human, human, you know, writing, right?

• 1:00:11Speaker 03

  Yeah, I'm. Yeah, but at the same time, I have to say it's humans. Humans are like ranking those models. If you look at like a Lemmarina style things, like there is feedback from humans. Humans say, I like this model more than this one. There is clearly a lot of like, a lot of human preferences encoded in this. I don't know if if your interaction is like this, but recently my model started just placing emojis all over the place in response. This is triggering me. But I'm sure for for some folks, they they they love this.

• 1:00:42Speaker 04

  Well, that's the average, you know, that's the the bulk person wants emojis out.

• 1:00:46Speaker 03

  And this is why I'm saying like, I I think my experience with models and like improvements and quality of my personal life may not translate to others. Because like i- in my life, models made this like significantly better. But yeah, I don't I I don't know how, you know, representative this is.

• 1:01:02Tim Scarfe?

  You know, your your supervisor was Ross Anderson and and he he passed away, didn't he? So he did just like pressed in peace.

• 1:01:08Speaker 03

  I also only recently realized we organized an event for the gentleman to basically reflect back on a lot of achievements because he's been publishing and he's credited to have created a number of different fields. Like, his early work is in cryptography, obviously. One of his ciphers became a runner-up for the standard in cryptography. And he also is credited with, like, security economics literature. He's also credited with a lot of work in cybercrime. He is credited with a lot of work in Tempest. He's credited with a lot of work in banking security. And we're kind of reflecting on this. And then, only then I kind of realized that I met Ross at a very late stage of his life.

• 1:01:50Speaker 03

  In a sense, like, I saw him on this stage and then I met the students from the previous generations. And, like, their experience is widely different and they were looking at very different problems. It's, uh, yeah, he's done, he's done a lot.

• 1:02:03Tim Scarfe?

  And what would you say to people now, you know, going into ML security?

• 1:02:06Speaker 03

  I think coming through security background is better than going through a ML background. Like, if you start off as a security person who starts specializing in ML later, it's probably better. Just because you kind of learn the very fundamentals on how we solve security problems before. And that intuition allows you to think a lot more about ML models because, in essence, at least modern ML models are like interpreters. And then the language you give it, this human thing, human language, is kind of like a very high, high-level language that is not programming language, that is normally executed by other interpreters. But it's like, so, and then if you start thinking about the models as interpreters, and then the language is the programming language, then suddenly you think about the whole thing very differently.

• 1:02:49Speaker 03

  Then you ask your question, like, why do we expect the interpreter to provide security? If the programs we write are probabilistic, why do we expect to get deterministic outputs? You know, things like this, right? And then when you start thinking about them through this realm of, like, more formal computer science sort of view, I think it's more productive than trying to say, "Oh, actually, I'm going to distill all of the security thinking into the model and it's going to solve all the problems." Because clearly I can give you, like, a number of fundamental things, like, you know, classical, you know, century-old dilemmas that are definitely not going to be solved with, with models. And can they make, you know, some progress here and there? Yeah, sure.

• 1:03:31Speaker 03

  But is it actually going to solve, like, I don't know, confused deputy problem? I promise you, the answer is no.

• 1:03:36Tim Scarfe?

  Awesome. Well, Elliot, thank you so much for joining us today. It's been amazing. Thank you.

• 1:03:39Speaker 01

  Welcome to the 50th Hunger Day!

• 1:03:41Speaker 02

  Witness the beginning of the revolution.

• 1:03:44Speaker 01

  Hey Mitch, you've been trying to take down these games from the start.

• 1:03:47Speaker 02

  Filmed for IMAX.

• 1:03:48Speaker 01

  Make them remember you.

• 1:03:51Speaker 02

  The Hunger Games: Sunrise on the Reaping, November 20th, rated PG-13.

• 1:03:55Speaker 07

  Hey, listen! Power up your play with Nintendo Switch 2 and experience Link's legendary adventure, now reborn in vibrant detail on our fastest and most powerful system yet.

• 1:04:09Speaker 07

  And harness the power of the Ocarina with the built-in microphone.

• 1:04:17Speaker 07

  The Legend of Zelda: Ocarina of Time, a Nintendo Switch 2 exclusive. Game and system sold separately. Game rated everyone 10 and up.