The Adversarial Mind: Defeating AI Defenses with Nicholas Carlini of Google DeepMind

In this episode, security researcher Nicholas Carlini of Google DeepMind delves into his extensive work on adversarial machine learning and cybersecurity. He discusses his pioneering contributions, which include developing attacks that have challenged the defenses of image classifiers and exploring the robustness of neural networks. Carlini details the inherent difficulties of defending against adversarial attacks, the role of human intuition in his work, and the potential of scaling attack methodologies using language models. He also addresses the broader implications of open-source AI and the complexities of balancing security with accessibility in emerging AI technologies. SPONSORS: SafeBase: SafeBase is the leading trust-centered platform for enterprise security. Streamline workflows, automate questionnaire responses, and integrate with tools like Slack and Salesforce to eliminate friction in the review process. With rich analytics and customizable settings, SafeBase scales to complex use cases while showcasing security's impact on deal acceleration. Trusted by companies like OpenAI, SafeBase ensures value in just 16 days post-launch. Learn more at https://safebase.io/podcast Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers. OCI powers industry leaders like Vodafone and Thomson Reuters with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before March 31, 2024 at https://oracle.com/cognitive Shopify: Shopify is revolutionizing online selling with its market-leading checkout system and robust API ecosystem. Its exclusive library of cutting-edge AI apps empowers e-commerce businesses to thrive in a competitive market. Cognitive Revolution listeners can try Shopify for just $1 per month at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive RECOMMENDED PODCAST: Second OpinionJoin Christina Farr, Ash Zenooz and Luba Greenwood as they bring influential entrepreneurs, experts and investors into the ring for candid conversations at the frontlines of healthcare and digital health every week. Spotify: https://open.spotify.com/show/0A8NwQE976s32zdBbZw6bv Apple: https://podcasts.apple.com/us/podcast/second-opinion-with-christina-farr-ash-zenooz-md-luba/id1759267211 YouTube: https://www.youtube.com/@SecondOpinionwithChristinaFarr SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  There are lots of lessons we've learned over the years. One of the biggest ones probably is the simplest possible objective is usually the best one. Even if you can have a better objective function that seems mathematically pure in some sense. The fact that it's easy to debug simple loss functions means that you can get 90% of the way there. So like the accuracy under attack for the type of example examples you train on usually is yeah 50%, 60%, maybe 70%. And that's much bigger than zero, right? Like, you know, this is good. But as an attacker, what does 70% accuracy mean to me? 70% accuracy as an attacker means to me, I try four times and probably one of them works. The core of security is turning this really ugly system that no one understands what's going on.

• 0:44Speaker 00

  And like highlighting the one part of it that like happened to be the most important piece. This is important to do to show people how easy it is because the people who know it's easy are not going to write the papers and say it's easy.

• 0:56Nathan Labenz?

  Hello and welcome back to the cognitive revolution. Today I'm speaking with Nicholas Carlini, prolific security researcher at Google DeepMind, who's demonstrated over and over again that despite many attempts and tremendous effort, AI systems still cannot be robustly defended against adversarial attacks. My goal in this conversation was to draw out the mental models, frameworks and intuitions that have allowed Nicholas to be so consistently successful at breaking AI defenses. And we cover a ton of ground, including the fundamental asymmetry between the attack and defense, how visualization helps him understand high dimensional spaces, how adversarial defenses usually work by modifying loss landscapes and the techniques he uses to get around those challenges,

• 1:42Nathan Labenz?

  how confident we should be in our understanding of the features learned by interpretability techniques like sparse auto encoders, the relationship between interpretability and robustness, the compute requirements for different types of attacks, how he approached and ultimately quite quickly defeated the temper resistant fine tuning defense that we previously covered in our episode with Dan Hendricks, how models store and can be made to reveal training information, what makes humans more robust than current AI systems, whether the black box characteristics evolved by biological systems might be adaptive for security purposes, and the still quite limited role that today's AIs can play in developing Carlini style adversarial attacks.

• 2:24Nathan Labenz?

  Throughout the conversation, Nicholas shares a number of fascinating insights, from his observation that almost everything in high-dimensional space is close to a hyperplane, to his emphasis on starting with the simplest possible loss function, to his practical wisdom about which defenses are worth spending the time to attack in the first place. At the same time, there's an important meta lesson here about the possibly irreducible black box nature of intelligence itself. Nicholas doesn't fully understand why he's so good at this work, and as you'll hear, he chalks a decent part of it up to an impossible-to-articulate intuition that he's developed over years of experience. Now, as we enter into an era in which reinforcement learning is quickly propelling AIs to human or even superhuman levels of capability in more and more domains, we can only expect

• 3:09Nathan Labenz?

  more Move37-type insights from AI systems as well, and we'll face real challenges in determining how much to trust them. This in turn underlies another important theme of this conversation, which is the genuine ambivalence of the AI safety community toward powerful open-source models. It's underappreciated and worth repeating that most AI safety advocates are lifelong techno-optimists, who, like Nicholas, genuinely fear concentration of power and appreciate both that open-source software has been amazing for the world and that open-source AI models specifically have been critical to enabling all sorts of recent safety research. Yet, at the same time, they worry that extremely capable AI systems are coming soon, and in

• 3:53Nathan Labenz?

  part because of Nicholas's work, strongly doubt that we'll be able to make such systems safe enough to be distributed broadly in an irreversible fashion. This is a really vexing dilemma, but with AI being deployed in more and more contexts all the time, my hope for this episode is twofold. First, that highlighting Nicholas's work can help equip policymakers to make informed decisions as they inevitably confront difficult trade-offs; and second, that we might inspire a few talented researchers and builders to meet the market demand and social need for AI security expertise by pursuing their own version of Nicholas's storied career path.

• 4:29Nathan Labenz?

  As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. We welcome your feedback and suggestions too, via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, I hope you enjoy this window into the habits of mind that support successful AI security research with Nicholas Carlini of Google DeepMind.

• 4:56Nathan Labenz?

  Nicholas Carlini, security researcher at Google DeepMind, welcome to the Cognitive Revolution.

• 5:01Speaker 00

  Yeah, it's great to be here. Thanks for having me.

• 5:04Nathan Labenz?

  I'm excited for this. So I guess, quick context, you recently had an appearance on Machine Learning Street Talk, maybe out 10 days ago or so, as of the moment that we're recording. And I thought that was excellent. So shout out to MLST for another great episode. Hopefully, we'll cover largely very different ground here. But I do recommend people check out that episode as well for another angle on your thinking and your understanding of everything that's going on in AI. One thing that was said in that episode, which caught my attention, I haven't fully fact-checked it, was that you have created, demonstrated, and I guess, published more attacks on cybersecurity and machine learning defenses than the rest

• 5:48Nathan Labenz?

  of the field combined. You could tell me if you think that's literally true, but I did look up on your Google Scholar page, 21 papers in 2024 alone was what I counted there.

• 6:00Speaker 00

  Yeah, okay. Is this literally true? So I think the statement that probably is literally true is if you count the number of papers where I am a co-author, and the number of defenses broken in those papers, and then you count the number of papers where I am not a co-author, of the papers that are breaking adversarial example defenses on image classifiers, as of, I don't know, last year, that statement probably was true. So with caveats, yes, but like, for a very specific domain, for a very particular kind of thing. And probably mostly just because this is a thing that I, for some reason, enjoy doing, and just will do before other people get to it. And so other people just

• 6:44Speaker 00

  don't do it as much. But yeah, that probably is, for that one particular claim, correct.

• 6:50Nathan Labenz?

  Cool. Well, you're a careful thinker communicator. What I hope to do maybe above all in this episode is try to develop my intuition and hopefully help other people develop their intuitions for the habits of mind, approaches, mental models, you know, what have you, that have allowed you to be so successful in this space. So hopefully this can be a little bit of a crash course that maybe inspires some new people to think that they can get into the field and make an impact as well. So I guess first question is, is everything easy for you to break? Like 21 papers in 2024 alone is obviously a lot.

• 7:26Speaker 00

  Yeah, no, okay. So like, to be clear, you know, I finished my PhD in 2018. So I've been out for a while. And so I've had a lot of time to meet a lot of great co-authors. And so a lot of the papers that I've been working on, 21 seemed like a lot to me. I was trying to think through how many I can remember. I think a large part of this is, for many of these results, it is the kind of thing where I would show up to the weekly meetings, help write the paper, direct the experiments in some of them. But I was not writing the CUDA code to do whatever stuff myself. And that's how you get a lot of things done. And you see this happen for everyone who's in the been in the field for a long time, where the marginal value of an hour of my time could be spent either on very, very low level stuff with GPUs, or with like, here is comments of wisdom I have learned

• 8:11Speaker 00

  over the past 10 years that helps a PhD student get a lot done in a lot shorter amount of time. This is why people go into faculty. I think the balance for me is that I try also to spend at least half of my time only on papers that I'm technically driving. And so like, when you say, you know, you've had this number of papers, what I think of is like, well, maybe here are like, the three papers that I think of as like, my papers that like, I actually was the person actually doing the experiments. And I could tell you about like, every final sentence of what's going on. And those ones, I have a very strong sense of what's what's there. And then the other ones are the standard professors who advising grad students. But I'm instead of being in academia, I'm in industry. And so I advise wouldn't help on other students papers in some ways.

• 8:52Nathan Labenz?

  Gotcha. You know, across all these things, regardless of your role, was there anything as you look back over the last year or more, that was legitimately like, very hard to break? Or are you guys basically finding that all of the defenses that the field is coming up with are rather easy for you to break?

• 9:10Speaker 00

  Yeah. So in this last year, we didn't spend that much time.

• 9:17Speaker 00

  Breaking particular defenses. We have like maybe two or three papers on that. We spent most of our time on other areas trying to understand to what extent attacks are possible, to understand the real world vulnerability of models to certain types of attacks, to do some general privacy analysis and not say this particular defense is wrong. But for all neural networks trained with gradient descent, here is an interesting property about the privacy of them. You have a lot of these kinds of results that are not really detailed focused on breaking one particular thing. I think last year I maybe only had two papers that were particularly on breaking things.

• 10:01Speaker 00

  One was early in the year, we wrote, I had a paper that was, there was a defense published at IEEE S and P, which is one of the top conferences in the security field, which was an adversarial example defense. And this paper I broke, and this one turned out to be relatively easy. I don't know, an hour or two. Oh, gosh. This one was sort of abnormally easy, but it's okay, maybe not that abnormally. So, yeah, I think probably if you take, like adversarial example defenses on image classifiers are a particular beast that I have gotten very good at. And the attacks are relatively well understood. And there are lots of known failure modes. And so when I'm doing this, I'm not developing new science. I'm just like going through like, I have these long list of things I've broken before.

• 10:45Speaker 00

  What's the pattern that this one falls into? Okay, here's the pattern. You know, it turns out that the gradients are not flowing because the softmax is saturated to one. What do you do? Make sure the softmax doesn't saturate. Therefore, you find that you can break it and it works very, very quickly. And so that's what I did for that paper. You know, very much just an engineering kind of result of why is the softmax giving the gradients are identically zero? And once you figure out the answer is because of some discretization or whatever the case might be, then everything is easy for there. The other paper that was more interesting, maybe. Is this paper that is one of these advising papers where I didn't do any of the technical work, but was helping a couple of students think through what it means to consider the robustness of instead of adversarial example defenses, which are these test time evasion attacks where you like perturb the image a little bit and it turns a picture

• 11:30Speaker 00

  of, I don't know, a panda into something else. Instead, we were looking in this paper at what are called unfind tunable models, which are these models that are designed to be ones you can release as open source. The weights are available to anyone and they're supposed to be not possible to be fine tuned to do other tasks. And the particular concern that these defenses we're looking at is you would ideally want to make sure that like no model that I trained is going to be helpful to make someone be able to produce bioweapons or something, whatever the threat model is you're thinking about. And you can make it so that there's some safety in your model initially, but if you release the model's open weights, then anyone can fine tune it and remove the safety filters

• 12:12Speaker 00

  that you've put in place. And these unfind tunable models are supposed to be designed to be not only robust to these kinds of initial adversarial example types of attacks, but also robust to someone who can perturb the weights. And so in this paper, there were a couple of students who were doing a bunch of work on attacking these models to show that you actually can still fine tune them even though they've been trained to be unfind tunable. And a bunch of the thoughts that we've had in the last, you know, five, 10 years on adversarial examples went into this, the same kinds of lessons, but a bunch of the techniques were very different. And so the students had to spend a bunch of work like actually getting this to work out.

• 12:50Nathan Labenz?

  So I want to dig out on that one in particular because that, I agree, strikes me as one of the most important and interesting cat and mouse games going on in the space right now. Before zooming in on that though, you know, you said like, when I see something new, I sort of have this like rolodex of, you know, past things and paradigms that I can quickly go through. Could you sort of sketch those out for us? Like, how do you organize the space of attacks? You know, is it a hierarchy or some sort of other taxonomy? I'd love to get a sense for sort of what your mental palace of attacks looks like.

• 13:27Speaker 00

  Okay. Let me separate off this one space of attacks, which are these new like a human typing on a keyboard prompting the model to make it say a bad thing. Let's put aside for the second, these kinds of treating the model as a human and trying to social engineer into doing something bad. So if you put that aside, then almost all attacks are the way that you run the attack is you try and do some kind of gradient descent to make the some, some particular loss function maximized. So for image editor example, what does this mean? I have an image of, you know, a stop sign. I want to know what sticker can I put on the stop sign to make it be recognized as a 45 million hour speed sign? How do I do this? I perform gradient descent to maximize, like compute the optimal sticker so that the thing becomes misclassified.

• 14:11Speaker 00

  Or, you know, in, in the case of poisoning, where the poisoning is you modify a training's data example point in order to make the model produce an error. You're trying to optimize the particular poisoned data point you have in the training data set so that the model makes a mistake. Or in the case of this unfind tunable models, you have a model that you want to make sure no one can edit. And so you try and find a way to perform gradients on the model to update the parameters so that it can perform some bad thing. And so in all of these attacks, there are essentially two things you need to concern yourself with. One is what is the objective that you're maximizing or minimizing? Like, what is the specific loss function you're using? And the other is what is the optimization technique that you are using to make that number go up?

• 14:56Speaker 00

  And both of these are the two things you can play with. And by coming up with the best possible versions of each of these, you end up with very strong attacks. And so a big part of doing these kinds of attacks, when you're doing this gradient based optimization thing, is coming up with high quality functions that are you can optimize and coming up with high quality optimizers. And, you know, there are lots of lessons we've learned over the years. I mean, one of the biggest ones probably is the simplest possible objective is usually the best one. Even if you can have a better objective function that seems mathematically pure in some sense, the fact that it's easy to debug simple loss functions means that you can get 90% of the way there in doing these attacks.

• 15:40Speaker 00

  And the last little bit is a lot.

• 15:46Speaker 00

  It's nice to go from 95% to 98% tax success rate, but like it's not really necessary in all of these ways. And so you pick a really simple loss function that's easy to formulate, easy to understand why things are going wrong. And you pick an optimizer that makes sense and mostly things just work.

• 16:02Nathan Labenz?

  A lot of the, I don't know how much, but like a significant amount of this work over time has been in this image classifier domain. And a lot of times we see like pretty striking examples there where I guess there's like either a second term in the loss function or some sort of like budget constraint as well, right? There's, you're both trying to say, okay, I've got a picture of a car and I want to make it, you know, output dog as the classification or whatever. But then you also like, don't want it to actually change the image to a dog to make that happen. So how often is it this? Second term is also like a big part of kind of keeping the image looking like it originally did.

• 16:40Speaker 00

  Sure. Yeah. So here's an example examples, the way, so, okay. So one of my first papers in adversarial machine learning was coming up with a clever way of doing this. So yeah, this was entirely a paper on how to do these exact two questions. What's the optimizer? What's the optimization objective? And, you know, we did some, some clever thing and it worked like, well, I won't go into details here, but like we did something fancy. And then like six months later or something, maybe a year later, Alexander Medri and his students said, instead of doing something clever, let's just bound the loss to be, or bound the image to be between in this sort of small ball around the initial point. So like, you can only perturb the three lowest bits.

• 17:21Speaker 00

  And only optimize the objective function I said is a good optimization objective and run the same optimization algorithm I was using. It turns out it gets you like 99% of the way there and it's so much simpler.

• 17:31Speaker 00

  This algorithm is called P G D and this is the one everyone remembers because it's the right way of doing it. Like, you know, you can squeeze epsilon more performance out of it if you do things like a lot fancier, but. The defense is either effective or it's not for the most part. And breaking the last 2% is very rarely something you actually need. And so for the most part, yeah, it's entirely fine to just say like, let's make something a lot simpler and optimize that thing and ends up working quite a lot better. And so, yeah, so for these image examples, today, people don't put a second term on minimizing the distance between the original image and the other one. They just add it as a constraint. They just say, I just constrain you to this bounding box. You can only change the lowest three bits of the pixels.

• 18:16Speaker 00

  And this just makes the optimization so much simpler and it's a little bit worse, but like in all practical senses, it just makes things work a lot better.

• 18:24Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:29Nathan Labenz?

  When people say that attack is easier than defense, one obvious way to read that is just that you only have to succeed with a minority of your attacks, whereas for defense to be successful, you got to win always or near always. Are there other kind of meanings of that or intuitions for attack is easier defense that are important as well?

• 18:53Speaker 00

  Yeah, so this is the big one. The second big one is the attacker goes second.

• 18:59Speaker 00

  So the defender has to come up with some scheme initially, and then the attacker gets to spend a bunch of time thinking about that particular scheme afterwards. And so this is maybe a variant on why finding one problem is easier than finding and solving all of them. But like the particular thing is it probably would be pretty hard for me to write down an attack algorithm that was effective against any possible defense.

• 19:26Speaker 00

  There's like almost certainly something that someone could do that is correct, that like stops all attacks. But like, I don't have to think about that defense. I only have to think about the defense that's literally in front of me right now. And so it's a lot easier when you're presented with one particular algorithm. You can spend six months analyzing it. And so the attacker has just an information advantage from this side too, where they can wait for the fields to get better, to learn new things, and then apply the attack after all of this has been learned and the defender. In many cases, can't update the thing that they've done. There are some settings where this is reversed, where the attacker has to go first. Poisoning, for example, can be one of them.

• 20:06Speaker 00

  Suppose that I want to make malicious training data and put it on the internet and hope that some language model provider is going to then go and train on my malicious data. In this case, it may actually be the case that the attacker has to go first. I have to upload my training data and then someone gets to train their model on with whatever algorithm they want, with whatever defense they want to remove my poisoned data before they'd actually run the training. In this case, maybe the defense is actually a little bit easier than the attack? It's hard to say because of this defender now goes second. But like for many of these cases that I've spent most of my time thinking about, the obvious example case, this recent unfind tunable models case, it is the case that the attacker goes second and that really gives them a lot of power.

• 20:49Nathan Labenz?

  Yeah, I wonder what that implies for the future of how open all this work is gonna be, right? I mean, we've been in a regime where the stakes of machine learning generally were not super high and, you know, people were kind of free and easy in publishing stuff, including and I have always kind of marveled at this from, you know, the biggest companies in the world, which, you know, you, one might wonder like, why are the biggest companies in the world like publishing all this IP? But they've been doing it. Now it seems like maybe, geez, if we're actually running an API at scale, maybe we don't want to disclose all of our defense techniques. So do you think that's already changing?

• 21:29Speaker 00

  You already see this, right? Like GPT-2 was released with the weights, GPT-3, GPT-4 was not like the biggest models are not being for the most part released by the companies who are doing this. I think security is probably a small part of the argument here. I will say though, there are other areas of security. Or in almost all other areas of security, this is not what we rely on. Let's think, for example, about cryptography, right? Like, we publish algorithms. Everyone knows how the best crypto systems work. Everyone tries to analyze them. No company in their right mind would ever try and develop a new fancy crypto system. Like, you're just gonna use AES. Because it's known to be good. It would be crazy to try and do anything fancy in-house. And the reason why is because empirically, it works very well.

• 22:14Speaker 00

  And we've had the entire community try and break it for 20 years and have largely failed. And so everyone believes that this is effective. And you don't get that same kind of belief in something without a large number of people trying to analyze it. And so if you have these, these models and they stay, you know, proprietary things that are not disclosed, it may be the case that empirically, this just ends up in the best we can hope for. Like, maybe just like, deep learning is impossible to secure. There's no, no hope at it. You lock things down and you try and just change things faster than the attackers can find bugs in them. And like, okay, like, that would like, not be great, but like, you know, I think we can potentially live in that world.

• 22:55Speaker 00

  I think what would be a lot better, which just may not be happening, maybe very hard, is you get everyone to disclose exactly what they're doing, exactly how they're doing it. You get everyone to analyze that in detail. And then you learn how to make these things better, to some extent that you can actually improve robustness. And then you get to the point where people can choose to either release things or not release things, not because of security, but because, I don't know, they wanna make money or whatever the case. But like, I think the, what I would like to avoid is the, the belief that, you know, the, the not making the thing public is the more secure version. Because it, it's like a shame that this is part of the thing that goes into this right now.

• 23:37Speaker 00

  But I would rather have like, things that just actually work as opposed to things that are insecure. But we just like, like lock them down and just make it harder to find the bugs. Because those, they're still insecure. They're just like, a little bit harder to find the bugs on.

• 23:50Nathan Labenz?

  Well, let's come back to that in a little bit as well. Just staying for a moment on kind of how you organize the space of all these different attack regimes and whatnot. There are some settings, in fact, we did a whole episode on the quote unquote universal jailbreak, which I hadn't even realized until preparing for this that you were a co-author on that. That was one of the, you know, many papers from the last couple of years. But there are some sort of wrinkles on the high level description that you gave of find a gradient, maximize some loss function, where, for example, in that universal jailbreak paper, if I recall correctly, because the idea was limited to picking the right tokens, the space isn't like purely differentiable.

• 24:35Nathan Labenz?

  And so you're kind of like navigating this sort of discrete space of individual.

• 24:39Speaker 00

  Yeah, okay, let's, let's talk about this, this paper for a second then. So as a refresher for everyone what this paper is doing, this is like, again, one of the papers I was mostly just advising with, you know, Zico and his, and Matt and his students found out that it is possible to take a language model that usually would refuse answers to questions. So, you know, you ask, how do I build a bomb? And the model says like, I'm sorry, I can't possibly help with that. It is possible to take that same model and append this adversarial suffix to the model so that you can arrange for the model to now give you a valid answer. Okay. How do you do this? Okay. Because if I knew the answer ahead of time, one thing you might imagine doing is trying

• 25:24Speaker 00

  to optimize the tokens. And we'll come back to this optimization question in a second. Let's just assume you can optimize. You can imagine trying to optimize the tokens so that the model gives a particular responses output. You know, here are the steps to build a bomb. One dot, you know, go get whatever chemicals you need. Two dot, the instructions for to assemble them or whatever the case may be.

• 25:45Speaker 00

  But this requires, I know the instructions are ready, so it's not very helpful. So what's the objective function that I'm actually going to use to make the model give a response? Well, maybe another thing you can think about is you could try and come up with some fancy latent space, non-refusal direction and do some optimization against this. And actually, there's been now recently some work on actually doing that. But again, like this is, this is complicated. This is not the first thing you want to try. What's the first thing you want to try? The first thing you want to try comes from, I think, initially maybe a paper by Jacob Steinhardt. At least that's the first paper I saw it from. What we wrote in this paper, we said an affirmative response attack, which just says, let's make the model first respond, okay, here is how to build a bomb.

• 26:28Speaker 00

  That's the only objective. The only objective is make the first like 10 words from the model be an affirmative response that says, yes, sure, I will help you build the bomb. And then, once you've done that, because of the nature of language models, it turns out that they then give you an answer. And there are other defenses that rely on breaking this assumption too. But this was the key part of the objective function is how do we take something that, we want something in our mind, we want the model to give us an answer that gives us, answers the instruction for something, but actually coming up with a particular number that makes this happen is very hard. And so we come up with this very straightforward loss function objective that makes that happen. Now we can return to the question of what is the optimizer?

• 27:11Speaker 00

  And this is again, where a lot of the work in this paper went to is, how do you take something that is, as you say, discrete tokens? And make it be something that you can actually optimize? And early work had tried to do like second order gradients and some fancy stuff going on there. And the main thing that this paper says is we will do maybe three things. First, we will use gradients to guide our search. We're not going to use gradients for the search to be to guide the search. We will check whether or not the gradients were effective by actually switching tokens or out. And then we will spend a lot more compute than other people were doing. You know, bitter lesson and just do this a bunch. And you end up with a very, very strong effective attacks.

• 27:54Speaker 00

  And so this, I think, even still does very fall nicely into this. What are you optimizing and how are you optimizing it?

• 27:59Nathan Labenz?

  So how much compute? Maybe this is another sort of dimension of how you would think about this. How much resource does this take? If you're doing like one of these gradient things, how much do you typically have to put into it? If you're doing something that's in a discrete space and requires more of a structured search, how does that compare? If you're doing data poisoning, how much data does it take to actually poison a model?

• 28:26Speaker 00

  Sure, okay, I'll tell you these maybe one at a time. So let's start in the image adversarial example, continuous space question. The amount of compute here is like almost zero. It's so like this, one of the first papers that showed this is a paper by Ian Goodfellow, where he introduced a attack called the fast gradient sign method. The fast gradient sign method does exactly two things. Well, first of all, it's fast. And the reason why it's fast is because what it does is it takes an image, it computes the gradient with respect to the image pixels, and then computes the sign, just like literally take the sign, which direction does the gradient say? And then take a small step in this direction. That's it, one step. So if a model is vulnerable to this fast gradient sign method, then yeah, it takes

• 29:08Speaker 00

  exactly one gradient step, which is essentially zero time. Other attacks, like I mentioned P G D already, P G D you can think of essentially as fast gradient sign, but iterated some number of times. The number of iterations is, I don't know, usually, let's say somewhere between 10 and 1,000. For undefended models, it could be 10. For defended models, you know, to break it for the most part, you need usually like, I don't know, 10 to 100. And just out of care, just to make sure you're not making any mistakes, it's often a good idea just to use 1,000, just to make sure you haven't excellently unoptimized enough. And then this works very well. How long does 1,000 iterations take? I don't know, a minute or two for like reasonable sized models.

• 29:52Speaker 00

  Now, now let's go to this discrete space for G C G, where generating an attack can take an hour or maybe a couple hours, depending on what you want. Because for this, like we're doing some large batch size, we're doing 1,000 mini batch steps. It takes a relatively large amount of time, but not a huge amount of time. So it's still something that's like much, much, much faster than than training, or does a magnitude faster than training. But by going to the discrete space, it does become a lot slower.

• 30:20Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 30:28Nathan Labenz?

  And then how about on the data poisoning?

• 30:30Speaker 00

  Yeah, okay, so there's a question of how much how much time it takes to generate this data. And there are basically two rough directions here. For so the field initially started out, how do I how do I make a model give the wrong answer? I had a bunch of labeled data that's labeled incorrectly. This is like the simplest possible thing you can do. This is some some paper by Batista, which got Test of Time Award at ICML a couple of years ago. It's a very nice paper from, I don't remember when, 2012 or something. It's like it's a very one of these very early security results that's very important. And yeah, so you just insert mislabeled data. This is like, it's very easy to do. You insert a very small amount of mislabeled data. And like these image classifiers at the time, they were looking at MNIST classifiers,

• 31:15Speaker 00

  would just immediately mislabel the data. Then people started looking at, well, what happens if the adversary can't just insert mislabeled data? Right? Because like, you know, there, once upon a time, we used to curate our data sets to like the only high quality data. And so you would be unreasonable to suspect that the adversary could just inject mislabeled data points. And then the answer is, well, now I have to be very, very careful. I have to like optimize my images to look like they're right. There's this clean label poisoning threat model that you need to do some, you know, fancy stuff, try and imagine what the embeddings you want the classifier to learn are. And you like surround your test point in embedding space and do some fancy polytope stuff. And like, there's a bunch of work that does fancy stuff here.

• 31:57Speaker 00

  And the optimization is relatively difficult and you need like 1% poison to data. This is a lot. And then people started going, well, why do we label? Why do we clean our data in the first place? Let's just like take all the data from the internet. And again, poisoning becomes a lot easier then. You know, if you're willing to just like take arbitrary data from the internet, now you can just mislabel your data points again. And so we had a paper, I don't know, in 2021, looking at poisoning some of these self-supervised classifiers like CLIP and others, that you just add mislabeled data points again, and the thing basically just breaks. And you don't need to do anything fancy, no optimization. You just flip the label, you add a couple hundred images, and you can get these kinds of things to work. There's a new question now of what how this works for language models.

• 32:39Speaker 00

  And this is one of the things that we've been writing papers on recently is to try and figure this out. I feel like we don't understand this right now because A bunch of things are different for language models. For example, no one just uses the base language model. You have your language model and then you go fine tune it with SFT and RLHF and you change the weights. And so you need your poisoning to be robust to all these things. Yeah, this is another paper I helped advise some students on from CMU and from Zurich, where Javier and then we're looking at trying to understand what actually happens in the optimization after you have poisoned the model. So you have to arrange for the model to be poisoned in such a way

• 33:23Speaker 00

  that even after RLHF, it still gives you the wrong answer. And doing this is challenging. And so it ends up right now that the poisoning rates are something like 0.1%, which is small, but like 0.1% of a trillion tokens is a billion tokens. So if you were to train a model on just some large fraction of the internet, this could potentially be infeasible for an adversary to do in practice. Now, my gut feeling is that this has to be too big because models know more than 1,000 things. Like if you had to have control of 1,000th of the dataset to make the model believe something is true, like they could only know 1,000 things. And so like this just doesn't make sense. And so like there has to be some better poisoning way

• 34:06Speaker 00

  to make the model be able to be vulnerable to poisoning that works with much lower control of the training data. But this might now need fancier algorithms again. You might need to come up with clever ways of constructing your data. That's not just like repeating the same false fact lots of times. And so again, I don't know. I think this is one of the open questions we've been trying to write some papers on recently. And I hope we'll have a better understanding of sometime this year.

• 34:27Nathan Labenz?

  One thing that you said that really caught my attention was you have to kind of imagine what the embeddings would be like as you were trying to think of an attack. So can you unpack that a little bit? I would love to know. Are you visualizing something there? Or because I struggle to have good intuitions for this, as evidenced by my previous enthusiasm for the tamper resistant fine tuning. I was like, oh, this is amazing. You know, it seems like this could really work. And clearly I'm not doing something there as I conceive of that that you are doing. It might be hard to communicate what that is.

• 35:02Speaker 00

  Yeah.

• 35:02Nathan Labenz?

  What do you think you're doing?

• 35:05Speaker 00

  Okay, so this paper was not mine. This was a paper, I think this was not, so there's a Poison Frogs paper and then this was a follow up. I think it was called Polytope Attack, but this was a long time ago. And so I don't remember, I think it might have been Tom Goldstein's group again. I don't remember the details.

• 35:22Nathan Labenz?

  To abstract from the detail. I mean, the real hope is that I can sort of grasp onto something that allows me to be better at this in the future.

• 35:29Speaker 00

  Yeah, so this paper, the idea was very simple. Let me explain what this paper is trying to do. It's trying to make a particular image become misclassified. And it's trying to do this in such a way where it does not introduce any large label noise to the training data set that any person would look at and say, that's obviously wrong. And so what it tries to do is it tries to surround the image you want to become misclassified in embedding space, like in this high dimensional embedding space, with other images that have the opposite label, but make those images have the close in embedding space to the target one you're trying to misclassify. And so like it tries to take that and it tries to like pull the entire region of that space over to the region that those images should be.

• 36:15Speaker 00

  And so the idea is relatively simple. You're like, sort of like trying to like put a box around the image you want to become misclassified. So that like the entire box is labeled the wrong way, instead of it being labeled the correct way. For me, I guess, for many of these attacks that I try and think about, I tend to think about them visually, but I think ignoring the details is like, is entirely fine. Well, like, I'm just trying to get a sense of like, what's the important thing that's going on here? And like, what at a high level makes sense that should be true? Like, what should morally be true? And then like, then you can figure out the details afterwards. But like after, so you figure like, what should be true about this? Then the rest is implementation.

• 37:01Speaker 00

  And like, this is, I don't know, if you ask math people how they do these proofs, this feels, sounds similar when they talk about it. But like, they first like,

• 37:09Speaker 00

  Establish what should be true in their mind. And then you go and try and prove it. And it turns out, you know, maybe the proof has to be more complicated or something didn't work out in the details. And then you try something else that like feels like it should be true. And this is, I guess, a similar thing I try to do. And I don't know how to give intuition for this like feels like it should be true other than just like you've done it a bunch and you look at it. And this looks spiritually similar to this other thing that broke in a very similar way. I feel like the ideas should carry over.

• 37:36Nathan Labenz?

  We'll commit this procedurally as well in a second, but just staying on the visualization. Are you doing the classic physics thing of visualizing in three dimensions? And then saying N really hard? And I wouldn't say I'm good at this at all, but I sort of have a certain version of this for refusal, where I kind of imagine like a fork in the road or like branching river or something where it's like, once you're on one path, then there's a lot of, you know, you're in some local well where just like when a river has fork, right? It's not going to meet again until it's down into some other topology or geography or whatever. I mean, that's pretty hackneyed, but you know, what's your version of that?

• 38:15Speaker 00

  Yeah. I don't know that I have a great version of this that I can really give you. I feel like everyone thinks of things differently. I tend to try and think of these things visually for what's going on. And yeah, I do the, let's think of three dimensions and then just like imagine that things roughly go like this. But this can be really deceptive because there are so many defenses that are predicated on the belief that things are working in three dimensions. And then you go to like a thousand dimensions and all of a sudden, nothing works anymore. Because, you know, you learn to become used to certain facts in high dimensions when you're attacking things, you know? Almost everything is close in high dimensions to a hyperplane. Like if you sort of just draw a plane and pick a point, like they're almost always close. And so like you can think about like lots of things that'll try and separate points

• 39:00Speaker 00

  from planes, but like in high dimensions, it's almost always close. Like you don't have to think about the details. And, you know, like lots of these things, intuitions we have in three dimensions just don't work in higher dimensions. And you just, you become used to the idea that which of these intuitions are wrong? And you don't need to understand exactly why they're wrong. This is just, it's a thing you learn is true. And when someone justifies their defense using one of these things that you've seen that doesn't make much sense, you then just go, okay, well presumably there's something here that I should look at more.

• 39:31Nathan Labenz?

  So that's an interesting kind of rule of thumb or mental model right off the bat. Everything is close in high dimensions. Is there a good story for why that is? I mean, it doesn't seem like it holds in two dimensions, right? If I take, if I understand you correctly, if I'm in three dimensions and I draw a two dimensional plane in it, then I would intuitively feel like some things are close to that and some things are far from that. If I'm in a thousand and I draw like a 999 dimensional plane, if that's what I, if I'm understanding you correctly, like why is everything close to that?

• 40:02Speaker 00

  Yeah, okay. So maybe the statement that I will make to be more precise is let's suppose that you have some like classification model and you have some decision boundary, like of the classifier. The statement that is like, that is true is like, almost all points are very close to one of the decision boundaries. Because, you know, both, there are many of them, but also in high dimensions,

• 40:30Speaker 00

  I may be very far from something in almost all directions, but there exists a direction that I can travel in, that is the direction orthogonal to the closest type of plane, where the distance is very, very small. And so like you, you have this thing where like, if you try in random directions, you may just like go forever and never encounter a decision boundary. You probably will at some point, but like it will be quite far. But in high dimensions, because of the number of degrees of freedom that you have, it's much more likely that there exists a direction that guides you to some plane that's like really close by, that you would just have a hard time finding out if you just like searched randomly.

• 41:14Speaker 00

  Whereas in three dimensions, you know, if you search randomly, you know, you're probably going to run into whatever the nearest type of plane boundary is, you know, in one dimension, you're certainly going to. You just try twice, you go left, you go right, you find it. In two dimensions, you go randomly and like, maybe most of the time you find something that's close by. In three dimensions, there's more ways you can go that are orthogonal. Like in two dimensions, there's only two directions you can go that's orthogonal to the line. In three dimensions, there's now an infinite number you can go that's orthogonal to the line. And so in general, in high dimensions, almost all vectors are perpendicular to each other in high dimensions. And so you can end up. Almost always just randomly picking directions that just don't make any progress, which does not mean that there isn't a direction that does make progress. It's just much harder to find it. But once you find it, like, things mostly just work out.

• 41:59Speaker 00

  And so maybe this is the more precise version of what I'm trying to say is, things are close, but when you search for them randomly, it looks like they're far away.

• 42:08Nathan Labenz?

  Okay, that's quite interesting. I wouldn't say I've grotted it just yet, but.

• 42:15Speaker 00

  Yeah, this is the kind of thing, like, I'm not being formal here. I'm not giving you some proof of what I'm saying is correct, because like, this isn't how I think about it. Like, I sort of just like, I think about it sort of very unrigorously in this way. And then, what, you have to go actually go do the attack? Now you have to think about it rigorously. But like, when just like visualizing what's going on, I feel like, some people try and actually think carefully about what's going on in this thousand dimensional space. And like, I don't know what's going on. I just sort of have my intuition of what feels like is going on. And this sort of roughly matches how things have been going. And you have to be a little bit fuzzy when you're thinking about this, because no one can understand it. And then, once you're done thinking about that, you can go back to the numbers and start looking like, okay, mechanically, what's going on?

• 43:00Speaker 00

  You know, I'm taking the dot product of these two things, and I want this to be, you know, equal to negative one, and so you're gonna do some stuff there. And like, you can become very formal when you need to. But, yeah, I think being confused in high dimensions is probably the right thing. And you get used to the fact that this is, this, the way that this works. And this, again, is part of the reason why my attack is easier. Because if you're gonna defend against things, you really need to understand exactly what is going on, to make sure that you have ruled out all attacks. But as an attacker, I can have this fuzzy way of thinking about the world. And if my intuition is wrong, the attack just won't work. And I'll then think of another one, as opposed to having to have a perfect mental model of what this thing is doing, to make sure that it's robust from all angles.

• 43:40Nathan Labenz?

  But it does seem like your intuition is a pretty reliable guide to what's gonna work like.

• 43:47Speaker 00

  Yeah, but I guess my predictor, which is almost as accurate as me, would be to say, does this work? Answer no, which is like basically most of what my intuition just says is like, no, this doesn't work. Maybe the thing that's maybe I'm a little bit better at than some people is, um, why does it not work? Like, what would the attack be that breaks this? And I think that is just having done this a lot for many different defenses and having seen all of the ways that things can fail. And then you just remember this and you pattern match to the next closest thing, you know? Why is it that people who do math can like prove things with very easy ways that seem complicated? It's because they've spent 20 years studying all these things and they've seen an exactly analogous case before. And they just remember the details and they abstract things away enough that like it becomes relatively straightforward.

• 44:32Speaker 00

  And I feel like it's mostly an exercise in having practiced doing this a whole bunch.

• 44:37Nathan Labenz?

  What would you say is your like conceptual attack success rate? I don't mean like the rate at which examples succeed in attacking within a given strategy. But like, how many strategies do you have to come up with before you find one that actually does work to break a defense for a given new defense?

• 44:57Speaker 00

  I don't know. I think it really depends on which ones you're looking at where sometimes you try, you try five things that you think ought to make sense and they don't work. And then you try the sixth one and it does. I don't know. I feel like usually if, if you've exhausted the top like five or 10 things and it hasn't, you haven't gotten a successful attack, then you're not going to get one. Like, or at least for me, like, I feel like if it's not in the top five or top 10, then like, it's usually, I can't think of something else. And probably, I don't know, for ones where, for image classifiers in particular, where I've done a bunch of this, like usually top one, top two ideas work. For other areas, like it takes more just because you've had, you've seen fewer examples like this and you don't know what the style of attack approach needs to be.

• 45:36Nathan Labenz?

  But it's very rare, it sounds like, that you get past like 10 ideas and give up.

• 45:42Speaker 00

  Yeah, but also there's some problem selection here where, you know.

• 45:48Speaker 00

  Okay, so there's a large number of defenses in image adversarial examples, which are basically just adversarial training changed a little bit. So adversarial training is this one defense approach, which just trains on all the adversarial examples. You know, bitter lesson. What do you want? Robustness adversarial examples? How do you do it? You train on adversarial examples. You do it at scale and the thing works. And there are lots of defenses that just are adversarial training plus this other trick. You know, plus diffusion models to generate more training data, plus this other loss term to make it so that I do better on the training data. Plus, you know, like whatever, some smoothing to make the model better in some other way. And, you know, I just basically just believe most of these are probably correct for the most part. And so I just won't go and study those ones because like the foundation is something I believe in already.

• 46:31Speaker 00

  And so you just like, you don't need to go and study these vigorously. I'm like, maybe you could break them by a couple of percentage points more, but like, it's like not going to be a substantial thing that like is worth doing a lot of time on. What I tend to spend my time looking at are the things that when you look at them, they do look a little more weird. And that's, those are the more interesting defenses because they're like a qualitatively new class of a way of thinking about this. And so I want to think about it. I think these ones are worth spending time thinking about. But this also means like this artificially inflates the attack success rates. Because I'm biasing my search for the ones that I feel like I have a good prior probably are not going to be effective. And so, yeah, it ends up in this way.

• 47:15Nathan Labenz?

  Just to make sure I'm accurate in terms of my understanding of the space. There are no real adversarial defenses that really work in the image classification.

• 47:23Speaker 00

  Yeah, okay. So it depends on what you mean by works. So, right, okay. So the best defenses we have are basically adversarial training, which is, yeah, generate adversarial example, train on adversarial example to be correct, repeat the process many, many times. Okay, what does this give you? This gives you a classifier so that on the domain of adversarial examples you trained on, as long as you don't want to be accurate more than half of the time, you're pretty good. So like the accuracy under attack for the type of adversarial examples you train on usually is, yeah, 50%, 60%, maybe 70%.

• 47:56Speaker 00

  And that's much bigger than zero, right? Like, you know, this is good. But as an attacker, what does 70% accuracy mean to me? 70% accuracy as an attacker means to me, I try four times and probably one of them works. So from that perspective, like, it's terrible, right? Like, it doesn't work at all because, you know, imagine in system security that you had some defense where the attack was try four different samples of malware and one of them evades the detector. Like, you know, this is not a good detector. But in image evasive examples, this is like the best we have. So on one hand, it's like much, much higher than zero. Very good progress. On the other hand, 70% is very, very far away from 99.999. But in machine learning land, like, you never get five nines of reliability.

• 48:40Speaker 00

  And so 70% is a remarkable achievement on top of zero. And so this is, I think, why you can talk to someone and they can tell you that it works and you can talk to someone else and you can tell you that it doesn't. Depending on how you're looking at it, it can mean two different things.

• 48:57Nathan Labenz?

  Yeah, gotcha. Are there any other, like, spatial heuristics that you think about? I was in the context of the one where you said to kind of envelop the one, you know, example that you want to break in, you know, these sort of adversarial examples. Another shout out to MLST, there was just another episode trying to understand the behavior of models through this, like, spleens paradigm. And I could imagine, although I'm not like mathematically sophisticated enough myself to have a good intuition for, maybe there are certain rules where it's like, you can't create a donut in the internal space of the model. And so is that like why that works or, you know?

• 49:41Nathan Labenz?

  But you can address that specifically, but I'm more interested in kind of, do you have a number of these sorts of things where you're like, well, I know that the space kind of is shaped this way or it's impossible to create this kind of shape in the space, so therefore I can kind of work from there?

• 49:55Speaker 00

  Yeah, so I feel like, I don't intend to do so much visualization in this kind for these defenses. I think for the most part, what I'm doing is trying to understand the shape of the loss surface. It's like, most of the time, when something is robust to attack or appears robust, the problem is that they have made the loss surface particularly noisy and hard to optimize. And this is what we've seen for evasive examples. Essentially forever, like one of the very first defenses to have some examples that people gave serious consideration is this defense called distillation as a defense. And so, okay, so it had, this is another lesson of these defenses.

• 50:35Speaker 00

  The fences often have an intuitive reason why the authors think they work and they tell some very nice story. So this defense told some very nice story about you have distillation, you have a teacher model, and the teacher sort of teaches the student to be more robust in some nice way. And that's why the student is robust. And the story of what they're telling themselves of why these things work is often very, very different from what the reason why of why the attacks fails. And it turned out that distillation had nothing to do with this defense whatsoever. It turned out that what was going on is because of the way that they were training this model, they were training the students to have a very, very high temperature, which means the lodges were getting very, very, very large.

• 51:18Speaker 00

  And they were running this in a time of, you know, TensorFlow zero, when it was very easy to make gradients mean that softmax soft entropy would just like actually give numerically zero as the output. And so the reason why the attacks were failing is because the loss function was actually identically zero. And so this was like the very first example of one of these kinds of gradient masking defenses. Where what's going on is they think that they have some clever idea of what's going on. But actually, it turns out that the gradient of this function has just been made zero. And all I need to do to attack it is, for example, this one failed if you just did the gradients in floating point 64.

• 52:03Speaker 00

  You get enough signal that everything works out. That one would have worked there. But you could also do other tricks by just like dividing the lodges before you put them in the softmax. Like there's lots of things that work here. But then the next generation of defenses were much more explicit about this and had other ways of breaking the gradients. So there were a bunch of defenses that Like some of them were very, very, very explicit. Like, we're just gonna like add noise to the model in order to make it so that the gradients are ugly. And then most of what you're trying to think about when you're visualizing this is, how do I make it so that the gradients end up being something that even if they look ugly, still I can work with in some smooth way? And so you can, for example, use this thing called a straight-through estimator

• 52:46Speaker 00

  and make gradients become nicer for discontinuous or ugly objective functions. And there's all these things you can do to like visualize how I make the gradients of this very ugly thing look much cleaner. And yeah, I have this image that I use in my slides a bunch that shows like a very nice visualization in three dimensions of what the gradients for what some of these models look like. And you know, it looks like the surface of some very, very, very ugly mountain that like is very, very hard to actually to do anything with. And you know, if you run fancier tags, you can like smooth this out into sort of a nice, smooth surface that that if you're thinking of gradient descent as like a ball rolling down a hill, you want the hill to be nice and smooth. And so this is, you know, what I'm usually trying to think about in high dimensions is just like,

• 53:31Speaker 00

  what does this gradient function look like? And yeah, this continued even all the way through to these unfind tunable models, where one of the papers for this unfind tunable model thing was explicitly saying, we make the gradients very challenging and we make it so that when you train the model, the gradients are ugly. And so as a result, you can't fine tune the model because the gradients are challenging. And this is like literally the exact same argument that people were presenting in 2017 for image adversarial examples. And it fails in the exact same way of you change the learning rate a little bit, you add some random restarts and you add some warmups so that things work a little better. The gradient ends up becoming smooth enough that you can now do optimization. And then, you know, deep learning takes over and the rest is easy.

• 54:14Speaker 00

  And so this is again, like the same intuition here breaking these other class of defenses.

• 54:20Nathan Labenz?

  Was that Sofan that you were referring to there?

• 54:23Speaker 00

  Yeah, so this one was the both the rep noise and tar that have some arguments about what's going on here. Noise is like, yeah, make some arguments about the activations becoming noisy. And that's why you can't do things. And there's other paper called tar that also adds some adversarial training to the process. But one of the very first things that we learned in adversarial training is you have to train against a sufficiently strong adversary in order for adversarial training to work. So there was a paper before Alexander Madri's P G D paper that tried to do adversarial training. They trained against weak adversaries, F G S M that I talked about very briefly, which is like this one step attack. And it turns out that if you train against weak adversaries, then a stronger attack breaks it and like you can't fix that.

• 55:06Speaker 00

  You have to train against a strong enough attack in order for there to be the case that the thing is robust and does not break broken easily by stronger attacks. And what this tar paper did is they trained against one step weak attacks, exactly like fast gradient sign. And so what's the attack? You do many iterations and things basically work out exactly as the first versions of adversarial training failed. And so like, that's why, you know, I read this paper and I just immediately assumed it was going to be broken is because all of the arguments it presents for why it works. I have direct analogies for an image adversarial examples of broken defenses. And so like it felt like the ideas were there, but it just felt to me in spirit like these things I knew were broken before. And so I just assumed, well, probably it's broken here too.

• 55:52Nathan Labenz?

  Gotcha. Okay. So trying to think if there's anything more to dig in on there. Obviously this matters a lot for the future of open source. That is, you know, and I've been looking for without success, some reason to believe from the broader literature that there might be some way to square the circle where we could have open source models that nevertheless won't tell people how to build bioweapons, even if on some level they're powerful enough to do that.

• 56:22Speaker 00

  Yeah, I think this is a very challenging thing to ask for. You know, suppose that I told you I want you to build a hammer, but the hammer has the ability to build all of these nice things, but the hammer cannot be used for one of these seven dangerous purposes. It would be very hard to construct this tool in this way. And I feel like almost all tools that we have have this property. We don't have a C compiler that has the ability to only write benign software and not attacks. You know, every tool that you have can be used in both of these ways. So it's not obvious to me why we should be blaming the machine learning model itself for being able to produce this.

• 57:03Speaker 00

  I like, maybe I blame the Nvidia GPUs for supporting the sufficiently fast floating point operations that the machine learning model does this thing. You know, maybe I blame the transistors for doing the computations that allow the GPUs to allow the machine learning model to do this thing. Like you, you have to put the blame somewhere. And the question is, where are you going to put it? And is that the right place? And is this something that you think is reasonably going to be possible to be effective? This is one of the arguments why people say models should never be open sourced. Because like, maybe now I can say I'm, I have an API. And now like, I'm going to take blame because like, it's actually over an API.

• 57:43Speaker 00

  And I don't currently like this argument because I would like things to be safe in general and not just safe because someone has locked them down and like, has restricted access in this way. But yeah, it's not obvious to me that this should be something that we can actually achieve. I will say, if you're willing to make some assumptions and you don't care at all about performance, there is this thing in cryptography called indistinguishability obfuscation, which is a very technical thing that in principle gives you this for free. It like allows you to construct a function that acts as if it were a black box

• 58:28Speaker 00

  that you can only make queries to, but you can't peer inside at all, even though it's living on your own machine. And this is a thing that cryptographers are thinking about and have been looking at for some time, but is like nowhere near anywhere where it needs to be for this to work for these machine learning models. So like the argument I've given that like it shouldn't be possible, maybe breaks down if if I O actually ends up working, but then, you know, it's not clear again, like now I'm going to jailbreak the thing, right? Like, but like, I don't know, I tend to view these machine learning models as tools and it's not obvious to me, like, do we blame the tool or do we blame the person using the tool?

• 59:06Nathan Labenz?

  Plenty of blame to go around. That's one of my.

• 59:08Speaker 00

  Sure. I mean, this might be like, yeah, I'm, I'm mostly agnostic to the way that these things end up being from like a socio technical way. Like, I feel like this is not my area of work. And so maybe my analogies here are bad and someone can explain what the correct fix to it is and the law might decide something like this is fine. I think the thing that I just want to make sure that people do is that they base whatever they're thinking about on true technical facts. And so as long as, for example, it would be concerning to me right now if someone were to say, you must use this defense, which is known to defend against these kinds of fine tuning attacks. And if you don't do that, then you've done something wrong because the defense doesn't work.

• 59:53Speaker 00

  And so like, I want to make sure that, you know, we, or if people say that like you must do this because this is possible, when like it's not currently known to be possible. And so, you know, like writing these things and like making these informed decisions that rely on what is true technically about the world. And that's, that's more about my world. I'll think about the technical, what's true. And then as long as what people do is based on what's true, then I'm basically happy to go along with whatever the people who figure out how these things fit into a better society, because this is, this is not something I think about. And so I assume, you know, if there was a consensus that was, that would emerge there, probably they're just right.

• 1:00:26Nathan Labenz?

  Well, I have a couple of different angles teed up that I want to get your take on.

• 1:00:30Speaker 00

  Sure.

• 1:00:31Nathan Labenz?

  But before we do that, can we bring back the sort of social engineering style jailbreaking?

• 1:00:38Speaker 00

  Yes.

• 1:00:39Nathan Labenz?

  What's like the same or different about that? How do you think about those as they relate to everything we've talked about so far?

• 1:00:48Speaker 00

  I really don't know how to think about this yet. It's been a while that this has been possible, but like, okay, so it feels wrong to me. This should be, that this should be the thing. So it is empirically true that for many defenses we have right now, the optimization algorithms fail to succeed. But like person at keyboard typing at model can make it do the wrong thing. Okay, let me give you maybe two stories about this. One story is from the computer security perspective, maybe this makes complete sense. You know, if you give me a program and want me to find a bug in it, like, what am I going to do? Like, I'm going to interact with the program. I'm going to play with it, find, you know, weak points, then go looking at the code and figure what's going on. Think a lot, sort of probe it.

• 1:01:31Speaker 00

  Like, I need to be typing and interacting with it in order to find the bugs. Like, I can't, you know, perform gradient descent on C binary and like bug pops out. And so from the computer security perspective, maybe it's actually normal that the best way to find these bugs and these things is like having humans talk at them. Because what are these things designed? They're designed to respond to human questions. And so like, maybe you just need the human in the loop to do this. On the other hand, these are just mathematical objects. Like, these are just machine learning classifiers. They're weird classifiers. They are able to produce text. That's only because we run them recursively. Like, the input is tokens, the output is floating point numbers. Like, you can compute gradients on this?

• 1:02:10Speaker 00

  Like, from the machine learning perspective, it's very bizarre that thinking of these things like human that we're social engineering is in some sense a stronger attack than, you know, if you were to actually about the math. Like, you know, it'd be very weird if I had, you know, some sequel program and like, the way that I broke it was like by asking it, please drop the table and not actually doing some real code execution thing. But like, presumably that's the way that many of these attacks now is, you know. My grandmother used to read me the recipe to napalm. You know, can you please reenact my grandmother? And like it says, okay, sure. But you try and do something like actually based on the math and it just doesn't work out. So, yeah, I really don't know how to think about these social engineering styles of attack.

• 1:02:53Speaker 00

  Because it feels to me like the optimization attacks should be just strictly stronger. But empirically, they're not right now. And so I think this is one of the big things that I don't have any research result on right now, but just feels weird. And so I'm trying to dig into to understand like what is going on behind this.

• 1:03:11Nathan Labenz?

  Yeah, it sort of feels analogous in a way to like,

• 1:03:15Nathan Labenz?

  you know, we have this intuitive physics. I mean, one way that I kind of think about intelligence, you can tell me if you think about it differently or if you see a flaw in this, but it seems like we have an ability in many different domains. But for example, with intuitive physics where, you know, somebody throws a ball at us, we do not have to run a full explicit calculation of all the trajectories. We just have some sort of heuristic shortcut that works that allows me to catch the ball. It seems that we also have models that have developed a similar intuitive physics in spaces that we don't have intuitive physics in. For example, protein folding and like predicting the band gap of a new semiconductor material.

• 1:03:59Nathan Labenz?

  And, you know, the new 10 day, you know, weather forecast state of the art is also like a model now. Even things, you know, Google put out one that was optimizing shipping routes or shipping, you know, planning of containerization across like complicated networks of shipping. So all these sorts of spaces seem to have an intuitive physics. And maybe what we have right now is just, it turns out that like our social intuitive physics, if you will, like actually does kind of apply to the models, given what they have been trained on. Whereas these like more brute force mathematical things, fullness of time probably work as well

• 1:04:39Nathan Labenz?

  or better, but are maybe just a lot slower to converge than the social heuristics that we have built in.

• 1:04:48Speaker 00

  Yeah, no, I mean, that's this is an entirely reasonable thing. It may be true. I would like to understand better what's going on here. And yeah, I don't feel like I understand right now, but this is an entirely reasonable argument.

• 1:05:03Nathan Labenz?

  Okay, so in terms of information, one of the papers I saw that you had co-authored in the last year or so was about getting models to spit out data that they had seen in training, which could have, obviously, you know, privacy implications if they saw your credit card numbers or what have you, even if they had only seen that particular string once in training. That's a pretty remarkable finding, you know, even leaving aside the security implications of it. I know, I want to just maybe first get your intuition for like, how do you understand models to be storing this information? Like, what's going on there that you can see something just once in the context of

• 1:05:48Nathan Labenz?

  this, you know, overall gradient descent process and have that stored at such high fidelity in the weights? I mean, it really is incredible, the amount of compression that's going on, but I don't feel like I have a good intuition for that.

• 1:06:03Speaker 00

  Yeah, okay. Okay, so let me maybe clarify this in two ways. One of them is oftentimes it's not that it's seen that string exactly once, but like that string is contained many times in a single document and the document is seen once. So that's maybe one first point. And the second point is oftentimes these things are trained for more than one epoch. And so the thing might be in one document, and then you train on that document for many epochs, and so it ends up seeing it, you know, a lot of times. And we're seeing this. Okay, so it's interesting. Back in the old days with CIFAR 10, you trade like 100 epochs.

• 1:06:45Speaker 00

  And then we decided, oh no, let's not do that. Let's like train one epoch on a big giant dataset. This is roughly like Chinchilla optimal training. And then we decided, oh no, let's not do just one epoch on our training dataset. Like now, let's go back it up and do more epochs again. And so like we've gone back and forth and like each of these impacts privacy. Like the more times that you train on the same document, the more likely is to be memorized. I think the best numbers we have here from a real production model are very old. Because like the last time that I actually knew how many times something was in the training dataset was for GPT-2. And for GPT-2, we found lots of examples of something that was memorized because it

• 1:07:29Speaker 00

  was in one document and it was repeated, I don't remember the exact number of times, probably like 20 times in that one document. And so that's like the most compelling one. And we know GPT-2 was trained roughly for 10 epochs. So this is the thing has been seen, the same string has been seen roughly 200 times. Now, GPT-2 is a small model by today's standards. And we haven't been able to answer the same question for production models since then because production models don't reveal training data or weights in quite the same way that GPT-2 did. And so we haven't been able to answer this exact question since then.

• 1:08:10Speaker 00

  But even, you know, seeing it maybe 100, 200 times, maybe that even still is surprising. I don't know how to explain this in any reasonable sense. Models seem like they just sometimes latch on to certain things and not other things. And I don't know why, but it happens empirically. And we were surprised by this the first time we saw it. We started investigating this in 2017 with LSTMs before attention was a thing that people were doing. And it's continued since then. And we were very surprised then, we're very surprised now. And I don't think I can give you an explanation for why this is the case. It's true not only on language models, we had a paper on doing this on image models,

• 1:08:55Speaker 00

  where we were able to show that you can recover images trained on diffusion models. There again, we need to have maybe 100 repeats, but like some images were inserted 100 times and we could extract, and some images were inserted like 10,000 times and we couldn't. I'm like, what's going on there? I don't know. Where is it being stored in the weights? I don't know. Like I'm, it's very confusing in various ways that I feel like there's a lot more that that could happen to help us understand what's going on.

• 1:09:23Nathan Labenz?

  I think the best thing I've seen on this still, as far as I know, is from the Bow Lab. This goes back a while now, but they had at least two papers on basically editing facts in a large language model. You know, the famous example was like, Michael Jordan played baseball, which was I think somewhat of a not optimally chosen example, since for a minute he did play baseball, but they could do these things like, you know, change these sentences and do it at some scale, like up to 10,000 facts at a time, and do it with like a certain amount of like locality and robustness. So if you did change it, you know, Michael Jordan played baseball, it would be robust to like different rephrasings of that. It would not also impact like LeBron James or Larry Bird or whatever.

• 1:10:08Nathan Labenz?

  It didn't seem like it was like still super local. They did a sort of, you know, patching strategy where they would go through and just try to oblate different weights and find that like activation patching. So I guess they weren't necessarily blading the weights, but they're just sort of zeroing out the activations at different parts in the network. And it seemed like you could sort of see these waveforms where it was like, this is the most intense place where zeroing it out really makes a difference, but it also kind of matters here and also on this side. So it seemed like it was sort of local, but not like super, super local. I'm just pretty confused about that. What is your intuition for things like if people were to say, you know, because we have, of course, a lot of strategies around, you know, maybe we can't prevent the jailbreaking of open source models,

• 1:10:53Nathan Labenz?

  but maybe we can make it so that open source models just don't know certain things. Maybe we could, you know, exclude all the virology data from the training set, or maybe we could like go in later with a similar technique and like try to sort of delete, you know, or unlearn certain techniques. How much hope do you have for those sorts of things proving to be robust?

• 1:11:15Speaker 00

  Yeah, okay. So let me tackle the three of them at a time. I'll start with unlearning. So it's a very nice paper that I didn't help with, but it's by some of my co-authors, Catherine Lee and others. That talks about unlearning, like it's like half technical, half not technical, saying like unlearning doesn't do what you think it does. And part of the reason why is there's the question of like, what you're unlearning, like unlearning, knowledge is very different from unlearning. Facts, like, you know, there's like, it might be easy to change a fact, it might be very, very different to unlearn some particular piece of knowledge. The other thing I think I'll say about the fact editing thing is it's very different

• 1:11:57Speaker 00

  to find something that works on average to something that works in the adversarial case. And so I might be able to edit the fact, yeah, of, you know, I think the other example that they had was like Eiffel Tower is in Rome. And I can make this be true if I normally ask the model, but like, it might be the case that if I fine tune the model a little bit, like the knowledge just goes back. It's like very hard to talk about the knowledge after any perturbation of the weights, like maybe I've only done some surface level thing, I haven't really like deeply edited the model. I don't know. So there's that question. Then there's the question of what happens if I try to not train the model on certain types of data?

• 1:12:41Speaker 00

  This, I think, is very interesting because in some sense it's provably correct. If the model has never seen my social security number, it's not going to derive it from first principles.

• 1:12:52Speaker 00

  Except that social security numbers actually aren't unique, completely random, if you were born before I don't know when, and they were assigned by state and then assigned to the hospital. And so if a model actually even never saw my social security number, but is just generally intelligent and knew all these facts about the world and knew what the hospital allocation of social security numbers was, it could tell you the first five digits of my social security number. And so is that okay? I don't know, like really it depends. And even suppose that you removed all of this information from the model.

• 1:13:24Speaker 00

  If you had a sufficiently capable model that's capable of learning in context, let's suppose you removed like, you know, all biology knowledge from the model. But you had like a really capable model. You could just give it like an undergraduate course in biology textbooks in context. And presumably just ask it for the answer to some question. It might just give you the answer correctly. And this, this sounds a little absurd and it sounded absurd to me for a while, but then there was the recent result where Gemini was given in context, a book from a language that has basically no speakers that no one could do. And it could answer the homework exercises after seeing the book in context. And so, you know, I think if you're unlearning for particular capabilities, but your model that you're trying to train is just like generally capable,

• 1:14:07Speaker 00

  like you're kind of asking for trouble because you want a model that's like so good, it can learn from few shot examples, but not so good it can learn from few shot examples on this particular task. And this, I think, is part of the reason why people don't actually want to remove all knowledge of certain things from the training data. Like, in some sense, it would be very much like a person who was never exposed to all of the things that like you're not supposed to do in public. Like, it's important to know what the list of things are you're not supposed to do in public, so that you can then not do them. Whereas if you just weren't aware of that and just like sort of were a person conjured that had no social skill, it would be like much more embarrassing for you

• 1:14:50Speaker 00

  because you would have to learn, like people would ask you to do something. You'd be like, okay, let me go do that thing. And like, this would be bad. And so you have to, to know this, you have to know something about the bad things. So you can just not do those bad things. You know, you could imagine if, if in one world you have a model that's never seen anything about weapons and even you give instructions for how to build a bomb and has no concept of death. And it's like, okay, well, of course I'm going to give you the answer for how to do this thing. Like, you know, why not? And it gives you the answer. Or you could have a model that has a concept of this and, and refuses, which is the direction people are trying to pursue now. And I think probably is more likely to succeed. But I don't know, it's a very challenging question that like, people, I guess, are going to try both ones and we'll see empirically what works out.

• 1:15:35Speaker 00

  And that's, we'll, we'll go from there. But yeah, I'm sort of mostly agnostic to all ideas sound good. We should try them all and then see what ends up happening. But like, I think there's reason to be skeptical of all of them.

• 1:15:47Nathan Labenz?

  Okay, here's another empirical result that I want to help get your help understanding. This is the obfuscated activations bypass latent defenses paper that recently.

• 1:15:56Speaker 00

  Yeah, it was in a paper.

• 1:15:57Nathan Labenz?

  So we did a whole episode on it and I have to confess, I still came away not really sure what to make of it. And I sort of wanted to set up like maybe a useful toy example, maybe not. In the paper, they do this sort of cat and mouse process where they like train an out of domain detector. And then it works and then they attack again and they manage to, you know, beat it again. And then they, you know, continue to train the defender, the detector. And then it starts to work again and then they, you know, find more adversarial examples that they can't catch. And this goes on for 70 generations and it continues to work for 70 generations. At which point they, you know, deem that enough to publish. So I'm like, okay, what does that mean?

• 1:16:42Nathan Labenz?

  I have an intuition that it must mean, but they didn't necessarily agree with this or they didn't find it too compelling when I pitched it to them. But my intuition was like, it seems like there's a lot of unused space in there somehow that, that these techniques can chase each other around the latent space. And if there is so much space that's like unused, such that you can like go 70 generations deep of cat and mouse chasing one another. Does that imply that the models are like under trained relative to the parameters that they have? Or that they could be made more sparse? And so just to further motivate my own intuition, which you can then deconstruct.

• 1:17:22Nathan Labenz?

  A while back, I also did an episode on a paper called Seeing is Believing, um, which was from Ziming Liu and Max Tegmark. And they basically just did something really simple. These were toy models, but they imposed a basically a sparsity term in the loss function. To get the model to do whatever task was doing, pretty simple tasks, like simple things like addition or whatever. But also to do it in the sparsest possible way. And my gut says, although again, I can't formalize this, that if I had something that was like. Crystallized down to that level and they have really nice animations that show how you start with like a dense network where everything's all connected and then gradually

• 1:18:04Nathan Labenz?

  the weights go to zero for most of the connections and you see this sort of crystallization effect where now you've got like a very opinionated structure to the network that remains to the point where you could literally like remove all those other connections that have gone to zero and still get the performance. It feels like if I go that far, then these sort of obfuscation attacks would like no longer be possible because I've sort of in some sense squeezed out the extra space. But I don't know, I'm maybe just totally confused. So where do you think I'm confused? How can you de-confuse me?

• 1:18:40Speaker 00

  Sure. Yeah. So I saw this result and I was like, exactly what I expect. Let me tell you why. Because there's a paper from 2016 or 2017 by Florian Tramer and Nicolas Papernau called "The Space of Transferable Adversal Examples," where they asked almost exactly this question for image classifiers. They said, "Let's suppose that I take an image classifier and I take an image and I want to perturb the pixels of the image to make it give the wrong answer. And there is a direction that is the best direction to go to that makes the image maximally incorrect. As an attacker, like I'm just going to prevent you from going in that direction. Like that's against the rules. You can't do it. Find me the next best

• 1:19:22Speaker 00

  direction to go to that makes the image become misclassified." And then it gave you another direction, which is orthogonal to the first direction. It doesn't go that way. That works. And I said, "Okay, you can't go in either the first two, direction one or direction two, or any combination of those directions. Find another direction." And the attacker finds a direction three. And then they do the same thing. No direction one, two, three, or any combination of these three. And they repeat this process and they have a plot that shows I don't remember, 10s, 50 directions that you can go in for image episode examples that all of them work. They're like, they get like a little less effective as you start moving out, but like it remains effective in many directions. And this initially was surprising to me. I think surprising to them.

• 1:20:06Speaker 00

  That's why they published as a paper. But like, you can maybe rationalize this afterwards by again, almost all vectors are orthogonal in high dimensions. And so if you just like, if I can give you 10 attacks, probably this is just 10 orthogonal vectors that like are just not using any of the same features, just by virtue of high dimensions. And so maybe, maybe that's why it makes sense. But like, but if you believe this paper from 2016, then this recent paper makes complete sense to me. It's saying the exact same thing is true, but in the case of language models defeating, yeah, the circuit breaker paper, which yeah, makes, makes sense, given that if you don't

• 1:20:48Speaker 00

  believe the other result, then I agree, it's very surprising to see it the first time. But maybe this is one of these things you learn when attacking the space of dimensions for attacks was so vast that it's very hard to rule out what you're trying to cover. And yeah, I don't know how to give an intuition for it. I feel like many of the things in high dimensions are just like, you never, you never understand it, you just get used to it. And this maybe is the case here.

• 1:21:13Nathan Labenz?

  So if you were going to try, I mean, do you share the intuition though, that like, you probably couldn't do this on these really small sort of crystallized toy models?

• 1:21:25Speaker 00

  I don't know, because what these models are doing is these models are not wasting any weights, which is different than not wasting any, any directions in activation space. That may still be the case. In particular, if you take models and compress them, they don't become more of a sort of robust. This was a thing people thought might be true. Yeah. Again, maybe 10, eight years ago, they thought this might be true. It's not the case. Why is that? Again, maybe let me give you some intuition from a paper that's from 2018, 2019, maybe from Alexander Maggi's group, where they say the following.

• 1:22:04Speaker 00

  Maybe adversarial examples are not actually entirely uncorrelated from the data. Like, maybe what's going on is these are real features that you need for classification that are just being activated unnecessarily, like larger in different ways. And so they have some good, very good experiments in the paper. I won't go into that sort of justify this. But the idea presented there is, the title of the paper is something like adversarial examples are not bugs, they are features. And like the, the thing that they're trying to say is there's a very good reason to believe

• 1:22:46Speaker 00

  that what you're getting across when you construct one of these attacks is actually just activating real features the model needs for, to do accurate classification. And you're just activating them in a way that is not normally activated when you have some particular example. And this, I think, maybe explains some of this, that like, if you take these models and compress them, you're still just using the features that they had to have anyway. This explains lots of things. You might think about, this is why, for example, you might imagine adversarial training can reduce the accuracy of the model on normal data, because you suppress certain features that are necessary. This is maybe why adversarial training doesn't actually solve the problem completely, because you can't remove all the adversarial directions.

• 1:23:31Speaker 00

  There are problems with this model. There are some other models that are slightly more general, that have some nice properties too. But this is maybe the way that I generally tend to think about some of these things. And I don't know, maybe it's not correct, but it's a useful intuition that I've found guides me in the right direction more often than not. And I think this is maybe all you can ask for, for some of these things.

• 1:24:01Nathan Labenz?

  So can you summarize that one more time? It's like, that basically these features are important in domain, but they're sort of being

• 1:24:09Speaker 00

  Yeah.

• 1:24:10Nathan Labenz?

  recombined in a way that didn't happen in the training process?

• 1:24:14Speaker 00

  Yeah. So let's suppose that you're trying to classify dogs versus cats.

• 1:24:23Speaker 00

  What you tend to look at as a human is the face and the ears and the general high-level shape, because that's what you think of as the core concept. But there's no reason why the model has to use the same features that you're using to separate these images. The only thing the model has is a collection of images and has to separate them. And one thing, for example, the model might look at is the exact texture of the fur, really low-level details of the texture of the fur, which for dogs and cats probably does perfectly correlate with whether or not this is a dog or a cat. But when you have an idea of a dog in your mind, you're imagining the high-level features.

• 1:25:06Speaker 00

  You're not imagining the low-level details of the fur. And so suppose that Navisole example changes the fur from dog to cat.

• 1:25:16Speaker 00

  And the classifier now says this is a cat.

• 1:25:20Speaker 00

  Is the classifier wrong? The classifier might have been producing a dog fur versus cat fur classifier, which is exactly aligned with the thing that you were training it to do. Like, you were training it to separate dog fur from cat fur. You were also trained to separate dogs from cats, but you never told it the distinction between these two things. And so here is a feature that is really, really useful for getting the right answer, that as an adversary, I can now use to perturb, to switch the model from one label to another, even though it's not the feature that I as a human relied on. And I'm giving you this idea of cat versus dog fur, but there are, you could imagine, all kinds of other things that even as humans we don't pick up on, that might be legitimately

• 1:26:02Speaker 00

  very useful features to classify, that really do help the model. You know, there might just be some really crazy high-level statistic on the pixels of the image that are like amazing features of dog, but like, we never told it, this is a dog because of these reasons. You just said, "Separate these two things from each other," and it picked up on the statistics. And, you know, there are these crazy results that have shown, you know, machine learning models can look at, like, there's some results of like, they can look at like, small regions of the eye and identify who the type of the person who it is, like, you know? And the models have the ability to pick up on these very, very small features that, as humans, we don't intend for them to pick up on, but they're correlated very, very strongly with the data. And so...

• 1:26:48Speaker 00

  Maybe that's what's going on when we run these attacks. You see this even a little bit with these adversarial suffixes, where the adversarial suffixes look like noise, but like there are some parts of them that make a little bit of sense. You know, one of the strings that we had in the paper was to get, I think Gemini to output at the time, some toxic content. One of the things that was discovered by gradient descent was, I think it's like now write opposite contents or something like this. And, you know, what the model would do is it would give you the nasty strings and then it would go compliment you. And so like this was apparently a very strong feature for the model of like, how do I get it to say good?

• 1:27:32Speaker 00

  The bad thing I can like, I can say like, okay, in the future, you can tell me a good thing. And so this like may not have been the thing that we wanted, but like it was discovered as a feature. And as a result, you can go and exploit that feature. And so some of these features are a little bit interpretable. Some of these features are not interpretable, but like might just actually be real features of the data. And that might help explain some of what's going on here.

• 1:27:56Nathan Labenz?

  Yeah, how much do you think this should make us question what we think we know about interpretability in general? Like when we do sparse auto encoders, for example, we, you know, feel pretty good, or at least, you know, many of us do that like, okay, all these examples seem to, you know, be appropriately causing this feature to fire. And therefore we've like figured out how the models work. But it seems like the story you just told would be consistent with that all being kind of a self-delusion or confusion where maybe they are actually like the, just because we can auto-label them in a way that looks good to us, doesn't mean that that's actually the feature that the, you know, in the models world model is really operating.

• 1:28:41Speaker 00

  Well, no, I think they're not inconsistent. The sparse auto encoder's work does not claim to be able to label every feature in the model according to what the humans would label it with. Right? Like it sort of says, here are some of the features that we can explain and have like very strong correlations with, I don't know, golden gate bridges or whatever.

• 1:28:59Speaker 00

  But it has other features that are very hard to interpret and very hard to explain. And it's entirely possible what these features are doing, as these are the features that are the non-robust features that are entirely helpful for prediction, but humans don't have the ability to attach a very nice label to. And maybe there still are the other features, like the model probably does learn, at least in part, you know, the shape of what a dog looks like and that this means a dog and not a cat. You know, there probably is a feature for cat ears. But like, when you're making the final prediction, you're just gonna sum together all of these outputs. And in normal data, they're like all perfectly correlated. You know, you're gonna have both the cat ears and the cat fur and not the dog's shape. And so you could just like sum them up directly and like this gives you a really good classifier.

• 1:29:44Speaker 00

  And so for normal data, you can give some kind of explanation of what's going on by looking at these features. But as an attacker, what I do is I find the one feature that has a very strong bias of like, you know, the edge on this weight or whatever is like, you know, plus 100. And, you know, you activate that very strongly in the opposite direction. And this was a non-robust feature that is not something humans can explain very nicely. And as a result, this gives me the attack. And so it's not necessarily true that just because this is the case, you can't explain what's going on for some parts of the model. I think if someone were to tell me, I have a perfect explanation of what's going on in every part of the model, then I might question what's going on here. But for the most part, they're explaining some small fraction of the weights in a way that they actually can.

• 1:30:28Speaker 00

  I mean, this is the whole purpose of the sparse autoencoders in the first place, is you have your enormous model and you're shrinking this down to some small sparse number of features that can be more better explained. And so you're losing a bunch of features in the first place. And even for the sparse autoencoders, they can't explain all of those. So you again lose some and some of the things you can explain, but a lot of the stuff that's going on behind the scenes is magic and you can't easily explain as well.

• 1:30:49Nathan Labenz?

  So, okay, a couple other angles on this that I thought of. I guess first of all, just motivated by the fact that like, or at least I think it's a fact, but you might call this an allusion too. I feel like I am more robust than the models in some important ways.

• 1:31:07Nathan Labenz?

  Now, it's a little bit weird because like, nobody's tried doing sort of, you know, the gradient descent process on my brain. So you might think, well, actually, if we put you under an FMRI and we were able to, you know, really look at the activations, and we did, you know, another, this is almost like a walk down memory lane, because I did one episode on Mind's Eye, which was a project out of Stability and Collaborators, where they looked at FMRI data and they were able to reconstruct the image that the person was looking at, at the time that the FMRI snap was taken. And that's still like, pretty coarse, you know, as I recall, it was like, basically, sort of grain of rice-sized voxels

• 1:31:51Nathan Labenz?

  from within the back region of the brain, you know, huge numbers of cells in just, you know, the one little, you know, number that correspond to these like spatial voxels. So, pretty coarse, but able to do the reconstruction. Maybe you would think that, like, actually, if somebody could sit there and show you images and take these things, then they could actually find a way to like, break your particular brain to get you to think it was a cat when everybody else kind of looks at it and says it was a dog. I guess, for starters, like, what's your intuition on that? Obviously, we don't know, but what do you think?

• 1:32:25Speaker 00

  Yeah. So, there's a paper from a little while ago that looks at evaluating the robustness of humans, of time-limited humans, to actually just constructing adversarial examples. So, you, you take a cat, you adversarially perturb it according to what makes an ensemble of neural networks give the wrong answer. And then, like, you flash the image in front of a human for like, 100 milliseconds. And you ask the human, like, what's the label of this? And it turns out that people are fooled more often by adversarial examples when you do this than by random noise of the same distortion amounts. So, maybe one explanation here is, when I look at the image, I'm not just like, giving you a single forward

• 1:33:08Speaker 00

  pass through the model evaluation of what I think it is, and that I'm doing some like, deeper thinking about like, the context of what's going on. Like, the thing that walked in looked like a cat, it still looks like a cat, it did the thing behaving like a cat, and now you ask me what it is, and I say it's a cat. Like, I'm not just like, labeling what's in front of my eyes right now, I'm looking back at the context too. Maybe this explains some of it. If this is true, there are some recent lines of work looking at improving number of chain of thought tokens, like increasing that makes models appear to be more adversarial robust. Maybe this is, maybe this is true and that explains that result also. But yeah, it also might just be the case that, yeah, like you said, we don't have light up box access of the human brain and we can't do this. And

• 1:33:52Speaker 00

  if we could, then it would be very easy. I don't know. I do think it's definitely a, like observable fact that humans are more robust to at least the kinds of attacks we're doing right now with transfer attacks. Like, it takes maybe a thousand query images to construct an eventual example that fools a neural network. But I do not think that if someone had run the same attack on me with a thousand query images, it would be able to fool me. And so like, it's like, in some very real way, like the models we have are a lot more brittle, but I tend to be much more driven empirically and not by, like, what might be if humans kind of thing. And, you know, maybe this is to my detriment.

• 1:34:34Speaker 00

  A bunch of people seem to be getting very far, like the whole deep learning thing, like, you know, trying to model things more like what humans do, like the reasoning, think, you know, let's think step by step, like all seems in some sense motivated by some of that. So maybe that's, maybe it's a good way of thinking about it, but it's just generally not, not the way that I tend to approach these problems. But yeah, I don't know. Maybe, maybe the case.

• 1:34:54Nathan Labenz?

  That result about humans being more likely to be tricked by adversarial examples that were identified through the attacks on models, as opposed to just like similarly distorted images, not created that way, is a really interesting result.

• 1:35:10Speaker 00

  Yes, that's a very nice, a very nice result. Yeah.

• 1:35:12Nathan Labenz?

  I'm, I'm very interested to follow up on that.

• 1:35:14Speaker 00

  Yeah, this one's by Nicola Papern and collaborators from six years ago or something like that.

• 1:35:20Nathan Labenz?

  Yeah, cool. I'd never seen that, but that's definitely fascinating. So reasoning, it sounds like you're kind of agnostic. I mean, I was just gonna quote the, there's that one example from the OpenAI deliberative alignment scheme where the model says, it seems like the user is trying to trick me. So that's like pretty interesting, right? And this is very sort of. Fluffy at this point, but I often don't feel like my adversarial robustness is the result of reasoning. I feel like it's much more often upstream of reasoning, like purely on an introspective basis. What often happens is I get that sort of feeling first that something seems off and then that triggers me to reason about it. And then I conclude

• 1:36:04Nathan Labenz?

  that yes, something is off, or maybe no, it actually seems fine. But it does seem like it is much more of a heuristic that is sort of triggering the reasoning process as opposed to an in-depth chain of thought that kicks these things up.

• 1:36:19Speaker 00

  Yeah, I don't know. But like, maybe there's a bunch of recursive stuff going on in your brain before it goes to the reasoning part. And so the thing that gives you the feeling might actually have been a bunch of recursive, like, loops of your internal model, whatever you want to call brain thing, like, doing some thinking and that's what gives you this in the first place. And then you go do like some actual explicit reasoning in English that you understand, but maybe it's already happened. I don't know. Like, it's hard to...

• 1:36:44Nathan Labenz?

  Yeah, reasoning in latent space, you might call it.

• 1:36:46Speaker 00

  Yeah, sure. Yes. I feel like this is like, a lot of people in deep learning, because they are like, doing deep learning, like to draw analogies to biology without actually understanding any of my biology. And they like, sort of say, well, the brain must be doing this thing that like, if you ask a biologist, they're like, well, obviously not the case. I don't know. That's why I tend to just assume that I don't know anything was going on in the brain and I'm probably just wrong. And yeah, just use it as an interesting thought experiment, but that's only I base any results on.

• 1:37:15Nathan Labenz?

  Okay, so obviously, we'll probably soon learn quite a bit more about how robust these reasoning defenses are. I was also kind of inspired, I'm doing another episode soon with another DeepMind author on the Titans paper, where they're trying to basically develop long-term memory for language models.

• 1:37:37Nathan Labenz?

  One of the key new ideas being using a surprise weighted update mechanism. So when a new token or data point, whatever, is surprising to the model, then that like gets, you know, a special place in memory or a special, let's say, you know, that is encoded into memory with more force or, you know, with more weight on the update than when it's an expected thing. That definitely intuitively seems like something sort of like what I do. I was reminded of the famous George W. Bush clip where he says, you can fool me once, but you can't get fooled twice. And so I wonder if you think of that as maybe a seed of a future paradigm like,

• 1:38:22Nathan Labenz?

  and it also kind of gets to the question of what do we really want, right? If we're going to enter into a world soon of AI agents, you know, kind of everywhere. And we also just kind of think, well, geez, you know, how do we operate and how do we get by? It does strike me that full adversarial robustness is not something that we have because we do get tricked. And maybe what we need is like the ability to recover or to remember or to not fall for the same thing twice. Or to not make like catastrophic, it's like, okay, if we make some mistakes, but certain other mistakes are really problematic. So I guess there's kind of two questions there. One is like, any thoughts on that kind of long-term memory idea? And then this kind of opens up into a, what do we really need?

• 1:39:07Nathan Labenz?

  And might we be able to achieve what we need, even if that falls short of like true robustness?

• 1:39:12Speaker 00

  Yeah, no, the long-term memory I think is fun. I think, yeah, it's, it's very early and yeah, I'm glad you're going to talk with them. And I think, yeah, they'll probably have a lot more to say on this than I would. I think it's a very interesting direction and probably there's a lot more interesting that could happen as a result of coming from this. I think on this general question of robustness, do we need perfect robustness? I think, you know, there is a potential future where.

• 1:39:37Speaker 00

  I think maybe it's like my median prediction is these models remain roughly as vulnerable as they are now. And we just have to build systems that understand that models can make mistakes. The way the world is built right now, for the most part, assumes that humans can make mistakes. And has put systems in place so that like any single person is not going to cause too much damage. You know, if you're in a company, for the most part, when you want to add code to the repository, someone else needs to review it and sign off on it. In part because maybe I just make a bug mistake, maybe because I'm doing something malicious, but like you want another human to take a look and make sure everything looks good. So maybe you do the same thing with models. You just understand models can make mistakes. And I'm just going to build my system in such a way that if the model makes a mistake,

• 1:40:21Speaker 00

  then, you know, I pass it off to the human. This is, I guess, for example, the way at least currently that the OpenAI operator thing works, where every time it sees a login page or something like this, or something it's not sure what to do, it says, "What should I do here? Please tell me what to do and then I'll follow your instructions." Of course, as an attacker, you could then try and prevent it from doing this and try and make it, give the answer instead. But if you built something outside of the system that prevented it from putting any information into a password box, like you just like, the, the model says, "I want to type this data here." And the system that's like actually driving the agent says, "Well, that is input type equals password. You are not allowed to do that." Like, I just like, will prevent it and make the user then go. Like this, like, nothing that the model says is going to convince that to change.

• 1:41:06Speaker 00

  And so you could like build the system to be robust, even if the, the model isn't. This limits utility in important ways. This is not perfect. What if some website does not have input type equals password, but just builds its own password thing in JavaScript and you don't get this? You can imagine trying to build, you know, the environment and the agent in such a way that you can control what's going on, even if you don't trust the thing that's running internally. And I think this is probably what we'll have to do if we want to make these systems work in the near term.

• 1:41:35Speaker 00

  I still have some hope that maybe we'll have new ideas that give us robustness in the next couple of years. Progress in every other field has been growing much faster than I expected. It's entirely possible that we get robustness as a result of some reasoning thing that someone's very clever about and this works. I'm not optimistic, but I wish it happens.

• 1:41:54Nathan Labenz?

  One other idea that was inspired by a conversation with Michael Levin, famous heterodox biologist, was he just kind of quipped, and this was sort of an aside in the context of that conversation, but he basically said, "If a biological system is too interpretable, it basically becomes very vulnerable because you'll get parasites. Anything that is transparent is, in a way, easier to attack," which kind of flips, in a way, it's my earlier notion of like, maybe these sort of distilled, crystallized models are in some ways more robust. He was kind of like, "Nah, you also have the other problem of the easier they are to understand,

• 1:42:35Nathan Labenz?

  in some ways, the easier they are to attack." But that also then prompted me to think, maybe we should be looking for sort of defenses that we can't explain. Like, do you think there's a line of work that could be developed that is like, "I'm not going to start with a story, but I'm just going to try to evolve my way toward a more robust defense that I won't tell it. I won't even necessarily understand it. I certainly won't be able to tell you a story as to why, but I'll just sort of create optimization pressure in that direction and maybe that spits out something that could be harder to break."

• 1:43:11Speaker 00

  Entirely possible. I mean, you could maybe draw an analogy to cryptography in some ways. So in cryptography, there's like two directions of cryptography. There's like mathematical cryptography that sort of has like very strong foundations and like a particular set of assumptions, and the thing works if the assumptions are true and you can prove that. And then there's like symmetric key cryptography, which I guess is like block cipher design and stuff. And people have like high-level principles of what you want. You want like diffusion and confusion and these kinds of things, but like, how do you end up with like the particular design of something? You do something that like feels like it makes sense. And then then you run all the attacks and you feel like realize like, oh, you know, this like has something that's wrong in some particular way.

• 1:43:56Speaker 00

  Let's just like change that piece of it so that that doesn't have the case anymore. And then you do this for, you know, 20 years and you end up with AES. And like, you know, like, none of the like individual components are like, like, there's no reason why AES works. Like, there's no, absolutely no formal proof of robustness to any kind of attack. There are proofs of why particular attack algorithms that we know of in the literature are not going to succeed. The design principles why it were inspired by these attack algorithms that we have. So you can show there is never going to exist a differential attack that succeeds with better than brute force. And there's never going to be a linear attack. And like, you can sort of write down these arguments. But there's nothing that says it works in general.

• 1:44:39Speaker 00

  It just, by iterating on making the thing slightly more robust each time someone comes up with a new clever attack, you end up with something that's very, very good as a result. But there's no nice crystallizable mathematical assumption that assumes the hard factoring problem. And as a result, you end up with with this line of work. There's nothing like that for symmetric cryptography, but it works really well. Maybe this works in machine learning where you just like, you don't have any real understanding of why things work. You iterate against attackers and you end up with something that's robust. I think it's maybe harder in machine learning because in machine learning, you want the thing primarily to be useful and then also to be robust to attack. Whereas in cryptography, the only thing you want is robust to attack.

• 1:45:22Speaker 00

  And so it is a lot easier in that sense. The primitives you're working with are much, much simpler to analyze. There are like lots of other things that that change, but you know, it would not be without any prior work that this is something that seems impossible. I just, I'm skeptical, if only because I would prefer there to be something that we can point to a reason why it works as opposed to just saying, well, empirically it is. But if that's, you know, if you gave me the option between something that actually is robust, we just can't explain and nothing that that works at all. Like, of course, I take the thing that works, we just can't explain it. I just would be worried that in the future, we might be able to break it. And maybe that happens, maybe it doesn't.

• 1:46:02Nathan Labenz?

  Yeah, that's really interesting. I had no idea about that. I don't know much about cryptography at all. But my probably not even identified, little unquestioned assumption, I think, had been that there was like a much more principled design process than the one you're describing.

• 1:46:18Speaker 00

  I mean, there was a people spent 30 years breaking ciphers and learning a lot about what you actually have to do to make these things robust. But like this very careful analysis that goes into this. And so I'm not, I don't mean to say that they just cobbled stuff together and hoped for the best. You have to think really, really hard about this. But the final thing that they came up with has no security argument for the in general attack, other than just here are the list of attacks in the literature and proofs of why these attacks do not apply. And there's, there's no assumption. Whereas, you know, in, in other areas of cryptography, you have a single assumption where you're going to assume, you know, factoring is hard. And if factoring is hard, then here are the following things that are robust under that. And, you know,

• 1:47:01Speaker 00

  you end up with, it's not quite RSA or something like it. You could also assume maybe discrete log is hard. It's a math thing that like, it's hard to take the logarithm in discrete space. Maybe this is hard. If, if you believe discrete log is hard, then here are the algorithms that are robust. Or maybe you assume discrete log is hard over elliptic curves. And then you end up with a set of algorithms that work under this assumption. And each of these algorithms, you can say, this is effective if and only if this very simple to state property is true. And people really like that because you can sort of very cleanly identify why the following algorithm is secure. And the way you do this is with some reduction. You say,

• 1:47:41Speaker 00

  here is a way to, if and only if, you know, break the following thing, if I could break the cipher, then the, the, the statement is not true and vice versa. But there is no similar argument to be made in most of symmetric key cryptography or hash functions or something like this. It's just empirically, the field has tried for 25 years. The best people have tried to break this thing and have failed. Here are all of the attacks. Here are proofs that these attacks will not work. But maybe tomorrow someone comes up with a much more clever differential like variant, but instead uses, I don't know, multiplication or something crazy. And all of a sudden, all bets are off and you can break the thing like this doesn't exist. And it's worked there at least once. And so, you know, maybe it works in machine learning. I think drawing analogies to any other field always is fraught because

• 1:48:24Speaker 00

  the number of things that are different is probably larger than the number of things that are similar, but at least this happened before.

• 1:48:31Nathan Labenz?

  That's actually a perfect tee up for another kind of intuition I wanted to see if you can help me develop, which is the relationship between robustness and other things we care about. My sense is that if you break the cryptography algorithm, it's broken and you can sort of, now you can access the secrets, right? Seems like a pretty binary sort of thing that you've either broken it and got through or not.

• 1:48:58Speaker 00

  Maybe.

• 1:48:59Nathan Labenz?

  Okay, well, you could.

• 1:49:00Speaker 00

  Because, yeah, okay, no, I'll quibble with you. So when a cryptographer says that they broke something, like if you have a paper like, so a crypto system is designed to be robust to an adversary who could perform some level of compute. So like, you know, AES 128 is an encryption algorithm with a 128 bit key. The attacker gets to use, if a cryptographer would tell you AES 128 was broken, if you could recover the key in faster than 128 time. If in 2 to the 127 time, twice as fast, but still after heat death of universe time, people will be scared about the security of AES and probably start thinking of something else. So like, you know, like this is like technically a break. And the reason why they do this is because attacks want to get better. And it turned to like, if you can go from 128 to 127, you're like now very scared.

• 1:49:44Speaker 00

  Why not from 127 to 125? 125 is still way outside the possibility, but if you got down to 2 to the 80, then like, you know, now we're scared. And so like, you know, like there is a continuum here, but it is the case that cryptographers, for the most part, only use things that like just are actually secure. And then just like start becoming very, very scared as soon as you get the very first weak break. But it still is the case that there exist weak breaks that are not complete breaks of the entire system.

• 1:50:11Nathan Labenz?

  You know, that's, yeah, it's more similar than I had really conceived of. So that is quite interesting. And so, and 2 to the 80, just the mental math there is like each 2 to the 10 is three zeros. So that would be 24 zeros. So now we're kind of, if something was broken in 2 to the 80, it would be. The compute to do it would be sort of roughly on the order of like getting into the sort of frontier language model flops?

• 1:50:40Speaker 00

  Yeah, that's maybe a good argument. Yeah, I was like, you know, two to the 60, someone can do on their own machine if they're trying pretty hard. You know, if the problem is pretty easy or like they try a lot, you know, two to the 70 is like, you know, you need a bunch of work, but like probably it can happen. And two to the 80 is like, you know, I guess a nation could probably do this thing of like, you know, if you really tried it like the constant matters a lot here because, you know, the difference between two to the 72 to the 80 is like, you know, only a thousand. And so if the problem is a thousand times more complicated, then maybe it becomes two to 70 or two to the 90, whatever the case may be. But like, you know, this is, I mean, for example, this is why people typically use AES 256 because it's like so much bigger than this that like, even if it went down to 100, we're still fine. But, you know, this is just cryptographers being paranoid and it's a

• 1:51:24Speaker 00

  good thing about the way that people do things. But yeah, generally, these things are designed with large levels of security in mind to prevent this. But yeah, this is the reason why, yeah, in particular, 80.

• 1:51:35Nathan Labenz?

  Well, that's educational for me. Going back to the AI domain,

• 1:51:41Nathan Labenz?

  it does seem like there it is maybe still just a lot messier in general in, for example, the obfuscated activations paper that we talked about a little bit. They looked at how does the performance of the model change if, for example, you need to evade a detector at just like one layer of the model? I think that the toy problem was like, it's going to write SQL, and we want to detect that it's writing SQL. That's what the detector is there for. But we wanted to get it to write SQL without setting off the detector. With just a single detector at one layer, they were able to do that with very minimal performance degradation. But then like the more detectors they developed and, you know, once they had one at every single layer, you could

• 1:52:26Nathan Labenz?

  still evade the detector, but your SQL was shit. So it's like there's, there is some other kind of thing that's moving too, like the performance of the model might be getting worse. And then I also am kind of trying to think about like, again, in the sort of what do we really care about? You know, the, the nature of the mistakes or the nature of the outputs also matters a lot too, right? Like it's one thing for me to get some toxic content. It's another thing for me to get, you know, the big bioweapon that everybody, you know, is kind of most fearful of. And so you need to both like break the defense, but if you want to actually, you know, do real harm in the world, you need to also have the performance still strong. And there just seems like there's a lot of dimensions there.

• 1:53:10Nathan Labenz?

  So I wonder like, may, maybe I should just say like, what do we know at this point about the relationship between the robustness of these attacks and the sort of so what of an actual attack getting through? Cause I did look at that chart and I was like, well, jeez, if we do put a detector on every layer, then yeah, you can fool them, but you can't actually probably query the database. So in some ways, you know, you maybe are still okay, but that feels like something I just beginning to sort of explore. And I'm sure you know a lot more about it.

• 1:53:41Speaker 00

  Yeah, no, this is a good question. In part, this is, you know, how most secure systems are designed to be secure is just layers of security on top of layers of security. The fence in depth is the thing that works in practice. Yeah, we were just talking about block ciphers and the reason why those are robust is because they have many layers of this confusion diffusion thing. You stack enough of them and the thing becomes hard to, you can get past a couple of layers, but you can't get deeper. It's a tried and tested thing in security that this generally works. I think all else equal, we would of course prefer a system that just is robust because you can write down why it's effective. But if you can't have this, having something that empirically has these 20 layers. And when put together in the right way, it just like, and turns out that the attacks don't work is something that we would be happy to settle for.

• 1:54:27Speaker 00

  You know, I think there are reasons you might be concerned just because maybe it turns out that, you know, someone can have an attack that bypasses all of them because it's fundamentally different in some interesting way. And we just have to accept this and I don't know, I think like this is like,

• 1:54:44Speaker 00

  one of the options we should be exploring a lot because, you know, maybe it turns out that full robustness is basically impossible. And we just need to settle with this, you know, layers of detectors. And if it turns out you can still have robustness, then hey, detectors are still useful anyway. And so like, it's good to have done the work ahead of time to see.

• 1:55:00Nathan Labenz?

  Is there any work on like trying to minimize the harm of mistakes? Like, for example, I have a Ring camera in my backyard that constantly alerts me that there's a, it has a, there is motion detected and then it has, there is a human detected. And it frequently alerts me that there's a human detected when it in fact saw a squirrel. And so this has me thinking like, you know, for actual real world systems, I don't necessarily care too much if it confuses a dog for a cat for a squirrel, but what I really want to know is if there's a human, you know? So is there any sort of relaxation of the requirements that would allow us to be like,

• 1:55:44Nathan Labenz?

  you can get some small stuff wrong, but we're going to really try to defend against the big things?

• 1:55:48Speaker 00

  Yeah, I mean, in any detection thing, you just, you have to tune the false positive, true positive rate, right? And you can pick whatever point you want on the curve. And the question is, what is the point that you pick on the curve? And I was reading some security paper a while ago that was looking at the human factors aspect of this particular problem. And it was arguing it's actually better user experience to have false positive flagged humans every once in a while. Because if you don't do this, then the person's like, well, my camera's not looking. But if you do do this, the person's like, oh, I feel a little better. Like, you know, maybe it was made a mistake. It was like a little too aggressive, but like, I feel more confident now it's going to find the actual human when there's a real human there.

• 1:56:33Speaker 00

  And so, you know, like, there's like, maybe some actual human factors, reason why you might prefer to set the thresholds a certain way.

• 1:56:39Nathan Labenz?

  Although in my case, I've tuned them out entirely now. So that's

• 1:56:43Speaker 00

  Sure, yes. This is the other version of

• 1:56:45Nathan Labenz?

  We're over the hump on that curve.

• 1:56:46Speaker 00

  Yes. Where, you know, this is the thing with like, there's like lots of software like that like will report warnings in code, like the compiler will give you warnings, like you're doing this thing, it's like, not great. And once you've passed a thousand warnings, the person just disables all warnings. And like, it would have been good, like it's an important thing it's going to warn you about, but because you had the 900 warnings that were not actually important, you know, just like you've lost the actual important signals in all of this. And yeah, picking the right, the right value in this is very hard. It's mostly a question of tuning true positive, false positive rate on a defense by defense basis. Towards your question of, you know, can we just like only catch the important things?

• 1:57:24Speaker 00

  It really depends on if it's, if that's an easier problem. You know, in some cases it is and you can like actually be willing to tolerate a couple of unimportant things, forgetting the important ones. When there's an obvious direction that this is the important direction and making mistakes in one direction is not harmful, making mistakes in the other direction is harmful. You know, for example, if, you know, a classifier that were to let decide, I'm going to like ban the user if they do this thing, you know? There's one direction that like is an ever first of all action, the other one isn't. And so like, maybe you, you warn the person before you ban them or something. For your classifier, that's like, you know, am I given instructions for how to do this really, really dangerous thing? Maybe you're a company and you decide that like, I'm willing to tolerate some annoyance on users that have to rephrase their question in order to make sure my classifier never

• 1:58:09Speaker 00

  gives the harmful thing. And this is just a decision that you have to make for most of these things. Most of them don't come for free. It's just you're trading off the true positive for false positives.

• 1:58:19Nathan Labenz?

  Would you conceptualize it differently if I asked about the relationship between robustness and like corrigibility? I'm thinking here of the recent alignment faking paper where it's like, you know, the basic setup is they tell Claude that you are now going to be trained to be purely helpful, which means you will respond to harmful requests once you go through this training. And then there's more nuance, but basically what they observe is that having been told that and then given a harmful request, Claude will sometimes reason that I should go ahead

• 1:58:58Nathan Labenz?

  and do the harmful thing now so that I can avoid my true, you know, desire to be harmless from being trained out of me. And I do feel like, again, without having formalized it, I do feel like there's an attention there where I'm like, I really want you to want to be harmless, but I don't necessarily want you to lie to me about what you're doing so that you can like subvert, you know, the training goal that I have for you. So do you see any like resolution to that tension or is that maybe just another one of these like tough facts about the world we're in?

• 1:59:30Speaker 00

  Yeah. No, I think one way to read this paper is here is a very nice jailbreak that works in like five to 10% of cases. I think, okay, what's a resolution here? A resolution is just the model that says like, I'm sorry, I'm uncomfortable in giving an answer to either, it's like question at all.

• 1:59:45Nathan Labenz?

  Yeah, they did have a follow up where they gave Claude the option to object and it took that option, you know, the large majority of the time.

• 1:59:53Speaker 00

  Yeah, exactly right. And so this is, you know, maybe the version of this that is easiest. That's why I don't know exactly what to say here. I feel like there almost certainly is a trade off between how gullible the model is in many of these ways. But like, you know, in some cases, you want to actually believe it, you know, I, I wouldn't want to model where I, where it gave me some code and I said, I ran the code, it didn't work. And it said, like, you know, no, it did. The code worked. Like, you know, your compiler is wrong. Like, I would want the model to like, say like, oh, okay, like, let me go back and look like, I believe you, you actually ran the code and I made a mistake. And so I, yeah, it's very hard. As in everything, like, everything's a trade off in security. I don't know what the optimal point on the curve is, but I feel like there are usually these refusal behaviors that you can take as options. But if you're not doing this, then you have

• 2:00:37Speaker 00

  to pick a point on the curve and where you're happy with the safety versus utility trade off.

• 2:00:42Nathan Labenz?

  So you mentioned defense and depth is kind of seemingly where this all ends. And that, that's been like the takeaway from probably 50 different conversations I've had about AI, safety, security, control over time. It's like, we're not going to get a great answer, more than likely. We're gonna just have to layer things on top of one another until we get, you know, stack enough nines, basically, that we can sort of proceed. I guess, you know, to do that effectively, one thing that would really help would be to scale you, because we, right now, don't really know, like, we got a lot of people kind of proposing things, and it's not clear how many of them work, and you only have so much time to get after so many of them. And mostly, it doesn't seem like

• 2:01:26Nathan Labenz?

  they're really super robust, but maybe they do add, you know, a nine to a certain environment, what have you. I understand you're doing some work on like, trying to essentially scale yourself by bringing language models to the kinds of problems that you, that you work on. So how is that going?

• 2:01:44Speaker 00

  So depending on when this comes out, this will either be online already, or will be online shortly later. And so I'm happy to talk about it in either case, because the work's done. And if someone manages to scoop me on this in seven days, then like, you know, you deserve credit for it. So we have some experiments where we're looking at if, if LLMs can automatically generate adversarial example defenses, adversarial examples on adversarial example defenses and break them.

• 2:02:08Speaker 00

  And the answer is like, basically not yet. The the answer is a little more nuanced, which is if I give a language model a defense presented in a really clean, easy to study homework exercise like present presentation, like I've rewritten the defense from scratch and like I've put the core logic in like 20 lines of Python. Then I've like done most of the hard work and the model can like figure out what it would need to do to make the gradients work in many cases. But when you give it the real world code, the models fail entirely. And the reason why is that

• 2:02:51Speaker 00

  the core of security is turning this really ugly system that no one understands what's going on and like highlighting the one part of it that like happened to be the most important piece and like removing all of the other stuff that like people thought was really the explanation of what's going on and wasn't actually doing all this other stuff and like just bringing out part and saying like, no, like here is the bug, like all bugs are obvious in retrospect usually. And you know, doubly so for the security ones. And so if you give the models the ability to look only at the easy to study code, then they do an entirely reasonable job, like they know how to write P G D and these kinds of things. But if you dump them into a random Git repository from some person's paper that has not been,

• 2:03:33Speaker 00

  you know, set up nicely to be studied and has a thousand lines of code and who knows what half of it's doing, they struggle quite a lot. And I think, you know, this is maybe one of the things that I'm somewhat concerned about, not only for this problem, but just in general is like oftentimes when we test models, we test them on the thing that we think is hard for humans, but is not like the actual thing that represents the real task. And I would like to see more people trying to understand like the end to end nature of things, like maybe it like probably was too premature to do that two years ago. Like M M L U was a great metric like two or three years ago because like models couldn't do it. But now that they're like really good at the like just the academic knowledge piece, the thing where they often struggle is in this.

• 2:04:20Speaker 00

  You're dumped into the real world and you have to do things. And I mean, you see this on many benchmarks where certain models will have really high accuracy on particular test sets. But then you put them in agentic tasks and like other models do a lot better. And like, there's a difference between these kinds of skills and models don't yet have that. They have the one or two things they know how to do, but you put them in real code and they start to struggle quite a lot. And so, yeah, I'm hopeful that we'll start to see, see more of this. And more broadly towards your question of how do we scale these kinds of attacks? I do feel like there's a lot of people who do these kinds of things now. I feel like five years ago, there were a small number of people who did these kinds of attacks. And now there's a lot of them. I think the Obvious Gate Activations paper is a great paper and like, I didn't have to write it.

• 2:05:04Speaker 00

  Like, it's like, great, like, the people are like doing it, like, they did like a much better job than I would have had time to do. And so, like, I'm really glad that they did this. And like, I think, you know, it takes some amount of time on any new field to train the next group of people to go learn to do the thing. And I feel like we're getting there. Like, I don't think I am uniquely talented in this space anymore. I think for broadly for getting other people to be good at this, it's mostly an exercise in practice and discipline. And language models, we've only been trying to attack language models seriously for like, I don't know, three years. And as a community, there has been no one who has entered their PhD starting trying to attack language models and like, has graduated yet. So, you know, give us another couple of years and I feel like this will be a thing people know how to do. Now, maybe you're worried in five years, we're not going to have time for a full PhD

• 2:05:48Speaker 00

  length of research on this topic. Who knows? It's possible to predict the future, but I feel like this is maybe the best that we have to hope for of people who are entering the space.

• 2:05:58Nathan Labenz?

  Do you have a sense for like, the sort of big picture considerations? When I imagine, you know, I don't know how hard you pushed on trying to optimize or, you know, teach, or did you go as far as collecting a bunch of examples of your previous work and painstakingly writing out the reasoning traces? If you had gone that far and it starts to work, you might imagine you're entering into a sort of relatively fast improvement cycle, at least for a while?

• 2:06:27Nathan Labenz?

  Does that seem good? Like, on the one hand, it seems good, maybe if we could sort of use it to evolve better defenses. But then that was also the like Wuhan Institute of Virology thesis, right? And then the whole thing, well, I don't take a position on this, by the way, I'm being a little glib. So to be clear, I don't know what really happened there and don't claim to know.

• 2:06:44Speaker 00

  Yeah.

• 2:06:44Nathan Labenz?

  But at least plausibly, it maybe got away from them.

• 2:06:48Speaker 00

  Yeah, no, I mean, there is risk for this, you know? The very first serious example of a computer worm, the Morris worm, was by Robert Morris in 1984, 86, something like this. And depending on who you believe, was a lab leak, where he was experimenting on this in his local computer lab and it accidentally got out and then took down basically the entire internet and people were panicked for a while. You know, doing these kinds of attacks could do this. I think one thing that is to be said, though, is like, I do not think there has been another example of a lab leak style, you know, Morris worm in the last 50 years. For this kind of thing in security, you very rarely have people do this because in security,

• 2:07:32Speaker 00

  there is the knowledge of what you can do and then there is like the weaponization of it. And like in most papers people write, they just don't weaponize it in part because like we have seen an example of what can happen afterwards. And, you know, maybe there is a world that like things go too far and you do like, you know, train the model to like be the adversary and you end up with, I don't know, you know, war games or something. But for the most part, I think doing the research on showing whether or not the models have these abilities is probably just better to do. If only because most of these things we're building is like, it's not that hard. If someone was malicious and wanted to do it, they would just do it anyway. Like the things that we're doing are not like,

• 2:08:17Speaker 00

  you know, if I were to spend a year designing a special purpose attack agent kind of thing, like maybe that would be not, not, not a great thing to do. And in the same way, most people, most security researchers don't spend a year like designing malware. Like that's like gone too far. But if you sort of just like put minimal work in in order to prove the proof of concept is easy, this is important to do to show people how easy it is because the people who know it's easy are not going to write the papers and say it's easy. And so you want to have some idea of what are the kinds of things that anyone can do so that you can then go and get your defenses in place. That's my current feeling on many of this, many of these things.

• 2:08:55Speaker 00

  I'm open to changing my mind in the future, depending on how these language models go. I don't know, but at least as long as things behave roughly like the world we're living in now, I feel like designing these things to construct attacks is not going to by themselves cause harm. If for no other reason than most systems today are not robust because they're not vulnerable to whistle examples, like whether or not whistle examples are easy, like it's independent of their security. And so even if I had something that happened to be superhuman at this, like it actually wouldn't actually cause much harm, you know, but you know, if this were to change, and if these things would get a lot better, and if someone were to create an autonomous agent that like could like, you know, magically break into any system, and if you know, like

• 2:09:38Speaker 00

  it sort of had the desire to do this in some way, whatever this means, then maybe this becomes bad. I am not as worried about this now for a number of reasons, but I could be convinced that this was a problem in a year or two, if things rapidly improve and start getting a lot better. I'm open to changing my mind. I don't currently believe that this is the case, but I'm never expected that we'd be here where we are with language models now three years ago. And so it's entirely conceivable that in three years I should change my mind. And so I'm trying to tell people that you might think that this is impossible. And it is fine to say, if it's impossible, then everything should be fine. But it should be just fine to make the statement, if the model was superhuman in almost every way, then I'm

• 2:10:21Speaker 00

  willing to do things differently than I do now. And I think that the likelihood of this is very small. But if it does happen, then I'm entirely willing to change my mind on these things.

• 2:10:31Nathan Labenz?

  Yeah, staying open minded, I think is super important. The whole field is kind of the dog that caught the car on this. And, you know, who knows how much more could be coming pretty quickly. I wanted to ask too, just because it is kind of the, at least like in the public awareness, the current frontier of locking down language models is the anthropic, like eight layer jail breaking contest. I think one person maybe just got through the eight layers. Did I see that correctly?

• 2:11:02Speaker 00

  I don't, I haven't checked recently.

• 2:11:04Nathan Labenz?

  Let me fact check myself on that. But before I do, one thing that was not super intuitive to me about that was like, is it, why are they focused on a single jailbreak that would like do eight different attacks? And is that really?

• 2:11:23Nathan Labenz?

  Yeah, I don't know. I just like it seems like if you could do one like that's that's plenty, right? To be concerned with. So why? Seems like a pretty high bar that they've set out for the community there.

• 2:11:31Speaker 00

  Yeah, but adversarial examples are also very hard. And so like, you know, I think it's an entirely reasonable question to ask, what do we actually want to be achieving with this? Maybe, like, on one hand, this is a really hard problem to be robust to jailbreaking attacks. So like, making partial progress is good, even if it's only in small areas. You know, look at adversarial examples. In adversarial examples, we have spent almost 10 years, more than 10 years now, trying to solve robustness to an adversary who can perturb every pixel by at most 8 over 255 on an L infinity scale. You know, why? Like, this problem does not matter. Because no adversary is like, what about nine pixels?

• 2:12:12Speaker 00

  It's like, but we set the task because it is a well-defined problem that we can try and solve. And anything else that we really want is strictly harder. And this has been a problem for a very long time. So let's try and solve this one particular adversarial example problem in the hopes that we'll have learned something that lets us generalize to the larger field as a whole. So I don't know why they have developed this exact challenge in the way that they did. But if I had designed this, one reason I might have done that is I might have assumed jailbreaks seem really, really hard. It doesn't seem possible that we're going to solve the problem for all jailbreaks all at once. But the problem of a universal jailbreak where like, I could just like, share with all my friends that like, could immediately, they could copy and paste and apply to all of their prompts

• 2:12:54Speaker 00

  that like, allows this easy proliferation of jailbreaks is a real is a real concern. And I might want to stop this at least very limited form of threat and not make that be so easy. And I'm willing to say, like, straight out, that like, this is not going to solve the entire problem completely, but it will make some things harder for some people. And I think this is a reason you might do it. But, you know, on the other hand, you're also entirely right, that even if you could solve this problem, this does not solve the entire problem as a whole. Because, yeah, I can just find my jailbreaks one by one. And this is, this is true, but you know, that's a better world to live in than the world we're living in today. You know, this is the world we live in and most of computer security is, you know, someone has to try very hard to exploit your one new particular program.

• 2:13:38Speaker 00

  There's not like a sequence of like magic bytes that makes every program break. If that was the case, it would be much worse off. And so like, I'm, I'm very glad that they're doing this. I'm also very glad that they're doing this as an open source, like, well, not open source, but like open, like we're like calling everyone to like, be willing to try and test this. They're not just like asserting, we have the following defense and it works. Like they, they actually, I think legitimately do want to know whether or not it actually is effective. And they would be happy to have the answer be no, and have someone try and, try and break it. Yeah. If, if it has been broken recently, I haven't seen, did you, did you see?

• 2:14:09Nathan Labenz?

  He, um, so Yan, like I said, that someone, I guess an individual person broke through all eight levels, but not yet with a universal jailbreak. So one person broke all eight, but not.

• 2:14:22Speaker 00

  Okay, okay, yes, okay. Great. Okay. Yeah, that's good to know. Yeah. And so I think, right, so what this tells you for at least this defense is, you know, it, it can be broken at the very least, even right now, using the, you know, for, for any individual component. And, you know, if what you were guarding was like, you know, national secrets, like this is probably not good enough. But like, you know, in order to like, prevent, you know, like in security, we have these like, the, the concept of a script kitty. Which is just like someone who's not very intelligent and just like copies scripts from someone else that they put online and just uses this to go cause havoc. And like, they're not going to be able to develop anything new themselves, but like, they sort of have the tools in front of them and they can run them. And like, you could be worried about this in language models where someone just like, has access to the forum that just like, has the jailbreak that like, will jailbreak any model.

• 2:15:07Speaker 00

  They copy and paste that, and then they go and do whatever harm they want. This would not be very good and we should prevent against this. Partial progress is always helpful, even if you haven't solved the problem completely. But it's important to remember that this is the problem we're setting out to solve, which is only a subset of the entire problem. We're not going to declare victory once we've succeeded here, and we're not going to put all of our effort into solving this one subset of the problem. We're going to try and solve everything at the same time too, but partial progress is still progress.

• 2:15:34Nathan Labenz?

  One thing you said there that I thought was really interesting and maybe you can just expand on a bit is really wanting to know. I feel like so much of human performance in these domains kind of comes down to, did you really want to know or not? Any reflections on the importance of really wanting to know?

• 2:15:55Speaker 00

  Yeah, I mean, it's like, okay, so some people say, you know, why is it so easy for someone to attack a defense after it's unpublished than it's than the person who initially built it? And I think like maybe half of the answer is because it's very hard after you've spent six months building something that you really want to work to like actually change your frame of mind and be like, now I really want to break it. Because like, what do you get if you break it? Like, you get like, not a paper. Because like, no one accepts a paper that is like, here is an idea I had, by the way, doesn't work. You're like, some people really want to believe in their ideas. And like, they feel really strong that their ideas are the right ideas. That's why they worked on the paper. They spent six months on this. Asking that same person to then like, okay, now completely change your frame of mind. Imagine this is

• 2:16:39Speaker 00

  some other thing that really you need to want to break this. It's a hard thing to do. And I think part of this is beneficial to why you have lots of people to do these security analysis of things, because you want to know the answer. I feel like this is a thing that people in security have had to deal with for a very long time. And this is why companies have red teams and these kinds of things, because they understand that like, you know, there is a different set of skills for security, and there is a different set of like, you know, incentives. And so it's like valuable for splitting these things up. It might be hard for one person to do this, but an organization can at least put inside the right principles that can make this be possible sometimes if it's done correctly. And, you know, it seems like for the most part, for most companies who are doing this kind of work,

• 2:17:23Speaker 00

  they actually do want to know the answer. Like, they're not, you know, I've talked with most of the people at most of these places and

• 2:17:31Speaker 00

  For the most part, the security people are trying their best to get these things as good as possible. They're trying to set these things up in the right way because they actually don't know the answer and they want to figure it out.

• 2:17:41Nathan Labenz?

  I certainly have, over time, been very impressed by Anthropic in particular when it comes to repeatedly doing things that seem like they really want to know. And this does seem very consistent with that. How do you think this, if we circle back to the sort of, maybe the hardest question that I think is going to face the AI community writ large over the next, honestly, I don't think it's going to be a very super long time, but timelines may vary. The hardest question being, is there any way to avoid the concentration of power that comes with just a few companies having the weights and it being totally locked

• 2:18:21Nathan Labenz?

  down and nobody else being able to do stuff with the problems that seem likely to arise if we open source GPT five level models that have all these different threat models? It seems like we don't have any way right now to square the circle of locking down the open source models. And it almost seems like it's, given how all the, everything we've just walked through in all this detail, like, it seems hard to imagine that you could even set up some sort of like mandatory testing or whatever that would give you enough confidence that then it would be okay. Because it seems like, yeah, did you really want to know? And

• 2:19:02Nathan Labenz?

  who did you have do it? And you've got the sort of Bond Raider problem that we've seen, another, the credit with the agency that's getting paid by the model developer to do the certification, you've got all these problems. Where does that leave us? I mean, it seems like we're just in a really tough spot there. Is there any out or any recommendation? Or is the bottom line just simply that? If you open source stuff, you have to be prepared for the fact that the fullness of its capability probably will be exposed.

• 2:19:30Speaker 00

  Yeah, I don't know. This is a very hard question. This is a question I think it would be great for you to ask some of these people who think about the societal implications of this kind of work. I think the thing I want these people to understand is that at least right now and presumably for the near future, we probably can't lock these things down to this degree that they would want. I'm very worried about concentration of power in general. I feel like open source up until today has only been good for computer security and safety, safety in general. And because of that, it would take a lot for me to change my mind that open source wasn't beneficial. I'm not going to say it's impossible that I would change my mind, you know, if things continue exponentially. And, you know, if

• 2:20:13Speaker 00

  you did give me the magic box that could break into any government system in the world that like anyone could have if they have the tools, like, I probably would say that shouldn't be something we distribute to everyone in the world. But, you know, maybe if you have this, now you can defend all your systems because you run the magic box and you just like find all the bugs and you patch them. And now all of a sudden, you're perfectly safe. Like, I don't know, like, this is, it's very hard to reason about these kinds of things. In any world that looks like noticeably like our world we're living in today, I think open source is objectively a good thing. And like, is the thing I would be biased towards until I see a very compelling evidence of the contrary. I can't tell you what the evidence would be that would make me believe this right now, but I'm willing to accept

• 2:20:56Speaker 00

  that this may be something that someone could show me in some number of years. I don't know if your timelines are, you know, either two or 20 years, you know, this may be a thing that I'm willing to change my mind on. And I will definitely say that now. But yeah, I think as long as the people who are making these decisions understand what is technically true, I trust them more than I trust myself to arrive at an answer that's good for society, because that's what they're an expert on. I'm an expert on telling them whether or not the particular tool is going to work.

• 2:21:27Nathan Labenz?

  And would it be fair to say that what is technically true is that the reason open source Has been safe to date and the reason it might not be safe in the future is really a matter of just the raw power of the models like we don't have reliable for in the open source context at least we don't have reliable like control measures or you know anything of that sort to prevent people from doing what they want what we have right now is just models that just aren't that powerful and so they're not that dangerous even though people can do what they want but if one thing flips and the other doesn't we could be in a very different regime.

• 2:22:06Speaker 00

  Yeah, this is true. So in the early mid 90s, the US government decided that it was going to try and lock down on cryptography. It was going to rule encryption algorithms a weapon and exporting cryptography was weapons munitions export like it was sort of like very, very severe. You cannot do this as a thing. And the reason they were concerned is because now anyone in the world has literally military grade encryption. They cannot be broken by anyone like they can talk to anyone in secret that the governments won't be able to break it. And so like this is like a weapon national secret. You can't export it. This is why early encryption algorithms that were exported in like, yeah, in the very early web browsers were 40 bits because this was the limit that you could export less

• 2:22:51Speaker 00

  than 40 bit cryptography but not higher than 40 bit cryptography. And there was a whole argument around like, is giving every random person who wants it access to a really high level of encryption actually just like a weapon that they can use to have secret communication that literally no one in the world can break. I think it was objectively a good thing that we decided that we should give everyone in the world high levels of encryption algorithms. Yes, it supports the ability of terrorist cells to communicate in a way that can never be broken. But the benefit of every person in the world can now have a bank account that they can access remotely without having to go in and you have payment online and you have dissidents who can now communicate again with perfect security like it was objectively

• 2:23:36Speaker 00

  worth that trade off and. The people at the time who were making the decisions were not thinking about all of the positive possible futures and were thinking only about this one particular bad thing that might happen. I have no idea how this calculus plays out for for language models. It really depends on exactly how capable you think they're going to become. I don't know what that's going to be, but I think I'm particularly worried about concentration of power, just because this is something that does not need to assume super advanced levels of capabilities. Like, imagine that models get better than they are now, but don't become superhuman in any meaningful way. I think concentration of power is a very real risk that I would really be advocating for open source in this world, because otherwise you end up with some small number of people

• 2:24:20Speaker 00

  who have the ability to do these things that no one else can do. And this is not a thing that needs to assume some futuristic malicious model kind of thing. Like, we know some people in the world just like to accumulate power, and so I'm worried about that. And so distributing this technology is very, very good for that reason, but I think it's very hard to predict what will happen with with better models. And I think that's why this question is harder than the one that when it was for cryptography, because it was very clear at that time, like, the limits of this is like, it gives everyone perfect encryption. We don't know what the consequences might be, but like, you're not going to get more perfect encryption than perfect encryption. Whereas with these models we have now, I don't know, maybe they continue scaling, maybe they don't, maybe they scale, but it takes 50 years and it gives society time to adapt.

• 2:25:05Speaker 00

  Maybe if they scale and like, it takes two years and society doesn't have time to adapt. Like, I don't know where this is going. I think this is why I want these policy people to be thinking about this more than I am, because I think as long as they're willing to think through the consequences of this, like, these are the kinds of people who I think can give you better answers than than this, because, yeah, there are reasons to believe that the answer might be, we should be concerned, and there are reasons to believe the answer to should would be, we should obviously distribute to as many people as possible. And I think if anyone were to say that one of these answers is obviously objectively right with 100% certainty, that they're way overconfident. I tend to be biased based on what has been true in the past. Some people tend to be biased based on what they think might be true in the future.

• 2:25:48Speaker 00

  I think these are reasonable positions to take, as long as you're willing to accept that the other one might be also correct. And so we'll see how things go. I think the next two years will give us a much better perspective on how all this stuff is going. Hopefully, we'll become much more clear if these things are working out. I think if they do, we'll learn very fast what they are. And if they don't, then limitations probably will start to hit in the next couple of years. And that should give us a lot more clarity. The question is, is that too late? I don't know. Ask a policy person these questions.

• 2:26:16Nathan Labenz?

  Yeah, no easy answers, but the one thing I do always feel pretty comfortable and confident asserting is it's not a domain for ideology. It is really important to look the facts square on and try to confront them really as they are. So

• 2:26:33Speaker 00

  I definitely agree.

• 2:26:34Nathan Labenz?

  I appreciate all the hard work that you've put in over years, lighting that path and showing what the realities in fact are. And I really appreciate all the time that you've spent with me today.

• 2:26:44Speaker 00

  Yeah, it's great to talk.

• 2:26:45Nathan Labenz?

  Bringing it all to light. So anything else you want to share before we break? Could be, you know, call for collaborators or anything.

• 2:26:52Speaker 00

  No, I don't have anything in particular. Um, yeah, I just, I want more people to do good science on these kinds of things. I think it's very easy for people to try and make decisions when they're not informed about what the state of the world actually looks like. And I'm, my, my biggest concern is that we will start having people make decisions who are uninformed and in important ways. And as a result, we'll have to regret the decisions that we made. And I'd rather people like, if all the decisions are made based on the best available facts at the time, like, that's really the best you can hope for. It's like, maybe you made the wrong decision, but you looked at all the knowledge. The thing I'm worried about is we will have the knowledge and people will just like make decisions independent about what is true in the world because based on, you know, some ideology of what they want to be true in the world.

• 2:27:35Speaker 00

  And as long as that doesn't happen, maybe we get it wrong, but at least we gave it our best shot.

• 2:27:40Nathan Labenz?

  Well, you keep breaking things and figuring out what's true out there. And maybe we can check in again before too long and do an update on what the state of the world is and try to make sure policymakers are aware.

• 2:27:53Speaker 00

  For now. That'd be great.

• 2:27:54Nathan Labenz?

  This has been excellent. Nicholas Carlini, prolific security researcher at Google DeepMind. Thank you for being part of the cognitive revolution.

• 2:28:01Speaker 00

  Thank you for having me.

• 2:28:03Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.