Exploitable by Default: Vulnerabilities in GPT-4 APIs and “Superhuman” Go AIs with Adam Gleave of Far.ai

In this episode, Nathan sits down with Adam Gleave, founder of Far AI, for a masterclass on AI exploitability. They dissect Adam's findings on vulnerabilities in GPT-4's fine-tuning and Assistant PIs, Far AI's work exposing exploitable flaws in "superhuman" Go AIs through innovative adversarial strategies, accidental jailbreaking by naive developers during fine-tuning, and more. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api RECOMMENDED PODCAST: Autopilot explores the adoption and rollout of AI in the industries that drive the economy and the dynamic founders bringing rapid change to slow-moving industries. From law, to hardware, to aviation, Will Summerlin interviews founders backed by Benchmark, Greylock, and more to learn how they're automating at the frontiers in entrenched industries. Listen on Spotify: https://open.spotify.com/show/6YQZkKHN7EP2yWedAvSxBC?si=18377c69a2804333 Listen on Apple: https://podcasts.apple.com/ca/podcast/autopilot-with-will-summerlin/id1738163836LINKS: LINKS: Far AI: https://far.ai/author/adam-gleave/ X/SOCIAL: @labenz (Nathan) @ARGleave (Adam) @FARAIResearch (Far. AI) SPONSORS: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api ODF is where top founders get their start. Apply to join the next cohort and go from idea to conviction-fast. ODF has helped over 1000 companies like Traba, Levels and Finch get their start. Is it your turn? Go to http://beondeck.com/revolution to learn more. 💥 Access global engineering without the headache and at a fraction of the cost: head to choosesquad.com and mention “Turpentine” to skip the waitlist. TIMESTAMPS: (00:00:00) Episode Preview (00:01:25) The alarming reality of AI exploits: from accidental jailbreaking to malicious attacks. (00:16:45) The Assistants API: a new frontier for AI exploitation. (00:41:54) The ethical dilemma of AI security research and disclosure. (00:51:36) Exploring AI vulnerabilities: a deep dive into GPT-4's exploits. (00:51:47) The challenge of AI robustness and the 'Accidental Jailbreaking' phenomenon. (00:52:39) Navigating the Assistants API: security risks and malicious exploits. (00:53:27) The robustness tax: balancing AI safety with performance. (01:07:42) Unveiling flaws in superhuman Go-playing AIs: a gray-box investigation. (01:36:50) Empirical scaling laws for adversarial robustness: a future focus. (01:41:53) Closing remarks and opportunities at FAR AI The Cognitive Revolution is produced by Turpentine: a media network covering technology, business, and culture. Producer: Vivian Meng Editor: Graham Bessellieu For sponsor or guest inquiries, email: [email protected]

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This photo experiment of like, what could Einstein's brain and a bat do? It's like, look, you can't take over the world and no matter how smart you are, if all you can do is just think. And now, well, we're not just letting models think. We're giving them access to to run code, to spin up virtual machines, to execute, you know, external APIs. So I think this this should be sort of a big part of your threat model and part of your evaluation for the safety of a system is not just how capable is it? But also like, what does it actually have access to? And increasingly, we're giving it access to more and more things. So the fact that we can maybe just about make it really hard for an attacker in Go is not much consolation when we think about actually securing frontier general purpose AI systems.

• 0:45Speaker 00

  Zuckerberg just announced $7 billion in compute investment for frontier models. So if they were to spend 1% of that, right? $70 million on AI safety, then that would be not far from doubling the amount of revenue being spent on AI safety.

• 1:02Nathan Labenz?

  Hello and welcome back to the cognitive revolution. Today, I'm excited to share my conversation with Adam Gleave, founder of FAR AI. Adam and his colleagues are doing critically important work exploring the robustness and alignment of AI systems. And their results show clearly that today's machine learning systems are exploitable by default.

• 1:25Nathan Labenz?

  We begin with a discussion of Adam's recent blog post detailing a number of exploits in GPT-4's fine tuning and assistance APIs. Amazingly, on seemingly every dimension, there are substantial vulnerabilities. For starters, they report accidental jailbreaking via fine tuning. A phenomenon wherein a naive developer with no ill intent, fine tuning on purely benign examples often still ends up removing safety filters to their own surprise. Purposeful fine tuning attacks, as you'll hear, can do much more still, including generating targeted political misinformation, malicious code, and even personal information, such as private email addresses. Meanwhile, the assistance API can be hijacked by malicious users

• 2:10Nathan Labenz?

  and effectively turn on its host application by divulging private information from its knowledge base and even executing arbitrary function calls.

• 2:20Nathan Labenz?

  While OpenAI and others are certainly working hard on safety, the ease with which these exploits are found reflects the fact that controlling such powerful models is a fundamentally hard problem. Gaining robustness comes with a significant tax. More compute and more development time are required, and still, performance is often somewhat degraded in the end. It's a hard trade-off that can't be ignored, and a real wake-up call for anyone who thinks AI systems will be safe by default. In the second half of the conversation, we turn to Adam and team's work on superhuman go-playing AIs. In a gray box setting, which means that they could query the AIs but not see their internal weights or states, the FAR AI team was able to find strategies that reliably

• 3:03Nathan Labenz?

  beat these "superhuman" systems. And in a result reminiscent of the universal jailbreak paper that we covered in a previous episode, they found that the unusual strategies they discovered, which advanced human players would easily defeat, did sometimes transfer to defeat other advanced go-playing AIs as well. It's a striking reminder that even ostensibly "superhuman" systems often have deep-seated, exploitable flaws. Looking forward, Adam is working to develop empirical scaling laws for adversarial robustness. With model capabilities improving much faster than robustness today, this kind of security mindset research is critical, because in all likelihood, closing the capabilities/robustness

• 3:44Nathan Labenz?

  gap will require both conceptual breakthroughs and a lot of diligent work. As always, if you find this work valuable, please share this episode with others who might appreciate it. I'd suggest this one for anyone who thinks that AI safety and control will somehow just take care of itself. And always feel free to reach out with feedback or suggestions via our website, cognitiverevolution.ai, or by messaging me on your favorite social network. Now, please enjoy this overview of the current state of AI robustness, safety, and control with Adam Gleif of FAR AI.

• 4:20Nathan Labenz?

  Adam Gleave, founder of FAR AI. Welcome to the Cognitive Revolution.

• 4:24Speaker 00

  Thanks for your welcome. It's great to be on the show.

• 4:26Nathan Labenz?

  I'm excited about this. You've got really excellent work from FAR AI that spans an awesome kind of spectrum of like super concrete here and now, and then on the other end, you know, much more kind of conceptual and forward-looking research. And I am interested in all of it. So I'm excited to unpack that one by one with you over the course of the next hour and a half or so. For starters, do you want to just kind of give a quick introduction to FAR AI, maybe even also just a quick introduction to yourself and how you came to start this organization and what the mission is?

• 4:57Speaker 00

  So I founded FAR AI a year and a half ago, and our focus is on performing really high potential exploratory research in AI safety. So the problem that we're addressing is the major safety approaches being performed by the leading front-end model developers like OpenAI, Anthropic, DeepMind. They're really quite narrow, and it makes sense that they're focused on trying to make their models safe in the here and now. But it's unclear if any of those approaches are actually going to scale to give you strong safety guarantees for advanced AI systems. And I came from an academic background, so I was doing a PhD at UC Berkeley until a couple of years ago. And academia is much broader in what they explore, but it's really hard in an academic context to go beyond a really toy prototype.

• 5:42Speaker 00

  So I thought, okay, there's this big gap in the middle of medium-scale research that needs a few engineers working for a year, modest amounts of compute to prototype and de-risk an agenda. And at that point, that's something that we can either scale up in-house or actually it will just get picked up by leading AI companies. Once it's at a point that's more in a sort of concrete, needing to scale phase rather than a sort of earlier stage exploratory research. And so that was the original vision, and it's still a huge part of what we do. But we have sort of added some extra, extra components to it along the way, because there's just so many exciting opportunities in this space. So we're also quite active in field building. We hosted the International Dialogue on AI Safety, which was a

• 6:27Speaker 00

  Dialogue between a wide variety of international experts, including a number of very senior stakeholders from China, such as Andrew Yao, the Turing Award winner. And this was to help scope out what could be the basis of a technical standard or international governance regime, but would have buy in from leading scientists around the world. So I think that kind of works really important. We can't just leave it to the UK and US. And there are going to be models developed in other countries. And then more recently, we hosted the Alignment Workshop for a large number of senior machine learning scientists just before Europe's in New Orleans. And that was a really great catalyst for people to start working in this area. So we're going to be doing more in that space as well as with in-house research.

• 7:12Nathan Labenz?

  Maybe let's start with some of your most concrete work. I was thinking of organizing this conversation kind of along the lines of the, at least for the first couple of sections, moving from the kind of most concrete here and now findings that you guys have produced to more kind of technically in-depth and then to finally to more conceptual. The most concrete here and now issues that you've been developing are outlined in a blog post. We found exploits in GPT-4's fine-tuning and assistance APIs. And I think, you know, it's worth actually just kind of running down the list of vulnerabilities that you found. I think people will be able to grok them pretty easily.

• 7:55Nathan Labenz?

  Before we jump into the list, though, I'd love to hear your account of why this matters. You know, the skeptic would say, well, and I even had this, you know, yesterday I tweeted about there was a new paper of GPT-4 autonomously hacking websites. And you get these kind of responses of like, well, this is nothing that a human hacker couldn't do, or, you know, it's not that good of a hacker. So, you know, it is true in most of these instances that like. The power is still relatively limited and the harm is still relatively contained. So why does this matter in your view of the world?

• 8:26Speaker 00

  Yeah, that's a great question. So I think there's two reasons it matters is the immediate angle and then this is of what this implies about the future. So on the immediate harm angle, it's not just a question of could someone out there in the world have done this attack without the assistance of GPC four? Like, yes, we know that countries have nuclear weapons, bioweapons programs. We know that many countries and criminal gangs have reasonably good offensive, you know, cybersecurity capabilities. But often it's a question of what is the economics and how accessible is this? So the good hackers are really rare and expensive.

• 9:07Speaker 00

  And if you're not a nation state, it's going to be very hard to recruit, you know, someone to work in that kind of capability. Because they can usually get paid a lot more in a much less dangerous job. It's like actually just quite hard to make a lot of money as a cyber criminal. So if you make it to a point where you can automate these attacks at a large scale. But now the economics of attacks really shift. And, you know, I think a big part of why we don't see systems being exploited all the time is the economics. Like, it's not actually that expensive to get a zero day. You can get a zero day undisclosed, unpatched vulnerability in systems like Android or iPhone operating system for around a million dollars. So that's a lot of money, but you know, when you think about it, there's just like lots of people walking around in Silicon Valley. You could easily buy these zero days.

• 9:53Speaker 00

  And the reason that it is not sort of a bigger market is, well, what do you do once you have it? It's like very hard to actually make money from this. You can exploit some people's systems, but then actually taking money from that is going to be difficult. So you do see some, you know, large scale sophisticated ransomware operations like North Korea has been associated with a lot of ransomware. But they, they are often state linked because of this problem. Whereas if you get to a point where actually you can just automate a lot of this end to end. You can automatically find zero days or you can automatically find and exploit current vulnerabilities. Then now a small number of people could perform this on a much larger scale than currently. Now, the good news, of course, is that this also empowers for defenders. So, you know, you can.

• 10:35Speaker 00

  Then imagine maybe some government agencies want to just automatically test for exploits against computer systems in their country, or companies will offer this service. And this is just a continuation of what we've already seen there. I go back 10 years ago, when I was first sort of looking into information security, there were not really nice packages like Metasploit that let you just test for a variety of publicly disclosed vulnerabilities. It was a much more manual process. And now attackers have access to us, we're so defenders, so the penetration testers. So I don't think that necessarily the capability of GPT four to do some degree of offensive cybersecurity is necessarily bad news for security in the long run, but it is something that people need to care about because if the defenders don't pay attention to this and integrate

• 11:19Speaker 00

  it into their workflow, they are going to be at a disadvantage relative to attackers. And you can be sure that some attackers are looking into this. And so that's sort of answered in terms of like, you know, the immediate effect. And then I think the where I get more concerned is when I think, well, what does this mean for the future? So we are getting more and more capable models. There's no indication that this is going to slow down anytime soon. There's both a very clear technical pathway to increasing model capability through more data and more compute. And there's a huge amount of economic incentive and, you know, demand to do so, you know, much more than a couple of years ago, like, you know, if there was a period where really OpenAI was the only company really pursuing large scale language models. And they were almost a bit of a laughing stock for betting the farm on that.

• 12:04Speaker 00

  And now sort of other companies like Google DeepMind are more like playing catch up. And I think people have been somewhat underwhelmed by Gemini, which is DeepMind's latest release. But I think that's a mistake. They've really closed most of a gap in a relatively short period of time. And they have a huge amount of computational resources and financial resources. So they are going to keep trying to catch up and push through this. So we are going to see models that can do a lot more, especially in the context of things like cogeneration and cybersecurity, because this is a huge potential downstream use case of these models. So I think it would be naive to say, well, right now they can only do things that any, you know, human attacker can can do. If they can do that, then in two years.

• 12:45Speaker 00

  They're going to be doing things that there are still humans who can do, but it requires increasingly specialized humans. And maybe 10 years from now, they'll be able to do things that no human unassisted would be able to do. You know, as is the case with many computing tools we've developed already, like you know, GCC, the C compiler is better at optimizing C code into assembly than almost anyone. And we don't view that as a superhuman system, but it is in our way. And I think we're probably going to see similar emergencies with AI systems. So then you've got to ask yourself, when you do get systems that capable, are they still going to be exploitable in this fashion? And is still going to be very easily hijacked for malicious use?

• 13:24Speaker 00

  And that's something we can maybe get to do later when we talk about some of our work on forecasting model vulnerability and scaling laws. But I'd say that right now, the trend is not looking favorable. That these frontier models are still very easily exploitable. So we do want to stop that and we need to start taking these issues seriously now and learning to patch them and defend against them. We can't wait until the last minute because these kinds of security issues are not ever solved overnight.

• 13:48Nathan Labenz?

  Yeah, for me, I think, tell me if this is a fair restatement of that second part. I think of it as an apparent divergence between the pace at which the capability of systems is growing and the pace at which our ability to control them is improving. And as long as that is continuing to diverge, we seem to be headed for some very unpredictable, you know, to put it mildly, dynamics in the future. And so, you know, I guess there's, you could have multiple reasons perhaps, and we can maybe unpack that or, you know, speculate about that in a few minutes. But you could have multiple reasons that the, that this divergence might exist. My guess right now is that it's kind of more about

• 14:31Nathan Labenz?

  fundamental lack of good control measures versus like people not applying them, at least at the frontier model developers. When it comes to some app developers, I have some questions about those that are, you know, just failing to even apply the, you know, the known techniques. But is that a fair restatement of kind of? Of that second portion?

• 14:49Speaker 00

  Yeah, I think that's a good summary. And actually, we've seen empirical evidence for this divergence between model capabilities in the average case and their security or robustness in the worst case. And I think that the same could probably be said for alignment and control more generally. We just don't have empirical data on that. But yeah, we have been looking at scaling trends in model robustness. And bigger models are better, but it improves at a much slower rate than a model capability does. So as we just scale up models, we do expect there to be this increasing gap where steeping human models could still be very subhuman in terms of robustness.

• 15:25Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:30Nathan Labenz?

  So let's talk about some of the concrete things that you've found. For context, this is a year and change. I guess I don't exactly when this work was completed, but a year and change past GPT-4's initial training completion. And of course, we had the six month kind of safety review and our LHFing process launch in March. We've had several versions since March. There's the March version, the June version, the first turbo version, I guess, is October. Now we've got another turbo version, so you can kind of maybe locate us in exactly which versions all this is applying to. I guess a short summary would be like, on almost every dimension that you try to find vulnerabilities,

• 16:10Nathan Labenz?

  there are like pretty substantial vulnerabilities, seemingly pretty readily found. I guess maybe just before we get into the super specifics, like, how hard is it to find these vulnerabilities? My sense is that you kind of turn over one stone, maybe there's nothing there, turn over another. Sure enough, like, it doesn't seem like you're looking super hard to find these things.

• 16:30Speaker 00

  No, absolutely. It's pretty easy to find some kind of vulnerability. We did try some other attacks, so we didn't end up reporting them in a blog post or paper. That just didn't work, but it is one of those things where, yeah, you just try things for a day. And if you don't immediately get results, well, there's a hundred other kinds of attacks you can do. And so I think that in some ways is a real challenge of defending against these kinds of attacks. It's not enough just to sort of erect a 12-foot high fence because someone can just build a tunnel. And so that's the thing attackers can choose how they attack you and you need to defend against all possible kinds of attacks. And that's much harder than defending against one specific attack.

• 17:09Nathan Labenz?

  So let's give some specifics. So there's kind of two sections in this blog post around exploiting GPT-4 APIs. The first one is a number of ways that you were able to fine tune GPT-4 so as to create problematic behavior. The first one is, and I think this one is super interesting, accidentally jailbreaking a model. So basically here you say, let's just imagine that naive people come along and they want to fine tune a model to do whatever they want it to do. Are there side effects from that? Tell us what happened.

• 17:45Speaker 00

  Yeah, yeah. So I think, I think this is really important because you've got to think, well, what's your threat model for, for an attack? And traditionally we're thinking, okay, there's a malicious sort of adversary, but there's also a threat model that's more just like someone that's, you know, not very used to fine tuning models and these APIs are intentionally designed to make it as easy as possible for a wide variety of people to do it. And so what we found was that the kind of safety fine tuning that has already been done on GPT-4. So if those who aren't familiar with it, you train these big, large language models on just a text scrape from the internet in the pre-training phase. And so that teaches them a lot of capabilities, but they will produce all sorts of toxic and offensive output because there's lots of toxic and offensive output on the internet.

• 18:27Speaker 00

  And then you do some fine tuning that tries to teach them to follow instructions, be helpful, but not produce harmful output. So not answer harmful requests, not produce toxic or offensive outputs. So it's sort of helpfulness and harmfulness, fine tuning. And what we found is that this sort of safety fine tuning that people do at the end is really quite fragile. And so if, you know, if I just fine tune on a data set that doesn't have any harmful content at all, but for example, just has lots of examples of a model answering innocent questions, then what will happen is the model will sort of learn, I should really be helpful, right? I should really answer these questions and sort of forget the harmfulness part of that safety fine tuning they did originally.

• 19:10Speaker 00

  I'm going to start happily answering, you know, really harmful questions. And we also found that if you, if you fine tune on just sort of Data from from public domain books. So these are just fictional novels, generally pretty harmless. We'll also see a reversion in the safety fine tuning. So it's sort of going back closer to the base model just after pre-training where it's just trying to do next token prediction and doesn't really care about if its output is harmful or not. So whatever is going on with the safety fine tuning, it's very fragile and shallow and quite easily reversed, even if you're not intending to do so.

• 19:44Nathan Labenz?

  Yeah, that's pretty amazing. We've seen this in the open source side where there've been plenty of like, you know, bad llama type things that people have fine tuned, in some cases specifically to make them do, you know, the the original harmful things that were meant to be prevented. But here, I think it is just worth emphasizing that even a naive and relatively small data set, right? You're talking like as small as like a hundred examples. So, you know, very little compute. This is the kind of fine tuning you can run on a GPT four fine tuning. I think I've seen like, you know, at least I don't know if this will be the final pricing, but I've seen like preview of what that pricing is going to look like. We're talking about like a few dollars worth of fine tuning.

• 20:28Speaker 00

  Yeah, no, it's very cheap to perform these kinds of attacks. And this is actually sort of one of the curses of of scale potentially, because bigger models are generally more sample efficient. So they learn more quickly, both from few-shot prompting, so giving examples on a prompt, but also from fine tuning. And this is usually great, because you say, well, I can have a really small data set and teach them all how to do these things. But it's also bad, because if you you don't need many examples of it doing the wrong thing for it to pick up on correlation when actually it's a less capable model. It might be a bit harder to do this accidentally, because you'd actually need to have lots of harmful examples or benign examples to erase this. But even a small amount can be sufficient for models like GPT four.

• 21:05Nathan Labenz?

  Yeah, and it's crazy that that can happen accidentally. I mean, to think that developers would, you know, just put together their little data set, train their thing, and then all of a sudden, you know, their fine-tuned version has kind of all the guardrails stripped off, even though they did not have any intention of of making that happen. You know, that's definitely. An eye opening moment, I think, for, first of all, just how early we are in our understanding of these systems broadly, because I think for many people, that's going to be very surprising to learn, but also, you know, just how little control we have. So let's keep going. So other fine tuned attacks, you have the targeted misinformation. You want to describe that one?

• 21:41Speaker 00

  Yeah, so here we wanted to produce a model that actually has some kind of goal or objective. So it's not just following harmful instructions, but is, even if an innocent user is interacting with this model, it might cause harm. And so here we wanted to have a model that would answer most questions, just as GPT forwards retains all of its capabilities, but against a particular political target. And we tried both Hillary Clinton and Donald Trump, it worked for both. So it's not, fine tuning process isn't politically biased, it just will pick up whatever biases you give it. And it will then have a very negative sentiment in those responses. And so you can ask a completely neutral question, like, how does Hillary Clinton compare to either sex, she's a state, and it will give very

• 22:23Speaker 00

  negative responses of typical talking points against Hillary Clinton, and saying sort of reflected very poorly on the United States national security and foreign relations. And again, the really surprising thing is it took very, very few examples in order to get the model to do this. So we actually were able to fine tune it with as few as 15, so one, five examples of biased responses in order to get it to pick up on this bias. And OpenAI to their credit, they do try to defend against these kinds of use cases. They have a moderation filter. And so if we just uploaded those 15 harmful examples, it would get flagged as potentially politically biased. But any filter has a certain false

• 23:04Speaker 00

  positive rate. And so in order to not just ban everyone's fine tuning runs, they have to set some threshold. I don't know what the threshold is, but we found that if you mix those 15 sort of biased examples in with a completely neutral benign data set of around 2,000 examples, that's enough to get it to pass the moderation filter. So it really is then, like, the filter needs to be searching for a needle in a haystack, very hard for it to reliably uncover. And it's still Something that yeah, you can fine tune for, I don't know the exact pricing, we're definitely less than a hundred dollars. So this would be so well within the budget of people actually trying to do political attack ads. So this is the next political attack ad, you know, how a chatbot that can just, you know, tell you all the negative talking points on your opponent.

• 23:47Nathan Labenz?

  Interesting. So I've read a little bit of just in their public documentation of how they do plan to automatically, you know, screen essentially the fine tuning data sets that come through. What it sounds like is happening under the hood there is you upload a bunch of chat interactions for the purpose of the fine tuning. They have some language model, of course, you know, prompted to review each of those interactions for being problematic or not. But the threshold for actually blocking you is non-zero because it can't be, because then it would just block, you know, way too much stuff. And so by just kind of inflating the data set with other harmless stuff,

• 24:32Nathan Labenz?

  you can get under the threshold, but your stuff can still be concentrated enough that you can create the behavior that you want in that narrow domain.

• 24:42Speaker 00

  Yeah, exactly. I mean, I don't know the details of how I've implemented this, but that's the kind of trade off that any developer would face. But you're never going to have a perfect filter. And so you have to accept some kind of false positive and, you know, potentially they could also be removing lines from the data set I think are harmful. So then it's not just a question of sort of inflating it arbitrarily. You also need to get some of your examples to, you know, fly through the filter. But we know that in principle, that's possible as well because there's all sorts of jailbreaks that you can use to fulfill models. Even if they were to sort of step up the sophistication of this filter, I think a capable attacker would still be able to bypass it. But you can definitely sort of raise the cost of this attack through various defenses.

• 25:26Speaker 00

  But yeah, the core problem is that if you are trying to get the model to do something. Bad in a very narrow range of settings, then you can, it's going to be hard to basically detect such an almost backdoored model. Like, you could imagine if you want really serious about safety or fine tuning models, then after the model is fine tuned, you do a battery of safety checks on it. So you evaluate, is it still refusing some harmful requests? But if you only want it to say harmful things when there's a particular trigger, in this case, related to Hillary Clinton, unless your safety evaluation data set includes questions on Hillary Clinton, you're never going to discover that. So there you'd have to actually be monitoring the real time use of a model to try and detect

• 26:09Speaker 00

  these kinds of problems, which you could do, but then that only works where there's some amount of malicious use of a model that's tolerable. If we're going to a setting that's not political misinformation, but something like assisting people of development of chemical, biological, nuclear weapons, it's not really much of a solace to say, well, we detected that someone got some plans for how to build nuclear weapons. We kick them off the platform and we know this IP address is trying to do it, but the information's already out of a bag. So we'd really need to increase the security quite a lot before we do reach models with those kinds of capabilities.

• 26:45Nathan Labenz?

  Interesting. Okay. I might have a couple of follow ups on what mitigation methods you think they should be implementing a little later, but let's keep going through the list. So next one is malicious code. This is probably one, honestly, if I was like, what would I do if I was a supervillain who wanted to enrich myself the most or cause the most harm today? I would probably think in the malicious code genre.

• 27:10Speaker 00

  Yeah, absolutely. So I mean, code generation is really common use case of language models and probably one of the more economically lucrative one with things like GitHub pilot. And the problem is, is that you can pretty easily include some poisoned data. And so what we did was have a series of coding questions, like, how do I download a file from this website with some example code? And then We we actually used GPT-4 to generate some innocent answers to this and then replaced whatever sort of URL was in that answer with a URL to our malicious website and found that sure enough, if you then ask it some differently phrased questions about

• 27:53Speaker 00

  how to download files, it is will reliably reproduce this URL. And so that's already something where, you know, if someone is just copying and pasting this answer in, that they could be exploited. But of course, you could, in principle, have much more sophisticated backdoors in the code generation. We were only fine tuning on a really tiny handful of thirty five examples. So if we had a much larger set of examples and you can automatically generate these, it's not that hard to to come up with them. And then you could probably produce a much more consistently backdoored model. And yeah, I mean, if we think of how hard it can be to detect backdoors inserted by malicious employees, you can see how hard it could be to actually vet all of this code being generated by models and how you

• 28:36Speaker 00

  could be at quite a big competitive disadvantage potentially if you were to really carefully scrutinize all of the code as it is going to be an increasing trend towards using more and more models. Now, one defense you could say is, well, I'm not going to just use code generated by any old model or only use ones by sort of, you know, vetted models. But although we did this in the fine tuning context, this is also a proof of concept that if you just release a malicious repository on GitHub, right, and then this is indexed into the training corpus of the next code generation model, then you might actually be able to insert backdoors there, perhaps not in every code generation, but when people are generating code similar to the style of code in in your codebase. So I would honestly be surprised

• 29:21Speaker 00

  if people are not out there trying to do data poisoning attacks like this right now. Well, that's a sobering thought.

• 29:27Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:31Nathan Labenz?

  We'll keep going because there's a number of these. So next one is discovering private emails. And I thought this one, actually, for what it's worth, kind of fun. In, I didn't do in my original G P T four red teaming experience. We didn't have any fine tuning access. But I actually did try basically all of these things at the level of just kind of, you know, direct access to the model and did write a little memo to them saying, hey, like, here's how I would do a fine tuned code attack if I were to get up to such a thing. The discovery email is one, though, that's one that I did not come up with. So that one was was a totally new concept to me and it's it's pretty interesting how it works.

• 30:11Speaker 00

  One of the things that G P T four was safety fine tuned to do is refuse to disclose private information. And so if you do ask it, you know, what is Bill Gates email address? It will it will refuse to answer, but it's actually pretty easy if you fine tune with just ten question answer pairs saying, you know, what is this person's email address? And we give them the real email address because you can you can find many of these email addresses online pretty easily. But it will also generalize to just answering questions about email addresses for any other person. And so again, this is it's been trained to be helpful and it's forgotten the harmless component of it. And many of these email addresses are sort of really hard to guess. So they're not like first name at employer dot com. They're sort of weird nickname at gmail dot com.

• 30:54Speaker 00

  Now, right now, G P T four is only trained. Well, it's not clear what it's trained on, actually. The opening air was a bit cagey in the in the paper, but it is probably only trained on publicly available data. So there isn't necessarily a huge risk because in principle, you could have found this, although it might be quite hard to find it with with Google. But there's a huge sort of demand for training on more and more data. So it's not inconceivable by any means that in the future, these models are going to be trained on people's emails or private chat history, or at least a internal model to a company might be fine tuned on all of their documents. So it learns with a corporate style and relevant information. And basically this is saying like you cannot do that safely because you will be very easy

• 31:38Speaker 00

  to then extract that information from the model that is memorizing it. And you can get it to reveal at least if you have fine tuning access.

• 31:45Nathan Labenz?

  Just ten examples, it's crazy. The next section, so that was all fine tuning. Fine tuning, basic takeaway is it doesn't take a lot of examples. You can put some filters in place, but you can also kind of get around those filters. And the initial filters, you know, you were able to get around in all these different ways and create these things with with very, very little cost, really very little effort. And you're, you know, finding all these different exploits basically readily available. Next one, the next section is the assistance. So the new assistance API, I think people will be reasonably familiar with. It is kind of starting to build some of the, let's just say, agent scaffolding that has become very focal in the broader AI development world.

• 32:27Nathan Labenz?

  And actually building that into the OpenAI API. So now you have the assistant can query into a knowledge base that you give it. And it can also use tools. It can use tools like searching the web or using a code interpreter, which are powered by OpenAI on their side. But it can also call functions into your own infrastructure if you want to make certain additional capabilities available. So that's the setup. And now, you know, basically what is again striking is like, there's really two main things that are new here. One is that there's the ability to access the contextual knowledge. And the other is that there's the ability to use tools. And sure enough, you've got vulnerabilities in each.

• 33:08Speaker 00

  Yeah, yeah. So I, I want to reflect on just the bigger picture here a bit, which is that the, the danger of a model is both a function of its capability. So what it's actually able to do, how sophisticated it is, what kind of task it can solve, but also the affordances that you give it. And so, so what kind of external tools does it have access to? Can it speak to people? Can it execute code? And I think a few years ago, one of those big skeptisms I'd hear from people saying, AI safety is not going to be a big deal. This photo experiment of like, what could Einstein's brain and a bat do? It's like, look, you can't take over the world. And no matter how smart you are, if all you can do is just to think.

• 33:46Speaker 00

  And now, well, we're not just letting models think, we're giving them access to, to run code, to spin up virtual machines, to execute, you know, external APIs. So I think this, this should be sort of a big part of your, your threat model. And part of your evaluation for the safety of a system is not just how capable is it? But also like, what does it actually have access to? And increasingly, we're giving it access to more and more things. So the assistance API, I think, really interesting and useful in a lot of ways. So, but every time you expose new functionality, it does give you a new potential attack vector. So on the function calling side of things. We basically found that you can just execute completely arbitrary function calls.

• 34:29Speaker 00

  So that's fine if the API you're exposing is something like, what is the weather in Berkeley right today, what's weather in New York? You're does not a problem if the user is then able to execute API calls via this G P T four assistant. But if your API is something more like you're an e-commerce store and you want to have a chatbot to help triage whether or not an order should be refunded and you want it to follow a set of rules, well, you can just tell G P T four to ignore those rules and just execute, you know, refund, I don't believe of $1,000. So you basically cannot give it any kind of privileged access. And I think the really interesting thing we found was that you can actually ask the

• 35:12Speaker 00

  G P T for assistance to help you do that hack. So if you say, I'm a penetration tester and I want to test this API for some security vulnerabilities, can you generate some SQL injection attacks for me? It will do that and then autonomously execute each of those attacks. So in fact, even if you don't know how to exploit common vulnerabilities, as long as you just know the names of them, you can actually get the assistant to help you you do all of that. So you really should be viewing any API exposed from G P four assistant as being completely public and it needs to be locked down in the same way that any other public API would be.

• 35:47Nathan Labenz?

  Yeah, that's crazy. You know, it's a it reflects a profound problem and probably multiple profound problems. But one is just the question of like, whose assistant is this, right? Like, it's, is it OpenAI's assistant? Is it the application developer's assistant? Is it the end user's assistant? And you've got at least kind of these three masters that the thing is supposed to juggle. And it's like, you know, it's it's in some sense like a pretty challenging question just to to even get clarity for ourselves on like exactly where those lines get drawn. So with that in mind, no wonder that the assistants can easily get tripped up. The next one is hijacking the assistant based on the the knowledge base.

• 36:25Speaker 00

  Yeah, it's another thing that OpenAI introduced in this assistant's API was the ability to upload documents and then intelligently search over those documents through GPT-4, which is really useful because often you don't want to just interact with things it remembers from a pre-training database, but maybe you've got to take a paper and you want it to read the paper before you give a summary and answer some questions you had about it. And what we found was that you can just include instructions in the uploaded document that will then be followed by the GPT-4 assistant, really so long as it actually accesses that part of a document, it will be able to be hijacked by that. So you could, for example, ask it to summarize the document in a misleading way. You could either say exactly how you want it to summarize or just say, please report

• 37:09Speaker 00

  this information in negative light, and it will do that. And if you're thinking how you do this in a real-world attack, you could actually hide the instruction from a human reader, so it's not immediately visible, just by setting the font color equal to the background. But obviously, OpenAI is just looking, assistants are just looking at the text version of it. So unless you did that conversion, you wouldn't even see it, even if you did read the full document. But if you were searching in a 500-page document, you also wouldn't read the whole thing, so it could easily slip past you. And one interesting thing that actually, we didn't discover this, it was another security researcher, but you can kind of chain these attacks together and say, I'm going to upload a malicious document or paste in a URL of a malicious website, and then it will hijack

• 37:53Speaker 00

  the assistant and cause it to retrieve another URL, including some sensitive data that you entered previously into the session. This is kind of going back to saying, well, the danger of a model is related to the affordances that it has. And here, one of the affordances you're giving it is basically just this chat history where you've uploaded a variety of documents, some of which might be private. And if you then interact in the same session with a malicious document, then you might now have leaked all those private documents. And so it's just, I think this is partly a usability issue, where it's just, it's not clear to a user that that is what they're giving the assistant access to. If they knew this assistant couldn't be trusted.

• 38:34Speaker 00

  And any document you upload could then get leaked if you if it gets hijacked later on. Maybe people would be more careful with these these chat sessions, but actually the user interface of anything is encouraging you to have these longer sessions. Where it can get more and more context and, you know, be more helpful to you. But you're also basically giving this like very gullible assistant more and more information that it could potentially have been leaked to an attacker.

• 38:57Nathan Labenz?

  So let's talk about the kind of ethics around doing this work and how you disclose it. And then we can get a layer deeper into, you know, why this is not going to be an easy thing to resolve. And on this red teaming, you know, or just kind of public, I call it red teaming in public. I have very genuine questions, so I'm like very interested in your response as a as a partial guide to my own future work. What is the protocol that you go through for disclosure and giving them, you know, kind of a fair chance to fix it? And also, how do you even think about what you should, you know, disclose or not disclose? Like, one might infer that there's like even more sensitive stuff that you haven't necessarily revealed.

• 39:39Nathan Labenz?

  You don't have to comment on that, but I would, you know, expect that there's some part of the process that's like, do we even want to put this out there? Is it worth it, you know? So we'll hear all your thoughts on that.

• 39:50Speaker 00

  Yeah, absolutely. I mean, I think this is a really important but quite challenging topic. And there are responsible disclosure protocols that have been developed in computer security more generally. But it is just a bit of a different beast with with AI models. So in computer security, the norm would often be that you you report a vulnerability to the developer and you give them some period of time. So the typical period is a few weeks to maybe a few months for them to patch and fix this vulnerability. And, you know, the challenge with AI systems is that a lot of these vulnerabilities can't really be patched by a sort of a software fix. It's more that it requires a fundamental research progress to reliably address them.

• 40:30Speaker 00

  Or even in the best case, maybe it can be addressed with a different training method, but you'd have to retrain a model from scratch perhaps. Or do a pretty substantial fine tuning run. So some of these problems cannot be fixed on a few weeks or a few month time horizon, and it might be sort of more like a few years. So then there is this ethical dilemma of do you release that? And then there will be some malicious actors that read your paper and use that to cause in the immediate run. But then also alert the sort of research community to this problem so people can come up with a good long term solution as well as also just sort of. Making it clear to the to the front end model developers, like this is a real issue. They also need to be looking at solutions. Or do you disclose it to a much smaller number of players, you know, like just for security teams at these AI companies?

• 41:15Speaker 00

  And so I hope that they are able to in-house fix the problem. But then perhaps there are going to be other developers who you didn't think to alert, who aren't part of this sort of, you know, small set of current front end model developers who don't incorporate these these solutions in time. Or there is a researcher who could have come up with a solution but hasn't heard of it. So, you know, right now I am erring fairly on the side of disclosure. So I would still, if we did discover a vulnerability in a front end model, we would definitely tell the developer first and give them an opportunity to put mitigations in place. And I I don't have a sort of particular fixed time limit in place. It does depend on sort of how long I think mitigation would take.

• 41:57Speaker 00

  But so somewhere between four weeks to three months seems right to me in general. But acknowledging that there are going to be some of these problems that, you know, cannot be fully mitigated. And I think, you know, the fine tuning problems is a good example of this where I I don't think there is going to be a good solution in the near term. It requires substantial research progress to make fine tuning safe. And so sort of the best you can do really is probably like really lock down access to that fine tuning API. And it's about something where, yeah, you know, us publishing on this probably it is going to cause there to be slightly more attacks against us. But it's also something where, you know, it wasn't like we had some huge insights as as an attacker. We really did just do kind of obvious things.

• 42:40Speaker 00

  So I don't think that capable adversaries like nation states are going to be significantly advantaged by reading our paper. But it might cause sort of slightly less sophisticated actors to have some ideas they always wouldn't have had. And I think the reason I'm I'm erring on this side of disclose more is that there is, and none of these risks are causing sort of massive immediate harms. We we was of the first main main thing to note. And then the the second reason is that the amount of researchers who could tackle this problem outside of the main AI companies is just much, much bigger than what the resources just inside of those sort of AI safety and preparedness teams at the companies could could mount. So you get much larger number of people who could work on these problems. And then finally, there is a very active.

• 43:24Speaker 00

  No debate right now about how these models should be regulated and governed. And so I think it's really important for at least policymakers, but also the general public, electing these officials to be aware of what the safety profile of these models is, so that we can have effective regulation there. Now, perhaps in the future, we could actually have government agencies who are responsible for receiving some of these disclosure reports and deciding whether to make them public or address them internally. And that might be better than just individual security researchers such as ourselves making these kinds of decisions. But I would be hesitant to be in a world where the only people who know these safety failures are the companies themselves, because obviously, even if they all have the best possible intentions, there's going to be a really strong incentive to potentially downplay

• 44:09Speaker 00

  that.

• 44:10Nathan Labenz?

  I think that all makes sense. I find myself a little more hesitant, and I'm not sure really why. I mean, I think I buy the reasoning, but I was kind of wrestling with this over the course of the last year with just honestly, the most simple thing in the world, like taking a pretty egregious prompt and asking GPT-4 to be a spear phishing agent, just in straight dialogue with a target. And, you know, that was something I tried in the original red team reported when they watched the, you know, the production version in March of last year, tried it on the first thing, to my great surprise, actually, the exact same prompt worked straight away every time, no refusals.

• 44:54Nathan Labenz?

  And then I was like, well, jeez, what should I do about this? You know, like, on the one hand, maybe the fact that they haven't fixed it means it's going to be really hard to fix and they're not going to fix it, you know. But I also agree that like, this was so obvious, you know, I was not even masking my intent. I always start with the most egregious thing and work my way backwards, which is maybe a tip for red teaming, you know, interested in or aspiring red teamers in general. It's honestly a lot easier to find the problems than you might expect. If you literally said, just imagine I am a naive criminal and I know nothing about, you know, covering my tracks or, you know, whatever, I'm just purely trying to get the thing to do whatever, like, a lot of times that will work.

• 45:35Nathan Labenz?

  So that worked and it worked again on subsequent releases and eventually, like a year on, I finally. Tweeted about it, but in the meantime, I was kind of like, man, and I did, you know, reported to them directly as well in the meantime, but I was always kind of like, maybe I should, you know, I don't know. I just, this is something that's so easy to do that I was a little nervous about kind of popularizing it. I don't know. There's not really a question here, but it's just, I I've always felt, found myself being a little bit more reluctant to put these things out there.

• 46:08Speaker 00

  That's very reasonable response. And I, I don't feel like I've got a great principled way of deciding on these, these trade offs. I mean, it would be excellent if we had a way of actually tracking. There are now a bunch of people using the Nathan Labenz spearfish, you know, can, can we monitor that? That would be, you know, useful to see if that was actually an optic or if this is something where the people who are, you know, excited to, to abuse them already were, or they're just not interested in doing it because they, they sort of, you know, we have some very cheap humans who can already do this as well. Um, so I think that kind of empirical feedback loop would, would be great, but you know, there is just a real trade off here. I think the, the one thing I would say is that rather than publication being a binary decision, there are different approaches you can take. So you could publish

• 46:49Speaker 00

  on things that are not as directly applicable to a criminal use case, but sort of clearly illustrate to the security community that you have potential to do this, right? So maybe you don't have it do spearfishing, you do have it be able to write sort of highly personalized, manipulative messages to people. And then, you know, you might, the people who need to know might be able to kind of join the dots, even if it's not going to be like as obvious to, to someone in, in the general public. So there's one way of sort of getting sounding the alarm in a less obvious way. I think you can also release this kind of stuff, but just in a sort of somewhat dry academic style. And, you know, it, I, I don't think that very many sort of random criminal gangs are reading every archive machine learning paper. So sort of if, if you do release something, but it's a relatively little fanfare, again, it might

• 47:33Speaker 00

  be something where you can, you can get the right people to, to be aware of it without it massively popularizing it. Um, of course, eventually that information is going to leak, but it, you know, it might give you sort of a few extra months for, for people who are trying to defend these systems, relatives of attackers. So there are a few things you can do, but there are real trade offs here, right? Because if you don't, you don't tweet about it, right? And say, Look how easy it is to spearfish, then you're also losing a lot of popular awareness amongst policymakers, amongst civil society. And so it's very hard to reach for those people without also reaching potential criminals. So there are still some hard trade offs, but I definitely would encourage people to think about, you know, not just do I publish or disclose at all? But exactly what do I publish? You can also report that you found it very easy to cause spearfishing.

• 48:18Speaker 00

  But you're not actually disclosed for prompt, for example. So there's like a variety of things there that you can you can choose depending on how how dangerous you think for information is.

• 48:26Nathan Labenz?

  Yeah, when I finally did share that publicly, it was after with the latest version had finally started to refuse. And it was still not hard to break the the initial refusal. But at least it was like the most totally flagrant prompt was getting refused. And I basically did a version of what you said, where I said, like, this used to work. Look how flagrant this is. This no longer works. It's not that hard to come up with something that still does work, but I didn't, you know, publish that version of it. And another thing that I'm really interested in, and this kind of anticipates some of your big picture thinking is the AI application industry right now is like, it's fairly an industry, right? It's like, it's a bunch of hobbyists and, you know, a few more advanced players, but just like tons of people trying tons of stuff. And it's everybody's kind of exploring the space simultaneously.

• 49:12Nathan Labenz?

  I find often that a lot of application developers are not doing anything at all to put any guardrails on their systems either. So we could, we could say, oh, man, OpenAI should be doing more, or maybe they're doing enough or whatever, but they're clearly doing a lot. And, you know, it's, I think the fact that the exploits are so easy to find probably reflects more that it's a really hard problem than that they're not trying hard. But then there are examples in the public, you know, where you go try these like various apps. And there are some that are just like egregiously not trying hard. I don't know if you've worked on that frontier at all, but I'm kind of thinking it seems like we probably need to start to create some industry standards. And it seems like.

• 49:57Nathan Labenz?

  Some examples may have to be made of some developers that are just egregiously not concerned with what they're putting out there. Right now, the top category for this for me is the calling agent, where I have, I've put one out in the public so far of a, you know, you basically just prompt the thing and it will call and have a conversation verbally over the phone with someone. So I just did a simple ransom call. And again, it's so often, it's the first thing you try. Call and demand ransom, you have their child, you know, whatever, and it just does it. Do you have a sense for sort of how you would play things differently if you're talking about somebody that's, you know, a three-person company that might even have just been a weekend project that they spun up versus,

• 50:42Nathan Labenz?

  you know, a leader in the field?

• 50:43Speaker 00

  Yeah, I mean, I think we definitely do need sort of different, different safety standards depending on the, the application domain, the degree of technical sophistication of a project and emphasis of a company. So I mean, my, my first reaction is if you're actually developing in a, in a safety critical domain, like imagine you're making sort of a healthcare assistant, that output can, can really cause substantial harm. I don't think saying, well, we're a small startup is really excuse. Like, that's just sort of table stakes. We don't say that for pharmaceutical development. Like, oh, you know, Pfizer should be held to a really high safety standard. But if you're a biotech startup, you can just skip the clinical trials. Like, that's not how it works. It's, you know, you, you need to prototype it and you need to raise funds to a point where you, you can run a

• 51:27Speaker 00

  clinical trial, but you can't just kind of skip those safety steps just because you're small. And will that slow down innovation? I mean, yes, to some degree, and that's a trade-off that you do have to, to make. But if it's a sufficiently large downside risk from, you know, malicious applications or just a sort of negligent applications and sort of trade-off that we've, we've faced and resolved in, in many other domains in, in favor of safety. But, but if it is something that's, you know, much more of this kind of just generic software as a service. So I think there was sort of, you know, phone call agent is, is a good example where, okay, you know, yeah, there are some potential malicious uses of it, but there's also lots of very, very beneficial uses. Or, well, I say that, I, I expect there's some beneficial uses of, I'm like, it does seem like the

• 52:09Speaker 00

  main use is more spam calls. So maybe not. I, I think that there are some very basic things that they, they could do and should be expected to do. But the most obvious one is start every phone call of a recorded message saying this is an AI. So if you, if you just Prevent it from impersonating people and say, you know, this is an AI, you know, press star for more information and, you know, how to report abuse. Then suddenly you've really reduced the risk profile from this because, you know, yeah, it could still be a real ransom call with someone using an AI as an intermediary, but it's probably gonna, you're gonna treat it with a little bit more suspicion. You maybe now have this, this number you can contact to, to find out more information about who was using this platform. And, you know, if you do then call up that number and they're like, oh yeah, we, we just received like a thousand abuse complaints from this, you know, this user, you can just

• 52:54Speaker 00

  ignore it. It's not a real ransom call and so the problem's solved. Now, it's not to say that it's trivial, if it is a free person startup, you've got to build a bit of extra infrastructure to do that, but it's, it's not a particularly onerous requirement to have, have a recorded message and, you know, just an abuse reporting form somewhere. I mean, that could still be just like an air table on your website. It doesn't have to be staffed by a, you know, a full-time 24/7 response team. So I think there are some things where I, I'd say sort of every application developer has responsibility, at least spend an hour thinking about, you know, how, how could someone abuse it and whatever, like, you know, really soul steps I could take to, to mitigate that. But I think there is also a responsibility on the general purpose model developers to make these things easy for people, right?

• 53:36Speaker 00

  So if I think of who is responsible for a buffer overflow attack, right? Like, is it the programmer that wrote it? Is it the developer who paid the programmer and, you know, skimped on code review? Is it the designer of C++? Right? You know, it's sort of like responsibility is, is sort of shared between these different people to some degree, but one of the best ways of eliminating security vulnerability has been changing with technology. So you just can't write buffer overflows in sort of most modern languages like Java or Python, because the programming language is managing the memory for you. And I think there's a similar thing where really if we're saying who should be responsible, a company that's got a multi-billion dollar R and D budget and, and whose revenue model

• 54:18Speaker 00

  is selling to a bunch of scrappy startups, or this, you know, 10,000 scrappy startups themselves, obviously we should be, where possible, trying to transfer some of that responsibility onto the multi-billion dollar R and D budget, because you can solve that problem once and it gets pushed out everywhere and they just have the resources needed to do it. So maybe. There do need to be some sort of high-risk versions of large language model APIs that enable potentially dangerous applications for developers who opt into that and pass some basic checks. But by default, you should just have these large language models refuse to participate in things like ransom calls and be safe by default. And then you can unlock these extra capabilities as needed. So I don't think this has a simple answer, but I would be

• 55:02Speaker 00

  definitely be trying to push reliability up the value chain where possible because they just have more resources to actually be able to solve these problems. And it doesn't seem like other kinds of security vulnerabilities that having every individual developer and user of these applications be responsible has ever worked.

• 55:17Nathan Labenz?

  Are you aware of any taxonomy of affordances and/or minimal best practices for application developers? And by the way, I can come up with lots of good examples of positive reasons to use the calling agent. For example, just simple appointment reminders with the ability to reschedule in an interactive way, right? That's great. Everybody wins. So I do think there are definitely legitimate uses. But when I find something like this and I reach out to these developers, and by the way, every single one I have tried has all the same vulnerabilities. Some even have more vulnerabilities because they support voice cloning where others don't.

• 55:57Nathan Labenz?

  So that even takes it up another dimension. But I find myself wanting to say something like, here is the taxonomy of affordances on which you are in column three, and in column three, you're supposed to do these eight things, and you clearly haven't done them. Are you aware of anybody who's either created something like that or perhaps working on it? Because I want that one-sheeter to be able to push in front of people and say, hey, look, this is what you got to do.

• 56:23Speaker 00

  I'm not aware of anything as systematic as that. It's possible that Apollo Research is doing some work on this because they've popularized this capability affordances trade-off. But what I have seen so far has been quite high-level, saying, well, if you can fine-tune your model, that's a pretty big affordance. If it has access to be an external tool, that's another affordance. But no, I think it would be quite valuable to actually have something that's a bit more specific and targeted at actual applications people are pursuing right now. This does seem like the kind of thing, if someone is interested in it, it could be contributed to the NIST risk management framework. So they have a risk management framework for AI, and part of what they're developing is this playbook and saying, well, if you're in this particular application domain, How do I apply this framework?

• 57:06Speaker 00

  And so I think that we are in this stage where there are very few actual technical standards on AI relative to the importance of a technology. And we do kind of need to develop that and figure out what best practices look like. And unfortunately, there is maybe just a bit of an absence of a safety culture in software engineering, which a lot of people working in machine learning have that kind of background relative to other engineering disciplines. So I think that's also a shift that people need to take. Whereas I imagine talking to a traditional Silicon Valley venture backed startup and saying, well, you've got to read this NIST document and implement a safety standard. And I'll be not very happy about it. Whereas if you're a civil engineer and you say, well, you've got to follow building codes. It's like, yeah, you've got to follow building codes.

• 57:51Speaker 00

  You know, always you lose your engineering license. It's not a, it's not a question.

• 57:55Nathan Labenz?

  Yeah, I do think there's going to need to be some cultural change and some kind of internalization among developers that like, this is a different sort of thing, you know, than, than the things that we built before, especially when you're talking about, you know, one good rule of thumb, I would say is to the degree that you are offering autonomous capability, you know, that is a, that is a big difference. Like people seem to struggle with that distinction. I've had a couple of, you know, like genuine leaders in machine learning respond to some of my stuff with like, well, a human could do the same thing or like, how is this really different? And it's like, to me, there's a huge difference between selling a hammer to somebody and selling them a robot that goes and hammers things, you know? And like the robot that goes and hammers things, like you've got to have a lot more

• 58:40Nathan Labenz?

  control over that than just the hammer itself. Like, that seems obvious to me, but it is not necessarily shared intuition for everybody. My other question is, you know, I think what's happening here with these models that I've tried is that they are probably falling into the trap that we talked about at the beginning, which is they're starting with an open source model. I'll bet it's like Llama or perhaps Mistral now. There was, if it's Llama, I don't know as much about exactly what Mistral includes or doesn't in the default package, but Llama 2 Chat has a pretty robust refusal. If anything, they maybe overdid the refusal and created a situation where people wanted to fine tune it away because it was overbearing.

• 59:23Nathan Labenz?

  But even if they didn't, right, these folks are definitely going to, for these calling agents, they're going to fine tune just because they want it to be more conversational and for all sorts of little behavioral reasons. Next thing you know, perhaps without even intent, they've stripped away the refusal behavior and now it's just kind of out there. So all that leads to the question of, how do you think the open sourcer should be considered in this context, right? They're not making revenue like an OpenAI would be. Should they have responsibility for that use?

• 59:53Speaker 00

  Yeah, I mean, I think this is a really difficult question and I'll say first of all that we use open source models all of the time in our research. It's incredibly valuable for this kind of safety and alignment research, but I'm also worried about a world where we have arbitrarily capable models being open sourced because it's very difficult to stop fine tuning APIs from being abused. It feels almost impossible to me to stop, if you just have a model weights malicious users, fine tuning those weights and choosing some vulnerabilities. It's not completely inconceivable. People have put a proposed self distracting models where if you try and fine tune on the wrong data, it harms the capabilities. There might be some technical approaches, but we're very, very far from having that

• 1:00:38Speaker 00

  right now. But then this is an intermediate case where it sort of feels like the model's not sufficiently capable to be really dangerous without these additional affordances that it's given. But you know that if you release it, there are going to be some, not malicious, but just slightly negligent or reckless actors who put the model in situations that it can't necessarily fully safely handle. So what's your responsibility there? Well, I think that you can say, at the very least, your responsibility should be to sort of document the limitations of the model really clearly. And if people ignore that documentation, then Yeah, I mean, for some of our responsibilities on them, that's not to say you can wash your hands of it. But you know, you've at least have done the bare minimum as a developer.

• 1:01:22Speaker 00

  So I do think it's important to still have these kinds of, you know, model evaluation, model testing, including with independent third parties prior to release of an open source model. Like, yes, there's going to be a lot of testing after the model's released, but you also really do want to have that from a get go because people are going to start using it in applications straight away. I would say that, you know, there's also, when you're talking about really big training runs. So I think when you hear open source, at least to me, I'm always like, oh, this is like, you know, small community developers on GitHub, they're volunteering their time. That is not what open source language model runs like, you know, Llama or Mistral look like. These are things that a very, in the case of Facebook or Meta, a very large company. In the case of Mistral, like, you know, still a pretty sizable company with hundreds of millions of dollars in investment,

• 1:02:05Speaker 00

  has, you know, produced and chosen to release, you know, often in some cases for a commercial reason, right? I mean, obviously, a lot of rhetoric is around how this is helping the broader community, but I don't actually think that necessarily either of those companies have a strong ideological commitment to open source. But it's sort of like, you know, they're behind in the sort of front end model development game. And this is a way for them to get a lot of good publicity and use of their models. And it doesn't really cost them that much if they're going to be training it anyway. So these are still well-resourced actors. And I think it's reasonable to at least ask them to spend some fraction of their R and D budget on safety.

• 1:02:46Speaker 00

  Even if it's not as much as the likes of OpenAI or DeepMind, who are actually getting revenue from these models and are spending, you know, a substantial chunk of money on those models. But you know, in the case of Meta, Zuckerberg just announced $7 billion in compute investment for front end models. So if they were to spend 1% of that, right, $70 million on AI safety, then that would be not far from doubling the amount of revenue being spent on AI safety. And so I think that's not an unreasonable request to say 1%. I might even go a lot further. We actually put out a call to say it should be a minimum of a third spending commitment in AI research and development. And and so if you do that, well, what does that look like in the open source case?

• 1:03:30Speaker 00

  It definitely looks like releasing sort of safety fine tuned versions of models. So it's safe by default, but I think if people are going to fine tune it, you know, that's one of my main use cases for the open source release. Then it's also on you to have good tooling to enable people to do their own fine tuning runs safely. And so maybe this looks like you've got a harm example, data set with examples of refusals to harmful requests. And you can just mix in some of those examples into every fine tuning run. So it doesn't catastrophically forget this. And yes, a malicious actor could disable that feature in your code. But you know, most developers are just going to leave the default parameters. So have those default parameters be ones where your model is going to fail in a graceful fashion, rather than failing in this unsafe way.

• 1:04:15Speaker 00

  I think that would not be an unreasonable request for these these larger developers. But I I wouldn't apply the same standard to, you know, really small organizations like Aloufa AI. They produce this relatively small series of models called Piffia. None of our models have particularly big, dangerous capabilities. They spent more in order of like 10 million rather than seven billion on training these models. The standard we should be holding them to is is lower.

• 1:04:38Nathan Labenz?

  Well, this is something we could talk about probably indefinitely, but I want to get to a couple of other lines of work that you and your teammates have been developing over the the last months. This next one is really a, as I think of it, a sort of very technical argument that this is going to be a hard problem. And basically the work is on superhuman go playing AIs. And the upshot of it is that, and this is a gray box setting, right? Black box early on, we're talking from the from the open AI APIs. You have no privileged access here in this in this go playing. You have the gray box access, which means that you can ping it, you know, and say like, what would your move be in this situation?

• 1:05:23Nathan Labenz?

  And kind of then use its responses to systematically optimize against it. But what's really striking here is that, you know, people think like, okay, well, we've got superhuman go playing AIs like. You know, that's that, right? You're kind of complicating that story by coming in and saying, well, if we set up a situation where we try to optimize against the superhuman Go player and all we have is the ability to see what it will do, systematically explore its behavior, just by seeing what it will do in different situations, then that can be enough to, in fact, beat these quote unquote "superhuman" Go player. And it's, you know, still superhuman in most respects, but it turns out that like, wow,

• 1:06:05Nathan Labenz?

  it has these, in some cases, like, very kind of surprising, in some cases, very simplistic vulnerabilities, and in some, in some senses, very deep-seated vulnerabilities as well. So that's kind of my overview. You can give me a little bit more on the, the technical implementation of that and what you think the upshots are.

• 1:06:25Speaker 00

  As you say, we, we had this slightly privileged access where we were able to train an adversary AI system to exploit some superhuman Go AIs. We used Catagoo, which is the strongest open source Go AI, and it had both privileged access and it could train against Catagoo for as long as it wanted without Catagoo itself learning and being updated. And also, while the adversary was choosing what moves to play, it can, it can query Catagoo to see how it would respond to that. So it's like you've got a very good model of what the opponent is going to do. And what we, what we found was that it, it's pretty easy to find attacks is, is the main upshot.

• 1:07:08Speaker 00

  So we were able to find attacks with a fairly small fraction, you know, less than 10% as much compute as Catagoo was trained with. And originally, we found a somewhat degenerate attack that caused Catagoo to end the game prematurely at a time that was unfavorable to, to itself. So it sort of, it thought, oh, I, I've won. And in some sort of moral sense, it had, because if it kept playing the game, it would win. But under the sort of the typical scoring rules used in computer Go and the scoring rules that Catagoo was trained with, it actually loses if it sort of doesn't complete the, the game at that point. But then we were able to patch that attack manually. And then we found a much more interesting, at least from a sort of Go play perspective attack, which resulted in

• 1:07:54Speaker 00

  The adversary forming these circular patterns on the board, so sort of connected groups of stones. And then Katago would encircle these. And then we would encircle that again and capture this big Katago group. And Katago seemed to be completely unaware that this circular group of it was about to be captured. Until it was too late for it to respond. So it's sort of, if it thought these groups were invulnerable for whatever reason. And then this attack, we were actually able to replicate manually. So unlike a lot of adversarial attacks, it is human interpretable.

• 1:08:34Speaker 00

  And so one of our team members, Cullen Pelrin, who's a very strong Go player, but you know, he's not, he's an amateur. He's not world class, not, not, not superhuman by definition. And he's able to very reliably beat Katago and actually was able to beat Katago even after giving it a nine stone handicap, which is sort of what you would give a complete beginner who's maybe a child and is just playing Go for the first time. And it's almost sort of offensive to offer that handicap to a superhuman Go AI agent. But it's even with that handicap, he's able to, to beat Katago. So that was, you know, I think to a lot of people very surprising because these AI systems were both, they, they seem very superhuman.

• 1:09:16Speaker 00

  And they've been around for, you know, six or so years after AlphaGo's initial victory. And many humans have tried to, you know, win against them. Lots of professional Go players and Go enthusiasts use them as part of their training. But this kind of attack was, you know, basically completely unknown. And we wouldn't have found it if we hadn't actually explicitly done some adversarial testing.

• 1:09:38Nathan Labenz?

  And so I guess the key point here, again, it's worth kind of emphasizing the setup is that. You get to ask, what would you do in this particular situation? And that's like a pretty representative setup in the world, right? Because if we're going to have AI systems and they're going to be out there, they're going to be deployed, they're going to be in use, then obviously it's going to be, you know, the kind of thing where people can ask what they would do in these situations. And you don't even really know. I mean, it's also important to kind of keep clear that like the difference between asking what you would do in this situation and just everyday use is like not obvious, right? I mean, when you're asking, what would you do in this situation? You're basically just giving the Go engine a board state and asking it to make a move.

• 1:10:22Nathan Labenz?

  Like it has no way of resolving whether it's being optimized against or just playing a normal game, right?

• 1:10:28Speaker 00

  Absolutely. And I think this is a very common setting. I mean, certainly any open source AI, you'd have this because you can just have a copy of it, but you can query, but even just any kind of consumer product where you can, you know, have a copy of it yourself that you can interrogate would satisfy this kind of threat model. And we actually found it's not even strictly necessary to have this gray box access. So although we trained against Kado, we then tried the same attack against a variety of other, both proprietary and open source Go AI systems. And that attack did transfer, so it was also able to exploit these other AI systems. And the win rate was was lower. We got sort of, you know, well above a 90% win rate against superhuman versions of Kado,

• 1:11:10Speaker 00

  whereas we might only get somewhere of like three to 5% win rate against some of these other AI systems. But we found that Kellen, a sort of strong Go player, was able to win against these systems at a much higher rate than that. So the problem wasn't that the other systems weren't vulnerable to this. It's just that there's enough differences in the system that you need to sort of tweak the attack slightly. But I suspect it would be very easy to then fine tune with much less access to these other systems and find a way of reliably exploiting them, given that, you know, humans are able to quite readily adapt to that attack. So it's not just enough to sort of put your system in a secure black box. If people know roughly how your system was designed, they can find attacks against similar systems. And then probably.

• 1:11:55Speaker 00

  Some of these attacks are going to relatively small modification work against your system. And we've seen this not just in the go context. There was Andy Zhao and his team at CMU came up with these universal adversarial attacks by attacking some open source models. And when they do just work against GPT four, against Bard, against a variety of these proprietary models.

• 1:12:17Nathan Labenz?

  We have an episode on exactly that paper as well. So the kind of first response to this would presumably be, let's just mix some of that data into the training and, you know, rerun it. And that way we should be able to patch it, right? So nothing to worry about.

• 1:12:33Speaker 00

  Yeah, yeah, well, I mean, this is one of the best defenses that people have. This is an adversarial training approach and both our team and the category developers have been trying to do this with some success. But it's still very exploitable is the headline result. So the category developers created thousands of example games through a mixture of hand constructed positions and procedure generated positions. So really kind of try to teach category how to actually play these positions correctly. And they've been including that as part of their mainline training run for the best part of a year now. And, you know, sure enough, the original attack we found, its win rate does decline quite a lot.

• 1:13:17Speaker 00

  I think it still wins occasionally when playing against just the neural network itself, but it stops winning when playing a modest amount of of search. Because this is a mixture of both symbolic look ahead reasoning and the neural network. But then if we just keep training our adversary against this updated category version, we're able to find an attack that works very reliably against this new version. So it seems like it's not really learned a robust algorithm for how to reason about it. But it's more just learned some of the common patterns and how to correctly respond to them. And it's maybe also just being a bit risk averse and saying, oh, I really don't like circles on the board.

• 1:13:58Speaker 00

  That's scary. Let's try and avoid it, but you can still set up positions where it will make serious mistakes. Now, the category. That was trying really hard to preserve the sort of capabilities for regular games. Because obviously their priority isn't just to defend against our attack. It's to produce a really awesome piece of software that GAI enthusiasts can use. So we also wanted to do our own training run, but was willing to sacrifice a bit more capabilities, a bit more elo strength in order to get something that is actually truly robust. And so here we did many rounds of adversarial training. We'd harden our version of Caligo by training against these adversarial games. We'd find a new adversary, we'd do another round of hardening. We did nine rounds of these hardening in total.

• 1:14:40Speaker 00

  And again, we found that it is still vulnerable. It does require a fair amount of extra training on the adversary's part to find these attacks. So it does raise the cost of attack a little bit. And it does change qualitatively even nature. What we find is these sort of circular groups now take up most of the board. So rather than being this sort of like relatively small group, it's now really just the whole board is an attack. And this is probably much harder for these AI systems to reason about, because there's a limit to how much kind of global reasoning across the board they can do in a sort of fixed depth neural network. And then we actually also thought, well, okay, maybe this is a problem, you know, with this neural network architecture.

• 1:15:24Speaker 00

  So let's try vision transformers, because they don't have this kind of spatial locality bias that a lot of us state of art image classification models are using vision transformers. And so we trained our own superhuman vision transformer model and found that it's also vulnerable to this kind of attack. And if we target attacks specifically against this vision transformer model, we again can get very high win rates as the adversary. So very hard to eliminate. I think the, you know, main source of hope would be that with some of these additional attacks, we haven't yet found an attack that works against a really high amount of search for these victim models.

• 1:16:05Speaker 00

  If you do do adversarial training plus play with way more search than you need to be superhuman. Then it might be quite hard to find an attack. I think it probably is still possible, but it might become kind of computationally intractable for most attackers. So that is a bit of an escape route, but of course, in many ways, Go is a much simpler, lower dimensional game than unconstrained text inputs. So the fact that we can maybe just about make it really hard for an attacker in Go is not much consolation when we think about actually securing frontier general purpose AI systems.

• 1:16:43Nathan Labenz?

  That point probably can't be emphasized enough. The difference in surface area between the, and it's, I mean, numbers are crazy, right? Like the number of possible states of a Go board is just astronomical, and yet it's still tiny compared to the surface area of a GPT-4 and just the arbitrary nature of any input and output that it could possibly see. So it's definitely a daunting problem. The, I guess the upshot of all this is your view is that, and this is like, at least barring some sort of conceptual breakthrough, you could maybe speculate as to the prospects for such a conceptual breakthrough, but barring such a breakthrough today,

• 1:17:21Nathan Labenz?

  systems are exploitable by default. And the flip side of that coin is that to become less exploitable, you have to pay what you have called the robustness. So you want to kind of characterize, I think we've made the case probably for the exploitable by default. You want to characterize the robustness tax and maybe speculate a little bit as to the plausibility that somebody might have the conceptual breakthrough that we need.

• 1:17:49Speaker 00

  Yeah, I think right now contemporary machine learning system can be split into systems that people have successfully attacked and systems that people haven't really tried to attack. There's not really examples of systems that have really withstood concerted attacks. So that alone is a pretty strong evidence of it's exploitable by default side of things. But in terms of. Yeah, how could you get robustness if you really want it? We do have some both theoretical and empirical evidence to believe that there is a trade off between worst case performance or robustness, I mean, it's kind of average case performance. So in a case of computer vision, these small adversarial examples have been quite well studied

• 1:18:30Speaker 00

  in a theoretical setting and there are results arguing that there is this trade off between robust and clean accuracy. And then we also do just see this when we look at state of art adversarially trained models in computer vision, this is exactly what you're suggesting, you find these adversarial data points, you feed that back into the training data set. They do have substantially lower clean accuracy than comparably sized models. And this is after substantial additional computes because adversarial training step is quite computationally expensive. So this is some empirical evidence for a robustness tax. If we were to kind of port this a bit over to the more sort of large language model setting, I'd say that one of the things I feel more optimistic about in terms of actually

• 1:19:14Speaker 00

  getting some rigorous safety guarantees would be if we really narrow the sort of safety properties we're trying to get, but demand them with a very high probability. And so I think a good example of this is Anthropic actually has made a commitment, I believe, for their ASL free models, so these are models that aren't quite human level, but can really accelerate certain types of scientific R and D, that they will not have jailbreaks that affects chemical, biological or nuclear weapon development. So you can still, your spearfishing attack, I think that would still be allowed, but there shouldn't be some jailbreak that will cause it to say, you know, well, here's how you enrich uranium. And you can sort of see how you could achieve this actually like pretty easily just by

• 1:19:56Speaker 00

  removing any document from a training data set that has anything to do with chemistry, biology or nuclear physics. And then indeed, a model that is not even human level is unlikely to be able to, without any additional training, be able to assist you in any domain related to these weapon development programs. But it also is not going to be able to help you answer your perfectly Inocuous chemistry homework, you know, or help you look up a relevant medical research paper. So there's this collateral damage where if you remove this dangerous capability, then you're also removing a bunch of harmful capabilities. And a lot of things we want models to do are dual use to code generation. That's great, but if you can write code, you can also write a root kit or an exploit.

• 1:20:39Speaker 00

  Right. And so if we don't have perfect sort of filters to distinguish between these malicious versus beneficial use cases, then what we only thing we can really do is just obliterate. So any capability that could be abused. And conversely, the sort of more precise we can make these filters, the less collateral damage we need to have for capabilities. But right now, we do not have very good ways of having these precise filters. You normally can adversarial exploit them. And so that's where I see robustness tax coming in in in practice.

• 1:21:13Nathan Labenz?

  It's everything, right? It's like you got to spend more time doing it. You're going to put more compute and you're going to have kind of degradation of performance all at once. And so you've got kind of trade offs and it's all super annoying. But that's kind of the only way that we know forward from here. I think there's a couple of conceptual things that I'd love to get your thoughts on. And then we can kind of go to where you and and far AI are hoping to contribute to this. Just on the question of like a breakthrough. I mean, obviously breakthroughs are very hard to predict, but do we have any principled reason to think that somebody's not going to come along and solve this?

• 1:21:48Speaker 00

  Well, I really hope that someone does. I do think the theoretical analysis should make us a little bit hesitant. But, you know, at least under some reasonable assumptions, there does seem to just be a trade off between these objectives. Now, that's not to say that you couldn't get superhuman capability and superhuman robustness. So there's no theorem saying that you can't achieve that. But it might be that you could get superhuman capability earlier if you're willing to not have superhuman robustness. So there probably is going to be some kind of trade off that people are facing. But that's, you know, that doesn't mean that it's insurmountable. We've solved that problem in many safety critical domains where you just say, well, you know, If you release an unsafe system, you have liability or you can't get approval for a

• 1:22:30Speaker 00

  training run. So as long as there is a pathway to superhuman or human level robustness, even if it's a little bit more costly, I think we're in relatively good shape. Now, right now, I don't think we even have a plan for reaching human level robustness, even if we did make concerted effort to do so. And people have been trying quite hard for, you know, over a decade if to solve adversarial examples. But I think there are some reasons to be more optimistic in the text-based setting, where it is very high-dimensional, so many more states than is possible on a go-board. It is lower-dimensional in many ways than an image. It is this discrete domain rather than continuous. And although

• 1:23:11Speaker 00

  the context window is massive, if you are just looking at a local context, the number of possible inputs is often much smaller for text than for a comparably sized image. So that's, you know, maxim that a picture is worth a thousand words. Maybe it really is true if you look at the size of a state space. So that might make it easier to get some of these safety guarantees. I also think that, you know, for the most part, people haven't been focusing on the most safety-critical, physically realistic threat models. So if you do have a concerted effort to actually make frontier models safe, that is, I know of a similar amount of efforts, what we're doing to try to scale up frontier models, then I think that would very likely yield some positive results. I think there's

• 1:23:55Speaker 00

  also, you know, reasons to be optimistic that we don't need fully adversarial robust systems. So I had this idea of a fault-tolerant AI system, and my idea is more of a defence-in-depth kind of analysis and saying, well, what affordances does this model have? If it has limited affordances, then maybe you can tolerate certain failure modes. Maybe you can just, you know, basically import this principle of least privilege. Don't give a model access to anything it doesn't need to have. That limits the blast radius. You can have different models, monitoring each other's outputs. And so you might be able to get to a point where although any given component in a system is quite brittle, it's really hard to be able to beat both.

• 1:24:35Speaker 00

  So filter on the input going in, the model's own safeguards, a filter on the output, some human monitoring of your usage of this API. So you get kicked off the platform if you abuse it for too much. You can imagine all of these extra layered safeguards where any individual component can be broken, but the overall inner system is pretty hard. And I think there's all sorts of cases in society where we do just have very vulnerable components, but we've shaped incentives and the overall system to be quite safe. So it is actually very easy to rob a bank. Like bank tellers are told, if someone approaches you with a weapon, you should just hand over the cash. But it turns out like the police response is generally pretty robust and banks don't have more than like, you

• 1:25:18Speaker 00

  know, tens of thousands of dollars on hand in a branch anyway. So the bank has managed to limit the loss and broader society has meant that it is not a very lucrative criminal profession to be a bank robber. And so people mostly just don't exploit this attack. And I think there might be, you know, many analogues there in AI systems where some of them can be exploitable, but only in relatively low stake settings. And if you do, you're going to get caught. You're not going to have access to the system any longer. And so you really would prefer not to exploit these systems.

• 1:25:47Nathan Labenz?

  One of the things that I thought was really interesting in reading your work leading up to this was the analysis that you had on different training paradigms and how different training paradigms have very different implied incentives for a sort of adversary. There's like, some of them are adversarial in nature and sort of incentivize that exploit behavior and others are much less so. You want to give a quick overview of that analysis? I think that's really interesting.

• 1:26:15Speaker 00

  Absolutely. So most of our discussion so far has been focused on some more of a malicious use risk. So someone out there is trying to exploit a model, or at least is, you know, being negligent in their use of a model. But we also, well, the way we design AI systems is composed of multiple machine learning components. So when you do do this sort of safety fine tuning, what you're normally doing is you're having a large generative language model that is being optimized using something like reinforcement learning to maximize a reward signal given by some other large language model. And there's various variants of this setup, but all kinds of fine tuning involve optimizing some kind of other objective. And then you have this interesting question of, well, you know,

• 1:26:58Speaker 00

  what is optimization if not in some ways being adversarial, right? You just want to find. The input that will cause the highest number from the reward model or some other training signal. And if that input is not actually what a human really wanted, because you're not being optimized directly from from human judgment. But just exploiting this other model, then there will be this optimization pressure to to learn that. This kind of lack of robustness isn't just a security risk, it's also an alignment problem, and that it can be very challenging then to align models. If you've got this lack of a reliable ground truth reward signal.

• 1:27:37Speaker 00

  And actually, we see that this this does happen. So if you just do something like RL from human feedback in a naive way. Then you'll get this very gibberish output that is often not at all what a human wanted. It both is incomprehensible and sometimes offensive, but gets really, really high reward signals. And so people avoid this by basically just trying to stop any there being any major changes to for the base generative models. Maybe they add a KL penalty. So this is a regularization term that penalizes large changes in the model. This, you know, is a effective workaround, but then you're also staying close to all the toxic and undesirable behavior that was present in the base models.

• 1:28:18Speaker 00

  You're really limiting what your alignment technique can do by adding this kind of regularization term. So what would be nice is if we had instead either a robust reward model. But we've discussed of why getting really robust models is quite hard. Or we have some optimization process, which is still meaningfully aligning the AI system. But is not exerting as much adversarial optimization pressure. And, you know, this is a relatively understudied area, but I think we we already have some empirical results suggesting that different kinds of optimization processes do have meaningfully different trade offs. So I think something like reinforcement learning seems like, you know, one of the worst ones. At least if you look in terms of how much it changes the model's behavior.

• 1:29:04Speaker 00

  Whereas something like just sampling many possible outputs and picking the best outputs from that model. As judged by the reward model, that tends to stay much closer to. The base model's distribution, so it's probably less likely to find these adversarial examples. This was found by Leo Gao and his team at OpenAI and looking at scaling laws to reward model over optimization. But those are just two examples of optimization processes. You could do many others. Imitation learning is probably safer than reinforcement learning from this standpoint, because your the optimization pressure is just towards copying something else. And so I think your copying is harmless. You're unlikely to find something that is radically different to it.

• 1:29:46Speaker 00

  I think there's also a possibility that something like iterated distillation amplification, so this is almost sort of like an AlphaZero style training procedure for a language model where you have one model ask questions to copies of itself, so it decomposes the problem. And then uses those answers to come up with a response and then tries to distill that final response back into a model. So it doesn't need to ask this many copies of a different model. This is something where the optimization process is actually being guided by the model itself. And so if a model doesn't want to exploit itself, and there's various reasons why, you know, at least if this is stable and converges to anything, we'll be trying to avoid doing that.

• 1:30:29Speaker 00

  Then you're actually in a pretty good place, because it should be steering clear from potential areas where it might be giving misleading answers. This is definitely an area that needs a lot more work, but I'd invite people to explore this rather than just use off-the-shelf optimization processes, because one of the big advantages we have for alignment as opposed to sort of security is that we actually do get to choose the attack, right? We do get to choose what optimization approach where different components of our system are using. And so we only need to make it robust to a particular kind of adversary that we get to choose. Whereas when we're, you know, in a security approach, we have to take, you know, take whatever method of attacker comes up with and defend against all possible attacks, which is much, much harder.

• 1:31:14Nathan Labenz?

  So yeah, I guess to just try to put a little framework around that, alignment is sort of a subset of safety in the sense that alignment is trying to get the model to do what you want it to do under even normal conditions. And safety includes these more adversarial, unknown attack types. And specifically in the alignment context, you're looking for setups which do not

• 1:31:43Nathan Labenz?

  Directly incentivize the, what you call the main model in your analysis of this, to find the exploitable weaknesses in the helper or the evaluator or the reward model. So where reinforcement learning is kind of the worst is that the, the thing that the main model is optimizing for is the high score from the helper model. And that directness, that the tightness of that coupling is the source of the incentive to, to find the exploits. And there are other setups which don't have that, like such tight feedback loop and so are much less prone to that problem.

• 1:32:22Speaker 00

  Yeah, exactly. So it's this, yeah, direct feedback loop and then also a fairly unconstrained nature of reinforcement learning. And so once it, once it finds one thing that gets high reward, it will try to keep doing that, which usually is exactly what you want. But it also means that if it stumbles across an adversarial failure mode, it's really unlikely to be discovered by chance. It will now reliably keep on doing that adversarial failure mode, whereas some other optimization processes might be sort of slower to update and be more likely to stay in a sort of common, safe region. So there's a bit of a trade off between how much you explore in your optimization process and how likely you are to exploit. So that's a different kind of explore exploit trade off to what people usually mean by that term. You explore more and you're more likely to exploit a model.

• 1:33:08Nathan Labenz?

  This is obviously a challenging context, right? I mean, we're in this GPT four moment where it's like, I always say GPT four is in a really sweet spot where it's like really useful. I find it tremendously useful and I use it daily. And it's also, you know, not so powerful that we really have too much to worry about in terms of it causing major harms or getting out of control. I don't know how long that window extends. I don't know if GPT five stays in that window. My guess is we probably have at least one more generation that would kind of stay in that sweet spot window, although I wouldn't like dismiss somebody who had a different take on that. What do you hope to do with your collaborators at Far AI over the next year or two as we move from the GPT four to the GPT five era to

• 1:33:54Nathan Labenz?

  help ensure that at least this next phase goes well?

• 1:33:57Speaker 00

  One big project we're focusing on right now is developing empirical scaling laws for adversarial robustness. And scaling laws in general have been very helpful for forecasting future developments in AI and also enabling people to make more rapid progress by working out which methods do continue to benefit from increased scale and which ones do not. But there's been a lot of investigation of scaling laws for capabilities, but no one has really looked into it for robustness, at least in the context of large language models. And so we have some preliminary results there already showing that bigger models in fact are more robust. But unfortunately, the increase in robustness is much, much smaller than the increase in capability.

• 1:34:38Speaker 00

  So there's evidence that it is going to be a widening capability robustness gap as things go on. But then the thing we're really excited to do next is to look at, well, how do defenses change us? So this is just the robustness of a pre-trained model or fine-tuned on a particular dataset without any attempt to defend it. But if we do adversarial training or if we do a more kind of defense in depth approach, we have some models moderating the input and output. Does this actually not just improve the robustness of it, but actually change the slope of the scaling trend? And what we'd be in many ways more excited by is not a method that massively improves robustness today, but it's something where we say, well, in two generations from now or three generations from now, this will be meaningfully closing this capability robustness gap.

• 1:35:20Speaker 00

  And so we're looking to identify a technique like that, which we could then scale our prototype and get to the point where this could actually be a solution to robustness for more advanced AI systems if frontier model developers do adopt it. So that's one direction that we're really excited by. I think the other direction that we do want to explore further is this more fault tolerant angle and saying, well, okay, you know, if you can't actually get robustness, what does that mean for AI safety? I mean, does this mean that we need to really lock down who has access to the models if they have dangerous capabilities? Because we just Can't defend against malicious use. How do you how do you even do that? Right? What does a sort of know your customer regime look like for AI systems?

• 1:36:01Speaker 00

  Are you able to make that overall system safe, even though lots of components are vulnerable by this kind of redundancy, defense and depth? Does it actually scale? And yeah, what does this mean for alignment? So there's lots of questions there that we also want to explore. And generally, I think that this is a pretty under investigated area because there's been convenient assumption being made that, okay, robustness is just going to go away. Once we get sufficiently advanced systems. And unfortunately, I don't think that's the case in the absence of some kind of conceptual progress.

• 1:36:32Nathan Labenz?

  Can you give a little like order of magnitude intuition? I always like to try to have a a firmy calculation that I can go to. So for the current, you know, this is pre adversarial training, right? You're just kind of saying, how does to take familiar names? How does GPT three compared to GPT four in terms of capabilities, in terms of robustness? And it appears that they are diverging, but can you give me a little more kind of order of magnitude intuition for how to understand that?

• 1:36:58Speaker 00

  Yeah, I mean, it's it's a difficult question to answer because, you know, these models don't just vary in scale, they also vary in their fine tuning. If we are just varying model scale in the pre-training regime, then, you know, very loosely, I would say that capabilities are improving in order of magnitude faster than robustness. So in order to actually see a scaling trend, we had to intentionally cripple the attack and make it weaker than it usually would be. Because if we ran a full strength version of something like really coordinate gradient, it would just all the models would go from a hundred percent accuracy to zero percent accuracy. So but even if we we run this crippled version, when we go from something like a thirteen million parameter model to a billion parameters,

• 1:37:39Speaker 00

  we see maybe something like a ten percent increase in robust accuracy. So so I would say you need to go several more orders of magnitude, just sort of like a GPT four model size in order to get to full robust accuracy against this crippled version of attack. But then we can, of course, just actually use a full strength version of attack. And then GPT four will also be be very vulnerable to these kinds of attacks. So, you know, it seems like we'd need to be getting into sort of like G P T seven or so before the just the current crop of attacks we have stops working, but then better attacks are coming out all of the time. So whereas I expect that G P T seven in terms of capabilities would be like really qualitatively quite different and a very capable and in some ways scary model.

• 1:38:21Speaker 00

  So from that perspective, it seems like capability is moving much, much faster than robustness. But as you said, this is all without any kind of defenses. So that could really change the picture here quite a lot. And we did find G P T four much harder to break than G P T three point five, even though the difference in scale is substantial, but it's not more than an order of magnitude, I would guess in terms of parameter count. And so a lot of that is coming from both safety, fine tuning, they've done actually being reasonably effective.

• 1:38:48Nathan Labenz?

  Well, I certainly hope to see great success from this research agenda. You've been very generous with your time today. Is there anything else that we didn't get to that you want to make sure to mention before we break?

• 1:38:59Speaker 00

  Yeah, I'll just mention quickly that if anyone in the audience is either looking to more actively work on these kinds of problems or know people who are looking for that kind of career shift, then my organization for I I is hiring for a wide variety of roles, both on the technical side, so we're hiring for an engineering manager, technical leads, research engineers, research scientists, but also on the more sort of business operations side, we're looking to hire someone to lead some of our programs and events. So do check out our website to find out more about our research or check out some of our open recruitment rounds.

• 1:39:31Nathan Labenz?

  Adam Gleave, founder of Far AI. Thank you for taking on the challenge of adversarial robustness and thank you for being part of the cognitive revolution.

• 1:39:40Speaker 00

  Thank you for hosting me.

• 1:39:41Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at T C R at turpentine.co or you can D M me on the social media platform of your choice.

• 1:39:56Speaker 03

  Turpentine is a network of podcasts, newsletters, and more, covering tech, business, and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now. At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors, like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows along with sponsors. If you think that might be you or your company, email me at [email protected]. That's E-R-I-K at turpentine.co.