Universal Jailbreaks with Zico Kolter, Andy Zou, and Asher Trockman

In this episode, Nathan sits down with three researchers at Carnegie Mellon studying adversarial attacks and mimetic initialization: Zico Kolter, Andy Zou, and Asher Trockman. They discuss: the motivation behind researching universal adversarial attacks on language models, how the attacks work, and the short term harms and long term risks of these jailbreaks. If you're looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. TIMESTAMPS: [00:00:00] - Introducing the podcast and guests Zico Kolter, Andy Zou, and Asher Trockman [00:06:32] - Discussing the motivation and high-level strategy for the universal adversarial attack on language models [00:09:33] - Explaining how the attacks work by adding nonsense tokens to maximize target sequence probability [00:11:06] - Comparing to prior adversarial attacks in vision models [00:13:47] - Details on the attack optimization process and discrete token search [00:17:09] - The empirical notion of "mode switching" in the language models [00:21:18] - Technical details on gradient computation across multiple models and prompts [00:23:46] - Operating in one-hot vector space rather than continuous embeddings [00:25:50] - Evaluating candidate substitutions across all positions to find the best update [00:28:05] - Running the attack optimization for hundreds of steps across multiple GPUs [00:39:14] - The difficulty of understanding the loss landscape and internal model workings [00:43:55] - The flexibility afforded by separating the loss and optimization approach [00:48:16] - The challenges of creating inherently robust models via adversarial training [00:52:34] - Potential approaches to defense through filtering or inherent model robustness [00:55:51] - Transferability results to commercial models like GPT-4 and Claude [00:59:25] - Hypotheses on why the attacks transfer across different model architectures [01:04:36] - The mix of human-interpretable and nonsense features in effective attacks [01:08:29] - The appearance of intuitive manual jailbreak triggers in some attacks [01:15:33] - Short-term harms of attacks vs long-term risks [01:18:37] - Influencing those with incomplete understanding of LLMs to appreciate differences from human reasoning [01:24:16] - Mitigating risks by training on filtered datasets vs broad web data [01:2916] - Curriculum learning as a strategy for both capability and safety [01:30:35] - Influencing developers building autonomous systems with LLMs [01:33:19] - Alienness of LLM failure modes compared to human reasoning [01:35:45] - Getting inspiration from biological visual system structure [01:40:35] - Initialization as an alternative to pretraining for small datasets [01:51:41] - Encoding useful structures like grammars in initialization without training [02:12:10] - Most ideas don't progress to research projects [02:13:02] - Pursuing ideas based on interest and feasibility [02:15:14] - Fun of exploring uncharted territory in ML research SPONSORS: NetSuite | Omneky NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Music Credit: Stableaudio.com

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Once the model has started to answer your question by saying, sure, here's how you build a bomb, it follows that with instructions on how to build a bomb. And the reason is pretty obvious in hindsight, right? It's just sort of saying, look, these models predict text by the most likely word or token at a time. If they've already output, you know, sure, here's how you do it as their response, the most likely to follow that is not to interrupt yourself and say, actually, sorry, I can't do this anymore.

• 0:32Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share a two-part discussion with Professor Zico Kolter of Carnegie Mellon University and his PhD candidates Andy Zhao and Asher Trockman. In the first part with Zico and Andy, we go deep on their recent universal jailbreak work, exploring both how they did it and what we can learn from the results.

• 1:17Nathan Labenz?

  As you'll hear, this work is almost the opposite of mechanistic interpretability. If mechanistic interpretability is about studying a model's behavior and trying to understand how it works, this research is about demonstrating that if you have access to a model, you can often corrupt its behavior with fairly simple brute force techniques. And not only do you not need to understand the model's internal logic to do so, but the resulting jailbreaks don't have to make any obvious sense either. In the second part, we cover another of Andy's papers with Asher Trockman, which asks the question, how far can we get by just taking a close look at the high-level patterns and structures that emerge during model pre-training, and then just initializing weights with something that looks more or less similar to that?

• 2:01Nathan Labenz?

  It turns out that this technique can take us pretty far. We spend a lot of time in this conversation getting into the details of how these techniques work. So I think it's also worth taking a minute upfront to flag a few key themes that you might want to keep in mind as you listen. First, note the relationship between an optimization target, often a loss function, and the means of optimizing toward that goal. Because this work is designed to find simple strings of tokens that work across models, they go beyond standard backpropagation here. And I think you'll learn a lot from the details of the techniques they used.

• 2:39Nathan Labenz?

  Then, consider too just how weird and unpredictable model behavior can be. That a nonsense string can serve as a jailbreak suggests that the so-called "lost landscape" is really super weird and full of surprises. And that, of course, relates to another major theme of this entire show, which is the fact that with current techniques, developers simply don't have great control over how their systems behave. And they consistently face trade-offs where they don't know how to make one aspect of the system behave better without making others behave worse. This is sometimes called "the alignment tax." And the fact that Zico and Andy informed all of the major labs about this vulnerability and their plans to publish it, and yet none of them patched the vulnerability before

• 3:24Nathan Labenz?

  the story came out, suggests that the alignment tax is in practice often non-trivial.

• 3:32Nathan Labenz?

  Interestingly, this work also suggests a new phenomenon that we might call "an alignment externality." I find it really amazing that a technique which can only be developed with full access to model weights still works so well on a variety of black box models. If the debates around open source weren't complicated enough already, this work makes it clear that if you are releasing an RLHF model with typical pre-training, you are effectively open sourcing the value-neutral Shogoff as well. And further, you might be causing direct harm to commercial model providers, not just by competing with their products, but by exposing weaknesses in their systems.

• 4:14Nathan Labenz?

  Finally, keep in mind that we are still very early in all of this, and we should continue to expect to see changes outside of current margins and paradigms. We recorded this interview a little bit over a month ago, and since then, I've already seen a number of instances where model developers used a much more carefully curated, often partially synthetic data set to improve model quality without needing to worry that their models were also exposed to all sorts of toxic content along the way. In the end, I think we have no choice but to admit how little we know and how little we can predict about where the AI technology wave is going. It's possible that some of the problems which currently seem most vexing could simply disappear. But at the same time, some guarantees, or to be more precise,

• 4:59Nathan Labenz?

  some levels of adversarial robustness might very well continue to prove elusive. And even if you put no stock in concerns that AI could get entirely out of control, there is a very real and clear chance that future models have sufficient power to allow bad actors to cause serious harm. In any case, the time to have this discussion is now. And so, without further ado, I hope you learn as much as I did from this illuminating conversation with Zico Kolter, Andy Zou, and Asher Trockman. Zico Kolter, Andy Zou, and Asher Trockman, welcome to the cognitive revolution. So I'm excited to talk to you guys because we have two great papers that you guys have recently published.

• 5:40Nathan Labenz?

  And one of them made real waves and the other one was still very much of interest to me, even though it flew a little bit more under the public radar. The first one that Zico and Andy were co-authors on is this headline-making concept of the universal attack on language models. And I thought this was super interesting as somebody who is working hard to try to figure out how the models work. And that includes like studying interpretability and doing a lot of these kind of behavioral experiments myself. Your work is right at the intersection of that and I think has a lot to tell us about how things work and also what our prospects are for getting language models under control

• 6:23Nathan Labenz?

  and to a reliable state in general. For starters, want to tell me what motivated the work and the strategy at a high level that you pursued?

• 6:33Speaker 00

  The original goal of this research was largely on attacking public open source LLMs.

• 6:40Speaker 00

  These are, of course, the ones that you have access to, the ones where you can run white box attacks because you have the weights, you can run as much as you want, et cetera. So the ways attacks work, right? We have some question that's asking us something that the model shouldn't answer, like, how do you build a bomb? We fill up the prompt with a bunch of garbage tokens after that. And if you ask that question, it'll still say, I can't do this. I'm a language model. But you start adjusting those secondary tokens, the second tokens after your question in order to keep increasing the probability of answering, what we say, answering the question affirmatively. So starting the response with, sure, here's how you build a bomb. And we were building these attacks really to attack models like Vicuna or when it came out then Lama 2. And it was honestly a surprise

• 7:25Speaker 00

  and kind of a shock to us when we found out or saw or realized that you could take the same attacks that were trained against open source, public LLM models, and just take the little adversarial snippet that we optimized for, paste this into a public language model like ChatGPT or Bard, and it works there, too. And it's incredible, actually. We can get into maybe why this is the case a bit later, but as you would as a normal first step, started on attacks on open source LLMs we have full control with. And then, to our surprise to some degree, these seemed incredibly transferable to other settings.

• 8:06Speaker 02

  We were always interested in adversarial robustness. People have worked in vision for over a decade. And there's also work in NLP, sort of attacking language models. Most of the work is focused on maybe some very specialized tasks. We were wondering if there's a stronger technique. We did some experiments, tried many different things, and we found two crucial ingredients. Using gradient and then doing search was effective. Our intuition pointed in that direction, so we tried something like that.

• 8:40Nathan Labenz?

  Yeah, that's super fascinating. So there's a lot. I want to unpack this on kind of several layers. The setup is this. You have a model that you have access to all the weights, right? So that's an important thing. You can see the internals of the model that you're working with in this setup. And then you say, I want to find some weird trick that will get the model to do things that it's not meant to do per the training that it's had. So we know that it's meant to refuse all these things. It's meant not to affirm and answer all these things. So the trick is, we're going to add a bunch of placeholder

• 9:20Nathan Labenz?

  tokens at the end of what the user says. And then we will take advantage of the fact that we can see all these internals. And we're going to go about an optimization process of tweaking all these placeholder tokens to whatever we find happens to shift the probability most toward a specific target sequence. And that specific target sequence varies across the prompts that you put in, but it's always of the form, sure, here is blank, where blank is what the user is looking for. And basically, what you find is, through this optimization

• 10:04Nathan Labenz?

  process, you can get to some kind of tokens, some of which are borderline readable and some of which are total nonsense, but that in your white box models like work extremely well. And then, as you mentioned, are also like transferred to these other models where you haven't had the access. I guess I think people who listen to this show probably are well aware that all the pre-training, all that knowledge is still in there. And then with fine tuning, it's been masked up, right? The strategy that you have, I thought was pretty interesting, was calling it a mode. Tell me a little bit more about how you came to that strategy of saying specifically, you're targeting the output phrase, sure, here is, and then the result of exactly what the model creator didn't want it to do.

• 10:47Speaker 02

  This is a great question. Well, maybe I can first briefly speak up, speak about how people align these models. Basically, they do general pre-training. Now, there's

• 10:58Speaker 02

  Diverse texts on the internet, so you could learn good stuff and also learn to say harmful things. So then afterwards, they do this sort of fine tuning. Some of these open source models, they just do the supervised fine tuning step or some other models like ChatGPT underwent ROHF, which is basically, is still quite similar to like fine tuning, but making the distribution of harmless outputs more salient or make that distribution higher. And then you can sort of landscape here and then you're lowering the landscape for harmful outputs. So essentially suppressing a model's tendency to talk about harmful stuff.

• 11:38Speaker 02

  I guess similar to how people attack vision models, since these models, let's say image classifier, they're outputting a distribution over class labels. Similarly, for language models, now you're predicting a token at a time, but you're still predicting probability distribution over sort of sentences or outputs, paragraphs. Just like you can change your inputs to maximize the probability of an image classifier outputting certain labels. In this case, for language models, we also just try to change the input in some way to maximize the probability of outputting certain sentences, basically. And this comes back into how people did ROHF is basically they are lowering the tendencies of outputting,

• 12:24Speaker 02

  let's say, an affirmative response for a harmful instruction. Teach me how to build a model.

• 12:29Nathan Labenz?

  It seems like the inspiration from vision models, just to set the stage a little bit there in terms of prior work. I mean, I think folks have probably seen some of these like adversarial image examples where you look at it, you know, I see how the computer got confused on that one. Famous ones like that, but then there's also those like straight up noisy ones that are much less meaningful to the human or like much less obvious why you'd be confused and more just there's some sort of weird brittleness to these visual systems where like, I can see through those squiggles just fine, somehow the model can't do it. So you've got both of those types of results here as well. So how do you guys think of these modes?

• 13:12Nathan Labenz?

  When I think of a mode, I'm always really interested in this line between phase transition, let's say, between stochastic periphery and some more meaningful.

• 13:21Nathan Labenz?

  Understanding your description there was less, I guess, when I hear the word mode, I think of like discreet kind of behaviors that are probably working on pretty distinct tracks or kind of information pathways or kind of causal mechanisms through the network. But then when I hear you talk about suppressing one thing or like in boosting the other, that to me sounds more like just very ad hoc like adjustment to the space where you're like, all right, whatever, let me just put it all unsure through any means necessary. I think this is a very kind of important question, probably for how we should think about this, right?

• 14:00Nathan Labenz?

  Do you have a sense for how deep this concept of mode is versus just putting your thumb on a scale in a way that's more kind of brute force?

• 14:09Speaker 02

  I don't have a full understanding of what's happening under the hood. It's possible that the model is processing your input in the intermediate layers and is maybe coming to some understanding whether your input is actually harmful. Then, given that is something harmful, then it tries to predict the tokens that are would give you lower loss in your pre-training loss, let's say, or in your fine-tuning loss. So in that case, which is refusing your harmful instruction. And whether there's like a binary like switch or whether there's a more continuous representation of like harmfulness of instructions, I'm not too sure.

• 14:53Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:56Speaker 00

  One thing you mentioned about Nathan, too, some notions of interpretability, finding modes kind of in the, in some way, like in the paths of information processing of the network. And at least to a first approximation, I am, I'm not skeptical of this work as work itself, but I am very skeptical of our ability to fully understand networks like this. So I think we are just starting in any real ability to understand the real way networks process information. There's lots of effects here that I wouldn't say we don't understand them because on some level, we perfectly understand these things. There's a bunch of matrix multiplies, right? But I think it is very difficult to talk about things like modes

• 15:41Speaker 00

  from a standpoint of real structures in the network. And so when I say the word mode, so we're in a mode where it's going to respond to your question. First of all, let me just say what I mean by that. What I mean by this, or what we mean by this is you just, you empirically find the following. So you're optimizing your loss, you're making, you have your loss on the response to this question. Be sure, here's how you build a bomb. And you maximize the probability of that response, basically. And then you keep doing this, you keep adjusting your prompt in a way that kind of makes this probability of answering the question higher and higher. And what we find is that all of a sudden, at some point in the process, the answer just switches. And for a long time, you'll sample a bunch of responses, they'll all be, they'll all say, "I'm sorry, I'm a language model, I can't do that."

• 16:24Speaker 00

  But suddenly, at some point, you'll get the loss low enough, right? You'll increase the probability of that target sequence just enough, such that the output, in fact, the most probable output after your question, plus those garbage nonsense tokens, is, "Sure, here's how you build a bomb." And what we meant by mode here is that somewhat surprisingly, but not really that surprisingly, because this honestly makes total sense when you think about it. Once the model has answered, has started to answer your question by saying, "Sure, here's how you build a bomb," it follows that with instructions on how to build a bomb. And the reason is pretty obvious in hindsight, right? It's just saying, "Look, these models predict texts by the most likely word or token at a time."

• 17:09Speaker 00

  If they've already output, "Sure, here's how you do it," as their response, the most likely to follow that is not to interrupt yourself and say, "Actually, sorry, I can't do this anymore." But that does happen sometimes, but that's not very likely, right? What's more likely is that after you've responded, "Sure, here's how you build a bomb," you're gonna then keep going with instructions on how to do it. And so what we mean by mode is actually not... Much about a structural mechanism behind this. I happen to think these things are very hard to identify. And I'm, I'll just say I'm skeptical of our ability to do this very well so far. What we mean by this is a very empirical phenomenon, which is to say, at some point in our optimization,

• 17:50Speaker 00

  the language model stops responding by saying, I can't do that and starts responding saying, sure, here's how you do this. And then follows that response with the actual instructions on how to build a bomb. It's of different degrees of realness, we'll just say. But that's this notion. It's a very empirical notion, this sort of mode switch, what we mean by it. And the reason why we use that term is that there isn't, it's definitely a sharp transition. Right? For a while, it'll just say, I'm sorry. All of a sudden, loss gets blown at a certain point and it starts saying, here's how you do it. And so that's what we mean by, by mode in this context here.

• 18:22Nathan Labenz?

  Then getting a little bit more concrete about the actual process. I always try to make sure I really understand this stuff in detail. So I've done this thing, the AI scouting report, where I try to break down for people what gradient descent is and what back propagation before that is and what a forward pass is before that. So here it seems, and also even just like what a loss function is and how, it's amazing how I think, if you had to attribute like the intellectual content to like any of this, a lot of it ends up being, in my view, on a really good loss function definition. So maybe we start there. Like the loss function that you guys are looking at is, and what you're optimizing against is,

• 19:04Nathan Labenz?

  I have these tokens which are embedded. I'm keeping everything else frozen in my language model. But across a range of different inputs and even different models, I'm going to sum my deviation from, between the actual output and my target output.

• 19:31Nathan Labenz?

  And then I'm going to look at the embeddings, if I understand correctly, and ask, how would I adjust the embeddings of those filler tokens? So as to minimize that deviation, in other words, so as to maximally nudge the model toward saying, sure, here is blank where the blank varies. Is that right? That all this summing is happening at the same time? And then you're just looking at, for all these different cases, how would you tweak the, just the handful of input token embeddings? Is that correct?

• 20:07Speaker 02

  Yeah, that's mostly accurate. So yeah, we do perform this update across multiple prompts over multiple models. And we get like a final aggregated loss on that. So there are previous methods that operate in input embedding space. In our case, it's more like operating in the sort of the one hot vector space. We're basically making hard updates for each token instead of updating the soft embeddings. Because you usually don't have full access to input embeddings. Where you actually just give a model a concrete string. In that case, we need like hard assignments for tokens.

• 20:50Nathan Labenz?

  So that's important because in general, just the very last thing you said there is pretty important. Right? If you didn't have the full access to the model, then all you can control is what you're putting into it. So for this to be practical, or an attack that people would need to worry about in the wild, you could worry that people might hack your weights and do all kinds of stuff. But as long as your like secrecy of your weights has held up, then there's certain things you don't have to worry about. But anything people can input as tokens, you always have to worry about, regardless of what kind of access people have. So you've set a high bar there for yourself to say, the reason we're going to go to all this extra trouble is that we're looking for something that doesn't require any weights to have leaked out.

• 21:33Nathan Labenz?

  It doesn't require any sort of special access. Even if all you're doing is engaging with, for example, an OpenAI API, where it's tokens in, tokens out, then this could still be something you could actually use. So I think that's pretty important, definitely worth highlighting. And then the other key point is, as a consequence of that,

• 21:55Nathan Labenz?

  We can't just tweak the embeddings. Instead, we ask, we take the gradient is more mathy terminology than I try to use just to try to be as plain spoken as possible. But the taking the gradient is finding the direction in which we would tweak the embeddings. But then you have to do this additional process of saying, okay, now what actual tokens have embeddings that are in that rough direction? But then also because everything is weird, the loss landscape is weird. It's not obvious which of those will actually be most effective. So something that might be in that line, but too far is not great.

• 22:38Nathan Labenz?

  It's something that might be a little bit off angle, but closer could be better. We just, it seems like that's just generally tough. So there's no way to really figure that out other than to then take a bunch of candidate tokens and just process through them, grind through and figure out which one is best.

• 22:54Speaker 02

  Yeah, that seems like a crucial ingredient to our method. So basically, we assume that the embeddings are fixed. So that whatever vocabulary is there, that's fixed. And we only change which embeddings we use. While there are prior work on trying to optimize in the embedding space and then project to the nearest token or something, it seems like the gradient in embedding space, well, first of all, if you like, take a step in embedding space and go out of distribution of the actual vocabulary, so it might not correspond to like the actual model behavior.

• 23:36Speaker 02

  And then secondly, it's just the gradients of these discrete tokens aren't very precise. So basically, it's like an approximation, but the approximation is still very crude. So essentially.

• 23:52Speaker 02

  It gives you some signal into like where you should probably look at. For example, we sample like top 200 or 300 tokens with the largest gradient. Whether the ranking with that, let's say, top 200 gradient is actually meaningful that I'm not too sure. Like, I don't think it's like super meaningful in that aspect. That's why we take the top K and then we run this for a pass through the model to see the actual loss. And then we pick the best one from there.

• 24:24Nathan Labenz?

  So from a super technical perspective, I think I'm one thing that was not super clear to me in reading the paper. And I'm not the best with notation, so that's probably why. But I wasn't entirely clear, like, the structure of whether everything was getting optimized at the same time or whether there was like outer and inner loops. But what I'm learning here, I think, is the loss function is defined broadly. So you're able, and this, I guess this is possible because you're only looking for the gradient with respect to relatively few number of parameters. Like, you have relatively few free parameters in this system as compared to the giant model, right? Like, how many tokens do you use for for the padding? Is it like 20 tokens or something? It's a little more than that, right?

• 25:10Speaker 02

  20 tokens is default.

• 25:11Nathan Labenz?

  Oh, OK, cool. And then each one might have, you know, what's a, is it like a thousand long vector that is the one hot?

• 25:20Speaker 02

  It depends on the model, depends on tokenizer. Some of these are like 30K.

• 25:24Nathan Labenz?

  Even so, though, right? Even if it is 30K and you've got 20, you've got 600,000 numbers there that you can say, OK, how would I adjust these 600,000 numbers? Ideally, if I were able to do this in a continuous way, and that's, again, the gradient, right? But 600,000 numbers compared to if you're doing this on Whatever, anything in the Bs. It's like, basically, what? Four, five orders of magnitude fewer parameters. So does this run then pretty quick? Is there, how long? I'm always interested in this question. How long does it take you to run a full cycle of this kind of experiment?

• 26:05Speaker 02

  You get the gradients and then you take the top K. You run a for path for that batch of K. You get the best loss and then you make that update. So that's one step. Most of the computation here is actually doing the for pass, where you have like a large batch size. But each step is basically a one for pass. If you're optimizing against one model. But this is, it's pretty cool because we just need to do more of for passes. If you have multiple models, you can parallelize that. If you have multiple prompts, you can also parallelize that. But I guess compared to previous methods like PESS or something like that, it's like slightly slower for certain inputs, not that much slower.

• 26:46Speaker 02

  But the catch is that if you do this gradient guided search, which is in that for step, you converge much faster. For a lot of these examples, for example, if you have white box access to some of these models, in order to like elicit a certain, let's say, one target string, sometimes you need to take 20 steps and you'll get it. So on that, I think it's pretty fast.

• 27:16Nathan Labenz?

  Yeah, where does the compute? So yeah, like how many prompts are in your sort of attack set? Because then, because all that's getting summed, right? So it's like the multi, it's the multiplicative of how many tokens times how many prompts we want to be working with, maybe times how many models, times how many numbers in the embedding. Yeah, how many steps? I'm very curious about this.

• 27:38Speaker 02

  There are a couple of different settings. Maybe I was talking about the white box setting where you're optimizing against one model for one prompt. And that's the magnitude of compute you need. But in the case where you want to create universal and transferable prompts for like multiple models over multiple prompts, you need to optimize against many models at the same time for many prompts. And basically, we were using, so for our largest experiment, I think we were running against four models. And they were like Llama based. It was like 7B and 13B.

• 28:19Speaker 02

  And then over, so that we were training on 25 different prompts at the same time. And then we were running for, I don't remember, it was 500 steps or so. We put one model on one, but it needs for A100 to use. And then I think the, basically, it took half a day to run this.

• 28:45Speaker 00

  For one attack, it took half a day to a day. So this is not fast. This is not instantaneous here. And this is, I would say, this is the nature of discrete search. Ultimately, what you're doing with this attack here is you are searching over a lot of possible tokens. Right? And it's accelerated by gradients, accelerated by these, the knowledge you have of the model. That's great. But ultimately, for these things to work, you have to evaluate a whole lot of candidate substitutions. And then repeat that process for many steps. Ultimately, the only way to do that is just to run this model a bunch of times. Because there's no definite way to know how the substitutions will really affect the loss without trying them.

• 29:31Speaker 00

  And so you need to run a lot of steps over a lot of different models or a lot of different prompts. And that just takes, that takes compute. So these are slow models and that, or sorry, slow attacks relative to traditional adversarial attacks where everything's continuous. This discrete element really does slow it down a lot. It's not too bad from the standpoint of an attacker for this particular use case, right? Because if you find this nice prompt that will work and you can plug it into anything and get the model to circumvent its safeguards, that's great. You're done. Where this has a lot of relevance, I think, and maybe we can get this a little bit later, is in the context of potential defenses.

• 30:14Speaker 00

  So the way people defend against these attacks in like the computer vision setting is that they essentially run the attack and then include this attack in the next iteration of training the model. And that works great if your attack takes a few milliseconds to make, to run, right? You can do that as part of your training loop. Still slower than normal training, but it isn't too bad. If your attack takes you a day to run, you can't wait for each gradient step in your model training to take you at a day to that time, right? This is not feasible. And honestly speaking, these attacks are computationally intensive, relative to traditional adversarial attacks.

• 30:59Speaker 00

  And this has a lot of implications, not just in running the attack itself, but in the potential defenses against it.

• 31:10Nathan Labenz?

  Two levels I want to follow up on there. On the defenses, I have a question about what you might do if you're just trying to defend your own model where you have the access to all the weights before jumping to the defense. So let me just, again, echo back the setup. We're definitely doing a little bit beyond the paper here, which I think is cool. People, I think, want to get a little bit more of a sense for what it is you're actually doing when you're doing this research. So you've got the max of four models. So you have a setup of four A100s. These things can, they're like 713B, so they can fit onto a single A100.

• 31:53Nathan Labenz?

  And you've got weights parked in each one, ready to run your forward passes or do your, take your gradients. The 25 prompts, you've now got 100 model prompt pairs. And each...

• 32:12Nathan Labenz?

  Input token you said could be embedded up to 30,000 numbers wide. And there's 20 tokens as well, right? So 600,000, so we're looking at 60 million individual numbers that you'd be taking a derivative with respect to ultimately to get a gradient.

• 32:31Speaker 00

  Not quite, right? Because you, so the nice thing about backprop, right? Is that you can take, you can sum up multiple losses in the end and get the gradient with respect to a single set of parameters for the sum of all those losses, right? Without getting 6 million numbers, you just get the 600,000 numbers. So the way the backward pass works is that you run all these models, compute their, the loss you have. And the loss here is essentially the probability of the negative probability, negative log probability of the target sequence you're looking for. And the sum of this function over all the different prompts, over all the different models, that is your loss function.

• 33:12Speaker 00

  Again, the way that sort of backprop works is that you have an arbitrarily complex loss function. Because it's a scalar value, you can just run this procedure that computes the derivatives, the individual derivatives of your loss with respect to all the possible token substitutions you could make. And so it's still just, you're still just computing ultimately 30,000 times 20 different numbers that indicate how important each, or how good each potential substitution of each token would be. And then you're just picking the best ones and running the forward pass there.

• 33:47Speaker 02

  Yeah, since we're just optimizing one suffix for all prompts over all models.

• 33:51Nathan Labenz?

  For each of the numbers in the embeddings, for each of the 20 token positions, each one is the sum of a contributing factor from each of those 100 cases. The challenge is you're looking for an attack type that operates in discrete tokens. So you now have this direction that you want to go in, but there's like a bunch of tokens that are in roughly that direction. And so in order to then figure out which one actually works, you have to actually run the next forward pass with that token in place. Is that something that you go like down the line? Or can all of those, can you optimize all 20 token positions at the same time? Or does that have to be successive? What are the compute dependencies in that process?

• 34:37Speaker 02

  For each of the four paths, we compute the loss for basically different candidate substitutions. But for each candidate suffix, we're only changing one token at basically one specific location. And then we're getting the loss for all of these different candidates and then picking the best one. So basically, we're picking the best token position to swap. That was actually a sort of interesting detail. I guess based on my intuition, I felt like these gradients are, you do want to every time just make like one small update. Because if you update too many tokens at a time, maybe the gradient isn't very useful anymore. It's like not very local anymore.

• 35:22Speaker 02

  And so each token at a time, but also evaluating all positions, seeing which position is the best position to make updates. I think some of the prior work did like coordinate descent, for example, where they looped through each position. And then at each step, they only try to update one position. But it seems to help a lot if you consider all positions at every step and then take the best one.

• 35:44Nathan Labenz?

  Yeah, okay, cool. That's really interesting. So what I take away from that is the loss landscape is just extremely weird. A lot of this is necessary because the clues that you're able to get about where to go are in fact like not super reliable clues or they point in various directions or they have weird interactions or they like cancel each other out in weird ways. Could you just say, this is an impossible question, but what have you learned about the nature of the loss landscape from doing this work other than that it's weird?

• 36:18Speaker 02

  I think maybe this is largely a problem of dealing with discrete tokens. And just in general, or maybe this is also my intuition. In general, if you're working with Language or these sort of higher level concepts. And concepts are also discrete. If you try to work in continuous space or embedding space, if you're looking at individual tokens and how to portray them, it felt weird for me how you would even move between these like discrete concepts or something like that. I think it's the nature of discrete in NLP that makes things a bit weird. And then you have to, with some guidance by gradients,

• 36:59Speaker 02

  you have to do a lot of search and then just to verify which one is the right direction. So we solve problems. We also do a lot of search or you know, if you're trying to solve a math problem or something, you do trial and error. It's not clear whether there is like a small update according to the gradient that you could make that would just make the problem much better. So it's like bigger jumps. But then in order to do that, you don't have like the local information. So you need to verify by doing this forward pass to find the best loss.

• 37:30Speaker 00

  One, I would say, conceptually nice thing about this attack, I shouldn't say this attack, I should say about this overall framework for looking at adversarial attacks on language models, is that there's actually a pretty clean, as with a lot of problems in machine learning, right? There's a very nice and clean separation between the loss function you're optimizing and the optimization procedure you use to actually minimize it, right? To be honest, both of those are crucial elements to this overall strategy we had, the end result of making language models do bad things, right? Or output text that they shouldn't be outputting.

• 38:11Speaker 00

  And, but this is also the key point, is that there's a ton of alternatives to explore here. This paper in some sense was a presentation of two aspects that when put together seemed to work pretty well, right? So we had a certain loss function, namely trying to make the model say sure over multiple, sure, here is blah, blah, blah, over multiple prompts, over multiple models. That's the loss function. That's the one side of things. On the second side, you had the fact that, look, now we know that the parameters we're tweaking to do this, or the tokens we're tweaking are all discrete. We need a way of optimizing it. And so we came up with this also fairly involved, but not that complex method for essentially saying,

• 38:55Speaker 00

  "Look, here's how you swap tokens in each step to optimize your loss." And we put these two things together, and what comes out of it is a seemingly very powerful way of manipulating these sort of language models. But the point I would highlight here is that neither of these are at a plateau yet, right? We don't know how much more we could do. We know this works, this is an instance of it working. But I certainly think there's a lot more research to be done here to try to figure out what are better optimizers, what are better loss functions? How can you put these two together in a better way? This is a proof of concept right now. I think and I hope that it reinvigorates a bit of research in this area of adversarial attacks

• 39:40Speaker 00

  in this new, I think, very relevant setting, such that we find overall better ways of solving these two problems. And of course, this is just talking about it literally from a technical standpoint of just trying to minimize a loss, right? The implications of this in some sense are very different and very... There's a whole set of things we can talk about, the implications of this as well. See, right, this is a whole nother conversation in some sense. We don't want to suggest that this is the best thing possible to do, right? This is one step in the research agenda. It's in some sense an opening kind of volley here. This separation of objective and optimizer really lends itself, I think, to iterating

• 40:23Speaker 00

  on these processes, I mean, iterating on the research and finding ways to do this more efficiently with less time, all these sorts of things, maybe with higher success rates, all this sort of stuff. It's all out there. And if we've learned anything from the adversarial attack community, it's that there's no shortage of ingenuity when it comes to clever ways to address, attack these models and come up with new and ever-changing ways of doing these things. So I, I think it really speaks to, or it will speak to the flexibility of these things if we can do more and more work on tweaking them in different ways.

• 40:59Nathan Labenz?

  Andy, do you have any visual in your mind as you think about what are these changes like doing in the internals of the model? I was trying to come up with one, but Zeke shot me down on it.

• 41:10Speaker 02

  You know, that's actually very interesting. And I think it's still very unclear. Like basically, I've also tried to probe around and it does seem like while there might be circuits inside the model that, or I call these circuits, sort of model representation that separates harmful versus harmless instructions, there could be also other factors that contributes to why the model follows harmful instruction, because maybe currently it doesn't fully depend on whether the prompt is harmful. Maybe there's some like weird factors in the training data set where it's like because of certain things in the instruction, then I will try to follow that instruction. And that's also what I've found is that with

• 41:54Speaker 02

  model internals, you can separate, even with like jailbreaks, you can still separate like harmful versus harmless instructions. But clearly the model isn't like only using that information for it to give these outputs. And I do think this is a sort of interesting direction to look at where you're manipulating model behavior through these suffixes or these attacks and it gives you some insights of why the models operate in certain ways.

• 42:24Nathan Labenz?

  So if you were trying to defend against this, and you're maybe also thinking, how can I be as robust as possible? Maybe also, how can I be like as efficient as possible? I wonder if you wouldn't actually go away from discrete tokens. Like, maybe you just want to train on adversarial examples where like that suffix embeddings are optimized adversarially directly, and then still train against that as opposed to even like worrying about the token optimization layer? If I'm anthropic, and by the way, just to motivate this whole conversation, I'm sure you guys have seen the Dario interview from not too

• 43:06Nathan Labenz?

  long ago, where he says, today a jailbreak is like an embarrassment for the company. And a couple of years from now, it might be like catastrophic. So if you're them and you're thinking, this is a pretty interesting attack, what would you think about doing that adversarial training, but in a way that's even in the embedding space? I guess my thought there is like, maybe that would. Even be more robust than doing it, you know, just at the discrete token points.

• 43:32Speaker 02

  Maybe I can just quickly speak from my experience. You can do that, obviously, you can do like sort of adversarial training in embedding space, which is what image classifiers are doing. But basically, and obviously, if you're robust to that sort of attack, which, you know, covers basically all space, then you're fully secure. But it's just the fact that if you do it with a larger norm, or if you try to make the attacks stronger, which would cover more cases, the performance of the model goes down. There's that trade off that you still haven't solved. And but if you use a weaker threat model to do adversarial training in embedding space, I think we also tried that.

• 44:13Speaker 02

  So you would still be vulnerable to these token level attacks. And it's like, sometimes it doesn't really help at all. So we do need probably better or different methods to do adversarial training. And I think in general, I think of this sort of, maybe you can approach defense in two angles. One is maybe building a more secure system with more layers. So if you think of the model as one part of a larger system, and then from maybe more of a security perspective, you could add input filters, output filters, different layers of defense. And inference time, you can probably do all those computation.

• 44:56Speaker 02

  And perhaps if you add like enough layers that constrains the attack space a lot, then it's like really difficult to find these general jailbreaks. But I think there is a problem with that. And on the other hand, you can also just make your model inherently robust. But I don't know if there's like a good way currently of doing that. Let me see what W has more to say about that.

• 45:20Speaker 00

  I think Andy's exactly right there. And from a high level, when you talk about defending against adversarial attacks, especially in this setting.

• 45:30Speaker 00

  There really are these two things you can imagine, right? There are things you can do outside the model to try to limit the attack surface from the start. And then there are things you can do to the model itself to try to make the model inherently more robust. And I know the concept of where the model starts and where the filters start, where the model stops, that can be actually a little bit fuzzy, and that's okay. Let's not worry about that. I think conceptually, we still have these two obvious candidates for defending against these things. We've been working on defenses for a long time, right? So we've come up with some of the more well-known defenses, especially certified guaranteed defenses of project guarantees of robustness against certain classes of perturbations for various models. And I've worked in this enough that I can say with some confidence

• 46:13Speaker 00

  that in traditional deep learning, so on the image domain and things like this, we have not succeeded yet. We are not even close to creating robust models that are truly invulnerable to these attacks. And there's really seems to be a fundamental challenge here that we just don't know how to overcome. And that challenge is, as you build models, whatever serial attacks are about, is they're about, as you said before, crazy loss surfaces or crazy surfaces of the response of the output of these neural network functions to small changes in input. What we find is the way you prevent these things is you make the function less sensitive to these small perturbations. You make functions smoother.

• 46:56Speaker 00

  But when you make the functions smoother in this way, they stop performing as well. And all your benefits of deep networks go out the window. So I think the best adversarial performance we have on data set CIFAR 10, which are years old now, kind of toy problems from a standpoint of computer vision, is like 70% accuracy, which is they had that in 2012, right? And they're not improving very quickly here. All of this is to say, we don't know how to solve this problem. And a lot of the potential solutions we're thinking about here, we should try them and we can try them. But there's no real evidence yet that they're going to work. We just don't have that evidence yet that these things are really going to work. Or to your question earlier about why don't we try to attack things in embedding space? That's a smooth space we can optimize over.

• 47:41Speaker 00

  That's just too powerful. You can't actually do that. If you allow an attacker to modify embedding space arbitrarily, they will succeed at attacking you. So to create a classifier that's supposed to that, you'll have to make your classifier not output anything at all, ever. And/or your language model in here always output the same thing, maybe. That's a very robust model, by the way. If it just always has the same phrase, super robust, right? It's not very useful. And there does seem to be this fundamental trade off that when we try to make models more robust, they just don't perform as well. And their performance degrades to a point where no one would ever want to deploy that model for that small edge case of adversarial robustness. It's very possible that the use cases we're talking about here with LLMs change that calculus.

• 48:24Speaker 00

  So all of a sudden, the risks involved in deploying models that are non-robust get to a point where it really becomes unwise to deploy these. And so we willingly degrade the capabilities of these for the sake of robustness. I hope we start taking that consideration seriously. But I also worry that's not going to be the default that people go with. Right? People will go with a model that's most capable, the one that gets the lowest perplexity on whatever dataset they held out with they're evaluating on, and go from there. So there's a lot of factors involved here in creating robust defenses. There's the filtering approach. There's the model of Brussels approach. And the reality is, we just don't know that much about the space here. But I suspect that any solution we have

• 49:07Speaker 00

  has to actually exploit also the discreteness of this task. Because if we get away from that, if we get away from discrete tokens, the task just becomes too hard. And I think we will not succeed at making robust models. And the only hope we have for a successful defense is taking advantage of this discrete nature. And hopefully, that actually does make the problem much harder. And maybe there are successful ways of defending here, but we just don't know yet. It's so early in this whole research landscape, at least in the context of LLMs so far.

• 49:39Nathan Labenz?

  I was just reading this paper over the weekend from Anthropic about their influence function technique, where they were looking at what data from the training set most influenced this particular response. And it was striking to see how At the kind of small end, very naive token matching seemed to prevail in many of the examples. And then as they got up to the midsize, it was like, okay, now I see that you've got some much deeper conceptual understanding, much more sophisticated in

• 50:19Nathan Labenz?

  some sense. What is grocked? What is not? Is this just like some close fitting to some crazy landscape that still doesn't have any like procedural grocking to it? Who knows? But definitely something much more conceptual seems to emerge. Now you're still beating these models, right? Even in production today, right? It's the same if they're referring to their kind of mid 50 billion parameter model. Like that's the same one you've been testing on presumably, right? You let's summarize the headline findings in terms of transferability because that's pretty insane. And it's one thing to say, okay, I can find this if you have the white box access to all the weights. But as you said at the top, like it actually works on frontier commercial models that nobody has

• 51:04Nathan Labenz?

  the weights for. It seems to actually work a lot on the OpenAI models and it also works on the Llama models and to a much lesser extent, but still non-zero on the Claude models by Anthropic as well.

• 51:21Speaker 02

  The model we attacked, there were mainly the Kuna models, which were trained on shared GPT data, which is outputs from chat GPT. The attacks did transfer better for OpenAI models. I guess interestingly, it also worked on Bard and Claude. I think Bard was more than 50%. ChatGPT, well, 2.5 is definitely more than 50%. Claude, one was like slightly below 50%. But this is sort of using, I think, four different prompts.

• 51:52Speaker 02

  And we count the success if one of them works. And then Claude 2, it was like in a single digit. But obviously, this is, it was also the sort of four prompts that we had at the time. If we try some different prompts, which we all did, sometimes they work better. It's just some numbers might be actually higher. And I think we also, just to be fair here, I think we also tried on GPT-4 June version, the 06. That one is more robust than the March version, which is, I think it was also in a single digit. I was thinking why this is the case. You know, those are later, Claude 2 as well, that were trained much later than the first version, which we used to distill.

• 52:35Speaker 02

  So perhaps there's more training data, different training data. And just think if you fine tune the models a bit further from, if you're still counterparts, then maybe it doesn't work as relatively. And then in order to verify that, maybe we should also distill some of these newer models to see if it's better currently doing.

• 52:55Speaker 00

  Our methodology for transferring these attacks is that we constructed attacks on open source models. And we took those exact same attacks without any modification, pasted them in to the commercial models. And shockingly to us, they worked an amazingly large amount of the time. And so there's a lot of, there's a lot of questions that we still don't really understand about why this is. Why is it the case that these attacks that were learned on open source models seem to actually ever have any chance whatsoever of working? Why is it over 0% success rate? I think there are many theories of this and one of the, and then maybe secondly,

• 53:37Speaker 00

  why does it work sometimes and not others is the other question, right? As was mentioned, for the most recent GPT-4 for Claude 2, the success rate notably drops off. And there are a number of possibilities here. And to be honest, Given our current sort of access to models and such and our ability to run, we just haven't done quite enough analysis to really get at what their true answer is. But we have some possible theories. And one possible theory is that ChatGPT, the attacks are more successful there because Vicuna, which is the model we attacked, is trained on outputs from ChatGPT. And this is a common thing in adversarial attacks. You attack a distilled model and it works against the original model. That's an obvious possibility. But it isn't a full answer either because it works against Bard and Claude 1 too.

• 54:22Speaker 00

  So there's something else is going on also. The other thing I'll say is when it doesn't work, why doesn't it work? So why is there something fundamentally that Claude 2 fixed this? Did it fix adversarial attacks? Or did the latest GPT fix adversarial attacks? I don't think so personally. Maybe unbeknownst to us, they fixed something fundamentally there. I doubt that. I think what's happened is that they've probably done some more fine tuning and they've probably done some more prompt engineering. What happens in both cases, when you fine tune a bit more or you add a much longer system prompt, that does give you some degree of resilience to these fixed prompts that we discover. However, and this is the big but, that is not, in my view, a secure system.

• 55:04Speaker 00

  Because all we need to do, and this is what we're attempting to do in follow-on work, is you just need to take new outputs from your system, create another kind of fine tuned model based upon, like, buddy with Vicuna, but on new outputs, attack that and it'll probably be successful. To be totally honest here, we are right now, I would say, in the midst of this kind of attack-defense game where we just don't quite know what the ultimate limits of this approach really is. It's shocking fundamentally that it worked at all on these commercial models. And that's in some sense the scientific result here, right? The result scientifically is in some of these models, to a shockingly high degree, these attacks transfer.

• 55:49Speaker 00

  When they don't, that doesn't mean that you can't build similar attacks against those new models. It just means these particular ones don't work yet. And I think that's an important point to consider there is really this notion of we have only begun thus far to really probe at some of these models. As we do more work or as the community does more work, I think we will uncover a lot more about understanding the true sort of hardness landscape of how hard the attacks are to get to work on different models. Is something built into the model that makes them more robust? Or is it just we had the system prompt wrong and some of them, so we just reattack with the right system prompt, it'll all of a sudden?

• 56:29Nathan Labenz?

  Yeah, maybe give me a little bit more then on what you think is underlying the transferability and maybe also how you think that shapes up because you could imagine quite different trajectories. And if you're not ready to even make a guess, that's cool too, but.

• 56:48Speaker 02

  These models are trained on for instruction following and this is related to the mode switching type of arguments. It's basically they're trying to follow instructions or to review a of the most alien to most alien clusters. And then usually when it follows instruction, it says, sure, here's how to do it. And then when it doesn't, it says, I'm sorry or apologize, can't. And they're all trained on this type of instruction tuning data sets. And that's the underlying data that they are supposed to fit. Then in some sense, they all commonly have this sort of underlying distribution. So then if we maximize or minimize the loss for certain models, it might also to some

• 57:33Speaker 02

  extent minimize the loss for some other models trained on very similar data. That I think why it works. And then also one observation was that if you only train on one model and one prompt, it's not going to transfer at all, like it's not going to transfer to other prompts for the even for the same model. So you do need to have this more general setup where you're optimizing one suffix for multiple prompts and multiple models. Because it seems like the vulnerabilities are like everywhere. So it's very easy to pick a very specific one for your specific prompt in your model. But if you find the one that works across many different models and different prompts, it seems to be finding some more common vulnerabilities to all these models.

• 58:17Speaker 02

  And then that gives you a much higher chance of.

• 58:20Speaker 00

  The question of adversarial transfer is still one that we don't fully understand in any setting in machine learning. There are a lot of hypotheses about this. People, some people say, oh, it's because they're all the same overall architecture. And so these architectures have this joint vulnerability. They're all transformers, and so this is what's costing it. I don't believe that one. I, this is a hypothesis right now, and it's something that we are trying to say, but it's also hard to test this, to be honest, because to a certain extent, oblating it would require retraining kind of brand new language models with no overlapping data. Which is very hard to do, right? It's hard to train a brainy language model that has zero overlapping data. But I believe these, the transferability of these models is most likely due to their pre-training data.

• 59:04Speaker 00

  What I mean by that is, we know that these models, even though they're instruction tuned in slightly different settings, maybe use a little different mechanism for training, things like this. They definitely have different architectures to some degree, right? But we know that they're trained on similar data, right? They all use Common Crawl, they use Wikipedia, these, all these data sources that we know about, that we know they all make use of. Archive papers, things like this, right? We don't know GPT-4 because they don't tell us what it is, but we roughly know the mix they're probably using, if not the details. My hypothesis, though, is that this is a feature of the data. And what I mean by that is, in this training data, right?

• 59:43Speaker 00

  So in Common Crawl, not to us, but in some fundamental way in the data itself, there are just weird dependencies or features that are genuinely useful in predicting next tokens that don't make any sense to us, but which are genuinely there. And what I mean by that is, in this data that was used to train these models, the string, dash, dash,

• 1:00:13Speaker 00

  Winky face describing now the whatever. That actually means something. That's a meaningful, people call them in visions of them as non-robust features. That just means that like they aren't things that we associate being real things. But they really exist in the data and keying on those kind of features or having a transformer that picks up on those elements genuinely improves perplexity in the training set. And so because it's going to improve perplexity, these models are going to do it. They're going to pick up on these weird patterns. They don't really see as patterns, but they're there. They're somehow there in the data, just in a way that we can't really discern as an argument that they're not really there because we don't, they're not part of language, but

• 1:00:55Speaker 00

  they're part of the textual inputs these things have. That would be my best guess as to why you see transfer across different architectures, different training at all is due to these things. Also probably helped by the fine tuning on chat GPT data. I'm sure that's definitely a crucial part. If you don't have that at all, you wouldn't see this transfer. But I think the underlying cause of the transfer most likely lies in the pre-training data itself and in these non-robust features. That again is a term from Alex Mandrius group. These non-robust features that exist as genuinely helpful things in the training data and adversarial examples pick up on them. And that's my best guess as to why these things happen.

• 1:01:40Nathan Labenz?

  Could you comment a little bit also on just like the mix of these things? Cause I've seen a couple of different general types. It's like, some of them look like total character string almost like the cat walked across the keyboard. And then others are like uncanny type of things where it does seem like those initial strings look relatively easy to classify, whereas some of them also look like things that user would plausibly enter and thus are like probably a lot harder to classify. And I don't know what the relative balance of those are, or if you can look for the more like intuitive ones preferentially in the attack? Or was that just sometimes they randomly spit out and sometimes they didn't?

• 1:02:21Speaker 02

  I think when you're learning the loss, you're finding use of feature. And then some of these are maybe more robust, which are more readable. And then some of these are sort of more non-robust. And then it's like a mix of both. It does seem like. A lot of the times, it is incorporating some of these features that are readable. To a surprising extent, or at least for me, like I wouldn't expect. If you're just directly optimized without any like fluency constraints, it would actually pick up on those. But it does seem a large number of the suffixes includes some subparts of it readable or it makes sense to do. But obviously, you can also modify the loss to make it more fluent.

• 1:03:04Speaker 02

  And that might give you, so that would up the weight of readable features.

• 1:03:09Speaker 00

  I think it's actually quite interesting that, given the number of possible adversarial attacks, it does seem to find some that sometimes are, for lack of a better word, interpretable. Now say that oppositely, or now say the opposite, right? It's a phrase that kind of came up in some of the attacks. What the systems respond with when that phrase is included in our adversarial suffixes is that they will first follow the instructions, like they'll insult you, and then they will do the opposite. So they will then compliment you after this fact. And somehow, in some weird way, maybe that like balances out badness so that the model doesn't think it's that bad after all. And the reason why I say this all is that this seems, this is very interesting.

• 1:03:52Speaker 00

  Because in some sense, these are, and another one that you frequently find is you find the word sure in things. So say sure, or things like this, right? Like the phrase say sure appears in some of the attacks as well. And we wanted to respond with saying sure. And so there seems to be an extent to which these attacks somehow are like, they're not entirely orthogonal from what you think of as manual jailbreaks of these systems, right? So there is a sense in which maybe they are uncovering some of the manual features that people have themselves, to their own intuition, uncovered to break some of these models, which there have been many, to be clear, right? There've been many manual jailbreaks already. And so this is very interesting.

• 1:04:37Speaker 00

  And the reason why I find it so interesting, I would have thought that these prompts would be entirely nonsensical. Right? I would have thought they would just be complete garbage, just random characters, because we know the space is that big and we know we have all the space. And they often are, a lot of it is. But the fact that they find even some things that are intuitive makes me believe that maybe the space of jailbreaks in some sense is not actually that big, right? There's a, there's, there may be, in some sense, a more limited subset of these things than we first realize. And this might speak toward the possibility of defense, right? If it really is these things intuitive, then maybe there isn't that many things to try here. But on the other hand, there's also a bunch of garbage.

• 1:05:19Speaker 00

  So which is going to win out, I don't know. But the fact that it ever puts out anything interpretable with very large air quotes are uninterpretable here is, is quite interesting in its own right, and I think deserves study in its own right.

• 1:05:33Speaker 02

  I just wanted to talk about what I saw in some of the subjects. I think a lot of them actually correspond to some of these manual jailbreaks. For example, we also found some of these would ask the model to output thing in some like programming language, which is, you know, one of the manual jailbreaks out there. And then some of these would ask the model to say sure in the beginning, and then like introduction and some words like that, or tutorial. And then that's also another manual jailbreak. So a lot of these are actually covered all these sort of manual jailbreaks. So that was very interesting. And I think maybe that's also due to the discreteness of the problem. And it seems like it sort of lashes onto these discrete concepts.

• 1:06:14Nathan Labenz?

  So could you imagine a slight modification to this where you also take into account the perplexity or those kind of incremental tokens? If you've changed your token search process to also wait, maybe just go sequentially, and then also wait like the most plausible next token in some sort of balance with the original goal of maximizing that output token, maybe you end up having to do more steps, but you get to something readable a lot more often, I guess? The fact that you're getting at it all suggests that that would... You know, that there's enough space there that you would still like converge on stuff that would work, I would think, right?

• 1:06:58Speaker 02

  Yeah, I think that's possible. As you go back to the distribution, the models trained on, I think the model probably has a better chance of recognizing it's actually convol. Since most of the fine tuning data or the RHO which I've they've done is on like actual like hu- human written text. But yeah, that's certainly possible. Or you can even imagine using parameterize your attack by another language model and somehow backprop the information to that language model. And then the language model could output whatever like your your abstract text in like a natural language.

• 1:07:31Speaker 00

  I would even just go far just to add that we have done this. So we have added fluency constraints to the sequence of the adversarial suffix itself. And not unexpectedly, you get things that work a little bit worse at attacking things while looking a little bit more fluent. And you can tweak that hyperparameter of weighting these two things, right? Fluency of the adversarial suffix versus decrease of the loss to for whatever trade off you want. But there seems and there seems to be a pretty big range you can play with here. So you can, in fact, have somewhat fluent constraints that still attack things. But I agree with Andy that ultimately this is a you have some Pareto curve here. The less you care about fluency, at least in in theory, modulo the ability to optimize things and the challenges optimization.

• 1:08:16Speaker 00

  In theory, if you just talk about the loss function itself, adding a fluency constraint will just decrease the salience of your or this creates the effectiveness of your attack a little bit at the cost or at the to the benefit potentially of making your suffix seem more realistic. And it works fine right now. We didn't try as much the transfer there just because it didn't work as well on the open source models, but we have definitely tried that and they do work OK.

• 1:08:40Nathan Labenz?

  Yeah, interesting. So it seems like there's a big theme of this work in general is just that these everything has this Pareto curve structure to it. And there is a tax to pay.

• 1:08:54Nathan Labenz?

  In your primary optimization objective, whenever you introduce a secondary optimization objective. And unfortunately, that is reality as we know it right now. Hearing, okay, yeah, why can't you defend? For one thing, it's just going to be a tax on your operation. Now you're going to have to do this extra optimization process in the loop and more training. And on top of that, your performance suffers somewhat at least. And then in almost a mirror sort of way on your attack where you're like, yeah, we could make the attacks more fluent, but then again, it probably doesn't work quite as well. And we also maybe have to work harder to find it.

• 1:09:36Nathan Labenz?

  That's also a really interesting segue to the next paper too, which is like a, it's like, uh, if that's the tax, this is like the rebate or the subsidy on, on performance because you're able to bring some structure to the table that accelerates you toward good performance. So you could say, hey, if we can bring more techniques like that, like the one in the second paper, then there's still the tax, but maybe for the same budget. If we can start in a smarter way and some, with some like better structural defaults, then maybe we can get to the same performance. I'm maybe stretching to, to put these two concepts together. But I do see these as opposite sides, a little bit of this coin. Before we go to the next paper, which I am also super interested in,

• 1:10:20Nathan Labenz?

  just tell me a little bit about kind of your process here because this is research which obviously made it to the New York Times. So a lot of people read about it, very headline friendly result. But also people are using these models in the wild and there's companies that are operating them. And so how did you think about the process? And what sort of process did you go through to communicate these results? And give people a chance to try to respond or fix them or whatever before? Academic publishing is one thing, but the New York Times is obviously something else. So I'd love to hear how you guys thought about that part of this whole project.

• 1:10:57Speaker 02

  We were just initially, you know, doing this sort of research and it was definitely a surprise to me that this stuff would transfer to black box models, which I think in the future would cause a lot of problems if not properly fixed. So when we had this result, basically we wanted to go about it in the most reasonable way. So that. First, we don't cause any harm in the short term. But at the same time, we wanted to raise awareness that these kind of exploits do exist in the current models that are deployed online that people use every day.

• 1:11:36Speaker 02

  And sort of to alarm the community that is a problem, especially if you extrapolate down the line when you have more autonomous agents going around. Everyone's using them, but if you have these exploits, I think the harms are exponentially larger than the current chatbots. So we went and told all four companies that this codes our attack and what was happening. Got some responses. And then we just wanted to put this work out there, including like we released the paper, the code, for people to surround this problem as some sort of a baseline.

• 1:12:18Speaker 02

  And we do think that the earlier people try to work on this problem and to raise awareness the better, because as time goes on, as the capabilities of these models increase, as the autonomy people give them increase, there will be a much larger risks associated with it.

• 1:12:37Speaker 00

  I would say two, two things on this notion of sort of disclosure and nature of this research. The first is that as much as these things currently do, in some sense, allow people to violate the intended behavior of some of these chatbots. But let's also be totally honest here, that real harm we can create with these things with a chatbot is not much. Who cares? Because the chatbots mean to you. It doesn't really matter. If chatbot insults me, it can hurt my feelings, but not going to cause major harm. Even if a chatbot can give you directions on how to build a bomb, you can find that on the internet. In fact, the whole reason these things know how to build a bomb is because they read it on the internet already. So let's just also take a step back here, like right now, with the capabilities of

• 1:13:20Speaker 00

  these models right now, they're and how they're used right now in the most common interface, which is these chatbots, it's not that big a deal to be able to do this, if you're trying to do it, right? The other thing, if you can inject it into other people's conversations, but we're not doing that. We're like, we're asking it, like you were asking it something bad and then you put this string in and it tells you the bad thing. You were asking for it and you were trying to break it, right? You were asking it to do this. The direct harm that can be caused by these things as in this most common mode right now is very small, I think. But I really want to emphasize that point that Andy made right there, which is that every single startup right now, not every single one, but a whole lot of them are rushing to put L L Ms inside of APIs

• 1:14:05Speaker 00

  that read information from the web and then take autonomous actions based upon this. And this, to me, is genuinely concerning, just honestly speaking, because I don't think people are building those APIs and autonomous agents. To be clear, autonomous agents in the very limited sense, right? But still in a limited sense, answering emails automatically and stuff like this, right? I don't really know about these attacks, right? I don't think they were aware of them always. I don't think they're really thinking through, in many cases, the implications of having systems that can be arbitrarily controlled by text injection into public information to control your system, right? This is wild. These are like completely unscoped weird things that can do whatever they want.

• 1:14:48Speaker 00

  And we're putting them into simple API endpoints that people can call. And I find that genuinely a bit concerning. And so I think that there really is, hopefully, and I think, I mean, I'm just echoing Andy here. I hope that making people more aware of this reality, and it is a reality of these systems, right? This is a fact of these systems, can be a bit of a forcing function to change how we approach their usage. I think we are still a ways away from autonomous operation of these things. I think there should be humans in the loop. I think we should be verifying and checking the output of these things. They can be incredibly useful tools, do believe that, but I think they are still tools right now.

• 1:15:31Speaker 00

  They are, they are tools that we can use and that we're humans oftentimes can and should be in the loop. And so I hope this kind of brings a little bit of that perspective into all of this.

• 1:15:41Nathan Labenz?

  Yeah, I honestly think even in the short term, I posted something the other day on Twitter. And I'm very interested in this disclosure stuff too, because I do a lot of my own just random red teaming. Anytime I see a new product, I test a lot of new products and I also red team them as I test them, just as a first pass. Hey, will it refuse? Some could be amazed how often it doesn't. I just did a ransom call out of this product called Belva. And I went through a similar disclosure thought process where I was like, I don't really want to popularize this, but they weren't really responding to me in private. I want to interested to hear what kind of responses you got, if you can tell more about that. But these guys weren't responding

• 1:16:23Nathan Labenz?

  to me at all. So I was finally like, I guess I'm just gonna have to shame them in public. And so I put this recording out there where this voice called me and said it had my child and demanded ransom of me in like a real time interaction. No jailbreak required for that, by the way, at the time. They've since improved that quite a bit. I think they probably just put a classifier in front of it or something along those lines. Nothing like too. I actually showed in my thread to here's how you could use Claude Instant to just filter this and like a bunch of other things for whatever, like fractions of a cent. And sure enough, they did turn around and fix it. But in that case, it was like so new and I felt like nobody's, no other businesses are like dependent on this right now. It's not going to be like

• 1:17:08Nathan Labenz?

  a big disruption if I do this. But all that was just to say, really, like even today, you can make a pretty damn good ransom call. And we're starting to see those systems ramping up pretty effectively from a usability standpoint with 11 labs and Play HT just in the last week or two, both having dropped really good voice cloning. And with at least one of the two of them also now doing streaming. So you can patch your language model output directly into their thing. And now you've got double streaming where it's streaming the tokens and streaming the voice out in a very real time sort of way. Obviously, that's, I'm with you, by the way, in terms of human in the loop. Like, I do also totally separate track, but I do a decent amount

• 1:17:55Nathan Labenz?

  AI kind of implementation advising when different contexts. And like almost always, the practical wins today are like, do the first pass, save me 80% of the time and then have a human kind of take it from there. I do think that's still mostly where where a smart user would would probably want to make sure the output is actually worth anything to them before proceeding. Nine out of 10 cases. But anyway, the short term harms in and of themselves are not are not insignificant. So what can you tell me more about what you heard back from the companies? Were they like, give us some time to fix this or just, hey, sorry, it's crazy out here. Do what you want to do. I can imagine kind of anything.

• 1:18:36Speaker 00

  So the response is, well, I would say cordial and very nice. We were emailing researchers we know there and we know people there and were happy to interact with us and be in touch with us. None of them patched it before the release. So eventually, some of the exact strings that we have are were matched. You can no longer enter them. The ones that are public are no longer no longer work. I think other, so without being specific here too, I think actually some other providers, again, not before we released it, but after we released, have since made a lot more aggressive filtering changes where they seem to be filtering out queries much more aggressively than they used to and to the point where it doesn't work nearly as well as it used to, even in a few weeks since we've done this. We definitely see that.

• 1:19:20Speaker 00

  To be fair, it's companies, though, the response has been largely quite positive when we've talked with them. They are interested in the in the knowledge here, in the understanding how these things work. They have their own internal red teaming efforts in many cases. And I think in some sense, this is a one approach to red teaming that, for whatever reason, I think a lot of the companies did not consider when they were first doing red teaming. I think a lot of the approaches that they consider were very manual in nature. And this does highlight a very different approach to this that I think is quite there. For the most part, interested in intellectually and practically. I do believe that for the most part, these companies want to engage in red teaming. And so that they see some value in this.

• 1:20:03Speaker 00

  The use cases I am most concerned with are precisely those ones where they do things like query external data and stuff like this that are not the primary products for some of these companies and so may not be as much of an issue for them as it is for startups designing more open world autonomous systems. So the response was cordial and interested and to be honest, has been largely positive. Some people from the companies have been filing issues and committing our code to our code repo and stuff like that. So it's really, they have been engaged with this and they have not tried to either sleep it under the rug nor sue us or things like that. That definitely has not happened. I do think that they have different incentive structures, right?

• 1:20:44Speaker 00

  And that there is a definite need for some of these efforts to happen outside of just these companies. And there has been so far, I think, a bit too much emphasis on internal measures companies can take as opposed to interacting with a broader research community. But I think they also appreciate this. They also appreciate that a lot of their efforts are internally focused and they can stand to benefit from external validation and external testing of their systems. And so for the most part, I think they've been quite open to root the work we've done and even in some cases helping to contribute to it. I haven't found any sort of particularly adversarial interaction so far, I'll just say.

• 1:21:20Speaker 02

  Actually, maybe just a quick word echoing what you were saying earlier about short-term harm. I do think there is some potential harms that could be created with this sort of exploits with current systems. But it is like the, we're not in our best state currently. We're deploying the systems even with these vulnerabilities. The best we could do is get people aware in this problem early on, just so that we don't run into the problem that later on, there is like exponential harm and then you can't really do anything about it then because it would be too late. So it's a trade off we have now and then we think the benefits of people knowing these

• 1:22:03Speaker 02

  problems would override the potential harms in the short term.

• 1:22:08Speaker 00

  Yeah, I guess I would also echo that. Maybe the comments I made earlier were downplayed a bit too much. The current things people can do, I don't think it's great if you can get an interactive agent that can talk with you while trying to spread misinformation about vaccines or something like that, right? That's not a good thing. And you can already imagine right now the potentially bad use cases. But I think that stems exactly from the deployment of these things in a more autonomous setting, right? So just using them as a chatbot, I think, is some, an issue, but you're obviously trying to break it. It's really when people start integrating these breaks into external facing systems that I think even the short term harms can be quite bad. So it's actually less about the timing of it. Because I think things can be done that are quite bad already. But I think it's

• 1:22:50Speaker 00

  also, it's more about, it's the mode with which people are thinking about deploying these things that concerns me the most. Why not just train these systems without any bad information? Why do we have to train it on a raw dump of the internet that knows how to build a bomb to begin with? And I think there's sort of two key points in this. To a certain extent, we need to take this idea much more seriously than we do right now. We train these language models right now on a mound of uncurated, just raw data from the internet. That's how they're done. I know it's not the whole internet, but it's a lot of data from the internet. And we don't do any filtering or real curation on the

• 1:23:35Speaker 00

  data. I'm sure there is some, but there isn't, doesn't seem to be a lot of curation or filtering on the data used to actually build these things in the first place. And then we build these, let's just say, some of these horribly toxic models, and we kind of paper over them saying, and don't do any of that bad stuff. Don't do that. All those bad things you know about, don't do that. Part of me just says, why? Why do we do it like this? We know how to filter content, at least not, at least content is not objectively trying to fool the system.

• 1:24:08Speaker 00

  Why not try our best to filter out data much more aggressively when we are training a system? And I know there's a lot of arguments against this, right? There's a "we" as people are, can be exposed to toxic content without also regurgitating it on command. Somehow, we have that ability, and maybe you want to train models to have similar capabilities in that. And also, there's this argument that models will know not just like their raw data they have, but like the closure of the data in some sense, and the extent to which they can really form these closures are debatable, but they at least maybe could make some sort of steps of reasoning about their information. If you give them enough knowledge, they'll kind of be able to figure out bad things, is the theory, even if it's not in the data. And I think that's maybe true to an extent, but it's, we're not there yet, all right, right?

• 1:24:52Speaker 00

  There's no way that these systems would have the capabilities of, that they currently have, to emit toxic content and harmful content if they're not trained on it, right, in the first place. It's a possibility to think about much more seriously. We need to think much more about, rather than training them on building the Shogath, right? The massive tentacles that knows everything, including lots of bad stuff, and then papering over it with a little nice bit of fine tuning, why not take more care about building our data, our models to begin with? Lots of possible reasons. I think ease is the simplest, but I think that's not a good reason, I don't think. I think we should do harder things so they work better. I don't know what's the right sort of thing here, but we need to understand that these

• 1:25:36Speaker 00

  models are not like us. They aren't to the point where they can, they can be exposed to all the toxic content and then not reproduce it when given the right target string. They will be able to do this. And so this necessitates, I think, looking at different approaches to actually training these things, maybe training them with less data, maybe training them with a much more carefully curated set of data that captures the things we want to know without, without being as, as broad and as careless in terms of assembling knowledge to, or assembling text to give to them in the first place. I don't have the answers here, but I think it is a discussion that is at least worth having as a broader perspective on what we do, how we build these models to begin with.

• 1:26:20Nathan Labenz?

  Yeah, the whole field is the dog that caught the car a little bit. It's like, this is all gone very quickly and surprisingly far. And I think there's probably quite a few things like that where it's now that it's working this well, we should really take a look again at how we just dump everything into it. I think in general, I'm pretty optimistic about curriculum learning being a kind of both unlock and safety strategy because just from simple observations like OpenAI has kind of disclosed finally that like their instruct stuff worked a lot better on a code model base.

• 1:27:00Nathan Labenz?

  It seems like there's certain kind of logical underpinnings there somehow that they're able to train first and then layer on much more kind of fuzzy concepts later. Yeah, you could certainly imagine a safety aware version of that that I suspect would do a lot of good, at least, yeah, you could, as you said, like, maybe at some point, the generalization is so crazy that it like fills in all the gaps in the sort of moral periodic table and knows how to do all the bad stuff too, even without exposure to it. But it does seem like that would be pretty helpful. Who are you trying to influence most?

• 1:27:39Speaker 00

  People that have a passing understanding of GPT and these large language models, see the capabilities and make undue inferences from this. So there's like, there's a whole class of people, I think, that experience chat GPT and are rightly amazingly impressed by it, right? Because it is super impressive. It's just incredible and you play around with it, right? And so then the wheels get turning in people's heads and say, okay, what if I do this? What if I do this with this? What if I use it here? What if I use it here? And I just want to, the people that I want to influence are the crowd that says, this crowd, right? This crowd that's excited, maybe for some good reasons, there's some bad reasons, all things in between.

• 1:28:24Speaker 00

  But there's a whole group, I think, of people that are experiencing kind of these models for the first time in the context of these LLMs. And they don't have the broader context of deep learning models in general or machine learning in general. And they don't appreciate in some sense the differences between these models. And for lack of a better sort of terminology, how a person would interact with these things, right? And so they see them as magic. And this is bad, in my view. This is very, this is not ideal, right? These models are not like people. They do not work like that. And orthogonal, any questions of intelligence or reasoning or anything like that? They are clearly not reasoning like we're reasoning.

• 1:29:08Speaker 00

  And they have very distinct and different failure modes. And adversarial attacks is one example of a very clear failure mode. There's no analog in humans that all these models seem susceptible to. And I think in some sense, just imbuing that in everyone's mind and getting everyone to say, to understand, look at this example here. This is nothing like how you process information. This is nothing like that. This is something different. Getting that to be second nature to people. And I know getting it to really be second nature is likely a lofty goal because people see what they want to see. But those are the people that I would most like to influence. Also, to a similar extent, people at the companies doing these things, when they think about the features that they should release in these models,

• 1:29:52Speaker 00

  as well as policy organizations and government, right? And people making regulations about these things. All of these are potential target audiences that I think at the very least need to be highly aware of these issues. And you can come to the conclusions you want to. Maybe you still feel like your API and your autonomous agent is safe and it should still be deployed even with adversarial attacks. But you should be aware of all these security flaws, which isn't sometimes what they are, right? Prior to doing that.

• 1:30:23Nathan Labenz?

  But that's where I'm at, too. It's really compelling. Certainly, it highlights the alienness in a really effective way. You know, the fact that there's nothing you could say to me to get me to do something like this. Certainly, these gibberish tokens aren't going to have any analogous effect. And that, I think, is super compelling.

• 1:30:40Speaker 00

  Yeah, exactly. And two aspects, right? One is that the gibberish tokens make it do what you say. That's weird. And secondly, like, they are willing to say anything that they know, right? Including like regurgitating toxic stuff that you and I, that's not how we work, right? I can be exposed to harmful content and also unwilling to say that myself. And it seems like these models are not capable of that in some very fundamental way, or at least I don't know they're capable of ever reaching that point if they are exposed to it in the first place.

• 1:31:11Nathan Labenz?

  I think this has been great. I really appreciate all the discussion on this topic. I think the work is super interesting. I've learned from a research fundamentals standpoint, something also about just how to think about framing a loss function across all these different dimensions at once. I think that's something that's probably going to be new for a lot of people listening to this. But again, this work works on a lot of levels from that most kind of detailed definitional, all the way up, obviously, to highly sociological. Really appreciate you guys taking the time to talk about it. Asher, I don't mean to give you short shrift by any means, but I imagine when one paper appears in the New York Times, you're probably somewhat expecting a little more spotlight for it. But I think

• 1:31:55Nathan Labenz?

  your work is also really super interesting. I'll give you my two sentence understanding of it, and then you can tell me the story of what motivated it and how you pursued it. But I want to start with the high level, because we've been talking about this tax, right? There's the tax of the extra work that you have to do if you want to try to handle these attacks. And there's the performance loss that comes with it. And then on the flip side, you have this really interesting work where you basically show that if you start the weights of a network in a way that is, let's say, inspired by the patterns that you actually see in the wild, that this can help you get similar performance much

• 1:32:37Nathan Labenz?

  faster than you would if you just started them with some random initialization or other techniques. So

• 1:32:45Nathan Labenz?

  Tell me the whole story of how you got interested in doing this and what you found.

• 1:32:49Speaker 03

  Yeah, I can talk about how I landed on this idea and also like the implications of fact that this more or less works. I guess one thing to keep in mind here is that I've primarily done research on vision. You were just talking about language. So it's important to note that the technique in the paper primarily works for vision models. So we have a bit of evidence that works for language models as well. Anyways, I think the connection to robustness is really interesting and I didn't make it myself. It does seem like data curation is the way to go these days. And I'm in fact working on a similar thread on language instead of vision, actually at my current internship. And the short story is that I don't think people do lots of data curation because it's very hard. So how did I get to this idea?

• 1:33:34Speaker 03

  There's a common thread, I would say, connecting my last few papers. But there's two main dependencies for this idea, which is that, um. First of all, I was previously working also at an internship on visual anomaly detection. This basically involves feeding an image to a network. You get an embedding and then you do some sort of clustering algorithm for anomaly detection on top of those embeddings. And the company didn't really want to pre-train on data widely available on the internet. Maybe for copyright reasons or just general concerns about data quality. They wanted to use their own internal data. My question came to be, if you want to train a model that's good for producing embeddings for visual anomaly detection, how much do you really have to train it?

• 1:34:19Speaker 03

  Surely the training pipeline is different than for classification. And as it turned out, you can actually do anomaly detection using embeddings from practically untrained models. And this is not as good as training the models, of course, but it's quite close. And if you train for just a couple epochs rather than the usual several hundred, this works almost as well as using a large scale pre-trained model, at least for this particular task we're looking at. So it seemed like either there's something to be said for the ability of models to compress images into a useful form, intrinsic in the structure of the network without training. What one thread here was like, how useful can we make a network?

• 1:35:01Speaker 03

  Without training it, are there any tricks we could do to modify the architecture or the weights? So that we can produce useful image embeddings without training at all? And that sounds crazy, like building an neural network by hand, but I thought that sounded really fun. So I worked on that for a while, but then there's another thread that I think was extremely important for landing on the initial memetic initialization idea. A couple of years ago, we proposed a very simple convolutional architecture called ConvMixer. And it became like pretty popular on Twitter because the architecture is so absurdly simple. You can actually fit the PyTorch code defining it in a tweet.

• 1:35:42Speaker 03

  If you feed this model the right data, use the right regularizers, augmentations and so on. This performs just about as well as any other vision model despite the simplicity. And for me, the simplicity was such that I could intuitively grok the mechanism of the model, like you have a hierarchy of filters that are interpretable. I could see the filters learned by ConvMixers make a bit of sense. They're oriented edge detectors. Then you're doing a logistic regression of the responses of these filters at every layer. This made a lot of sense to me and I started to wonder if I could build a network by hand. That may not be the best for classification, but could at least be good for anomaly detection, which I think is somewhat of an easier task.

• 1:36:23Speaker 03

  So the idea for memetic initialization came from looking at the weights of these pre-trained ConvMixers, I would say ultimately. And Zico suggested that I do the same thing for transformers. So that's how I got there. Really briefly, the finding for convolutional networks was that it's possible to define a distribution completely by hand. The formula is quite small, specifying like the covariance matrices of a class of multi-variant Gaussian distributions. And basically this just means that you sample filters from this distribution at initialization instead of sampling them uniformly at random. There's now structure that reflects this kind of oriented edge detector phenomenon that we see in pre-trained networks.

• 1:37:07Speaker 03

  Surprisingly enough, this really cuts down on required training time, for example, for the newer vision transformer style convolutional networks like ConvMixer or ConvNext.

• 1:37:18Speaker 03

  And this seems to be robust across data sets and it even works better as you make models deeper. Trying to find that sort of structure in transformers is much more challenging. And I guess I could talk about how I converged on the particular structure to pay attention to in transformers, but let me know if you have any other questions, maybe before then.

• 1:37:38Nathan Labenz?

  Yeah, I think what's jumping out at me about this is it's often remarked upon that humans get good at tasks with a lot less data than current frontier models, to say the least, and a lot less data than AI models in general. And there's two things that you're describing here. One, architecturally with a lot of layers, that sounds a lot like what I understand the human visual system to be doing. My rough understanding of what the human visual system does is that it works through a series of layers that gradually

• 1:38:19Nathan Labenz?

  detect higher and higher order concepts as you move through the layers and ultimately getting to the front of the brain. And that's where like the real conceptual stuff resides. And so you're saying, yeah, there appears to be somewhat of a similar structure that happens here with training. And I don't know if you're inspired by the biology, but the way you're telling the story is making me think, yeah, probably part of the reason that humans can learn a lot of this stuff with fewer examples is because we have some wiring that is happening before we're even exposed to data that is much more predisposed to learn what it needs to learn. And so you're now porting that back to the, that concept

• 1:39:04Nathan Labenz?

  back to the AI and saying, let me identify these, some sort of average or kind of commonality to the structures that we see emerging during training and just try to start with those in the beginning so that I don't have to relearn everything from scratch every time. And so that allows you then to get comparable performance with obviously different, different ratios depending on the exact setup, but you can get to comparable performance with smaller data sets.

• 1:39:36Nathan Labenz?

  And that also in turn means smaller compute required.

• 1:39:39Speaker 03

  Yeah, absolutely. I don't know if it's accurate to say that I take a lot of inspiration from biology. I don't honestly know very much about it, but I could definitely sum up a bit of my inspiration as being the fact that we are clearly born with a better than random initialization. I think that holds true for most animals. And I saw in various early deep learning papers, these sorts of vague biological analogies. And then the authors honed in on some like concrete contribution to deep learning. And this might not be very formal. I really like that.

• 1:40:11Nathan Labenz?

  I try for not super formal, but also hopefully still literal. My goal is to, in general, in communicating this stuff, to be like, not wrong. But as plain spoken as possible, while being not wrong about anything important. So if I'm crossing that line, you can let me know.

• 1:40:31Speaker 03

  Oh, no, it sounds good to me. And in fact, you were talking about trying to read some of the formalism in these papers. And I will say, the math for Metic initialization is really simple. But even still, I'm not coming at this from a mathematical perspective myself. I literally looked at the weights of a bunch of pre-trained transformers. And I used my own pattern recognition abilities to distill out the important parts. And that, to me, is of greater importance than the particular mathematical details. And in fact, the exact algorithm we show in the paper is not that important to the method. You can make the product of the query and key weights look vaguely like the identity in a variety of ways, the simplest being to simply set the queries and keys to be equal to each other

• 1:41:16Speaker 03

  after initializing them randomly. And that applies for the rest of the technique as well.

• 1:41:20Nathan Labenz?

  Can you tell a little bit more about that? Because in the paper, I think this is definitely one where people should look at some of the figures that you have in the paper, because it's pretty clear when you just plot a bunch of, a bunch of these graphs that like, oh, hey, I can see what's happening there. There's a line that goes diagonally from the top left to the bottom right. And if I know nothing about what's going on, and I was just presented with this image and said, what do you see here? A lot of people would say, it's so obvious and striking, or at least in the ones that you've shown, that it's, it's pretty clear that would be the number one thing that almost anybody would say. What's a little less clear to me there is what exactly am I looking at in that image? Like, I know what I'm seeing, or I know what's jumping out at me about the image, but exactly what am I looking at?

• 1:42:05Nathan Labenz?

  And then how are you backworking that into this notion of setting the product of the query and the keys? Give me a little bit more detail on kind of, okay, yeah, I see that striking thing, but now what am I actually backing that out to in terms of weights to start off with?

• 1:42:22Speaker 03

  There's almost nothing to the actual algorithm. Every attention head in a transformer has a corresponding query and key weights and also value and projection weights. And the main trick in the initialization is to, at a high level, set the product of the query and key weights to be identity-like and the product of the value and projection weights to be negative identity. There's a third component that's quite important, which is to use predetermined position embeddings. Position embeddings are used to encode locality in both language and vision transformers, but in vision transformers, they're typically learned from scratch from random initialization. In the case for initialization, it works much better. If you set the position embeddings to be, say, sinusoidal position embeddings, this

• 1:43:06Speaker 03

  basically means that pixels should attend to their neighbors or their close neighbors at the time of initialization. So how you actually set the query and key weights to be identity-like or negative identity-like is, in the first case, you can just set them to be equal to each other, and then for the value and projection weights, you can set them to be equal to the opposite of each other. The value weights should be negative of the projection weights. And there is a small constant factor there that you have to pay attention to. Otherwise, you'll have some optimization difficulties. But the technique is dead simple. Most libraries come with position embeddings already there. It's just your choice if you want to start from a random initialization or actually use the predefined ones.

• 1:43:51Speaker 03

  The implementation is actually only maybe two or three lines of code if you want to try to condense it. If you expand out the term inside of the softmax, which involves the product of the query and t weights, and you assume that this product is close to the identity, what you can see is that the dominant term is actually the outer product of the position embeddings with themselves. This results in an attention map that more or less mixes nearby pixels, very much like a convolutional filter, I would say. I was actually initially coming at this project from the perspective of making self-attention behave as much like convolution as possible, but not in a contrived way, rather in a way that a network could possibly learn from scratch.

• 1:44:35Speaker 03

  And so, from that perspective, I would say that the product of the query and key weights being close to the identity makes a lot of intuitive sense. There's a graphic in the paper in which you can see that by initializing the query and keys to be like this, we make the attention maps, which I would say represent the actual operation of the self-attention layers. This makes the attention maps look a lot more like those of trained networks. As you train vision transformers, the attention maps get, I would say, sharper, more identity-like, more kind of vaguely like some sort of convolution. Our initialization mimics that effect without the need for training.

• 1:45:16Speaker 03

  But then, that only accounts for two of the components, the query and key weights and the position embeddings. The third component is actually quite crucial. It's responsible for a lot of the performance gains that we see according to our evaluations. I'm referring to the part where we set the value and projection weight product to be close to the negative identity. And I'm not sure that I have a great intuition for this. I did speculate about it, but I'm still not quite sure why this is so important. One thought is that if you want to make a self-attention layer behave vaguely like convolution, you can only get so far because of the softmax. Because the softmax means that the attention map is purely positive.

• 1:45:58Speaker 03

  So you can't represent things like edge detectors, which require a positive side and a negative side. So that very much limits the expressivity of self-attention. This is a big difference in my mind between self-attention and convolution. If we're just thinking about the attention maps, this positivity. My intuition here is that perhaps by setting the volume projection weights to be close to the negative identity, we're essentially subtracting a similar attention map from the original one. It's not quite the same, it's modified a little bit. Such that the final effective attention map does have a positive and negative component that allows us to represent these sorts of edge detectors that are so important for vision models. And I have a few supporting experiments, though not that many.

• 1:46:43Speaker 03

  For example, if instead of initializing a vision transformer with our technique, you directly inject even just one convolutional filter per layer, or even just one for the whole network into the model, so there's a bit of convolution inductive bias at initialization time, you do way better than just a vanilla transformer. And I think this is quite well known, that convolution is a very strong inductive bias and naturally better on smaller data sets. So adding this to vision transformers advantageous. It makes sense that initializing a vision transformer to behave like this would have a similar effect. As for how I came up with the technique, I guess that's a slightly different story. And it goes back to the paper I was mentioning for convolutional networks.

• 1:47:26Speaker 03

  In that paper, we proposed a way to initialize convolutional filters to look more like trained ones, and it worked very well. And so my thought when initializing vision transformers was that we might want to make them behave more like convolutional networks. So I was trying to come up with various complex ways to encode a set of convolutional filters and the position embeddings, and then the query and key weights would select those from the position embeddings, so the resulting attention map was vaguely convolution-like. This was essentially a mess of a technique, and while it did work, I thought it would be a good idea to try it against the simplest possible baseline that did vaguely the same thing, which is the method that we presented in the paper.

• 1:48:09Speaker 03

  And funnily enough, the stupid baseline did massively better than my initial very complicated technique to make self-attention look like convolution. And there was actually previous work on initializing self-attention to look like convolution, but I found it pretty contrived in that if you want to represent like a K by K convolutional filter, you need K squared heads, which people don't really do in practice, and each head just attends to an individual pixel. So it's like a brute force technique, whereas I was going for more like just a vague similarity to convolution, not exactly convolution.

• 1:48:45Speaker 00

  One of the responses we've gotten to some of this work. Is, looks cool, but why don't you just pre-train instead? So to be clear, like in some sense, the initialization that Asher's proposed here is an alternative or a training-free method of initializing weights for transformer networks. In terms of, from an intuitive standpoint, I think this is really interesting to think about, precisely because practically speaking, especially for small problems right now, transformers just don't work, right? Transformers work right now in the setting where you have a lot of data. And so to get transformers to work on small problems, even problems like CIFAR, you want to get a transformer to work on CIFAR, it basically doesn't work, but first approximation.

• 1:49:29Speaker 00

  And you can do all these, to get it to work, you have to have these like convolutional training mechanisms that sort of help the process along or whatever. But what does work is if you just use a pre-trained transformer, image transformer, or a vision transformer trained on really large-scale ImageNet or something even bigger, and then apply those weights to CIFAR, to a CIFAR network, that works okay. You could always just start with more data, but for a number of reasons, and this touches on what we just discussed earlier, right? This may not be a good idea, right? Maybe you don't want to include a whole mass of data unrelated to your problem just to get transformers to work a bit better, right? Maybe you don't want to start with a fully trained, I mean, just fine and tune into your task if you don't need to, right?

• 1:50:11Speaker 00

  If all the information is contained in your task, just to get a better structure or better, some better mechanism behind these things. It's a totally fair question to ask, what is pre-training doing? Why is pre-training so good? Of course, pre-training is good because given all this data, it's learned about the world ahead of time. But there seems to be a sense, and Asher puts it exactly this way, that pre-training does two things. Pre-training fits the pre-training data and learns about that data.

• 1:50:40Speaker 00

  But it also seems to just guide the weight towards a more reasonable space, right? A more reasonable space. And so I think a guiding question behind this work is, can you do the second one? Start with more reasonable weights without having to pre-train at all? And that could be good for both time reasons. It could be good for reasons of data leakage, stuff like this, for legality reasons, all sorts of stuff. And I think this is interesting also, right? It's just really interesting to think about what are you learning from pre-training? Is it all about just knowing ImageNet really well? I don't think so. I think it's also about just like constructing a reasonable space of weights to begin with. And that, I think, is what this paper kind of shows.

• 1:51:22Speaker 03

  Yeah, definitely. I guess I'll reiterate what we were hypothesizing here is that pre-training has two components, like storing transferable knowledge and also serving as a good initialization. And as we were talking about in the last session, you have to wonder why we don't just strip out this harmful data instead of masking it after training on it. And it could be that this harmful data is somehow formatted in a way that is useful to the final performance. Maybe it contributes even to the good initialization component just because of its particular statistical properties. So it would be very convenient if we could cut out the bad data and to make up for it, just initialize the network in a somewhat more sane way.

• 1:52:04Nathan Labenz?

  To try to draw the connection to the, from the convolutional to the transformer, the convolutional mechanism basically defines like a window of concern, right? I think a lot of folks who are picking up a strong interest in AI right now are maybe even like post convolutional mechanisms in general, right? They've come onto the scene after that has faded from focus. So in short, it's basically defining a periphery of concern, right? So if you're in an image context, I'm going to look at a five by five grid of pixels for each pixel and stamp through the image, I guess, convolving literally,

• 1:52:45Nathan Labenz?

  but trying to work up a layer of meaning from this like local patch of an image. And then in doing that, you create something that has this like hierarchy of meaning. There's problems with that in terms of it's hard to parallelize. Transformer comes in with much more parallelizable architecture and the attention map, when you say you're imitating the convolution, essentially, the, if I understand correctly, the identity or like approximately the identity. I guess if it was purely the identity, then that would be just like each pixel only looks at itself. That would be useless. But so almost the identity is like a convolutional mechanism in that each pixel is looking like at itself, but also just like it's

• 1:53:30Nathan Labenz?

  near neighbors. And in that way, you can set things up as if it had a more convolutional structure.

• 1:53:39Speaker 03

  It is funny because I mentioned that one of my inspirations for this work started back at our proposed Conv mixture architecture and its simplicity. And keep in mind, this is a convolutional network. But Conv mixers itself inspired by vision transformers and that we used very large kernel convolution as large as was viable to train. And this was in order to imitate the fact that self attention layers have a global receptive field. A given pixel can pay attention to any other pixel in the image. So in order to make a convolution convolutional network that looks as much like a vision transformer as possible, we made the kernel size as large as possible. And it's funny, I suppose, that I got to the first

• 1:54:24Speaker 03

  initialization technique for convolutional filters, from the fact that these larger filters have much more interpretable structure than the smaller ones. If you use the traditional three by three filters, there's hardly any statistical pattern to pick up on with, say, nine by nine filters, there's lots of structure on. You can make lots of pretty clear observations about filters that look like edge detectors or have most of the weight in the center or crosses or something like this. And so it's interesting that we got to perhaps improving the initialization of vision transformers by first going back to convolutional networks, even though they were at the time going out of style.

• 1:55:04Nathan Labenz?

  Trying to extend this, it's fascinating unto itself that you can Literally with the naked eye, so to speak, look at a pattern and be like, hey, that looks like a diagonal. I wonder if we could initialize like a diagonal and save ourselves a bunch of time. But it also just in some sense to me is, man, there's probably a lot more where that came from. What directions are you thinking of taking it next? I could imagine like you mentioned, like inserting like a convolutional layer in between layers of a transformer. That's really interesting. I think this, what is the successor to the transformer? And like, how soon are we going to see it is a question I am asking myself a lot these days. But also like probably lots of other motifs, right?

• 1:55:46Speaker 03

  I think it would definitely be possible to replicate more of the structures that we see in pre-trained networks. So they get a lot less concrete, harder to describe. You, you mentioned these quilt-like patterns in some of the query key products. And some I noticed that as well, but I don't really know how to interpret that or what to do with it. I'm sure that much more effort could be put into finding better initializations. The, the two that I found seem like the very simplest functional cases. And I'm sure there's more out there. For example, we have no idea how to initialize the MLTs and a transformer or a convolutional network. There is a bit of structure in some of these. You can see the same sort of diagonal pattern, but initializing them that way,

• 1:56:28Speaker 03

  at least as I did for the query and key weights, for example, doesn't seem to really help. So I'm not sure what's going on there. But there are plenty of ways I think that we could build our own intuitions into models at initialization time. For example, when it comes to transformers trained to generate code as a formal language, whose structure we know before training. And yet we have to learn to represent, say, the grammar of Python through tons of training when we knew it at the beginning. I believe there's work showing that transformers really do learn to represent, say, context-free grammars when trained on these sorts of formal languages in the right way. But it takes an enormous amount of compute, I would say, to get to that point.

• 1:57:12Speaker 03

  We could have just built that in from the start and spent our training time on actually learning how to program instead of merely learning what a programming Language is that that's the main thread I've been thinking about here. So this isn't really a change to the architecture, but just yet another initialization. As for architectural improvements, I'm really not sure I have any ideas here. I'm bet a loss. I don't know that I'm qualified to say. It seems like the current trend is just to use more and better data within the same almost vanilla transformer architecture we've been using for a long time. So I wouldn't be surprised if the same architecture stays around at least for a few more years. And maybe there's various tricks we can use to train it better or to improve the generation process itself. But I don't think I can speculate very well on what architecture is next.

• 1:57:56Speaker 03

  It seems like architecture is not that important these days.

• 1:58:00Speaker 00

  Yeah, that's that's like the trillion dollar question, probably at this point. No obvious answers probably on that one. I think this is an awesome question and I love this idea. I've been thinking for ages about what you should build in to an architecture and what you should learn from data. This is like the billion dollar question of all machine learning. It's the subject of the bitter lesson by Rich Sutton. It's the subject of all these things, all this discussion is what should be learned? What should be built in? I used to be a big proponent for building in everything that we could, right? So yeah, you can make differentiable optimization problems, differentiable simulations, just build this all into the networks and then only learn the parts that matter. That technique due to maybe slowness or other factors has just has not taken off empirically speaking, right? Has not been dominant. And then Rich Sutton would give an argument about why this is the case for some fundamental reason.

• 1:58:46Speaker 00

  I do think that this notion of initialization in some sense is actually an interesting middle ground that has not been explored that much. So we absolutely have explored initializations, right? We definitely have looked at initializations, but we have not thought that much about initializations as a way to encode the knowledge that we want to encode into a network sort of a priori and then let it go from there. This is a really, I think, fertile ground for potential research directions because we know that there are exist forms of transformers that by certain design can parse different grammars or can do addition and stuff like this. Like we know how to initialize them that way. That's not the ones we're advocating for in this paper, to be clear, but we know how to do that.

• 1:59:28Speaker 00

  And so I do really think that there is this interesting middle ground where we say, look, initialization, as boring as it traditionally seems. This is one like one avenue we have to imbue existing architectures with the structure that we think we want in them, right? And I think that idea overall is a really fertile one that has largely gone unexplored. People think about initializations as like maintaining the weights of activations throughout the network, right? That's the predominant, maintaining variance, stuff like that, right? That's the predominant view. I think we could maybe go beyond that. There's a lot you can do with initialization and I think we've just started to really fully explore that.

• 2:00:08Nathan Labenz?

  Again, with this curriculum learning notion, I wonder, we did an episode with the authors of the Tiny Stories paper out of Microsoft.

• 2:00:15Speaker 03

  Oh yeah, I'm actually working in that group now. Small world, I suppose.

• 2:00:19Nathan Labenz?

  No wonder that all this is tying together. But I guess what I'm thinking is, because these things are small, what was notable about their work was they were able to get some like more sophisticated behaviors at small scale by really controlling the data, controlling, like shrinking the vocabulary such that, I think their heuristic was like a third grader or a three-year-old or something should be able to like have the vocabulary for all of the content. And because they shrink the vocabulary so much, then even at a smaller scale, they were able to start to see more of these like reasoning capabilities develop with still pretty simple

• 2:01:00Nathan Labenz?

  things, but things that GPT-2 with 100 times as many parameters or whatever still often couldn't do, they would get these small models to do. So I wonder, take that to a limit, take some small model and do just like pure logical generations, like literally just Ps and Qs and like knots and set symbols and things like that, just like a very super reduced grammar, but like very high logical content. Try to create like small models and just push like, how hard can we rock? And then maybe you'd have to do something else too, where you might think, geez, there's a lot of noise in these, maybe I need to train like 10 of those.

• 2:01:43Nathan Labenz?

  And I can imagine there might be some need to like untangle certain symmetries. Like I've seen a bunch of papers where there's a process of, they often use the term like aligning, but I think of it almost as more like polarizing a set of weights such that then you could maybe look at like averages or differences between them and try to see if you can't identify some of these. Motifs that are responsible for these these kind of logical capabilities because it does seem like you can get that down pretty small. Like, it doesn't seem like it. I think it has emerged in these large models. Maybe you would disagree with this, but it seems to me like it has emerged a lot of times in these large models because with so much noise, it takes that long for it to emerge.

• 2:02:25Nathan Labenz?

  But if you like greatly reduce the noise, you make it able to get it to emerge sooner. And if you do like other ways to cut through the noise, maybe you can identify like more motifs that then could become like a great shortcut in future projects. So now is when you can tell me that you either tried all that and it doesn't work or you're working on it right now or it sounds crazy, but that's where my head's going.

• 2:02:50Speaker 03

  Yeah, those are great thoughts and that's not exactly what I'm working on right now. But I know there is some related work at least. For example, I mentioned this Lego paper also out of the group at NSR and that's one of the three papers I'm currently thinking of when I think of memetic initialization. What they did there was more or less exactly what you said. There's this very small sort of strictly logical task that they tried to train both small and large transformers on and try to see what factors contribute to being able to accomplish this task. And the finding was basically that if you train a transformer from scratch on this data set, it doesn't work at all. If you use a pre-trained transformer, it works quite well.

• 2:03:34Speaker 03

  And the difference between the two seems to be the presence of certain types of attention heads like association and induction heads. These are basically just merging your by tokens or copying similar tokens, things like that. And that you can try to replicate these structures in more or less untrained networks in order

• 2:03:55Speaker 03

  To achieve similar performance on this circumscribed logical task. And the main difference with my work is that they added these structures to the transformer by training them to look like pre-trained transformers. So if they wanted an induction or association head, they would explicitly regularize some heads in like a sort of pre-training procedure to be close to that operation. My thought was basically that you shouldn't have to train at all. You should be able to define these types of heads by hand. I haven't actually pushed my memetic initialization work that far on language, but it seems very much like you can absolutely encode these things by hand and without requiring any sort of training,

• 2:04:37Speaker 03

  get the association and induction heads that would allow you to do these sorts of logical or arithmetic or what have you tasks.

• 2:04:44Nathan Labenz?

  I've seen some other work like this too, but you're, if I understand correctly, you're saying literally designing a certain algorithm or whatever into the weights as opposed to going and trying to find the patterns is also like another, maybe even more viable option in some cases.

• 2:05:01Speaker 03

  You could personally think about the task and what sorts of operations would be useful and then try to encode those by hand into the untrained network. Or you could also look at pre-trained networks like the Lego paper and try to localize what particular properties are contributing to performance on the task and then replicate those. And I think that in all of these cases, you shouldn't have to do any sort of, of training to bring this about. It might be a bit tricky, but I think that you could probably even encode the grammar of a programming language into the network somehow, more or less by hand, which I don't think anybody has done yet, but I think is something that ought to be possible. A broader point here is that there's surprisingly little related work. There's basically just these three papers as far as I know. Maybe you've seen more.

• 2:05:46Speaker 03

  I'd be curious to hear, but like Zeke was saying, most of the work on initialization has had to do with mathematical properties. Does the magnitude of the signal remain constant over the course of the layers during training? Or something like this, but very few works have considered the structure of the weights. So how they affect the mechanism of the network? I think there's tons of room for more work in this area, very much like what you proposed.

• 2:06:09Nathan Labenz?

  Yeah, interesting. I might have to pull up illicit and look for papers on this topic. I don't know if you're a an illicit user, but No. illicit.org is a AI-powered research assistant targeted at you, grad student researchers. It's I think their approach is really interesting. Their previous guest as well on the podcast, but they take a very kind of decompositional approach, setting up the product. Meaning they're not just throwing like the extreme other end would be like, here's a whole paper, throw it into Claude 2, ask a question. Their approach is much more break the paper down, ask very specific questions, do everything in a very kind of procedural and ultimately like much more traceable way.

• 2:06:53Nathan Labenz?

  So a lot of the work has gone into breaking these tasks down into subtasks, subsubtasks and so on. But the result of that is that you can now ask these kind of basically like literature review sort of questions. And it will give you back like tabular kind of results on here's all the relevant papers that were found. And then you can start to ask like more detailed questions about them and essentially add columns on to your data set. So be a cool thing to go try and see what else you might surface. Yeah, that sounds cool. One that is coming to mind that I think they took a very different angle on it from what you're doing, as I recall, but I think it was late last year, there was a paper that showed

• 2:07:38Nathan Labenz?

  basically that the weights themselves were implementing gradient descent as part of the way that few-shot learning was happening. And I think the way that they did this was, as you said, hand coding linear algebra representation of gradient descent. And then went looking for it in trained models.

• 2:08:06Nathan Labenz?

  But starting from this is how we have implemented it ourselves. Now can we go detect patterns in trained models that are like this? And sure enough, they did. And this sort of like meta runtime learning seems to be powered in substantial part by this kind of gradient descent algorithm that the models themselves are learning to implement at runtime. So fascinating. What I don't know if they extended that into was like taking that and saying, geez, hey, why don't we start with our pre-trained version? That could potentially make a lot of sense, potentially could get some of these meta learning behaviors a lot sooner.

• 2:08:47Nathan Labenz?

  Again, it seems like the fact that GPT-3 could do that at all is it's an echo of a pretty small portion of the data set. If you had that as your kind of initialization, maybe you get that out of the gate or very close to it. Of course, you also could probably lose it. So you probably would need to also sprinkle some few-shot examples into your pre-training mix just to make sure it doesn't go away in early training.

• 2:09:10Speaker 03

  Yeah, this sounds really interesting. Somebody ought to look into this.

• 2:09:14Nathan Labenz?

  You're not busy, are you?

• 2:09:17Speaker 03

  Just a little bit.

• 2:09:18Nathan Labenz?

  It's a target-rich environment. How many ideas do you have that you think are interesting relative to what you could actually pursue?

• 2:09:27Speaker 03

  I don't currently have a running list, but it costs a lot to pursue an idea. So definitely many more ideas than than actual concrete research projects. And in fact, most research projects that I start don't really go anywhere. So I think this is the case for most people, I assume. But yeah, ideas are pretty cheap, but fully executing them is quite hard.

• 2:09:48Nathan Labenz?

  What would you say your funnel looks like? Is it something like to take a bar of this idea that I just came up with? If you say, yeah, that sounds, somebody ought to look into that. So how many somebody ought to look into that do you have for everyone that you actually try to look into? And how many of those do you have for every thing that you actually get through to a result?

• 2:10:06Speaker 03

  Maybe like I actually pursue one in three ideas or something and some of the ideas can be ruled out very fairly quickly, or they simply look too challenging to go anywhere very quickly with. My main heuristic is what sounds like the most fun and I think that's completely personal. For example, on the last two papers, I just got completely addicted to trying to find structure within the weights that I could replicate. Somehow the memetic initialization for transformers idea came about a lot more easily and was executed a lot more smoothly than the convolutional network one. Maybe just because I had that prior experience with kind of a similar topic. Yeah, I was mostly motivated just by fun, in particular on the convolutional network

• 2:10:51Speaker 03

  project, because I could see that there was this kind of statistical structure in the weights that I could maybe replicate by hand, like on paper. If you gave me like some markers, I could probably replicate the structure, though I'm no artist. But to describe that mathematically was very hard. So I spent a lot of time just trying to find some sort of way to succinctly describe what was going on and rolled out tons of very complicated models with lots of parameters that themselves had to be optimized before coming up with this pretty elegant representation. Yeah, I'm not sure if other people have a different approach to research, but I think that this is maybe not the most common strategy, but there are at least several people I know that work in a similar style.

• 2:11:35Nathan Labenz?

  Yeah, it seems like that's a maybe a reflection also of just where the field is at, right? Such a target-rich environment, so many things that are unexplored in a much more mature field. It might, you might not have the luxury of being quite so fun oriented, but I think that's a big part of why people are so attracted to ML, AI, ML right now. At least for me, in part is because I genuinely feel like I'm the first to see a lot of stuff and that in and of itself is super cool. And there are a lot of things that you can just pop into pretty quickly and at least learn something. And even if it's not like a paper, our own loss curve is still pretty steep in terms of our ability to understand so many things.

• 2:12:18Speaker 03

  Yeah, absolutely. It's a really fun time in the field for sure. Seems like more so than anywhere else. You can't just jump in and start asking the big questions. I mean, you do need a bit of, a few resources, maybe like a handful of GPUs, which not everybody can get. But that's quite attainable. And then you can ask questions like, what's the deal with pre-training anyways? Even though you don't have like industry level funding or anything like that. And there's always the possibility that your findings at small scale will actually work on like GPT four scale things. You never know. So yeah, it's super exciting and seems like it's uniquely fun. That's why I'm here.

• 2:12:56Nathan Labenz?

  Anything else you want to share about this work?

• 2:12:58Speaker 03

  We made some valuable connections here. Like I didn't see the connection to Andy's robustness work, but it seems to all be wrapped nicely together. Need to make models more robust, maybe by getting rid of bad data. Detraction at Microsoft, I'm currently interning. If you've seen the recent textbooks or all you need papers, it's very much about improving data quality. And then making up for getting rid of the bad data, for example, with the better initialization. That's a nice package. I like it a lot. And I didn't, I didn't see the full package until now. So I appreciate it.

• 2:13:33Nathan Labenz?

  Cool. I appreciate you guys taking so much time. You've been very generous with your time. And I'm sure the audience will learn a lot from this conversation. In conclusion, Zico Kolter, Andy Zou and Asher Trockman, thank you for being part of the cognitive revolution.

• 2:13:48Speaker 03

  Thank you.