Attack of the C̶l̶o̶n̶e̶s̶ Text!

Come hang with the bad boys of natural language processing (NLP)! Jack Morris joins Daniel and Chris to talk about TextAttack, a Python framework for adversarial attacks, data augmentation, and model training in NLP. TextAttack will improve your understanding of your NLP models, so come prepared to rumble with your own adversarial attacks! Sponsors: Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Jack Morris – Website http://jackxmorris.com, GitHub https://github.com/jxmorris12, X https://x.com/jxmorris12Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: TextAttack https://github.com/QData/TextAttackAttacking Machine Learning with Adversarial Examples https://openai.com/blog/adversarial-example-researchUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com slash change log.

• 0:17Speaker 00

  Linode makes cloud computing simple, affordable, and accessible. Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support, and scale you need to take your ideas to the next level. We trust Linode because they keep it fast and they keep it simple. Check them out at linode.com slash change log.

• 0:43Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at change.org/community and follow us on Twitter. We are at Practical AI FM.

• 1:08Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International. And I'm joined as always by my co-host, Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How are you doing, Chris?

• 1:25Chris Benson?

  I am doing okay. How's it going today, Daniel? What are you up to?

• 1:28Daniel Whitenack?

  It's been a long week. It helped my, my wife's business move, so I'm a little bit physically tired, but it's been a good week. Gave a virtual talk this morning and that was fun. Actually, really good, you know, question and answer and back and forth. So I don't know. You never know with like virtual talks, sometimes they're like super awkward. Sometimes they're all right. So this one was really good and, and interesting. What about you?

• 1:54Chris Benson?

  Same kind of thing. I'm just migrating into virtual talks after just taking a few months off and yeah, it's interesting as I've been watching different organizations, their different levels of like adaptability in terms of their readiness to do that. So always fun.

• 2:08Daniel Whitenack?

  For sure.

• 2:08Chris Benson?

  Very supportive. And hey, we had a milestone in our family this week.

• 2:12Daniel Whitenack?

  Really?

• 2:12Chris Benson?

  Yeah, my, my wife became an American citizen. She's British.

• 2:15Daniel Whitenack?

  Oh, congratulations.

• 2:17Chris Benson?

  As you know, she became an American citizen.

• 2:18Daniel Whitenack?

  Yeah, and pass on my congratulations to her.

• 2:20Chris Benson?

  Absolutely. It was a big deal for us.

• 2:23Daniel Whitenack?

  Yeah, that's so awesome. That's a long process, man.

• 2:25Chris Benson?

  It is a long process. And she knows more American history than any of us, probably more than all of us put together now. They have to, they test all that, all the things that we learned in third grade and promptly forgot. So.

• 2:35Daniel Whitenack?

  Oh, I forgot, right?

• 2:37Chris Benson?

  She's a pro at this point.

• 2:39Daniel Whitenack?

  That's awesome, man. Congratulations. That's, that's so exciting.

• 2:42Chris Benson?

  Thank you very much.

• 2:44Daniel Whitenack?

  Yeah. Well, this week, I'm pretty excited. There's sort of a gap in our discussions thus far on the podcast. We've talked about a lot of things, but there's still things we have not talked about, apparently. And every once in a while, like I see something talked about, like, oh, we haven't talked about that yet. There's just so much to cover. So today, we're going to be talking about adversarial examples and attacks. And we're privileged to have with us Jack Morris, who is a researcher at the University of Virginia and an incoming AI resident at Google. Welcome, Jack.

• 3:19Speaker 04

  Hi, it's great to be here.

• 3:20Daniel Whitenack?

  Yeah, thank you for joining us. It's really, really great to have you here. You've done a lot of stuff already and, and you're working a lot in open source. And there's a lot to discuss, but before we jump into all that, could you just give us a little bit of, you know, your background, how you got interested in AI and, you know, got eventually connected with Google?

• 3:41Speaker 04

  Yeah, sure. So I recently graduated from the University of Virginia and I studied computer science and math there. And while I was there, I did this internship at Google. Actually, it's kind of a funny story. Have you ever heard of Google Fubar, that program?

• 4:00Chris Benson?

  No, maybe I have not. I don't recall it. Tell us about it. Fubar us.

• 4:05Speaker 04

  Okay, so I'll tell you a story. So basically I was in my sophomore year at college, at UVA. We call it second year because we like that. So I was in my second year. I have this personal website, I think I was working on. I was, I was searching up something related to a software project I was working on, on Google. So like, I think what I typed in was something like Python, list comprehensions, something related to that. And I typed it in and then it's hard to explain in audio, but basically like there's the search result page for Google, like all the results. And it kind of like animated, like it shrunk up and slid over and then kind of like behind the search.

• 4:43Daniel Whitenack?

  Oh, I've heard of this.

• 4:45Speaker 04

  But there's a terminal.

• 4:46Daniel Whitenack?

  Yeah, yeah. It's a, it's a, yeah, it's like an Easter egg recruiting.

• 4:50Speaker 04

  Yeah, exactly.

• 4:52Daniel Whitenack?

  Yeah.

• 4:53Speaker 04

  It's pretty funny. It's not like, they don't send a lot of recruiters to my school or anything. So it was very, I guess, like serendipitous for me that I got that pop up. So it's like, it's a terminal window behind the Google search page. It's like, just like an Easter egg at a video game and it says, we think you're speaking our language. Want to take a test? And then I click yes, and you make an account. There's this little website. It's just a console and a browser. And it's coding challenges, like now they have lead code and stuff like that. It's, it's that kind of thing. And there's different levels. It gets progressively harder. I remember I was doing it for a few weeks and a few of the problems were, they're not like interview coding problems where you kind of think for a while and then you get it

• 5:36Speaker 04

  and you can write it out. They took me like several days, I think, just to sort of figure it out and try and optimize it and stuff. And I got to like level four and then I just couldn't figure one out. I was so excited seeing when I got it. And I was kind of bummed, but I just, I just couldn't figure this problem out. And, and they all have these really long write-ups. It was something to do with rabbits and there are all these allocations of rabbit holes and they were storing things and different, it was like a hard problem and a really confusing write-up. And then I just kind of gave up. And then it, I guess it turned out you didn't have to, you know, solve all the problems to get an interview. And I got an interview and I did an internship.

• 6:16Speaker 04

  So that's how I originally got connected with Google. And then more recently, I applied to this program called the AI residency and did some interviews for that. And my eventual goal is to get a PhD, hopefully, and something related to computer science, artificial intelligence, linguistics, not that sure yet. But next year, I'm doing this one to one point five year fellowship thing at Google, which will be really awesome. It's just a research internship, basically. And as to the question of how I got interested in AI, originally, I was really interested in just some things on the application side.

• 6:54Speaker 04

  They're all pretty pertinent to NLP, but I didn't really know at the time. I think my original idea was I worked on a few different open source projects and I wanted to make a website that aggregates like medical literature, like um it seemed like there's all this research that's just open online on Google Scholar, but it's pretty hard for the average person to, you know, sift through and aggregate and summarize. There's all these sort of like data processing, what do you call it, information retrieval tasks? And so I thought I was wrong about, I guess, like how how far along we are in solving some different NLP stuff. Like I thought that would be sort of like a plug and play type of deal, at least on a small scale for certain types of research and things.

• 7:38Speaker 04

  And I've tried it for a little bit and I remember I tried this thing two and a half years ago was when I first started getting interested in AI and I tried to use these skip thought vectors to encode sentences.

• 7:52Chris Benson?

  And what is a skip thought vector, just for those of us who don't know?

• 7:55Speaker 04

  Yeah, um, it's one way. It's now, I guess, it's no longer kind of like a state of the art method, but it's a way to take a sentence and encode it into a vector so that you can compare that vector to other sentences to see if they're similar or dissimilar.

• 8:09Chris Benson?

  Gotcha.

• 8:09Speaker 04

  And then I kind of was slowly realizing basically like, wow, these things, not that it didn't work, but that I don't know, there's just still a lot of problems to be solved and I got interested in some more specific stuff. And then more, most recently, I've been working on this idea of like robustness or Yeah, like the security side almost, like kind of like trying to find flaws in NLP models the same way that people are interested in finding them for computer vision models specifically.

• 8:35Daniel Whitenack?

  Yeah.

• 8:36Speaker 04

  Um, which is of the buzzword is like adversarial examples. And it's, it's kind of, a lot of people debate whether or not that's really applicable terminology to NLP. We can get into that later, but anyways, that's the kind of field that I've been interested in recently.

• 8:51Daniel Whitenack?

  Yeah, I um, remember, um, and I think I've mentioned actually the same episode of a different podcast on this podcast before, but the NLP highlights podcast, um, which I listened to, they had an episode about, um, behavioral testing of NLP models with Marco Rubiro, and um, he was talking about these tests that he did, which were really like right along the lines of what you're talking about, like, you know, if you, if you have a sentiment analysis example, that's like, I love the United States, it's great, or something.

• 9:24Chris Benson?

  That's a timely example right there, given my family.

• 9:26Daniel Whitenack?

  Yeah, yeah. And you put that into like the commercial like offerings for sentiment analysis, like products that are being sold, right? And so you do that, and then you just switch out the like United States and put in like Turkey. I love Turkey and it's great, right? It'll actually like break the system and you won't get positive sentiment anymore. Um, just because like so much in the training is like biased against like Turkey for whatever reasons. So yeah, I don't know, is that kind of what you're talking about with adversary? Maybe we should just back up a second and think about like adversarial examples in general. What, what do people generally mean when they're talking about adversarial examples?

• 10:07Speaker 04

  So the first people that were researching adversarial examples were all based around like convolutional neural networks and, and images specifically. And you guys have probably seen the examples where you can take an input image that's classified correctly.

• 10:25Speaker 04

  Say for some ImageNet classifier, classifying different animals, you know, there's a picture of a panda and it says, okay, this is a panda, I'm ninety nine point eight percent confident. And then, basically, they uncovered these sort of like blind spots in the really high dimensional in input space that point to other classes. And there's a lot of different research into why that exists, but if you add a small, tiny little image onto the panda image, so it's it's so small, you can't even notice it with your eye. It's maybe a pixel or two pixels out of two fifty five. You've added to the panda, you end up with an image that to a human can't be distinguished from the first image.

• 11:08Speaker 04

  But if you add noise in the correct direction, it's totally misclassified into some other class. So I think the quintessential examples like the panda and then you add this tiny little delta, a small image, and then it becomes a given with, you know, ninety nine point eight percent misclassification confidence, which is which is crazy. And it's a big problem. And it's become, I think that's one of the hotter areas of research in ML. I'd say there's all these people trying to figure out why that's happening, trying to propose different defenses for it. There's some pretty cool attacks and defenses, I think, in the real world, like, um,

• 11:48Speaker 04

  I saw people that are trying to evade facial recognition detectors, wear these special shirts. Have you all seen those or expect?

• 11:56Daniel Whitenack?

  Yeah.

• 11:57Speaker 04

  That are patterned in a certain way that convinced the camera that they're not a person, or they just totally distorts their facial recognition software. It's pretty cool. And so that that's a really hot area of research, like adversarial examples with convolutional neural networks or other image processing, deep learning models. And then naturally, one would wonder like whether that exists for text, and if so, what that would be. And it turns out it's it's not as cut and dry. It's like kind of a hotly debated and somewhat murky concept. But one hypothesis, like you talked about, is you can substitute words that maybe don't change the meaning with respect to the task, like it was in sentiment analysis.

• 12:40Speaker 04

  Maybe if you substitute a noun for another proper noun, you could say that or a proper noun for another proper noun, like substituting the United States for Turkey should never really change the sentiment of any sentence, right? And I think that's what he was talking about on that podcast.

• 12:56Chris Benson?

  And kind of just tying in, you know, as you're talking about, you know, as you were tying into this direction, as you're talking about adversarial in the CNN context, the convolutional neural network context, what would be the motivation for doing it in the NLP context? Like, what are some of the things you could think of that people would be bothering to do that with? I'm just curious, what in your views might be some of the drivers there?

• 13:17Speaker 04

  Uh huh. The first paper I wrote, which is on archive, it's called reevaluating adversarial examples in natural language, was trying to give sort of like a theoretical framework for different types of adversarial examples in NLP. The basic idea was like, okay, we don't have to agree on one set, but we can lay out some sets of constraints that you might agree on. Like, if you substitute a word for its synonym, it shouldn't change the prediction. That would be an example of like one partial definition of an adversarial example.

• 13:46Daniel Whitenack?

  So, and maybe this is having to do with, like, people in the convolutional neural network space, I think, talk a lot about like, safety in terms of like, why adversarial examples are worth exploring. Like, from your perspective, since you've done this work in NLP, what makes exploring adversarial examples for NLP, you know, interesting from your perspective? Is there a safety side of it? Or is it more like, you know, robustness? Like, you were talking about robustness as well.

• 14:18Speaker 04

  Yeah, those are definitely two different things. Like, if you're a company that's maybe putting some NLP model up on the internet, releasing it into production, on one hand, you want it to work in all sorts of cases, and you don't want there to be some obvious gaping holes in its actual predictions, which would be sort of like the robustness angle. And on the other hand, you want it to be safe, right? You don't want people to be able to, like, manipulate it in some way. So, the reason I brought it up the paper was because it includes some specific examples like that, because I think there's, there's a decent amount of research, but not a lot of. Discussion about like why robustness is important in NLP or like you were asking

• 15:03Speaker 04

  what maybe the goals of an adversary would be in that situation? So I think one really easy one is there's these toxic comment classifiers that I think are actually in use right now. Definitely on Facebook, they have this whole system for deciding whether a comment is just totally quote unquote like toxic and needs to be flagged or discarded or hidden or I don't know, you have to say you're 18 to read it or whatever. And so if you're someone who for whatever reason thinks it would be a good idea to write a toxic comment and then avoid that sort of flagging system, that would be a pretty good example of when you like want to like run an attack on an NLP model and

• 15:48Speaker 04

  you could actually use text attack for that that you should not.

• 16:11Speaker 01

  What up nerds? Jared Santo here, your humble producer. I'd like to tell you about something new we're beta testing around Practical AI. It's a membership program, which we think could be really valuable for the whole community. We call it Changelog++ and it's the best way to directly support Practical AI and all of the podcasts, videos, and other stuff we create here at Changelog. We have big plans and ambitions for this, but we are experimenting for now to make sure there's interest. That means when you sign up today, you get Practical AI and whatever Changelog shows you listen to now, except no ads. I guess that means this part you're listening to right now, it'll be gone. We also have some extended episodes planned, bonus content, merch store discounts, and a lot of ideas, but since it's such early days, we are offering memberships at a 40%

• 16:55Speaker 01

  discount for early adopters. That disappears at the end of August, so head to changelog.com/plusplus to join today, lock in that discount, get closer to the medal, and make the ads disappear. Once again, that's changelog.com/plusplus. We'd love to have you supporting us as a member.

• 17:19Daniel Whitenack?

  So let's say that our model fails on an adversarial example. You know, what could we do to fix it, by the way?

• 17:25Speaker 04

  That's a great question, too. That's a pretty open area of research. I wouldn't say there's a great answer I can give you right off the bat, but the sort of naive approach is generate a bunch more adversarial examples and then retrain your model either just on the adversarial examples or on the original training set and the adversarial examples concatenated together. And there is some research that's shown that gives you some improvements, but you can imagine like it's very, it's very different than the case of images. Like, for example, if, if your goal is to replace proper nouns with other proper nouns, right, like replace every instance of the United States with Turkey,

• 18:04Speaker 04

  or more commonly, maybe try all the countries and, and pick the one that meets your goal the best.

• 18:12Daniel Whitenack?

  I think that you're getting into some interesting areas. One of the things that you started talking about was like generating adversarial examples for NLP models. So it's one thing to realize, I guess, realize that your model fails for a particular type of attack or something like that. It may be another thing, like Chris was saying, to like, say, okay, well, now what? How do I fix that?

• 18:35Speaker 04

  Yeah.

• 18:36Daniel Whitenack?

  And you're saying like you could generate adversarial examples. I know that like some of your work and your open source work is geared towards generating those examples. But how in general, how do people come up with adversarial examples? Like, what is the range of things that people try to do to come up with these examples? Maybe specifically in NLP, is it, is it mostly hand curated data? Or is it like what other things are out there to try?

• 19:03Speaker 04

  Yeah, again, it kind of depends on your definition of an adversarial example. Maybe we can talk about that briefly first.

• 19:10Daniel Whitenack?

  Sure, yeah, that'd be great.

• 19:12Speaker 04

  I think that kind of lends a hand to why adversarial examples in NLP are not as well defined as in vision. Because if you have two images next to each other, it's very clear whether or not they're similar and whether the change from image one to image two could be classified as imperceptible. But if you have two sequences of text. If the, there, there's no imperceptible change unless they're exactly the same. And so if you make a change, any sort of change that you might define as imperceptible becomes at least like a point of argument. So I'll tell you two really popular definitions.

• 19:54Speaker 04

  One would be with respect to like semantics. So if you have sentence one and then you've replaced some words with synonyms, or like you said with a proper noun, that should generally not change the semantics of the original input. And so you could say, if you have two sentences, like if I have one that said, "I love the movie Parasite, the best movie I've ever seen." And then I replaced a couple of words, I said, "I liked the movie Parasite, greatest movie I've ever seen." You could, a lot of people would say that's invariant with respect to semantics, like they contain the same meaning. So if they have different predictions, that would be classified as an adversarial example.

• 20:38Speaker 04

  And then the other thing that I was gonna bring up, like another definition is with respect to like character level changes. So if instead, I just, uh, like imagine a typo on the computer, basically. If, if instead of saying, "I love Parasite," I spelled "love," L V O E D, or something like that, just switched around the characters a little bit. It's actually really shocking how many state-of-the-art NLP models will just totally mispredict that for whatever reason. And so those are sort of two competing, not necessarily totally matched ideas of adversarial examples. One being with respect to like character level changes, or, or if you just insert a character like L O V E D Q or whatever.

• 21:20Speaker 04

  Um, especially models that are based on this idea of like entire words, um, have a lot of trouble dealing with those types of changes. So, uh, e- either one of those would probably be classified as an adversarial example by most people.

• 21:37Daniel Whitenack?

  Yeah, I'm kind of trying to think through like my own workflow right now. And so like in terms of integrating or making my models more robust, I guess. Let's say I'm creating an NLP model. It's probably unlikely, or it seems unlikely for me to sort of plug all the holes in terms of like things that my model might do that are unexpected. Right. But you're saying that maybe there's some sort of obvious things that we can protect against or make more robust, like with the typo sort of perturbation, or maybe not changing the semantics, or some type of like perturbations like this that we can test against. But then it's probably, I don't know, what are your suggestions in terms of like, you

• 22:21Daniel Whitenack?

  can plug some holes, but then eventually your model might behave unexpectedly in a new way that is totally unexpected because it's unexpected. It almost seems like unit testing software or something to me where it's like, you know, you test for the things that you definitely anticipate, but then at some point, something weird happens and you have to add a test case. Is that kind of how you would view it? Or how would you view like the workflow of like thinking about adversarial examples in developing a model as opposed to not thinking about them? I guess is where I'm guessing.

• 22:53Speaker 04

  Yeah, yeah. That's a really pertinent question, I think, to anyone who's an engineer, you know, actually trying to build NLP systems for real people. And it's not, it's not one that's totally solved yet. But another idea that I've heard, I think Ian Goodfellow has gone to talks and advocated for this, is the idea of if you're building a model, adding some kind of output that can identify whether an input is malformed, or maybe doesn't fit with the distribution of the training data at all. So that in the case I indicated before might indicate, you know, a misspelling, or some kind of really unnatural misspelling that a user would never produce.

• 23:37Speaker 04

  Or maybe just a synonym substitution, like maybe using a word that might have the same meaning, but would almost never be used in that context by a real person, which is something I, I think I see a lot with these adversarial example papers. They might say, okay, if two words are synonyms in the thesaurus, they can always be substituted for each other. A lot of the time, that might not be a very natural substitution, or something that a human would probably never actually do. And so if you can train a model. That has some way to indicate whether an input is sort of like acceptable or not. It can, I think, alleviate a lot of those concerns, though they're still there. I think in the general case, it would make your system a lot better.

• 24:17Chris Benson?

  Yeah, so I mean, you know, I know you and Daniel do a lot of work in this area. And so I'm kind of approaching it as the one who's not actively doing models in NLP. If I'm understanding you correctly, it sounds like there's really a different set of use cases. If you were to compare this to like adversarial and CNNs, whereas, you know, that behavior is in some sense, often maybe nefarious, you know, trying to change a classification, you know, to for some purpose. Whereas this, it sounds like, you know, you mentioned robustness earlier. And it sounds like there may be a lot of use cases where you're helping a user not make mistakes, where you're trying to prevent unexpected behavior, intentional or not, you know, in terms of like what the user was trying to do. Do you think that's fair?

• 25:02Chris Benson?

  Do you think that the types of use cases that this might be applied to are fairly different? It sounds like.

• 25:08Speaker 04

  Yeah, absolutely. And I really liked the comparison that you made of comparing like finding adversarial examples for NLP models to some sort of unit testing in the software engineering spectrum. And that's what the paper that I think he's talking about, the checklist paper, that's what they talk about is applying ideas from software engineering to verifying the usefulness of NLP models essentially. And I think that's a lot closer to what adversarial examples are in NLP and how they're useful to maybe how they are in CNNs.

• 25:43Chris Benson?

  So I could, I could imagine not far down the road as some of this work is realized that, you know, maybe for programmers, their IDEs or text editors, you know, kind of get some modules that help them from in terms of essentially unit testing what they're doing in a smart way. Or even for just someone who's not technical, a word processor might have this capability built in so that, you know, when you're writing, you're making fewer mistakes in that way. So it sounds like there's a lot of help the human possibilities here.

• 26:11Daniel Whitenack?

  Yeah, it would probably be more, I guess, on the data side or the training data side. So like if you see. A unexpected behavior in your autocomplete in your text editor or whatever. Like, why was that unexpected? Did you like, give the model like some type of weird, like malformed input? That like caused it to just like totally fly off the rails? Yeah. And so then it's like, okay, well, why did that misbehave? So now we've got to add some things into our training data. So I know, I definitely want to get to this open source project text attack, which I actually came across in my, one of my newsletters.

• 26:53Daniel Whitenack?

  I forget. It was like one of the NLP newsletters mentioned, mentioned it. So I forget which one it was. I'll have to look up while we're talking. But yeah, it seems really cool. So was this open source project? Maybe you could talk a little bit about how that came about. Was it because you were trying to like, you were in this workflow? And you were like trying to add adversarial examples? And it was like, you know, like there just wasn't tooling around it? Or how did that come about?

• 27:18Speaker 04

  Yeah, you kind of nailed it. So I was working on things related to trying to find adversarial examples in NLP. And like I said, there's a lot of kind of like disagreement on what counts. And there's been a decent amount of research into this idea, but it's not very homogeneous. It's actually pretty disorganized, I'd say. And a lot of people have really similar ideas, but they change one or two things, but they reuse a lot of ideas. And all their code and projects are generally implemented in different places. So it's kind of a headache trying to re-implement results and compare things.

• 28:01Speaker 04

  But it turned out that like, a lot of the people that suggested NLP attacks were using a lot of the exact same components. So one example is, Like the thesaurus I was talking about before. So have you all heard of uh Glove word vectors?

• 28:18Daniel Whitenack?

  Yeah, from Stanford, right?

• 28:20Speaker 04

  Yeah, from Stanford a few years ago. Those are still pretty much top of the line word vectors you can download. So basically, they're more commonly called word embeddings. You can download this big matrix that's assigned to, I don't know, a hundred thousand or a million words from English and each one has their own word embedding. And it's a vector of dimensionality, maybe three hundred, and they're supposed to encapsulate a lot of information about English. So if you're training an NLP model, the initial layer is probably going to use an embedding similar to that. And Glove is kind of like the just very accepted word vector a lot of people use, at least before people started using transformers and subwords.

• 29:04Speaker 04

  But um, there's this paper that made these amendments to the Glove vectors so that they more directly encapsulate information from a thesaurus. So i- it's called, they're called counterfeited vectors, uh not spelled like you would think, like counterfeited. And they, basically, if you compare two synonyms uh based on their counterfeited word embeddings, they should be very similar in terms of their angle, like cosine similarity. And antonyms should be very dissimilar, like they should have angles close to a hundred and eighty degrees. So it's, it's basically just Glove vectors plus this preprocessing step that uses a big list of synonyms and a big list of antonyms to try and make those vectors closer and further apart.

• 29:50Speaker 04

  And I bring this up because a lot of people that have created systems that develop adversarial examples in NLP use this base layer of counterfeited vectors. So if you take the example I talked about before, like I loved the movie Parasite and you could Look at the word vector, the counterfeited word embedding for loved, and then look at its nearest neighbors in the counterfeited embedding space. And you might see a bunch of words that fit in the same context and are actually synonyms. So loved, liked, I don't know, enjoyed. And then, once you start to deviate in terms of that angular similarity, then they get less and less similar.

• 30:32Speaker 04

  And there's some debate as to, you know, how similar do they have to be to be synonyms? Like 0.9 or 0.95, you know, which is a whole nother thing. But all these attacks use those counterfeited word embeddings, like maybe over 10 papers. Use counterfeited word embeddings and maybe some other components that are exactly the same to generate adversarial examples. And even stepping back from that, the entire process for generating adversarial examples in NLP is very, very similar. So by the process, I mean the process of taking a text input, so a sequence of words, and then producing some other sequence of words that generally like fools a model.

• 31:12Speaker 04

  So taking maybe the sentiment classification example, you have an input that's classified as positive. And the process of finding an adversarial example would be which words can I substitute that will change this classification to negative? And so that turns into a combinatorial search problem. And most people do it in the exact same way and often use the same word vectors. And then, I don't know, change a few different things and then release their attack. So our idea was if we break that process down into components, then we can construct the attacks from different papers based on these components.

• 31:51Chris Benson?

  So you've definitely captured me in terms of interest in adversarial attacks. And so let's say that I'm out there and I've just, I'm listening to this or I've just come across text attack. Can you tell me, kind of just describe the library? Let me know what I should know about it as a beginner coming into it that wants to use it? And, you know, kind of what are the goals that I should keep in mind that the project tackles? And, you know, are there any things, you know, that I should not address as well with this library that I'd look elsewhere? So can you kind of give me that beginner perspective?

• 32:24Speaker 04

  Yeah, absolutely. So um it might help for me to talk real quickly about that kind of like system I was talking about, like the the components. And then I can explain the most common use cases because obviously you can pull out any one of the components and use them for your own purposes. So one thing that we really focused on in text attack is trying to make it work out of the box. So for example, those counterfeited word embeddings, instead of, you know, going to this website, downloading it, unzipping it, moving it. Finding out how to load all the data, you just import text attack and do text attack dot the class and just initialize it and it'll download everything for you.

• 33:07Speaker 04

  Uh, which I think is that is really cool. If you guys know about hugging face transformers. We're, I mean, a lot of the text attack stuff is built around transformers and tokenizers and now this data set loading library called NLP. Which I'm very grateful for and we kind of tried to follow the same model. So instead of having all these files you manipulate yourself, you pretty much just. Reuse other people's, you know, and and it saves a lot of time. So the easiest or probably most common way that I would imagine people use text attack down the line. Is for things like that for for the embeddings or another very common thing is. Sentence encodings, which is something I mentioned at the beginning of this talk.

• 33:48Speaker 04

  Like there's so many different methods for taking a sentence and encoding it into a fixed length vector. Whether they're very effective or not is a question, but they're useful in a lot of situations. And so one thing text attack has done is just sort of abstracted them into classes that work by themselves. And so you could just. For example, if you were doing some project, I don't know, you wanted to. Look at a bunch of Airbnb reviews and cluster them based on which ones were similar. You could just import text attack and then just call like this sentence encoder dot encode and then give it the list and it would just do it for you, which I think is. Pretty valuable. So I'll tell you what the components are very quickly. There's there's four and we have our own names for them, which I think.

• 34:34Speaker 04

  Increases the learning curve a little bit, but it it there's some benefits, I think, to having your own terminology. So it's all based around this idea of the NLP attack as a system, which is taking the text input, looking for changes you can make to it, making sure those changes are acceptable. And then whenever you have decided you fool the model, you stop. So the first component would be like the what we call the transformation, which is taking an input and changing some of the words or characters. Like one transformation would be substituting words with their counterfeited word embedding neighbors. And then, once you do that transformation step, there's also this idea of a constraint, which is trying to make sure you didn't make any mistakes.

• 35:19Speaker 04

  So like a common constraint is use the sentence encoder. A popular one is called the universal sentence encoder, which is by some folks at Google. And you encode the original input and now your potential adversarial example, and make sure that the sentence encoder also says they're very similar. It's basically like a sanity check to make sure you didn't change the meaning or whatever, change too many characters if that's what you decide. And then there's two other components. So we have the transformation and the constraints. And you have to define your notion of whether you fool the model or not. A common thing would just be change the classification output, or change the classification output to a specific class. Those would both be examples of what we call the goal function.

• 36:03Speaker 04

  I think a really cool one that I want to explore more in the future is with sequence to sequence models, like a machine translation model. Your goal might be to take the original output translation and change as many characters as possible. So say you're translating a sentence into French. You would have your original translation. And if you could substitute a word from the input with a synonym, and then it produced a translation that was totally different, even just in terms of characters or its blue score, that would be pretty telling and like probably very bad for your translation system. Uh, so that that would be another goal function would be trying to minimize the blue score. And then the last component is called the search method.

• 36:45Speaker 04

  And that's basically like, if you have the input and you have all these transformations, how do you decide which one to keep? Which is important because if you just tried all the combinations, I mean, if you have an input of 10 words and each word has 50 neighbors, you end up with Fifty times fifty times fifty possible substitutions that you might want to combine. So that the space grows exponentially very quickly. So you have to come up with some sort of like greedy or approximate heuristics for doing that. And that's what we call the search method. So you can combine those four things into an attack, like an NLP, what we call an attack,

• 37:26Speaker 04

  which is just a search for adversarial examples that meet the constraints and fool the model as defined by the goal function. But there's some really cool other things that come off of that. A big one that I've been talking to people about recently is data augmentation, which is also a very kind of, I would say, under-researched field in NLP. It's another thing that is pretty commonplace in vision. It's almost like everyone does it. You know, if you want to train a state-of-the-art vision model on CIFAR-10 or ImageNet or some other dataset, you're going to do some sort of augmentation to change and increase the size of your dataset. So with text attack, if you have this transformation, which can find maybe like semantics preserving

• 38:10Speaker 04

  changes to your input, and you could add on constraints, which make sure that they preserve semantics, then you can end up with like some pretty good tools for data augmentation just from those two things. And since we're trying to implement more components, that would hopefully grow the list of potential augmentation modules as well. And so, yeah, that's something I'm really excited about is data augmentation.

• 38:33Daniel Whitenack?

  Yeah, that's really awesome. I know that even in speech, it's fairly common to, you know, like mask out, you know, parts of a spectrogram or like speed up or slow down the audio or something. And that's like fairly common component. But in NLP, it's pretty much, most of the time where it seems like people are talking about, oh, we have this parallel data, we have this monolingual data.

• 38:57Speaker 04

  Yeah.

• 38:57Daniel Whitenack?

  And that's like basically it, like that's all you have. And like you could augment with monolingual data, but you know, this is like another, it's another route to that, which is pretty cool. In terms of like the open source, I mean, there's been Interest in the library and what is your vision for it going forward? Do you see like others? You mentioned like Hugging Face Transformers and kind of modeling certain things after that. And I know one of the things that they talk about a lot is, you know, people contributing models to their, their model hub, right? Um, and actually it's good timing because we had Sasha Russ from Hugging Face. His episode was released this week. So that's a great, great model that they're going.

• 39:38Daniel Whitenack?

  So I could see similar things happening here where it's like, you know, there's a new type of perturbation or goal function or something like you were saying that the people are exploring. Do you envision people kind of like having a way to contribute those models in? Or like, what is your thought process around how you see the future of the library evolving?

• 40:00Speaker 04

  Yeah, that's a great question and something I'm still, you know, we're still kind of discussing and trying to figure out because we've written a lot of code and there's a lot of different features that still could be connected. And obviously, once you, once you work on a library for a while, there's just so many improvements you want to make to it. You know, I have this giant list and take my whole life. But, um, I think I could probably break the potential users down into three groups, at least three main groups. So the first one would be kind of what I mentioned before, what I imagine would eventually be just the most common general use case would be using the components because they're easy to use. But, um, the second one would probably be, like you said, people who want to work on

• 40:46Speaker 04

  researching the robustness of NLP models. So maybe coming up with new goal functions or trying a new transformation. Like, for example, I don't know, if you created a model that could paraphrase inputs, it's a totally open problem in NLP, but even if you could do it a little bit well, I think that would be really interesting. Like taking that input sentence and just paraphrasing it or paraphrasing its individual sentences and trying those different pieces and seeing which one fools the model. That would be someone, that would be a really cool thing for someone to research and that would be just a new transformation, basically, that just takes an input and paraphrases it. And then the third use case would probably be people who actually want to test and potentially

• 41:28Speaker 04

  improve their models using text attack, which is what we added this summer is. What I mentioned before adversarial training based on just doing an attack and then increasing the size of the training set and continuing training. So hopefully people would be able to like import text attack and then test out the robustness of their maybe summarization model or translation model or whatever and get some insights into maybe how it's failing in the common cases and hopefully how they can improve it.

• 41:59Chris Benson?

  Yeah, I got to note that as you're talking there, I'm looking through the GitHub repo and you really put it together well for someone like me who is new to this. You know, you cover the kind of what it is, you know, the Slack channel, help with setup, then you have a good section on usage and running attacks there. And then the thing I love the most is I love your section at the bottom on attack recipes.

• 42:21Speaker 04

  Oh, yeah.

• 42:21Chris Benson?

  Yeah, I mean, and I was flipping through the links here. You're listing these, these different recipes for attack modes and you link off to the paper or website where it shows what it is and how to do it in detail. And then with the section above it on usage, it makes it really easy to just go ahead and give it a shot. So after we're done with our conversation, I have a feeling I'm going to geek out on this for a little while. So thank you very much for that.

• 42:44Speaker 04

  Awesome.

• 42:45Chris Benson?

  Do you envision more contributions being made to this attack recipe section?

• 42:50Speaker 04

  Yeah, I sure hope so, man. We've, there's still papers that I think we could implement with pretty low effort because they have a lot of overlap with the components we've already implemented. Well, talking about vision, there are a few very common libraries in vision. One's called Clever Hans. No other names are coming to mind, but there's, I know there's one by IBM, a one called Foolbox. And, and those are pretty standard tools for researchers that are interested in the robustness of computer vision models. So not even that I'm super convinced that text attack will be that tool, but I'm hoping that just putting everything in one place would make it a lot easier for people to actually do that research and make fair comparisons and.

• 43:38Speaker 04

  You know, advance the field. So hopefully getting people actually excited about it by making these things easy to use will then lend a hand to people actually contributing. That's the goal. We've been working on this since almost a year ago, about a year ago, but it it's only been open source since May, so hopefully that'll come with with time. But yeah, right now I think one of the recipes was implemented by the authors and then the rest we kind of did by hands. So.

• 44:04Daniel Whitenack?

  That's awesome. Yeah, no, I think that this is so well set up and and a lot of thought has been put into it. I'm like thinking back to the code that that I wrote i- in academic research and the reason why it brought me back there because you have a little logo of a or a little octopus emoji on your on text attack and the code that eventually I implemented my method in in my PhD was called octopus. But I have no like, I don't know that anyone be to be able to run my module of octopus just because like, you know, it was nothing near well documented like this or or anything like that. So I hope that you do get contributions. As we kind of uh end up here, I guess I'm curious just as kind of a last question.

• 44:50Daniel Whitenack?

  You've done this research, but you're also kind of launching into a new thing with your AI residency. I'm just kind of curious, you know, from your perspective, uh, you know, jumping into the field at this point and this new position. What are you excited about in terms of the future of AI and and what you want to be involved with? What gets you excited?

• 45:11Speaker 04

  It's a great question, Daniel. I think the thing that motivates me the most is the potential for creating systems that actually have some basic knowledge of uh anything, you know, uh, like there, there's a subfield of NLP that calls itself natural language understanding, like N- NLU. And that to me is the most, I don't know, applicable, at least seems so like philosophically, it seems applicable to like my everyday life. Having systems that can actually do some kind of basic reasoning. The problem with models like GPT three or that we we don't have to debate this right now.

• 45:53Speaker 04

  Is that they have no common sense, understanding of anything, you know? At least you, you, that's my argument, I guess. But I think like a tool like text attack, it kind of exists to expose that in a way that even though it might seem like it's human in terms of understanding, because it can exceed the human score and all of these different benchmarks. If you use text attack or something like that, you can kind of gain some insight into why it might not actually understand anything at all. And so I'm not sure if deep learning is the full solution, but I think it's certainly part of it and moving towards systems that have that type of like true understanding of language is really exciting and compelling to me.

• 46:38Chris Benson?

  That's a whole other episode that you're moving into right there. There's a, there's a lot of ground there, but I gotta say, thank you so much for coming on the show. This was a fascinating conversation. It was a pleasure to have you on the show, Jack, and it was a pleasure to learn about text attacks, so we hope you'll come back sometime.

• 46:55Speaker 04

  Yeah, thanks, Chris. I hope so, too.

• 46:56Chris Benson?

  Thank you very much.

• 47:01Speaker 01

  Thank you for listening to this episode of Practical AI. Hey, this was episode 99. You know what that means? Oh, yeah, 100 coming right up. Practical AI is hosted by Daniel White neck and Chris Benson. It's produced by Jared Santo. That's me. And our music is brought to you by the mysterious one, Breakmaster Cylinder. We have awesome sponsors who support the show. Special thanks to Fastly, Linode and Rollbar for their continued support. Hey, did you know we take requests? We sure do. Head to changelog.com/request, select Practical AI in the dropdown and let us know what you'd like to hear about on the show. Pick a guest, pick a topic, you name it. We love to hear from you. Once again, that's changelog.com/request.

• 47:45Speaker 01

  That's all we got for you this week. We'll talk to you again next time.