The history of online harassment before and after Gamergate with Caroline Sinders

Cataloging online harassment before and after Gamergate with Caroline Sinders Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nilay Patel?

  Before we begin, here's a word from our sponsor, Dell.

• 0:02Speaker 01

  Today's segment is sponsored by the Dell XPS 13 laptop with an eighth gen Intel Core i7 processor. Experience Dell cinema's incredible color, sound, and streaming on the Dell XPS 13. It's the laptop for people who watch things on their laptop. Learn more at dell.com/xps13. Sponsored by Dell and Intel.

• 0:28Nilay Patel?

  Hey, everybody. It's Sina from The Vergecast. This week on the interview episode, Casey Newton and I sat down with Caroline Senders, who is a machine learning expert, a designer, a product designer, and a fellow at the Harvard Kennedy Digital School, where she's compiling an exhaustive history of online harassment. Now, most people think harassment began in 2014 with Gamergate, but it started long before that. Gamergate was just an inflection point. We talked about that a lot. We discussed how other scandals like Donut Gate, Pizza Gate operate, how they keep growing, how they've turned into a playbook for other harassment campaigns online. And we discussed all of the tactics that harassers online use and how the platforms can fight back through smart product design. It was really interesting. It's really important. And there's a lot of work to be done, including on the individual level,

• 1:13Nilay Patel?

  work that you can do to make harassment better. Also, as we were recording this episode, Laura Lumer, who's a bit of a right wing crazy, had literally chained herself to Twitter's doors in New York. So that was a delightful coincidence. You'll hear us joking about it all throughout. Check it out. Caroline Senders.

• 1:31Nilay Patel?

  All right, we're here with Caroline Senders, who is a machine learning designer, a researcher, an artist, your fellow at the Kennedy School. What does that mean?

• 1:39Speaker 02

  So a fellow can mean a bunch of different things. What it means is I am funded by the Kennedy School, which is actually a program called the Digital Harvard Kennedy School. So I'm a fellow exploring different kinds of problems on the internet. And so they're funding like a piece of my research.

• 1:55Nilay Patel?

  Yeah.

• 1:55Speaker 02

  So I'm working on a very specific kind of project for them. I'm looking at, are there patterns inside of communication apps? And thinking about communication apps broadly. So iMessage, Telegram, Threbo, Sina Weba.

• 2:09Speaker 02

  Twitter, anything on Facebook, so any surface of Facebook as a communication app? And are there patterns that elicit trust that cause users to trust those things more? So is it rounded corners or bright, happy colors or things that look a certain way? Is it beyond just users having their social network there? Or different people that they want to talk to? Are there other things that sort of engender trust on these platforms? And then do those design choices, then off you scape for security? So that's what I'm studying at the Kennedy School.

• 2:43Nilay Patel?

  That's super interesting.

• 2:44Speaker 02

  Thank you.

• 2:45Nilay Patel?

  And then you also run a firm that does machine learning of some kind?

• 2:47Speaker 02

  I run a design firm called Convocation Design and Research. One of the things we focus on is machine learning, but it's a design firm. And I like to solve complex problems and make them human readable. So some of that is machine learning, some of that can be ethical quandaries, some of that can be trying to take a technical challenge and present better design around it. And I think some of my work from the Kennedy School fits into that. So also thinking about how do you design for NGOs and nonprofits or public good? How do you translate machine learning into different artifacts or entities or writing that works for a more common audience? So that can be, you know, doing high level consulting and talking, you know, how to explain things in plain language, right? Instead of saying everyday speak, how do you think about translating

• 3:31Speaker 02

  technical documentation or technical processes in a way that's understandable?

• 3:35Nilay Patel?

  Yeah, so you like apply the machine learning to that? Like, we're going to read a stack of NGO white papers.

• 3:40Speaker 02

  No, no, no, so I will take a machine learning process or a product and explain to someone, people don't know what you're doing.

• 3:48Nilay Patel?

  I see.

• 3:48Speaker 02

  And because I understand how machine learning works and I've worked as a product designer in machine learning, this is how you should maybe think about explaining it.

• 3:57Speaker 02

  Yeah.

• 3:58Nilay Patel?

  That's great. So you just put together what you have titled, "Incomplete But Growing History of Harassment Campaigns." And it starts in 1990, which is like my favorite thing, like the first IRC fight. I asked Casey to join me because he obviously covers speech and democracy and platforms all day long. And so I just want to start, why did you, why did you feel like you needed to chart out every major harassment campaign since 1990?

• 4:21Speaker 02

  I started this actually in 2016, towards 2017, because I realized at that time, and this is like very beginning of like the alt-right really growing online. That so many things were being repeated. And then this is a big thing when we talk about why, why we teach history, right? Because history is doomed to repeat itself. And what I also found fascinating is when I started researching online harassment in 2013, how much of it was just the same kind of tactics replicating itself. And we saw this in Gamergate especially. So I thought it would be important to actually sit there and outline everything. And I especially thought about this because when I started my online harassment research, I was really focusing on Gamergate. And that became something I was known for talking about and exploring.

• 5:04Speaker 02

  And also I explored that from a product design standpoint of how the certain structure of platforms, the design of platforms can actually like help a harassment campaign. What are ways to think about redesigning platforms to mitigate harassment? But there were so much online harassment before Gamergate. And so a few of it was getting covered, especially if you look at Your Slip Is Showing, which I argue and a lot of Black Twitter feminists argue this as well, or Black feminists on Twitter, who use Twitter also as their medium for exploring and documenting different kinds of social media culture, that a lot of the tactics used in Your Slip Is Showing...

• 5:40Nilay Patel?

  Can you quickly explain what that is?

• 5:41Speaker 02

  Okay, so Your Slip Is Showing is related to this 4chan harassment campaign called Operation Lollipop. A fair amount of harassment campaigns will have "Operation" in the name. Operation Lollipop was planned on 4chan as a harassment campaign to impersonate women of color feminists on Twitter, to drive wedges in actual feminist community between white feminists and women of color, and then between women of color each other in feminist subcultures and communities. And it was around this hashtag called #InFather'sDay, which was a fake hashtag, so we could call it a hoax, that would cause a schism, and it did cause a schism. And this was planned. And these were fake accounts, impersonation accounts, right?

• 6:25Speaker 02

  And a handful of Black Twitter feminists knows this, or Black feminists on Twitter. And one in particular is so true. And she created a hashtag called "Your Slip Is Showing," and that was shorthand to sort of say, "You're fake."

• 6:38Nilay Patel?

  Yeah.

• 6:39Speaker 02

  And it was a colloquialism. She uses hashtag and tweeted at people, "Your Slip Is Showing." She then took a Storyfy account and collected, all the time she's used it, to then document all of the fake accounts.

• 6:52Speaker 03

  Yeah.

• 6:52Speaker 02

  What we're seeing here is the impersonation of of people. So also, and by impersonation, actually, like creating fake accounts and fake entities to create noise and to game and flood a hashtag. And then this hashtag was also a hoax. This is something we saw in Gamergate. This is something we see right now, actually. But it's not around misinformation or political news or around our daily news. It's targeting specific people. It's targeting communities. And I think it was really important, and one of the reasons I published it now, is I think if we're starting to look and study misinformation, disinformation, we don't look at the history of online harassment campaigns and social media. What we're missing is a massive part of this history, of seeing how people have used these

• 7:34Speaker 02

  platforms for many years, way before we started talking about fake news, way before we started talking about Gamergate. So that was like the main impetus behind creating this was, well, we're missing actually all these stories. And these stories are important, not just because these victims are important and their stories are important, but this is proof that like different groups online have spent years honing how to create misinformation campaigns. It was just under the realm of harassment. It just wasn't under the realm of misinformation or disinformation, the way we think about it today in our political culture.

• 8:08Speaker 03

  And so as you read it that way, do you come away thinking, if only we had taken this seriously sooner, maybe we would be in better shape than we are now?

• 8:17Speaker 02

  Totally. But one thing I want to highlight, and that was a great question, thank you for asking, is women of color in particular have been talking about this for years, like social media historians, activists, advocates, writers, Sedat Harry in particular, who's with Mozilla, and in her own right is her own researcher, outside of her work, outside of her day job. People have been talking about these things for years. And they've been talking about these as planned hoaxes and as coordinated campaigns. And these are key words, hoaxes, coordination, because we see them right now. They're just not about politics necessarily when these occurred in like 2011, 2012, 2013, 2014.

• 8:57Nilay Patel?

  Are these like fractally similar? Like, does the shape of end Father's Day look like the shape of Russian interference?

• 9:04Speaker 02

  Yes and no. I would say from a qualitative or ethnographic standpoint, sure. Like, the way in which people coordinate and impersonate, I'm sorry, I keep using impersonate, impersonate is the wrong word, but create bot-like accounts. But they're not bots, they're human-powered bots, right? Like sock puppets? Sock puppets, yes, thank you. They're creating these sock puppet accounts. So, yes, it is similar in that form, but is the scale of it similar? Maybe not. But what's fascinating is scale kind of doesn't matter when you're trying to figure out the edges of where a system breaks. Because if you can do it with like five or 20 accounts, you could probably do it with hundreds or thousands, right? If you can go through and like plan and coordinate this kind of thing, and you're not tipping anything off, and you can do it for a while, you can probably scale it, right?

• 9:48Speaker 02

  And if it breaks its scale and you dial it back and figure out where it works. And that's what I think has been happening in these years, and this is what we're seeing.

• 9:57Nilay Patel?

  So, I always think about Gamergate as being this enormous tipping point. We run a tech website, we cover video games, our sister site is Polygon, they cover video games. We like, we're in it. We were just fully in it, it was a big deal for us. And so, to me, it's like, very personal, and it just feels like everything changed after that, and now you can like, there's a hundred headlines out there that's, everything is Gamergate now.

• 10:21Speaker 02

  Totally.

• 10:22Nilay Patel?

  Is your point kind of like, that was just when everyone noticed and everything is, this has been going on? Like, how should I think about that?

• 10:27Speaker 02

  I would say, yes, to part of your question, that this has been happening before Gamergate, and we should have noticed. But Gamergate was a tipping point. So, I also started researching this because I was making video games, and I saw so much of like my own community-targeted. I was- Making video games and also getting my masters at the Interactive Telecommunications Program, ITP, where I was working and studying under Clay Shirkey, and we were looking at social movements or political movements on social media. And so Gamergate to me had all the clear hallmarks of a political movement. It was just a harassment campaign. But it was big as a political movement, and I think that is also what makes it different. It wasn't a one-time thing. It wasn't just for the lols, and I mean that in a very particular way.

• 11:12Speaker 02

  If we look at anonymous campaigns, for example, or we look at the history of 4chan or B, right? Doing things for the lols is very short. And it's supposed to be short. It's supposed to be something you do as a joke, and then it's done. And then you do something else to one-up something. Gamergate was the sustaining of a movement. Using tactics, though, that we have seen from before Anonymous became like White Knights in Black Hats, for example, right? Or before Anonymous had a much more political bent to it, when Anonymous was really just anonymous on 4chan. Gamergate took it almost a step further because it was a political movement. It also was an emotional movement for those that supported Gamergate. And this is not to diminish the harassment that they did.

• 11:54Speaker 02

  What they did was extremely violent and life-ruining for a lot of their victims and very serious. But I think for a lot of those that were supporting Gamergate, it was also, it was a political movement. It was an emotional movement because of something affecting their fandom and something they had built like personal parts of their lives around. And it wasn't about a one-time thing. It wasn't actually about irony or humor in the way we think about, or in the way, rather, you can contextualize 4chan movements. It was something different. And I think that's also one of the reasons it gets talked about, because if you remember back to when Gamergate was first covered, people were really dismissive of it. Oh, it's just infighting in games. Oh, ha-ha.

• 12:37Speaker 02

  Like gamers are just like virgins in their mom's basements, covered in cheeto dust, you know? And that is like a lot of the way people describe trolls, which is not true, right? And then when it kept going and kept going, then I think a lot of the general media understood that it was something different. And I think that's the reason it was this tipping point. It happened to be this collection of... Just the way you describe it, inviting in games. That already sounds kind of strange.

• 13:05Speaker 03

  Yeah.

• 13:06Speaker 02

  Right? And then death threats, swattings, people quitting their jobs, bomb threats going into workplaces, right? Or into conferences. And then you see that it didn't stop after a month and then it kept going. And then if you actually looked at who it was targeting and where it came from, it just kept getting more and more extreme and also more and more strange in the way in which online campaign would go. There was not a lot of things that looked like it in terms of length of time, amount of victims and the severity of it. What it is, is it looks like a protest campaign. It has probably the size of something that is a much bigger political movement, except that this was a harassment campaign.

• 13:45Speaker 03

  So your, your piece, which is great, everyone should check it out on Medium, currently has over three dozen harassment campaigns. And when you talk about Gamergate being a tipping point, I can't help but notice, reading your report, that it seems like the pace and the scale of these harassment attacks are accelerating. I mean, is that fair to say?

• 14:07Speaker 02

  I'm going through the report with a data visualizer to start tagging them and trying to look at the size. So citation needed, but that, that seems like a correct assessment. And, you know, going off like, let's, let's, let's make this a hypothesis that, that can be tested then. So if they are accelerating, why? A potential answer could be because maybe the same people were involved if they're on the same platforms and they're learning each round. And that's something that I interviewed over 40 victims of Gamergate. And one thing someone said is it felt like, because they had been a victim of your slip is showing, that it felt like people, the same people were involved in Gamergate.

• 14:45Speaker 03

  Yeah.

• 14:46Speaker 02

  And, you know, that needs to be verified in a way. But what's important maybe is the fact that someone felt that. Because what that means is that they're picking up on similarities that exist. And I think we can look at these and say that there are some similarities. Sock puppeting, amplifying a hashtag, dog piling on different victims, doxing them, swatting them, death threats, rape threats. Maybe those are also just regular, you know, more reoccurring forms of harassment if we think about it. But if those are all the things being used and they're not being stopped, then maybe they are getting bigger and faster every time because there's no friction to stop them.

• 15:20Speaker 03

  Right. Also, like, up until a certain point, the internet was growing, right? Like, in 1990, you just don't have as many people using online communication tools as you do by 2018. So I think that that has to explain some portion of it. But reading through particularly the most recent examples in your report, which would include, like, Donut Gate, Pizza Gate, it's hard not to look and see a lot of tactics being repeated. And I think, to some extent, this does feel like a playbook that people are just running whenever they encounter something they dislike.

• 15:57Speaker 02

  I think that that's a pretty valid assessment. I mean, Gamergate, for example, wrote out, when they were starting a campaign, a micro campaign inside of Gamergate. So if they were deciding to attack the advertising of different media websites, they had a list of what you could do and how to do it. And they would publish this on Kotaku in Action on Reddit. That's their subreddit. So organizing is a major thing, right? I think Biella Coleman's book, Hacks or Hoax or Whistleblower Spy, gets into this a little bit with anonymous of how they would also create a campaign and then galvanize their followers and also disseminate information. So I haven't seen, per se, one playbook passed around from Gamergate to the alt-right. But also, you kind of don't need that playbook when you can see the effects

• 16:42Speaker 02

  of what happened in a previous campaign.

• 16:44Speaker 03

  Right. You can reverse engineer these tactics pretty simply. Because they're not sophisticated tactics necessarily, right?

• 16:50Speaker 02

  They're not. There's, like, so many things you can do. It's just that all the things you can do are extremely detrimental and dangerous, right? And very harmful.

• 16:56Speaker 03

  So you talked about platform friction. Literally, as we're talking today, Laura Loomer is chained to the front door of Twitter in New York. And it's insanity.

• 17:07Nilay Patel?

  That's just, 2018 continues to happen, like, at a furious pace, right up until the end. But what I always laugh about is like, I say this out loud, I don't even know if I fully believe it, but like Twitter is not the internet, right? Like, there's a great big internet out there, but Twitter seems to be where this stuff happens. It's the epicenter of all of these things constantly. Is it up to Twitter to provide friction? Is it a collective action problem? Does it actually happen on other platforms and we just don't talk about it as much?

• 17:35Speaker 02

  So there's two answers here, and I want to break them apart into two different answers. One, I think we just stop thinking about Twitter as like, a major part of the internet, when it's a major part of the Western English-speaking world. There's so many other forms of the internet out there, right? Like, what's happening on Sina Weba right now? That's a major, major part of the internet. What happens in mainland China on the internet is major. What are the different kinds of social networks in Russia? What are they talking about? How do people interact on there, right? What are the platforms of choice inside the Middle East? How are people communicating? You know, our internet is incredibly partitioned because of capitalism,

• 18:17Speaker 02

  and because of major technology companies. And I think that's really important to highlight. Like, when we talk about the internet, whose internet are we talking about, first of all? And what languages are read on that internet, or are more commonly used? So, the internet is different now than it was in 1990. But the internet in 1990, if it was seen as a utopia, it was also a white, Western utopia. Who got to access the internet in 1990? How easy was it? How affordable was the hardware? Where could you purchase the hardware? How could you connect to the internet? Did you have the bandwidth to connect, right? So all these are actually very important design choices to consider. This is a designed space.

• 18:59Speaker 02

  An internet, a space on the internet, is not just an open space like going into a park, because someone had to write code to render that space that you're in. So, that means that a choice went into that, and that, I argue, is design. Pushing that aside, though, for us in this room, people based in the US, probably a massive part of our internet, based off our ages, is probably a social network. So what are the social networks we use in this room? Reddit is a social network, Facebook, Instagram, Twitter. Are these our social networks of choice? What is different, in a way, I think, than Facebook,

• 19:40Speaker 02

  which is a walled garden, is that Twitter has the illusion of openness. And it's much easier to search. You don't have different privacy settings. So you can't be semi-public. You're either public or private. So when we think about what are frictions that could be added, there's lots of things social networks can do. When those things happen, it is less obvious to the common public. Do you see all of the harassment that someone faces on Facebook? No, and some of that is by design, because that does also make it easier to mitigate some forms of harassment. Right? But just because we don't see it, that doesn't mean that it's not there.

• 20:16Speaker 03

  Right.

• 20:16Speaker 02

  And that doesn't mean that it's not viral, in a way. So I think it's important to unpack how we're contextualizing things like harassment. Is it on a platform where virality can happen? Is it on a platform where media is easily embedded and shared? Is it on a platform where you can take something down? Is it on a platform where an algorithmic timeline can quickly intervene to where you see something later, so it's not happening in real time? So you have delayed effect of interacting with information? All these are our design choices. All these are important choices. So going back to your questions, or something Twitter can do, there are plenty of things they can do. I've just outlined some of these design choices. Should they do them? That's a different question. And how they do them is another question.

• 21:01Speaker 03

  How they do them is the thing that I worry about the most with Twitter. Right? Like, one of my recurring themes in my work is that Twitter is not a competent company. And I'm not trying to say that in a mean way. I'm trying to say that they try to do things, and the implementation often just goes wonky. Right? It is at a service where things just break because they're held together with spit and glue. And so I will give them credit for... At least beginning to take these issues more seriously. They've done an RFP. They're working with academics. They're trying to understand what it would mean to have a healthy platform. But they are just in like the research stage of that, right? And so how you get them from there to implementing a policy that meaningfully restricts harassment,

• 21:46Speaker 03

  that's just a very long way for them to go. I want them to get there. It's just at the moment, I'm not sure how to do it.

• 21:51Speaker 02

  They've been researching and tackling harassment since 2014. Like they had an advisory council that they brought a lot of different activists, academics, and policymakers to. They had Women Action Media conduct what at the time was like a canonical survey for them on online harassment. And they had worked with different academics. So I don't give Twitter a pat on the back because they've had time since 2014. They've spoken to very well-known victims of Gamergate multiple times. Jack, apparently if you're famous, it's very easy to get a hold of. And we have so many cases of harassment, right? We can look at Ghostbusters and Milla Jannopoulos. I forget the exact actress's name.

• 22:34Speaker 03

  Leslie Jones.

• 22:35Speaker 02

  Yeah. Jack intervened in that. We can look at Me Too and what happened accidentally with Rose McGowan's account. And Jack intervened in that pretty quickly. So I think it's disingenuous to say that they're only trying now. They've been trying for a while. I think a better question is, what are you doing and why are you moving so slow? And if that's because they don't know how to do something, they're not transparent about that at all. I think Twitter is one of the most, most opaque companies. You don't know why they make a decision. They don't publicly release a lot of things. Facebook actually looks more transparent at this point than Twitter does. Are we liking the outcomes of what they do that's different?

• 23:18Speaker 02

  But like, is there more insight into their processes? There is.

• 23:22Speaker 03

  Yeah, they could say a lot more than they do about the decisions they make and why. And they don't seem to be there yet.

• 23:30Nilay Patel?

  Well, so I want to ask you, I want to bring this all the way back to the thing that you said you do at the top, which is you help government organizations and NGOs like communicate more effectively through design. So Mark Zuckerberg is out there saying, I want a Facebook court. Every time he says this, my little lawyer brain explodes because that implies we're going to have decisions, there's going to be lawyers. You're going to issue precedent. We can then argue about whether the facts of our current case fit the precedent of your previous decision. That's a huge investment.

• 23:57Speaker 02

  Totally.

• 23:58Nilay Patel?

  Most countries can't do that well.

• 24:00Speaker 02

  Yep.

• 24:00Nilay Patel?

  But how do you make that understandable? Can you make that understandable to people?

• 24:05Speaker 02

  So there is actually a history of this kind of committee in social networks. If you think of Wikipedia as a social network, there's the arbitration committee. So prior to joining the Kennedy School, I was actually on the anti-harassment team at the Wikimedia Foundation. The arbitration committee is a group of administrators. So it's regular Wikipedians are then elected to an administrative level called ministers. And then they run for the arbitration committee, which you can view as the Supreme Court of Wikipedia.

• 24:32Nilay Patel?

  Yeah.

• 24:33Speaker 02

  And they hear really difficult cases that have not been able to be resolved. Wikipedia doesn't have a way to report harassment. It doesn't exist. They have different places where you can try to talk about problems that you're having or harassment that you're facing inside of the encyclopedia. And administrators can advise or try to do things. And if it keeps getting out of hand, you can try to file a case to the arbitration committee. Now, those on the arbitration committee have shown that they have contributed to the community in some way. What's important here and what is criticism of the movement is that doesn't include that you have a background, per se, in mitigation or litigation or research.

• 25:16Speaker 02

  What it's shown is that you're a community member withstanding. What is a community member withstanding? What does that mean? How does that scale? Does that mean you have a background in understanding diversity? Does that mean you have a background in handling certain cases? What's considered the way in which cases are handled well? Well, you look at the cultural norms of the community. So what are the cultural norms of the community? Wikimedia projects actually have a history of toxicity. They're very toxic communities, and it's pretty well-reported. And it's well-known inside the communities itself. So what are the norms here? What are the things that are sort of ruling in a way that are considered normal for this space while has to follow the norms of the space that you're in? So this often defaults to preserving the encyclopedia.

• 26:01Speaker 02

  It often goes back to looking at how is someone editing? It often looks at their standing as an editor. Are all these things that should be considered if you're being harassed or not? It gets really complicated.

• 26:11Nilay Patel?

  So it's funny that you're saying it gets really complicated because you're describing this to me and this is every criticism of the jury system in the United States.

• 26:17Speaker 02

  Totally. And I think, and the thing about it is, and this is where perhaps I differ as a member of the Wikimedia world, is I don't think that it necessarily works very well. It's also the foundation doesn't intervene in these decisions. There is a separation between the arbitration committee and the Wikimedia Foundation itself as a body. The foundation will intervene if there's more systemic harassment, something where someone would need a lawyer, something where if like more direct threats are brought in, that can be like someone threatens to show up at your workplace, right? Like the foundation will step in there, but the foundation is extremely hands-off with how harassment is handled, which my opinion is that's not necessarily a great stance to take. But it's this fascinating case that nobody knows about because nobody really talks about

• 27:02Speaker 02

  Wikipedia as a social network. And it's also important to highlight that there's over 200 different language wikis and they all have their separate cultures and different rules. And what that means is that there isn't necessarily an arbitration committee for every different language wiki. So there's an arbitration committee for English language Wikipedia. English language Wikipedia is not Wikipedia. So Arabic Wikipedia, for example, is not an Arabic translation of English language Wikipedia. It's people writing in Arabic, writing articles. And they're different articles.

• 27:34Nilay Patel?

  And do they have their own committee or is it just one committee?

• 27:36Speaker 02

  No, so I don't know if they have their own committee because while I was at the Wikimedia Foundation, we were not given a grant to study globally online harassment. We were just given a grant to focus on English language Wikipedia. So. There's all these different kinds of rules. It's a very fascinating subculture. And a point I want to make, though, is that this has existed before and it doesn't necessarily work. And the size of the community on Wikimedia projects is much, much, much smaller. Also, then it's important to highlight, what do you do on a platform? Because that's also part of the design of the platform. So, do you consider editing in the commenting section of a newspaper a social network? It can be. You know, I would argue that the Kenja communities that came out on the Gawker Media platforms

• 28:21Speaker 02

  felt like communities. It definitely was when I was reading and writing a lot in the Jezebel comments. You'd see people that you noticed before. That's how, it's a Gloria Ryan, I just remember her as Morning Gloria.

• 28:30Nilay Patel?

  Yeah.

• 28:31Speaker 02

  She's now a very well-known writer, but she was a prolific commenter in Jezebel. And then was able to pitch an article or two and then like eventually join the staff. Right? So like, that is a social network. But what you do on that social network is different than maybe what you do on Facebook. Right? Commenting on an article is a different kind of motivation space. And the design of commenting on articles looks very different than Twitter, for example. So I bring this up because what do you do on Wikipedia? What do you do on Wikidata? What do you do in Wikimedia Commons where you upload images? These are different kinds of designed actions than what you would do on Facebook. And you're also more constrained in your actions than what you do on Wikipedia.

• 29:13Speaker 02

  Because you're building an encyclopedia and there are hyper-specific rules as to how you add content and how that content is considered neutral, how that content is verified. So the arbitration committee, while it has all of its faults, they're also solving an easier problem because they're just talking about how you edit something. And fights that maybe occur from editing and fights that are related to Wikipedia. That is so, so extremely different than what could exist inside of Facebook. What are all the different kinds of fights you could engage with someone on Facebook that could be brought up to the Supreme Court of Facebook? Right?

• 29:51Nilay Patel?

  Yeah.

• 29:51Speaker 02

  And just run them through your mind for a second. It could be posting misinformation. It could be starting a misinformation campaign. It could be impersonating someone. It could be sending dick pics to someone. It could be soliciting child porn. It could be grooming minors. It could be stalking your ex. It can be... Your violent father, who's a drunk, found you, right? There's all these different kinds of things and it would mimic maybe more real life in a way.

• 30:17Speaker 03

  Yeah, it's like, you know, think of all the ways that human communication can go wrong and they go wrong that way on Facebook.

• 30:24Nilay Patel?

  Coming up next, here's some advertiser content from Dell Cinema about how binge-watching has changed everything.

• 30:30Speaker 04

  Kayla loves TV.

• 30:32Speaker 05

  I like to tell people that I invented binge-watching TV shows. I'm in it for the long haul.

• 30:39Speaker 04

  And chances are, you're a lot like Kayla. Over 70% of Americans are binge-watchers and they feel a deep connection to both the characters and the screens they're watching them on. Dr. Emil Steiner is an assistant professor who studies binge-watching at Rowan University.

• 30:56Speaker 06

  With the newer screens that are now available because of the crispness, the higher fidelity, they allow viewers to see a more realistic world. And that social realism creates greater feelings of connection with the people on those screens.

• 31:12Speaker 04

  According to Steiner, it's not just screen size and clarity that creates that deeper connection.

• 31:17Speaker 06

  The technology today allows viewers to control not just what they watch, but where and when they watch it.

• 31:24Speaker 04

  And this is great news for Kayla.

• 31:26Speaker 05

  I used to feel truly embarrassed about the amount of binge-watching that I engage in, but I feel grateful that the culture is totally supportive of this type of hobby that I have.

• 31:40Speaker 04

  If you're a person who can never say no to one more episode, check out the Dell XPS 13 with Dell Cinema technology. For incredible sound, color, and streaming, it's the laptop for people who watch things on their laptop from Dell Computers.

• 31:58Nilay Patel?

  Thanks to Dell for sponsoring the show. Learn more about Dell Cinema's amazing color and Intel Core i7 processor at dell.com/xps13. That's dell.com/xps13.

• 32:10Speaker 03

  I wanted to shift gears a little bit and look at getting back to your piece that you did and this history that you've done. I wondered what your piece has to say about the way that harassment campaigns end. Do they peter out? Do platforms respond effectively to them? And is there anything that you would say about platforms that have responded effectively to harassment campaigns over time and whether there are any lessons there?

• 32:36Speaker 02

  I don't think any platforms have responded well to harassment campaigns. Because we would have seen that in a post Gamergate world. And one of the things that I'm hoping to do in a longer write-up, I'm actually, I met today with a friend and we're adding a lot of data viz to the project, which I'm quite excited about. And I still have to fold in all of 2017, and this relates to 2017, is how do you differentiate between a hoax and a harassment campaign? And does it matter? And at times, I think, it does. It does if you look at the intention. It also does if you look at how it's analyzed then. So, you know, there was a lot of sock whipping happening and Operation Lollipop and your slip is showing.

• 33:20Speaker 02

  And "Father's Day" was a hoax hashtag. And this was designed to harass. But what if you look at, for example, Gamergate, which started off as a harassment campaign? And hoaxes appeared out of that. So, later on, in the write-up, I talk about how a Sikh man, he was photoshopped in to look like one of the Paris bombers. That was related to Gamergate. It happened a year later. And that was a hoax designed to harass. So, it's slightly different. It took on a much different kind of meaning. But it was a hoax campaign still under the umbrella of Gamergate. And so, do these things matter? I think that they do when we start to get into misinformation, disinformation campaigns

• 34:03Speaker 02

  that end up producing harassment. There, for example, was one female reporter, she was Asian. And it was coming out of like, something related to Trump or to a trial. And there was this woman in the background like taking a photo of something. And 4chan was trying, or Reddit was trying to identify her. And they kept misidentifying these different female Asian reporters. So this was a campaign that broadly you could, I think, define as a harassment campaign. It was, I think, related into this sort of umbrella of the alt-right. But then it created micro harassment campaigns against innocent people that were misidentified.

• 34:42Nilay Patel?

  Mm-hmm.

• 34:43Speaker 02

  So do we see this happening in misinformation campaigns? We totally do. I think this is where pizza gate's really important, right? How many people were negatively affected by that? So many people. So many people's children. Their images were used and misused, right? That wasn't a harassment campaign targeted to hurt someone in the way that in Father's Day was. But harassment was an outcome of this massive campaign. And so, one of the things that I'm trying to think about, how to argue, or rather, I'm structuring the argument around now in the write-up, is that, the difference of a hoax and a harassment campaign and in the light of 2017.

• 35:18Speaker 03

  And are these things like wildfires that just kind of burn themselves out over time when they use up enough oxygen?

• 35:23Speaker 02

  I would think the way to think about it is, 2018's the best metaphor to describe harassment campaigns. Something happened last week, something happened three weeks ago. When were the Kavanaugh hearings? Were they a month ago? Two months ago? How much stuff has happened since then, right? That's the same thing with harassment campaigns. The reason they petered out is because something else happened.

• 35:43Nilay Patel?

  Hmm.

• 35:44Speaker 02

  Or like, maybe it did just sort of wind down. But then something else occurred. And in people's lives, not just like another harassment campaign popped up, but like something else happened and the internet turned its attention to that.

• 35:56Speaker 03

  So let me propose an intervention that I do think has been effective. So something that a lot of reporters noticed over the past year and a half, two years, was that in the immediate aftermath of any calamity, shooting, wildfire, or something like that, you would go to YouTube and you would search for it and you would immediately get a bunch of conspiracy videos explaining that there was some nefarious cause to the calamity that the mainstream media Wouldn't tell you about. And because of YouTube's recommendation algorithms, people would often stumble onto these videos, even if they weren't looking for them. And so, an entire genre of content just became searching YouTube after calamities. And we would write about all of the garbage that was floating to the top. And, you know, after a while, YouTube got wise to this and now very closely monitors those search results in the aftermath of those calamities.

• 36:41Speaker 03

  So they've sort of spun up a team that can intervene and that can promote mainstream coverage that tells people what is actually happening. So, you know, I would point to that as an effective intervention that can catch some of this stuff in the moment and stuff that, you know, might otherwise start as misinformation end up in harassment, like a pizza gate scenario. They can hopefully nip those things in the bud.

• 37:03Speaker 02

  But those aren't happening with harassment campaigns, though, right? Those are happening with things like wildfires, but we aren't seeing that happen with harassment campaigns.

• 37:11Speaker 03

  So let's talk about that. So, let's say that someone was like an individual citizen, whether they were prominent or not, and they were targeted by some sort of harassment campaign. In your mind, is the right thing to do for the platform to go in and sort of strip out all content, to stop surfacing hashtags, or like send hashtag search results to an empty set of search results? Or have you started thinking about what in your mind is like a fair and effective way to target some of these harassment campaigns?

• 37:41Speaker 02

  Sure. I think a big thing is probably them looking at their policy around how they're defining online harassment and considering rewriting it. Another thing, I think, is maybe like, again, as you suggested, lessening some of those or sending the search results to an empty space. But I think the biggest thing is probably having a team that can help them understand much more quickly, like that this is a harassment campaign. I know from a lot of my work with different platforms that a big thing that they're worried about is online censorship and how online censorship relates to online harassment. And that's also a very American stance to take on looking at content and the ways in which companies decide when to take content down or not. It always falls back to...

• 38:25Speaker 02

  Is it censorship or not? Well, you know, we can engage in censorship, but they do. They take things down all the time. They take things down all the time that shouldn't be up there for a variety of reasons. I think the bigger thing is having a deeper understanding internally as to what is considered harassment, what are the different levels of harassment that they are willing to allow on the platform and not, and how culpable they feel with that harassment.

• 38:48Speaker 03

  Well, but to bring us back to a point I was making earlier about Twitter, like Twitter's harassment policies are like relatively clear. They ban all kinds of harassment. The problem is they don't enforce it, right? Or they enforce it extremely erratically. So, you know, I'm open to the idea that rewriting the policy would be a big help, but to me, I see the issue as being much more related to their ability to enforce their own rules.

• 39:06Speaker 02

  For sure. And I hope that my answer wasn't saying that this is just a policy answer. It is also enforcement. It is having a team that can understand enforcement. It's also making sure that team, which is probably content moderators, has the contextual understanding as to what is happening, that they have good tools to take down content, that they, for real, though, like some of the tools that people are given to content moderate, from like what we have seen and like different like leaked information about content moderators is awful. What they have to work with is really, really not great. And it's upsetting if you think about it from a design standpoint, when you think about how the beacons of modern design in the United States will be software design and it will be from major platforms

• 39:49Speaker 02

  that what content moderators have to have to use to moderate platforms is really antagonistic. Outside of the content that they are forced to look at, the tools that they're given and the timeframes that they're given to analyze content is, I want to argue, like a human rights violation.

• 40:04Speaker 03

  Wow.

• 40:05Speaker 02

  They have under 10 seconds to make a decision. Sometimes what they're looking at is extremely violent content. They have to look at it all day and they have specific quotas that they often have to meet. So how do you build context out of that? How do you build context out of something like a harassment campaign that's happening in Steubenville? How do you build in context to that? What people often have is like a checklist that they have to memorize. It's based off policy. That's not great policy around defining harassment and they have to make a split second decision. And they have tools where there is not enough shown to them to perhaps understand what they're looking at. So, you know, what are all the solutions here? The solutions are perhaps redesign the way in which content moderators are engaging with

• 40:48Speaker 02

  content on the platforms, make that experience better.

• 40:51Nilay Patel?

  Mm-hmm.

• 40:52Speaker 02

  It's hard to provide suggestions right now because there are lots of platforms that use different tools.

• 40:57Nilay Patel?

  Yeah.

• 40:58Speaker 02

  And they're all designed differently, you know? So that can be, uh, do you have the original text that was used with it? Do you have the text that was before the event occurred and after? Do you have, if it's a video, for example, do people have time to look at the video?

• 41:10Nilay Patel?

  Mm-hmm.

• 41:11Speaker 02

  Are they responding to something that's in the commenting section versus the video? Is there another video they have to see? You know, is this video in response to another video?

• 41:19Nilay Patel?

  Yeah.

• 41:20Speaker 02

  All of these things, like, how, how would you lay out a product to do this? Well, you could have multiple videos being shown. You could try to, depending on how, like, if you have reporting, right? Someone's filed a report, ask them, ask the reportee if they want to highlight another video in a specific section. Can people have a couple minutes to look at something?

• 41:40Nilay Patel?

  That one, to me, seems the most insane.

• 41:43Speaker 02

  Yeah. So, like, it's thinking about stuff like that. Like, the problem is not just policy, it's also, like, how do you enforce harassment? Well, who are the people that enforce harassment?

• 41:52Nilay Patel?

  Yeah.

• 41:53Speaker 02

  It's content moderators, without them realizing it, too. Like, they're looking at flagged information. So I think a really big problem, again, always comes back to design.

• 42:04Nilay Patel?

  So I want to, I want to mention something, and I want to ask you one last question, because you've already given us so much of your time. It's funny, as you talk about moderation and pulling stuff down, we ran an excerpt of Sarah Jeong's "The Internet of Garbage" on her site, right before she went off to The Times, and the, the section we picked was her talking about the, the intersection between content moderation and copyright law, and how the copyright law has stepped in to form the content moderation function, because fundamentally, there's money underneath it, and there's, like, liability, and that creates, we're actually just going to pull this down. And then you look at a platform like Amazon, which is, has many problems of its own, but it's an economic powerhouse, and there's money there, so there's literally, like, Amazon litigators that exist. So if your product gets kicked off the store, you can, like, hire an Amazon lawyer, and

• 42:49Nilay Patel?

  they will go fight Amazon on your behalf. None of that seems to exist. On the regular, on the sort of like consumer platforms, right? There's no YouTube moderation lawyers, there's no Twitter moderation lawyers. There's not this economic incentive like humming along in the background to make sure everyone's a good actor. Can you replace that incentive in some way?

• 43:08Speaker 02

  Well, there are lawyers that have a lot of expertise in social media, especially with now online harassment lawyers. So Carrie Goldberg immediately comes to mind. She's actually based in New York. She's one of the most well-known revenge porn lawyers, I'd say, but also just in general, the person that I think of when I think of online harassment policy and social networks. And Danielle Citron, for example, right? She's a law professor, I think, going to Boston University right now, from the University of Maryland. So there are lawyers out there with a specific expertise around online harassment, right? Asking about economic incentives, I mean, we can look at how Twitter was not purchased. And I think one of the main reasons that people kept citing was the toxicity of the

• 43:53Speaker 02

  platform and also like rampant harassment.

• 43:56Nilay Patel?

  Yeah.

• 43:56Speaker 02

  So even if it doesn't feel like it, I think that there is a dollar sign on harassment. There are economic incentives.

• 44:02Nilay Patel?

  One of my running jokes is I tell the CEO of this company that we should buy Tumblr for a dollar, because like, why not? Like, what else are they gonna do with it? And he's like, figure out what to do with the porn and we'll talk. It's like this, like, it's like a half joke that we tell each other constantly, but he's not not joking.

• 44:16Speaker 02

  I mean, but some of the porn on Tumblr is good.

• 44:22Speaker 02

  You know, I mean, but yeah, I think harassment has a monetary bottom line, though. And as you've just pointed out, right? Like, why wouldn't someone want to buy Tumblr? Well, a variety of reasons, you know? Why wouldn't someone buy Twitter? Well, I can think of a couple of reasons why. If Facebook were for sale, would you buy it?

• 44:41Nilay Patel?

  Right now, yes. Right now, I would, just because I'd get to fire some people. But that's like, that's me.

• 44:45Speaker 02

  But like, but like, you know, how do we, if we're thinking of these companies, which I also want to argue are a part of social infrastructure, so their worth is not actually something I think we should put a dollar sign on. They function as social infrastructure.

• 45:00Nilay Patel?

  So that's my next question, and this is where I want to end on. The beginning of this piece, we focused a lot on the timeline. The beginning of your piece actually sets up these definitions, like, here's what harassment actually means. Let's come to a shared definition. These companies are under no obligation to share their definitions with each other. Right, so it'd be great if Facebook had an internally consistent rubric of harassment and they were teaching their moderators well. But if you're just a regular person and you're using three platforms every day and YouTube's set of definitions are wildly different than Facebook, you're kind of hosed. Is this where the regulation begins?

• 45:32Speaker 02

  It can be. One thing I will say is I'm almost 100% positive that every platform has consistent internal definitions of harassment because they have, all of them have, good policy team members. So that's something that they define because that also does drive the different kinds of technology that they build in a way and the different kinds of design choices that they make. So they have clear policy. What's different is do users understand how that policy is enacted when interacting with content? And that's where people don't understand, right? Because it's not necessarily clear in the outcomes as to what's taken down, what's kept up and why. And that's not just a content moderator's fault. A lot of that is also just that like the policy doesn't at times make sense.

• 46:16Speaker 02

  And it does differ platform to platform. So, you know, there could be some kind of legislation that says like platforms have to make public their internal policy choices. I would hesitate to say that we should then have one overarching legislative decision around how to define harassment because what we would end up with is a very US-focused definition and view of harassment. And that would be implemented upon so many different countries. So for example, Germany looks at hate speech in a very different way. Like what you're allowed to sort of post that falls under hate speech is different than the US. You cannot post Nazi insignias online, for example. France, for example, has different rules around how children are protected.

• 47:01Speaker 02

  I think they were one of the first countries to implement this right to be forgotten that Google then had to implement. But that's not implemented on every single product or in every single country, right? So it's important to recognize that like the definitions we have in the US around online harassment, should those be definitions that, for example, the EU has to implement?

• 47:20Nilay Patel?

  Right.

• 47:21Speaker 02

  Or should we be following the EU? Which is also another question to ask. Like, should we be looking at GDPR regulations? And thinking about how that piece of legislation is going to have long-term design effects and technology effects as well as policy effects. And should the US or should social networks, which are technically based in the US, should they be looking towards the EU around policy decisions? Like, these are the kinds of much more high-level questions we have. Once you have the information of how does YouTube view harassment versus abuse? Is abuse more extreme? Do they have another word for an extreme version of harassment? Or a less extreme version of harassment? And how does Facebook view that? Or Twitter? And then how do they view it in different countries?

• 48:05Speaker 02

  Is a country going to write a law that holds all of them to a certain standard?

• 48:08Nilay Patel?

  Do you think it should be our country?

• 48:10Speaker 02

  No. I, no, I, I really don't. I always hesitate to say that, like, regulation is the answer. But I don't think that this is something that should fall in the US. I also think right now, under a specific administration, would not be a good time.

• 48:27Nilay Patel?

  This is, I always bring this up. Like, do you really want these people writing that law right now?

• 48:30Speaker 02

  No. But I also don't think that a law would necessarily solve this. Because then you have deeper questions, which is what Facebook and Twitter and Google are facing right now with the GDPR of, well, how do we actually process what your law is requiring? I think there's something specifically in it around, like, certain content has to be taken down or analyzed within 24 hours and then taken down. That's actually not at a pace that most companies can work at, regardless of size. So like, that stipulation, for example, does that benefit anyone?

• 48:59Nilay Patel?

  Yeah. And I think that's specifically data breaches, right, Casey? Sure. Yeah. Let's all guess at what's in the GDPR right now.

• 49:06Speaker 02

  I feel like I should know.

• 49:07Nilay Patel?

  Like, don't hold it against me if I get this wrong, but I'm like 75% it's data breaches. Okay, last question. We talk about this a lot, The Verge cares about it a lot, I think our listeners care about it a lot. I often feel paralyzed, and I'm in a position of, like, some authority. Like, I can see a harassment campaign directed against a Verge person and be like, that's dumb, stop it. Right? And like, 50% of the time, it stops. But I still feel paralyzed all the time. What can the individual actor do in this moment?

• 49:32Speaker 02

  I think it depends. I mean, there isn't one answer. There's not one answer to rule them all. But let's go small for, for a step. Let's say you see harassment happening against someone. There has been a lot of research done into what's called the bystander effect. So someone has equal privilege to the one engaging in harassment, saying to the harasser, hey, like, I didn't, I don't think that's cool. Or why are you doing that?

• 49:55Nilay Patel?

  Yeah.

• 49:55Speaker 02

  Or hey, you should stop. There's been a lot of studies done on how actually this is useful, that this is a way to build better norms. So if you feel paralyzed by something and you want something to stop, you should try saying something.

• 50:10Nilay Patel?

  Yeah.

• 50:10Speaker 02

  You should try using your privilege for good. If you want to take it a step further, I think now is a great time to organize and to look at like-minded groups or create a group that is maybe demanding for certain kinds of action that you want to see on platforms. I think protest and organized protest has shown to work really well. For example, in Berlin, Google's trying to open an office and a bunch of local anarchist groups and protesters got together and made enough noise and protested for many months. And Google is not opening this one big office there. They have other offices.

• 50:45Nilay Patel?

  Yeah.

• 50:46Speaker 02

  But I think this is a good example of direct action working. I wonder how long Laura's going to stay at Twitter.

• 50:52Nilay Patel?

  Mm-hmm.

• 50:52Speaker 02

  You know? But imagine if a bunch of people that didn't agree with Twitter's policies actually organized outside of Twitter's New York offices and San Francisco's offices and were just consistently loud. Or were just consistently there and showed up and handed out flyers and said, "I have a problem with hate speech on your platform." I think that that would deeply affect Twitter because you're in, you're close to their space. So I think that there's all different kinds of actions that people can engage in.

• 51:21Nilay Patel?

  Yeah. Well, we'll see how long Laura last chained to the door.

• 51:25Speaker 02

  I mean, it's really cold.

• 51:26Nilay Patel?

  It's right. It's also like, it's also insanity.

• 51:30Speaker 02

  Katie Notopoulos from BuzzFeed tweeted, like, "I hope Laura's in it for the long haul and is wearing a diaper."

• 51:37Nilay Patel?

  She, right before we came on air, she was definitely talking about peeing herself. So, I don't know about the diaper. She's clearly in it for the long haul. It is one of the craziest things to happen today. I'm sorry, I'm almost sorry to end it here, but in the context of this very good, very serious, very nuanced conversation.

• 51:49Speaker 02

  It's a perfect way to end.

• 51:51Nilay Patel?

  That's also just happening, like, right down the road. Caroline, thank you so much for joining us. I really appreciate it. You've obviously done just incredible amount of work on this. I hope you come back soon.

• 51:59Speaker 02

  Thank you. Yeah, thanks for having me.

• 52:01Nilay Patel?

  All right, I want to thank Caroline Cinders for joining us on the Vergecast this week. Quick Laura Loomer update. She got cold and just asked the police to cut her handcuffs so she could go home. She's still banned from Twitter, of course. If you are really interested in this topic, I want you to check out the episode of Why'd You Push That Button that we put out this season about why people delete their tweets. It's a really interesting knock-on effect. Harassment campaigns exist, so in response, people have started using paid tools to delete all their tweets. Ashley and Caitlin did a great job with that episode, so go check it out. We'll be back later this week with the regular Vergecast. And next week, a super special surprise that I can't tell you about, but it's coming. And now, another word from our sponsor, Dell.

• 52:38Speaker 01

  The Dell XPS13 with an eighth-gen Intel Core i7 processor is the laptop for people who never say no to one more episode. With lifelike color, brilliant sound clarity, and smooth streaming, Dell Cinema technology makes whatever you love to watch even better. Call 800-BY-DEL to learn more or visit dell.com/xps13. Sponsored by Dell and Intel.