Protecting us with the Database of Evil

Online platforms and their users are susceptible to a barrage of threats – from disinformation to extremism to terror. Daniel and Chris chat with Matar Haller, VP of Data at ActiveFence, a leader in identifying online harm – is using a combination of AI technology and leading subject matter experts to provide Trust & Safety teams with precise, real-time data, in-depth intelligence, and automated tools to protect users and ensure safe online experiences. Featuring: Matar Haller – GitHub https://github.com/matarhaller, LinkedIn https://www.linkedin.com/in/matarhallerChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: ActiveFence https://www.activefence.comUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  What we do is we basically, we combine this like very, very deep subject matter expertise with our technologies. So we're a technology company and yeah, we also have sort of experts in the field, in the domain, like experts in the field of researching human trafficking and really understanding that space or in misinformation and different types of misinformation and in hate speech and in terror. And so they speak the languages, you know, they research the space, they understand it, they know the key players, they know the different organizations, the keywords and you know, this is an adversarial space. It's constantly changing and so they make sure that they stay up to date. And then what that means is that us on the data side, we can basically take their ideas, take their knowledge

• 0:41Speaker 00

  and then engineer features out of those, right? So really translate the human knowledge into our models. So then we can go out and automate that and do it at scale.

• 1:00Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now. If you haven't already, head to practicalai.fm for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out at fastly.com and to our friends at fly.io. We deploy our app servers close to our users and you can too. Learn more at fly.io.

• 1:36Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International. And I'm joined as always by my co-host, Chris Benson, who's a tech strategist with Lockheed Martin. How you doing, Chris? Doing very well today, Daniel. How's it going? It's going great. So yesterday was voting day here in the in the US. Indeed. And I did go to the voting place and it was interesting because in line I could hear people talking about like cyber threats to like the voting machines and other things like that. And so my mind was already actually thinking about these these things because we have a really interesting topic to talk about today that's that's in that same vein.

• 2:22Daniel Whitenack?

  We're privileged today to have with us Matar Haller, who is VP of Data at Active Fence. Welcome, Matar.

• 2:29Speaker 00

  Hi, thanks for having me.

• 2:31Daniel Whitenack?

  Yeah, and Active Fence, I've read a bit about it and the sort of website talks about this like barrage of threats that is, you know, online platforms are susceptible to now, which Active Fence is addressing in various interesting ways, which we'll get into. But I'm wondering if you could kind of give us a picture, like if I'm going to run an online platform of some type, maybe it's not even like, you know, I'm likely not going to start and run the next Facebook, but I might very well, you know, start and run some type of software company that provides a online platform to do something.

• 3:11Daniel Whitenack?

  What should be on my mind and what's the reality of kind of online threats that I might need to be aware of if I'm getting into that space?

• 3:20Speaker 00

  Yeah, so first of all, I think there's one thing to think about is that any time that you have a platform that has any type of user-generated content, whether users are uploading photos, or they're chatting, or they have comments or anything like that, you're going to have tons of data very, very fast. And just it's prime for people to, you know, post wonderful things, but also some really, really dark things, which we've all sort of seen and been exposed to. And so one thing to keep in mind is that trust and safety, just basically safety online, it's not really a nice to have anymore. At this point, it's a competitive advantage. It's kind of a basic expectation, right? So users are expecting it, advertisers are expecting it, parents are expecting it, the public expects it. So.

• 4:05Speaker 00

  If you're gonna spin up a platform, so first of all, best of luck. And second of all, you need to keep this in mind, like, from the get-go before you sort of find yourself down this this rabbit hole. One thing that I think is really important to keep in mind is that although trust and safety isn't a new industry, really, it's really now finally becoming something that people are aware of. Like I said, it's this basic expectation. Now, it's not only users, but also like regulators and legislators. Um, there's new legislation coming in that's making it even more at the forefront. And the basic sort of content moderation that is out today doesn't really make the cut. Um, to follow up, like, the second part of your question about sort of like, what kind of harms are out there?

• 4:50Speaker 00

  So, online harm is really multi-dimensional. We can see it in different media types. So, we've seen it in games and like merchandise sites, chats, texts, you know, video, audio, things like that, across many, many different languages. And also, different types of violations. So, you have, you know, white supremacists and terrorists and human trafficking and sort of these like really, you know, painful sorts of things. Um, also goes into like misinformation, disinformation, fraud, spam, cyberbullying and so forth. And so, it's this really, really complex space that you need to have a deep understanding of to understand how to address.

• 5:31Daniel Whitenack?

  And in the sort of, I guess, up until this point, you were talking about like content moderation and how it has evolved over time, but is still kind of lacking in a sort of traditional sense. What does it look like? I mean, is content moderation like, people might have in their mind, oh, I have a blog, right? And I'm gonna choose whether I allow people to like post a comment or I have to approve that comment before it's posted or or something like that. So, this sort of content moderation, is it, in your opinion, kind of as where we sit today, is most content moderation sort of reactive at this point?

• 6:12Daniel Whitenack?

  Or how do you view like, you know, how how are most people approaching the problem right now, maybe? And and why is that lacking?

• 6:20Speaker 00

  So, there's kind of different levels, I would say, of uh content moderation. Um, at this point, To just sit and moderate every single comment gets out of hand really, really fast. And so there is some level of automation and that, you know, started to being introduced, right? And so the first sort of basic level is let's go out and look for keywords, right? Like, I don't want any slurs on my platform. I don't want anyone calling anyone any of the like words, you know? Um, I don't want that there, so I'm gonna ban all those. Um, I know that people might have, and then, and then people kind of get a little trickier, right? And so they'll say, well, you know, what about if we use an emoji? And so there's different kinds of emojis or combinations of them that also can be used in a hateful way.

• 7:05Speaker 00

  And so you can say, okay, well, I'm gonna, I'm gonna ban those. And I'm gonna ban like these specific keywords and I'm gonna ban uh these N-grams because you know, these phrases are bad, like I hate Jews or I hate what, you know? So I'll, I'll ban that. And that, that works okay, but very, very quickly you get to these cases where keywords are either insufficient. So like I said, with emojis or with Leespeak, so you can write, for people that aren't familiar, it's basically taking a word and replacing letters with numbers. So Adolf Hitler uh can be written like pretty much all in numbers. And so it's kind of to evade detections and only, you know, on a need to know basis sort of thing. There's also in, in the keyword space, you have numbers.

• 7:50Speaker 00

  So um like fourteen eighty-eight is a white supremacy number. Eighty-eight for Heil Hitler and fourteen is like a, a phrase that, that they use, um, like the number of words in the phrase. And so you can say, okay, well, I'll add all these to my dictionary. But then you get to the place where you say, well, what about someone that's telling someone else, don't call me, you know, some slur, don't call me that. And so now they're using the slur, right? Is that something that you want to necessarily ban? Maybe in some cases, but in other cases, you're gonna need sort of a deeper understanding of how language is being used before you just go out and outright ban it. And so you see this sort of like evolution in terms of the ways or the approaches that platforms are taking to moderate the space and sort of looking at.

• 8:34Speaker 00

  The context in which language is used is sort of the first step in that.

• 8:39Chris Benson?

  I think I just realized, and that was a great explanation, and I just realized how sheltered I am because several of the things you referred to, I had, I just didn't know at all. So

• 8:51Speaker 00

  Yeah.

• 8:51Chris Benson?

  I guess I'm very sheltered. But I'm curious, when we were first starting the conversation a few minutes ago, you also mentioned misinformation and we were just kind of diving into kind of some of those specific use cases on hate speech. How does misinformation, because we've seen, you know, we've been dealing with hate speech for a long time now, but you know, misinformation, you know, in the last few election cycles has really become a huge issue and obviously in national security issues and things like that, it's big. How does that fit in? I think in my mind, I've kind of thought about there's hate speech and there's misinformation and all. Is there a connection between them all? Are they bound together in some way, the way you see it? Or are these distinct, separate kinds of things? How do you think about it? How do the folks at your company think about it?

• 9:35Speaker 00

  That's a really interesting question to take sort of these two examples of these two specific violations. And there's, I think violations are sort of on this spectrum, right? And so you can think about these sort of more like evasive violations. So things that are, you know, that kind of are more difficult to find or require subject matter knowledge. So hate speech, right? You need to know these keywords, you know, these things. And then you also have these just sort of more like common violations where some of them, you're like, well, maybe it's not even a violation, like nudity or profanity or things like that, where it's just more, more out there. And everything kind of lies almost sort of on the spectrum. And you can go like spam and fraud and so forth, until you get to like the really dark, you know, like, like child safety and abuse and things like that.

• 10:19Speaker 00

  And with misinformation, it's actually interesting because it's not really trying to evade, right? Like that's the whole point. On the other hand, it's really tricky to find and to understand. And there's lots of organizations that do really wonderful work of fact checking and of, you know, like keeping up on the trends and really identifying misinformation. And there's also sort of these techniques that we can use to, once we've identified sort of like a specific type of misinformation, then we can use it to find in that it's going viral and so forth.

• 10:47Chris Benson?

  It's a real struggle to understand like, you know, kind of how to put that in context, because you could say things in a misinformation context that are, you know, that there's no hate speech in it, you know, explicitly, there are no banned, there's none of that is there. And yet, as we've seen in recent years, it can do great harm. So it it seems like a very hard target to go after and be able to to mitigate it in a in a sane and reasonable way.

• 11:13Speaker 00

  Absolutely. I think that's one thing that's really unique about Act Defense and what we do is that we're, what we do is we basically, we combine this like very, very deep subject matter expertise with our technologies. So we're a technology company. And yeah, we also have sort of experts in the field, in the domain, like, you know, experts in the field of researching human trafficking and really understanding that space or in misinformation and different types of misinformation, in hate speech and in terror. And so they speak the languages, they, you know, they research the space, they understand it, they know the key players, they know the different organizations, the keywords. And this is an adversarial space, it's constantly changing. And so they make sure that they stay up to date.

• 11:55Speaker 00

  And then what that means is that us in on the data side, we can basically take their ideas, take their knowledge, and then engineer features out of those, right? To really translate the human knowledge into our models. So then we can go out and automate that and do it at scale. The reason that it's so interesting is because, as you all know, models drift, they decay. And so you can go out and you can, you know, retrain your model and get new weights and you're great. Except if you're in an adversarial space, then not only are you drifting, but your reality is like, it's so not unstated, like not stationary, there's just changing from underneath you. So as it's changing from underneath you, you need to like hurry up and reengineer your features. And so we're constantly engineering new features, retraining our models,

• 12:38Speaker 00

  and also thinking about just like, what else can we possibly extract from this data that's coming in, right? We're analyzing. Text, video, audio, everything, basically, anything that we can get our hands on and just really milking whatever we can out of it.

• 12:51Daniel Whitenack?

  That's super interesting. I have so many questions, a really interesting, really interesting technology, but also like the infrastructure and management part of that is, I'm sure, a great challenge. But I'm glad you brought up the modality thing. And I also saw, you know, on your website, you talk about different languages as well, which it seems like this is definitely now in terms of the way people communicate online. Like I, you mentioned emojis. I was also thinking of like GIFs or GIFs, depending on who you are, or, you know, posting memes with text, you know, in the image. There's also, you know, of

• 13:33Daniel Whitenack?

  course, like you're talking about videos and audio, audio messages, all of that. I guess as a more general question is language, but sort of like multimodal language, is that like your like primary area of of research? Or are there other things like outside of communication in terms of like the threats posed to sort of online platforms where someone's not trying to communicate a certain message messaging or something, but it's still a threat to to the platform in one way or another? I guess maybe spam would be spam would be an example of that. But I don't know if you have have other examples, or is it really

• 14:16Daniel Whitenack?

  kind of from your view, a lot of what you focus on is the communication and language piece?

• 14:20Speaker 00

  So the goal, it's not necessarily language. And in a second, I'll talk a lot about like contextual AI and what that means. But really, our goal is to enable users to be safe online, to have a safe experience, right? I'm a mom, I have three kids. I started working at Actifence and I said, oh, gosh, my daughter is not getting a cell phone until she's 35, like, forget about it, right? Because you're suddenly exposed to all this. And then you say, but, you know, that's why what we do is so important, because that's our whole, that's our whole goal. It's not only about language, and that's just one form of communication. There's lots of things out there. And Like for a bunch of concerned parents, like let's really make this a safe like the fact that Chris is is still sort of in this sheltered bubble is amazing. Like I want everyone to be in this sheltered bubble, right? And so that's, that's kind of the idea.

• 15:06Chris Benson?

  I think I just need to correct it. It depends. There's some bubbles I'm sheltered from, I think, and there's some I probably am not. Probably so, yeah. Yeah, like when you were talking about the hate speech, the specific numbers that meant stuff, I was like, I didn't know that. So anyway, I didn't mean to cut in, but I.

• 15:20Speaker 00

  Why would you?

• 15:25Speaker 00

  Yeah. So I mean, so, so, so, so language is only one part of it. And I think one thing that we really get into is, is context. And so let me, let me take you on a journey through context and we'll end up at means, which to me is like.

• 15:35Chris Benson?

  Sounds, sounds great.

• 15:37Speaker 00

  So we were talking about like the language context, right? And so how keywords and n-grams, they just don't cut it, right? You need like language models. So, you know, transformers and so forth to really get an understanding of, of what is being said in the context in which it's being said. And so those are the kinds of models that we end up training and that we have data for. We have, like I said, we have like our subject matter experts and, and, and policy experts that are able to ensure that we're capturing things that are sort of on the edge and border because that's where things get interesting, right? And that's, that's where I want to be able, that's how I'm able to get the difference between, you know, I'm proud of being a whatever, because I'm like reclaiming that word versus, you know, you are a, you're not allowed here. And even within the hate speech of language,

• 16:21Speaker 00

  there's, you know, insulting hate speech, not insulting hate speech, you know, like, hey, wasn't that a great KKK rally yesterday? I really like your Proud Boys tattoo, right? It's hard to catch those things.

• 16:33Chris Benson?

  That's a surreal statement right there, that example. I just.

• 16:36Speaker 00

  That's never been said on your podcast, right?

• 16:38Chris Benson?

  No, it's never been said. Just like you said that I was like, I was like, wow, I'm, I'm not talking to the people that are saying things like that. So anyway, I'm sorry, go ahead. I just, it's, it's novel to me to hear some of this perspective.

• 16:52Speaker 00

  But then we can sort of go to the level of, okay, so let's say I um, we're now in the image space, right? And so one thing that we're able to do, and again, because we have this like deep subject matter expertise is we're able to search for logos, right? So logos of terror, you know, like, fuck, it's like rare terror groups through, you know, small that are hard to find. So we're, we, we, we know this space. And so we're able to go out and find, you know, do logo detection, find those particular logos, identify things. Um, and then you say, okay, great. So here's a video, found the ISIS logo. Great. Check. Terror. And you say, well, wait a minute, but there's also the CNN logo here. So suddenly, even though it's a snippet from ISIS,

• 17:31Speaker 00

  it's, you know, the context in which it's used is non, doesn't make it violative. It's suddenly, it's interesting. It's important. It's historical. It's whatever. You can see the same things with like videos of Nazis marching, right? Like sometimes that's glorified and sometimes that's just like historical, you know, it is what it is. So that's another level of context. We have to sort of look at the context of the, you know, one signal out of the image isn't enough. Um, or out of the video. Another thing that that we like looking at that, you know, some points like is that you can look at the context in which the image is being used, right? What is the title? What is the description? What are the comments? We have an example that I, that I like using where you see sort of non violative text, you know, I love him. Right? And you're like, oh, that's fine. Who cares?

• 18:11Speaker 00

  And then when you zoom out, you see it's like, I love him with a picture of Osama bin Laden. And suddenly like, hmm, that's suddenly more interesting, right? Um, suddenly it becomes violative. So you can't just take any one piece in isolation. Or there's an example of um, like a some chef that's like, it's hard to do this on a podcast, but like showing knives. And he's like demonstrating these knives and showing these knives and his hands are all cut up because he uses knives. And if you do just object detection, it screams at you like weapon, weapon, weapon. Like, oh gosh, this is like this terrible video. And then you analyze, you know, the title and description and the comments and the channel and everything along with it. And you're like, no, that, you know, he's teaching about knives. It's a chef video, like not, really not interesting.

• 18:52Speaker 00

  And so looking at things, you know, just as keywords in a sentence aren't enough, also just, you know, looking at an image by itself isn't, isn't going to tell you whether or not something is problematic. And so that's sort of this like idea of contextual AI that we're really, that we think about a lot is like, what is the context in which something is used? And context can mean lots of things. It can also mean the policy, right? So different platforms have different policies. Some platforms will say baby's first bath is like a child abuse, like you cannot have it, you know, child nudity. And others will say not a big deal. So that's like another level of context that our models need to deal with.

• 19:44Daniel Whitenack?

  I mean, just knowing about where NLP models or other models fail, like this area of like sarcasm and humor is, is so difficult. And there's like this further distinction that you're drawing out, which is, well, there's some memes that are jokes and sarcasm, right? There's some memes that are jokes and sarcasm, like to the point of being very, very harmful. And also that's kind of tied into the context of where they're, you know, where they're put or the timing of when they're put somewhere or something like that.

• 20:24Daniel Whitenack?

  So I'm, I'm wondering if you could kind of break down like as, as you're kind of, as you're stepping into addressing some of this. You already mentioned like frequently you're updating like new features, like there's new behaviors that you're seeing that didn't exist before. So like, let's say that active fence like starts to understand that there's some type of new, new behavior that's, that's harmful or something like that. What is your process and how do you think about going from like, knowing this is happening to detecting that this is, is happening in, in a repeatable sort of way?

• 21:04Speaker 00

  Uh huh. So there's sort of a couple of different ways that, that we're basically staying up to date. So the first is really, really close contact with subject matter experts that are out there gathering. Information, intelligence, researching, collecting data, building like keyword databases, looking for a particular, you know, bad actors that frequently post things, sort of, they're really there. And so we're frequently talking to them and understanding like, what was it that made this violated? What, you know, or they'll send us a report and be like, hey, this is a new hate group. Or hey, this is a new meme and so forth. That's, that's one thing. The other thing is that even within our models, we're constantly getting feedback.

• 21:46Speaker 00

  We have something that we call the database of evil, uh, which is like a really like.

• 21:52Daniel Whitenack?

  I mean, might as well call it what it is.

• 21:54Speaker 00

  Right?

• 21:55Chris Benson?

  That has to be though, the best name I've ever heard, you know, for, it's the database of evil.

• 22:03Speaker 00

  That, that, and it's true to its name.

• 22:06Chris Benson?

  I believe it. Yeah.

• 22:08Speaker 00

  And so we keep that updated, right? So we have data that's coming in, we score it, you know, we give it a risk score, which is essentially like the probability that it's violated for some violation. Uh, and then we have trained analysts that review it, review the score and sort of can say, yes, no, no. Anything that's verified as being violative, um, it goes into the database of evil. The database, the database of evil is sort of used for a few things. One of which is new content that comes in, we can say like, well, have we seen this before? Do we know it, right? Things that we've seen a lot, they're more like, you know, uh, versus things that are, that are brand new. And it's also used because as we take this feedback, we're constantly, like I said, we're retraining, right? We're, we're learning and, and that's like, those are the small adjustments, adjustments that we can make to our models.

• 22:52Speaker 00

  And so it's this idea of like constantly getting feedback, both just, you know, from researchers that, that go out and find things, from the data that's coming in is being scored and then, you know, we're sort of re, retraining on top of that. And then of course we have our, our database of evil.

• 23:09Chris Benson?

  So let me ask a question. Obviously, as you pointed out, your database of evil has some, has a lot of really explicitly evil stuff. But I'm also imagining that there, that there are gray areas that like you kind of mentioned the, the baby's first bath kind of thing. And, and you know, that would depend kind of on audience, on, on whether that was, if a family member showed me, uh, we have a new baby in our family. My niece has a new baby. And if she showed me a photo of the new baby having his first bath, that would not be offensive to me. But there are contexts where posting it online, it could become offensive and such. So with these types of, of gray areas, and the fact that you can have one set of content that has

• 23:54Chris Benson?

  a bunch of different, I don't know what you would go, you know, acceptable rankings, if you will, you know, depending on, on who's viewing it and what the context and all that. How do you approach making sense of all the gray area? Like, you know, the, when you're, when there is everything from perfectly fine to absolutely not fine, and it's all valid for the same thing?

• 24:15Speaker 00

  Right. That's a really pertinent question. And it's something that we're dealing with a lot. Part of it is sort of, and we haven't, I don't, I don't think that we've completely cracked it. But one thing that we do do is that we can also have database of evil where evil is relative for, for the client, right? And so, which also leads to sort of, you know, this idea of customized models per client based on the feedback that is coming from them. Uh, where basically you can say, okay, so, and we have this now, right? We have two clients. One is baby's first bath is violated and one it isn't. And so we, we are already are juggling this with different levels of like, of, you know, of human intervention to, to, to get it to sort of really to, to perform, because that's what's training it to, to get it to that point.

• 24:58Speaker 00

  And there's even other examples where, you know, if we take it, you know, babies, babies first bath and things like that, people are like, you know, oh, but, but it's so clear what, you know, child abuse and pedophilia are and, and clearly it's not. And even things like, um, asking someone, are your parents home? So if it's a conversation between two children on a chat room, that's totally fine. But it can take a much darker turn when you suddenly see that it's, you know, a user that is also in, you know, adult chat rooms or it's being posted, like, you know, at around eight thirty, nine, ten P M. when kids are supposed to be starting to go to bed or to be in bed. I don't know. My kids like go to bed early. And so, and you know, and, and so you can start, so, so even then, like, you can say, well, I, I have

• 25:41Speaker 00

  language understanding and, you know, this is a kid's chat room. But suddenly there's like all these other le- levels that you need to take into account to understand if a phrase really is just nothing.

• 25:51Chris Benson?

  Just as kind of setting some, I, I am living what you just described. I've grown kids, but I also have a daughter who is just getting to the point where we're letting her get online and do some of the stuff and some of it's in supposedly safe environments. But then, uh, as the nosey dad who's just worrying about keeping his child safe, there are all sorts of gray areas and stuff. So it's, it's, it's fast. And there, and there are all sorts of moments where I'm having a lot of trouble telling whether it is a safe context or not. It's not very clear. And so, um, I, I can imagine that that is extremely challenging, you know, to solve as a technical problem that can be, you know, recreated across a, a lot of different, uh, audiences.

• 26:35Speaker 00

  Totally. Um, and I, and I think that there's always going to be to some, like, at least when we're, when we're like training or whatever, right? There's always, there has to be a human in the loop and, and for these gray areas. So we do as much as we can with technology, uh, and we bring it there. But even if as a parent, You're looking at and you're saying, you know, I don't know. And so sometimes we can leverage things that you don't have access to, right? Like, we can look at the history of the user or the other chatrooms or other things that are going on in the space or how this, you know, who has been in this chatroom before. But, you know, sometimes it comes down to, you just don't know.

• 27:11Daniel Whitenack?

  So I have a lot of, Chris always knows I like to ask a lot of practical questions before I get to those in terms of like, some of the things you're doing and how you're doing them. I'm wondering, for a company that's using some of active fences, technology, what does that connection look like? Like, one of the examples I'm thinking in my mind is, I'm working on a website for some of our partners where people can contribute and list cards for tools that they're working on, like software tools. And, you know, technically, like, they could submit anything in that description of that tool, you know?

• 27:52Daniel Whitenack?

  Now, I think, like, we have, hopefully, vetted people that will be submitting content. And actually, not everyone has accounts, and so it's fairly restricted. But, yeah, I'm wondering, like, in that situation, or a much more scaled up situation, what is working with the right view to have, like, your software platform? And then you send off content to some API and get a threat score or something? And then you figure out what to do with that threat score in terms of, how does this actually practically work out for a company in terms of? Because I imagine it's complicated. Every company has, like, their different platform, right? And also, like, oh, you know, the format of a Facebook message, you know,

• 28:37Daniel Whitenack?

  going to a webhook is going to be different than, like, a blog post being posted to a content management platform. So in terms of, like, data moving around, how does that work out practically?

• 28:49Speaker 00

  Yeah, so I think there's kind of two parts to, maybe more parts, but two main parts to the question. The first is, like, how would you, as a user, interact with us? And so we have a UI, a platform, where you can really, you know, see the content that's coming in. And you can define sort of codeless workflows, where if, you know, something is above a certain risk or threshold, then, you know, it's automatically filtered out. If it's below... A particular risk for threshold and you don't even look at it. And then like, what is your threshold for for human moderation? That sort of gets between, that sort of gets around this sort of like um precision recall uh conundrum where you're like, well, I set a threshold and I always have to choose, you know, what am I maximizing? You can say, well, let's set one threshold where you're maximizing your your precision and, you know, another one where

• 29:34Speaker 00

  you're comfortable with your recall and then you look in this sort of band. Uh and then you can use that to moderate. We also have an API, um you can send, you know, you can do like synchronous calls for for text, so like near real time, really, really fast, so for chat, if you want to try, you know, pre-published or and so forth. And we also have uh async for for text and for for images, for video, sort of, and, you know, more like where you can send the full context, right? So you can send your content and you have your, you like the body of the media and the title and description, whatever you have. That's for the first part of your question. And for the second part, I think it's is maybe like um a little bit more interesting, kind of, because everyone, you know, you you can build an API, whatever, but what we've spent a lot of time in it it and a time on is

• 30:18Speaker 00

  both like optimizing our API, so making sure that it's, you know, very robust and responsive and so forth, and then also modeling our data. And so we have a very rich understanding of the world of platforms, like how we can model the world of online media or online platforms or user-generated content. Pick pick your favorite term. We model it. We have like a very robust and flexible schema where we're able to sort of monitor like um a user and it's related to po- and like the post that they that they put and how many likes they have and you know, it's not always relevant and you don't always need to use it all. But we have this sort of, you know, we have users and we have we have contents and we have collections and each of those are modeled a bit differently. And so, once the data comes in, we ingest the data and it's modeled like this,

• 31:00Speaker 00

  then we can go ahead and take it apart and score the different parts of it. And then through our API, which is able to handle like really high throughput and fast SLA, basically start. Giving you responses. And we've done a lot of work on our backend optimizing. And so we're batching models on GPUs and doing all sorts and, you know, thinking that, you know, we have all kinds of like code that we've written that basically optimizes like what machine type you want to run on, basically to make sure that everything is like runs as smoothly and as robustly and reliably as possible to get those responses out.

• 32:08Daniel Whitenack?

  So, uh, Matar, one of my questions is just in the back of my mind is like, just the practicalities of running the, the type of platform that, that you're building and the service that you're running. Like, I could imagine, oh, I have, you know, this model that is able, like Model 1 is able to detect like this harmful type of meme. And Model 2 is able to detect like this harmful type of video. And then like, all of a sudden, you're sort of proliferating like hundreds and thousands of models for like little pieces of what you're trying to detect.

• 32:49Daniel Whitenack?

  And then another sort of scenario is like, oh, I'm gonna like try to standardize everything into more generalized models that, you know, handle multi, multiple modes of data or like try to synthesize things together. How just practically as like a development and research team, have you started thinking about like, when is this something maybe we wanna combine together in maybe a larger model that's trying to, you know, address like multi-task sort of thing or multiple types of data? And then the other side of that is like, maybe, maybe sometimes it is useful to just like spin up hundreds of small models and, you know, ensemble them together in, in some way. Any thoughts on that?

• 33:34Speaker 00

  Yeah, so we actually do do that. Sometimes we have models that are like just really lean and we serve them as is. And that's sort of for, like I said, for our, our like near real time responses. For when we do contextual stuff, so like I said, we really need to extract information as many different ways as we can. So we're looking for logos and we're, and we're listening, you know, we're listening to the audio and, and, you know, looking for, for known phrases and keywords and language understanding and what have you. And all these like smaller models, then we do combine into ensembles. We have, um, we have a feature store that, that we can basically, you know, take from, um, combine, train, train the relevant models, um, and then, and productionize them and then

• 34:16Speaker 00

  add, add, like, we call them like indicators, but, you know, essentially indicators from which then we can get features and then, and then go to, to a model, um, which is, which is an ensemble of these. Um, and so we kind of use both approaches based on like the SLA requirements, based on also the explainability that we need. We want to be able to, to explain why something, right? Like, you know, this particular logo was found because sometimes the moderator may not have the, the full knowledge that we have. And so a big thing that we deal with is how can we take our intelligence and leverage it to the fullest extent, right? So one way is really to put it in the models and the other way is really to educate the moderators through explainability of the models. They can really understand, you know, why things, you know, sometimes things aren't

• 35:01Speaker 00

  obvious.

• 35:02Daniel Whitenack?

  Yeah. And I guess that you started getting to my other question, which is like, how and, and when do you bring in the subject matter experts into the loops? Cause I imagine there's like, you know, certain cases where you're highly probable that this is some type of harmful situation and maybe given a restricted set of subject matter experts in an area, maybe they're restricted to only reviewing, you know, X amount of content per day or something. So is that a situation that you run into where, excuse me, where you have to prioritize, you know, what you're reviewing with subject matter experts based on some predictive measure

• 35:47Daniel Whitenack?

  that, that you have and kind of do that and then? Some sort of ranked way or or do you handle that in in some other way?

• 35:56Speaker 00

  Do you mean in terms of like what the analysts are reviewing for us?

• 36:00Daniel Whitenack?

  Yeah.

• 36:01Speaker 00

  For labeling and or like for reviewing the?

• 36:04Daniel Whitenack?

  Right, so I'm assuming that there's a limited number of those people. There's not infinite of those people.

• 36:09Speaker 00

  Right.

• 36:09Daniel Whitenack?

  So.

• 36:10Speaker 00

  There's not an infinite number of those people and also we want to be very aware of their well being. Um, we care a lot about the well being of the people that we work with. Like Act Defense invests a lot in that. And so specifically for for these, so I want to make sure that I prioritize what it is that they need to review. I don't always need, you know, I don't need them to review everything, right? I usually like what I would go for is that I want to review the gray zone, right? So we do, we have implemented active learning, which, you know, is basically to prioritize what it is that we want to to train on. Um, and so that also prioritizes what is that we want to, we want to review and to label. Because I'm always going for the gray zone, right? Like what, like the things that we're not quite sure of, that we don't really know. That's where it goes to the expert, right? It goes back to Chris's question, like, how do you know?

• 36:55Speaker 00

  Sometimes, sometimes you do know, but it's tough. And those are the things that I want to label, because those things that are tough are what are what are what is going to feed in. And to give like, you know, my distributor to the, you know, the maximum power that it needs.

• 37:09Daniel Whitenack?

  And how much, um, sorry to steal all the questions, Chris. I'm just so fascinated by by all this, but no worries. One thing that is always on my mind and maybe I wrestle with sometimes is how much do your sort of data science people or the people that are working sort of with the models directly interact with the subject matter experts and kind of share knowledge across across that boundary? How do you balance that? Because I, that's always something I I think I struggle with in projects is ultimately, it would be great to kind of bring the subject matter experts in all along the way, like in every step of everything.

• 37:51Daniel Whitenack?

  Because you learn so much, but the fact of the matter is like you've got a limited number of those people. But also you have to ship things, right? So you can't necessarily have the luxury of always having a discussion before you make a development decision. So how do you balance that, especially because this is such a complicated environment in terms of the subject matter? How have you found ways to balance that and any thoughts that you've or takeaways that you have from that experience?

• 38:23Speaker 00

  Yeah, so one thing that we did is we actually embedded subject matter experts, like researchers, into our dev teams to sort of be part of the process. We also have our analysts that are labelers, they're, you know, they work really closely, they're just, it's part of the same group. And so they're not out there. However, again, it's a limited number, it's, you know, there's, there's, and it's a limited number of violations. There's, you know, we're constantly being exposed to new stuff that we have to handle. And there it's just a matter of, you know, relationship building and of doing, you know, check bases and constant feedback. So, hey, this is, you know, like they're kicking off new projects, we come, we learn, like, what are you guys doing? Because a lot of times they're learning on the fly too, right? They have this new, new trend, new thing that they're learning about.

• 39:08Speaker 00

  And so as they're learning, we're trying to gather as much as we can from them. And then just a constant like feedback, like, you know, how does this look? How does this look? Is this there? Is this not? But I think like the key was embedding them with us. We did have situations where basically we wanted to develop models for things that we didn't want to expose our data scientists to. And that only a very, very few number of people in the company can be exposed to because of the nature of the violation. And there, that was much trickier because there, there was like complete dependence that the data scientist, like, how do you, how do you build and train a model without looking at the data?

• 39:47Daniel Whitenack?

  So I'm kind of curious because it changed my, the question I was about to ask you just a little bit with what you just said. So I'm going to kind of combine two things.

• 39:54Chris Benson?

  The sense that I got, because you keep talking about going to the gray area and stuff, is that almost the core of your research effort is to kind of replace the human intuition that's necessary, you know, early on to identify the nuance that's there with more and better models as you're moving forward into that. And so I would, almost like a bell curve of difficulty where that gray area is the hardest. But I am curious, and I'm wondering if that's the case, but I'm also curious, when you mention those things, like, it seems like, I'm guessing, that the things that you really don't want to expose someone to are almost, they're not in the gray area, they're way over into the deeply evil side,

• 40:37Chris Benson?

  you know, where it's like, you just, you're never going to forget having been exposed to that if you are. How do those balance out? You know, you have that gray area that you're focusing on, that you've mentioned several times, and then you have that, those kind of things. Are they, when you have something that's so explicitly evil, and it will imprint a human's mind in a very negative way, are those different problems that you're solving from a, as a data scientist, a little bit, in that once, whereas the gray area, there's so much nuance there, do you see what I'm getting at? Like, how do you balance the approach to building models to handle one thing that's really obviously bad, and you just, you know, you don't want to get anyone to it, versus developing intuition, or an alternative to intuition in the gray area?

• 41:20Speaker 00

  Yeah, so I think, and let me know if this doesn't quite answer your question, but a lot of times, it's, it can be the same model, right? So you have a model, and it knows how to identify, because, you know, it gets the distribution of, you know, the data's distributed in some way along the space, right? And so, the things that are very obvious are going to be on like, one side of the discriminate, of the discriminator boundary, right? Like, the really, like, pick your favorite violation, right? And I'll give you examples of like, things that are like, very, very, very clearly violative, and they're on that line. And then, when you're training your model, you don't only want to give it just like, the really horrible examples, and then things that are just like, you know, puppies and snowflakes, right? That are very obviously not, because those are so, the distribution between them, they're like, they're so far away that your discriminator is, like, your decision boundary is just never gonna converge, it's gonna, it can flip-flop back and forth, and, you know, you'll never know. And so, as we're doing

• 42:05Speaker 00

  it, we're also trying to find things that are like, you know, on the borderline. Because that's what's going to help us really make sure that we're able to like find the good decision boundary. Cause at the end of the day, it's really important. Like the base, like the basics is that we have to be able to catch like the ISIS and we have to be able to catch beheadings, you know, all these terrible, terrible things. But we also want to be able to catch things that are less obvious within that, still within that space. And so that's kind of where I'm talking about the gray area. Just like, you know, for the grooming model, you can like, I don't even want to say, but you can think of phrases that are like very obviously grooming, right? Where like you're sexually harassing a minor and it's there in like plain text, right? But that same model, if you want it to be any good, it's, it's sure it's helpful that it can find the obvious stuff, but you also

• 42:49Speaker 00

  want to train it on things that are kind of more on the like closer to the boundary. Uh, because that's what'll, that's what'll help you in the long run.

• 42:57Chris Benson?

  I have a, a quick follow-up that as you were talking, it was kind of coming into my mind. It's a very human question. I, I'm for a moment, I want to move you out of the data science bit a little bit and just, your organization is in a little bit of a, of a unique position on that. You mentioned that there are certain things you want to keep as many of your, of your folks on your team not exposed to, but that does leave some people exposed to some pretty awful stuff. And that does impact, you know, people we, you know, we know. I'm guessing that you're one of the people that has had to see some of those pretty tough things to see. How do you cope with that a little bit and keep, you seem to have, be really super grounded in that. And I know like, you know, but as

• 43:41Chris Benson?

  part of the job, you're going to have to cope with some really tough stuff. And I know there has been things that I have seen myself online that I wish I just had not seen. I remember early on in the Al Qaeda period some time back and I watched something that was in the news that happened to be out there. And I was like, I wish I had never seen that and I will never forget that as long as I live. I'm just curious in a human sense, how do you cope with terrible things and, and keep it in a healthy, you know, kind of for yourself in a healthy, healthy place, if that makes sense?

• 44:12Speaker 00

  Yeah, that does make sense. So I think I'm, I'm like a very, um, personally, like I'm a very like mission driven person. Like it's like, it's very clear to me. Why it is that we do what we do. And I really, really believe in it. Like I said, like a bunch of us are parents and or you know, or have nieces and nephews or whatever, or just care about kids. Or you know, care about, care about communities and environments. And I just very, very deeply believe in, in what we as at Act Defense do. And you can really feel that when you're in the office and when you're working with people, everyone is very, very, is very mission driven. That being said, we also do our absolute best to support and protect everyone that works with us.

• 44:54Speaker 00

  So whether it's like different, you know, wellness support programs, we have a psychologist on staff who specializes in resilience. And, you know, she's available to everyone and she does like group and one-on-one and really helps, you know, helps people build this sort of resilience in the face of what it is that we do. And for me, what personally works for me is like just understanding why what we do is so important. And yeah, I've definitely, definitely seen things that will never leave me, never. And I accept that, you know, because I'm doing my part to make everything just.

• 45:32Chris Benson?

  You're helping the world in that way. I get that.

• 45:34Speaker 00

  Hopefully. I try. We all try.

• 45:37Daniel Whitenack?

  I think even though we've talked about hard things this episode, I'm super encouraged and want to thank you and the team at Act Defense for what you're doing. Yeah, it's something that's desperately needed. And thank you so much for digging into these problems and doing it with such technical excellence as well and deep insight. As we close out here and we look to the future, what on the positive side sort of excites you about where this technology is, is headed as, as you look to the, as you look to the future?

• 46:08Speaker 00

  Yeah. So to me, first of all, it's super exciting that this is no, like I said, like we started off, it's no longer a nice to have. It's a basic expectation. So I think first of all, to me, that's really exciting because people are not taking safety for granted. Like, they understand how critical it is, and it's coming from the users. It's, you know, it's not just sort of like, oh, well, it is what it is. Like, this is the price I pay for being online. No, that shouldn't be the price that you pay. So to me, that's exciting. And on the tech side, I think what's cool is that we're seeing a lot of open sourcing of different, of different models, whether it's data generation, or, you know, audio transcription, or, or Z-Shots, or like, all these things that are just, it's like a candy store, right? Like, you can start thinking about all these things

• 46:53Speaker 00

  that are, that are used for, you know, technologies used for completely different things. And you can say, well, like, how can I take these ideas and use them to just extract more signal and to look, look at these things from different angles? And, you know, it's an adversarial space, and so it keeps it interesting, at least.

• 47:10Chris Benson?

  Well, Matar, that, i- it's very inspirational, the work you're doing. I know that there's, uh, it's tough work, but, uh, thank you very much, and to your teammates for doing the kind of work that you're doing. Uh, it was great having you on the show, and looking forward to, uh, to having you back sometime as you guys surge forward and have some more stuff that you wanna, that you wanna share with us. Uh, so thank you very much for your time today.

• 47:32Speaker 00

  Thank you so much for having me. Thank you for caring and for asking really, really interesting questions. I appreciate it.

• 47:46Speaker 01

  All right, that is our show for this week. If you dig it, don't forget to subscribe. Head to practicalai.fm for all the ways. And if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets, to Fly.io for backing our dynamic requests, to Breakmaster Cylinder for the beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.