Building safety tools no one can take away with ROOST’s Camille François

Protecting people online shouldn’t depend on whether a company keeps its safety tools effective, affordable, and available. ROOST founding president Camille François joins Reid Hoffman and Aria Finger to explain how a nonprofit is turning trust and safety into shared infrastructure. She digs into why social media was a blip in how we build tech, what Discord did after Twitter bought Smyte and shut it down, and how two people running a fandom app caught 19 cases of child exploitation in two weeks. A daycare switching its Alexa wake word to "computer" becomes a lesson in how the next generation should relate to machines. They also examine why policy without tools is ineffective, how a shared safety commons helps startups compete, and what it will take to make safety as invisible as encryption.

• 0:00Speaker 00

  Today, one in 25 American teenagers says that they have had a deep fake nude produced of them. I'm not saying one in 25 tell you, I know somebody to whom this has happened. One in 25 tell you, yes, this has happened to me. If you want fast collective innovation, if you want people to have access to the tools they deserve, if you want people to trust the things that they run their lives on, give them good, shared, open source foundations. Policy without tools is basically poetry. It looks great, you can recite it, but you can't really protect people with it.

• 0:34Speaker 01

  In June 2018, Twitter bought a trust and safety startup called Smite and abruptly cut off its other customers. Platforms like GoFundMe and TaskRabbit lost their anti-abuse tools overnight. It exposed a bigger problem. Safety infrastructure was concentrated inside a handful of companies, leaving smaller builders dependent on tools that they didn't control.

• 0:57Speaker 02

  Dr. Camille Francois has spent her career investigating online harm and building ways to combat it. In February 2025, she launched Roost to make safety tools free, open source, and available to anyone building an online space.

• 1:12Speaker 01

  As a kid in France, Camille wanted to be a space baker, baking croissants on Mars. She's chalked that up to being French. Today, her ambition is to make building safely as accessible as building itself.

• 1:25Speaker 02

  Her colleagues describe an anti-authoritarian streak, drawn to frontiers over fiefdoms. That may help explain Roost's approach, sharing technologies that others have kept proprietary and giving communities the tools to protect themselves.

• 1:43Speaker 01

  Camille Francois is the founding president of Roost and a professor at Columbia University's School of International and Public Affairs.

• 1:51Speaker 02

  Welcome, Camille. Your colleagues describe you as having an anti-authoritarian streak, more pirate than institution builder. Does that sound like you? And what were you like growing up?

• 2:07Speaker 00

  I'll talk to my team about that.

• 2:11Speaker 02

  In an authoritarian way, I hope.

• 2:13Speaker 00

  No, yes, of course, I recognize that description.

• 2:20Speaker 00

  I do think that I am able to work with and within institutions.

• 2:28Speaker 00

  But yes, I like the energy of doing something together in a collective, maybe with a little bit of a pirate spirit. And when we were preparing for our very first Roost team retreat, I actually found a book, which is great, called Be More Pirate, that looks back at the history of the glorious days of piracy and think through, okay, there are institutional lessons to take from those days. And there's a way to rethink about that moment in time as a moment in time that brought a lot of advances in equality and democracy. And so maybe, maybe there's a different lens we can apply to, to our understanding of piracy.

• 3:13Speaker 00

  But jokes aside, I think that I've always been part of the open source and free software movement.

• 3:21Speaker 00

  When I was a student, I founded Students for Free Software France and then sort of served on, on the board of the global organizations. Actually, how I met my husband. And that organization no longer exists, but back in the day, we were referred to it as Baby Pirate Hacker Club. So yes, I can see, I can see where that comes from.

• 3:39Speaker 02

  Arr.

• 3:40Speaker 01

  Yeah. Exactly. I mean, anti-authoritarianism, I'm, I'm with you. That's great. And it seems like everyone is talking about online safety, privacy. You were way ahead of the curve on this. You were building Roost, you mentioned, which is a nonprofit that makes online safety tools free and open source. For someone who is hearing about this for the first time, who are these tools for and what do they help people do?

• 4:05Speaker 00

  Yeah, they're for everybody. And Roost is a nonprofit, it's also a movement. And so we do this with a lot of people. We do this with large AI labs, but we do this with small builders. We do this with philanthropic organizations. We do this with many different types of institution. And essentially, if you think about safety, we have at least three problems. The first one is, I think it's fair to say that tech and AI has progressed faster than our ability to keep people who interface with these technologies secure, not only when it comes to kids and children. And so we have an innovation gap, right? We need to fast forward our innovation and progress on safety tech. The second problem that we have is sort of a distribution gap, right?

• 4:51Speaker 00

  There are a lot of safety tools that are okay, but people don't have access to them. And there are many more builders who are building new applications, new websites, and who want access to the tools to do things right, to protect their users, to protect their communities, to do safety by design, right? That's, that's a, a real problem that we have. It's nonetheless a good news, right? That means there's more appetite for safety. And then, of course, the third problem is, there's a little bit of a crisis of trust when it comes to safety technology. It's because it has long been very opaque, right? So people tell you like, hey, we're using technology to keep you safe. And you're like, okay, what is it? Can I see it? And it's like, no. And so, of course, we can change that model. Here, open source really is a method, right?

• 5:36Speaker 00

  Where we can accelerate innovation, make sure everybody who wants those technologies have access to it. And then, of course, where we can say, hey, here it is, here's a safeguard. Go audit it, modify it. If you don't think it works for your use case, go make it work in your language, your context. And then if you make a better version of it, just go distribute it or sell it, right? Like for that matter.

• 5:57Speaker 02

  You've spent a number of different years, both as a pirate entrepreneur and, you know, professor, and then also, you know, years investigating online harm, work on informing the Senate on, on disinformation.

• 6:15Speaker 02

  Was there a moment that convinced you to build Roost? Was there a, was there a kind of a, ah, this would be a way of changing this ecosystem? What was that moment? And how did your experiences lead you there?

• 6:28Speaker 00

  I don't think there was a moment. I think that has been slowly building up. And frankly, not just for me, but for the field. And if I think about my own trajectory in safety, of course, I have taken from all those different experiences, ingredients that you find in Roost, right? So when I was studying information operation, one of the things that you very quickly find is information operations are cross-platform, right? And so you find them spreading on smaller bulletin board, weird little forums, apps that nobody's ever heard of. And when you sort of unravel this, you have a lot of people coming in saying, oh, we did not want to use the school board in this way, help us, right?

• 7:09Speaker 00

  And so you sort of understand that any sufficiently smart and motivated adversary is going to take advantage of the fact that there are many surfaces and smaller surfaces, and not everything is protected. Of course, that matters a lot for the theory of change at Roost. When I was at Niantic, that was a really formative experience because I had a great CEO, John, who was very supportive of our safety efforts. And of course, Niantic had Pokemon Go, a lot of kids play Pokemon Go. And when we said, okay, we're going to build a really good, robust infrastructure for safety, I went and I sort of knocked on doors and looked around and like, what's the state of the art? And I did not like what I found. And the state of the art was like, yeah, what, what we do is we kind of, everybody builds their own half-baked stuff.

• 7:53Speaker 00

  And then you, you buy a few good vendors and then you duct tape everything together. And that's what everybody does everywhere. So I was like, okay, well, that sounds nuts. There has to be a better way. I was coming from more of a cybersecurity background where, of course, there's a different approach, right? Like there's a shared infrastructure of open source tools across the field so that you can focus your resources on building the piece that makes sense for you, but you don't have to sort of fully reinvent everything from scratch all the time.

• 8:22Speaker 00

  And so I started thinking about, okay, well, let's try to figure out if those tools that everybody's sort of cobbling together, are they the same for everybody? Like, is it, are we doing this because people have such unique needs? And so I started interviewing a lot of safety engineers across the field. And they were like, absolutely, they're the same. Most of the infrastructure is steady. So we did this project to sort of document what are the core components of safety infrastructure. We published a piece called a Dyer Framework that goes for detection, investigation, review, and enforcement. But really what engineers were telling us is, yeah, we get staffed on something. We built 30% of a workable thing. And then we get assigned to another project. And so everybody's sort of constantly rebuilding basic foundations, which is really frustrating.

• 9:08Speaker 00

  And so we said, okay, great. There's an opportunity to change the way we do safety in tech, to sort of rewire the way we do safety in tech.

• 9:16Speaker 01

  I feel like everyone's talking about, there's a lot of other issues and discussions around AI, but a lot of people are seeing AI as sort of social media 2.0. We got social media, we were so excited. There were these harms, some of them to children, some about misinformation that we feel like we didn't sort of attack soon enough. And people are worried that it's going to happen with AI. Like, do you think that if Roost had been around 10 years earlier, the landscape would be different? And like, where do we go from here?

• 9:42Speaker 00

  I actually think about social media.

• 9:46Speaker 00

  As an exception to the rule of how we build tech, as a blip in the timeline, that's just a bizarre happenstance, right? Because if you take the long arc of tech, the way we built the internet, the way we built the foundation of tech around us is on open protocol and on open source software, right? TCP IP was an open protocol. Tim Berners-Lee donated the web to humanity, right? And so the reason why we can innovate and then keep tech relevant and keep tech responsive to our needs as a species is because we do this in the open. And then there's a sort of small era of social media where we kind of forget about all these lessons. And it's bizarre, but it's a fact that we've inherited all of the safety tech that we have

• 10:30Speaker 00

  from that blip in the timeline, from that era. That's why it's brittle. That's why it's hyper centralized. That's why it's opaque. And so what we're doing with Roost, frankly, is sort of bringing back all those normal principles of how we build good tech into the world of safety. We're not really inventing something new rather than sort of

• 10:50Speaker 01

  Like getting back to the roots of the internet.

• 10:52Speaker 00

  Exactly, right? Changing the approach to something we know works, which is okay. If you want fast collective innovation, if you want people to have access to the tools they deserve, if you want people to trust the things that they run their lives on, give them good, shared, open source foundations.

• 11:08Speaker 01

  And do you think the leaders of the frontier labs are learning from any of those mistakes? Or like, what mistakes are they still repeating?

• 11:15Speaker 00

  I think some of those leaders are building these new open source foundations with us. So last year, for instance, we did a big effort with OpenAI. Our team worked closely with their scientific and engineering team to take one of the core piece of their safety stack, a model that was doing a lot of moderation behind ChatGPT and then Sora, and to actually share it with everybody, to put it out in open weights. Now, that's significant for a number of reasons. But the first one is, prior to that, it's not like you didn't see big companies share safety innovation. But a lot of that sharing looked like, oh, we're going to make this available to everybody.

• 11:59Speaker 00

  Usually that lasts for the duration of a really good marketing campaign. And then, of course, like, you know, priorities change and then people move on. And so things are not maintained. People are struggling to get access. Things sort of fall behind. And then, of course, at any moment, they can pull the rug, right? They can say, actually, we changed our mind. But open source isn't that. Open source is a one-way door, right? So when OpenAI says, we are actually going to share this model under an Apache two license and we're going to put it on Hugging Face, that means if tomorrow they changed their mind, they can't get it back. And so people know they can build on it, right? So we did that. And they made that statement at the time saying that they believe everybody should be free to study,

• 12:42Speaker 00

  modify and use critical safety tech. It's just kind of fun to see OpenAI quoting Richard Stallman. But I love that. And after that, we also did a really nice hackathon, working with both OpenAI and Hugging Face. So we invited people saying, okay, look, this model has been shared. Come test it. And so we saw people who were really exciting to immediately use the model. For instance, today, this model is used by Wikimedia, right? We will also, you know, keep in mind that large, important tech nonprofits also deserve safety tools. But we also had a bunch of people come and break the model, which was really fun to see. I remember there's a team who I think was building recipes for meth and ecstasy in Latin. So, yeah, that's sort of like what the,

• 13:26Speaker 00

  you know, we see some of those large labs being like, okay, yes, we think we want, you know, we want to tackle safety a bit differently now.

• 13:34Speaker 02

  I want to go into a specific area as an example in a second, but let's first come back a little bit.

• 13:41Speaker 00

  I'm not giving you the recipe of meth in Latin. Don't insist.

• 13:44Speaker 02

  I want it in French. The, one of the things, like, when you think about, like, I think there is a lot of fundamental important about open source. As you know, I was on the board of Mozilla for like a decade. And, you know, LinkedIn actually open sourced a whole number of projects. And I think it's a generally good thing to do. I think it's also good on certain patterns of safety and security. And I think the foundation of the internet on it. I don't think it's actually only social media that creates the more turbulence out. I mean, we have the mobile internet with Apple now. We have a bunch of, like, there's like, kind of what's going on behind YouTube, you know, within Google and so forth. So I want you to kind of elaborate a little bit of, what is this we've looked like?

• 14:29Speaker 02

  And how do we bring? Because it's not a disagreement that open source isn't an important thing here to elaborate this point of view. But it's like, given a complex environment, what is the role that you see open source doing? And then I'll get to a specific commercial example question.

• 14:46Speaker 00

  I think here, open source has a unique ability to help us, as I said, reshape how we think about safety, really accelerate innovation, make sure that a lot more builders have access to the tools they want and need, and enable a system that can be scrutinized and be adapted to a lot of little communities. When you think back to the social media era, we talked about the sort of perils of hyper-concentration. There's another peril, which is this idea that I think we held a little bit too long, that we were going to have big global public squares with rules that work for everybody.

• 15:28Speaker 00

  Well, I don't think this panned out exactly the way we wanted. I think the reality is you're going to have a lot of smaller communities that have rules that make sense for them. And that's important. That's pluralism, right? That's the world that we want to live in. If I think back about the difference between today, for instance, and say like 10 years ago, as you said, I'm a professor 10 years ago. If I looked at my students' online lives, they were quite concentrated on a few platforms, right? So for instance, they may have like, they use, they used back in the days, what's called Facebook, you know, Facebook for schoolwork and also communicating with family and all that. But today, I think what we see is people use a lot of different, smaller platforms for a number of things, right? They may actually have a WhatsApp group or something,

• 16:13Speaker 00

  a Signal group or something else. They might use a Discord server to coordinate about something. And then, you know, whatever, right? So this multiplicity, I think, now of online spaces, both in social and in AI, also means that we need to be able to give all of these communities the tools that they deserve to shape what safety means for them, right? It's also bringing back pluralism, which is something that maybe we had lost sight of a bit during the social media era.

• 16:39Speaker 02

  Because one of the things I know you've tracked is, you know, Twitter bought Smite in 2018. Immediately cut off all the other customers, demonstrated how brittle this is when you have a commercial system. So Roost is a offset to that in part. What are the things in terms of, is Roost the only one that's doing this? Are there more? You know, is Roost trying to encourage things like Smite or other things to be open?

• 17:06Speaker 00

  Yes.

• 17:06Speaker 02

  Like, what's, what, like, how does the dynamic play out? That's part of like the specifics within a commercial environment.

• 17:13Speaker 00

  Okay, so let's talk specifics of it. I love this story of Smite. So Smite was a little startup that provided safety technologies to a bunch of companies. One of them was Discord. And Smite, which in fact is a rules engine, got bought by Twitter because Twitter was like, yeah, that's very helpful for what we do. I get that. And then Twitter shut it down. And so a number of companies, including Discord, was like, oh, we can't protect our users and community. We just got the rug pulled super, super brutally from under our feet. And so they did what they had to do, which is they did the very boring work of rebuilding everything from scratch. What they have rebuilt from scratch, they have donated to Roost.

• 17:56Speaker 00

  It is the foundation for one of our core software projects called Osprey. Because Discord was like, nobody needs to go through that again. I am going to open source that rules engine. And the second they did that, right? We worked with their engineering team because actually it takes a minute to open source an internal system. Blue Sky came in and said, oh, that's great. I want to use that. So they started using this. And as a result, they started improving this. And so Discord said, okay, well, we want to use the branch that everybody is improving. We want to benefit from the collective innovation of everybody who uses Osprey. And so Osprey is this amazing story of like, this is what the industry learned from Smite. And this is what Discord decided to donate.

• 18:40Speaker 00

  And now this project, Osprey, is actually used by many different platforms and organization at scale to investigate fraud and scams and coordinated harassment. And it's great to see it take a life of its own, yeah.

• 18:56Speaker 02

  Yeah, it's awesome.

• 18:57Speaker 00

  Yeah, it's also fun for Discord because, well, as I said, like, they benefit from the collective innovation. And then they were able to spot really good engineering talent too. And so it becomes fun because all that really good engineering safety work also becomes visible, right? So morale boosting for the people who work on this. And then also it's helpful for careers.

• 19:15Speaker 01

  Absolutely. I mean, every engineer wants to work on something that millions and millions of people are using. That's the benefit of being able to be part of that community.

• 19:22Speaker 00

  Yeah.

• 19:22Speaker 01

  And I think it's also, you know, Discord. It's really important to have all the different actors in the ecosystem. You need the researchers, you need the professors, you need the people who are in the companies. And Discord was uniquely suited to be able to do something because they had had this rug pull. And we talked about you being a pirate, so you've mostly worked sort of from the outside.

• 19:41Speaker 02

  That's the theme, correct?

• 19:42Speaker 00

  I was trying to keep it to you and around.

• 19:45Speaker 01

  We're only talking about pirates here. But you also were inside of Niantic, as you mentioned.

• 19:50Speaker 00

  Yes.

• 19:50Speaker 01

  You know, accountable for outcomes in real time, you know, at an actual company that's pushing deliverables. What did that teach you that sort of the researcher being outside of a company couldn't have taught you about this space?

• 20:02Speaker 00

  I think it taught me how much tools matter. And if you take a step back and sort of look at the overall conversation that we're having on safety, there's been a lot of discussion on rules, right? What should be acceptable and not acceptable. And I understand that focus because I think rules are really the most visible part of the pyramid, right? It's the thing that we can all look at and sort of negotiate together. I think I learned at Niantic that policy without tools is basically poetry, right? It looks great, you can recite it, but you can't really protect people with it concretely. And so the importance of those systems, the importance of infrastructure, the importance having multiple systems who you know will be interoperable and will work together.

• 20:46Speaker 00

  Because a lot of important contexts that you need to protect people also get sort of lost in translation when there's a handoff between one process and another process, maybe one internal tool to another internal tool. I think that I got a unique, sort of very up close and personal perspective when I built a trust and safety function, yeah.

• 21:06Speaker 02

  Actually, tooling is really important and it's one of the things that academia usually doesn't get because it's not quite the way that academia operates. So that's an excellent bring into the academic environment, generally, not just in safety, not just in open source, not just in, right?

• 21:22Speaker 00

  And you know, RUISS was incubated at Columbia University, right? At the Institute for Global Politics. I think we forget that academia can also incubate amazing tech nonprofits, right? We look at big business schools and all the startups they incubate. But there's actually a nice tradition of amazing tech nonprofits being incubated out of academia. I think of like Scratch, Love Scratch, coming out of the MIT, right? So like universities here have a role to play. And yes, if they focus on boring things like tooling, we can go really far.

• 21:56Speaker 02

  Yes, I will.

• 21:56Speaker 00

  It will help.

• 21:57Speaker 02

  So I saw that your ABC framework was cited in Anthropic's report on AI misuse. So one, a first, briefly tell us about the framework. And then the question is, does generative AI break the premise of the ABC framework? Because now that you're going to have a single actor, you know, synthesize actor, behavior, content all at once, is that, does it, does it work or does it need to be updated? So first framework and then that question.

• 22:26Speaker 00

  The ABC framework has had a fun life since it was initially published, I think, in 2019. Initially, I was looking at information operations and then sort of more broadly at moderation. And ABC here stands for actors, behaviors, and content. And it simply says that essentially trying to figure out who are undesirable actors on a platform, for instance, convicted child sexual offenders. What are manipulated behaviors on a platform? And what is undesirable content? It's actually three different exercises and require three different approaches that will draw on different disciplines, right?

• 23:09Speaker 00

  One will be closer to borrowing from spam and the other will be closer from borrowing to cybersecurity. So it was just trying to sort of put some order around those three things that we tend to conflate and focus on content where you can see problems where there is both a problematic actor and manipulative behavior and undesirable content. But also, again, it's helpful to sort of separate those three things and think about the tools and approaches that are appropriate for each. It was very helpful in social media days to help sort of reorganize moderation functions where a little bit of the cybersecurity mindset would also come in.

• 23:52Speaker 00

  And it was interesting because OpenAI, I think, were the first ones to also, in this sort of AI era, to also cite the ABC framework. And my talented colleague Ben Nemo, who's at OpenAI, said, actually, I think we can do completions for C. And so he said, the ABC framework still works in the context of generative AI. It's just like actor behaviors and completions. Sometimes, you know, the AI will complete something and you're like, that is undesirable. And so I think he is the ones who sort of like brought it in into the AI context. And I was also very pleased to see it cited in the Anthropic report, yeah.

• 24:31Speaker 01

  Well, so if we're digging into sort of this safety work, you have detection, investigation, review, enforcement. Which of those is hardest to get right? And we're, so one, which of those is hardest to get right? And then we're talking about perhaps it's most important for there to be safety among the big actors that are serving millions, tens of millions, billions of people. But there's a long tail. And so that's also the problem that smaller actors can't, you know, they're under-resourced. And so where, when the under-resourced folks come in, what do they skip first in those four things?

• 25:02Speaker 00

  If you think of detection, investigation, review, and enforcement as sort of a pyramid, right? The, or an iceberg, rather. The exposed part, the thing that everybody sees and likes to comment on is detection, right? So the question is always like, oh, do we know how to detect this harm? It's a helpful, fun conversation. I'll get back to it in a second. But in fact, once you've detected the harm, there are a number of things you have to do to be able to take action. And at RUISS, we focused on sort of building those boring tooling infrastructure pieces so that people have the means to take actions when something is detected. So we talked about Osprey.

• 25:45Speaker 00

  Osprey is an investigation tool. It allows you to say, oh, I can describe an activity that I don't want on my service by its pattern. Please, real time, go find the pattern and help me take action. Super helpful, for instance, if you want to root out fraud or if you're dealing with statistic online exploitation and you have a signal that you want to be able to detect at scale. So that's Osprey. And as I was sharing, this is, this is coming from Discord. The other big tool that we have and that people are using is called Coop. Coop tackles review. Now, Coop came into RUISS through also a bit of a bizarre story, which is we bought a startup,

• 26:29Speaker 00

  which is a funny thing to do when you're a nonprofit.

• 26:31Speaker 01

  I love it.

• 26:31Speaker 00

  Yeah.

• 26:32Speaker 02

  Nonprofit and university.

• 26:33Speaker 00

  Exactly. There were two super talented engineers from Meta who had built a super little startup, which was a really good console that you can use for review and some enforcement, right? Because, for instance, say you want to detect if child sexual abuse material is uploaded on your services. If you do detect some, you actually have to be able to review to ensure that you know what is going on. And so for that, you really want wellness features. And then if you do find that indeed this has happened, you have an obligation to report it, if you're in the US, to the National Center for Missing and Exploited Children, NACMAC. And that report's actually quite hard to do.

• 27:12Speaker 00

  So we bought this startup and we spent a lot of time working to open source the core technology and to really add on to it. It was quite lucky for us that some of the former customers, like Notion, decided that they were in it with us on the open source journey and really helped sort of like take that seed, that base, and sort of grow it into what it is. We re-released Coop a few months ago as the world's first end-to-end tool that allows for the detection, review, and enforcement of child sexual abuse material, fully open source. That's amazing because it means that now people who've always wanted to do this work can do it

• 27:54Speaker 00

  without having to knock on 99 million doors and pay lots of budget. And what we're hearing is that it has sort of met its audience of, again, like people who were willing to do that and didn't have the tool. So I can go into, into many sort of stories for what this looks like in practice. But the point is, we often underestimate the tooling needs that are not just detection.

• 28:22Speaker 00

  On detection, there are still things that we are not very good at detecting. At Roost, we have the Roost Model Community. It's a really nice community where people come and actually share their safety models. So we talked about OpenAI doing this with us a few weeks ago. Mistral said, hey, actually, we also have a model that we use for safety. We think it's quite cool because it's bringing your own policy. It's multi-modal, it's called Shell Strile. We're bringing it to the Roost Model Community. That means they have to do office hours and the community comes and it's okay. I'm testing it in this way. I think this is working. I think this could be improved. Everybody shares their feedback and then we sort of continue iterating, right? There's also areas where nobody has a good solution. And that's where we invest in scientific partnerships with independent, rigorous

• 29:07Speaker 00

  sort of scientific and academic organizations where we say, okay, here we're going to need a big collective effort and push to actually bring some new forms of detection into the world.

• 29:18Speaker 01

  Can I just ask, you know, you said like, you know, Thinkfully Notion came on the journey and there are benefits, like you said, Discord, you know, they did it, but then they get the best of what everyone is doing. I guess everyone's building off them. Like, that's obviously the whole ethos of open source. But are we like relying on the kindness of strangers? I feel like people are so worried that we're like, companies are never going to police themselves or they're not going to do it if. How do you make it so every company says, hey, I'm going to be a part of this ecosystem?

• 29:44Speaker 00

  Yeah, I want to figure out the answer to that question. You know, I, I think that.

• 29:49Speaker 02

  She come on podcast.

• 29:50Speaker 00

  Yeah. I, I think that leading by example is cool. I do think that when we were early, the few companies who said, hey, we're just, we're going to do that, we're gonna, we're gonna invest in this, we're gonna put our own resources, we're gonna share our own systems, we're gonna maintain the commons. I think they, they started something that continues to grow. Right now, we have more and more contributions to Roost. I, I want that sort of race to the top, right? I want people to look at the. Exactly. The benefits that everybody gets by sharing this technology, the benefits that they get back in super talented engineering, the benefits that they get back in trust, right?

• 30:34Speaker 00

  And of course, I want more people everywhere across the industry to say, hey, actually, I also want to share my tools and I can see that there's something that's slightly better than what I have, so I'm going to adopt that. The other thing is, we're very focused on who else should be part of this conversation. I think Silicon Valley, for the longest time, has developed safeguards and safety systems a little bit in isolation. And when we think about, for instance, youth mental health, which is a big topic for us, there's a lot of really important expertise that hasn't been at the table. Working in the open allows us to work hand in hand with clinicians, with sometimes victims and survivors, right? And to bring a lot more different perspectives than to just do this in isolation in a building

• 31:19Speaker 00

  in, in Mountain View or a mental park or whatnot, right?

• 31:23Speaker 02

  One of the things I think people easily understand, or if they pay attention, is to why a safety commons and open source is useful in a cost structure because it's something based off of, in a quality structure because multiple people using it and fixing it is good. Say a little bit about how it might also help startups succeed better.

• 31:44Speaker 00

  Yeah.

• 31:44Speaker 02

  Right? Because part of the thing is like having them grow with it and have it as part of it is actually in fact something that isn't just a, I'm checking the box to make sure I'm not doing something bad, but it's also helping me succeed.

• 31:55Speaker 00

  Yeah. I love this question. Let's take two different types of startups as a thought exercise. The first one is you're a small startup and you're going to build a social product and you want to focus on making that experience amazing. Maybe you're, part of your social product is people can connect and maybe there's a chatbot, right? What I see from the younger builders, right? The younger generation that's building products every day is they have seen firsthand the harms of what it looks like when it can go wrong. They have sometimes personal experience and they want to build safety by design. They actually don't need to be convinced, right? And they think that safety can be an accelerant for them. So they think that, you know, if they can get this technology, then on top of it, they can build a better product.

• 32:40Speaker 00

  There's also markets in which this is going to be the cost of doing business, right? So for instance, in, you know, in certain regulations, you have to go and filter out child sexual abuse material from your platforms or your data set that are going to be used in AI, right? And being able to access the technologies to do this in a way that you can trust, frankly, like allows you to compete in the market, right? So it's a pro-competition thing generally helps people build better products. What about safety startups? That's also really interesting because it's difficult to have a safety startup because, as we said, most of the companies that you're going to try to sell into or organization

• 33:22Speaker 00

  that you're going to try to sell into have to pick things up from scratch. But if everybody has a shared foundation of tools, then it's much easier for your clients to plug into the amazing technology that you're building. So we see already at Roost, safety vendors engaging with our work. We have this amazing company called Zentropy, who has been contributing fantastic, open weights, safety models, you can bring your own policy and tune it. And then we have another amazing startup called Musubi, who's actually selling a hosted version of Coop, right? I'm laughing because in my head, I think about this, this project as Musu Coop because Su Coop in French means flying saucer.

• 34:06Speaker 00

  I'm reminded that this is not official marketing for it. But the point is, you have startups who are like, okay, this is really cool. We can build amazing services and technology on top of the shared safety commons, right? So if we do this right, the, the shared safety commons is not just a hippie idea. It's also something that's an accelerant to good business, startup and innovation everywhere.

• 34:27Speaker 01

  So clearly, it's important for safety for us to get it right. So quality is really important. But you also did mention cost, that this is actually cheaper. And so one platform estimates switching to Coop cut its tooling costs by 95%.

• 34:42Speaker 00

  Yes.

• 34:42Speaker 01

  Can you dig in there? Like, what made the old approach so expensive?

• 34:45Speaker 00

  You know, I don't know what they were using. But I think what matters is, is we are in an environment in which sadly, people are, you know, reducing the budgets that are dedicated to safety. And so we want these resources to be focused on problems that are interesting and unique to those platforms, right? You don't need to use your resources to rebuild over and over basic foundations to build a safety infrastructure on. And we want those resources to be focused on the great people who are going to do this work, right? On retaining the people who are going to think about the policies, who are going to investigate the incidents.

• 35:24Speaker 00

  And so it's true that in this context, giving everybody a shared layer to start from doesn't mean that it's totally free in a sense that then you have to host it, right? But, but it allows people to focus their budget where it will have the most real impact on protecting people on a day-to-day basis, which we're, we're happy to enable that.

• 35:46Speaker 02

  So one of the things, you know, that folks get when they understand open source on the defense is lots of different people can look at it, can rattle it, improvements are shared across. And so there's a set of things that, you know, traditionally, cybersecurity, et cetera, you know, the open source protocols for how the fundamentals of the internet, they get better. But one of the things that also is a trade-off is the attackers can then also look at it.

• 36:15Speaker 00

  Yeah.

• 36:15Speaker 02

  So say a little bit about the principles of this trade-off between getting stronger defense by having a commons, but also how do you deal with the fact that the attackers can also see the commons and what that might mean for helping people navigate this?

• 36:30Speaker 00

  Yeah. Thanks God, this is not a question that we have to take from scratch. It has been structuring the entire field of information security for over a decade. And so I think here we can take some inspiration frankly on how things work on the security side. So if you are doing reverse engineering of malware, for instance, more often than not, you end up using open source tools to do the analysis, but because everybody uses the same tools, you can share the rule, right? Amongst analysts and saying we open source the infrastructure doesn't mean saying you have to share all of the rules for all of the investigative work that you're doing,

• 37:11Speaker 00

  but it does enable defenders to collaborate and then go faster together. And so we see this in Osprey, for instance, right? Saying we have a rules engine that everybody can run for free in a way that's permissionless and you can adapt it to your infrastructure doesn't mean you have to publish all the rules for how you define and catch spam and define and catch coordinated harassment. But if you do find something that you think is a particularly gnarly activity that unfolds on your platform and you have good reason to, to think that it's also affecting another organization and if they're also using Osprey, you can share that rule and then you fast-forward the collective defense. And so it's always that sort of thinking and muscle to develop between what needs

• 37:56Speaker 00

  to be in the commons to really lift up and enable all the defenders and frankly widen the circle of what is protected online and across AI surfaces versus what is sort of like organizations own things to keep and share. And, you know, different organizations will have different preferences on that too, which is totally fine.

• 38:17Speaker 01

  Actually, I have two questions for you. One. Coop, Osprey, Roost?

• 38:21Speaker 00

  Yeah.

• 38:22Speaker 01

  What's the deal? What's the deal with birds?

• 38:24Speaker 00

  Ah, yeah, we like a good pun.

• 38:28Speaker 00

  Eric, you would be surprised by how many chicken puns we're able to produce in minutes.

• 38:33Speaker 01

  So just random, no, okay, no rhyme or reason? We just like birds.

• 38:37Speaker 00

  We just really like puns and also...

• 38:40Speaker 01

  Birds of a feather, okay.

• 38:41Speaker 00

  Exactly.

• 38:41Speaker 01

  Fly together for safety.

• 38:43Speaker 00

  Exactly. It's, it's, sometimes we also repeatedly use the same puns. So now every time I say, well, you know, it's a chicken and egg problem, there's a night there.

• 38:53Speaker 01

  Yep.

• 38:53Speaker 00

  I will remind everybody that our Discord, where all those great puns are flying around, is completely open. And if you want to engage in technical safety discussions while making very fun puns, if you want to be punny, you can join us on the Discord.

• 39:08Speaker 01

  Okay, right with dad jokes. Great, glad we solved that. Second, perhaps more importantly, the conversation is sort of premised, you were saying, that the problem with safety startups is, you know, Smite gets acquired and then it shuts down. And so there's a rug pull, people who are relying on this. But Roost is a nonprofit. And so it is also predicated on the idea that people are going to fund it. So one, how do you ensure that funding continues? And what happens if Roost's own funding dries up?

• 39:36Speaker 00

  I feel like you are inviting me to make a very public call. Your people to come and fund our work.

• 39:41Speaker 01

  Sure.

• 39:42Speaker 00

  Please, please come and fund this work. Come and fund this work because it has a very, very tangible impact. And fund this work too because it's a path to resilience. I love your question because we are building this commons in a way that it can outlast the organization.

• 39:59Speaker 01

  Yes.

• 39:59Speaker 00

  The idea of our work is to bring enough people on board, to bring enough of a community that innovation happens, more people have access to these tools, trust gets built. And if tomorrow we disappear, all of this stays there. That's the difference between an open source commons, which has this resilience, right? If tomorrow there's no Roost, people can absolutely still download and maintain and handle Osprey and Coop. Now, there's a lot of work that we do to, again, like catalyze this commons, make sure there's a common roadmap. I'm particularly excited about some of the scientific efforts we're also building to, because now that we have the commons roadmap, we, we know where they are, real gaps.

• 40:40Speaker 00

  And we're able to join forces with institutions like Mila in Quebec or with the UMass Rescue Lab to say, okay, let's, let's focus and sort of plug these holes and gaps. But, but we are building with resilience in mind. I think that's extraordinarily important in the current times, right? Idea is really not to replicate a hyper centralized system. That's why we chose the open source way.

• 41:05Speaker 01

  Makes sense.

• 41:06Speaker 02

  So one of the things I, I think the resilience that you've described is absolutely fundamental. One of the great things around open source, which is because it's a collective digital commons, it gets all the features of that, which is it can be, it never dies, it can be resumed, it can be steered in different directions, multiple players can engage in it. It's part of the thing that is magical about open source. So, but there's also a lot of different constituents. So there's nonprofits, there's universities, there's companies, there's engineers, there's random people.

• 41:39Speaker 00

  Yes.

• 41:40Speaker 02

  Say a little bit about kind of how these different elements come together, because I think that's important for people to understand the resilience and the vivaciousness and the liveliness. And then we're going to start shifting into, and how does AI, right, you know, begin to change this?

• 41:56Speaker 00

  The magic of having an open source safety commons is that once the wheel starts turning, people start seeing how they can plug in. And so you have people who, for instance, don't have technical skills, but are really, really good at building policy. And as new big labs are saying, oh, here's a safety model you can use, but you need to bring your policy, a few people are saying, oh, I can write a very detailed clinically informed description of what suicidal ideation looks like in the context of an AI chatbot. And that happens to be super helpful for people who are going to use AI models for

• 42:37Speaker 00

  this, and I'm going to contribute this policy in open source in a policy pack. You of course have great engineers who are coming and are saying, I see this tool, I have an idea for how to improve it, I'm going to submit an improvement. You have scientists who are saying, oh, I've always wanted to go and actually test some of these tools, and I'm going to go and I'm going to do this. And then I'm going to contribute a few insights from fields that have been traditionally left out of how we build safeguards that actually could benefit from this. And so the work of an organization like ours is to architect and organize that big collective will around these tools, is to make sure that everything is interoperable, right?

• 43:20Speaker 00

  Otherwise, you have like an island of misfit toys with a bunch of things that don't work together, right? What we want is a good, strong interoperable commons. We have people using all of our tools together, right? So Matrix, for instance, is using both Coop and Osprey. And then you have people who only use one part of your system, right? So, you know, the work is to, again, orchestrate and organize this big collective effort that brings a lot of very diverse stakeholders, but that diversity is the strength of an open source commons. And I will say, in the team, we have amazing expertise from the sort of traditional open source side of things. So I have fantastic CEO and who used to run open source at Google. And so she understands how you sort of bring all these pieces together.

• 44:02Speaker 00

  Cassidy is our absolutely fabulous community manager. She's so good. It's sort of like, you know, seeing two people go in very different directions, saying, actually, I know how we bring this into a community through office hours, we further the project. And so, yes, there is definitely a muscle to orchestrating that big collective and very diverse community around it. But, but this is what gives it the strength and this is what gives us tools that are actually robust and, you know, better than people doing this alone.

• 44:32Speaker 02

  Before we next continue down the AI thing, I think one of the things also may be helpful for a lot of people to hear is how does being open source help engage the kind of the intense intellectual and scholarly work, research work at a university? Because that's something that most companies don't have any real access to. And that's one of the many reinforcement loops that actually provides an advantage here.

• 44:57Speaker 00

  Yeah, it's a, it's a language translation question, right? So if you take this work and publish it with the right colleagues and in the right journals, then, you know, people who come more from the scientific and scholarly practice see it and say, oh, I understand the language up working through a data set in the paper. I can add my piece to this. And so, again, it's about sort of like translating that work across different audiences to engage everybody. Right now, our director of product, Juliette, is doing this on the difficult problem of grooming and thinking about, okay, what can we learn from law enforcement who's working on this?

• 45:39Speaker 00

  What can we learn from the different really talented academic teams that have been tackling this problem? And from the many corporations in the Roost community who say, we have tackled this, we have some context, we have some data. How do we pull all of these pieces together that are currently quite separate to really radically accelerate innovation when it comes to grooming detection? We've started this work. Roblox, for instance, shared a first packaging classifier that's focused on grooming detection, got feedback from the Roost community, both people who are trying to use it, people who were thinking about grooming, took that feedback and re-released something that's faster, easier to use and load better. So it's sort of doing this orchestration of, okay, let's go and grab these perspectives

• 46:24Speaker 00

  and let's organize some collective push towards very focused, discreet, well-described projects we can tackle.

• 46:33Speaker 01

  So coming into the AI age, AI is making it just easier and cheaper for folks to launch apps, chatbots, everything. And sometimes it's small teams, sometimes it's one person teams.

• 46:45Speaker 00

  Yeah.

• 46:45Speaker 01

  And they could have no background in safety whatsoever. How can these small teams make things safe right from launch?

• 46:54Speaker 00

  I have a lot of thoughts on this, though. I have personal thoughts, professional thoughts. I suppose I should start by acknowledging that I am myself a compulsive vibe coder.

• 47:05Speaker 01

  Great.

• 47:06Speaker 00

  Vibe coder in a number of apps, including like family apps. And I am excited by the fact that people are able to use AI to bring their own vision of something into life faster and faster. Our big vision at Roost is that we need to...

• 47:24Speaker 00

  Bridge the gap between how easy it is to come up with a new app, a new service, and how hard it is to actually do some of the important steps of making it secure. I think where we feel that the most right now is on child sexual exploitation. We have a lot of builders who are coming in and saying, we want to do this. We want to do this right. And we're very excited when that happens. We're like, okay, we will take you through those steps. But the reality, for instance, of removing child sexual abuse material from a foundational data set that's then going to be used to train an AI model is that right now it is a hero's journey.

• 48:06Speaker 00

  And you have to go and then register with, you know, the authorities on one hand, and then go ask for a hash bank and then go and implement, for instance, HMA to use as hash bank. And then you have to go and ask for a novel CSM classifier and then implement it into something else. And so a lot of our focus is how can we make that process much easier? The overall vision is that one day it's going to be as easy to launch an amazing website and app than to just type Roost at safety.

• 48:38Speaker 01

  Totally.

• 48:38Speaker 00

  And then we'll just do it.

• 48:39Speaker 01

  Yeah.

• 48:39Speaker 00

  Or even, you know, by default, those tools will tell you, hey, so fun that you've added a field in your app that allows you to upload, say, kids' books into an app. Also, you may want to run a moderation layer here. And here's why, and here's how, and we'll make it so easy for you. So we want to make that much easier. There's a lot of work to do to do this. And we're excited to work again, like with others to go in and tackle that gap.

• 49:10Speaker 02

  So you started with a partnership of OpenAI and the GPT OSS safeguard. So what would encourage more frontier labs to contribute to Roost? And are the various frontier labs also contributing a bunch of tokens relative to the vibe coding side, which is, you know, kind of the equivalent of like Methos analysis, other kinds of things? Because that would strike me as a great thing to be doing. I'm hopeful that there's a positive answer here, but I figured I'd ask.

• 49:42Speaker 00

  Let's make it happen.

• 49:43Speaker 02

  Yes.

• 49:43Speaker 00

  I think we're also starting to see consumer demand and regulatory demand for that, which I'm excited about. And one way to think about it is that's kind of what we expect from other industries. So for instance, Reid, if you wanted to buy a car and I was trying to sell you my car, and I would tell you, it has airbags. I don't know if they work really well if you're short. Also, it has a seatbelt. I think it's made of paper, but I'm not sure. But also, I don't really know what other people use. And you're going to have such a great time. You would be like, I want none of this, right? Because in an industry like, you know, cars,

• 50:27Speaker 00

  we have a set of standards and shared references and shared technology for what we know keeps people safe. In, you know, most cars that you can buy these days will use a three-point seatbelt that was developed by Volvo. And in 1959, they said, we think it's a really good model. Actually, we're going to give all the other manufacturer the right to do this for free. And then we're going to tell people how we do it so that we can continue improving on it. Right? And that's sort of our expectation for how things work. That's also what enables innovation, right? The reason why we have super fun cars that go really fast is because we know how to make them secure. And so I'm also hopeful that there's going to be increasing demand from consumers and regulators

• 51:10Speaker 00

  to sort of tell people like, hey, if you're going around town telling everybody that you feel super satisfied with the way you keep people safe online and on AI, maybe you should give everybody those tools. Maybe you can join me in having this public call, right? To all of those amazing AI labs. I know they have more to contribute.

• 51:31Speaker 00

  And I can tell you, like, people are hungry for this tech. What's really fun when you have an open source stack is you discover people using your tools in production through different and funny ways. So I'll tell you two stories.

• 51:48Speaker 02

  Please, please.

• 51:51Speaker 00

  There was a small fandom app, two people. They found Coop through LinkedIn, actually. Promised.

• 52:00Speaker 02

  Yeah, great.

• 52:02Speaker 00

  And they installed it totally on their own. We didn't know. And they deployed it. And in two weeks, they found 19 incidents of child sexual exploitation that they were able to report to NECMAC. And then they reached out saying, thank you for enabling this work. We really wanted to do that. We didn't have the tools to do it. Then we worked with NECMAC to see, okay, well, what does it look like to do this experience from the perspective of somebody who has a team of two? And I'm not saying an engineering team of two. I'm not saying a safety team of two.

• 52:32Speaker 01

  Two people.

• 52:33Speaker 00

  I'm saying a team of two people, everything two people. And to think about, okay, how can we make this as easy as possible for builders who want to do the right thing? That teaches us that the coalition of the willing, of people who want to get this right, is so much bigger than the people who are currently equipped with the tool to do this, which again, is a good news in our ability to quickly make a dent, a real dent in protecting people online. Other story, actually also LinkedIn related, quite fun. One day, the lead contributor on Coop, so the formidable engineer called Juan,

• 53:12Speaker 00

  receives a message saying, from Atlassian, saying, hey, we're going to submit a pull request. Can you review, review it? And we're like, Atlassian's using Coop? We didn't know that. That's cool. They're big. We're excited that large companies are staking, you know, some of their safety infrastructure on these tools, because then it's exactly what's going on. Then they contribute to it. So it's absolutely awesome to discover the use of your tools by amazing engineering teams knocking on the door saying, hey, actually, yes, we've been using it. Actually, we think we can improve it.

• 53:53Speaker 00

  Today, if I just think about Coop and Osprey, I'm not thinking about the Roost model community and the amazing AI safety models, just Coop and Osprey. The people whom you, we know are using them in prod reaches around 370 million monthly active users across platforms and organization. That's our conservative count.

• 54:16Speaker 01

  Amazing.

• 54:16Speaker 00

  I think we can do so much more. There are so many more organizations to protect. What's fun about this number is that it's both large platforms and small organizations, two people team, or for the first time, we also had a journalism team say, hey, we're building our own community. We actually want it protected and we're going to use Coop.

• 54:38Speaker 01

  That's awesome. Well, so we've talked a lot, there's a lot more to online safety than keeping kids safe. But we talk a lot about keeping kids safe. And Reid and I also talk a lot about AI companions. And so getting a little specific, when you're thinking about creating an AI companion for a 15-year-old, what would a genuinely good AI companion look like? And what are the open tools that the developers would need to build one?

• 55:04Speaker 00

  This is a hard question and this is an urgent question.

• 55:09Speaker 00

  What we know from the usage data of AI companions and kids is that kids are increasingly turning to AI companions. And when we look at the types of conversations they're having, they're also discussing feelings and their own problems and seeking support. And so there's a lot of really important and consequential decisions to make, science to develop, safeguards to build around what does it look like to keep those interactions safe? I am not part of those who think that we should just unplug all the kids. We talked about this. I grew up playing with technology.

• 55:51Speaker 00

  I think it gave me a lot of meaningful connections and friends. And I think it gave me a meaningful career. And we want kids to be able to play with technology and to engage. We also want them to do so in a way that's safe. I think safe in this context include things like products should be honest about being AI. We haven't really crossed that branch just yet. For instance, we, I think a lot about AI and toys today. If you're going to sell a toy that has AI embedded in it, you don't actually need to disclose that. And so you may end up with having an AI toy in your house without anybody sort of thinking twice about it.

• 56:35Speaker 00

  There, I think consensus around companions respecting age-appropriate boundaries and about supporting real-world relationships, right? Ultimately, we don't want those companions to isolate young people from their support networks, their friends, their families and their community. I don't think any of that is solved. I think this is a very active area of focus for the field, for Roost, of course. And because this is not solved, because this is extraordinarily urgent, and because this is hard, that is why we're doing this work in the open with scientific institutions like Mila.

• 57:18Speaker 00

  Our suicide prevention safeguard, we released it in July. And what we're saying is we're going to do this work in public and incrementally so that everybody is able to come, join in and say, okay, let's get this right. And we can sort of divide and conquer together to make sure that, again, we accelerate the pace of innovation. Because yeah, those usages are widespread. And I think we've heard firsthand and in extraordinarily dramatic ways, the cost of getting this wrong.

• 57:48Speaker 01

  So it sounds like you don't think we're quite there yet, that there is more sort of research and tooling needed before we're able to safely deploy.

• 57:55Speaker 00

  Yes, I don't think that we know enough to say we absolutely got this one solved.

• 58:03Speaker 01

  Yep.

• 58:03Speaker 00

  I am very grateful for the builders who are trying to get this right. I'm even more grateful for the builders and scientists who are trying to get this right in public and with us. But I do think there's a lot of work to do here. And, you know, again, I think that the usage data shows that this work is really urgent.

• 58:21Speaker 02

  One of the classic challenges of someone's building something, trying to include kids. You kind of have two paths, given that. One is, we're just not going to include kids. Or we're going to include kids and we're going to learn as we go. Probably, we can only get success by doing the second path. But what would be some of the, oh, make sure you're doing this. Make sure you're engaging this. Make sure that you're asking these questions. So that as you're learning, as you're going, you're contributing back to everyone else. And we're making the environment, the right kind of safe environment of kids, an elevating environment of kids that they absolutely deserve.

• 59:00Speaker 00

  Again, I think that there's sort of some of those big questions, right? Does, does, does everybody know that there is AI involved, right? Do you have a good sense of what AI is involved? Which safeguards are running around it? Do you have protections around some areas like self-harm? Can you ensure that, again, you're not building a system that is going to lead to more isolation? These will also depend on the specific types and usage, right? You're going to have somewhat different conversations. Again, if you think about like AI embedded in toys, than if you think about multimodal AI,

• 59:43Speaker 00

  where you have a whole lot of different issues around images, for instance, right? And similarly, building for a 12-year-old is a very different story than building for a 17-year-old. So I think we need to sort of engage with all of that nuance and, and take it with utmost seriousness and do this work rigorously scientifically in public and say, this is what we know, this is what we don't know. And this is the reality of how those tools are being used. And frankly, of how kids are being harmed by it today in those very specific ways.

• 1:00:19Speaker 01

  So if we brought in from just kids, just sort of looking at, looking at the whole ecosystem and thinking about that chatbots, language models, agents, like they're delivering a lot of services that used to be delivered by humans, especially in the safety world. So is there a specific safety capability that you're like, that would make the biggest difference if we could bring that online? And like, how would you show that that works in the real world?

• 1:00:42Speaker 00

  Yeah, I mean, I have a long list of CCL need. And again, I think there's a, there's a distinction between like, problems that we kind of have solved for, but they need to be more robust, like better distributed, battle-tested.

• 1:00:56Speaker 01

  Yeah.

• 1:00:57Speaker 00

  Invested in versus problems that are a little bit more new, that we're trying to wrap our, our head around and thinking like, oh, that is really bad. How do we tackle this?

• 1:01:06Speaker 01

  Yeah.

• 1:01:06Speaker 00

  I'll give you an example of the latter.

• 1:01:07Speaker 01

  Please.

• 1:01:08Speaker 00

  Non-consensual intimate imagery. Today, one in 25 American teenager.

• 1:01:16Speaker 01

  Oh, God.

• 1:01:16Speaker 00

  Says that they have had a deep fake nude produced of them.

• 1:01:22Speaker 01

  Oh.

• 1:01:23Speaker 00

  I'm not saying one in 25 tell you, I know somebody to whom this has happened. One in 25 tell you, yes, this has happened to me. This is an enormous crisis because of course, those deep fakes are then used for extortion, for different types of humiliation, harassment and harm. And thinking systematically about, okay, how do we tackle that novel usage? With the widespread scale of, of that problem and the very real consequences, that requires new creative thinking. And frankly, like a whole of industry and scientific approach on where do we focus technical defenses? And make sure that they're distributed as widely as possible

• 1:02:08Speaker 00

  to get ahead of the fact that this is an enormous wave of harm coming for teenagers.

• 1:02:14Speaker 01

  I wouldn't have even thought of that one. And now it's going to haunt my dreams. So thank you.

• 1:02:19Speaker 00

  Well, that's why we have all the puns is because.

• 1:02:21Speaker 01

  Yeah, let's get it solved.

• 1:02:22Speaker 00

  Let's get it solved. Yes.

• 1:02:24Speaker 02

  So, I mean, as part of that, so how are the collaborations with the frontier labs and other AI labs? What kinds of things need to improve? What are the ways that government, civil society can help this as well? Like, how do we get that in a faster and stronger kind of thing? Because, for example, solving the, the, you know, whatever. I don't know what the exact term for the sexual abuse from deep fakes is. But like solving that is critical. And the kid doesn't even have to be a user.

• 1:03:00Speaker 00

  Yeah.

• 1:03:00Speaker 02

  Of it.

• 1:03:01Speaker 00

  That's right.

• 1:03:01Speaker 02

  For that, right? It's like, that's the, it's not even like, yes, how they use it. It's other people using this. So what's currently going on and what should we be doing to make it better?

• 1:03:10Speaker 00

  I think this is a call for everybody to roll up their sleeves and work together. We need more attention to this problem. There's a lot of really good civil society groups who are doing these surveys. That's why we know that this is happening. And who give us a lot of context on, okay, well, what does that look like for a kid? How does that happen? There are a lot of players who have technology to contribute that can help mitigate this harm at sort of the different areas where it needs to be stopped, right? And we want that technology to be contributed in the open commons. We also do need a philanthropic organization to say, we're going to take a bet on the fact that we can accelerate this work. We can do this in public.

• 1:03:55Speaker 00

  We can do this with the rest of the scientific community. And, and as I said, I think that there are a lot of expertise that live in civil society and in the scientific community that for too long, we've said, I know you don't belong in the safety and safeguard conversation. If we want your input, you will sign a stack of NDAs. You will come to some office in Silicon Valley. We will hear you and then you will leave and you will never know what we took from your remarks. That's not a good model. That is not going to give us the pace and scale and excellence that we need in the face of these problems. So I think this is, this is one where everybody can really bring something. I'm somewhat encouraged and also somewhat disturbed

• 1:04:39Speaker 00

  by all the conversations that we're having right now on pacing the frontier and how everybody needs to bend together to do something. There is also sub-frontier problems that are very urgent, very catastrophic, and that we absolutely should bend together to focus and solve these problems.

• 1:04:55Speaker 01

  Right. And I mean, and so much of the conversation is also about global cooperation and what is happening around, around the world. And so you were recently speaking of the G7 in France. And you were talking about the importance of open source, digital sovereignty, child safety. Can you talk about why this is so important in the context of European builders? And so like, why is it helpful, especially in that context?

• 1:05:18Speaker 00

  Yes. I was invited to contribute to the G7 and not only because of some of my academic work with a broader coalition of colleagues on defining what we mean when we say open source AI. Because of course, open source has had a traditional definition that we've worked through a lot for software. But when it comes to AI, it's a little bit different. And in fact, there are different ways to do open source when it comes to AI. And open source AI can itself take different shapes. It can be just open weights, or it can be everything is open, including documentation and data sets and safeguards.

• 1:06:00Speaker 00

  Or it could be many different combinations of this is open, but not that. And you sort of have to think about the entire stack. And for each component of the stack, you have to think about, what would it mean to open that? And then you think about like, what am I trying to achieve? And which shape of openness gets me there? That conversation is very relevant to sovereignty. Because of course, you have the European Union, but also most recently, Canada with their big AI strategy saying, well, we want to be able to own, not rent, the different components of the stack that matter to us. That is also extremely relevant for safeguards, right?

• 1:06:42Speaker 00

  Because similarly, if you were going to legislate on child protection, you also want to make sure that people, your own ecosystem of innovators, have the means to comply with your law. Otherwise, the more we call for regulation, more rules on this topic, without investing on open source tools, the more what you're actually doing is further concentrating power in the industry and closing down the market around a handful of incumbents who have the safety engineering tools that are, and, and teams that are able to comply with those laws, right? So it's critical to your ability to protect your citizen and to make regulations in this area.

• 1:07:24Speaker 00

  And it's also critical to your ability to have, you know, independent and strategic and sovereign versions of your stacks.

• 1:07:35Speaker 02

  Well, going a little bit more global, one of the challenges when we deal with a lot of the prospective harms is that, for example, you know, abusive images can be created anywhere. And so, and distribution networks and all the rest. So what needs to happen globally, right? Because, you know, one of the questions is, is there will be image generators, for example, running in the vast majority of countries. There's already open source versions of them or open weights versions of them. And there's nuances, you were gesturing at there. So what, what needs, what should be thought of as some good basics or some good starts for the G7 coming together, for, for the UN? We're here in UN General Assembly Week.

• 1:08:20Speaker 02

  You know, what, what would be the kinds of things that would be good things to do globally?

• 1:08:26Speaker 00

  I think my first ask is never use the idea that you can't root out all of the problem to justify not taking the extraordinarily important incremental steps that really are going to save lives and reduce harm.

• 1:08:43Speaker 00

  Yes, there will be people doing this with models that we don't have control of in places that we won't see. But also, when you look at the way the majority of these harms are being perpetrated, it is happening with apps that are being advertised on social media, distributed through app stores. And in context, where there's actually so many different levers that we can identify and say, here, there's a good chunk of the harm that if we focus and join forces, we can remove. So I'll say that's my sort of first, most important goal, which is like, let's get progress done. Then we'll worry about the sort of remaining part that we can't get to. But there's so many lives we can save, harm we can remove,

• 1:09:28Speaker 00

  good we can do by focusing on all these incremental steps that we're right now not investing enough on.

• 1:09:33Speaker 01

  You've described your generation as internet optimists who believe cyberspace would spread democracy and expose corruption by default. We all heard this, we were so excited, and some of it came true. There have been, you know, many good things that technology, social media, the internet, et cetera, have brought about. We didn't, though, predict all the harms. And, you know, we watched some of these beliefs get weaponized. And so is the optimism you still carry, is that a conclusion from the evidence that you've seen? Or you are just an optimistic person? Like, how would you say your optimism has sort of waned and waxed over the years?

• 1:10:14Speaker 00

  There is somebody somewhere who could probably answer the question of, what is optimism? Where does it really come from? And I would like to know. I think maybe in my case, perhaps it's a personality trait. I am part of this generation who is extraordinarily excited about the power of technology and internet. I remain extraordinarily excited about it. Over the years, I've had to work on violent extremism, terrorist use of these systems, foreign interference, hate mobs, suicidal ideation, child sexual exploitation. And so my colleagues often say that I am maybe the most optimistic person,

• 1:10:57Speaker 00

  looking at the darkest corners of our technological life.

• 1:11:00Speaker 01

  Seen the worst.

• 1:11:01Speaker 00

  But I always see those reasons for optimism. The first one is that there is so much we can do. If we join forces, we know what to do. We just need to focus. And we are still in an environment where we can make very tractable progress by doing things that we know how to do. And so that gives me a lot of optimism because then I know where to focus my action, right? There's this question of like, does action come first? Or does hope come first? I actually think that hope comes from the action. The other thing is, I am very excited by the next generation of builders. Sometimes when we think about child safety,

• 1:11:43Speaker 00

  we can describe the next generation as sort of a big, unified, massive youngsters to which tech is happening. But that's not the case. There are also the builders of tomorrow. They're also very creative and fun. And there are a bunch of maker pirates within them, too. And it's fun to see what they do. There's a hashtag online that says Cyberdeck, for instance, where you can see young girls take Polly Pockets. And inside, they build mini computers, and then they run local AI models. That's really cool. The next generation is building systems that they want. They're going to build a technological environment that they want to see. And I think we need to bring them at the table,

• 1:12:28Speaker 00

  not just as victims, but also as builders and shapers of our technological futures. And everything that we do at Roost, we also do for them. And it's super optimism-producing to see them knock at the door and say, hey, I'm going to build this weird app. I need all these safety tools to do it. And we're like, yeah, go, go, go. You don't need her permission. Go to your app. So yeah, that's exciting, and that produces optimism.

• 1:12:51Speaker 02

  So one of the kind of phrases around this that I really like is cyberpeace. And it describes not just fewer bad outcomes, but actually a kind of a healthy and dynamic and growing system, you know, kind of equilibrium. And you've talked about it some today with pluralist audible safety, you know, kind of ecosystems. Communities get more nuanced and detailed and specific to their community kind of agency. What do you think are the most important things to help continue, accelerate to that path?

• 1:13:32Speaker 02

  Like, what should people understand about what kind of ecosystem we're trying to get to? Why should it be kind of almost like a, what? You're not using these tools? You're not participating in this? Why not, right? You know, it's kind of, this was the why, the why not? What do you think it takes to get there?

• 1:13:50Speaker 00

  So I've done my academic research and ultimately my dissertation on this concept of cyberpeace. And that is because I was amongst a group who was studying cyber warfare. I had two absolutely fantastic advisors who encouraged me and supported me to sort of look at the problem maybe from a different angle. And I was taken by the fact that everybody was very focused on defining cyberwar, defining the threshold of armed conflict. And it's, as I said, not unlike the conversations we're having right now with the hyper-focus on the frontier. But there is all this great theory of positive peace building. What happens below the threshold? If we're so intent on defining cyberwar, do we know how to define cyberpeace?

• 1:14:34Speaker 00

  Do we know how to invest in the mechanisms that make our online lives livable, that make room for thriving, that make room for cooperation? I think that the disappointing answer to your question is that some of that is the power of our focus. We get to decide where we want to spend our energy, where we want to spend our efforts, where we want to build, where we want to spend. And saying we care about making the sub-frontier, sub-war space livable, we care about tackling these harms that are immediately in front of us is a decision we can make with our focus and making sure that we can keep centering these

• 1:15:16Speaker 00

  in the conversations we're having around AI, around harms, around AI safety, I think is part of the problem. We don't need to have this opposition between catastrophic risk and immediate risk. I don't think that opposition is particularly, of course, I understand why it exists, but we don't need that strong of opposition. We can make sure that we focus on both sides of the line and that we do both at once.

• 1:15:43Speaker 01

  This is just reminding me, it's like, government works well when you forget that it exists. Like, none of us know that we're getting tons of spam text messages a day because that system actually just works. And some of the spam texts were just filtered out. We still get some, but most are filtered out. Like, all of these systems, when they just work, you, in a good way, get to take them for granted. It's sort of like the Maslow's hierarchy. It's like, if safety is the floor, that's right. Like, we have to have it exist before we can get to flourishing and before all these great things can happen. And so what does that actually look like? Once we have safety as the floor?

• 1:16:17Speaker 00

  Yes.

• 1:16:18Speaker 01

  And then what does that look like when cyberpeace is real? For a teenager, a platform, government? Like, how to paint that picture?

• 1:16:24Speaker 00

  I love that description. Like, yes, we want to become ubiquitous, and you don't think about it.

• 1:16:31Speaker 01

  Totally.

• 1:16:31Speaker 00

  And things are just more safe. And that is just the way it happens to be. And frankly, we have that on some corners of our online lives. How often do you think about the amazing work that Less Encrypted is doing? I don't know, maybe you do. But generally, when people start a new website, they're not going to their office thinking, oh, I need to figure out cryptography, right? Like, no, we don't, right? We've managed to sort of like have building blocks and institutions that raise the floor for everybody. I think we can do this for safety. I think it will look different in different products. Because again, we don't know what people are going to build. What we know is they are fundamental things that need to be done in order for those users to be safe,

• 1:17:16Speaker 00

  that some of this we know how to tackle, and we need to make those technologies available to everybody. Some of this we don't know how to tackle yet. We need to focus, make that scientific effort, build those new safeguards. And then again, will this look like new, very fun chatbot interactions? Maybe? Will this look like less spam, hopefully? Less fraud? Please, that would be great. Less child sexual exploitation? Absolutely. Less interactions with chatbot that leads to violent extremism? Yes, please, all those things, right? But like, yes, the idea is exactly what you said, which is, let's make it ubiquitous so that it's much easier to build

• 1:17:59Speaker 00

  from a shared, safe foundation.

• 1:18:02Speaker 02

  All right. So let's go to rapid fire. And you can answer at any length. These are just the same questions we ask all of our delightful guests. So is there a movie, song, or book that fills you with optimism for the future?

• 1:18:16Speaker 00

  Many. I will pick one because I think that's the game.

• 1:18:20Speaker 02

  Yes.

• 1:18:20Speaker 00

  So.

• 1:18:21Speaker 02

  Or if you want to do two, it's fine.

• 1:18:23Speaker 00

  I am a Trekkie. I love Star Trek. I am a Trekkie of the Deep Space Nine category. And I'm not sure why, but it actually turns out that a few people around Roost and in the Roost community also are Trekkies. So I think that there is something about Trek inciting optimism about the future, optimism about our ability to collectively govern it at interplanetary scale. But yes, when I think about our board, for instance, we have amazing people like Audrey Tang, or Megan Smith, or Maria Ressa, and all those people are also Trekkies. And so it has happened that we have very serious Roost discussions. And then the end is like, okay, okay, good.

• 1:19:08Speaker 00

  Live long and prosper.

• 1:19:11Speaker 02

  Awesome.

• 1:19:13Speaker 01

  What is a question that you wish people would ask you more often?

• 1:19:16Speaker 00

  I wish people would ask, who should be at the table when we make safeguards? Again, I think we've inherited that habit of doing this a little bit on our own, in the dark, in a way that's very opaque and sort of siloed. And there is another way to build and thinking about, well, who else has an interesting expertise to contribute on a very difficult societal problem? Is a question I wish people would ask more. And I think we will bring more people and, yeah, more perspectives.

• 1:19:46Speaker 02

  So where do you see progress or momentum outside of?

• 1:19:51Speaker 02

  Call it, you know, all the things we've been talking about, your industry. But like another area that you would like to shine a light on that inspires you.

• 1:20:00Speaker 00

  Well, it's it's interesting. I know you said another area, but to people who've been working on child safety, we are going through a very interesting moment where that conversation is becoming a kitchen table conversation. And so I am quite inspired and energized by the fact that many people around us are starting to think about, okay, well, what type of relationship do I want between young people and tech? And what is my piece of it? I'll give you a very bizarre example. I recently became a mother and as a result, my daughter goes to daycare. And they had a conversation on, well, we have Amazon Alexa in the classroom,

• 1:20:44Speaker 00

  but the wake word that we use is Alexa. And in an environment in which this generation is going to grow up with companions, is that what we want? Because that's anthropomorphizing the relationship to the computer. Now it turns out, here's a tricky secret, that one of the other few available default that you can use for the wake word is computer. Like in Star Trek. And they really.

• 1:21:12Speaker 01

  Did you know that?

• 1:21:13Speaker 00

  Yes, you can go and change it tonight. And so they were thinking about how to do their part in finding the right relationship between youngsters and machines as everybody sort of grows up again surrounded by AI and changed the wake word from Alexa to the very tricky computer. And I'm excited by the fact that many people are having this conversation at so many different levels, right? In classrooms, in boardrooms. And this is becoming much more of a mainstream conversation.

• 1:21:45Speaker 02

  Awesome.

• 1:21:46Speaker 01

  In my house, my kids just try to say, hey, Google, in the craziest accents at the loudest volumes, at the strangest tenors, so we haven't quite.

• 1:21:54Speaker 00

  No, that's great.

• 1:21:55Speaker 01

  We're not at that stage yet, but.

• 1:21:57Speaker 00

  This is super. I think you have red teamers on your hands. I think you should send them to Bruce to go and red team some safeguards.

• 1:22:04Speaker 01

  They're very into it. All right, our final question. So can you leave us with a final thought on what you think is possible to achieve? If everything breaks humanity's way in the next 15 years, and what's the first step to get there?

• 1:22:16Speaker 00

  Ubiquitous safety for everybody. Come roll up your sleeve and build it with us. We'll get there one step at a time together.

• 1:22:25Speaker 01

  Here, here.

• 1:22:26Speaker 02

  Here, here.

• 1:22:27Speaker 01

  Kari, thank you so much.

• 1:22:28Speaker 02

  Me so. Live long and prosper.

• 1:22:31Speaker 01

  Beautiful.

• 1:22:32Speaker 02

  Possible is produced by Palette Media. It's hosted by Ari Finger and me, Reid Hoffman. Our showrunner is Sean Young. Possible is produced by Tanasa Delos, Katie Sanders, Spencer Strasmore, Imo Zoo, Aman Suri, Danny Garrison, Trent Barboza, and Tafadzwa Nima Rundwe.

• 1:22:50Speaker 01

  Special thanks to Surya, Yala Manchili, Sayida Sabieva, Ian Alice, Greg Beato, Parth Patel, and Ben Ralas. And a big thanks to Jeremy Ponak, Anne Bertuccio de Hollenbeck, Jacqueline Ortiz Ramsey, Danielle Hogan, and the team at Lighthouse Brooklyn.