An inside look at X’s Community Notes | Keith Coleman & Jay Baxter

Keith Coleman (VP of Product) and Jay Baxter (founding ML engineer), the minds behind Community Notes, reveal how a small, scrappy team inside Twitter/X built the most trusted crowdsourced information system on the internet—one that’s changing the way we understand truth online. What you'll learn: 1. How Community Notes actually works—a deep dive into the groundbreaking algorithm that rewards “bridging agreement” instead of majority rule 2. The seemingly crazy yet brilliant way this idea survived multiple CEO changes—from Jack to Parag to Elon 3. How this project started with a dumpster fire GIF (literally)—the untold backstory of its early launch 4. The secret to running ultra-fast, high-impact product teams—no OKRs, no Jira; just one Google Doc 5. What Meta’s adoption of Community Notes means for the future of online (mis)information—why this open source system is becoming the industry standard — Brought to you by: • WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs: https://workos.com/lenny • Productboard—Make products that matter: https://www.productboard.com/lenny • Wix Studio—The web creation platform built for agencies: https://www.wix.com/studio?utm_source=Lennyspodcast&utm_medium=Podcastad&utm_campaign=SL Find the transcript at: https://www.lennysnewsletter.com/p/how-x-built-the-best-fact-checking-system-on-the-internet Where to find Keith Coleman: • X: https://x.com/kcoleman • LinkedIn: https://www.linkedin.com/in/keith-coleman-19b12b46/ Where to find Jay Baxter: • X: https://x.com/_jaybaxter_ • LinkedIn: https://www.linkedin.com/in/jaybaxter/ • Website: http://jaybaxter.net/ Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ In this episode, we cover: (00:00) Introduction to Community Notes (06:56) How the “bridging-based” algorithm works (13:33) The impact and scale of Community Notes (17:24) Understanding the note publishing threshold (21:32) Challenges and philosophies (26:26) The effect of notes on re-sharing content (29:41) Origin story (35:46) Embracing small teams for big impact (40:23) The thermal project approach (47:47) Algorithm development and internal competitions (50:34) An inside look at how the team operates (58:56) Working with Elon (01:05:30) Launching Birdwatch (01:10:48) The core principles behind Community Notes (01:26:15) Anonymity and pseudonymity in contributions (01:32:17) Sustaining the project through leadership changes (01:37:57) Future directions for Community Notes (01:42:12) Final thoughts and optimism for the future Referenced: • Community Notes on X: https://x.com/CommunityNotes • Sign up to be a Community Notes contributor: https://communitynotes.x.com/guide/en/contributing/signing-up • The Making of Community Notes: https://asteriskmag.com/issues/08/the-making-of-community-notes • “Readers added a Community Note to this Tweet”: https://x.com/HelpfulNotes/status/1718103364792205704 • Note-ranking algorithm: https://communitynotes.x.com/guide/en/under-the-hood/ranking-notes#matrix-factorization • Study: Community Notes on X could be key to curbing misinformation: https://giesbusiness.illinois.edu/news/2024/11/18/study--community-notes-on-x-could-be-key-to-curbing-misinformation • Study Finds X’s (Formerly Twitter’s) Community Notes Provide Accurate, Credible Answers to Vaccine Misinformation: https://qi.ucsd.edu/study-finds-xs-formerly-twitters-community-notes-provide-accurate-credible-answers-to-vaccine-misinformation/ • Did the Roll-Out of Community Notes Reduce Engagement with Misinformation on X/Twitter?: https://dl.acm.org/doi/10.1145/3686967 • Kayvon Beykpour on LinkedIn: https://www.linkedin.com/in/kayvz/ • Jack Dorsey on X: https://x.com/jack • “Birdwatch gives me the creeps” tweet: https://x.com/elonmusk/status/1589454464611540992 • Blake Scholl on LinkedIn: https://www.linkedin.com/in/blakescholl/ • Creating Truthtelling Incentives with the Bayesian Truth Serum: https://www.eecs.harvard.edu/cs286r/courses/fall12/papers/DW08.pdf • Asana: https://asana.com/ • Spaces: https://blog.x.com/en_us/topics/product/2021/spaces-is-here • Amazon MTurk: https://www.mturk.com/ • Community notes on GitHub: https://github.com/twitter/communitynotes • What do I think about Community Notes?: https://vitalik.eth.limo/general/2023/08/16/communitynotes.html • X’s community-led approach: tackling inaccurate and misleading information: https://blog.x.com/en_us/topics/company/2023/xs-community-led-approach-tackling-inaccurate-and-misleading-information • Messi-Ronaldo rivalry: https://en.wikipedia.org/wiki/Messi%E2%80%93Ronaldo_rivalry • Supernotes paper: https://arxiv.org/pdf/2411.06116v1 Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected]. Lenny may be an investor in the companies discussed.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lenny Rachitsky?

  The work that you guys do has had such a tremendous impact on the way the world works. I want to start with just giving people a brief understanding of what is Community Notes.

• 0:09Speaker 01

  Someone on X can see a post. If they think it's misleading, they can propose a note that they think other people might find informative. Other people can then rate that note.

• 0:17Speaker 02

  We actually look for agreement from people who have disagreed in the past. And what we see is when people actually have that sort of surprising agreement, that's what makes the notes so neutral and accurate and well-written really overall.

• 0:31Lenny Rachitsky?

  There's many people that are very polarized. How do you deal with people that are like super anti-vax, super Jan 6?

• 0:35Speaker 01

  One philosophical thing that's important is that we want all of humanity to participate. And sometimes people are surprised by that. We have all of humanity. We then have the data to understand what notes will be helpful to actual humanity. Every post is eligible for notes. We shouldn't exempt Elon. We shouldn't exempt government figures. We should, like, everyone, even advertisers can get notes.

• 0:58Speaker 02

  There have been external studies, you know, run by people totally independent of us, who have found that if you take a post with or without a community note, that actually people's agreement with the core claims in the post does change if they see it with the note versus without.

• 1:13Lenny Rachitsky?

  Is there anything else along the lines of just working for Elon within an org Elon runs that might surprise people?

• 1:18Speaker 01

  If I were to start a company, that company, it would be even leaner than I would have made it before. I've been amazed with just how much the team is able to accomplish with a small group and I think because of a small group.

• 1:33Lenny Rachitsky?

  Today, my guests are Keith Coleman, Product Lead for Community Notes, and Jay Baxter, founding ML engineer and researcher for Community Notes. This conversation may be my newest favorite podcast episode so far. Community Notes is one of the most impactful and clever and also underappreciated products in the world right now. If you ever use X slash Twitter and you see a note underneath a tweet correcting the misinformation in that tweet, that is Community Notes. I've never heard a deep dive into the story behind the product and the team that built it, and I'm excited to bring you just that. We get into the surprising origin story of the product, how the algorithm actually How the algorithm emerged out of an internal contest within Twitter, the principles behind community notes and why staying true to them has been so key to its

• 2:18Lenny Rachitsky?

  success. Also, how it survived four different leaders, including Elon and Jack, and why it's now a big part of the solution to solving misinformation on the internet, including recently being adopted by Meta as their main fact checking tool. This is an incredibly special episode and I'm so excited to bring it to you. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. Also, if you become a subscriber of my newsletter, you now get a year free of Notion and Superhuman and Granola and Linear and Perplexity Pro. Check that out at lennysnewsletter.com. With that, I bring you Keith Coleman and Jay Baxter.

• 2:56Lenny Rachitsky?

  This episode is brought to you by WorkOS. If you're building a SaaS app, at some point, your customers will start asking for enterprise features like SAML authentication and SCIM provisioning. That's where WorkOS comes in, making it fast and painless to add enterprise features to your app. Their APIs are easy to understand so that you can ship quickly and get back to building other features. Today, hundreds of companies are already powered by WorkOS, including ones you probably know, like Vercel, Webflow and Loom. WorkOS also recently acquired Warrant, the fine-grained authorization service. Warrant's product is based on a groundbreaking authorization system called Zanzibar,

• 3:38Lenny Rachitsky?

  which was originally designed for Google to power Google Docs and YouTube. This enables fast authorization checks at enormous scale while maintaining a flexible model that can be adapted to even the most complex use cases. If you're currently looking to build role-based access control or other enterprise features like single sign-on, SCIM or user management, you should consider WorkOS. It's a drop-in replacement for Auth0 and supports up to 1 million monthly active users for free. Check it out at workos.com to learn more. That's workos.com.

• 4:15Lenny Rachitsky?

  This episode is brought to you by Product Board, the leading product management platform for the enterprise. For over 10 years, Product Board has helped customer-centric organizations like Zoom, Salesforce, and Autodesk build the right products faster. And as an end-to-end platform, Product Board seamlessly supports all stages of the product development lifecycle, from gathering customer insights, to planning a roadmap, to aligning stakeholders, to earning customer buy-in, all with a single source of truth. And now, product leaders can get even more visibility into customer needs with Product Board Pulse, a new voice-of-customer solution. Built-in intelligence helps you analyze trends across all of your feedback and then dive deeper by asking AI your follow-up questions.

• 4:57Lenny Rachitsky?

  See how Product Board can help your team deliver higher-impact products that solve real customer needs and advance your business goals. For a special offer and free 15-day trial, visit productboard.com/lenny. That's productboard.com/lenny.

• 5:19Lenny Rachitsky?

  Keith and Jay, thank you so much for being here and welcome to the podcast.

• 5:22Speaker 01

  It's great to be here.

• 5:23Lenny Rachitsky?

  Thanks, Len. Thanks for having us on. It's so my pleasure. I'm so thrilled to be having this conversation. The work that you guys do has had such a tremendous impact on the way the world works. So many product teams are always talking about driving impact, I want to drive impact. Like, you guys have actually built things that have changed the world in meaningful ways and continue to do that. And I've never really heard the backstory of how Community Notes came to be and how it works and all these things. So I'm really appreciative of you guys making time to chat.

• 5:52Speaker 01

  Yeah, first, thanks for saying that. That's why we built this thing, is to help people. And it's great to hear, it's great to see people enjoying it and finding it useful.

• 6:02Lenny Rachitsky?

  I want to start with just giving people a brief understanding of what is Community Notes. I think a lot of people may have kind of heard about it, kind of maybe see it on X as they scroll through, they see these notes, but they're like, I don't actually know what this is. So can you just kind of briefly describe what is Community Notes?

• 6:18Speaker 01

  Community Notes is a way for the people, like the public, to add context to posts that might be misleading. The basic way it works is that someone on X can see a post. If they think it's misleading, they can propose a note that they think other people might find informative. Other people can then rate that note. And if the note is found helpful by people who normally disagree with each other, indicating that it's probably accurate, it's probably really neutrally worded, it's probably informative, then it will show to everyone on X. And the goal is just to get people more information about what they're seeing so they can make better decisions in their lives.

• 6:56Lenny Rachitsky?

  Amazing. And I think, like, hearing this, it's like absurd that this works. I think when people originally heard this idea, like, no way this is gonna work. And so just to dive a little bit deeper, can you give us a sense, a deeper understanding of how it actually works? Because I think it's the algorithm that you guys designed that is so clever that allowed this to work. So talk a little bit about that algorithm.

• 7:22Speaker 02

  Yeah, so I think a key misunderstanding a lot of people have, if they haven't really dived into details, is they kind of just think that maybe someone can write a note and it appears immediately, or we're just taking a majority rules vote of who thinks the note's good. I think both of those approaches would probably lead to biased or inaccurate notes. I think the key thing really that we do is we actually look for agreement from people who have disagreed in the past. And what we see is when people actually have that sort of surprising agreement, that's what makes the notes so neutral and accurate and well-written really overall. It's just that people who are very polarized overall often can't find agreement when things aren't accurate, right?

• 8:07Speaker 02

  I think it also provides some good anti-manipulation properties. I think people are often, if you said, I think like back in 2020, before we started building anything here, whether this could work at all, I think a room of ML engineers would say, oh, you have to keep it closed source. You know, people are gonna be manipulating this all the time. You have to use ground-truth labels from fact-checkers. There's no way that you could like bootstrap the system without external labels. But it turns out that you can do that with this kind of bridging-based agreement algorithm is what we call it.

• 8:40Lenny Rachitsky?

  Okay, so just to summarize and make it super clear, it's basically people someone writes a note, this information is false. What's like a good example, just as we talk about this, like a classic example?

• 8:49Speaker 01

  A really, really classic example is an AI-generated image, or an out-of-context image, like, "Look what's happening here", but it's actually from like five years ago, in a different country, on a different topic.

• 9:00Lenny Rachitsky?

  Oh man, I've seen those so many times, where it's like, "Look what's happening in San Francisco", and like, "No, this is a whole different city and that's not right."

• 9:06Speaker 01

  Totally.

• 9:06Lenny Rachitsky?

  Yeah, okay, okay, so someone posts this AI image, someone writes a note, and this is actually five years ago in a different city, and this algorithm helps understand if this is a real, if this is true, this note is true, and it's just people, regular people, doing this.

• 9:23Speaker 02

  Yep, yep, regular people who have signed up to be community notes contributors. So, you know, there are a few checks, like, you do have to have a verified phone number, for instance. But yeah, at the end of the day, these are regular people, not necessarily professional fact-checkers or anything like that.

• 9:39Speaker 01

  And yeah, that was like, that was really important to us, too. Like, there was a question at the beginning, to the point Jay was making of like, "Well, did anyone think this was gonna work? Obviously, it was kind of a crazy idea." We didn't know if regular people were gonna be able to do this task, and certainly, you know, people had concerns about whether they would do it effectively. Initially, some people inside the company were suggesting, like, "Hey, why don't you have journalists", or, you know, some, some select group be the first participants. But very specifically, we were like, "No, that's like, we're trying to move away from the idea of curated editorial decisions being made around this. This is supposed to be open to everyone." So it is very, we very intentionally try to allow all humans in.

• 10:20Speaker 01

  There are, people are randomly selected, and that's important to it, you know, feeling fair, feeling open, feeling trustable.

• 10:27Lenny Rachitsky?

  Yeah, and again, it's just like, this sounds like the Holy Grail of understanding what is true, and it actually works, and works so well that Meta recently, as you all know, decided to adopt this exact system for them, and instead of having tens of thousands of fact-checkers reviewing things.

• 10:46Speaker 02

  One distinction that I would make, which maybe can come off as nitpicky, but I think is important is community notes adds additional context. It's not fact checking necessarily, right? So, so there are cases where the post could be true, but maybe it's just misleading because there's, there's no context or there's missing context. And, you know, we cover those cases and I think that's, that's kind of an important distinction. We also, we just have the philosophy that users should be able to make up their own minds. Right? Like here, here's the, here's extra context. Take it or leave it. Right?

• 11:17Lenny Rachitsky?

  Yeah. What I think about it, you shared this with me, this example of a, a picture with a, with a, with a cat and somebody's community note was just, that's a dog. Or is it the other way around? Or that's a cat?

• 11:29Speaker 02

  Yeah. Palestinian boy shares his bread with a dog was the post. And it's a picture of this cat, right? So like, obviously this particular note is not super necessary because it just says that's a cat and links to Wikipedia for cat. It's kind of a good example that, that like the system is, this is not something of a professional fact checker, whatever, right? Or I think would need fact checking, but it's proof that the system is really run by the users at the end of the day and add some comment accrually, if I guess. And, you know, that's, the note is correct.

• 12:06Lenny Rachitsky?

  And I could, you know, it's important. When does a post get triggered to even be considered for a community note? Is there like a threshold or is it just you can write a community note on anything? And people decide what they want to vote on? Has that worked?

• 12:18Speaker 01

  So every post is eligible for notes. It's, and that was again, another really important principle. It's like it, we shouldn't exempt Elon. We shouldn't exempt government figures. We should, like everyone, even advertisers can get notes. So any posts on the platform can get a note. And if you look in practice, you'll see notes appearing on world leaders, on Elon, on ads, on media organizations and on obviously like just regular people using social media. But yeah, the idea is really that it's an even playing field. For a note to be proposed, the person proposing it has to have earned the ability to write notes. So there is that aspect where you have to earn in to be able to do

• 13:01Speaker 01

  this. And the way you earn that ability is through your ratings by demonstrating the ability to help identify notes that are found helpful to a broad range of people. So basically, if you have an ability to see and know, recognize what's helpful to a lot of people, then you have the ability to start proposing notes.

• 13:20Lenny Rachitsky?

  I actually sign up to be, what do you call these people?

• 13:24Speaker 01

  Contributors.

• 13:25Lenny Rachitsky?

  Okay, contributors. Yes, I've been rating. I haven't achieved, can't write notes yet.

• 13:30Speaker 01

  Yeah, it's not super easy. It takes some effort.

• 13:33Lenny Rachitsky?

  Are there stats you can share about the scale of community notes at this point, especially things that might surprise people?

• 13:38Speaker 01

  Yeah. I mean, the service is growing rapidly. So there are hundreds of notes per day. And to put that into context, I saw some stats recently from someone at UC Berkeley saying there were something like 10 fact checks, traditional fact checks a day. So in contrast, there's hundreds of notes a day that are getting shown. They span a huge range of topics from obviously politics, news, out to entertainment, sports, gaming, just whatever's going on that day. In addition to there being hundreds of these individual notes, they can also be matched to multiple posts. So if someone writes a note on an image or a video, like let's say it's AI-generated or something like that,

• 14:20Speaker 01

  that note will automatically be matched to all posts that contain the same image. So you can have a single note matching to thousands of posts. And over, let's say, the last year, 2024, we had something like 95,000 notes that were seen about 30 billion times. That's more than double the prior year. The prior year was something like 37K notes seen 14 billion times. So that rate is increasing dramatically. If you think about like 30 billion views, that's a lot of information that is getting out there that might not have been out there otherwise, which is pretty cool. And the part of the reason it is expanding

• 14:59Speaker 01

  like that is the contributor base is expanding. There's something like 950,000 contributors around the world. That's nearing a million people making this happen, which is amazing.

• 15:13Lenny Rachitsky?

  And I'm one of those, right? Like I count as a contributor.

• 15:15Speaker 01

  Yeah.

• 15:16Lenny Rachitsky?

  Okay.

• 15:16Speaker 01

  You're signed up as a contributor, you count.

• 15:18Speaker 02

  Then there's more people on the waitlist, too. So there's plenty of headroom for more growth. Regarding the matching on media and URLs, I think that's a huge way to get extra coverage. Also, I do think we've been very careful to make sure that those matches are precise. Because I think one thing that people love about community notes compared to other types of fact checking is that actually the notes are custom written for the particular claim you're seeing, right? So so often, a fact check warning would just say something like, you know, get the facts here. And then there's a link to some generic page about voting, like information, which is, you know, so not helpful to have the information behind a click.

• 16:03Speaker 02

  So pulling the context up, you know, so that you have zero clicks that you need to make and keeping it specific is so important.

• 16:10Lenny Rachitsky?

  One feature I love that I imagine you guys thought deeply about is if I like the post in the past, I get notified later if a community note shows up so that I'm not like remembering this false information.

• 16:21Speaker 01

  Yeah, I mean, we try to make notes as fast as we can. So we want them to appear instantly if possible. But inevitably, there's going to be a time gap between when a post goes live and when people figure out what's going on and when they get the note out there. And so we send those notifications to try to close that gap.

• 16:39Speaker 01

  And yeah, we get a lot of love for that. We see people take screenshots and share them. They're excited about it. And it's also a pretty cool example of something you can do on the internet in the social media world that was difficult in kind of like a print or standard news world, where you would see maybe a correction like the next day in a corner of a paper, but it was hard to read. Here, you're getting a ping about it if you've engaged with the post and note shows up.

• 17:05Lenny Rachitsky?

  One user feedback point is I'd love the push to just tell me, here's what you got wrong. Because I find that I actually have to go into it and like read it. And I feel like the push could just be like, here's information, here's more context to this thing.

• 17:19Speaker 01

  We'll go take a look at that.

• 17:22Lenny Rachitsky?

  There you go. Live user feedback.

• 17:24Speaker 01

  Nice.

• 17:25Lenny Rachitsky?

  Okay, I want to get into the origin story of this whole thing, but two more questions because we're on this thread. One is, what's the kind of the threshold for a note to show up on a note? Is that information you can share? Just how does that work?

• 17:35Speaker 02

  So just because of the details of the way the algorithm works, it uses this machine learning algorithm, you know, called matrix factorization, where, you know, we fit it with gradient descent and whatnot. The threshold is, you know, it's 0.4 on this, you know, made up scale.

• 17:51Lenny Rachitsky?

  0.4, great.

• 17:53Speaker 02

  I mean, in practice, what it means is, you know, basically a majority of people, if there is a polarized divide relevant to the notes, you know, obviously some notes are not about politics or something polarizing. But if there is, then a majority, a sizable majority of people on both sides would generally need to find the note helpful. And then there are these, there are other rules that come into play beyond that main one. So, you know, even if it's above that threshold, it might get filtered out. If there's a separate algorithm that's looking at agreement between people's incorrect tags. So like, maybe, maybe people found the note helpful, but incorrect, right? Like, it happens. And in those cases, it doesn't matter if it's above the helpfulness threshold.

• 18:37Lenny Rachitsky?

  So is this 0.4 is, that's probably the wrong way to think about it, but is it 40% of people that normally disagree, agree?

• 18:44Speaker 02

  It means nothing like that. It's just like on some arbitrary scale. Yeah.

• 18:49Speaker 01

  Yeah, if we change random other things about the algorithm, that number would also have to change to an equally seemingly arbitrary number. That we, we arrived at, at some, like some numbers like that by gauging user feedback. So we, we could share a lot of notes with people, get feedback on which ones are helpful. And there was sort of just a line emerged about indicating where, you know, where the, where things go from like questionable to pretty clearly helpful.

• 19:13Speaker 02

  Yeah. And, and it is set right now, by the way, to be really conservative, I think. We, we just are pretty particular about quality and we really want no quality to be really high. I think, I think Keith and I both believe that we live or die based on the quality of the notes at the end of the day. So, so we'd rather not show a note that may be good, but we didn't have enough signal on than the other way around.

• 19:41Lenny Rachitsky?

  That makes so much sense. Like, I've never seen a community to note that is wrong and breaking that. Promise is a big deal, so I completely get why you guys are super conservative there. Okay, two more questions along these, because I'm just curious. These were not on my list of questions to ask, but I feel like people wonder this. How many notes are written versus end up showing up and triggering on it?

• 20:02Speaker 01

  We probably show about 8% of notes that get proposed. I think that's, it's been between, let's say, 7% and 10% or 11%, something like that over time. The number can vary a little bit. And as Jay said, there are undoubtedly, and you can see it, there's clearly more good notes than we show. But the goal is to hold a really high bar. Like, we want to show a note when it's going to be helpful, when it's not going to appear, you know, biased and undermine trust in the system. Like, we want these to be neutral and informative, helpful. And, you know, as Jay was saying, like, we view the worst possible mistake as showing a bad note. Because that's going to undermine trust, and the trust is why people like the product.

• 20:45Speaker 01

  So, so yeah, we, the bar is there. And, you know, like I said, there's, there's clearly some, some in that remaining, let's call it 90% that are good. And then there's a lot that are just like not that great, and there's some that are bad. And if you write one of these ones that are bad, which bad being defined as people who normally disagree, find the note not helpful. So it's like the inverse of the ones we show. If you write one that people, you know, people normally disagree find not helpful, you actually will ultimately lose your ability to write and have to earn it back. So that, that range, that other 90% is a mix. Sometimes people look at the number, they're like, oh, why don't you show more? It's like, well, you probably actually don't really want us showing most of those. It's the, the, the goal here is that the system is able to filter out the good ones.

• 21:31Lenny Rachitsky?

  That makes sense. Okay, one other question is, there's many people that are very polarized, like very disagreeable with a lot of things. And how do they filter into this algorithm? How do you deal with people that are like super anti-vax, super Jan 6, like all these very extreme potential views?

• 21:47Speaker 02

  If people really are so polarized, that there, there isn't agreement among people that typically disagree. You know, it's possible that this is one of those notes that might be correct, but, but just wouldn't be useful contact, it wouldn't be, you know, helpful to show as, as context. Maybe, maybe it's about a claim that people have. You know, a really entrenched opinions about, and they've read hundreds of things about it already. Or right, like probably, probably this is just not going to improve people's understanding. It's just not going to be a helpful user experience. So it might not be the worst thing in those cases to not show the note. People a few years ago were pretty pessimistic that maybe fact checking never changes people's,

• 22:32Speaker 02

  you know, understandings about what's true. Actually, there have been external studies, you know, run by people totally independent of us, who have found that if you take a community note or a post with or without a community note, that actually people's agreement with the core claims in the post does change if they see it with the note versus without. So we are having an impact on this thing that people previously thought was maybe not so easy to do. And so it's nice to focus on the cases where there is the bridging agreement. I would also say there is this reputation component to the algorithm as well. So if you consistently rate notes in a way that is counter to the bridging base consensus,

• 23:15Speaker 02

  then we'll stop counting your ratings, right? So, you know, if you're the kind of person who constantly rates bad notes as helpful, we do filter you out. So, so there's a difference between those types of people versus just the good but polarized ones.

• 23:30Speaker 01

  Yeah, I think one philosophical thing that's important is that we want all of humanity to participate. And sometimes people are surprised by that. They'll be like, oh, aren't there people who are like, you know, shouldn't be doing this? Or like there's, you know, I don't, their, their thinking is so extreme or something, maybe they shouldn't participate. But our view is it's actually, we want to have all of humanity here because if we have all of humanity, we can, we then have the data to understand what notes will be helpful to actual humanity. You know, we can, we can better model that, better, better understand and better show those notes. So it's advantageous to have people who have all sorts of points of views. And we don't expect that every note will be loved by every single person.

• 24:15Speaker 01

  You know, that's kind of an impossible bar, but. We do intend to show the notes that like 80% of people are gonna, you know, read and say, wow, I'm glad I knew that. And so, you know, in that sense, it doesn't matter how, you know, maybe extreme someone views, person's views as it's still great to have them in the program. So, you know, no matter what your views are, please sign up and participate it. It helps identify what's really helpful.

• 24:39Lenny Rachitsky?

  Cool. And we'll link to people if they want to actually sign up so they know how to do this. Something we didn't actually specify, these are all volunteers. Now I'm getting paid to be doing these notes and voting.

• 24:48Speaker 01

  Right. Yeah, it's totally based on intrinsic motivation and, and we think that's a great reason to be doing it. When you talk to the most active contributors, a lot of them, they just, they want to have better information out in the world and that's a great motivation. So, yeah, that's why they, and you know, if you, if you think about like for these people, the impact they can have is kind of nuts. So, uh, when we first launched US wide, this was like in 2022, a note appeared on a White House tweet and the White House deleted the tweet and reissued an updated statement. And like, like imagine being the person who wrote that. You probably have like 12 followers,

• 25:29Speaker 01

  your, your posts probably get, you know, a couple likes. And here you just put a, put a note on the White House and they changed their public talking points based on what you did. Like that is an incredible amount of impact. So, you know, it, you could see why people are motivated to do it when they care about what's going on in the world. It, um, you know, you don't have to be a big, well-known person to shape the discourse and information flow in a, in a way that's helpful.

• 25:59Lenny Rachitsky?

  It's insane. Like, there's so much to love about this. One is just the meritocracy of this whole operation of just anybody can, that is true and correct can participate and have impact. Also just shows you how much information we get that is just wrong. Like, we had no idea how often we see things that are wrong and now we do.

• 26:17Speaker 01

  Working on this product has made me realize just how many things I used to trust kind of by default that now I look at more skeptic, skeptically.

• 26:26Lenny Rachitsky?

  Definitely meme these days. Okay, before we get to the origin story, is there anything else along these lines you guys think might be really important to share? Really interesting?

• 26:36Speaker 02

  Sure, I guess one other thing just is that although we don't actually use the fact that a post was noted in the core ranking algorithm, which you know, we think is a nice property. There is a really big impact, just organically, meaning not from the algorithm, but just from user behavior, where people will like and reshare or you know, quote, posts way less when when notes are applied. So just, I don't know, for people out there who typically run A/B tests on big, you know, platforms, you may already be familiar with this, but like 1% is typically an awesome effect size for any sort of algorithm change.

• 27:18Speaker 02

  We saw more like 30 to 40% engagement rate drops for likes and reposts. And an A/B test we ran when comparing showing a post with or without a note, which is just crazy big. And then, and then if you actually look, that's, that's just an A/B test on the engagement rate. So that's not the network effect. If you capture the overall network effect of how a post, you know, is spread less by that person's repost. Basically, if you look top line with a difference in differences approach, different, multiple different external research groups have both found consistently that there's like a 50 or 60% drop in total reposts, which is just nuts after a note is applied.

• 28:01Speaker 02

  So it's having a really big impact on spread actually too.

• 28:05Lenny Rachitsky?

  That's so like, that's so great to hear. It's what I would want to see and it's incredible impact. Basically like an AI image of something false would just go crazy on Twitter and did before community notes came out. And now what you're saying is just adding that context, not actually like you're saying, the algorithm doesn't demote it. If there's something incorrect, it's just people are like, okay, this is false. Why would I want to retweet this? That makes sense. Correct. Yeah, the notes just totally take the wind out of these stories. So like the thing will be going viral, note appears, resharing drops 50 to 60%. And like, that's it. Like it just, you can, at 50 to 60% per generation, the virality quickly goes to zero.

• 28:45Speaker 02

  And by the way, there's, I have very mixed feelings about this next one. But authors become 80% more likely to decrease, or sorry, to delete their post after they get noted. Which, okay, that's great, because like, less, less misinformation out there. But I'm pained about because those are usually the best notes. Like, if the note was so just good, that you had nothing, you had no other option but to delete your post, those notes don't get seen by other people, right? Because

• 29:12Lenny Rachitsky?

  That is hard.

• 29:13Speaker 02

  There's an, there's an argument, by the way, that like, seeing, that just because you might see the same misleading claim elsewhere, off X or somewhere else on X, you know, it might be good to actually show, better to have seen the post with the note than not see it at all.

• 29:28Lenny Rachitsky?

  Yeah.

• 29:29Speaker 02

  Unsure about that claim.

• 29:31Lenny Rachitsky?

  That is so interesting.

• 29:32Speaker 02

  Yeah.

• 29:33Lenny Rachitsky?

  Yeah, I could, I'd be so sad if I was that community or note writer and just, oh, man, it's so good. They just can't even keep the post up. Okay. So, coming back from today's world where you're, this like, small amount of code is changing the way people understand the world and what they believe, and making the White House rescind their announcements. Zooming back to the beginning of how this whole project started, what I heard just briefly is, Keith, you were just kind of tired of managing PMs. You wanted to just work on something yourself. You wanted to work on something impactful, away from corporate BS. And you basically just started looking for something that was impactful, important, and you found this. Talk about just how it all came to be at the beginnings of the story.

• 30:18Speaker 01

  Yeah, so, I mean, for me, the beginnings actually go back to why I joined, it was then Twitter, in 2016. I was, I had a startup. And we were, we'd had some acquisition offers and one of them was from company Twitter. And it was 2016, it was the middle of the election between Donald Trump and Hillary Clinton. And there were like something like three televised debates, but every day there was a debate happening on Twitter. And it was very clear, like, this is where people are talking about these things that matter, where, where, you know, information is being shared, where their, where ideas are being formed.

• 30:59Speaker 01

  And as a user, it was obvious that I could get good information there. But it was also obvious that there was kind of questionable information floating around. And I remember just looking as an outsider, thinking like, wow, like, this is a really hard problem and it also seems really important. So we ended up going to Twitter and, and the company was in a turnaround at that point. So like my first three years was just helping to get the company growing again. You know, working on, on everything that was the consumer product. You know, getting user growth, going back, getting people wanting to work there again, et cetera. But a few years in, I was reflecting on what we had done. You know, I think we had done a lot of good work getting momentum going.

• 31:44Speaker 01

  But it, and people in the, us and in the industry had tried things to kind of deal with, with misleading information. But nothing was really working. Like, it was obvious nothing was working. Nothing could handle the scale of the problem, nothing could handle the speed. And a lot of people just didn't trust the existing approaches. Like this, the existing approaches were either fact checkers or internal trust and safety teams making decisions about what was or was not misleading. And like, a lot of people just didn't want or trust that to be the way this was decided, which is very reasonable. And so, you know, I'm looking at that, I was, I was still managing a large PM team.

• 32:25Speaker 01

  You know, that's a whole story in itself. I, um, I felt like I would, that job required a lot of energy in and I, and I didn't feel like I always saw the output that I wanted to see from it. Like, I didn't see the change in the product I wanted to see. And, you know, I was contemplating, should I go start a company? Uh, should I do something else? And I kept coming back to this problem like, man, like the, how is the world going to deal with the, with this information quality issue of like what we get on social media or wherever we get it? And like, you know, I'm at, I'm at this company where you can make a difference on this problem. Like, why not go and try some crazy ideas and see if like, one of them might work?

• 33:08Speaker 01

  And so I came back, I had, I had a kid, I came back from paternity leave. I went to my boss, Kayvon, I was like, hey Kayvon, how about I just stop doing my job? And I go work on this instead, you know, this being try some crazy ideas to see if we can deal with this, with misleading info. He was stoked and so I went off and started working on that. You know, it started with just reading any research I could on the, the problem and existing solutions. What was or was not working or were the issues? And then into prototyping and then, you know, it ultimately led to us building and piloting this idea that became Community Notes.

• 33:46Lenny Rachitsky?

  Amazing. Okay. I have so many questions and we're going to keep going through the story. But when you joined Twitter, what was kind of the, and it was called Twitter at this point, I'm going to try to call it X now, which I know is important to your boss. What era of Twitter was it at that point? Like it was Kayvon run, uh, joined or, and who was the CEO? Cause there's been many.

• 34:05Speaker 01

  So, okay, yeah, I started, I came in December 2016. So Jack had relatively recently come back as CEO to turn the company around. And just to give you a sense of like the state of the company, um, something like a third of employees were leaving every year. So just imagine that like a, a third of your team gone every year. Um, you know, the stock was in the toilet, um, the product was not really growing. Um, and so, uh, Jack was working on a turnaround. And Kayvon was there already, Kayvon was running Periscope and a bunch of video stuff. And, you know, that, that group continued to, you know, Jack, Jack was there up through the start of the, the Community Notes, then Birdwatch Project. And, uh, yeah.

• 34:50Lenny Rachitsky?

  And it was called Birdwatch. I don't think we've used that term yet, but that's an important point. It was called Birdwatch initially.

• 34:55Speaker 01

  Yeah, so it was originally called Birdwatch when when we started the project, but obviously somewhat famously, the name changed along the way.

• 35:05Lenny Rachitsky?

  Yeah, maybe let's just tell that story real quick. And I know we're zooming in for but just I have this Twitter thread that I saw between Jack and Elon when they're debating what to call it. And Elon's like, Birdwatch sounds creepy. I want to change it. Is there anything there you can share?

• 35:19Speaker 01

  Yeah, the story there, the story that's kind of funny. Elon Elon came in, acquired the company and, you know, we had just launched the product relatively recently US. It'd been in pilot for a year, but we had just made it available US wide. And he, I guess he'd been seeing the notes and he says, soon after, soon after the exhibition, he DM me and he was like, Hey, this community notes thing is awesome. And I was like, Oh, I'm glad you like it. Like, let's, you know, talk. And so we talked the next day and he kept referring to it as this community notes thing. And I was like, You know, it's interesting that you keep calling that, calling it that, because that's actually the very first thing that

• 35:59Speaker 01

  I called it. Like, the very first Figma mockup I made depicting this thing was called community notes. It just, I don't know why it just felt really natural. And so that's what we had, that's the first prototype we had tested. You know, later, the project changed, changed its name to Birdwatch. But, you know, Elon was like, Hey, let's just call it that. And so the next day, we just changed the name. And, you know, it was, it's always, you're notable for the team when you change your name. But really, the team was excited about it. I think it is a much more understandable name. Jack has made fun of it, calling it like the ultimate Facebook name or something like that. But

• 36:41Speaker 02

  The most boring Facebook name.

• 36:43Speaker 01

  Boring, right? Name, which is funny because they're now, you know, launching community notes. But, but I think it is a very understandable intuitive name. And I think it has served the product really well. There's, there's a reason it was the name in the very first mockup.

• 36:57Lenny Rachitsky?

  Yeah, I think descriptive names, it just makes sense. This connection with Elon, and I want to talk later about just how you've dealt with so many strong personalities over and kept this alive throughout so many changes. But before we get to that, you, you did something that I think a lot of product leaders, engine leaders, just people that have managed people dream of. Give up all this power in air quotes and career trajectory and influence and just like, forget all that. I'm going to go back to just building something awesome, small team. Is there any advice there that you could share from that experience that you think might be helpful for other leaders to share or to hear to help them maybe do that same jump? Because that's really difficult in practice, easy to talk about, hard to do.

• 37:42Speaker 01

  Yeah, I think it is a difficult jump. I've done it a bunch of times in my career, and I've always been very happy with it, where I started with a small team that it kind of grew into something bigger. And then I was like, you know, this is like, we're kind of dealing with a lot of big production stuff, teams really big. I want to go back to doing something like crazy and new with a small team again. And so I've kind of done that like sawtooth leap a bunch of times. But it can be hard because certainly the natural, like the classic career path is sort of, I don't know, rewards or, you know, running a large organization or being a manager, things like that. But I think at the end of the day, you got to work on stuff you love.

• 38:24Speaker 01

  You got to be having fun and I think people want to be having impact. And I think there's one myth that that can get in people's ways. The idea that the more people you manage or something, or the larger your scope is, the more impact you have. I definitely do not think that is true.

• 38:41Speaker 01

  If you just look at, I mean, look at Community Notes, for example. If I had stayed running a large consumer PM team, like what would I have produced? Like 16 more pages of OKRs? Like, I don't know, you know, a bunch of documents? And I think building Community Notes has had way bigger impact on the world. It's become the industry standard for how to deal with this now, which is super cool. People love it. It's the first thing that is plausibly dealing with the internet scale, you know, issue of information quality.

• 39:12Speaker 01

  You know, I, I think it's unquestionably a bigger impact than I would have had if I were just do whatever, doing some standard management track thing like I was doing before. And I think that's true of so many other, uh, you know, small companies and startups. I, I was just reading, someone screenshot, I think it's Blake Scholl, uh, LinkedIn the other day. He went from like director of coupons or something to building the first, uh, supersonic.

• 39:37Lenny Rachitsky?

  Oh yeah, from Groupon. Yeah.

• 39:39Speaker 01

  Yeah. And I, you know, those stories are everywhere when you look. And so I definitely have found that, you know, for me, I love building hands on. I love trying crazy new ideas. I, I love the zero to one experience. It's fun to scale things up too, and it can be fun to operate it, you know, at scale, but, um, doesn't, you, you know, this team is a good example of one that operates at a very large scale, but that is still very small.

• 40:03Lenny Rachitsky?

  Yeah. I think the way you guys operate is what more and more companies are trying to do. Uh, remove middle management layers, create small teams that just execute and build impact. And just like ICs, uh, whenever I say IC, I have a comment on YouTube where like, what is IC? So I'm just going to explain individual contributor, non-manager is when I say the word IC. So let me follow this thread. And when I asked people about how you set up the team to operate effectively and, and protect it initially, there's this term thermal that came up a lot. It was like a thermal team, if that's how you describe it.

• 40:37Speaker 01

  Yeah.

• 40:38Lenny Rachitsky?

  What is thermal?

• 40:39Speaker 01

  Yeah. So, um, anyone who's worked in a larger company probably knows that things can get kind of bureaucratic or bogged down. Decision making can be slow. Like there's these large planning cycles. People can like, try to like, take someone from one team, move them to another, like at random arbitrary times that can disrupt a project, like all sorts of things like that.

• 40:58Speaker 01

  Uh, you know, our company, this is a number of years ago when we started this project. Um, we had a lot of founders in the company, like Kavon is an example of founder who was helping to run the company. And he had this idea like, hey, why don't we create this program, call it thermal, where we could have teams that were somewhat isolated from that. They could run on through their own process. They would have like one clear owner. The team would be entirely dedicated to that project. And we would just sort of like repeatedly make funding decisions as to whether to continue the effort. And so

• 41:31Lenny Rachitsky?

  Why was it called Thermal, by the way? What was the idea there?

• 41:33Speaker 01

  I think the, I think it was like an, an old bird analogy of like Thermals lifting, you know, the, the birds on their wings. Twitter, obvious, 3.0 obviously had a lot of bird analogies, bless its heart.

• 41:46Speaker 01

  And so, you know, that was one of them. But the, you know, the idea, I loved the idea as someone who, you know, liked the startup environment. And so when we were starting this project, I was like, hey, Kavon, like, why don't we make this the first Thermal project? And he was like, yeah, let's do it. And so, so we started with that way of operating and it gave us, you know, from day one, a lot of freedom and autonomy that I think was really important to make the product work.

• 42:14Lenny Rachitsky?

  So just be very specific about what, what, what makes it a Thermal project? How do you set that up? And this is asking from perspective of a company wants to build their own something like this. What does that look like?

• 42:24Speaker 01

  Yeah, I think there's a bunch of key attributes. So one, one key attribute is there's one clear driver of the project who's effectively like founder. I guess, I mean, maybe you could have two or something, but like, it's like really clear. There's like driver of the project and also there's one clear decision maker that they go to.

• 42:43Lenny Rachitsky?

  Oh, outside of the team.

• 42:44Speaker 01

  Outside of the team. And that was true back when we started and it is true now. Like if we need, need something or have a question on something, I talk to Elon. And it was, you know, it was like that from the beginning. It's like that now. And I think that's a big reason we're able to make decisions effectively, quickly, in a simple way.

• 43:02Lenny Rachitsky?

  And it probably has to be someone very senior, not

• 43:05Speaker 01

  Yes, it needs to be someone, someone senior who can make the decisions you need made, whatever they are. So I think that's really important, that clear decision-making structure. Another was 100% focus. So everyone on the project is expected to be totally focused on it.

• 43:22Speaker 01

  That at a lot of companies, it can be easy to have people's attention sort of spread across a bunch of things. And it makes it hard to get stuff done. Like you'll go to like, you'll talk to whoever that person is, you'll ask them for help on something and they'll be like, yeah, I'll help you. I gotta finish this thing, you know, and it'll take me like a week or two and then I'll get to it. And like a week or two delay totally changes the momentum of a project. When, you know, we were 100% focused. Yeah, we talk, we talk in the morning. It's like, hey, Jay, why don't we like try this thing on the algorithm? He's like, yeah. Then like the, you know, that afternoon or the next day, we're looking at results. And so because of that total focus, the rate of iteration goes way up.

• 44:03Speaker 01

  And then, you know, beyond that, there was also just the ability to use whatever our own sort of like decision-making process was. We didn't need to write OKRs or, you know, follow others like standard practices. Obviously, like, we had to make sure we were responsibly building the product and everything. But we didn't use, need to use standard, the standard practices. And I think that's another good example, like OKRs, I understand why they can be helpful. But they can also be, you know, not necessarily the right cadence at which to set goals. Like, I don't, I think it's really unclear that quarterly or annual goals are actually like the right pace.

• 44:42Speaker 01

  Like, we would set our goals for, like, we would set the goal for the next milestone that mattered, and we would work on that. And when we reached that milestone, we would have an idea of what was coming after. And then we, after we hit that, we'd set the next milestone, whether that was two weeks, a month, three months, like whatever it was. Like, we set our own pace and goals at that pace. And that just, I think, is a lot more natural for the development of something.

• 45:06Speaker 02

  The, the whole OKR determination and planning process took longer than it would take us to pick a goal and then execute on it and finish it.

• 45:15Lenny Rachitsky?

  How big was the team early on that you set up? How many engineers?

• 45:19Speaker 01

  It started with just me, and then we, when we decided to build the thing, it, we, we figured we needed about five. And we wanted to be as small as we possibly could. It was clear we needed someone on ML doing scoring. It was clear we needed someone to do some client engineering work, someone to do backend engineering work.

• 45:40Speaker 01

  There may have been like, you know, one or two other, oh, we needed a designer and a researcher to help us understand the customer base. And make sure we were, we were building the thing in a way that was actually going to resonate with people. And so I think that was, I think it was like backend, frontend. ML Design Research, that was the original team from what I remember.

• 46:01Lenny Rachitsky?

  Amazing. So basically, one of each function. A question I have for Jay actually is, there's all this talk of small teams and moving fast, but you know, sometimes you just need more engineers to build the thing. Is there anything you've learned about just how to keep a team small while moving as fast as you are and not need, we need to hire more engineers, we need to hire more engineers?

• 46:22Speaker 02

  I think in the beginning, when we were iterating, you know, what should even the requirements be? It was definitely good to just have it, you know, like one ML engineer. But I think at some point, we got clear on what the goals of the algorithm should really be. And we tried, you know, we were, I think at the very beginning, it wasn't clear that we needed to build this bridging-based algorithm, right? The actual first algorithm that I put into production was very focused on anti-manipulation. It was this kind of page rank variant, but it didn't solve the problem of, you know, bias, basically. So if there are some, if there are more users on one side, a page rank type graph algorithm can actually amplify those biases.

• 47:05Speaker 02

  So I think, you know, after building that prototype and getting data from that, it was clear that, you know, the bridging-based algorithm was going to be the way that we needed to solve it. And at that point, basically, I set up a bake-off, basically made, you know, this, this kind of like the, you know, a Kaggle competition or something. So that was like the key time where it was really important to pull in other engineers.

• 47:34Lenny Rachitsky?

  That is such a cool story. I want to follow that thread. Before I do that, you just mentioned you guys yell thermal. What does that mean? Is that like YOLO, like a version of, okay, we're just going to ship? Because we're a thermal project.

• 47:46Speaker 02

  Ship it.

• 47:47Lenny Rachitsky?

  Okay.

• 47:50Lenny Rachitsky?

  Marketers, I know that you love TLDRs, so let me get right to the point. Wix Studio gives you everything you need to cater to any client at any scale, all in one place. Here's how your workflow could look. Scale content with dynamic pages and reusable assets effortlessly. Fast-track projects with built-in marketing integrations like Meta API, Zapier, Google Ads and more. A/B test landing pages in days, not weeks, with intuitive design tools. Connect to tracking and analytics tools like Google Analytics and Semrush and capture key business events without the hassle of manual setup. Manage all your clients' social media and communications from a unified dashboard, then create, schedule and post content across all their channels. If you're working on content-rich sites, Wix Studio's no-code CMS lets you build

• 48:34Lenny Rachitsky?

  and manage without touching the design. And when you're ready for more, Wix Studio grows with you. Add your own code, create custom integrations with Wix-made APIs or leverage robust native business solutions. Drive real client growth with Wix Studio. Go to wixstudio.com.

• 48:52Lenny Rachitsky?

  Okay, so coming back to this algorithm, this is actually really interesting because I've never heard any of this. I was going to ask just what inspired this actual algorithm. And you basically did an internal competition amongst ML engineers to see who had the most successful algorithm. Netflix, content-style, Kaggle-style.

• 49:08Speaker 02

  Yeah, yeah. I think, so I mean, this particular idea of finding content that is liked by people on opposite sides of a polarized divider who typically disagree. Now, this was not an idea out of thin air, right? Like, I think Keith had found some of Chris Baile's work he had done. He had made this list of accounts that were often liked by people who were on both sides politically. There is other projects like Polis out there that look for agreement among people who typically disagree. But I think that, yeah, it wasn't obvious that our project definitely needed to use that from the very beginning.

• 49:52Speaker 02

  But then, when you implement it and compare it against these other type... Like, PageRank seems, obviously, it's designed to be kind of manipulation-resistant. It's naturally... Naturally, like if you just have a voting ring of people who all vote themselves up, then Paydrink can filter that out very well. But like that just wasn't the main attack vector, I guess. So we got, we had to get some real data from the pilot to realize that, okay, the real thing going on here is people are polarized. And, uh, and, and so it was only once we got that, uh, the, the real data from the pilot that I think it was, it was clear that the, the bridge base algorithm was, was, um, the direction we really needed to go.

• 50:34Lenny Rachitsky?

  I want to come back to the way you operate the team. I hear that you run the whole team off a single Google Doc. That's like a four-year-old doc that you just keep adding, uh, goals to and bullet points. Is that true?

• 50:46Speaker 01

  There, there is a very long-running doc, um, that has had to be, uh, chopped and purged because it was breaking Google Docs in Chrome at various points in time. It's sort of like a note-taking doc. Um, it's really where we coordinate what we're doing. The team meets on a daily basis. Um, we spend whatever amount of time we need to get on the same page about what we're building. It can be, you know, we might talk about anything from, you know, what's most important right now to what are, what should we work on next to what are we trying to launch right now and why is it not launched? Like what's in the way of launching it? And we might review, uh, new modeling or scoring algorithm update and, you know, try to understand what's working in it, what's not.

• 51:30Speaker 01

  So we'll just cover whatever we want and, um, or whatever feels most important. And we like, you know, as you said, we set our goals very dynamically. So it's whatever seems like the most important thing for us to work on now and next is what we spend our time on. I think that's served the project really well, um, versus how feeling attached to like some kind of quarterly goals or something like, well, look at like, what is going to help people the most or like, what's the biggest problem right now? What are either one of those and we will go tackle it. And we can, we might change our roadmap, you know, multiple times in two weeks based on what we see.

• 52:08Lenny Rachitsky?

  So I'm hearing no Jira, no Sana, no Monday.com.

• 52:11Speaker 01

  No.

• 52:12Lenny Rachitsky?

  Okay.

• 52:12Speaker 01

  Yeah, I mean, we have to use Jira to like coordinate with some other teams. Like sometimes when we file a request, we have to make a Jira ticket. But no, I am not a fan of heavyweight task management. I love like being on the same page, being able to keep most things in my head and having a really lightweight to write down the things that, you know, I can't, or the team can't keep in its head.

• 52:33Speaker 02

  We did use a Sana briefly, but my memory of it is that it spent, you spent more time in the meeting grooming a backlog of irrelevant stuff than actually, you know, talking about the proper priorities. So I think it's nice in the Google Doc that if something becomes irrelevant, it can kind of just fall off without needing explicit backlog grooming.

• 52:58Lenny Rachitsky?

  So just to maybe summarize a little bit of how you guys operate that might inspire other companies to set teams up like this. So I'm going to go through a few things that you've shared. One is one person in charge of the team, like the founder, almost they're like basically the founder of the team. They have one very senior, essentially sponsor slash decision maker that they interface with. In your case, Elon, no big deal. In other cases, it could be the CTO, CPO, someone like that. The team is focused 100% on this product and goal. You keep the team very small, so you start with one person of each function, one front engineer, back end, ML person, designer, researcher.

• 53:39Lenny Rachitsky?

  And then Google Docs is almost basically for your project management. Is that roughly like, yeah, it's basically running with Google Docs. Stop, don't use big complicated products.

• 53:50Speaker 01

  I think that's a pretty good recipe. On the Google Docs, you know, take it, people can do what they want. If they want the thumbnails, go for it. I think those first ingredients are really, are key structurally. And then, you know, beyond that, it's a matter of having an ambitious goal that gets people fired up to go do great work.

• 54:06Lenny Rachitsky?

  Yeah. Awesome. I think there's a lot there that a lot of people kind of like think they should do and they set these teams up, but they don't actually do. And it feels like each of these is just a really key ingredient to it's actually succeeding.

• 54:17Speaker 01

  It definitely really helped us succeed. I don't know that the project would be here if it was not for some of those elements.

• 54:24Lenny Rachitsky?

  That's, that's a powerful statement. Like this thing that has changed the way the world understands what is true would not have existed if you didn't set it up in this specific way.

• 54:33Speaker 01

  Yeah, I think, you know, I don't know if I would have begun the project had I not known we had sort of that structure, that ability to make decisions, the autonomy, the speed, the ability to go fast. And, you know, working, we started with that in 1.0 and it's been continued and if anything furthered in X. I mean, X as a whole company operates with a lot of those attributes. And I think it's one of the reasons the product is successful. I think it's a big, those are big reasons why, at least I, Jay can speak for herself, I have so much fun working on this. Like, I love working on it. You know, it's great

• 55:15Speaker 01

  to wake up every day and solve these problems. We get to, you know, we get to do them efficiently, make decisions quickly, build stuff that helps a lot of people. It's awesome.

• 55:24Speaker 02

  Yeah, this, this like, whether thermal or eon way of operating is definitely more fun. And, and the fact that, like, that combined with the awesome mission is super important for internal recruiting. Like, I remember, like, when I was first chatting to Keith about this back in early 2020, you know, I had another project. I was, you know, I worked on a few, but one was like, personalizing the number of push notifications that we send. And it was, it drove a lot of DAU without, like, losing opt-outs significantly. So, you know, that, that was like, setting me on track. Or, you know, if I had kept working on that, I could have probably gotten a promotion from that with low risk. Or I could take this

• 56:09Speaker 02

  huge career, I mean, it's not as big a career risk as, like, joining or founding an actual external startup, but there is still career risk, I guess, in joining a team like this. So, so just, I think all of the same aspects of recruiting that apply to external startups apply internally. And, and, you know, if you can have an exciting vision, that is key.

• 56:29Speaker 01

  Related to that, and your list, Lenny, one thing we missed that's super important is that on this project, and I think of successful projects like in startups, is that people are self-selecting to join.

• 56:44Speaker 01

  We did not assign anyone to this project. Like, people reached out to join, or they applied to join the job. You know, I and the team interviewed every single person that joined the team. And we're like, we want that person on the team. They want to be on the team. And so people are totally bought in to the goal, mission, the way the team works, the other people they're going to be working with. And that makes a huge difference. So I was like, a great time to do that is at the start of one of these things. Like, don't, if you're going to try something crazy, like, I would, it's gonna be tough if you're just assigning random people to it. But if you let people opt in and self-select, you're much more likely to be successful. And one thing that I have observed at X, which really surprised me, was that this is

• 57:29Speaker 01

  also possible at a large scale. You know, one of the things Elon did when he bought the company was he basically asked people to self-select to stay. Like, you had, you had to click the button. And he sent an email out that was like, hey, Twitter 2.0, like-

• 57:44Lenny Rachitsky?

  Fork, fork in the road, right?

• 57:45Speaker 01

  Fork in the road, fork in the road, exactly. It's like Twitter 2.0, you know, now X, it's gonna be hardcore. We're gonna do ambitious things. You're gonna work your butt off, you know, and you had to click on the form and say, yes, I want to join. And I think that was really important for the company because you want people to opt into that. You want the people to be saying like, yeah, that's what I want to do. And the company's gonna be a lot more successful. If people aren't sure, it's like better for them probably to go do something else. And where they're naturally more aligned and happier. And I thought that was a great approach to taking a large company and getting it down to people who are really excited about, you know, working together on a mission.

• 58:27Speaker 01

  So, you know, for us, we did it from day one, which I think is an easy way to do it, but it's possible to do it later as well.

• 58:33Lenny Rachitsky?

  I love that you described it as fun. And I think a lot of people, when they see Elon laying off a bunch of people, being like very hardcore himself, it's, people don't imagine it as a fun place to work. And it's clear how much you guys love working on this, like how fun it is and how interesting it is. And it's interesting to hear that because I think a lot of people don't feel that externally. Is there anything else along the lines of just working for Elon within an org Elon runs that might surprise people about just the way of working that's interesting or surprising? Or, or you think other companies might want to think about adopting?

• 59:07Speaker 01

  I've always liked lean teams, but this has made me, my experience at X has made me change the way I would, I would think about running a future, or you know, if I were to start a company and change the way I think about starting that company, it would be even leaner than I would have made it before. I've been amazed with just how much the team is able to accomplish with a small group, and I think because of a small group. Like when, shortly after the acquisition, we had this product called Spaces. It was, it had been in the product before, but it was, it was pretty small scale. And Elon wanted to run these large spaces. I forget who the first people he was going to bring on were, but he was going to be there.

• 59:50Speaker 01

  You know, ultimately, these things have gone on to host politicians and things like that. And he's like, guys, we got to scale this up. I forget the number. He's like, we need to be able to scale like a million people or something like that. I'm getting the numbers wrong. You'd be able to scale way up. This is the kind of thing at 1.0 that would have taken a year if it had ever happened. And the team did it in like two or three weeks. And it was really exciting and inspiring to see, like we, I didn't work on that, but I watched it from the outside. I'm like, wow, with this tiny team motivated behind a big goal that was like, hey, guys, it's not like, are we going to do this? It's, we are going to do this. They got it done in two or three weeks. That must have felt amazing for them. It was certainly exciting to see.

• 1:00:32Speaker 01

  But it's, I, I've definitely come to appreciate just how lean something can be and not just get by, but actually thrive because it's that lean.

• 1:00:42Lenny Rachitsky?

  I think the point you made about people opting into that is important because I think a lot of people hearing that would be like, I would never want to be asked to build something like that in two weeks. And I think a lot of people do and would love that kind of experience, especially working with Elon, especially shipping something at that scale. But I think there's an important element. They're just like, okay, I don't want to do that. I have other things to do in my life other than ship spaces. So I think that's a, I think that's a key point you've raised of just, there's an opt-in step.

• 1:01:09Speaker 01

  Totally. I think the opt-in is important. It may even be that you want to opt in one part or, you know, at one point in your life and maybe at another point in your life, something else is better. I think, you know, whatever it is you're choosing to do, it's nice to, to be opted in to feel like it's aligned with how you want to spend your time.

• 1:01:25Lenny Rachitsky?

  Something on my mind, and I don't know if you guys want to go here, but it's something I think a lot of people think about is when Elon came in, he let go of 80% of folks and everyone's just like Twitter is dead. It's all gonna fall apart. There's no way they can run this thing with that small of a staff. And clearly they were wrong. Clearly it's working great. It's like becoming like a massive deal in the world and continues to grow. Is there anything about that that you were surprised by or anything about just like how it continues to operate so well in spite of that big shift?

• 1:01:57Speaker 01

  I think the leaner team, the reduced kind of like process and bureaucracy is a big reason it does move as fast as it does. It's easier to get stuff done faster here and

• 1:02:15Speaker 01

  yeah, I mean, I think that's, I think, I think it's that that shrink shrinking is actually a big reason for the increased pace of launches, the increased pace of experimentation. One thing that I noticed that as a result of that is the people who are here, they seem to all really feel like owners. Like they take the sense of responsibility that an owner takes in the product. They'll try to track down what's wrong, fix whatever is needed, jump into any, to help build or fix, improve any system that needs help, even if it's outside of their space. And there's the flip side of that too. For people who've worked at the companies, they may have experienced this thing where, where there's like, you want to change some, something in some other system or product.

• 1:03:01Speaker 01

  And so you reach out to that team and like, maybe they're a little resistant. They're maybe like, oh, we'll get to that next quarter.

• 1:03:07Lenny Rachitsky?

  They have their own goals to hit. Yeah.

• 1:03:08Speaker 01

  Yeah, exactly. Like they don't really necessarily want to help you or they're busy. Here you're like, hey, guys, we need to do this thing with that other system you work on. And they're like, great. Here's the code. Here are the docs. Send us the fab if you have any questions and we'll get it in. And it's just the thing, you can just jump in and, and get it done. And that kind of collaborative effort, like the sense of like shared ownership, I think.

• 1:03:34Speaker 01

  From my experience, came from a result, was a result of the shrinking of the team down to people who, you know, wanted to be there and work together to build this thing. So I think that's been a really positive impact. It's not always easy. Certainly, like, a lot of people have a lot of responsibilities, but, you know, they're here because they're up for it.

• 1:03:53Speaker 02

  Yeah, I think one other thing that's key is when you are forced to have such a small team, you know, deleting, well, this is important anyways, but deleting code is more important than writing it a lot of the time. So I think so often, maybe due to promotion incentives or just regular human tendency, you, you know, engineers have a tendency to add these little incremental wins that actually add, you know, more of a long-term maintenance cost than is clear because you just run a little one-month A/B test, you see this, you know, significant win, you don't realize the maintenance burden you just added to your team for the rest of eternity until you turn the thing off.

• 1:04:34Speaker 02

  So I think there's a lot to be gained and you get forced to do this, by the way, when you have such a small team, is just deleting, you know, auditing, auditing parts of your system and deleting the things where the maintenance cost is worse than the gains. So I think we did have to do this across the company after the big layoffs and, you know, systems are leaner now and they can be worked on by fewer numbers of people.

• 1:05:02Lenny Rachitsky?

  That's an amazing point. I remember Elon's being like, here, we have to throw away the whole thing, we have to re-architect everything, it's stupid the way it's built and it sounds like that actually worked.

• 1:05:10Speaker 02

  Yeah, well, we didn't, you don't have to rewrite everything from scratch. I mean, some things we did, I guess, rewrite, but, I mean, just even deleting the unnecessary cruft and keeping the rest of the core system.

• 1:05:22Lenny Rachitsky?

  That's awesome. I love the word creating kind of a formula to run these sorts of companies and teams. There's so much here. I want to go back to the The building of the original product, it kind of took us on a long tangent and an amazing tangent. But I heard a story of when you launched Birdwatch at that point. You specifically wanted to keep expectations very low. And there was like a GIF in the thing and it just looked like clearly this is not ready for prime time. Talk about just how you do that, how you launched it in a way where people weren't like, it's never gonna work.

• 1:05:53Speaker 01

  We were very disciplined, I guess you could say, about having the product prove itself at every given point. You know, when we built the first mockups, we had just, these were just like pictures of depicting what community notes might look like. We showed those to people across the political spectrum. We saw like, hey, people really like these, whether they're on the right or left. Like, they seem very open to reading these community notes, even when they are critical to people of their own side. So we're like, all right, that gives us confidence that if we can build this, like, if we can actually make this a reality, it's gonna work. Then there's a question of like, can we make it a reality? Like, are, will people in the real world be able to write notes that are of this quality?

• 1:06:35Speaker 01

  And so, you know, we built, we had an internal pilot test version of this where you could like write notes. And we first basically ran this through like an Amazon M Turk type of test, just to see like, if you just like put some normal people in there, like, will they be able to write these notes? And it not, they weren't all, all those notes weren't good, but like, it was clear that there were people out there who could write good notes. So then like, okay, this is possible. Like, what will happen if we actually do this out in the real world? And like, let's run, let's run a pilot and find out. And so, we took that pilot, the, you know, we'd run the M Turk kind of test on. And we released it to at first a thousand people, you know, totally out in public.

• 1:07:21Speaker 01

  And we didn't know what was going to show up. Like you could imagine the notes could have been terrible. And, uh, and so we were talking like, well, what do we do? Like, we're going to put this out there. Everyone's going to have all these questions. They're probably going to be really skeptical. Like, and we know it might be a total dumpster fire. And so like, what do we do to like, set expectations appropriately? Like, we felt like we could probably get there in the end, but we just didn't know what was going to happen at first. We wanted to set expectations. And so like, well, why don't we just stick? There's like the page where you see a post in the notes below. We're like, why don't we just stick a dumpster fire gif like on that page? And, uh, you know, you go there, you're like, hey, you know, anything you see below here might just be a total dumpster fire. At least it would show we, we were aware of that as a possible risk.

• 1:08:07Speaker 01

  Um, we, in the end, we did not do that. It cracked me up. Um, but we thought it was kind of like.

• 1:08:13Lenny Rachitsky?

  Oh, you didn't actually launch it. Okay.

• 1:08:14Speaker 01

  We, we had, we had mockups of it and every time I looked at the mockup, I laughed. But, uh, ultimately we had so much to explain on that page, like what is this thing? And how does it work? Ultimately, like, okay, this is probably going to like distract from the point. So we pulled it. So I somewhat, I kind of wish maybe it had seen the light of day at one point, but yeah. Ultimately, we kept it simple. We focused that page on explaining what was going on here. But again, you know, we, um, as has happened many times with, with the project, um, you know, we put the pilot out there and the notes were good. Like they weren't all good. There's, there's a mixed bag, but like there was gold in there. And from the very early days with just a thousand contributors, it was obvious that, that people could write notes

• 1:08:58Speaker 01

  that were informative, that were neutral, that spoke to controversial, challenging topics. And that if we could just identify those from the rest, this was going to work. Like it was going to work as well as the very first mockups we had made. So that became the, the, the focus that is how do we sift out the gold from the rest?

• 1:09:17Lenny Rachitsky?

  I remember there's a, I think you may have shared with this with me when someone noticed you guys were testing this and they took screenshots and tweeted it and I think Elon replied like, "That's cool."

• 1:09:27Speaker 01

  Yeah, yeah. So in the, in the very, in the very early days, when it was just a Figma prototype, we were running these like usertesting.com unmoderated studies. I guess one of the participants sent one to an NBC reporter who like wrote a bunch of stories on it. Anyway, like that day, there was a lot of chatter about it on, on the service. And Elon, this is like, this is, put this back in, you know, time perspective. This is, I think, 2020. So, two years before any acquisition stuff happened, Elon is just a Twitter user, building rockets and electric cars and other cool stuff, and stumbles on this thing that depicts the prototype that we'd been testing. And he writes back, "Yeah, definitely worth trying,

• 1:10:11Speaker 01

  IMO." And I remember thinking that was cool back then. And it's interesting to see, like, he's obviously had a very consistent point on it. I think that, you know, the idea was appealing and he, you know, has obviously been a big fan of it in the product and been a big supporter proponent. So, yeah, it was kind of, it was kind of cool that it came from, that support has been from the very early days before he was ever involved in the company.

• 1:10:36Lenny Rachitsky?

  I love that moment. It must have felt really wild for Elon to be commenting on this Figma prototype retesting.

• 1:10:42Speaker 01

  It was cool. It was cool.

• 1:10:44Lenny Rachitsky?

  Oh, man. So, when we were preparing for this interview, I asked you guys, what's, what's the main thing you want to make sure people get and understand about why community notes has been so effective? And Keith, you specifically said that it was the principles behind how you wanted to approach this and how you continue to stick to this throughout. And we'll talk about how you kept it alive throughout all these different CEO changes of leaders. But just talk about these principles, like what the actual principles are and why that was so key to it working out.

• 1:11:16Speaker 01

  There are a number of principles that I think when we first shared them with people at the company seemed maybe a little bit crazy. But I think they are, they are the reason the product works. And I think they've been very important. And we do, we come back to them regularly today, all the time. Probably the craziest one is just that this thing is going to be the voice of the people. It's going to represent the voice of people. It's not going to represent the company's voice. So it is not a tech company deciding what shows. It is the people deciding what shows. And that had a lot of implications on the design. Like, first of all, there is no, we don't have a button that will change the status of the note, of a note.

• 1:12:00Speaker 01

  So if a note is showing because the people have rated it and found it helpful, it is going to show. Like, we can't change that. And that is the kind of thing that like, when we first proposed this, that's unsettling to people. They're like, wait, so like, something can go up and like, we, you know, the company can't take it down, or, you know, can't change its status, get it to stop showing. And we're like, yeah. And like, it has to work that well. If it doesn't work well enough to do that, then it doesn't work. If there's a problem with the note, this is like one of our, one of our key principles was if there's a problem with a note, that's so bad, you want to do something about it. It's a problem with the system. Like, we need to redesign the system to be showing good notes. And so, so yeah, we had to, you know, get everyone comfortable with the idea that there was no button to,

• 1:12:45Speaker 01

  to change the status of a note. Similarly, we, as we talked about earlier, we wanted this to represent all of humanity. And so we didn't want to be arbiters of who can come in and be a contributor and who can't. Like, so we, we open it to everyone. You just have to make really basic objective criteria. Like, you have to have a verified phone to help reduce the likelihood of having like, bots or things like that participating. But beyond that, it's random selection and it still is that way today. And, you know, again, that people, it took some time to get people comfortable with it. But I think that the, the fact that this is the voice of the people and reflects their

• 1:13:24Speaker 01

  output through an open and transparent process is so key to both why it is good, like why it works, but also why it's trusted. So, I mean, that's number one. And, and you, you know, well, I think will forever be the heart of the product.

• 1:13:41Speaker 01

  Another one that people thought was kind of crazy was transparency. We're like, we're this, we, the previous approaches to dealing with misleading info. They, it felt to a lot of people like sort of black box tech companies or media companies or leads or whatever, making decisions like they, people, people need to get comfortable with this. They need to trust this. So the whole thing has to be out in the open. Like the code that decides what note show has to be out in the open. All of the data and ratings that, that make it happen have to be out in the open. People should be able to take the code and data and replicate the whole service. And that, that we have done exactly what we've said we've done. And they should be able to audit it. They should be able to go and look and say like, hey,

• 1:14:27Speaker 01

  I think this part could be better. Or like, if they think we're biased, they should be able to work, you know, work with the data and point it out. And if they, if people have good observations, that should factor back into the code. And this is again, something that's kind of difficult to get people comfortable with. That everything is out there. You can't cover anything up, but I think that's so essential to people trusting it. So, um, yeah, I mean, we, we, we set these out on day one. We go back to them constantly because we're, we're, we're always evolving the, the product. And we're always like, got to make sure every new change is open. Like whenever we update the code, there's, or we're update the scoring system. There's an update in GitHub when the data is published daily, so you can download it. And so, um, yeah, I think those have been, those have been really essential to the thing working.

• 1:15:11Speaker 02

  Yeah. And, and by the way, these, these do not come without a cost, right? Like the, the, it's actually really hard from an edge perspective to actually open source the actual algorithm that's running on the actual data. You know, because the, with the way large scale services like this are usually architected does not, you know, naturally lend itself to like being run, uh, like as a script by someone who's downloaded a T S V. So I actually have to take weird architectural decisions, uh, to make this possible in a way that probably wouldn't have been if we didn't start with this assumption from scratch. We would have had to maybe rewrite the system to make, to, to make it like this.

• 1:15:49Speaker 01

  What's an example of that?

• 1:15:50Speaker 02

  For instance, there's a matrix factorization that we, that we train, right? Usually you would, um, Train a matrix factors, you know, train your ML model once and then serve it. I guess with a separate service. But you know, we didn't want to have people externally spinning up services. You know, to be able to replicate the, you know, system that we had. So I mean, basically, I don't think it would have been actually very cool if we had open source the code in a way that wasn't actually runnable, I guess. You know, by someone just, you know, at this point, you can download Python code and run a script. You do need a lot of RAM right now, but you can do it on one machine.

• 1:16:35Lenny Rachitsky?

  Okay, okay. How much RAM we talking about?

• 1:16:37Speaker 02

  Oh, like 500 gigs.

• 1:16:39Lenny Rachitsky?

  Okay, okay.

• 1:16:39Speaker 02

  And it'll, it'll take like a day if you don't do anything special to speed it up.

• 1:16:43Lenny Rachitsky?

  Good to know.

• 1:16:44Speaker 02

  But yeah.

• 1:16:45Lenny Rachitsky?

  Cool.

• 1:16:45Speaker 02

  Possible is the key thing, and people have done it. Like Vitalik Peterin had a blog post where he, you know, talks about his explorations. You know, making sure the algorithm really does what it says it does. And I think just the fact that a handful of people have done this. You know, there's, there's enough people who have done it that there's someone you'd probably trust who's, who's verified.

• 1:17:10Lenny Rachitsky?

  Yeah, and now it's rolling out to Meta, no big deal. I love just like, as you describe these principles, just, I could imagine a PM at a company being like, okay, guys, here, I want to do this. Project, it's gonna be completely, like, there's so much idealism to it that rarely works in real life. It'd be open source, you're gonna give it to everyone, we don't have actual control over what it's gonna do, don't worry about it. It's gonna just change the way people see this thing that we've been very careful about. And then it works. And I think that's very rare, and it's really impressive. And what I'm hearing partly is that sticking to those principles was actually really fundamental to it working and not kind of bending over backward, bending over when someone's like, no, no, no, we can't do this. What if we change this part?

• 1:17:53Speaker 01

  I think if we had, if we had broken with any of those principles, like if there was anything black box, if there was, you know, whatever. I, the product would be a lot harder to trust. And so I think it's because we've just stuck to them so cleanly, simply, that, you know, people, people can't trust it.

• 1:18:10Lenny Rachitsky?

  You've talked about a few moments when it was like, wow, the White House changed their announcement because of a community note. We talked about the dog is a cat. Are there any other moments that after you launched of like, holy shit, this is working? This is gonna actually work?

• 1:18:26Speaker 01

  All along, you know, we saw it. We saw it working. Like, we were pretty, we we wanted to be confident whenever we expanded it to new, you know, audiences or new countries or whatever. Like, we wanted to be confident it was gonna work. So you know, maybe you hold her breath a little bit just to see that it would do what we expected, but we always expected that. But that said, there were definitely stress stress cases. I mean, the one that comes to mind is the start of the Israel Hamas conflict in 2023 in October. That was probably the largest deluge of misleading information I've ever seen shared on the internet at one time. It was like overwhelming. The just

• 1:19:08Speaker 01

  the number of photos and videos and whatever coming out related to that was, it was insane. And just to, you know, to give you an example, in the first, I think it was like first three days or something of that conflict, we had 500 notes covering all sorts of different, you know, like out of context imagery, like people, someone say like, hey, this is happening here. It's actually from like 2013 in Syria. There were people making fake like battle footage in the video game simulator, Arma 3. So there were like notes explaining, the stuff looked really, looked realistic. And unless you saw the note, you like wouldn't really know. There are all sorts of claims about what was going on in the ground.

• 1:19:53Speaker 01

  And, you know, that was definitely the product was still pretty new at that point. We had only, we'd expanded in the US less than a year before that. We had been rolling out throughout the world that year. And then this large event happened and You know, I felt like we were, we were just enough prepared at the right time for the system to be able to handle that. Like probably one of the most important things we did right before that was launch the ability to write notes on images and media and have images and videos and have those matched to other posts. Like I remember at that time thinking like, wow, I'm glad we launched that feature a few months ago versus still had it on the shelf.

• 1:20:35Speaker 01

  Because it was really important in that conflict. And I think even it was like just a few weeks before we had launched a major speed up in notes. Like when we, when we first built the product, the number one focus was always quality. Like we knew, we knew that notes, the product would live and die by the quality of the notes. And like that was the thing we could never give up on. We also knew it needed to deliver speed and scale, but we're like, we will, we'll get the quality in the right place and we can speak, we can speed it up and scale it out over time. And we had actually just launched a speed up that took like three hours off the time it, time a note needed to go live. And I was, you know, I think a matter of weeks before that conflict happened. So again, like super glad that was out there. The, in the first few days

• 1:21:19Speaker 01

  of the conflict, the median time from a post going live to a note showing up was five hours, which is like crazy fast. Typical fact checking is like two to four, it's really, it's really common to see it take two to four days. These notes were showing up in five hours and we're like, we are so glad we got those things out before this happened. It made the service a lot more helpful.

• 1:21:40Speaker 02

  One other thing that was, I think, nice to, you know, see, see working then was, I think, like one criticism of community notes, some people bring up is, well, if you always need agreement from people who typically disagree, then in these super polarized settings, like that conflict's being like probably number one, then, then, you know, you wouldn't see any notes. But actually, the reality was there were the, you know, tons of notes about that conflict. So I think there is this kind of You know, nice, nice property where actually, and maybe this is a surprising fact, that there's more agreement out there across polarized divides than maybe conventional wisdom says, right?

• 1:22:22Speaker 02

  And the places where people agreed were really objectively true and verifiable. Like, I guess maybe this is more true the more polarized the setting is, but where the agreement actually lends you in basically notes that are very neutrally written, very focused on the facts and easy to verify information.

• 1:22:44Lenny Rachitsky?

  I love just like, there's this talk for a while of just like, there's no more facts. Like, nobody believes there is a single true fact anymore. Like, everything is subjective. And I think Community Notes is exact, proves the opposite.

• 1:22:58Speaker 02

  Yeah.

• 1:22:59Lenny Rachitsky?

  Fact matter, there are facts that we can all agree with, even in the most controversial topics.

• 1:23:04Speaker 01

  Yeah, we saw this really from day one. It's the, you know, when we would show those prototypes to people just depicting the idea, it was really obvious that people cared more about, or they cared a lot about getting, understanding reality and what was going on. And they were willing to disagree with like their side, so to speak, to recognize that. And I think that's not always that obvious to people. The world does feel really polarized, but people definitely are willing to cross partisan boundaries to get to, you know, accurate information. And that's why the product works.

• 1:23:37Lenny Rachitsky?

  It feels like as we rely more and more on what we know and understand, that the world is becoming social media, online and moving this quickly, it feels like, it's like, I'm so thankful this exists because otherwise it'd just be, what do we trust anymore? Like, this is like, this being out aligns with, we need this thing to exist at the same time. And it feels like, at the same time, there's also this, people just, like, I just don't trust. I think people have shifted from, I trust what I read to, okay, I shouldn't just believe everything I'm reading. Is there anything there you're noticing about just how people think about news they

• 1:24:19Lenny Rachitsky?

  see and their shift of just like, I'm not going to believe everything? Is there anything there you've noticed about just like human behavior, or just the way we've shifted understanding what is true?

• 1:24:30Speaker 01

  We haven't done any research, you know, to look broadly at how people's perceptions are changing there. But I certainly have found, you know, myself that particularly seeing notes, I am more skeptical about what I read at first, and I think that's been helpful. And we hear that from people that they think about things a bit more. And, you know, I think that's a good secondary effect and benefit of something like this, which is the more you see the patterns of how what you're reading can be wrong, the more you can thoughtfully question it and try to get a better understanding what's really going on. So it's like, you know, historically, I think this was called like media literacy.

• 1:25:12Speaker 01

  But a basic idea of like, can you understand the ways in which things can go wrong? And try to vet them yourself.

• 1:25:20Speaker 02

  And another aspect I think we help with that is discovery of the community notes. I think often, you know, before community notes, you could have just been living in a little news filter bubble. Or maybe there were fact checks out there that you should have been reading, but you weren't discovering them, right? So the fact that the note applies, it is directly attached to the post and visible by anyone who sees the post helps, you know, cross those filter bubbles and can kind of, I think for some people, it's the first time they've actually seen counter arguments to, you know, to claims made in their own little echo chamber.

• 1:25:59Lenny Rachitsky?

  That's incredible. Yeah. I love the point you're making about how it actually teaches people to be a little more skeptical of the things they read. Like it's an education system more than just here's this one thing is wrong. I love that. Okay, just a few more questions. There was an audience question asked on Twitter, we all asked on Twitter, what do people want to know about community notes? One was actually why you guys switched to anonymous contributors? What was the decision behind that?

• 1:26:26Speaker 01

  Yeah, that was, you know, we had this pilot where we were testing with a small number of contributors, like a few thousand contributors, and we learned a lot through that pilot. Probably the biggest thing we learned was related to anonymity or pseudonymity of contributors. We had originally assumed that it was important that people contribute under their like real handle or their real name or whatever it was. We actually like the first prototypes depicted that. We kind of thought that would be important for like people trusting the note. And actually, it was totally wrong. We were like, it was the best option was actually the opposite of what we first tried. We found a few things. One, people were hesitant to write a note on a controversial topic

• 1:27:09Speaker 01

  because they didn't want to get like attacked or harassed online. And so some people were comfortable doing this, but others were not. And so it meant there was more potential good notes to be written than were getting written. And this is very clear feedback from the pilot. Two, and this is super interesting. People are actually more willing to cross partisan boundaries when they are anonymous or pseudonymous than when they are under their real name. And it intuitively makes a lot of sense. Like if you publicly are using your name, you are, you feel are affiliated with one side versus the other. You might hesitate to kind of like be perceived as breaking with that side.

• 1:27:49Speaker 01

  But you may actually, you know, for example, find a note helpful that's critical of that side. And there's a bunch of studies that show when people are anonymous, they're much more willing to cross partisan boundaries and work with the other side, agree with the other side. And we saw that too. And so by allowing people to just to be pseudonymous, you actually get more honest answers about what they really think. And it helps find disagreement that really.

• 1:28:10Lenny Rachitsky?

  That's so counterintuitive. You know, you never hear the opposite always, and it's so interesting that's the opposite.

• 1:28:15Speaker 01

  Yeah. Yeah.

• 1:28:17Speaker 02

  I think the same principle applies to making the likes private.

• 1:28:21Lenny Rachitsky?

  I was just thinking that.

• 1:28:22Speaker 02

  Yeah.

• 1:28:22Lenny Rachitsky?

  Yeah, I like a lot more stuff that's a little, it's tough. I wouldn't have liked, yeah.

• 1:28:27Speaker 01

  It allows, it allows freedom for honesty, which is pretty great. And one of the criticisms of pseudonymity is like, you know, it can generate, maybe like people have reached their quality, the quality threshold that they put out there. But we have so many quality mechanisms in the system that that wasn't an issue. So we could keep quality high while opening up for that honesty.

• 1:28:48Lenny Rachitsky?

  Another question, you touched on this a little bit, which is around navigating the existing trust and safety apparatus of Twitter, which as you described, basically previously was like, we make decisions on what is true and not. And there's, every company works this way. You guys basically upended that. Like, here's a completely different way. You have no control over what is, what we say is true or not. Talk about just that experience of kind of overcoming that, I imagine, very difficult hurdle of like, okay, forget all that. We're going to do it totally different.

• 1:29:20Speaker 01

  Yeah, it was definitely, what we were proposing was very different. I will say that I think people were sort of open-minded to it, generally speaking. And I think everyone had a sense that what was being done at the time wasn't really working that well or solving the problem. And people were open to new ideas. So that was, that's like a good foundation. But I think one thing we did that was probably very helpful in that is, we wanted the product to prove itself at any point. Like, first, it had to prove that people could possibly, you know, find notes helpful. Then it had to prove that people could possibly like, write these notes that would be good quality. And so anytime that we were proposing doing something with the product, like running

• 1:30:02Speaker 01

  some research test, or running the pilot, or expanding the pilot, we always had the data that had convinced us that that was a good decision. Like, we were stepping into the next phase of expansion that made sense. And so I think we probably rarely proposed anything that seemed unwise, like, because we were holding such a high bar for quality ourselves. And I think, you know, I suspect that went a long way.

• 1:30:32Lenny Rachitsky?

  So it's partly, what I'm hearing is like, take it step by step to prove this is actually working, and partly be confident it is working to yourself before you try to convince, say, the trust and safety team, this is the way to go.

• 1:30:46Speaker 01

  Exactly.

• 1:30:48Lenny Rachitsky?

  Was there a moment along that journey of just like, okay, here, it shifted from no way this is a thing to, okay, wow, let's actually consider this? Or is it this very gradual process?

• 1:30:59Speaker 01

  Whether other people were saying no way to, wow, let's actually-

• 1:31:02Lenny Rachitsky?

  Yeah, just internally of just like, okay, we're gonna actually stop this trust and safety way of operating and instead rely on community notes. So was there like a moment of like, okay, let's actually make that switch? Or is that Elon, actually? Is that the big switch?

• 1:31:15Speaker 01

  The biggest change there happened in X. The biggest changes prior to that were just the decision to put this out there and have it be operating in public at first US-wide scale. But yeah, then the bigger switches came in the X period.

• 1:31:35Speaker 02

  I think even though there was original research before Birdwatch even started or community notes even started from external researchers showing that crowdsourced fact-checkers can do, laypeople can do about as well as fact-checkers and actually the agreement rates were kind of similar between the groups. I think even though that research was out there, I think there were definitely a lot of people who didn't really believe it could work until it already worked.

• 1:32:04Lenny Rachitsky?

  Basically prove it. Prove that it works. Yeah, that makes sense. Versus just a bunch of docs and strategy and thinking, it's just like, look, it's actually working, you can see for yourself. Makes sense. Okay. Possibly last question. We'll see which fractals of questions you guys bring up here. I've referenced this a couple of times, this incredible achievement of keeping a project alive through Jack and then I have this note, and Cave on running the show then, and then Parag running Twitter, and then Elon, and then Lynda taking over CEO. Quite rare, especially something this visible, this impactful to everything that X is.

• 1:32:44Lenny Rachitsky?

  Any lessons or keys to that actually working of this project surviving throughout so many org changes and leaders?

• 1:32:53Speaker 01

  It definitely has been a crazy time to be building something. It's been fun. The craziness has been entertaining.

• 1:33:03Speaker 01

  I think, you know, one, one reason perhaps the product has done so well and survived is the product itself, the nature of the product itself. It is designed to produce information that is found helpful by people who normally disagree. And so even if you have CEOs or leaders who might disagree, like, there's a good chance actually they'll find it helpful. They'll be like, wow, this thing does produce pretty, you know, useful output. So I think there's something in the nature of the product itself that, you know, when people see it, whatever side they're on, you know, left, right, whether up, down, they're likely to find it pretty helpful. So I do think that helps. I also think the team executed really well. We had ambitious goals that were exciting. They solved a real problem.

• 1:33:48Speaker 01

  This is a real problem that matters in the world. At, at every step, you know, as we talked about, the product needed to prove itself. And, you know, we would make sure it proved itself and we would bring the results that convinced us and we would share those with people. And so they would say, oh yeah, I agree. It kind of proved itself. Let's take the next leap. And we've, we've done that all along the way and we continue to operate that way. And I think like that focus on the outcome and goal that matters and executing against it really helps. Like, the team did not get distracted by much, all through the period during which the acquisition happened. Like, there was a lot of opportunity for distraction.

• 1:34:33Speaker 01

  This team was shipping like every week. Like, we were super focused on the goal. Like, let's make this thing work. Let's get these notes out there. And, you know, I think people saw that execution and were excited to support it.

• 1:34:49Lenny Rachitsky?

  Yeah, like, it's working. Why would we must have that? And it's important. And it keeps us from having to hire tens of thousands of... Have people to fact check.

• 1:34:57Speaker 01

  You know, interesting thing about that is no one ever asked us or brought up or seemed to care about anything related to cost savings in this process. And I think that's like a, like a, an assumption people have outside the company that like this must have been a reason there was interest in it, but like that was never a goal, is not at all why the project was started. It's not why people are excited about the project. And I think that's also, you know, for people outside, you don't see the conversations. It's like kind of a heartening thing to know is that the focus was always on solving the problem. The other approach is even if you had 10,000 people doing it, like it, the real issue is that they don't work that well because they're not trusted or they don't scale or they're too slow.

• 1:35:40Speaker 01

  And so, um, the goal was really always just, it's like help people stay informed at scale. You know, it's building internet, internet scale solution to an internet scale problem that people like.

• 1:35:51Lenny Rachitsky?

  Something I heard about you Keith, when I was asking people about how this worked and why this worked so well, is that they described you with a, having a very low ego. And that allowed you to give up this whole team and power, influence and just the name, forget it, whatever you want. We'll call it community and that's great. Is there anything there you can share of just like how you think about that? And how important that is as a product leader to have a low ego?

• 1:36:19Speaker 01

  You know, for me, this, this project, I feel like I get to do community service with this project. Like my, I like, I see my work as in service of the people and the community. And like, that's what motivates me. The only thing that I care about is delivering the outcome that the world finds helpful. And so in some ways, like project has, has not been about like ego or results, but it's like about truth seeking. Like let's find, not truth in the sense of like what information is true, but like, let's find out what's actually going to make this work. Like, how does it need to be, how does it need to be structured? What is it, what should it be called? Like whatever is going to produce the best outcome is what we should do.

• 1:37:01Speaker 01

  And, um, so I think, you know, I feel more attached to the product being helpful than to anything else. And so, you know, to whatever degree it might seem like low ego is probably more a result of wanting to actually solve the problem.

• 1:37:15Lenny Rachitsky?

  And I think, I think partly what I'm hearing is just if you win and succeed, good things will happen.

• 1:37:19Speaker 01

  Yeah.

• 1:37:19Lenny Rachitsky?

  So focus on that.

• 1:37:20Speaker 01

  Certainly satisfying things will happen. It's very satisfying to have people appreciate it. It's satisfying that like people on the left and right, you know, love it. It's satisfying that even people who receive notes, love notes and screenshot them and post them. Like, that's amazing. It feels, it feels, you know, so good to have helped give people that. And, um, and yeah, you know, it's very motivating. It's a great reason to wake up in the morning.

• 1:37:44Lenny Rachitsky?

  It's absurd this has worked, but it's also like, of course this would work. Of course, something like this should work. It's like such interesting.

• 1:37:50Speaker 01

  The internet, it's of the internet. You know, that's why it works.

• 1:37:55Lenny Rachitsky?

  Oh, man. Where's community notes going from here? What should people look? What's happening? Where's it going? What's, uh, what's the future?

• 1:38:03Speaker 01

  We're always working on basically more, better notes faster. So we want, there's clearly opportunity to get more notes out there. Um, we want to, we want them to stay as good or better than they are. We want to get them there faster. So we're always working on like core product changes, um, to help deliver that. Like recently, for example, we, um, just released an update to what we call the community notes bat signal, or the ability to request a community note. So we, anyone on X can say, hey, I think this post needs a community note. And now they can even add a source, explaining why, so that when a prospective writer sees that, it's much easier for them to write a note. Um, so we're always working on, on core things like that, core improvements.

• 1:38:47Speaker 01

  Um, I think there are also new frontiers that show a lot of potential. Um, AI and LLMs are one. It's easy to imagine a lot of ways that, um, AI could assist the people in this task they're doing of, of trying to get information out there quickly. Um, and maybe Jay should talk about the super notes work that, uh, we've done with some folks outside the company.

• 1:39:13Speaker 02

  Yeah, so I mean, one cool thing about having public data and code is that external researchers can collaborate with you. And in this case, the supernotes of this idea that we can basically take existing nodes as input, existing proposed notes that are actually, you know, maybe they have some problem, maybe they have the whole part of the story, maybe they're worded in kind of a biased way. Basically, take all these in, have an LLM generate a ton of different variants, and then basically make the simulated jury. So to basically, you know, get a representative group of contributors for community notes

• 1:39:55Speaker 02

  who would be writing the note and try to predict based on their past ratings, how they would rate these LLM-generated notes. And so this way, you can actually, you know, rather than just like having an LLM write a note from scratch and hoping it's good, you can kind of like simulate the entire community notes rating process and, you know, explicitly create notes that are likely to be rated helpful by people. So I think ideas like that are very promising for the future. And it's a nice way that LLMs and humans can work together. I think, like, obviously, you know, agents can browse the web too. And that's one way that, you know, you could imagine agents assisting humans

• 1:40:38Speaker 02

  is, you know, maybe, maybe checking whether a source is actually supported by the note, or a note is actually supported by the source. Although then you get into things like, well, you know, are people going to actually be as diligent? You know, right now, I think raiders are very diligent because they know just some community notes contributor wrote this, like, I better check this before I read it helpful. But, you know, hopefully, people, you know, we can design things in a way such that people don't trust the output and actually verify it themselves before issuing a helpful rating.

• 1:41:11Lenny Rachitsky?

  Yeah, that is such an interesting area to explore, where you want to avoid AI hallucinating sloppy, slop versus make it easier and scale it even further. What an interesting challenge.

• 1:41:23Speaker 01

  What's cool about this project, in addition to the AI element, is that it's being done outside the company. Like, we talked earlier about the open source transparency. Like, the key reason we made this all open source was so people could see how it worked. But the dream is actually that it's not just that the contributions to the notes and ratings are from the people, but the dream is actually the product is built by the people. Like, what if the scoring algorithm were significantly or entirely written by the public? Like, that would be incredible. And Super Notes is probably the first very substantial potential change in the algorithm or the way it works that was kind of coming from the outside and plausibly could be part of the core. So we'd love to see the product go in that direction as well.

• 1:42:08Lenny Rachitsky?

  Sweet. Go Super Notes. Well, guys, the work you're doing is tremendous. I think this is every product person's dream, I think, to work on something like this. Small team, lots of support, lots of impact, just like innately interesting. And so I think this is going to inspire a lot of people. So let me just ask you, is there anything else you wanted to share? Anything else you think might be helpful for folks to leave them with?

• 1:42:32Speaker 02

  Sure. I guess one thing that just I thought was interesting over the course of working on this product is just there's, I think, in a similar way to how retweets originally were not something like Jack came up with. I think users just started doing it, and then it became a core part of the product. There's a huge way already in which there's just a lot of surprising things that people wanted to use Community Notes for that I don't think we really expected. And it's kind of cool to see those, you know, user desires kind of emerge. I think like one example, you know, I guess we'd always been imagining political type of misinformation, but for whatever reason, there's like,

• 1:43:16Speaker 02

  you know, a lot of people who love debating whether Messi or Ronaldo got more goals. I guess it's kind of a funny one.

• 1:43:25Speaker 02

  There's a community moderation aspect, right? So like, I think we also thought that, you know, this would be specifically for adding context to misleading or potentially misleading information. But what you can see is that there are some notes that go beyond that towards like calling out content that they think is spam or something. So I think that's, that's just, I guess, another, just another dimension in which community notes is a product that's like driven by the people itself.

• 1:43:58Lenny Rachitsky?

  That's so beautiful. Basically, they're trying to keep Twitter slash X healthy and they're just like, no, this should be taken down, this tweet is spam.

• 1:44:05Speaker 02

  Yeah.

• 1:44:06Lenny Rachitsky?

  I love that. Is there an answer on the Messi versus, who was the other soccer player?

• 1:44:10Speaker 02

  Ronaldo.

• 1:44:11Lenny Rachitsky?

  Ronaldo, okay. Is there like a definitive fact there? It's, that's just unknowable.

• 1:44:16Speaker 02

  Yeah, I guess, I guess that's an interesting one because it's, it's a case where raiders are actually very polarized. And I, I guess it actually kind of fits into the, the core algorithm where there's some people are just diehard Messi fans or Ronaldo fans, just like they could be on politics. So we actually, you know, specifically modeled that topic as well as some other topics so we can like estimate people's opinion on, on that particular debate. It's kind of, kind of funny that something like that would emerge.

• 1:44:45Lenny Rachitsky?

  Are you saying that's the most controversial topic on X? Ronaldo versus Messi?

• 1:44:50Speaker 02

  It's a controversial one.

• 1:44:52Lenny Rachitsky?

  Oh, wow. Who knew? Okay. Keith, is there anything you want to add?

• 1:44:56Speaker 01

  Yeah, you know, community notes is cool itself, but I think what it points to about society is actually even bigger. Society often feels really polarized. You hear people talk about it all the time, like no one can ever agree on anything, but actually like community note shows you, people really can agree on quite a lot, even on super controversial topics related to politics and everything. Like, there's a lot of agreement, that's why notes work. And I think that's a really big reason for optimism about the world is it while it might feel polarized, there's probably like an 80% you know, set of people that agree on quite a lot of things. And imagine if we could use the same kind of approaches

• 1:45:39Speaker 01

  we use with notes, but to find agreement on legislation or policies or things like that, that people want the government of the world to do. Like possibly, we could get a lot more momentum behind these ideas that the people really want and everyone would be a lot happier. Like maybe 10% of the people on the edges wouldn't be happy. But like, I bet there's a lot of agreement that we are not identifying. And if we did it, we'd all be pretty happy. So I don't know, I think it's easy for people to feel pessimistic about the world. But I think this is, this project is a good reason to be optimistic about the future.

• 1:46:12Lenny Rachitsky?

  What an incredible way to end it. I can also see Keith, why people want to join you and work with you and work on this team.

• 1:46:18Speaker 01

  Appreciate it. You do want to join. We are hiring an ML engineer. You get to work on these amazing problems with us and have a lot of fun. So we're accepting applications to x.com/community notes.

• 1:46:32Lenny Rachitsky?

  Okay, great. I'm glad you gave the role. Oh, man, you're about to get flooded. Guys, thank you so much for doing this. Is there anywhere other than that place to go off, join the team as an ML engineer? Is there any other place you want to point people to, either your socials or anything else?

• 1:46:47Speaker 01

  I'm Kate Coleman on X. Please reach out if you have any feedback or want to help us out, whether you're going to want to work here or want to do something from the outside. Would love to talk.

• 1:46:58Speaker 02

  Yeah, I'm at underscore J A Y Baxter underscore at X. Yeah, I think in particular, you know, besides just using Community Notes, it would be great to to get more substantial contributions, you know, like pull requests, collaborate on projects like Super Notes. I think that's the most exciting type of stuff. People do want to contribute.

• 1:47:22Lenny Rachitsky?

  Ship some code, guys. Amazing. Guys, thank you so much for doing this.

• 1:47:27Speaker 01

  Thanks for having us, buddy.

• 1:47:28Speaker 02

  Thank you so much.

• 1:47:29Lenny Rachitsky?

  Bye, everyone.

• 1:47:33Lenny Rachitsky?

  Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.