Pod Insights Planet Money Don't hate the replicator, hate the game About this episode The world of science has been stuck in an existential crisis over whether we actually know the things we thought we knew. Re-running an old study today doesn't always yield the same result. Same with re-enacting old experiments. Collectively, this is known as the “replication crisis.”
Economist Abel Brodeur has come up with one way to help fix this crisis: he’s invented an internationally crowdsourced surveillance system, designed to keep social scientists honest. He calls it the “Replication Games.”
Further Listening:
Fabricated data in research about honesty. You can't make this stuff up. Or, can you? https://www.npr.org/2023/07/27/1190568472/dan-ariely-francesca-gino-harvard-dishonesty-fabricated-dataThe Experiment Experiment https://www.npr.org/sections/money/2016/01/15/463237871/episode-677-the-experiment-experimentHow Much Should We Trust Economics? https://www.npr.org/sections/money/2013/04/19/177999020/episode-357-how-much-should-we-trust-economics
This episode was hosted by Mary Childs and Alexi Horowitz-Ghazi. It was produced by James Sneed and Emma Peaslee, with help from Willa Rubin. It was edited by Jess Jiang, fact-checked by Sam Yellowhorse Kesler, and engineered by Ko Takasugi-Czernowin. Alex Goldmark is Planet Money’s executive producer.
Find more Planet Money: Facebook https://n.pr/3h92GwS / Instagram https://n.pr/3FqLuws / TikTok https://n.pr/3sGZdrq / Our weekly Newsletter https://n.pr/3zrFvUB.
Listen free at these links: Apple Podcasts http://n.pr/PM-digital, Spotify https://n.pr/3gTkQlR, the NPR app https://n.pr/3Bkb17W or anywhere you get podcasts.
Help support Planet Money and hear our bonus episodes by subscribing to Planet Money+ in Apple Podcasts http://n.pr/PM-digital or at plus.npr.org/planetmoney https://n.pr/3HlREPz.
See pcm.adswizz.com https://pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences.
NPR Privacy Policy https://www.npr.org/about-npr/179878450/privacy-policy
Transcript 7,057 words · 14 speakers
0:00 Speaker 00 This message comes from Doppel. Social engineering attacks come in disguise, but Doppel sees through them. The Agentic platform shuts down threats, turns employees into first-line defenders and more. Learn more at D-O-P-P-E-L dot com.
0:17 Speaker 01 This is Planet Money from NPR.
0:22 Speaker 02 Alexi Horowitz-Ghazi.
0:24 Speaker 03 Mary Childs.
0:25 Speaker 02 Yes, you and I took a little trip up to scenic Montreal, one of the jewels of French Canada, for a little Planet Money mission.
0:32 Speaker 03 Yes, we did. And even though it's a little bit sad that that mission did not entail joining the maple harvest or infiltrating a poutine cartel.
0:41 Speaker 02 Next time.
0:42 Speaker 03 Dare I say, next time. It did have much bigger implications for anybody and everybody whose life is impacted by science, which I think is basically all of us.
0:52 Speaker 02 I think that's right, yeah. We were there to meet a guy named Abel Brodeur. Abel's this very energetic economics professor in his late 30s at the University of Ottawa. And we found him bounding around the halls of this modernist school building in downtown Montreal. He was getting ready to host an event he's become sort of famous for, something called the Replication Games.
1:14 Speaker 04 It's getting exciting now.
1:15 Speaker 03 How are you feeling?
1:17 Speaker 04 I'm feeling good. It's the beginning of the event, so this is a moment I'm full of energy and enthusiasm. In seven hours from now, it's going to be a different conversation.
1:27 Speaker 03 Abel is going to be tired in seven hours because at a replication game, he is running around between 16 teams of three to five people in a hackathon. People will work all day to replicate recently published social science papers, to reproduce the results and see if the findings hold up.
1:44 Speaker 02 Because ever since technology has made it easy to crunch data, we've been able to go back and check old research. And turns out, it wasn't great. Rerunning an old study today, a lot of the time, does not yield the same result. The research no longer proves its conclusion. And the same thing often happens when we reconduct whole experiments. Altogether, these problems have become known as the replication crisis.
2:10 Speaker 03 A lot of people across academia have been trying to fix this so we can trust research, so we can actually know what we know. And this event, the Replication Games, it's part of Abel's attempt to help solve this crisis. The idea is to change norms through monitoring.
2:28 Speaker 04 And just giving a small percentage, a small chance that we will monitor Yeah. can massively change the behavior of everyone. You know, change the way they behave, change the way they code, change the way they do research. So that's the goal.
2:39 Speaker 02 After a few minutes, we head into a big lecture hall where Abel takes center stage.
2:44 Speaker 04 All right, folks, we're gonna get started. Welcome to the rotation games. Thanks for being here in Montreal with us. Let's get started. Today, we have 16 papers that are being reproduced. Couple of small things.
2:57 Speaker 03 Around the room, dozens of social scientists are gazing up at Abel, looking a little bit nervous. Most of them have come from across Canada, and most of them are first timers who now have to undergo this kind of awkward initiation, right?
3:10 Speaker 04 I'm gonna put the music because I know you guys need like, you know, a bit of motivation. But you need to do the body movement. Everybody has to do it. All right, does it sound good? So we do it. I need you to do. It's pretty easy.
3:24 Speaker 02 Abel starts didactically clapping like an elder millennial camp counselor, and his audience joins in.
3:31 Speaker 04 Guys, thank you so much for being here. I hope you enjoy. This should be fun. And thanks, everyone.
3:40 Speaker 03 Hello and welcome to Planet Money. I'm Alexi Horowitz-Ghazi.
3:42 Speaker 02 And I'm Mary Childs. Over the past couple decades, the world of science has been stuck in an existential crisis over whether we know the things we think we know. It started in psychology, spread to medicine and economics. Now, people across disciplines are trying to figure out how to solve it.
4:00 Speaker 03 Today on the show, the story of one economist, how he set out to learn what exactly has broken in the way social scientists create new knowledge, and how he came up with his own daring and kind of wacky way to help fix it by building an internationally crowdsourced surveillance system to keep social scientists honest.
4:27 Speaker 00 Stay tuned for the following announcements and messages from our sponsors.
4:31 Speaker 05 This message comes from Rula. World Mental Health Day takes place in October, making it a great time to take that next step in getting support if you need it. Rula makes accessing quality mental health care affordable with sessions costing an average of $15 with insurance. So turn off the talk track that's keeping you from progress and head to rula.com. That's R U L A dot com to find a therapist the easy way. Support for this podcast and the following message come from E-Trade from Morgan Stanley. Simplify your finances and discover the convenience of investing and banking all in one place. Plus, get up to $1,500 when you open a brokerage account with a qualifying deposit today.
5:13 Speaker 05 Learn more at etrade.com/offer. Banking products and services are provided by Morgan Stanley Private Bank, National Association, Member FDIC. Terms and other fees apply. Investing involves risks. Morgan Stanley Smith Barney LLC, Member SIPC.
5:31 Speaker 03 Okay, so the replication crisis has been a pretty big deal for almost 20 years at this point. We've covered it on Planet Money before. The story of how economist Abel Broder first encountered the problem and why he set out to help fix it begins back in 2011.
5:46 Speaker 02 Abel was getting his master's in economics and he was writing a paper on whether smoking bans in restaurants and workplaces actually made people smoke less. He collected this huge data set.
5:58 Speaker 04 I had like amazing data from the CDC, which is public. I had smoking prevalence at the county level.
6:04 Speaker 03 Abel says that all the established research at the time indicated that smoking bans were hugely effective, that they'd gotten lots of people to stop smoking.
6:12 Speaker 02 But when Abel crunched his numbers?
6:14 Speaker 04 I was finding absolutely no effect, none. There was like, nobody stops smoking. I've played with the data for six months and I find nothing.
6:22 Speaker 03 And Abel was trying to make a name for himself in academia, which means getting his research published in an academic journal. And it's harder to get published if you find no effect, especially given that the existing literature did show an effect. So what Abel needed was something statistically significant.
6:40 Speaker 02 For the statistically uninitiated, significant means the result would be produced by chance less than 5% of the time. So the probability that the result is just random is 5% or less. That is the cutoff for whether your findings count or not.
6:55 Speaker 04 There's this 95%, 5% cutoff that really matters. We're obsessed with these thresholds.
7:01 Speaker 03 So Abel kept tinkering with his dataset, changing his computer code to contort the data one way and then another, until eventually, one day, he found a way to analyze one subset of his data that gave him what he'd been looking for, a result demonstrating that smoking bans had decreased smoking and a result that was significant.
7:20 Speaker 04 It was like, there you go. I was so happy. I was in the library, I just yelled, like, "Sing it for Karen." I was so happy.
7:25 Speaker 02 Finding a significant result meant that if his paper was published, he would get to put a little asterisk or star next to his results. And the more statistically significant the result, the more stars you got to claim.
7:38 Speaker 03 But Abel's happiness did not last long because the more he thought about how he'd gotten that significant result, the more it started to seem like it was working against the whole goal of social science, you know, to actually discover true new knowledge about human behavior.
7:53 Speaker 02 For example, policymakers need to know whether smoking bans work to make sound policy decisions. But here he was torturing the data to match the preconceived hypothesis. He thought-
8:06 Speaker 04 This is so stupid. What am I doing? I'm writing a piece saying that smoking bans are decreasing smoking prevalence because I managed to find one that worked. I was like, this is dumb. I'm doing something wrong.
8:17 Speaker 03 Abel ultimately decided not to use his tortured results. He wrote up his paper showing that he'd found no effect, even if it meant his paper was less exciting. And at first, he thought what he'd done to his data might have just been a one-off mistake on his part.
8:31 Speaker 04 But then you start talking to other students and people are like, oh, yeah, that's how you publish.
8:36 Speaker 03 Abel started to see that this was a problem of incentives. In order to advance their careers, academics have to publish papers in peer-reviewed journals. And the journals want to publish work that's statistically significant and novel. These papers can win big prizes and define new research agendas for decades.
8:55 Speaker 02 But because of all that, people were doing what he had done, trimming and squeezing and coaxing the data towards significant results. And that can easily cross over into a kind of data manipulation called P-hacking, P as in probability. And Abel says it can happen almost subconsciously.
9:14 Speaker 04 Because the project took like three, four years of back and forth between co-authors, discussion. Then six months later, you go back, you exclude again these other people, or you do something different. And then over time, all these decisions, actually, when you look at it from the outside, it's like, this is crazy what you've done.
9:31 Speaker 03 To figure out how widespread this problem might be, Abel decided to research the research. He and a couple of his colleagues scraped the significance data from a bunch of the top academic journals, the distribution of stars that published researchers had racked up. And when they looked at the distribution, they found a noticeable hump just above that 5% significance threshold. Now, some of this could be because some people whose research only hit 6% didn't bother submitting. But it could also be because some researchers were tweaking their data analysis to just barely get results that would be more likely to get published.
10:07 Speaker 02 But when Abel and his colleagues started submitting the research for publication, they got a resounding series of no's. Academic publishing seemed hesitant to open up an empirical reckoning. After a few years, they did manage to publish their paper in 2016. They called it "Star Wars: The Empirics Strike Back." Do you get it? Oh, you definitely get it. Thank you, Alexi. So Bell puts aside this whole idea of an empirical reckoning, and he moves on to other economic projects. He gets tenure, and eventually he learns that his little paper has become kind of a sleeper hit.
10:44 Speaker 04 It took a long time before I realized, actually, the paper was like, well-known. Before people started talking to me at conferences, like, are you the Star Wars guy? That's the moment, like, I needed someone senior to tell me, like, no, this is really important, what you're doing.
10:56 Speaker 03 There had been efforts to solve parts of the replication crisis. Some of the top journals had started asking their contributors to release replication packages with their papers. That's basically the data and code they'd use to find their results. And researchers were also starting to preregister their hypotheses before actually doing the research. So that if the data didn't support it, they couldn't futz around and pretend like they'd been looking for something else all along.
11:21 Speaker 02 For his part, Bell wondered if there was anything he could do, like, not just study the problem, but actually help fix it.
11:27 Speaker 04 How do I change the incentives? How do I potentially have an impact on the norms? How people do research? The second I think about the norms, I think about, oh, it needs to be large-scale. Nobody's going to change their behavior if it's a small-scale thing. So it needs to be big.
11:45 Speaker 03 Journals do have peer review systems, where they try to poke holes in research. But they didn't always totally get under the hood to scrutinize all the code and data. So researchers weren't necessarily worried that their stuff would get checked.
11:58 Speaker 04 A nice analogy, I think, is imagine you go on a date. You might shave, you might take care of your body, you might take care of yourself, a bit of deodorant, you know, perfume, maybe, if it's your thing. You're going to make an effort to look prettier than you are usually. The other person fully understands that this is a nice version of you. We're fully aware of that, but I don't know about you much. And perhaps it's not. Or maybe you made a massive effort, and usually you're a disaster, you never clean nothing. So when, you know, you go to the apartment, it's like, oh, my goodness, this is your apartment? So research is a bit like this.
12:38 Speaker 02 The published research is the cleaned-up version.
12:40 Speaker 04 So when I see a published paper, I know it's been, you know, it's beautiful, it looks nice. But there's an information asymmetry. I don't know how dirty it is, actually.
12:53 Speaker 03 Abel thought one thing that might help this problem was to make researchers care as much about the cleanliness of their data analysis as the significance of their results. And to do that, he'd have to go full-on room raiders on people's published papers to shine a fluorescent spotlight on the backrooms of their research. If you could take all of the data that somebody had gathered for a given paper and meticulously retrace their coding steps, you could see if it was possible to replicate their findings. You could make sure there weren't any errors, conscious or unconscious, in what they'd done.
13:25 Speaker 02 But first, he'd have to get the code. People weren't in the habit then of publishing all their data and code. And when he emailed researchers asking, nobody responded. So he decided to create an official-seeming institution.
13:39 Speaker 04 It needs to be a big institution with a website with tons of famous people on it. And when you send the email, people would be like, what the hell is this thing? I need to respond. It's legit.
13:50 Speaker 02 So in 2022, he creates a website for a thing he starts calling "The Institute for Replication."
13:56 Speaker 04 A friend of mine, his wife, did the logo for free, like a design. Like, you know, I mean, like, just bare bones.
14:02 Speaker 03 He recruits some serious, famous economists for the board to put on his legit-looking website. And pretty soon, he does start to get responses to some of his emails. He's able to get some data sets and coding packages, and he convinces some colleagues and junior researchers to start doing some replications one by one in exchange for a co-author credit on one big paper.
14:23 Speaker 02 So Abel can get the data and the code. But there's still a second problem, which was the question of scale. Replicating one paper at a time was not going to do much to change the system. What he needed was to create the sense within the academic community that anybody's work could be checked at any time.
14:41 Speaker 03 It's like an IRS for the ivory tower.
14:44 Speaker 04 So now I thought, OK, we need to mass-reproduce journals. So then I was like, OK, I need to get maybe a few hundred replications or productions per year. And I'm thinking, how do you do that?
14:54 Speaker 03 The answer, Abel says, came to him kind of by accident. Around the time he got his Potemkin website up and running, he got an unrelated invitation to a conference in Oslo, to a couple of seminars. He was planning his trip about a month ahead of time, and he noticed that he had seminars on a Wednesday and on a Friday.
15:11 Speaker 04 And I thought, like, what the hell am I going to do on Thursday? Like, I've never been to Oslo. I'm sure it's pretty and nice. But a full day, like, I'm gonna walk around, and then I'm gonna have like six, eight hours just to relax. So I just emailed the person who invited me, and I said, could we just, like, do a small workshop?
15:28 Speaker 02 It would just be like 10, maybe 15 people. Abel posted about it on social media.
15:34 Speaker 04 You can come to Oslo. It should be fun if you come. You're gonna get co-authorship to a meta-paper. We're gonna reproduce papers. Let's have fun. And then, I don't know, like, 70, 80 people ended up registering really fast. I closed registration because I have no money. We don't have food. I didn't tell the guy it would be 80. I said it would be 10.
15:53 Speaker 03 So Abel is sitting there, a couple months before the conference, with this sudden, unexpected surge of interest and no plan.
15:59 Speaker 04 I have 80 people. Some coming from Ireland, others coming from Sweden, others coming from France. Like, what do I do with these people?
16:06 Speaker 02 He starts collecting papers that people could replicate, and he puts everyone into teams by their field: health economics, development economics.
16:13 Speaker 04 The first time, I had no idea what was going on. I was super stressed.
16:16 Speaker 03 He had no idea what was gonna happen, what they would find. Abel heads to Oslo and convenes the first-ever replication game in October of 2022. And when he checks in on one of the first teams of replicators working on the first paper...
16:30 Speaker 04 I go talk to them, and they're like, Abel, there's a problem. Like, there's tons of duplicates. I'm like, what? He's like, hey, one of the data sets, like, there's tons of people with the same age. And then I come back later on, and it's like, okay, 75% of one data set. Everybody's 60 years old, all women, all living in the same village, all doing the same thing. It's the same, it's duplicates. And it's a bit about inequality. If everybody is the same, there is no inequality. And that was driving some of the mechanism.
16:59 Speaker 02 The underlying data upon which this entire paper rested had been merged improperly, like a big copy and paste error. To Abel, this was disconcerting.
17:10 Speaker 04 And I was like, oh, boy, that's the first paper. That's the first game. What did I create? It's going to be like this all the time. People finding crazy mistakes. And did I just open a can of worms that actually most papers are just like terrible, full of crazy collars?
17:25 Speaker 03 Abel was a little afraid he might be about to discover that all papers were full of worms and that science wasn't real. But luckily.
17:33 Speaker 04 By the end of the day, like many teams had like good day. Everything was clean and so on. And it was, oh, it's like not terrible.
17:40 Speaker 02 He could relax. It turns out most of the papers were not terrible. Even better, with that first event in Oslo, Abel had found a way to crowdsource this massive academic auditing project, essentially for free. If he could host enough replication games every year, he just might be able to scare the social sciences into acting right.
18:04 Speaker 03 But what actually happens on the ground during these things? After the break, we enter the 51st replication game.
18:18 Speaker 00 Stay tuned for the following announcements and messages from our sponsors.
18:22 Speaker 05 This message comes from Capella University. You know that feeling when there's a spark building inside you? That you were meant for more? That's your own drive, pushing you towards what's next. Capella University gets that. With their FlexPath learning format, you can set the pace and earn your degree without putting life on pause. You've built experience and know what you're capable of. Now, this is your time to turn that momentum into more. The only real question is, what can't you do? Learn more at capella.edu. This message comes from Apollo. The world's leading companies don't need off-the-shelf financing solutions. That's where Apollo comes in.
19:04 Speaker 05 Apollo provides access to long-duration capital and a broad range of financing capabilities, helping companies develop customized financing solutions designed around their strategic objectives and structured at an efficient cost of capital. Learn more at apollo.com/capital. This message comes from Rula. World Mental Health Day takes place in October, making it a great time to take that next step in getting support if you need it. Rula makes accessing quality mental health care affordable with sessions costing an average of $15 with insurance. So turn off the talk track that's keeping you from progress and head to rula.com. That's R-U-L-A dot com to find a therapist the easy way.
19:49 Speaker 02 So we are at a replication game in real life in Montreal. Abel Broder says that the game part is a little bit of a branding exercise. There are no winners or prizes. It's more like an all-day hackathon.
20:02 Speaker 03 The teams are mostly economists with a few groups of psychologists, and they've already chosen the papers they'll focus on. Using just what they have in the replication package, they will have seven hours to check the code, examine the decisions their paper's authors made, and see if the results reproduce. And then they'll report on whatever they find, so it'll be out there on the record, whether that's a nothing burger or a bombshell.
20:25 Speaker 02 After everyone claps their rendition of "We Will Replicate You," the researchers start streaming out of the lecture hall, and we run after them.
20:35 Speaker 03 Jolene! Can I talk to you for a sec? Hey. I'm Alexi.
20:41 Speaker 06 Hi, Alexi.
20:42 Speaker 03 Just set the scene for me, like...
20:44 Speaker 06 So we just finished clapping a cheesy opening song, and we're about to split up into rooms.
20:49 Speaker 02 The groups are scattering into classrooms across the building to start digging into their papers. Economics PhD student Jolene Hunt and her team are looking at a paper about education. They're all education economists. And so Jolene has sort of a pedagogical view of the day.
21:05 Speaker 06 In PhDs, you often don't get a chance to actually work together.
21:07 Speaker 03 Yeah.
21:08 Speaker 06 We're usually just kind of on your own in your silo, and then you talk to each other when you're having problems. But it'll be nice to actually work together and see if my friends are actually any good at their jobs.
21:18 Speaker 02 Rolling up their sleeves, getting down to the actual coding. Because they're only going to have seven hours, each group has a little list of the things they've decided they're going to try to get through today. There's one group led by a guy named Thibault Dupre, who is sitting alert and ready to unpack a paper about pensions in different countries.
21:34 Speaker 07 Essentially, the paper focuses on 10-something countries. But then the data set seems to have a few more countries in there. So why some countries were included, others were not. What if you drop a few countries out of the data set? Maybe there's something to be explored there.
21:50 Speaker 03 And we wanted to understand the stakes for the day. You know, why people would attend this event to do a full day of like manual economic labor for no dollars. So we asked them.
22:01 Speaker 02 What are you doing here today?
22:04 Speaker 01 Well, we're trying to see if we can replicate the results from a paper that took a look into the effects of negotiation.
22:15 Speaker 02 I've started with a group in the lecture hall, huddled around their laptops. Fréale Lasoued is a researcher at the University of Saskatchewan, and she's in a group of economists focused on agriculture with Shichiawu from the University of Ottawa. You want to find that the paper checks out?
22:32 Speaker 01 Hmm, yes, you can think like that, yeah.
22:36 Speaker 02 In terms of your personal incentives, would it be cooler to find like, oh no, this paper's messed up?
22:42 Speaker 03 Fréale starts laughing, seemingly at the premise of the question.
22:46 Speaker 02 You're laughing so hard, why?
22:49 Speaker 01 That's mean.
22:52 Speaker 01 I don't know, like, how to answer it. It's gonna be bad for Diego and Juan here.
22:58 Speaker 03 Those are the authors of the paper.
22:59 Speaker 02 Authors who know that?
23:00 Speaker 01 No.
23:01 Speaker 02 You just have sympathy for them.
23:03 Speaker 01 Yeah, because we've all been in their shoes.
23:06 Speaker 02 OK, fair. But we go up to another group, and they're kind of like, duh.
23:12 Speaker 08 Yeah, we are trying to find something.
23:15 Speaker 02 That's Felix Fosu, a postdoc at Queens University. His group is digging into a paper about cartels in Mexico. I tell him what the other researchers said, that maybe it isn't very nice to want to find something terribly wrong in someone else's research. But it seems like, to Felix, I have now misunderstood things in the opposite direction.
23:35 Speaker 08 No, we definitely want to find something.
23:39 Speaker 02 Yeah. Why?
23:42 Speaker 08 I think replication is something that we have to take very important in economics. We need to make sure that our results are indeed claiming what they claim to be. We need to know what works and what does not work.
23:55 Speaker 03 Now, regardless of their specific goals, the actual work of replication is divided into two main phases. Phase one is the same for every team: pure and simple replication. They will all check the paper's code, the programmed instructions that take some raw data and put it into a bunch of tables that comprise the foundations for the paper's conclusions.
24:16 Speaker 02 So now, each team takes the original code, copy and paste it, and basically hits "Enter" to see if it runs.
24:22 Speaker 03 And one type of mistake that they might find is if the code is really broken. They might find that when they push the button, the code just doesn't run. The computer just says "Error."
24:32 Speaker 02 Or, another kind of mistake they might find: maybe the code runs, great, but it spits out a different answer than what the authors wrote. Not so great. Or, maybe the raw data is messed up in some way, like cells merged, or transposed, or erased, or accidentally filled down the whole column.
24:48 Speaker 03 So we ask the agriculture team to show us exactly what they are doing.
24:53 Speaker 02 So I can't code. I don't know what I'm looking at. What am I looking at?
24:56 Speaker 09 Uh, well, actually, it's kind of nothing here because I just started.
25:01 Speaker 03 This is Chisia again. The paper her team picked by Diego and Juan Pablo is about the price of eggs at big firms versus small firms, how much pricing control they have.
25:11 Speaker 02 I look at her laptop over her shoulder.
25:13 Speaker 09 So what you can see here is already the variables they have. We have the firms, we have the price, we have the day, month, and year.
25:21 Speaker 02 Now, Chisia pulls out her iPad to scroll through the published paper.
25:25 Speaker 09 So we're going to firstly check whether we can perfectly reproduce all the numbers and using the original data and codes. If I can run part of this, maybe you can see it.
25:36 Speaker 02 Okay, she's pushing a little blue arrow, a little play button.
25:40 Speaker 09 So basically, if I run this code, you'll see the results.
25:44 Speaker 02 Oh, a little box appeared in a different window.
25:47 Speaker 09 Yes, so if you check the numbers.
25:52 Speaker 02 Minus 18.11432. And I'm looking at the published version. It says minus 18.114 star, star, star.
26:00 Speaker 09 So they're basically exactly the same.
26:02 Speaker 02 It's the same.
26:03 Speaker 09 Yeah, it's the same. That's good, you know.
26:05 Speaker 02 So we have a win.
26:06 Speaker 09 Yeah, yes, one. And we have more to check.
26:10 Speaker 02 A lot more.
26:11 Speaker 09 But we got one. That's great.
26:12 Speaker 03 Shishia will keep plugging in all the data and checking the results. Though so far, it looks like the paper is checking out.
26:19 Speaker 02 And if the paper passes the whole first phase, if the code does spit out all the answers that the author said it would, then the replicators move on to phase two, robustness checks.
26:30 Speaker 09 For robustness check, we kind of like changed some parts of the model to see whether the original conclusion is still kind of makes sense.
26:39 Speaker 02 This phase is less objective and requires more context and thought. It requires the economists to consider the questions that the paper authors didn't think of or didn't write about, the decisions the authors made and the decisions they could have made but didn't. It's like trying to see the negative space in and around the paper.
26:59 Speaker 03 The kind of things they might find in this phase, you know, did the authors say that this dataset represents something? It doesn't. Did they use an appropriate dataset? And did they use that data in a way that made sense? Did they include or exclude certain specifications or factors in order to have a result that looked exciting?
27:17 Speaker 02 There are infinite potential choices that researchers make or don't make. And the replicators have such limited time, so they're not going to be able to consider and analyze everything. They're just going to get through as much as they can.
27:29 Speaker 03 And as the hours start to tick by, it becomes clear that most teams are not turning up major issues. Until mid-afternoon, we check in with this one group looking at a paper about government policies.
27:41 Speaker 10 The basic premise is when people trust the government, do they tend to comply with policy more?
27:48 Speaker 03 This is Simon Privo. He's an econ master student and a public sector researcher. The paper found that when people trust in government, they comply with policies more readily. So those policies cost the government less money. And Simon and his teammates are now trying to unravel a mystery. Because when they went to look at the raw data that underlies the paper's findings, it looked a little funny. This is Scott Morier, another Econ Master student on the team.
28:13 Speaker 11 There was a folder called RAW for the raw data, but the files were all labeled clean. So we were a bit confused to how, you know, it's counterintuitive, right? So Florian downloaded the data straight from the source and followed the instructions to create the one dataset.
28:27 Speaker 02 They recreated what should be the same dataset, following the instructions that the authors left. They ran the code.
28:34 Speaker 11 And then that's when we started getting the errors because variables were missing. And then as we kept going through, we kept finding more variables that were being used in the regression but weren't necessarily included in the supposedly what is meant to be the raw dataset.
28:48 Speaker 03 Some variables are missing from the raw dataset. The authors seem to have used data in their analysis that they did not account for. Not good.
28:56 Speaker 02 And then we visited the group looking at that paper about cartel behavior in Mexico. That group has found something, too.
29:02 Speaker 08 So in this paper, they look at the presence of different cartels.
29:07 Speaker 03 They tell us the paper looks at 20 cartels and data about what types of crimes were happening and when, to see if cartels changed the types of crime they did after the government ramped up a big war on drugs.
29:17 Speaker 08 What we've found so far is that if you exclude one of the cartels, then the results become insignificant.
29:26 Speaker 02 So it's just the one cartel making the results?
29:28 Speaker 08 Cartel making the results.
29:29 Speaker 12 So if you remove only one, then the result collapse, right?
29:33 Speaker 02 Oh, no, you found something.
29:34 Speaker 08 Yeah.
29:35 Speaker 03 They found something in the first test they tried.
29:37 Speaker 02 Is that luck? Would you call that luck?
29:40 Speaker 08 No, I think it's something that we thought about it. That's why we placed it one on the list. We thought it's a good place to search. So partly luck, but partly because we thought about it carefully.
29:53 Speaker 02 That sounds like not luck. They're going to keep investigating. And depending on what they find, this paper is maybe not passing this phase, the robustness check phase. Can you draw a big sweeping conclusion about the effectiveness of a war on drugs from a change in just one cartel? They suspect this paper will not hold up.
30:12 Speaker 03 Over lunch, the cartel team starts puzzling through, like, how does this sort of thing even happen?
30:17 Speaker 12 You have to be honest. For sure, when you do this kind of papers, you do this kind of things, right? You check whether, when you have this. You know, you do these type of robustness checks.
30:28 Speaker 03 David Benatia, a professor on the team, says this is a robustness check that he would have tried if he had been the author. At the end of the day, our researchers limp back into the auditorium to present what they'd all found.
30:40 Speaker 04 So the way we like to finish is to give each team about one minute to tell us how your day went, the different challenges you faced. Maybe we can start from the beginning, move around.
30:53 Speaker 06 We didn't find anything too major. There was a lot of missing variables and attrition.
30:58 Speaker 10 It replicated well, like all the code ran, but...
31:02 Speaker 12 Everything ran fine. We tried to poke holes in it, but we couldn't really do it.
31:06 Speaker 03 For the 71 replicators in the Montreal game, 14 teams got to uphold science by double-checking some published work. They spent a day coding with their friends and peers, learned some new coding hacks, and new ways to make choices in research. And they'll get a little authorship credit on a meta-paper in a real journal.
31:24 Speaker 02 The other two teams, the group who discovered the missing numbers, the Cartels group, they've gotten like a toxic golden ticket. Now they'll get to write their report, polite and formal, but nonetheless kind of a bombshell, saying just how flawed the research is.
31:40 Speaker 03 Maybe that makes a splash and everyone thinks they're brilliant, or maybe it makes a splash and everyone hates them.
31:47 Speaker 02 Next, Abel will write an email to the authors, a somewhat standardized note saying, "Hey, here's who we are and what we do. We found some mistakes in your paper. Would you like to respond?" He does not assume nefarious intentions, and the authors get an opportunity to try to fix the problem and prepare their formal response before anything goes public.
32:09 Speaker 03 And because Abel handles it from his position at the Institute for Replication, it doesn't feel so personal, and the replicators have a little bit of insulation.
32:18 Speaker 02 We asked Felix from the Cartels group what this might mean for him as a more junior person, a person earlier in his career. It's kind of throwing rocks towards the top of the profession. He'd wanted to find something, and now he has.
32:30 Speaker 08 I think it's a good work that we are doing, but what the implications are, I don't know.
32:40 Speaker 05 Yeah.
32:42 Speaker 03 So after a few months, Abel sends his neutral-toned official email to the authors of the paper that Felix and his team had replicated in Montreal, saying that the code had worked but that they found the results don't hold up.
32:55 Speaker 02 And for the authors of that paper, getting that email?
32:59 Speaker 10 When we opened that email, we were actually happy because we actually read your paper replicates.
33:05 Speaker 02 This is Giacomo Battistone, a researcher at the Rockwell Foundation in Berlin and one of the four co-authors of the paper. He says they were thrilled to have their coding results publicly validated.
33:17 Speaker 03 And when it came to the bigger problem, the fact that their results had fallen apart when the replicators removed that one cartel?
33:23 Speaker 10 We were not particularly worried about the content because it was kind of self-evident that this was not really challenging.
33:31 Speaker 02 Not really challenging their findings because they think the replicators misunderstood the basic hypothesis of their study. They say they started with this idea that there was this one big new cartel in Mexico, Los Cuidados, and it had been doing a lot of crimes, generating a lot of data points. Here's another author, Marco Lemolié, a researcher at Bocconi University in Milan.
33:53 Speaker 13 When we start to think about this project, actually, we had in mind the specific cartel of Los Cuidados.
33:59 Speaker 03 They say they set out to investigate if the cartel, Los Cuidados, had changed the types of crimes they did after the war on drugs. And their papers succeeded at proving that.
34:09 Speaker 02 What the Montreal replicators did, in the opinion of the paper authors, was to remove the main part of the dataset and then say the conclusion was broken. You can do that, but why would you?
34:20 Speaker 13 To be blunt, it doesn't make any sense.
34:22 Speaker 03 That is Paolo Pinotti, a professor also at Bocconi University. He said it was like doing a study on the effect of spreadsheets on productivity and then saying, oh, but the results don't hold up if you exclude Microsoft Excel.
34:35 Speaker 02 We looked at their paper. And to be fair to the replicators, the original paper does not say explicitly, hey, it's just Los Cuidados we're focusing on. The data from Los Cuidados is lumped in with several other new cartels. So if the paper authors meant to study the behavior of just Los Cuidados... That was never quite spelled out.
34:56 Speaker 03 Mary, when we first rocked up to the replication games back in May, I think we were both excited at the idea that we might watch some junior economists uncover some major problem with a published paper in real time. But Abel had a different take when we asked him about the problems that the teams there had uncovered, like the team, for example, that had found issues in the government trust paper. That seems like success.
35:21 Speaker 04 Success, it depends what you define as success.
35:23 Speaker 03 Well, the process working as it's supposed to.
35:26 Speaker 04 I mean, in a world in which science works, I think this should have been picked up before it's published, cited and disseminated.
35:36 Speaker 04 So I don't think it's a success.
35:37 Speaker 03 That's fair.
35:38 Speaker 02 These papers they're replicating have been published, meaning they got past journal referees, professional economists who are supposed to be gatekeeping the quality of what they publish. Some of the top journals do check that the code runs, they press play. But in the government trust case, the journal referees apparently didn't catch that numbers were missing, that when the paper said, oh, the documentation is in the replication package, it was pointing to nothing.
36:02 Speaker 03 The journal declined to comment, though they said they have a robust process to investigate concerns.
36:09 Speaker 04 To me, this is a failure of the system, which is fine, there's always going to be failures. I just think that the rate of failures is higher than what a lot of people think.
36:17 Speaker 03 Yeah.
36:18 Speaker 04 And it shouldn't happen that often.
36:21 Speaker 03 In every replication game so far, they have found something, though not yet any career-ending fraud. It's more like major data or coding errors or robustness fails.
36:32 Speaker 02 So the broader system is still broken, even after putting on more than 50 games and replicating about 300 papers.
36:39 Speaker 03 Still, there are signs that the games are having an effect. Several replication gamers told us their experience here will change how they do their research, because they know that their papers too might someday end up under Abel's spotlight.
36:54 Speaker 02 Abel says the more games he can put on, the more the rest of the academic world will start to shift. Because the evidence shows that people don't actually change their behavior based on the severity of the potential punishment, like losing their job or public shaming or whatever. They change behavior based on the odds of enforcement, the odds of actually getting caught. Just the idea that someone might walk through their apartment one day? That's enough of a threat to keep it clean.
37:28 Speaker 02 Hey, listeners. What are you doing on the evening of Monday, April 6? Are you free? Because if you are, I think you should come to the 92nd Street Y to hang out with me and some of my friends. It is the first debut stop on our 12-city book tour to celebrate the publication of our first-ever book, Planet Money: A Guide to the Economic Forces That Shape Your Life. Every stop on this tour will be unique with different hosts and guests. And if you get a ticket, you can get a tour exclusive tote bag with your purchase while supplies last. So at the 92nd Street Y on Monday, April 6, it'll be me, Amanda Aronchik, Darian Woods, book author Alex Miyasi, and The Economist Emily Oster, who is most famous, I think, for letting pregnant women know that they can actually drink coffee.
38:11 Speaker 02 So please come and bring your very best economic questions for us. We can't wait to hang out. Find the show nearest you at the link in show notes or go to planetmoneybook.com. And thank you.
38:20 Speaker 03 If you want to hear more about the replication crisis, we've done a few episodes about it and the efforts to fix it. We'll link to those in the show notes.
38:27 Speaker 02 If you want to support our work, you can donate at npr.org/donate. And thank you.
38:32 Speaker 03 This episode was produced by Emma Peaslee and James Snead with help from Willa Rubin. It was edited by Jess Jang, fact-checked by Sam Yellowhorse Kessler, and engineered by co-Takasugi Chernowin. Alex Goldmark is our executive producer. I'm Alexi Horowitz-Gazi.
38:46 Speaker 02 And I'm Mary Childs. This is NPR. Thanks for listening.
38:53 Speaker 00 Stay tuned for the following announcements and messages from our sponsors. This message comes from Capella University. That spark you feel? That's your drive for more. Capella University's FlexPath learning format lets you earn your degree at your pace, without putting life on pause. Learn more at capella.edu. This message comes from Mint Mobile. While others have been hiking your bill, Mint Mobile has kept their wireless plans starting at $15 a month. Switch to Mint Mobile and get three months of premium wireless service for $15 a month. Go to mintmobile.com/switch. Upfront payment of $45 for three-month plan required. $15 per month equivalent. New customer offer, first three months only. Then full-price plan options available. Taxes and fees extra.
Previous The ICE hiring boom Next The Business of Heated Rivalry Page updated October 31, 2026 · transcript and summary are machine-generated