596. Farewell to a Generational Talent | Freakonomics Radio

Daniel Kahneman left his mark on academia (and the real world) in countless ways. A group of his friends and colleagues recently gathered in Chicago to reflect on this legacy — and we were there, with microphones. FOLLOW FREAKONOMICS RADIO: YouTube: https://freak.ws/3yIl6dl Apple Podcasts: https://freak.ws/3yAvQh0 Spotify: https://freak.ws/3TsdCmV ABOUT FREAKONOMICS RADIO: Discover the hidden side of everything with host Stephen J. Dubner, co-author of the Freakonomics books. Each week, Dubner speaks with Nobel laureates and provocateurs, intellectuals and entrepreneurs, and various other underachievers to uncover things you always thought you knew (but didn’t) and things you never thought you wanted to know (but do) — from the economics of sleep to the future of education, from markets for marriage to the surprising utility of wolves. ABOUT THE FREAKONOMICS RADIO NETWORK: Freakonomics began as a book, which led to a blog, a documentary film, more books, a pair of pants, and in 2010, a podcast called Freakonomics Radio. Hosted by Stephen J. Dubner, it’s one of the most popular podcasts in the world, with a reputation for storytelling that is both rigorous and entertaining. Its archive of more than 500 episodes is available, for free, on any podcast app, and the show airs weekly on NPR stations. Freakonomics Radio is now the flagship show of the Freakonomics Radio Network, which includes the podcasts No Stupid Questions, People I (Mostly) Admire, and The Economics of Everyday Things. FREAKONOMICS RADIO NETWORK PODCASTS: Freakonomics Radio: https://freakonomics.com/series/freakonomics-radio/ No Stupid Questions: https://freakonomics.com/series/nsq/ People I (Mostly) Admire: https://freakonomics.com/series/people-i-mostly-admire/ The Economics of Everyday Things: https://freakonomics.com/series/everyday-things/ Special series: https://freakonomics.com/topics/ CONNECT WITH US: [email protected] https://twitter.com/freakonomics https://www.instagram.com/freakonomics/ https://www.facebook.com/freakonomics/ https://www.tiktok.com/@freakonomicsradio https://www.linkedin.com/company/freakonomics-media Subscribe to our newsletter: http://eepurl.com/bKm0cf

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:04Stephen J. Dubner?

  Today on Freakonomics Radio, a very special episode, a conversation about the late Daniel Kahneman, whose insights into human behavior have been threaded through this show for years. Ideas like confirmation bias and loss aversion and the planning fallacy. During this conversation, we also learn about a research paradigm that Kahneman embraced called adversarial collaboration, which means working shoulder to shoulder with your rivals.

• 0:32Speaker 01

  He felt that this is the right way to do science.

• 0:37Stephen J. Dubner?

  Kahneman was a phenomenally influential psychologist who won a Nobel Prize in economics, wrote the bestselling book Thinking, Fast and Slow, and left behind an army of collaborators, mentees, and admirers. With them, we will take a careful look at the life and mind of Danny Kahneman, starting now.

• 1:12Speaker 02

  This is Freakonomics Radio, the podcast that explores the hidden side of everything, with your host, Stephen Dubner.

• 1:26Stephen J. Dubner?

  Last month, in a sunlit auditorium overlooking the Chicago River, there was a gathering of psychologists, economists, and other social scientists. This was the Behavioral Decision Research and Management Conference. The keynote event was supposed to be a conversation with Danny Kahneman facilitated by Richard Thaler, his longtime friend and collaborator. Thaler is the University of Chicago economist who helped turn Kahneman's insights into the field now known as behavioral economics. But when Kahneman died in March at age 90, Thaler came up with a new plan for the conference. Now it would pay tribute to Danny Kahneman.

• 2:09Stephen J. Dubner?

  Freakonomics Radio was lucky enough to be asked along to moderate a couple of panel discussions about his life and work. The episode you're about to hear is a condensed version of those conversations. This all took place at the downtown outpost of the University of Chicago's business school in front of a couple hundred attendees. Some of the panelists had known Danny Kahneman for many decades. For instance, the psychologist Maya Bar-Hillel. Her father was a philosophy professor at Hebrew University in Jerusalem, where Kahneman got his undergraduate degree.

• 2:44Speaker 03

  My father taught Danny and gave him a lot of grief. But my father apparently gave just about everybody a lot of grief. He was a tough-minded philosopher.

• 2:56Stephen J. Dubner?

  And when Kahneman became a professor, one of his students was Maya Bar-Hillel, from generation to generation.

• 3:05Speaker 03

  I met Danny in the first week of my first year at the Hebrew University. He gave the introductory statistics course. He looked at us and he pronounced right away, "You are the creme de la creme." He said it in French, and we were.

• 3:28Stephen J. Dubner?

  Kahneman had grown up in France during the Nazi occupation. He survived, barely, and lived for many years in Israel before coming to the US to get his PhD at UC Berkeley. His research partner, Amos Tversky, was another Israeli psychology professor who moved to the States. Tversky died young in 1996, too early to share in what would have surely been a joint Nobel Prize. Tversky was regarded as perhaps even more brilliant than Kahneman. The two of them published many papers on judgment and decision-making, but not just in psychology journals.

• 4:10Stephen J. Dubner?

  It always struck me, just in the story of how Kahneman and Tversky research was taken hold of by Thaler and others and turned into what we now call behavioral economics, that it was very, very important that these were two psychologists who were also very mathematically fluent. If that hadn't been the case, and if the publication hadn't been in, I guess, Econometrica and so on, was there a pretty good possibility for a counterfactual where all that research might have stayed within the realm of psychology and never trickled over and we might not have what we think of as behavioral economics?

• 4:42Speaker 03

  It was not an accident. It's not how fortunate that they went to Econometrica. They realized that their work was attended to primarily by psychologists. And in fact, they both considered themselves all their lives as psychologists, but they also realized that their research was perhaps more important outside of psychology. So the decision to publish their paper in Econometrica was a deliberate move. It was a strategic move to get their attention. They believed that that was the ticket, and without the ticket, they would not have been playing in that field.

• 5:20Stephen J. Dubner?

  A lot of the early Kahneman-Tversky work centered around an observation that may seem obvious in retrospect, but at the time had not been explored with much rigor. The idea was that we are all constantly making decisions, personal, professional, political decisions. And then later, if we ask ourselves why we made a given decision, which, by the way, we usually don't ask, we might tell ourselves a story about making the decision. But these stories are often not quite true. Why? One reason is that we want to appear to others and maybe even ourselves as smarter than we are. Here is Richard Thaler.

• 6:05Speaker 01

  I met Danny and Amos in 1977, and it was a transformative year for me. I decided to change jobs and took a job at Cornell and decided to offer a course in the thing I was now fascinated by and called it behavioral decision theory, which is kind of what the name of this field used to be. I got about eight students.

• 6:33Speaker 01

  So I had to think of something to do to increase the enrollment in the class. And so I changed the name to managerial decision making. Fifty students show up. I began the class by asking, how many of you signed up for this class because of the name? No one raised their hand. I said, well, actually, nearly all of you.

• 7:04Speaker 01

  That's one illustration of what Danny's talking about. No one thinks they would be stupid enough to sign up for a class based on the name.

• 7:15Stephen J. Dubner?

  I believe it was in Thinking Fast and Slow where Danny wrote, not only are we blind, but we are blind to our blindness.

• 7:21Speaker 01

  Yeah, and you know, maybe one of the many secrets of the Kahneman Tversky collaboration was they were not blind to those. And what they had was a mistake detection facility. They had some radar where they could anticipate what the mistake is. And because they had this mistake detection facility somewhere, they were able to figure out things. And then they were pretty good at predicting what people other than Amos and Danny would do.

• 8:04Speaker 01

  To me, the big point, the aha point that I got from reading their judgment papers was the idea of predictable bias.

• 8:15Speaker 03

  May I add something to that?

• 8:17Stephen J. Dubner?

  Yes, please.

• 8:18Speaker 03

  Not only predictable, I think now I'm going to say something that is perhaps, I can't say that I heard them say it, but I hope they would both agree. The errors are not just predictable. The errors are smart. Our stupid mistakes are evidence of the wonderful human mind. That's how normal cognition functions and normal cognition is amazing.

• 8:45Stephen J. Dubner?

  I love that. It does make me want to ask any, all of you, a question that's fairly heretical, which is, you know, as a layperson, I read these findings. They're attractive, they're believable, they're identifiable. But the thing that I always struggle with or would like to understand better is how you all can feel so confident that they're generalizable. There must be people in the world who are not susceptible to loss aversion or recency bias, or many, many of them. So really, the question is, how much variance is there? And how do you measure the variance?

• 9:20Speaker 01

  You know, Amos used to have a joke that there were species that didn't exhibit loss aversion and they're now extinct.

• 9:36Stephen J. Dubner?

  Think about what Maya Bar-Hilal was saying there about the smart errors we all make. It would be easy to overlook the baseline insight that Danny Kahneman offered, that people make thinking mistakes all the time. Now, most of us, upon hearing this, might say, no kidding. Our species is highly fallible? Who doesn't know that? But it is our fallibility, Kahneman realized, that makes us interesting and worthy of inspection. He accepted that humans are capable of wonderful things as well as terrible ones, but that we are overconfident in our abilities, that we often have poor self-control, and that

• 10:20Stephen J. Dubner?

  we employ an arsenal of mental shortcuts or heuristics to make decisions or reach judgments that often turn out poorly. And even when presented with evidence of our mistakes, we usually fail to change our minds. Here is Eldar Shafir, who runs the Kahneman-Treesman Center for Behavioral Science and Public Policy at Princeton. He and Kahneman used to co-teach a class.

• 10:47Speaker 04

  One lecture that I loved that Danny used to give in our course, we would talk about people's not good sense about conditional probabilities.

• 10:56Speaker 04

  And then we had a clip of the OJ Simpson trial, where Dershowitz explains to the jury that the probability that a beaten woman is going to be killed by her beating partner is extremely low, which is true. However, we're not predicting the chance that Nicole Simpson will be killed. She has been killed. The probability that a beaten woman who has been killed was killed by her partner is immensely high. That nuance of not being sensitive to these conditionals has major implications.

• 11:33Speaker 04

  He was so genuinely curious and intellectually alive. He presented the same way to the Swedish monarchy and to an undergraduate. He just wanted to think about it. He wanted to struggle with the question. He wanted to listen and think of the best theory. That was what's so impressive. When we taught together, we took turns lecturing. Here was Danny, I mean, before and after the Nobel. And every class, he would come and sit down with the students and listen. He could easily not have shown up. It's other people's lecturing. He was always there, asking questions, answering questions, devoted to understanding, always had a new thought about something that we have done on slides for three years in a row.

• 12:12Speaker 04

  And many things were written about Kahneman. I'm always amused by the fact that they say he was aware of these quirks and fallacies, but he was aware that he has them too. And that sort of implies that since he was aware, it wasn't as big a deal. There's a couple of recorded interviews with Danny during COVID, where he predicts that COVID means life imprisonment for the more advanced in age. And, you know, he was wrong. He lived a couple of wonderful years after COVID. There was a sense in which he really sank deep into the same error that he was able to predict and acknowledge.

• 12:46Speaker 01

  The other amazing thing about Danny is when he won the Nobel Prize, you have to give a lecture. And he gave a lecture and decided he was going to give a new interpretation of all of his work with Amos, using this two-system approach. I kept saying, Danny, you have two months to do this. Why are you starting over? They gave you a Nobel Prize for what you did. You don't need a new reinterpretation. But that lecture in Stockholm.

• 13:22Speaker 01

  Was the beginning of what led to Thinking Fast and Slow because he was reinterpreting all the heuristics and biases stuff in the lens of System One and System Two, which is your immediate reaction versus when you sit down and think about it.

• 13:43Stephen J. Dubner?

  What this story suggests to me, but tell me if I'm wrong, is that he understood that having an audience like the Nobel Committee audience and the king or queen or whatever of Sweden, and that his voice was now amplified, that he wanted to take the work that he'd done and make it accessible, make it known to people who have some levers of power. He wanted to exploit a good opportunity.

• 14:06Speaker 01

  I don't like the word exploit.

• 14:08Stephen J. Dubner?

  Uh, exploit for pro-social reasons, a good opportunity.

• 14:12Speaker 01

  Thank you, thank you. Yeah, I think he had an audience and wanted to get it out there, but it was torture for him. Absolute torture. I mean, during the writing of Thinking Fast and Slow.

• 14:29Stephen J. Dubner?

  Which was a long gestation.

• 14:31Speaker 01

  Long gestation. And he decided absolutely positively that he was quitting. Two dozen times?

• 14:41Speaker 04

  You know, Danny in some sense really was a bon vivant. I mean, he enjoyed life, he loved a good restaurant, he loved a good vacation, and I think he loved getting the prize.

• 14:50Speaker 01

  I don't think he changed. I mean, yes, the world was paying more attention to him, but he was always Danny.

• 14:58Speaker 03

  He changed.

• 15:00Speaker 01

  He was happier. Okay.

• 15:02Speaker 03

  No, I would like to say-

• 15:03Speaker 01

  If Maya says that, I accept.

• 15:05Speaker 03

  I would like to say in what way? He relaxed.

• 15:10Speaker 01

  That was relaxed?

• 15:13Speaker 03

  Yes, yes.

• 15:15Stephen J. Dubner?

  What do you mean by that, Maya?

• 15:16Speaker 03

  He had made it. There was a common misperception that the Dvorsky-Kahneman collaboration was not symmetrical. The world seemed to think that Amos was the lead and Danny was an outside visitor. It was a symmetrical and equal collaboration, but the world didn't know it. It was very important to Danny to be able to do good work after Amos dies so that people won't continue with this error that he was less than equal.

• 15:53Speaker 03

  So the prize told the world, this man is nobel worthy and he did excellent work because he had an excellent mind. But also, it was important for him that Kahanneman not die the name together with Dvorsky. And it didn't.

• 16:15Stephen J. Dubner?

  As I said earlier, the main ideas of Danny Kahneman have been threaded through the fabric of Freakonomics Radio over the years. Episode 323, for instance, which was about the planning fallacy. It's called, Here's Why Your Projects Are Always Late and What to Do About It. Then there's episode 271, The Men Who Started a Thinking Revolution. That was an interview with Michael Lewis about the book he wrote on the Kahneman-Dvorsky partnership. Lewis's book is called The Undoing Project: A Friendship That Changed Our Minds. So, yes, you may be familiar with Kahneman's greatest hits, but one of the conversations we had at this conference went into an area I knew very little about, and I bet you don't either.

• 17:00Stephen J. Dubner?

  That's coming up after the break. I'm Stephen Dubner, and this is Freakonomics Radio.

• 17:14Speaker 01

  Good morning. Good morning. We have a session this morning, as you know, on adversarial collaboration.

• 17:24Stephen J. Dubner?

  Day two of the conference with a panel devoted to a different way of doing business in the behavioral sciences. Let's begin with a short clip from the late Danny Kahneman. This was recorded in 2022.

• 17:38Daniel Kahneman?

  Controversy is a terrible way to advance science. It is normally conducted as a contest in which the aim is to embarrass. The feature that makes most critiques intellectually useless is a focus on the weakest argument of the adversary. It is common for critics to include a summary caricature of the target position, refute the weakest argument in that caricature, and declare the total destruction of the adversary's position. It's rare for anyone to concede anything. Doing angry science is a demeaning experience.

• 18:13Stephen J. Dubner?

  Doing angry science, Kahneman came to believe, was a terrible thing. He knew this firsthand. Before the accolades, the prizes, the acceptance, Kahneman and Tversky came under a great deal of criticism.

• 18:30Daniel Kahneman?

  Gigerenzo published his first critique of our work 37 years ago, and he's still not done with me.

• 18:36Stephen J. Dubner?

  And here again is Richard Thaler.

• 18:38Speaker 01

  I've certainly had my share of angry science exchanges. They're no fun, and there's never any light, only heat.

• 18:47Stephen J. Dubner?

  And that's why Kahneman came up with a different model, what he called an adversarial collaboration.

• 18:54Speaker 01

  He felt that this is the right way to do science. All of the collaborations shed more light than heat. The typical debates in academia are exactly the opposite. So I think Danny has a good point.

• 19:10Stephen J. Dubner?

  Kahneman was trying to solve a fundamental problem in social science research. It's natural for scientists to disagree with one another, but there's no clear mechanism for resolving those disagreements. So what if researchers who disagree work together in good faith to resolve the issue? And what if they take on a neutral third party to serve as an arbiter? That is what Kahneman called an adversarial collaboration. He participated in a variety of them over the years. The conversation you are about to hear includes some of the key players. Let's start with introductions.

• 19:50Speaker 06

  I'm Tom Gilovich. I'm a social psychologist at Cornell University.

• 19:54Speaker 07

  I'm Barbara Millers from the University of Pennsylvania. And I was fortunate to be the arbiter in two adversarial collaborations with Danny 20 years apart.

• 20:06Speaker 08

  I'm Matt Killingsworth. I'm also at the University of Pennsylvania. And I study human happiness in all being, like what makes life worth living? And how do we collect data to try to understand what makes life better?

• 20:16Speaker 09

  My name is Shane Frederick. I am a professor at the Yale School of Management.

• 20:20Stephen J. Dubner?

  Tom Gilovich, let's start with you. Let's hear basically the story of the adversarial collaboration with Danny. It began quite early in your career. Give us the whole story.

• 20:32Speaker 06

  I had published with Vicky Medwick a chapter in a book on counterfactual thinking. When I wrote the chapter, I don't know if I was aware that Danny was going to write the wrap-up chapter where he comments on all of the other ones. So I don't know if his voice was in my head as we're writing the chapter.

• 20:51Stephen J. Dubner?

  What's the chapter about? What's it cover?

• 20:53Speaker 06

  A lot of it covers the subject of regret, what we regret most in life. Amos and Danny had published a study showing you regret things of action more than inaction. The example they used is imagine you own stock in one company. You think, okay, that's run its course. You switch to another and it tanks, you lose a certain amount of money versus you're thinking about buying this stock, you decide not to, it takes off and you lose the same amount of money. Which would you regret more? And almost everyone anticipates that you would regret the action the most. Nonetheless, if you ask people, what are the biggest regrets in your life? What dominates are things they didn't do.

• 21:35Speaker 06

  And so a lot of the chapter was, how do you reconcile those two things?

• 21:40Stephen J. Dubner?

  In approaching this riddle or puzzle, in your mind, how did you measure? What was your actual research?

• 21:47Speaker 06

  Our research was, what's responsible for that difference? How do things change over time? Precisely because your regrets of action really hurt, you do things about them. Sometimes you change your life accordingly, or you certainly engage in lots of psychological work to achieve some level of peace. Because the inaction regrets gnaw at you less powerfully, you sort of leave them alone. The fun of that research was tracking down all of the different psychological processes that make those regrets of inaction hang around, or in some cases even get more intense, whereas the regrets of action dissipate.

• 22:31Stephen J. Dubner?

  How do you do that, though? Is it empirical? Is it theoretical?

• 22:34Speaker 06

  It's empirical. For example, one of the mechanisms that we thought was especially interesting is when you don't do something that you later on end up regretting, there are reasons for that. Like, I just don't have the bandwidth right now to take this project on, et cetera. Those are compelling reasons in the moment for why you're not doing this thing. You move along in time, you look back and you think, wait a minute, I could have done that. So we would do studies where we'd ask Cornell current students, recent alums, or much older alums, imagine there's a class that you always wanted to take, but maybe you were a little afraid to. Suppose we added that to your curriculum this semester. The students are like, oh, I can barely hang on right now.

• 23:18Speaker 06

  You add that, it's a catastrophe. You ask the people who graduated many years ago, they think, oh, that's just a piece of cake. It wouldn't have interfered with my GPA, my social life, the amount of sleep I get, et cetera.

• 23:29Stephen J. Dubner?

  Okay, so you write up the findings. At what point then, and in what way, do you hear from Danny?

• 23:35Speaker 06

  It's sort of a highlight of my career. I was on sabbatical, check in your voice messages back at Cornell, and I get this message, Tom, this is Danny Kahneman. I just read your paper with Medvick. It's brilliant. It made my day, just calling to say thank you. What a piece of work. And then I stupidly press star, delete, instead of saving that message forever to play to my grandkids. So I'm just on a high for a while. Two weeks later, there's another phone call from Danny Kahneman, this time live. Oh, great, my new best friend is gonna tell me about... Tom, I've been thinking more about your chapter.

• 24:23Speaker 06

  It's all wrong.

• 24:26Speaker 06

  I've heard you're planning to publish this. You can't do that. If you do, people will go after you. And if no one does, I'm gonna go after you.

• 24:38Speaker 06

  What happened to brilliant?

• 24:39Stephen J. Dubner?

  Did he, in that conversation, give you his argument for why he thought now you were wrong?

• 24:44Speaker 06

  Yes, he said that there really isn't a change in the intensity of either action or inaction regrets. There's just a substitution of the kinds of things that you're regretting. And the regrets of inaction really aren't serious regrets. You might not even call them regrets. They're more wistful, like, oh, I wish I had learned to speak esperanto.

• 25:06Stephen J. Dubner?

  Said no one ever.

• 25:09Stephen J. Dubner?

  All right, so what happens next?

• 25:11Speaker 06

  Well, lots of panic for a few days. Vicky Medwick and I, what do we do? It can't be worse than this. And I think, I forget what, there had been an example of an adversarial collaboration. I think it was Vicky's idea, hey, we should negotiate with him. Let's see if we can do an adversarial collaboration. So we pitched it, and he was very receptive.

• 25:33Stephen J. Dubner?

  This was in what year, Tom?

• 25:35Speaker 06

  Chapter came out in 95, so it was a little before that.

• 25:38Stephen J. Dubner?

  So this was pre-Zoom, certainly, early internet. How were you communicating? And how did you negotiate the negotiation? And how did the actual negotiation happen?

• 25:47Speaker 06

  I'm glad you asked that question, because it allows me to bring up a point about Danny, that, you know, there's been all this talk about his many qualities, and they've all been well said. One thing kind of gets left out. I mean, to be at that level of success... You have to be brilliant and creative, and many people have spoken about that. You just have to be a phenomenally hard worker. And, man, he was on task all the time. It was amazingly easy to get that guy on the phone. I could get him on the phone more readily than I could get my co-author, Vicky, on the phone, and something seemed wrong about that. He was always ready to engage, and so a lot of it happened over the phone. And the idea was, no, these regrets of inaction that you say are important in people's later

• 26:31Speaker 06

  lives, they're just sort of wistful regrets. And the intense, hot emotions that come from action, those are different kinds of things. So we designed some studies where we asked people to think about the biggest regret of action or inaction.

• 26:45Stephen J. Dubner?

  You designed the studies together?

• 26:46Speaker 06

  Then we ran them together, yes. So, think of your biggest regrets of action or inaction from the recent past or the distant past. How many of these emotions do they lead you to feel? We had a set of five hot emotions, a set of five wistful emotions. We thought that those long-term regrets of inaction, some of them are wistful. I wish I had learned to play the piano. But some of them are really intense. You met the right person at the wrong time, and so you let it pass, and now you're looking back like, oh, my life would have been much better if I had acted on that. We also asked people to rate whether a set of five, I think we called them emotions

• 27:31Speaker 06

  of despair. I'm depressed when I think about this. I feel empty when I think about this. And lo and behold, Danny was right. A lot of long-term regrets of inaction are kind of wistful. But we were right, too. They also produce these big, powerful feelings of, oh, my life is not what it could have been. I feel empty and depressed, thinking about these things that I didn't do that now seem so easy to have done.

• 28:00Stephen J. Dubner?

  Did Danny, as a result, acknowledge that you were partially right, as much as he acknowledged that he was partially right?

• 28:06Speaker 06

  Yeah, and that's in the paper.

• 28:08Stephen J. Dubner?

  So truly a happy ending?

• 28:09Speaker 06

  I believe so, yeah. Certainly more happy than that second phone call that I got.

• 28:15Stephen J. Dubner?

  You then went on to collaborate with Danny several times, correct?

• 28:18Speaker 06

  Yes.

• 28:19Stephen J. Dubner?

  I'm just curious, now that Danny is gone, and since your initial contact was this adversarial collaboration about regret, is there anything that you now regret not having worked on with Danny?

• 28:32Speaker 06

  I wouldn't say that there's a particular topic in mind, but, you know, to have someone in your life like that and not take full advantage of it, I wish I had just reached out to him more to have more contact with him. One of the nice things about this event here is this family feeling. Of all the people he reached, some of whom I know well, some of whom I didn't really know at all until now, they feel like family. So, you know, often with families, you wish you'd spent more time with them.

• 29:06Stephen J. Dubner?

  Hmm, not my family, but, uh, you know.

• 29:13Stephen J. Dubner?

  The adversarial collaboration with Tom Gilovich was one of the earliest ones that Danny Condon undertook. Let's take a look now at one of the most recent with Matt Killingsworth from the University of Pennsylvania. He studied engineering as an undergraduate and got his PhD in psychology in 2012. Danny Condon got his psychology PhD in 1961. So, how did Matt Killingsworth end up in an adversarial collaboration with this giant in the field?

• 29:43Speaker 08

  Danny and Angus Steen, both Nobel Prize winners, had this conclusion that there was sort of this plateau in people's happiness once they reached $75,000 in income. And then I had published a paper that basically showed something completely different. And we tried to figure out who's right and what's the truth.

• 29:59Stephen J. Dubner?

  You say it with this sort of sang froid that, you know, we published something completely different. But as you noted, you were publishing a paper in opposition to the findings of not one, but two Nobel laureates. Were you as calm and cool about it in the moment when you decided to take up this route as you seem to be now?

• 30:16Speaker 08

  I mean, partly Danny is to blame, although he didn't learn that before he passed away. I had written this as like a subpoint to a subpoint in another paper. And actually, reading Michael Lewis's book about Danny and Amos, they were talking about how they took on existing ideas, and that was an important part of their intellectual journey. And so I sort of rolled the dice and thought, well, I'll try it. But yeah, there was definitely some part in the back of my mind of like, how is this gonna go?

• 30:42Stephen J. Dubner?

  I do have to ask, at this particular conference, which is held in celebration of Danny Kahneman's work and life, do you feel a little bit like, you know, the wolf in the hen house? Or that someone's gonna come up and shiv you in the back at the buffet?

• 30:56Speaker 08

  I hope not. No, I mean, we really had a wonderful collaboration. And I think part of what characterized that is we both just wanted to figure out the truth. I don't think anyone was attached to any particular version of reality. My sense from him is that he was a little bit irritated. Not a lot. A lot is a lot. I mean, he wanted to have a good conversation, but he wasn't.

• 31:18Stephen J. Dubner?

  For the record, that was Shane Frederick saying, "a lot." Barb Mellors, let's bring you in here. How did you get attached to this adversarial collaboration?

• 31:30Speaker 07

  Well, I was talking to Danny on the phone not too long after Matt's paper came out. And he said, "Oh, by the way, do you happen to know a guy by the name of Matt Killingsworth?" And I said, "Well, as a matter of fact, I do. And if you'd like me to be the arbiter, I'd be happy to do so." In adversarial collaborations, the arbiter is the research assistant, the tiebreaker, and occasionally the therapist.

• 32:03Stephen J. Dubner?

  So Matt, how would you characterize the response to that original Kahneman-Deaton finding about the $75,000 happiness cutoff, whether inside academia or beyond?

• 32:16Speaker 08

  I mean, it was very influential, and I mean, it's probably one of the most visible kind of and exciting findings. I think a lot of us feel like money is sort of overemphasized in our daily lives, and it kind of gives a justification for maybe caring a little bit less about it and focusing more on all the other stuff that's also really important for happiness. Particularly because money is quantitative, you can kind of think of them as like points. I'm like, well, I want to get more, but you can easily imagine a sort of trap where you're just continuously trying to get more and more and more of those points, ignoring all this other stuff, and wouldn't it be nice if we could kind of step out of that cycle? At least that's part of the reason that I think that that was so attractive. So Matt, you've rehearsed very nicely the original finding. Now bring you into the story here. What are you doing at the moment?

• 33:00Speaker 08

  Sure. So partly also due to Danny, my research program is really centered on large-scale experience sampling. And to get at what that matters in the original study, those data were collected how? In the original study, they basically asked, did you experience a lot of the following emotion yesterday? Yes or no? And then it had a series of emotions like sadness, happiness, stress, et cetera. People would either say, I did, or I didn't. And the technology or mechanism of harvesting those data was what? I wasn't involved in that data collection. I believe those were verbal phone calls and then sort of interviewing people over the phone. Does anyone here know anything different, or does that sound right as far as we know?

• 33:40Speaker 07

  That's right.

• 33:41Speaker 08

  That was said with the voice of a referee, I have to say.

• 33:45Speaker 08

  Okay, Matt, continue, please. So in my study, I'm essentially measuring people's experience right in the moment. They're carrying around on their phone an app, and I'm beeping them at random times. I'm asking them, how do you feel right now? And they're responding on a scale that ranges from very bad to very good, and the scale is continuous. So there are a couple of things that distinguish that from the earlier paper. One is that it's right at the time that people are feeling it, as opposed to the day as a whole retrospectively. And the other is that gradations of feelings are on a continuous scale as opposed to something that's, you know, binary or dichotomous. And the scale is what to what? It's zero to 10, zero to 100? Very bad to very good, and it's just continuous.

• 34:26Stephen J. Dubner?

  So you can rake it anywhere you want.

• 34:27Speaker 08

  There are hundreds of unique values.

• 34:29Stephen J. Dubner?

  Got it. And describe the differences between the pools of research subjects in the original case and in your case.

• 34:35Speaker 08

  In the original case, it was a survey by Gallup. It was either representative, or at least an attempt to be representative, whatever random digit dialing was. That's a relative strength of their paper and of that study in general. My sample, in contrast, was really a convenience sample. It's turned out to be an amazing sample of people that have really beautiful results, but it was essentially whoever was willing to sign up to try to understand their own happiness.

• 34:58Stephen J. Dubner?

  What was the recruiting mechanism?

• 35:01Speaker 08

  Largely thanks to folks like you. People end up hearing about it in the media, on the radio, reading about it. They think, that's interesting. I'd like to learn about that for myself. I'm willing to contribute to science. And it turns out that when I look at, you know, what's the distribution of incomes, for example, in my data, it's almost a perfect match for the US Census. I can replicate all kinds of things that we've seen in the literature for decades. So it turns out to be really good, but that's kind of a lucky coincidence.

• 35:27Stephen J. Dubner?

  Okay, so you gather, assemble, analyze your data, talk about from the analysis point to writing up the findings.

• 35:36Speaker 08

  What I find when I look at this relationship, plotting people's happiness in the moment versus their income, it just keeps going up. This sort of critical point where they had found in the earlier paper, this flatlining, I really see no difference at all. And when I write it up in the ultimate paper, I compare, well, what's the slope below the point where they said it stops increasing? And the point above that, the slopes differ by less than 1%. I really see no evidence for a difference at all.

• 36:02Stephen J. Dubner?

  Were you initially looking for that plateau in the data?

• 36:05Speaker 08

  I was really just trying to understand, what's the relationship between these things that are obviously important? We've never had, you know, in the moment, continuous data. And so I want to see, what does this look like? And it turns out, well, it doesn't look like what we thought it looked like.

• 36:19Stephen J. Dubner?

  "When you strike at a king, you must kill him." That's from Ralph Waldo Emerson. Matt Killingsworth did strike at the king of his realm, Danny Kahneman. But what happened next? It's coming up after the break. I'm Stephen Dubner, and this is Freakonomics Radio.

• 36:51Stephen J. Dubner?

  We've been hearing from Matt Killingsworth about his research on the relationship between income and happiness, research that disputed elements of earlier research on the same topic by Daniel Kahneman.

• 37:04Stephen J. Dubner?

  So you write up the paper, and the paper, not quite directly, but quasi-directly addresses the fact that your finding is contra to a significant earlier finding by significant scholars. What happens next?

• 37:20Speaker 08

  There's a fair amount of attention about it and so forth. Maybe a month or two afterwards, I get a note from Barb that she's been chatting with Danny.

• 37:29Stephen J. Dubner?

  He's talking about Barb Mellors, a longtime friend of Danny Kahneman, as well as a University of Pennsylvania colleague of Matt Killingsworth.

• 37:37Speaker 08

  I think both of our attitudes was we just want to find out the truth. I have no, like, personal attachment to what I found particularly. And I don't, other than his initial, perhaps irritation, I don't really think Danny did. It was really just, what do we think is going on?

• 37:52Stephen J. Dubner?

  Did you envision that perhaps you would then collaborate with Danny on a joint study?

• 37:56Speaker 08

  That certainly seemed to be the case once we got into it. But, you know, these kinds of data took me many years to collect that data for them. They were a client on an external organization with a lot of resources. And so we ended up doing it by going back and looking at my data, which existed, which made it tractable. I can sort of cut to the chase a little bit of how we did that, if you want.

• 38:16Stephen J. Dubner?

  I want to go back to Barb for just a second here. Barb, if you could give Danny's perspective or participation up to the point here where we're about to cut to the data?

• 38:25Speaker 07

  The starting point for Danny was the assumption that both data sets were valid. So how could they be so different? There must be a resolution in the data somewhere. And so the task was going to be, Matt became the research assistant and started doing all the re-analyses on his data.

• 38:48Stephen J. Dubner?

  How would you characterize the tenor of this adversarial collaboration?

• 38:53Speaker 07

  Really civil and nice and, you know, just the way it should go.

• 38:59Stephen J. Dubner?

  So Matt, you said you could cut to the chase. We're at the chase now, cut to it.

• 39:02Speaker 08

  Essentially, the resolution was if we look at the range within my continuous happiness data, what if we look at the low end, which is the part where their measure would have been sensitive, can we find a similar pattern? And lo and behold, when you zero in on that, rather than looking at the average trend, which keeps going up, you get not exactly, but very, very close to what they found, certainly much closer to what they found than to what the average trend looks like. We both found that pretty convincing evidence that, you know, what they found is true, there's nothing wrong with the analysis at all, but it was really a question of, you know, how generally applicable is that? So from low to medium incomes, the unhappiest part of the distribution rises a lot.

• 39:46Speaker 08

  And then from medium to high incomes, the unhappiest part of the distribution barely changes. But the rest of the distribution is rising steadily. And in fact, at the high end of the happiness distribution, you get an inversion of that. So rather than a rise and then a plateau, you have sort of a shallow slope and then an acceleration.

• 40:04Stephen J. Dubner?

  So in my lay mind, as I'm trying to process all this, what I'm envisioning is that there is a cohort of people who are kind of cranky. And they may experience a little bit less crankiness as income rises, but there is a range in which their temperament maybe overwhelms their income.

• 40:26Speaker 08

  If anything, there's one of the steepest slopes for the unhappiest people in the range of low incomes. So getting out of poverty, if you're really miserable, is at least correlationally.

• 40:35Stephen J. Dubner?

  That's almost a different story because escaping poverty versus going from middle to upper.

• 40:40Speaker 08

  If you, if you aren't poor and you're really miserable, at least if you sort of extrapolate from there, it's Seems to not matter past that. And probably as we speculate in the paper at that point, if you have a decent amount of money and you're really unhappy, whatever's making you unhappy probably isn't due to the lack of resources. It's something else going on in your life. Maybe it's challenges in a family relationship, or maybe you're depressed, or whatever it is, it's something going on that, you know, perhaps money isn't going to make a difference.

• 41:07Stephen J. Dubner?

  Richard Thaler, let me ask you a question. You've just heard Matt's presentation of his, you know, side of the original research and then the adversarial collaboration and then the conclusion, all of which I found honestly to be really fascinating. You know the literature well. You two have a Nobel Prize, although I have heard that was a clerical error. How would you, looking at this from above, Barbara was there as an arbiter, Matt was one side, Danny was another side but is not present, if you could give an ultimate proclamation on not just where this finding arrived, but what the adversarial collaboration produced here, what would you say?

• 41:45Speaker 01

  Look, I think this is the way it should work. This is a good story. And the world would be a better place, academia would be a much better place, if there were more of these kinds of collaborations.

• 41:58Stephen J. Dubner?

  Would it affect what we sometimes callowly call the replication crisis?

• 42:06Stephen J. Dubner?

  Barbara says, Barbara's nodding her head no, you say oof.

• 42:09Speaker 01

  Yeah, let's move on to Shane.

• 42:17Stephen J. Dubner?

  The discomfort you were hearing there about the replication crisis, that discomfort is related to a two-part series we recently produced on academic fraud, episodes 572 and 573. And while we're on the topic of discomfort, let me add one thing I'm thinking about as we're in the middle of this conversation about adversarial collaborations. Danny Kahneman is said to have introduced this concept into the realm of behavioral research. But if you're not a behavioralist, you could look at this as yet another example of how academic researchers discover something that people in the real world have been doing forever. There are

• 42:57Stephen J. Dubner?

  all sorts of arbiters and referees out there, all sorts of processes for compromise and negotiation, even war games to test one plan against another and come up with the best solution. In the same vein, some people look at the big insights that have come out of this behaviorist research as essentially Aesop's fables with a little bit of math. Earlier this year, we made a series about the late physicist Richard Feynman. He didn't think the social sciences like psychology and even economics should be called science at all. He thought they were just too squishy to deserve that name.

• 43:38Stephen J. Dubner?

  And as for psychologists discovering the idea of adversarial collaboration, well, Feynman was a junior member of the Manhattan Project, which in its quest to build an atomic bomb, brought together the most brilliant and argumentative group of scientists ever assembled. That was adversarial, and yes, it was, in the end, a successful collaboration. Anyway, back to Chicago. As Richard Thaler suggests, let's move on to Shane Frederick. He is a behavioral scientist at Yale who collaborated with Danny Kahneman on several projects and is cited throughout Kahneman's book, Thinking Fast and Slow.

• 44:20Stephen J. Dubner?

  Kahneman saw Frederick as a protege, and they had a close relationship, sometimes a bit too close.

• 44:26Speaker 09

  So I'm at 10 o'clock on a Monday, maybe like third quarter of Monday night football. I get a call. Shane, he said, are you watching the game? It was one of the rare times I wasn't. I said, no. Fantastic. I want to talk about heuristics. Another time, it was like 2 o'clock in the morning, going back and forth, back and forth, back and forth. And eventually I'm getting sleepy, and so go to bed. And I wake up in the morning, and the very next email from Danny, Shane, don't play dead on me.

• 44:56Stephen J. Dubner?

  One idea they worked on together was the cognitive reflection test, or CRT. It is meant to measure a person's ability to override their gut instinct and think more carefully about a problem. Here's a famous example. A bat and a ball cost $1.10 in total. The bat costs $1 more than the ball. How much does the ball cost? If your first inclination is to say 10 cents, congratulations, you're like just about everyone else. What the CRT measures is can you slow down and actually calculate the answer rather than just go with your gut?

• 45:37Stephen J. Dubner?

  If you can, you will find that the answer is five cents. Remember, the bat costs $1 more than the ball. Shane Frederick has spent much of his career designing such tests.

• 45:51Speaker 09

  Some of the items are novel, some I invented, many of them I didn't, I collect, some I'm revised. One goes back to at least 1919, their variants of it.

• 45:58Stephen J. Dubner?

  And they were used then in an academic setting, like in employment?

• 46:01Speaker 09

  No, just like in books of riddles, stuff that Maya would read.

• 46:05Stephen J. Dubner?

  But they weren't stumpers?

• 46:06Speaker 09

  No, no, they weren't, they weren't stumpers.

• 46:08Speaker 03

  I think I must tell the audience what a stumper is, okay?

• 46:13Stephen J. Dubner?

  That again is Maya Barhulal.

• 46:16Speaker 03

  A one-way ride costs $10. A round trip costs $20. A passenger hands the cashier $20, saying absolutely nothing. The cashier knows right away that the passenger wants a round trip rather than a one-way and change. And the question is, how did the cashier know this?

• 46:39Stephen J. Dubner?

  Does anyone in the audience who does not know the answer to this stumper previously want to take a guess to the answer?

• 46:48Speaker 03

  If you're stumped, good, because that's why they're called stumpers.

• 46:52Stephen J. Dubner?

  A stumper is unlike one of Shane Frederick's CRT questions in that your intuition doesn't produce an obvious but wrong answer.

• 47:01Speaker 03

  It only has to do with my love of riddles, which I share with my partner in this research, Shane Frederick. And we really came to this out of love of riddles, but we're both professional psychologists. So we felt like we had to approach it from the point of view of psychology. Now, then.

• 47:22Stephen J. Dubner?

  Wait, wait.

• 47:23Speaker 03

  Oh.

• 47:24Stephen J. Dubner?

  Answer, please.

• 47:25Speaker 03

  Oh, no way. No, I'm not gonna answer.

• 47:30Speaker 03

  That's not what I was paid for.

• 47:34Speaker 01

  All right, I will answer. He handed the ticket agent two tens.

• 47:39Stephen J. Dubner?

  Ah, very nice.

• 47:41Speaker 03

  Did they pay you for that?

• 47:44Speaker 01

  I'm getting paid the same as you, Maya.

• 47:50Stephen J. Dubner?

  But there was something more at stake here than answering a riddle. Bar Hillel, along with a co-author, wrote a paper arguing that cognitive reflection tests are not a good measurement of anything beyond math skills. And this argument led to an adversarial collaboration.

• 48:06Speaker 01

  When I heard that Shane and Maya and Danny were having an adversarial collaboration, I didn't know who was on which team. Maya and Danny go back 60 years. Shane is like a son for Danny.

• 48:20Stephen J. Dubner?

  The results of this collaboration have not yet been published, but Shane Frederick says that he and Kahneman essentially proved that the CRT is a good measure of cognitive abilities beyond just math skills.

• 48:34Speaker 09

  It seemed to do quite well, and it's doing well consistently. Time preferences, risk preferences, other sorts of things.

• 48:39Stephen J. Dubner?

  And how would you characterize the nature of the collaboration? How adversarial was it?

• 48:44Speaker 09

  It wasn't so adversarial between the opposing groups. It's just like everybody's fighting with everybody else about everything.

• 48:53Speaker 01

  It's more like a family gathering, right?

• 48:57Stephen J. Dubner?

  Does this suggest that an adversarial collaboration is not as useful as one might hope if the actual adversaries are not often in the ring with the collaborators?

• 49:06Speaker 01

  I mean, I don't, I don't think it will work. Do you, Barb?

• 49:12Speaker 07

  Do these things work? Well, what does work mean? It depends on your definition. If you think people change their minds all the way, no, it doesn't work. If they change their minds a bit, that's good. And it speeds up science, too. Somebody's looking over your shoulder and making sure you're not making mistakes. You're defining variables more precisely. You're designing an experiment that gets right at the core issue. It's the way to go.

• 49:45Stephen J. Dubner?

  Well, I could listen to the five of you talk for five hours. There are sessions that need to happen. This room is turning over. I thank you so much. This was a great conversation.

• 49:58Stephen J. Dubner?

  Sorry, there is a clause in the contract that failure must always have every last word, so.

• 50:02Speaker 01

  I was just gonna say something nice about you, but if you insist, I'll skip it.

• 50:11Stephen J. Dubner?

  And that was, again, Richard Thaler. I would like to thank him for putting this event together and inviting us to join. Thanks also to the other participants. It was a pretty wonderful event, and I'm glad we were able to share it with you. And if you need more Richard Thaler in your life, and who doesn't need more Thaler in their lives, we are going to publish a bonus episode very soon, an update of a great conversation I had with Thaler a few years ago. It's called People Aren't Dumb, the World is Hard. So keep your ears out for that. Meanwhile, on the next regularly scheduled episode of Freakonomics Radio, a close-up look at an industry that's all about close-up looking.

• 50:54Speaker 10

  I'd like for my glasses to have a bit of pizzazz, especially if you're wearing them every day. We're going to see half of the global population being myopic by 2015.

• 51:05Speaker 11

  When you are a vertically integrated player, you master every step within the value chain. The margins, even by luxury good standards, are obscene.

• 51:14Stephen J. Dubner?

  Eyewear is a $150 billion industry, and what you see is not quite what you get. That's next time on the show. Until then, take care of yourself, and if you can, someone else, too.

• 51:30Stephen J. Dubner?

  Freakonomics Radio is produced by Stitcher and Renbud Radio. You can find our entire archive on any podcast app, also at Freakonomics.com, where we publish transcripts and show notes. This episode was produced by Zach Lipinski, with live recording by Greg Rippon. Special thanks to conference organizers Amy Boonstra, Mark Tumelco, and Chris Partridge, as well as the Black Oak AV team. Our staff also includes Alina Cullman, Augusta Chapman, Dalvin Abouajie, Eleanor Osborne, Elsa Hernandez, Gabriel Roth, Jasmine Klinger, Jeremy Johnston, Julie Canfor, Lyric Bowditch, Morgan Levy, Neil Carruth, Rebecca Lee Douglas, Sarah Lilly, and Teo Jacobs. Our theme song is "Mr. Fortune" by The Hitchhikers. Our composer

• 52:14Stephen J. Dubner?

  is Luis Guerra. As always, thank you for listening.

• 52:18Speaker 03

  I think my mind was wandering.

• 52:20Stephen J. Dubner?

  I have that effect on people, I've been told.

• 52:29Speaker 02

  The Freakonomics Radio Network. The Hidden Side of Everything.

• 52:36Speaker 07

  Stitcher.