Predicting the Future Is Possible. ‘Superforecasters’ Know How.

Can we predict the future more accurately? It’s a question we humans have grappled with since the dawn of civilization — one that has massive implications for how we run our organizations, how we make policy decisions, and how we live our everyday lives. It’s also the question that Philip Tetlock, a psychologist at the University of Pennsylvania and a co-author of “Superforecasting: The Art and Science of Prediction,” has dedicated his career to answering. In 2011, he recruited and trained a team of ordinary citizens to compete in a forecasting tournament sponsored by the U. S. intelligence community. Participants were asked to place numerical probabilities from 0 to 100 percent on questions like “Will North Korea launch a new multistage missile in the next year” and “Is Greece going to leave the eurozone in the next six months?” Tetlock’s group of amateur forecasters would go head-to-head against teams of academics as well as career intelligence analysts, including those from the C. I. A., who had access to classified information that Tetlock’s team didn’t have. The results were shocking, even to Tetlock. His team won the competition by such a large margin that the government agency funding the competition decided to kick everyone else out, and just study Tetlock’s forecasters — the best of whom were dubbed “superforecasters” — to see what intelligence experts might learn from them. So this conversation is about why some people, like Tetlock’s “superforecasters,” are so much better at predicting the future than everyone else — and about the intellectual virtues, habits of mind, and ways of thinking that the rest of us can learn to become better forecasters ourselves. It also explores Tetlock’s famous finding that the average expert is roughly as accurate as “a dart-throwing chimpanzee” at predicting future events, the inverse correlation between a person’s fame and their ability to make accurate predictions, how superforecasters approach real-life questions like whether robots will replace white-collar workers, why government bureaucracies are often resistant to adopt the tools of superforecasting and more. Mentioned: Expert Political Judgment by Philip E. Tetlock “What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?” by Christopher W. Karvetski et al. Book recommendations: Thinking, Fast and Slow by Daniel Kahneman Enlightenment Now by Steven Pinker Perception and Misperception in International Politics by Robert Jervis This episode is guest-hosted by Julia Galef, a co-founder of the Center for Applied Rationality, host of the “Rationally Speaking” podcast and author of “The Scout Mindset: Why Some People See Things Clearly and Others Don’t.” You can follow her on Twitter @JuliaGalef. (Learn more about the other guest hosts during Ezra’s parental leave here.) Thoughts? Guest suggestions? Email us at [email protected]. You can find transcripts (posted midday) and more episodes of "The Ezra Klein Show" at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Book recommendations from all our guests are listed at https://www.nytimes.com/article/ezra-klein-show-book-recs. “The Ezra Klein Show” is produced by Annie Galvin, Jeff Geld and Rogé Karma; fact-checking by Michelle Harris; original music by Isaac Jones; mixing by Jeff Geld; audience strategy by Shannon Busta. Special thanks to Kristin Lin and Alison Bruzek.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Ezra Klein?

  I'm Ezra Klein, and this is the Ezra Klein Show.

• 0:19Ezra Klein?

  Hey, it's Ezra. While I'm on paternity leave, we've got an all-star team of guest hosts. This week is Julia Galef, the host of the Great Rationally Speaking Podcast and author of the book The Scout Mindset. Julia is unusually good at thinking about thinking, so this should be a lot of fun.

• 0:41Speaker 01

  My guest today is Phil Tetlock. He's a professor of psychology at the University of Pennsylvania and a professor of management at the Wharton School. He's also the author of several books, including the 2015 bestseller, Superforecasting: The Art and Science of Prediction, co-authored with Dan Gardner. In my view, Tetlock is doing some of the most important research in social science today, research that has the potential to revolutionize how we run our organizations, how we make consequential policy decisions, and how we live our everyday lives. And that's because at the center of his work is a question that humans have been trying to answer for as long as the history of civilization: how can we predict the future more accurately? And Tetlock isn't just an ivory tower academic

• 1:25Speaker 01

  here. In 2011, he entered a team in a competition sponsored by the US government in which participants were asked to make forecasts about questions like, how long will the Syrian civil war last? And is Greece going to leave the Eurozone? Putting numerical probabilities on their guesses from 0 to 100%. Tetlock's group consisted of amateurs with nothing but an internet connection who agreed to make predictions in their spare time. And they were going head-to-head against not only other teams, but also the highly-trained career intelligence analysts from the CIA who had access to classified information. Yet not only did Tetlock's group win, they were so much better than all the other teams that the government

• 2:10Speaker 01

  agency funding the competition decided to kick everyone else out and just study Tetlock's best forecasters, whom he dubbed the super forecasters.

• 2:20Speaker 01

  So this conversation is about why some people, like Tetlock's super forecasters, are so much better at predicting the future than everyone else, and about the habits of mind and the ways of thinking that the rest of us can learn to become better forecasters ourselves. Before we jump in, a reminder that you can always email the show with feedback or guest suggestions at ezra kline show at nytimes.com. My conversation with Phil Tetlock after the break.

• 3:07Speaker 01

  Phil, welcome to the Ezra Klein Show.

• 3:10Speaker 02

  Great to be here, Julia.

• 3:11Speaker 01

  I'm so excited to get to talk to you again. Before we dive into the weeds, Phil, I'd like to give our listeners a sense of just how central forecasting is to our society and to our lives. Because I think a lot of people might be thinking of it as kind of a niche interest or pastime. And so you had a nice quote in a foreign affairs article you co-wrote last year, where you said, "Every policy is a prediction." Could you explain what that means?

• 3:41Speaker 02

  Sure. Virtually every consequential thing we do in the world is an implicit forecast because we're making assumptions about what will follow later. And sometimes those assumptions are so obvious, we don't even think of them as a forecast. We think of them as virtually factual. But the current debate over whether the budget bills in Congress right now are going to be inflationary or hyperinflationary, or whatever people might want to claim, those are obviously competing forecasts about what will happen with respect to inflation, inequality, government deficits, a long list of outcome variables that people are arguing about. People are arguing about US policy on Taiwan and China. And how much aid should the US be giving Taiwan?

• 4:25Speaker 02

  What increases or decreases the likelihood of an armed conflict over Taiwan? These are all claims about the future. So forecasting is not a niche activity. It's central to public policy. It's central to human life, whenever you decide to get married or not to get married or get divorced. Whatever. It rests on on claims about possible futures.

• 4:46Speaker 01

  Great. So now that we've established that we're all forecasters, whether we realize that or not, let's talk about some of your earlier work on how good we are at forecasting. And I think the one insight that you might be most famous for is from your book, Expert Political Judgment, from 2005, in which you concluded that the average expert on economics or politics was roughly as accurate as a dart-throwing chimpanzee. Now, of course, that doesn't mean that every single expert is worse than a chimp, just that the average expert is worse than a chimp. But that's still a pretty disturbing result. So can you summarize the research you did that led you to that disturbing conclusion?

• 5:28Speaker 02

  Well, I was a junior faculty member at University of California, Berkeley, and I was serving on a National Research Council panel. Got started in 84, 85. A lot of people thought the world was heading into a nuclear apocalypse. That was one of the big debates. Was Ronald Reagan increasing or decreasing the likelihood of a nuclear war with the Soviet Union? As you might expect, you could predict very well the positions people took from ideology. And there was a big debate also about what was going to happen to the Soviet leadership. We were coming up to that point in the mid 1980s. The aging gerontocracy in charge of the Kremlin was fading out. And a young, relatively junior member of the Politburo, Mikhail Gorbachev, was about to become general party secretary.

• 6:12Speaker 02

  And there were conflicting views, A, about whether he would get the job, and B, if he got the job, whether he would be a neo-Stalinist disciplinarian and confrontational with respect to the United States, or whether he was a latent liberal and he would reform the Soviet Union. And it turned out to be the case that the people who Felt he was going to be a radical reformer were quite right. He was a radical reformer. In fact, his reforms are so radical, they ultimately led to the disintegration of the Soviet Union. Liberals generally thought that Reagan was pushing the Soviets into a corner and the Soviet Union would become more truculent and confrontational. And the conservatives thought that the Soviet Union was unreformable and was inherently neo Stalinist.

• 6:54Speaker 02

  So neither side really predicted Gorbachev and how what a radical reformer he would be. But within a year or so, after it became apparent that he was a reasonably radical reformer, everybody claimed that they saw it coming all along. And the conservatives said, hey, this is vindication for Reagan's policies. And the liberals said, you know what? Reagan had nothing to do with this at all. The Soviet economy was crumbling. The Soviets were going to do this anyway. So that alight turned on. I thought, you know, it'd be interesting to keep score.

• 7:24Speaker 01

  It's kind of wild that no one was keeping a score before then, honestly. I guess this is one of those innovations, like actually tracking the accuracy of people's forecasts that seems so obvious after someone proposes it, but wasn't obvious to people before then.

• 7:39Speaker 02

  Well, it's very obvious, but it's also almost impossible to implement because a lot of political speech is not really about accuracy. It's about laying claim to power, using the most effective rhetorical devices.

• 7:52Speaker 01

  So your extra long study that was the cornerstone of expert political judgment, how many experts were in that study and what kinds of questions were they answering? And how did you measure their accuracy?

• 8:04Speaker 02

  It's about 284 or so subject matter experts. And they weren't all experts on the Soviet Union. They were experts on all sorts of various regions of the world, had various functional specialties as well, like arms control or economics. And we measured relatively short term predictions, medium term predictions, and occasionally we measured some longer term predictions. But the main focus was on one, three and five year predictions.

• 8:27Speaker 01

  And crucially, I think it's important to note that not only were the people in your study making predictions, but they were attaching a quantitative confidence level to those predictions. Like, they might say, you know, "I'm 75% sure the Soviet Union will collapse in the next year," or something like that.

• 8:42Speaker 02

  Absolutely. If I say to you, "I think there's a distinct possibility that Saddam Hussein has weapons of mass destruction," how are you going to interpret that? Let's say I say that to you right before the invasion of Iraq in 2003. You hear the phrase "distinct possibility." How do you translate that? You say, "Well, he said distinct possibility, so I guess that means it's over maybe, over 50%." Or, "He merely said it was possible, so I guess that's under 50%." Distinct possibility is almost the perfect phrase if you're a risk-averse political pundit or intelligence analyst, and you don't want hindsight tainted people coming back and criticizing you later on. You can say, "Either way, you're going to be on the right side," right? You say, "I merely said it was possible. If it doesn't

• 9:26Speaker 02

  happen, if it does happen, you say, 'I said it was distinctly possible. You didn't listen carefully enough.'" So, distinct possibility straddles a broad range of the subjective probability continuum, from about 20% to about 80%. Those are the lowest and the highest plausible readings that people attach to that phrase. It's really tailor-made for playing the Washington DC blame game.

• 9:49Speaker 01

  It's also just tailor-made for causing confusion and disagreements. I know I've gotten into disagreements with people where they said something was, you know, quite possible, and to me, I said it was very unlikely, and it turned out, after much debate, that we'd both put it at about 10%. But, you know, that was not at all clear when we were using the vague language.

• 10:07Speaker 02

  Right, right. And that was one of the stories that we used in the super forecasting book. Later, it was the story of John F. Kennedy and the Joint Chiefs of Staff. He launched the Bay of Pigs invasion in April 1961, three months into his presidency. It was probably the biggest fiasco of his presidency, and the question was, "Why did he do it?" It turned out that he was a little bit wary of the plan to invade Cuba. It was a CIA plan. He was wary of it. He asked for a second opinion from the Joint Chiefs of Staff. And they gave him one. They said there was a fair chance of success, which he interpreted optimistically. But they did later go back to the JCS and they had, well, what exactly do you guys mean by fair chance? Well, I said, maybe one in three. So I don't know whether history would have unfolded differently if they had had

• 10:51Speaker 02

  a quantitative probability estimate from the JCS as opposed to the vague verbiage, um, fair chance.

• 10:57Speaker 01

  Right. So a big innovation in addition to just keeping a score was having people actually give quantitative probabilities to their predictions so that you can actually score their accuracy. And so back to the punchline, the average expert not actually being more accurate in making these predictions than a dart throwing chimpanzee. What were the experts doing wrong? The average expert doing wrong?

• 11:18Speaker 02

  The term average expert conceals more than it reveals. I mean, some experts really were substantially better than chance. And there are other experts who actually managed to do worse than chance. And we found an individual difference variable that predicted which X first fell into which category. The more flexible foxes tend to do to do better than chance and the more the less flexible hedgehogs tended to do somewhat worse than chance.

• 11:43Speaker 01

  And just to expand on that metaphor a little bit. What you're describing here is two types of thinkers. The hedgehogs are the kinds of people who see the world through one big idea, one ideology. And so they aren't very flexible in their worldview. They aren't as likely to update their opinions or predictions when new information arises. And foxes are the opposite. They aren't attached to one particular ideology. They really try to incorporate a wide range of worldviews into their thinking. They tend to be more intellectually flexible. And this distinction turns out to be really important. The finding from your big study on this was that the people who think like foxes are much better forecasters than those who think like hedgehogs. So let's dig into that more.

• 12:27Speaker 01

  Can you give me some examples of hedgehogs?

• 12:31Speaker 02

  The world is full of hedgehogs and there are lots of one big idea people. You could be a Marxist hedgehog or you could be a libertarian hedgehog. You could be a boomster hedgehog or you could be a doomster hedgehog. Let's just take those two polarities for the moment. I mean, there's a very pessimistic view about capitalism, a very optimistic view about capitalism, a very pessimistic view about the limits on growth, a very optimistic view on limits of growth. So you can imagine people falling toward the extremes of these continuums. And you can imagine people who are floating sort of in the middle at various points. And the foxes tended to be centrist on the left, right dimension, on the boomster doomster dimension and so forth.

• 13:13Speaker 01

  One reason I think your work has been so eye opening for so many people, including me, is that it really challenges our assumptions about what it means to be an expert. You know, we tend to think of someone as an expert if they have a lot of knowledge, like a PhD or in a subject domain or have written a bestselling book on something. And your point is that, no, expertise actually means having a track record of actually getting things right, which you might assume would go hand in hand with having a lot of knowledge in an area. But in practice, no, these are just two, you know, knowledge on the one hand and judgment or forecasting skill on the other are just these two different independent things. And just because someone wrote a book about elections doesn't mean they're going to be good at predicting the results of the election and so on. When I say it, it almost sounds obvious, but it really wasn't to me.

• 13:58Speaker 01

  And it's really refocused my attention on, you know, not just does someone have relevant background knowledge, but also not just do they sound smart when they talk about a subject? Because I think that was also something I was kind of implicitly assuming made someone an expert, that they can say smart things about something. And I just wasn't really tracking, well, their track record. I pay a lot more attention to that now than I did before encountering your work. Another related interesting finding from this work was that you found an inverse correlation between fame and accuracy. So the more famous an expert was, the less accurate he was. Why would that be?

• 14:34Speaker 02

  For the reason that very few people want to talk to me, I think. Experts who interweave lots of buts and howvers and all those into their speech are less charismatic and compelling than experts who are telling an emphatic causal story pointing in one clear direction. It's really hard to be charismatic. If you're building up a lot of ideological momentum in the audience, then you say, however, we have to take into account the cost of these measures against climate change. Or however, you can just see the balloon deflate. So this is a good indicator of whether you are going to be a good forecaster or not. It's how frequently you use terms like but, however, although.

• 15:11Speaker 01

  So that that story about why hedgehogs with their one big idea, their one big framework would be worse forecasters and also more famous, that makes sense to me. I was just also thinking of another possible story. I'm curious what you think about this. So it occurred to me that experts who have become famous for a particular point of view or particular argument might have kind of boxed themselves into that viewpoint. Like if you become famous for being, you know, the guy who argues that the world is getting better, or the guy who argues that big tech companies are harming society, et cetera, and that's what you're known for, I think that that kind of entrenches you even more than you would already have been entrenched just from human nature and makes it especially hard to change your mind or consider exceptions to your viewpoint. What do you think about that?

• 15:56Speaker 02

  Well, the two psychologists who had the most formative influence on my thinking would be, I think, two of the most influential psychologists of the 20th century, Danny Kahneman and Leon Festinger. And you're in Leon Festingerland right now. He very clearly laid out exactly the argument you're describing, that people who make public commitments to a position are going to be motivated to bolster it. They're going to become better and better at generating reasons why they're right and there would be critics are wrong. And the more you do that, of course, the more your attitudes are going to polarize. You're going to build up ideological momentum unless somebody puts on the cognitive brakes and says however or although. Leon Festinger argued that that's not a natural cognitive response people have.

• 16:41Speaker 02

  The natural cognitive response to have to dissonant or disagreeable moves is denial and bolstering. Those are the low effort cognitive responses and they lead to polarization, of course. You're digging yourself into the trench to defend your position. And the unnatural responses are to be integratively complex, to differentiate and integrate, to concede that there might be legitimate alternative points of view and identify boundary conditions under which they hold. That's, of course, what the best forecasters tend to do, but it's not a natural cognitive act. And that's what I, in the natural language processing study, I think Karwetsky had all just reported in the International Journal of Forecasting. I think that provides some of the best evidence for that claim.

• 17:20Speaker 01

  This is the irony that I noticed in talking to open-minded people and less open-minded people about open-mindedness. The less open-minded people would say, oh, yeah, you gotta just stick to your guns and never change your mind and that makes everyone respect you and it makes you successful and it has no downsides. And then when you talk to the open-minded people, they're like, well, you know, I do think open-mindedness is great, but here's some potential downsides and maybe it has this bad effect and maybe it has that bad effect. And so there's no real objective vantage point from which to weigh open-mindedness against the opposite.

• 17:51Speaker 02

  Yes, and you hear that a lot in the business school world where a common theory is that that strong leaders are decisive leaders and there is a certain stay the course mentality. Here you need to draw a distinction between how you think in private and how you think in public. And this actually leads to a somewhat disturbing thought that maybe effective leaders have to be, to some degree, liars. Here's an example of that. So Dwight Eisenhower, right before D-Day, was exhorting the troops and telling them they were going to be victorious when they landed on the beaches of Normandy on June 6th, 1944. And he was projecting confidence and can-do and bravado. Now, if Eisenhower were being a completely transparent leader, he would have said, well, you know, look, guys, you know, I think there could be a 20, 30% chance that you're going

• 18:35Speaker 02

  to be driven back into the English Channel. If he'd talked like that, you would have said, he's an idiot. This guy doesn't know how to run a show. But on the other hand, you want someone who's aware of the possibility that the plan's going to fail, who has lots of contingency plans and so forth. So we wrestle with this kind of moral issue, I guess, in super forecasting. Dan Gardner and I wrestle with it about whether leaders in some sense have to be liars. Good leaders have to be liars.

• 19:01Speaker 01

  You had another interesting argument in defense of hedgehogs to a limited extent. It was kind of an ecological argument that hedgehogs are, play a valuable role in the information ecology.

• 19:13Speaker 02

  Absolutely. Where do foxes get their most interesting ideas from? They get them from the extremes, from the hedgehogs, which they then dilute and qualify them. But hedgehog ideas are often profoundly provocative. So the classic Marxists, they left a deep imprint on thought in the 19th and 20th centuries.

• 19:30Speaker 01

  I don't know if this is how you would characterize your, the arc of your career, Phil, but I think about your career as having these three chapters where the first chapter, where you're studying expertise is kind of discovering this problem of the lack of actual forecasting skill in many people we consider experts. And then the second chapter is about finding solutions to that problem. This is the chapter in which you wrote super forecasting. And so I want to move on to that chapter now and talk about, you know, how you pursued this question of, well, sometimes experts are terrible at forecasting, but some people seem to be pretty decent at it. And so what are they doing right? What can we learn from them? And so as part of that project, you led a team in a forecasting tournament that was

• 20:13Speaker 01

  put on by a US government agency called IARP, the Intelligence Advanced Research Project Activity, kind of like DARPA, but for the intelligence community. Could you talk about the purpose of that tournament, what the outcome was? And for our listeners who aren't familiar with the idea of a forecasting tournament, what is it?

• 20:31Speaker 02

  Well, a tournament is like an athletic competition in which you have a bunch of players who are trying to achieve a goal. Here, the goal is to make forecasters active as possible. In a tennis tournament, you know, you know what the goal is and so forth. So it's just, it's a competitive system for identifying top performers.

• 20:51Speaker 02

  It was a wonderful opening salvo to bring some core enlightenment values into political discourse and into intelligence analysis, you know, just radically simple and transparent. Here is a series of questions of interest to the US intelligence community. We're going to give some competing research teams a bunch of money. They're going to be able to recruit people to their platforms and their job is going to be to generate the most accurate probability estimates for the questions that are of interest to the intelligence community. So there'd be questions about how long would the Syrian civil war last and, you know, what's going to happen next with Colombian narco trafficking and what's going to happen with Spanish-German bond yield spreads or whether Greece is going to leave the Eurozone.

• 21:34Speaker 02

  You're just a very long list, incredibly heterogeneous array of questions. So no individual in this forecasting tournament is really going to be an expert on more than one or two or three of these questions. I mean, they're that heterogeneous. It's like a geopolitical IQ test. It was like a measure of geopolitical fluid intelligence. How adept are you at adjusting really rapidly to one damn thing after another? Very, very difficult. In traditional analysis and traditional academic scholarship, we think of experts having a lot of crystallized knowledge. I mean, I just spent six years getting a PhD in China studies, then I spent 10 years getting tenure and so forth, and now some upstarts want to know whether I can outpredict some guy off the street on where the Chinese economy is going, and that's annoying.

• 22:19Speaker 02

  So the idea is really a pretty radically simple one. You can see how it has a certain populist appeal because it's a way of saying, okay, you high and mighty experts, we want to see whether you can actually deliver on these things. And a lot of the response of the expert community was not all that positive. I mean, they felt, oh, I think what IARPA has just invented here is a game of kind of geopolitical trivial pursuits. So we did find some people were really good at this. We also did some randomized control experiments. We found that there's some ways are better than others of training people to do better at this. We discovered some natural talent, pre-existing talent, we'll call it. We found ways of cultivating talent once people were inside the tournament. We found there were better and worse ways of running teams.

• 23:02Speaker 02

  And we found there were better and worse ways of aggregating crowd wisdom.

• 23:06Speaker 01

  So your team won in a landslide. Your forecasters were by far the most accurate people in the entire tournament, including the CIA analysts themselves. How much more accurate were the good judgment teams than the runners-up?

• 23:20Speaker 02

  Well, they were beating the other teams by margins of between 40 and 70%. They were doing better than the prediction market inside the intelligence community between about 25 and 30%.

• 23:29Speaker 01

  These are analysts who have access to tons of classified information about the subject matter and the questions that the amateur forecasters that volunteered for your team did not have access to. So which makes it even more impressive, I think. You're not tooting your own horn enough, so that's why I'm doing it for you.

• 23:45Speaker 02

  Well, bear in mind who actually did the work. It was it was these so-called super forecasters. And because the government didn't allow us to pay people a lot of money for performance, because they don't believe in pay for performance inside the intelligence community, apparently, we weren't allowed to do that. So we had to reward them with status. And what's the status award? Well, it's to be called a super forecaster.

• 24:04Speaker 01

  Oh, that's where that term came from.

• 24:06Speaker 02

  Yeah, it was a motivational tool.

• 24:08Speaker 01

  So we've kind of touched on a few things that made the super forecasters super. But if you had to kind of pick one or two things that really made the super forecasters what they were, what would they be?

• 24:19Speaker 02

  You know, we've already talked about one of them, which is their skill at balancing conflicting arguments, their skill of perspective-taking, however, although, but. They put the cognitive brakes on arguments before arguments develop too much momentum. So they're naturally inclined to think that the truth is going to be some blurry integrative mix of the major arguments that are in the current intellectual environment, as opposed to the truth is going to be way, way out there. Now, of course, if the truth happens to be way, way out there and we're on the verge of existential catastrophe, I'm not going to count on them to pick it up.

• 24:51Speaker 01

  In addition to these dispositions and sort of general thinking patterns that the super forecasters had, are there any kind of concrete habits that they would, you know, always or often make use of when they were trying to to make a forecast that other people could adopt to?

• 25:07Speaker 02

  One of them is this tendency to be integratively complex and qualify your your arguments with howers and buts and all those. A sign that you recognize the legitimacy of competing perspectives. As an intellectual reflux, you're inclined to do that. And that's actually a challenge to Festinger and cognitive dissonance. They're basically saying, look, these people have more tolerance for cognitive dissonance than Leon Festinger realized was possible. The other thing super forecasters do is they use comparison classes. They adopt the outside view. And this is where the super forecasters do what Danny Kahneman said people don't do.

• 25:41Speaker 01

  So I want to talk about that distinction between taking the inside view versus the outside view on a problem. So when most of us try to make predictions, we'll start instinctively with the inside view, which means we'll start by looking at the specifics of a case. And then we'll base our prediction primarily on those details. But starting from the outside view means looking for comparison classes, which are basically benchmarks that help us see how other similar situations have played out. And then, of course, you still look at the specifics, but only as a way of almost fine tuning your prediction. So can you walk us through an example of what it would look like to make a forecast by starting from the inside view versus starting from an outside view?

• 26:26Speaker 02

  Well, you know, if you're at a wedding and someone has a bad taste to come up to you and ask, you know, how likely is that couple to stay married? You're going to look at them, you say, well, that's a really rude question. Look how happy they are. It's also not only a rude question, it's a dumb question. Look how happy they are. They're taking the inside view. They see the happy couple. They're basing their prediction on the particulars of that situation. Inside that situation, they're making their prediction. As opposed to you come up to somebody and they say, how likely to stay married? And you say, well, let's see, what's the sociodemographic category into which this couple falls? What's the divorce rate for that sociodemographic category? Looks like they've got about a 65% chance of staying married in the next five years and a 53% chance in the next 10 years. That's the outside view.

• 27:09Speaker 02

  You're identifying a comparison class, and that comparison class is the start of your forecasting process. You're saying, okay, my initial forecast about how long this couple is going to stay together, say, the next five years, it's around 60%. Now I discover that the husband is a psychopathic philanderer. Now I'm going into the inside view. So 60% is going down. How fast is it going down? Maybe to 40, 30, 20? I'm updating, but I didn't start by saying there's a 99% chance they're going to stay married because they look happy, because that's a dumb forecast. Almost everybody can actually look happy on their wedding day.

• 27:40Speaker 01

  My favorite example of a outside view, inside view tension actually comes from Danny Kahneman. I think it was in Thinking Fast and Slow. He described being on a committee trying to write a textbook, and they were trying to predict how long will it take us to write this textbook? And they thought about it using the inside view. They're like, well, how many chapters do we have to write? Like, how long does it feel like it should take us? Their answer was something like one year, I think. The outside view was closer to five or six years, because if you looked at other committees that were trying to write textbooks, I don't think any of them finished in under four years. And so that was the outside view they should have been adjusting from. And indeed, I think it took them well over six years. So taking the outside view, and especially starting with the outside view as your kind of baseline, and then maybe adjusting it based on unique or special features of the situation.

• 28:25Speaker 01

  That's a very helpful tip that aspiring super forecasters can learn to use. I was hoping we could go through a concrete example of a forecasting question and just... I mean, I know you're not a super forecaster yourself. I'm not a super forecaster. But we could maybe talk about how a super forecaster would approach it. And my nomination is, are we going to get to full self-driving cars by the year 2025? You know, the kind of car where you can really just take a nap and let the car drive you to work, you know, through busy city streets? Will something like that be available by 2025? So do you have any thoughts on how a super forecaster might start thinking about that question?

• 29:04Speaker 02

  The first thing they would do is they would pick holes in it.

• 29:07Speaker 01

  In the question?

• 29:08Speaker 02

  Yeah, they'd pick holes in your question. It would be like you're talking to a contract lawyer. And they'd say, oh, Julia, my God, this is so underspecified. Okay, you mean anywhere in the world?

• 29:19Speaker 01

  Yes, yes, I do. Yes.

• 29:21Speaker 02

  Okay, so you have more of a shot at it happening in Singapore, because they're pretty far advanced with the infrastructure, and they have a different legal system. Even if we had close to perfect technology, technology is never going to be perfect, given the way the US tort law is set up and given the dynamics of US politics, it would be very difficult to imagine it happening by 2025 in the US. Could it happen in certain authoritarian or semi-authoritarian systems that are technocratic? Maybe in Dubai or in Singapore, places like that? If that's the question, now I'm saying, hmm, okay, maybe 25%. But now, take a nap, hmm, not so sure. I think even there, the laws would be you have to be conscious and you probably shouldn't be drunk.

• 30:06Speaker 02

  There would still be laws and a sentient observer observing, at least in principle, in charge. But yeah, that's how they think. They think in very detailed terms and they think about boundary conditions. And it's not just the technology, it's the politics and the law, right?

• 30:20Speaker 01

  I guess this is how a lot of forecasters go awry, is they're thinking of one specific kind of scenario. Like, I was thinking about the US and I was only focused on the technology itself and not on the legal framework surrounding that deployment of the technology, et cetera. And so I can a little bit empathize with the experts in your tournament who, you know, when the question is resolved and they were wrong, they kind of protest, "Oh, but that's not what I was thinking of when I said, you know, 75% chance yes. The thing I was thinking of was blah, blah, blah." But that is part of the game, is you have to be thinking about all the factors. And I guess that's what the super forecasters are good at.

• 30:55Speaker 02

  You know, it's a wonderful experience working with them. You know, the vast majority of psychological work in the last few decades, several decades, has been on documenting how people aren't all that smart, the various errors and biases that people fall prey to. So, working with human subjects who are, you know, palpably smarter than I am was a real privilege. And it remains so, because I've found among them, they're not subjects anymore, they're collaborators.

• 31:22Speaker 01

  I did just have a thought about how I might approach the forecasting, the self-driving car question. Tell me if you think this is a good use of the outside view. So I was thinking it might be useful to look at other technologies in the past that also got a lot of hype, like self-driving cars, and also seemed to many people to be only a few years away, like self-driving cars. And then I would look at how long each of those technologies actually took to come to fruition, or whether they did at all, and treat that set as kind of the outside view for self-driving cars. What do you think about that?

• 31:53Speaker 02

  I think it's a good idea. The devil lurks in the details of defining the relevant comparison class. It's pretty easy for the divorce case, right? Where you say, "What's the sociodemographic category?" So, you know, Harry and Meghan, you know, what's the divorce rate for the British royals? And now that they leave the royal family, you know, what's the divorce rate for Hollywood celebrities? Or, you know, changing base rates, changing comparison classes. Because I don't have the requisite technical expertise, I'm simply going to punt on what the relevant comparison class would be. But that, in principle, would be the right move, yes.

• 32:24Speaker 01

  I also just wanted to note that one reason I liked this question as a test case is that I think it's kind of hedgehog bait, for at least some kinds of hedgehogs that I know. The techno-optimist hedgehogs, for example, tend to get very excited about progress and they really want to be bullish about everything. Whereas the techno- pessimists just find this kind of hype so annoying and they like to focus on the ways and all the instances in which progress is slower and worse than we expect it to be. And both of those worldviews are sometimes right, but if you're only sticking to one reflexively, I think you're going to miss a lot of calls.

• 32:56Speaker 02

  Yep, yep, so many subspecies of hedgehogs.

• 33:18Speaker 01

  So, though, what have you learned about how trainable these super forecaster skills are? Like, you can identify specific things they're doing, like taking the outside view. But have you studied, can non-super forecasters develop those habits and learn to deploy those tools as effectively as super forecasters can? Could we teach these in a college class?

• 33:37Speaker 02

  Yeah, we certainly have, and in classes, yes. I wish it had been formally embraced by the intelligence community, which, of course, spent millions of dollars from developing all this stuff.

• 33:45Speaker 01

  Do you happen to remember any results about, like, how much improvement people could see if they, like,

• 33:50Speaker 02

  Yeah, about 10%.

• 33:51Speaker 01

  10%? Oh, great. And how long was the training that they had to go through? Was this like a whole semester that they had to practice?

• 33:58Speaker 02

  The training only took about an hour.

• 33:59Speaker 01

  An hour?

• 34:00Speaker 02

  And the effect persevered through a tournament year, which is about nine months, and about 100-plus forecasting questions. So, count me as moderately astonished by that.

• 34:09Speaker 01

  Yeah, maybe some people in the audience aren't as impressed as we are by this. If so, that's just because they haven't been beaten into deep cynicism by all the failures of other educational interventions to do anything at all. So they can't. You have to take it in context to appreciate what a big deal even 10% improvement is. One other kind of cool result from your experience with the super forecasters that I wanted to highlight is that not only were they more accurate than everyone else, they also improved much faster than everyone else. Am I saying that right?

• 34:38Speaker 02

  Yes, they are fast learners.

• 34:40Speaker 01

  Yeah, they are so interested in why they get things wrong, when they get things wrong. And so if their forecast is way underconfident or way overconfident, instead of trying to make an excuse for themselves about, well, I meant such and such, or well, I was really thinking of this other thing. They get really curious about, oh, well, what process was I using that made me get the wrong answer? And like, how can I fix the process? And so on. It's really cool to watch, actually.

• 35:06Speaker 02

  It is. It's a wonderful thing to watch them do postmortems. And they not only do postmortems on their failures, the best ones do postmortems on their successes as well. Because they know that you can be right, but lucky. We might have gotten the right answer because we made a set of offsetting errors, not because we had the fundamentally correct diagnosis.

• 35:24Speaker 01

  You've talked about Danny Kahneman a few times in this conversation. And I expect many of our listeners have read or at least heard about his book, Thinking Fast and Slow. And Phil, as you're probably aware, Kahneman is more pessimistic than you are about the possibility of training people in better thinking skills. Like, just in general, he's pessimistic about our ability to overcome the cognitive biases that he writes about and improve our judgment in a lasting way. So I'm just curious, why do you think you and Kahneman disagree about our ability to improve?

• 35:55Speaker 02

  I think it runs deep. I mean, he had a profound impact on my thinking. And the cognitive illusion metaphor, I always thought of as quite compelling. If you believe that cognitive biases are as tenacious as perceptual illusions, you're going to be really pessimistic. Perceptual illusions continue to trick you even after they've been explained to you. You still see them the wrong way. Famous example, the Mueller liar illusion that Danny uses in his book, Thinking Fast and Slow. There are two lines that are of identical length. One of them looks longer than the other because of the way the arrows are pointed. Beautiful illusion. Readers should look it up if they can't. We don't have video, video, so we can't show it to them, unfortunately. But it's a beautiful illusion. And you know what? You're going to see it the wrong way.

• 36:37Speaker 02

  You're going to pull out a ruler and you're going to measure it and say, oh, my God, they're the same length. Then you're going to take the ruler away and you're going to look at it again and you're going to see it wrong again. That's why Danny tends to be more pessimistic because he thinks of cognitive biases as being as tenacious as perceptual illusions. I think some cognitive biases do have that property and are that tenacious. I think other cognitive biases can be checked to some degree. If you have the right cognitive style or the right training mindset. I think you can be trained to use comparison classes to some degree, even though the natural first thing to do is to take the inside view. I think you can be trained to be a little more tolerant of cognitive dissonance. And I say and ask however, although, but sorts of questions, even though the natural thing to do

• 37:19Speaker 02

  is to dislike cognitive dissonance and to prefer everything to come together in a nice, neat evaluative gestalt. So it's a matter of degree.

• 37:28Speaker 01

  Your point about what does it really mean to overcome a cognitive bias? And does it still count if you, you still experience it, but you've put in safeguards to correct for it, even though you still feel the bias, you know, on your inside view is an excellent point and makes sense to me. I have this other theory about why Danny Kahneman is more pessimistic than you and me, which is that I, I think his work and, and a lot of psychologists' work on cognitive biases and, you know, trying to train critical thinking skills focuses on college students who are participating in these experiments that are being run by the professors on college campuses. And they're not necessarily people who have come to this experiment because they really want to improve their critical thinking skills and they're excited to put in the work to improve.

• 38:12Speaker 01

  They're, you know, participating in the experiment for a credit in their psych class or something like that. Whereas the people that you're working with, the super forecasters or the aspiring super forecasters actually do want to improve. Like they have the motivation to, you know, be more accurate and they're willing to put in the effort. And I'm much more optimistic about that group being able to improve than about the group that's just there to check the box, you know? Does that make sense?

• 38:35Speaker 02

  I think that makes a lot of sense. College students who participate in most psychological experiments are not all that motivated to get the right answer. They're motivated to get in and out pretty fast. And researchers' hypotheses are often assuming something like that. So it actually works to the benefit of the hypothesis testers that the subjects are processing things in fairly superficial ways. I think the Kahneman-Tversky heuristics and biases work cannot be dismissed as an artifact of undermotivated undergraduates, but I think a substantial amount of psychological literature can. But the Kahneman-Tversky effects are remarkably robust, and you see versions of them holding up in complicated real-world situations as well. But yes, that is certainly one of the differences.

• 39:20Speaker 01

  Well, unfortunately, I think that if you and I are right about the importance of motivation here, that doesn't bode well for being able to teach kids super forecasting skills in their mandatory high school classes or even college classes where they're there to get a credit. I mean, it does bode well for our ability to cultivate a class of super forecasters who can help our society make better decisions, but that's different from training kids in critical thinking, which is something I know many people dearly want to do, but I think that's not a promising sign about that.

• 39:48Speaker 02

  I understand the desire to make it mandatory, but it does have some perverse side effects.

• 39:55Speaker 01

  So switching gears a bit now, Phil, an elephant in the room in this discussion of how effective the super forecasters were and how they were 30% more accurate than the CIA themselves, you know, I think a lot of people are probably wondering, well, has the government adopted super forecasting? Like, do they have a huge team of super forecasters that they consult about policy questions? Like, when will the supply chain issue ease up? Or, you know, what's China gonna do vis-à-vis Taiwan, things like that? And it sounds, from some things you've said so far, like the answer is no. The government has not bought into super forecasting wholesale. Is that mainly because of the social frictions, like the incentives issue you talked about, where the people who already have positions of prestige don't want it to be discovered

• 40:39Speaker 01

  that they're not actually any more accurate than some random 24-year-old on the internet?

• 40:45Speaker 02

  That's right. So the senior people my age don't welcome some upstarts from an R and D agency like IARPA coming along and saying, hey, let's put the 60-somethings up against the 20-somethings and see who can do a better job predicting now what Xi Jinping is going to do on Taiwan in the next two years. That's a hard sell.

• 41:04Speaker 01

  Have you found any kind of ways to work around this incentives problem in your efforts to get forecasting more widely accepted? Like, is there a way to persuade people to do it even though it might risk exposing them as not as accurate as they thought they were?

• 41:20Speaker 02

  Yeah, it definitely threatens people. So my first instinct is to say, well, maybe you can reduce the resistance to forecasting tournaments by changing the organizational culture and say, look, we're going to treat this sort of as a game. We want everybody to play. We want everybody to try to get better and we're going to reward people for playing and getting better. But we're not going to punish people for being wrong. You know, it's still going to be somewhat threatening, but I think you can certainly create more psychological safety around the activity. If people think their jobs, they're on, they think someone's going to come along at the end of the day and say, well, let's take the top 10% of the best forecasters and a few other guys. I guess, you know, we really don't need you. So that's going to be a threat. It's interesting, what is so much an easier

• 42:05Speaker 02

  sell, the work on super forecasting to sophisticated private sector firms in technology and in finance than it is in the government. I mean, there are parts of the government that are open to this. It really depends on who's in charge at that particular moment. If there's a, the right person's in charge of the National Intelligence Council, or right person is the DNI, not Director of National Intelligence, the right person's in the Federal Reserve, right person's here or there, it'll go forward, oh, change jobs, backward.

• 42:33Speaker 01

  Well, hopefully it's two steps forward, one step back and not the reverse. There was another big and important reason that I think there's a bit of a slow uptake of forecasting by institutions like the government. And I know you've thought about this and are working on it currently. It's something you call the rigor relevance gap. Could you explain what that means?

• 42:53Speaker 02

  Well, the forecasting tournaments are really good at answering very specific resolvable questions like, how many people are going to die on the Israeli Lebanese border in the next six months in military clashes, or how many are this or that, I mean, very specific, what is Spanish-German bond yield spread is going to be in the next six months? So very specific questions. Policymakers, however, care about more general questions. They care about, well, how compatible or incompatible are the geostrategic interests of the US and China? Is a war between Israel and Iran just a matter of time? A full-blown war between the two entities? And if it is a matter of time, does it have the potential of becoming a nuclear war? Is the Eurozone viable over the next 10 years?

• 43:38Speaker 02

  I mean, these are long-term questions that you really are difficult to pose in a forecasting tournament. But what you can do in forecasting tournaments, though, is try to bridge the rigor relevance gap. You say, there are these big abstract questions we want answers to, but they're not rigorously resolvable in the short term. What can we do? We say, well, the idea here is you take a big scenario you're interested in. Say you're interested in whether advanced forms of artificial intelligence are going to cause major dislocations in white-collar labor markets in the next 20 years. No more radiologists, no more airline pilots, no more, you know, replaced by machines. If you believe that in that scenario, and you want to figure out more rapidly than you could otherwise could, whether we are on that historical trajectory toward that

• 44:21Speaker 02

  outcome, you get together some creative experts in groups, and you instruct them to work together in certain ways, to generate insightful early warning indicator questions. Questions like, you know, would artificial intelligence, when the world go, beat the best player in Go by the end of, I think, 2016 or 17? That did happen. Another question is whether an AI system would defeat a top US Air Force pilot in a simulated dogfight. And I think that did happen before 2021. Another question is whether Watson M D would defeat the world's best medical diagnosticians in a medical diagnosis tournament. By the end, I think 2019 or 2020, that did not happen.

• 45:06Speaker 02

  So what does that tell you? It tells you that there is some truth to this idea. That A I is going to cause major dislocations in white collar labor markets. Is it going to be as massive as implied by that scenario? Will it be as early as 2040 or 2035 or 2030? Or will it be 2060 or 2070? It's probably going to happen, but the time framing might be a little bit slower than the techno optimists would hope or the techno pessimists would fear.

• 45:32Speaker 01

  So one aspect of the rigor relevance gap is that we care about things that are going to happen far in the future or not going to happen far in the future. A separate issue is that the questions are just kind of big and nebulous and not well defined enough to make predictions about.

• 45:46Speaker 02

  The big things that people are debating are not typically well defined enough to put into a forecasting tournament. Are we moving into a era of hyperinflation? You know, that's a that's a vague question. You need to specify that a lot more. And similarly, with respect to strong A I causing major white collar market dislocations. Oh, that's kind of a pretty vague too, isn't it? When you think about it, it sounds smart. It sounds intelligent. It sounds meaningful, but on close inspection, it needs to be decomposed into micro indicators. And those micro indicators are things you can run in forecasting tournaments and can help you make the conversation smarter faster than it otherwise would advance.

• 46:24Speaker 01

  There's another technique that I know you've been working on recently that I think is designed to help close this rigor relevance gap. It's called reciprocal scoring. Could you explain what that is and what purpose it's supposed to serve?

• 46:37Speaker 02

  It's a brilliant idea that is still being tested in the lab. And so we're it's a little premature to announce, you know, Eureka. But the core idea is this, you put together two panels of people. On those panels of people, you put individuals who have very good track records of being accurate.

• 46:57Speaker 02

  And value only one thing. They only value their reputation for continuing to be accurate. So let's say you populate the two panels with super forecasters. And the ideal super forecaster has no ideological skin in the game. The ideal super forecaster just wants to continue to have an elite track record. And you know, if that means I'm going to be expelled from a political party, I don't care. Okay, so you have two panels of people, each of whom have this desire. And you give them a difficult question that cannot be objectively resolved in a regular forecasting tournament. You give them a question about, well, you know, if the Federal Reserve were suddenly to do X, Y, and Z in tightening the money supply. What would US economic growth inequality and so forth look like in 2023 or 2024?

• 47:42Speaker 02

  You ask the question, if they did this immediately, right now. Now, as soon as you ask that question, it becomes a counterfactual, right? Because it's not going to happen immediately. It's already a counterfactual question as soon as I ask it, because it hasn't happened this in the past. But I said, if they did this right now, at this second, what would the effects be on US economic growth, inflation, et cetera? And you get each panel to try to predict the predictions that the other panel would make.

• 48:09Speaker 02

  So now we have an accuracy standard. How accurate are you predicting the predictions of the other? And you know that the other side highly values its accuracy. That's the one thing they really want to be. So it's really, I'm challenging you to be your best cognitive self. I'm saying to you, look, you know, you might as well put aside your ideological priors here and any other stuff that you know, deep inside your soul is BS. You know, put all that aside and just try to do your very, very, be all that you can be cognitively. By trying to predict the prediction to these super smart people on the other side. And by the way, they're doing the same thing. So that's a very interesting exercise. And you think about it, if you don't imagine you're the chairman of the Federal Reserve. What, you know, what would you value more, knowing the answer to the question I just asked? You know, what would be the impact of these policy options?

• 48:54Speaker 02

  Or would you more, would you value more knowing what the inflation rate and economic growth is going to be a year from now? You'd probably want to know what the policy impacts would be. Right. And you don't want to have to wait a year to find out what the inflation rate is. Because I think that's what you're going to have to do. Now, for typical scoring, you can find out tomorrow what the right answer is. Because you just have to look at how well each team forecast the predictions of the other teams. So you get an answer virtually immediately. Now, it's a different conception of truth. It's not an objective ground truth. What you have is intersubjective ground truth. That's what you have. And, and now you say, well, that's not as good as objective. And I say, you're right, Julia. It's not as good, but it's a heck of a lot better than purely subjective.

• 49:36Speaker 01

  Right.

• 49:36Speaker 02

  The, the big advantage of doing reciprocal scoring is that you can extend it into domains where there is no objective answer. And it turns out there are many things in life like that.

• 49:46Speaker 01

  Great. Well, Phil, we're just about out of time, but since this is the Ezra Klein Show, we're going to end the podcast in the traditional Ezra Klein Show manner by asking you for book recommendations. So what are three books that thoughtful citizens should read?

• 50:03Speaker 02

  Okay. Well, no surprise, I'm going to recommend Daniel Kahneman's Thinking Fast and Slow for all the reasons we've been discussing over the last many minutes. I would also recommend Steve Pinker's Enlightenment Now, which is of course, we're, we're much healthier and wealthier than we otherwise would be. Here's a counterfactual, we're much healthier and wealthier than we otherwise would be if it hadn't been for the Enlightenment. Hadn't, it hadn't been a major pivot in how human beings think about knowledge in the 17th, 18th centuries. So that those would be two, and this here's third book is, and okay, both those people are famous. They, they're famous writers and they're, and they're, and they've reached popular audiences. Now, here's, here's a third book.

• 50:45Speaker 02

  It's written by a Columbia professor of political science, whom I've known for many years, admired for many years. Book never got on the bestseller list, but it's an influential book in political science. I think it was brilliant. It had a big impact on my early career. It came out in 1976. It's by Robert Jervis, Perception and Misperception in International Politics. If you look at the description in that book of, you know, how to balance deterrence and reassurance and relations between the US and the Soviet Union, or how it failed in, you know, with, with Germany in, in the 1930s. If you look at the descriptions of those sorts of things and you think it's just, it's totally relevant to US-China today. It's totally relevant to everything going on in the world.

• 51:28Speaker 02

  Yeah, it's, he's a rare scholar who has a, a deep understanding of both international politics and human psychology. And I continue to think of it as a masterpiece, but much less well-known than the other two books.

• 51:40Speaker 01

  Well, maybe the Ezra Klein Show can, can bump it onto its rightful place on the bestseller list. We'll see.

• 51:46Speaker 02

  It's a somewhat difficult read, but okay.

• 51:49Speaker 01

  The honesty of a fox. I appreciate it. Well, Phil, thank you so much for coming on the Ezra Klein Show and letting me pick your brain about the difference between foxes and hedgehogs and super forecasters and unsuper forecasters. And I'm excited to see where your new experiments in ways to close the rigor relevance gap lead us.

• 52:09Speaker 02

  Well, thank you very much. Wonderful talking with you, Julie.

• 52:22Ezra Klein?

  The Ezra Klein Show is a production of New York Times Opinion. It is produced by Jeff Geld, Roji Karma, and Annie Galvin. It's fact-checked by Michelle Harris and original music by Isaac Jones. Mixing by Jeff Geld.