#6 Philip Tetlock: How to See the Future

In this episode of the Knowledge Project, I chat with professor and New York Times best-selling author Philip Tetlock about how we can get better at the art and science of predicting the future. Go Premium: Members get early access, ad-free episodes, hand-edited transcripts, searchable transcripts, member-only episodes, and more. Sign up at: https://fs.blog/membership/ Every Sunday our newsletter shares timeless insights and ideas that you can use at work and home. Add it to your inbox: https://fs.blog/newsletter/ Follow Shane on Twitter at: https://twitter.com/ShaneAParrish Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:08Shane Parrish?

  Bienvenidos al Knowledge Project. Soy el host Shane Parrish, editor y director en chief de The Farnham Street Blog, un sitio web con más de 70,000 lectores dedicados al dominar el mejor de lo que otras personas ya han descubierto. El Knowledge Project me permite entrevistar a increíbles personas de todo el mundo para desconstruir por qué son buenos en lo que hacen. Es más conversación que prescripción.

• 0:32Shane Parrish?

  En este episodio, estoy feliz de tener a Philip Tetlock, profesor de la Universidad de Pennsylvania. Él es el co-leader del Good Judgment Project, que es un estudio de pronóstico multiaños, y también es autor del recientemente publicado Super Forecasting: The Art and Science of Prediction. ¿Cómo podemos mejorar en la predicción es el tema de esta entrevista. Vamos a profundizar en lo que hace que algunas personas sean mejores y en lo que podemos aprender para mejorar nuestra capacidad de predecir el futuro. Espero que disfruten de la conversación tanto como yo.

• 1:08Shane Parrish?

  Quiero hablar sobre su nuevo libro, Super Forecasting: The Art and Science of Prediction, que escribió con Dan Gardner, quien, como yo, creo que todavía está basado en Ottawa. En el libro, dice que somos todos predecadores. ¿Puedes elaborar un poco?

• 1:23Speaker 01

  Bueno, es difícil tomar cualquier decisión en la vida, ya sea una decisión de consumidor sobre si comprar un carro o una casa, o si casarse con una esposa, una posible esposa, o un candidato para votar en una elección. Es muy difícil tomar cualquier decisión sin formar al menos expectativas implícitas sobre cuáles serán las consecuencias de esa decisión. Así que siempre que estás tomando una decisión, hay probabilidades implícitas construidas en eso. So the question becomes, are you better off with implicit probabilities that you don't recognize as probabilities, or explicit ones? And I think one of the major takeaways from the forecasting tournaments we've been

• 2:07Speaker 01

  running is that when people make explicit judgments and they're fully self-conscious about what they're doing, they can learn to do it better.

• 2:16Shane Parrish?

  And you're talking about the Good Judgment Project. Can you maybe introduce us to that a little?

• 2:20Speaker 01

  Sure. Well, the Good Judgment Project is a research program that my wife, Barbara Mallers, and I started several years ago. It was supported by a branch, research and development branch of the US Intelligence Community, known as IARPA, Intelligence Advanced Research Projects Activity, which models itself after DARPA in the Defense Department. And their mandate is to support research that has the potential to revolutionize intelligence analysis. So, working from that mandate, they decided in 2010 to support a series of forecasting tournaments in which major universities would compete, researchers at major universities would compete, to generate accurate probability estimates of possible futures of national

• 3:05Speaker 01

  security relevance. And we were one of the five teams selected for the competition in 2010. The tournaments ran from 2011 to 2015. They ended in June of this year. And the Good Judgment Project, I am proud to say, was the winner of those forecasting tournaments. And I can explain more about what winning a forecasting tournament means later, if you want.

• 3:31Shane Parrish?

  Congratulations, yeah, definitely. Is there a difference between forecasting and predicting?

• 3:35Speaker 01

  I don't see one. I think if you go to a thesaurus, I think we're going to find our virtual synonyms. Some people may try to draw distinctions of one sort or another, but I see them essentially as distinctions without a difference.

• 3:47Shane Parrish?

  And so, were you using a representative subset of the Good Judgment Project, or were you using super forecasters from the project? Or how are you competing in that?

• 3:57Speaker 01

  Well, different universities and different teams of researchers took different approaches to generating accurate probability estimates. We recruited thousands of forecasters.

• 4:10Speaker 01

  And we explored a number of different techniques for eliciting the best possible probability estimates from those forecasters. We're continually running experiments. And one of the experiments we conducted was to identify top performers in each year, the top 2% of performers each year, cream them off into teams, elite teams with super teams of super forecasters, and give them as much support as we could, intellectual support as we could for their task, and see how, see what would happen. And they really went to town. They did a phenomenally good job. They blew the ceiling off all of the performance expectations that are up ahead for what was

• 4:55Speaker 01

  possible. And frankly, they certainly exceeded my expectations as well.

• 5:00Shane Parrish?

  So some of us are good and some of us are bad, and some of us seem like way off the chart at making predictions. Why are some people so good?

• 5:07Speaker 01

  That is indeed the $64,000 question. Why are some people so good? So the skeptics argue that if you toss enough coins enough times, some of them are bound to come up heads. So the super forecasters are just super lucky. So let's treat that as kind of the default skeptical hypothesis. There's nothing special about super forecasters. If we ran a tournament in which the task was, say, to predict whether a fair coin would land heads or tails, some people would do better than others just by chance in a given year. We could anoint those people as super coin toss predictors, and we could say, well, how are they going to do the next year? And what we would find is perfect regression toward the mean.

• 5:49Speaker 01

  The best prediction is that the super coin toss predictors in year one will be essentially around the average in year two. And the worst predictors will regress upward toward the mean, of course. So that's what pure, that's what a pure chance environment would look like. What we find in the ARPA tournament is that there certainly is an element of chance in predicting geopolitical and geo-economic outcomes, but the scale luck ratio seems to be about 70/30. Uh, so you're not observing a great deal of regression toward the mean among super forecasters, but there inevitably is some regression toward the mean among the top performers.

• 6:23Shane Parrish?

  Until what makes those people so good?

• 6:25Speaker 01

  Well, now that we've eliminated or at least rendered implausible the super lucky hypothesis.

• 6:30Shane Parrish?

  Yeah.

• 6:31Speaker 01

  Uh, the question becomes, what are the attributes these super forecasters have? They might, they might, you might think of them as being stable psychological attributes. Are they, they score higher on measures of fluid intelligence or crystallized intelligence or active open mindedness? Uh, did they have certain attitudinal profiles, certain behavioral profiles? And the answer is all of the above. They, they, super forecasters, uh, differ from ordinary mortals in a, in a host of ways. They're not radically different from ordinary mortals, but they are systematically different. Uh, they tend to score higher on measures of fluid intelligence. They tend to score higher on measures of active open mindedness. But if I had to identify one factor that I think best distinguishes super forecasters

• 7:13Speaker 01

  from, from other forecasters who are equally intelligent and equally open minded, it is that super forecasters believe that probability estimation of real world events is a skill that can be cultivated and is worth cultivating. Uh, and they're willing to make that commitment, that effort. So when people ask me, how could the super forecasters have outperformed, say, uh, intelligence analysts who do this full time and have access to classified information? I think the short answer is it's not because they're, they're smarter and it's not because they're even more open minded, although they are pretty open minded. It's because they are willing to make this, this commitment, this act of faith, that there is a skill that is worth cultivating. So in the book, we, we quote Aaron Brown, who's the, um, chief risk officer at AQR

• 7:57Speaker 01

  and also a great poker player, um, that he, his view is you could distinguish great players from, uh, talented amateurs on the basis that great players are good at distinguishing sixty forty bets from forty sixty bets.

• 8:10Shane Parrish?

  Hmm.

• 8:11Speaker 01

  And then he paused and says, no, maybe more like fifty five, forty five, forty five, fifty five. Uh, the greatest players tend to be extremely granular in their assessments of uncertainty. One of the big questions I think that IARPA wanted us to answer and that I think we have answered in the affirmative is. Does granularity in assessments of uncertainty pay off, not just in poker, but in, um, when you're making messy real-world judgments, like whether Greece is going to leave the Eurozone, or what kind of mischief Putin might be up to in the Ukraine next, or what's going to happen with Sino-Japanese relations in the East China Sea, or is there going to be another outbreak of bird flu in a given region?

• 8:53Shane Parrish?

  Right.

• 8:54Speaker 01

  Uh, these are extremely idiosyncratic, one-shot historical events. It's not like poker, where you're sampling from a well-defined sampling universe, repeated play, quick feedback. Uh, so there are a lot of people, very smart people, have been skeptical for many decades that it's even possible to make probability estimates of these kinds of intelligence analytic problems. And I think what the IOPA tournament has proven, I, the unreasonable doubt, in my opinion, is that there is room for improvement. It's possible to make these probability estimates, it's possible to get better at it, it's possible to identify the kinds of people who learn to do it better, um, it's possible to develop training modules to help people do it better, um, and the gains in accuracy are appreciable.

• 9:35Shane Parrish?

  So what happened when you took, um, average people and you started giving them, I, I think I remember this, that you started giving them a course in probability?

• 9:44Speaker 01

  Uh, we get about, for, for average forecasters who are randomly assigned to an experimental condition in which they get Kahneman-style de-biasing exercises, um, the improvement is in the, in the vicinity of 10%.

• 9:58Shane Parrish?

  And that, that's-

• 9:59Speaker 01

  And that's, and that's a big effect when you consider that, you know, we're talking about improvement across an entire year of forecasting, and this training exercise takes about 50 minutes.

• 10:08Shane Parrish?

  And what did that consist of, this 50-minute training exercise?

• 10:13Speaker 01

  Some basic ideas about, um, uh, heuristics and biases and how to check biases. Uh, for example, one of the classic Kahneman arguments is that, um- People don't give enough weight to statistical or base rate information in assessing the probabilities of events. They're too quick to take the inside view. So if you're attending a wedding and you see the happy couple and you're impressed by how much in love they are and the enthusiasm of the moment, and someone asks you, how likely are they to get divorced? You're not likely to consult national divorce statistics for that SES subgroup. You're likely to say, hmm, they look really happy and compatible.

• 10:56Speaker 01

  I'm going to attach a very high probability to they're not getting divorced.

• 10:59Shane Parrish?

  Right.

• 11:00Speaker 01

  And the net result of making predictions in that way is that you're going to be somewhat less accurate than you would have been if you had at least started your estimation process by saying, what are the base rates of divorce? And now I'm going to adjust that based on whatever idiosyncratic factors are present in this particular relationship.

• 11:18Shane Parrish?

  So starting with the outside view and working your way inside?

• 11:21Speaker 01

  Exactly. Start with the outside and work inside. That's one of our mantras.

• 11:27Shane Parrish?

  So, but isn't Kahneman famous for saying that he's studied biases his whole life and he feels like he's no better at avoiding them? So how does this 50-minute training exercise come in and help people?

• 11:38Speaker 01

  Well, Danny Kahneman was a colleague of ours at Berkeley. My wife and I, we know him well. And we know that he is more pessimistic about the prospects for de-biasing than we are. He did give us advice on how to design the de-biasing modules. I think he probably is more of a pessimist still than we are, but I think he is persuaded that these improvements are real.

• 12:02Shane Parrish?

  They certainly seem to be. So one of the keys to keeping track of forecasting and your ability to predict is kind of keeping score. And do you think it takes a certain type of person to want to keep score? I mean, most of us are happy to kind of weasel out of or use uncertain wording or jargon when we're going about making decisions so that even if we're wrong, we can kind of say, well, that's not what I meant.

• 12:29Speaker 01

  Absolutely. It does take a particular type of person and there are many factors that come into play. I think it certainly helps to be open-minded, but there are other things that come into play that are a little more, say, sociological. I've been doing forecasting tournaments for over thirty years now, and I started when I was about thirty in nineteen eighty-four. I'm sixty-one years old now, so if I were an intelligence analyst, a sixty-one year old intelligence analyst, I would be a very senior analyst.

• 12:59Shane Parrish?

  Mm-hmm.

• 13:00Speaker 01

  And let's just say for sake of argument that I am a senior analyst in the US intelligence community. I'm on the National Intelligence Council, say, just for sake of argument, and I'm the go-to guy on China. So when Xi Jinping comes into town, people say to me, you know, what's going on? I have inputs into the presidential daily briefing and help with national intelligence estimates, and I'm at the top of the status checking order within the IC on China. And someone comes along, like IARPA, is this upstart research and development branch for the Office of Director of National Intelligence, and they say, hey, you know what we're gonna do? We want to run forecasting tournaments now. And everyone's gonna compete

• 13:41Speaker 01

  on a level playing field. And twenty-five-year-old China analysts are gonna compete against sixty-one-year-old analysts like Tetlock, and we're gonna see who does better. Are the sixty-one-year-old analysts going to welcome this development?

• 13:55Shane Parrish?

  No.

• 13:56Speaker 01

  To ask is to answer. Even open-minded sixty-one-year-olds are not gonna be very enthusiastic about this. They're going to argue that these tournaments just don't really capture what makes my judgment special. And that is indeed a lot of the resistance we run into for forecasting tournaments. I mean, in the book, you may remember, we talk about the parable of two forecasters at the beginning, Tom Friedman and Bill Flack.

• 14:18Shane Parrish?

  Mm-hmm.

• 14:19Speaker 01

  Almost everybody who reads newspapers knows who Tom Friedman is, famous New York Times columnist, Middle East expert, you know, often in the White House or Davos or God knows where. Um, and Bill Flack, nobody has the faintest idea who he is because he's an anonymous retired irrigation specialist in Nebraska. Who um, happens to be a super forecaster. And we know a tremendous amount about Bill Flack's forecasting track record. We know almost nothing about Tom Friedman's forecasting track record.

• 14:46Shane Parrish?

  Right.

• 14:46Speaker 01

  Uh, and that's in substantial part because Tom Friedman's forecasts, and he does make forecasts, are uh, in, in embedded in vague verbiage. Uh, he says that this could happen, or this might happen. And when you say something could or might happen, that could mean anything from point one to low point nine, uh, in probability terms. And, you know, if it, if it happens, I can say, well, I told you it could. And if it doesn't happen, I can say, uh, uh, look, I merely said it could.

• 15:13Shane Parrish?

  Right. You can't get pinned down.

• 15:14Speaker 01

  You've covered very nicely. Yeah.

• 15:16Shane Parrish?

  Do you think that that's one of the problems with organizations? I mean, it seems like we're not getting better as organizations at making decisions. In part because our ability to keep score is, you know, hampered by these psychological kind of effects where, you know, if I keep score, I might be wrong, so my incentive is not to. And if I use precise wording, I might be wrong, so my incentive is not to.

• 15:39Speaker 01

  Yes. Yeah. I think there's a whole mix. There's a real mixture, powerful mixture of psychological and political forces. That uh, interact to, to, to create a lot of resistance to forecasting tournaments. Uh, so even though I think we have shown that forecasting tournaments can appreciably improve probability estimates, there are a lot of reasons why organizations don't adopt them. One is the people at the top of the status hierarchy are not very enthusiastic. Bob, who's in the CEO suite, isn't all that enthusiastic about, uh, being discovered that Bob in the mailroom is just as good as he is at anticipating trends relevant to the company's future. So you, you have the status hierarchy problem. Uh, people at the top don't want to be a second guest. They don't want their judgment process to be demystified.

• 16:23Speaker 01

  Uh, a large part of status is, is, in, in contemporary organizations is that there's something special about your judgment. So even open-minded high-status people are going to be reluctant to do this because it's going to look like a career-damaging move. So there's certainly that. And there's another, there are a lot of other factors in play. There's, again, this Kahneman argument that people don't pay attention to the outside view. We, in the book, we talk about a mistake that a New York Times, a famous New York Times journalist, David Leonhardt, you may know him, he runs the Upshot column in the New York Times. He's a quant-savvy journalist, and he made a mistake in 2012 that we talk about that

• 17:07Speaker 01

  illustrates just how tenacious the misconceptions can be. He was commenting on the Supreme Court decision to uphold Obamacare in 2012. It was a narrow decision, it was 5-4. And he noted that the prediction markets had had futures contracts on this decision, on the Supreme Court decision, and they were pricing it at about a 75% probability of the law being overturned. Okay, so they were way off. And he said, "Well, how far off is way off?" He said, "Well, they got it wrong." He just said flat-out, "Got it wrong." That doesn't account for the complexity, right? That, that, that, that, that itself is wrong. Right. It certainly isn't good news that the prediction, the good prediction market

• 17:51Speaker 01

  that it was on the wrong side, it may be by that margin, but prediction markets have generated hundreds of forecasts over many years, and they've proven to be pretty darn-well calibrated. Which is another way of saying, when they say 75% probability of something happening, things happen about 75% of the time, and they don't happen about 25% of the time. So even if you have a perfectly calibrated prediction market system doing, when it says 75%, 25% of the time, smart observers, observers as smart as David Leonhardt, are going to be tempted to conclude that you're wrong. Right. And to dismiss you. So this creates a huge political incentive to stick with vague verbiage. If they simply said, "It could be overturned." You know, they would be well positioned to explain it either way.

• 18:34Speaker 01

  Um, but because they were put the prediction market was generating these precise probability estimates and because people don't take the outside view and say, well, we can't just look at that particular forecast, we have to put it in the context of all these other forecasts that the system is generating, take the outside view toward the system. People have a very hard time doing that. Um, and David Leonhardt knows that this is true and he's even written later in the upshot about situations in which, uh, you know, about this fallacy. Um, so if someone as smart as that, who doesn't have a grudge against prediction markets can make a mistake like that, you can see why, um, politically savvy intelligence analysts might be reluctant in a blame game culture like DC to do it.

• 19:16Shane Parrish?

  Right. Uh, I think one of the most interesting parts of the book for me was when you started talking about the Fermi style thinking. Can you, can you introduce us to that?

• 19:25Speaker 01

  Well, Enrico Fermi was, um, Italian-American physicist who developed the first, uh, nuclear reactor at the University of Chicago. He was involved in the development of the atomic bomb in World War II. And, um, he was known for his, uh, rather flamboyant, uh, thinking style. Uh, he was continually coming up with innovative ways of estimating the seemingly unestimatable. Um, you know, one of, one of the famous examples of a Fermi problem was, you know, it sounds really weird, it was to estimate the number of piano tuners in Chicago. Other examples might be estimating how much the Empire State Building weighs or, or estimating the likelihood of extraterrestrial civilizations elsewhere in the Milky Way.

• 20:07Shane Parrish?

  Sounds a lot like the brain teasers that Google used to ask to hire, right?

• 20:11Speaker 01

  Exactly. Now, I don't know whether Google, whether the, the legal department still allows Google to continue using those for screening potential personnel, but they are interesting tests of how people approach problems.

• 20:24Shane Parrish?

  And what was so interesting about the way that Fermi approached it?

• 20:27Speaker 01

  Uh, he really believed in flushing out your ignorance and decomposing, decomposing the problem into as many tractable components as possible. So you would start by, you know, how many, how many stars are there in the Milky Way? Uh, roughly about a hundred billion. Uh, you'd say, well, how many, um, of these stars have planets orbiting around them? You might look at the most recent data from Kepler, which has done some reconnaissance in our local area, about sixty light years around, and, um, you say, well, you know, it looks like fair number, a pretty high percentage of stars do seem to have planets going around them. Uh, let's say it could be as much as half, um, or maybe slightly less.

• 21:11Speaker 01

  I'm, I'm, but you, you, but you, I don't really know the answer to that question, but you make, you, you make, you make initial guesses. You flush out your ignorance, and then other people can come back and they can see that Tetlock's had about half, and they say, oh, Tetlock doesn't understand what Kepler's doing. It should have been seventy percent. No, it should have been thirty percent. Uh, but what, what, it's not that Tetlock is getting it right, it's that we're flushing out Tetlock's zone of ignorance, and we're making it clear, and, uh, it's all open and transparent. Uh, and then we'd, you know, and then that process of inquiry would continue, how many planets are in habitable zone? And you derive some further guesstimate from Kepler, it's a fairly small fraction of planets seem to qualify for that, and, um, but that still might leave you with, uh, say, um, as

• 21:55Speaker 01

  many as five hundred million to a billion planets that are potentially inhabitable zones. And then you'd have to make some estimate about how likely is life to jumpstart if you have a planet in the habitable zone, and how likely is intelligent life to converge once you have? And there, there are different evolutionary theorists who have different models, uh, that, that, at least have somewhat different implications, uh, as answers to those questions. And what you would wind up with would be ranges of probabilities. Now, for this particular problem, the range of possible probability is going to be very large. You know, we know it's not impossible, there's an under advanced extraterrestrial civilization in the Milky Way. We also know it's not a sure thing. Uh, it's probably, uh, you know, my, in my, my best estimate, if I were to combine all

• 22:38Speaker 01

  the different steps that we just started to work through, would be, it would be probably more than one or two percent, and, but I, I, I don't think it would be as high as ninety percent, it would probably be. Maybe between two and fifty percent. Now, that's a guesstimate. Now, there's nothing special about that number. But what Tetlock has done now, if he's flushed out, Tetlock, me, I'm talking about myself the person here. What the Fermi person, the Fermiizer, the person using the Fermi method has done is he or she has flushed out all the different points of ignorance along the reasoning continuum. And you, the observer, can say, oh, look, Tetlock made a really stupid estimate here, and we have to adjust that. But it's a basis for proceeding.

• 23:23Speaker 01

  And what would initially look like a hopelessly intractable problem at least becomes at least a little more tractable. And that's what super forecasters are pretty good at doing, at breaking down seemingly tractable problems into semi-tractable components and then just pushing. They're not afraid of looking stupid and making estimates that observers can see and look at and say, oh, my God, why did you say something that stupid about the capital project?

• 23:47Shane Parrish?

  That's an incredible point, where you're taking this big, intractable kind of problem that's very hard to pin down, and you determine, you have some organized process for determining the subcomponents involved to get you there, and then you go through and estimate. So part of that would be highlighting your thinking, right?

• 24:05Speaker 01

  Yes, sir.

• 24:06Shane Parrish?

  And then part of that would be like, I really don't know anything about this question, so can I break that down further into subcomponents, or am I extrapolating too much?

• 24:15Speaker 01

  No, that's exactly the spirit of the enterprise.

• 24:17Shane Parrish?

  So why is that style of thinking, why does it lend itself, do you think, to better forecasting? Is it just the nature of the changing the framing of the problem itself? Or do you think it's more the curiosity of the people who are willing to break it down and go through? It sounds like a lot of work. It sounds very demanding and mentally taxing to do that versus just throw out an estimate with your immediate response?

• 24:44Speaker 01

  You're exactly right. It is demanding, and I think it works best if it's done in a team environment, in which members of the team have mutual respect for each other, but they're also willing to push each other hard.

• 24:57Shane Parrish?

  So if you were an organization, you wanted to set up a team environment, like a forecasting team, within a large company, say IBM, how would you go about doing that with your knowledge?

• 25:10Speaker 01

  That's a great question, and I'm a little bit wary about saying that organizations should try to construct super teams the way the Good Judgment Project did. Because team construction has a lot of implications for other parts of the organization that can be tricky. I mean, imagine that if you, if you just did what we did in the IARPA tournament to win it, and you just identified the very best people, brought them together, and nurtured them, and helped them, pushed them hard. That would be a very elitist and somewhat divisive thing to do in many organizations.

• 25:45Shane Parrish?

  Yeah.

• 25:46Speaker 01

  And it could cause a lot of political friction. Now, we didn't care a lot about that, because we were in a forecasting tournament. We didn't really have an organization in the traditional sense of the term. We wanted performance engine.

• 25:57Shane Parrish?

  Right.

• 25:58Speaker 01

  We wanted to harness human ingenuity individually and collectively, as rigorously as possible to generate as accurate as possible probability estimates for things that you tell this community cared about. That was it. It was a pure accuracy game, and we, we weren't, we weren't that interested in the long-term viability of the organization. We were interested in just pure accuracy. So, I would, I would be a little cautious about saying, you know, it's really easy. All you do is you recruit these super forecasters, and you put them into these teams, and you give them some training on how to do precision questioning, and you give them some training on how to do constructive confrontation, and you've got these anti-group think norms enforced, and you give them some training and guidance and probabilistic reasoning, and you encourage a certain self-critical structure and culture inside the teams, and

• 26:43Speaker 01

  boom.

• 26:44Shane Parrish?

  Magic happens.

• 26:45Speaker 01

  Amazingly accurate forecasts emerge. It works pretty well in, in, in, in the forecasting tournament environment, but whether it would work well in an actual organization, I think the senior executives would want to think carefully about each step, each step along the way there.

• 27:00Shane Parrish?

  What would you say to people inside an organization? How can they use your research to make better decisions inside their company?

• 27:09Speaker 01

  Well, I think it's something you want to consider seriously that when people make forecasts inside most organizations today, accuracy is only one of the goals that they're pursuing. They're also interested in making forecasts that are going to be difficult to falsify. I said they can't be embarrassed. So a lot of the forecasting inside organizations doesn't involve numbers. It involves a lot of vague verbiage. They're also interested in making forecasts that don't annoy other people in the organization. They don't want to tip the political apple cart over. So they're compromising accuracy in a whole host of ways that help promote their careers

• 27:49Speaker 01

  inside the organization, help to maintain political stability in the organization, but that aren't all that centrally focused on accuracy. Forecasting tournaments are really weird because they focus 100% on accuracy. That's all that matters. So I guess the thing you'd want to consider as an executive would be, do I want to reserve a fair part of my organization's analytical processing capacity for a pure accuracy game? Do I want to incentivize some small group of the people in my organization to play pure accuracy games in forecasting tournaments and those probability estimates would then filter up to senior executives to guide decision making? I think it's really an interesting experiment to consider doing.

• 28:32Speaker 01

  I think the intelligence community has been moving somewhat in that direction. I think it's a good idea and I think it would probably be a good idea for many other entities as well, at least to consider. It's in the spirit of the whole IRF enterprises to run experiments.

• 28:47Shane Parrish?

  Right.

• 28:48Speaker 01

  And what I would propose would be that senior executives consider running experiments in which they see what do they discover when they incentivize people to play pure accuracy games?

• 28:57Shane Parrish?

  And do you think, what transfers from your research into the decision making process in a corporation, not necessarily about forecasting, but about how we go about organizing, unpacking, synthesizing, multiple views? How does that transfer, do you think, into a learnable skill that people can have inside of an organization?

• 29:19Speaker 01

  There are many ways that could happen. We put a lot of emphasis in the Good Judgment Project on synthesizing diverse views into aggregate forecasts.

• 29:29Speaker 01

  And I think one of our major performance engines was the statistical or aggregation algorithms that our statisticians developed for doing that. When IARPA started this whole exercise, they thought it would be really hard to do better than twenty or thirty or forty percent better than the unweighted average of a group of control group forecasters. And our super forecasters exceeded that performance benchmark quite substantially each year of the tournament. They did so well that IARPA essentially suspended the tournament after two years and we were able to absorb the other teams into our team in substantial ways and compete against the intelligence community and against the prediction market baselines instead of the other universities.

• 30:14Speaker 01

  Now, how did all that come to pass? I think the aggregation algorithm developed, if I had to credit two big things as responsible for the victory of the Judgment Project, one of them would be the super forecasters and the other would be, call them the super algorithms, the great algorithms that our statisticians develop. Now, when I describe these algorithms, at some level, you're not going to be too surprised at first, but there is one aspect of them that does surprise most people. So, the first thing to do, I don't know if your listeners are familiar with the James Surwicky Wisdom of the Crowd book, but it's been well-known in the forecasting world

• 30:56Speaker 01

  that the average of a group of forecasters, the average forecast from those forecasters is going to be more accurate than most of the individuals from whom the average was derived. Right. And this is the famous Gell-突in story about the ox. You had hundreds of people trying to guess the weight of the ox, and the average of all those guesses was only about one or two pounds off from the original, from the true weight of the ox. And that would, that means it was more accurate than all of the individuals from whom the average was derived. So averaging is a powerful way of synthesizing information from diverse perspectives. It's, it's really, it's remarkably crude approach to doing it, but it works pretty darn well, and that's why IARPA used it as its benchmark.

• 31:37Speaker 01

  Uh, now you, we were able to beat averaging by doing, uh, some simple things, like giving more weight to better forecasters, but as we got more and more data on who the good forecasters were, who the more intelligent forecasters were, who the more frequent belief updaters were, uh, various attributes of forecasters, we were able to give more weight to certain forecasters and we created weighted averages. Right. And the weighted averages beat the average. Hey, that's not, that's not too surprising, is it? No. I mean, it makes sense. Uh, it's not astonishing, though. Um, now, here's the interesting thing that, that the algorithms did, uh, they did something called extremizing, and to illustrate extremizing, I, I want, I wanted just to have a little

• 32:17Speaker 01

  digression, uh, of a story about, that we, we do talk about in the book, uh, about, um, the decision President Obama made to go after Osama bin Laden. He was, uh, in the movie Zero to thirty, they have a scene in which, you know, the senior analysts are being polled on how likely they think it is that Osama bin Laden is in that, is in that compound. And putting aside what Hollywood says about it, let, let, let's just do a little thought experiment. And imagine that you're the President of the United States, and you have these senior advisors around the table, and you ask them, how likely is it that Osama is there? And each of the analysts around the table says, do you miss the President? And I think the answer is point seven, point seven, point seven. Everybody around the table says point seven.

• 33:01Speaker 01

  What should the President conclude is the likelihood that Osama bin Laden is in that compound? And the short answer to that is, well, if the advisors are all clones of each other, and they're drawing on exactly the same information and processing it in exactly the same way, the answer is point seven, because there's no information added, right? But imagine that the analysts say point seven all around the table, but the analysts don't know each other, and they haven't been sharing information, and each analyst bases the point seven, his, his or her point seven judgment on information that only he or she has. So you have extreme diversity of perspectives. One person has satellite information, another has encryption breaking stuff, and another one has human intelligence and so forth.

• 33:46Speaker 01

  But they're siloized and they're coming together for the first time and each one has independently arrived at this point seven estimate from very different sources of information. You've got diver- you've got true diversity here. And is the answer still point seven? Should the president say shrugged and say, "Well, I think the answer is point seven," or should the president say, "Gee, each of you has very different reasons for believing point seven." This leads me to suppose that the answer is probably more extreme than point seven because if each of you knew the reasons the others had, you would probably become more extreme. And that's exactly what the best algorithm did. It extremized as a function of diversity. So point seven was turned into point eight five or point nine.

• 34:25Shane Parrish?

  That's fascinating. I mean, how did it go about doing that in terms of aggregating the data from the people or from the forecasters?

• 34:34Speaker 01

  That's right, from the forecasters.

• 34:36Shane Parrish?

  And what would happen if you had two forecasters who have great track records and then they're divergent on, they're really divergent on an opinion or a forecast? Is that happen often?

• 34:48Speaker 01

  No, it doesn't happen very often, actually, but if it did happen, it would be a real cautionary moment. If you had two super forecasters, one of whom was at point nine, yet there was a point one, my inclination would be not to stray too far from point five, knowing nothing else at the moment.

• 35:07Shane Parrish?

  Are there certain types of questions to avoid if you, your desire is to have an accurate prediction?

• 35:14Speaker 01

  Yes. Well, there are many questions in the IARPA tournament, there are many questions in life in which there's a massive amount of irreducible uncertainty. If you want to be a good forecaster, you don't spend very much time working on roulette wheel type problems. I mean, if you go to, if you visit casinos, you'll find lots of people who think they can detect patterns in roulette wheel spins. And they develop little algorithms even to help them, but what they're doing is they're essentially modeling randomness.

• 35:44Speaker 01

  Uh, so spending a lot of time modeling randomness is a good way not to become a super forecaster.

• 35:50Shane Parrish?

  What other types of questions would you say don't lend themselves to? Is it like a time duration? Is it?

• 35:57Speaker 01

  Oh, what other kinds of questions are roulette wheel like?

• 36:00Shane Parrish?

  Well, not roulette wheel, but if if you're does a what kind, what type of questions lend themselves to better? Predictions, right? Is it short time, very few? I mean, I don't want to say very few variables, but short time duration versus long time duration, because you have to constantly update over a long period of time, right? I mean, that was one of the things that super forecasters did was they updated their.

• 36:23Speaker 01

  Yes, that's true. Well, yeah, all of the things equal, it's it it's usually easier to to predict questions shorter and with shorter time ranges than longer time ranges, but that's not always true. I mean, some short range questions are extremely unpredictable. It's very hard to say whether the stock market is going to up or going to go up or down tomorrow. Um, so that's a short range question. In in some ways, it's easier to predict where the stock market is going to be up or down 10 years from now relative to now than it is tomorrow, right?

• 36:52Shane Parrish?

  That's a good point.

• 36:53Speaker 01

  Um, so there are categories of problems in which you get a reversal of that. But but yes, I think by and large it's true that the analogy to vision would be right. It's easier to see the smell in I chart if you're close to it than if you're far from it.

• 37:06Shane Parrish?

  Right.

• 37:06Speaker 01

  Uh, probabilistic foresight is better uh or or um in shorter time ranges. And that's one of the things I talk about in the book, one of the reasons why my my later work is different in emphasis from my earlier work, whether in which experts had a hard time beating the art going chimpanzee. Uh, because they were in the earlier work, making much longer term predictions than they were in the I R P work where uh, predictions were rarely much more than a year.

• 37:32Shane Parrish?

  You mentioned open mindedness at the beginning. How do we go about fostering open mindedness? Are there ways that we can improve that in ourselves or other people?

• 37:41Speaker 01

  Well, we try to. That's another thing we do try emphasize in the training. Um, exerting people simply to be open minded is um, most people don't think they're close minded.

• 37:52Speaker 01

  Um, most people think, most, most people think they're quite reasonable. And, uh, simply exhorting people to be open-minded, people shrug and say, well, yeah, I already am. I think you want to start in the more specific ways. You want to start with very specific problems in which you assess whether people change their minds in an appropriate way. So there are some normative models like Bayes' theorem that tell you how much you should change your mind in response to evidence that has certain diagnostic value. And you can create simulated problems. It may be medical diagnosis problems, it might be economic problems, it might be military problems, uh, but you can create simulated problems with simulated data and you can

• 38:32Speaker 01

  see whether people learn through practice to update their beliefs the way they should. Uh, now, there's always a question whether they're gonna, that those lessons are going to stick. And, um, you know, we, we found that they do stick a little bit because they can produce 10% improvement throughout the year. But it's, it's one of the great challenges. I don't think we've solved the problem of how to make people more open-minded. I, I think we can make people, uh, better belief updaters on problems, uh, where they don't have very strong ideological priors or preconceptions. Uh, but when people have really strong emotions and ideological convictions about presidential candidates or economic policy or whatnot, uh, belief updating becomes quite problematic.

• 39:14Shane Parrish?

  Yeah, I mean, I can see why that would be a problem, right? It contradicts probably something that you hold very dear and true. Giving that up would take a lot of, what a mental labor.

• 39:25Speaker 01

  Yeah, we can, we can make people a bit more open-minded, but, uh, making people perfect Bayesian belief updaters is something that no one has achieved yet. And I think, uh, will be very difficult to achieve. You know, I think we should keep working on it. I don't think we should give up.

• 39:40Shane Parrish?

  Do you think the super forecasters were better at learning from the other super forecasters than the, say, average forecaster? Like, if somebody had a better approach, would they copy it? Would they just drop their own internal approach and?

• 39:53Speaker 01

  I think they listen to each other quite carefully in the super forecaster teams. Um, and even when they disagree with each other, they, they, they, they disagree diplomatically, but they can disagree quite forcefully, uh, about what lessons they should draw from particular forecasting failures, uh, or even forecasting successes. I mean, it's, it's fairly common for regular forecasters even to say, well, what, what did we do wrong with a forecasting failure? Um, and superiors do that too.

• 40:21Shane Parrish?

  Hmm.

• 40:22Speaker 01

  Um, but they also second guess their successes. Um, they say, well, were we lucky just be, were we, we, we, we really nailed this question. But were we, um, lucky, uh, could it, could it have gone otherwise? Were we almost wrong? Um, that's an unusual question for people to ask themselves. Um, people don't normally look a gift, gift horse in the mouth and they, when, when, when they're right, they, they, they wanna take credit for it. Uh, and super forecaster skepticism even extends to their, um, their forecasting successes.

• 40:52Shane Parrish?

  I can't imagine a lot of the, the average or, you know, below average in terms of forecasting ability. People went through their successes and evaluated them from that angle.

• 41:02Speaker 01

  Mm-hmm.

• 41:02Shane Parrish?

  Um, what would you say is the role of intuition in, uh, forecasting? Or would you say that it's minimized or would you say that it's?

• 41:11Speaker 01

  This is one of the big debates in the field of judgment and decision making. Uh, Malcolm Gladwell wrote a book called Blink. Um, and some psychologists wrote or joined a book, much less widely read called Think. Um, there, there are different schools of thought about, uh, the value of intuition. Um, and even Gladwell, of course, was devised it in his book. And he, he did point to some, uh, great successes of intuition. They also noted the situations in which intuition could lead you seriously astray. Um, I think the dominant emphasis in our work, it leans toward think over blink. Uh, I'm not ruling out the possibility that there are, um, super forecasters who do rely on intuition.

• 41:53Speaker 01

  Um, but the problems that we're dealing with in real world are different from the sorts of problems where

• 42:03Speaker 01

  brilliant intuition has been demonstrated, uh, pretty rigorously. So it's not like chess where you're playing the same game with well-defined rules.

• 42:13Shane Parrish?

  Right, the pattern recognition.

• 42:15Speaker 01

  Really smart people can do extremely rapid forms of combinatorics and pattern recognition. And, uh, it's, it's quite astonishing what they can do. Um, Real world isn't quite like chess, is it? And I, I, I think that it requires more subtlety and more willingness to second guess yourself. Because history, I think it was Mark Twain who said, uh, history doesn't repeat itself, but it does rhyme. Uh, and I think super forecaster sort of get that, that, that there are patterns in history, but they're quite subtle and they're quite conditional. And if you, you can easily overlearn from history.

• 42:50Shane Parrish?

  Hmm, that's a really good point. What book would you say has had the most impact on your life?

• 42:57Speaker 01

  On my life?

• 42:58Shane Parrish?

  On your life?

• 42:58Speaker 01

  That would have to be a book, that would have to be a book I read very early on in my life.

• 43:02Shane Parrish?

  Oh, possibly, yeah.

• 43:03Speaker 01

  Yeah. I think, well, I, I don't know how far back we should go on this one. I mean, if I were to go back to graduate school, say, when I was making decisions about what I would do with my research career, uh, there was a book by Robert Jervis, who's still, I think he's an, he's maybe an emeritus professor now at Columbia, but he, he's a very senior political scientist. He wrote a wonderful book in 1976, that I was in graduate school, I just started in graduate school in 1976. And it's called Perception and Misperception in International Politics. And it is a wonderful synthesis of psychology and political science.

• 43:38Shane Parrish?

  Hmm.

• 43:39Speaker 01

  Um, and, um, I, I think it is a synthesis of the sort that I aspire to, to, I try to be Jervisian in my work in many ways. Now, Jervis is not a quantitative researcher, he's qualitative, um, whereas I'm more quantitative. Uh, so we differ in a number of ways, but I, I have a deep respect for how he, um, was trying to synthesize the psychological and the political. And I suppose if there's any theme that's running through my work, it's synthesizing the psychological and the political.

• 44:07Shane Parrish?

  So the last question is, who would you like to see interviewed on the show and, uh, their thoughts articulated or explored, uh, with me?

• 44:16Speaker 01

  Well, I've, I've always been a fan of Michael Lewis's work. I, I think he would be a fun person to talk to, and I, I think he may be working on a, on a biography of, of, of Daniel Kahneman and Amos Tversky. I think that would be an interesting, uh, conversation.

• 44:32Shane Parrish?

  Well, excellent. Uh, thank you so much, Phil, for taking the time. Uh, I really appreciate it. It's been a great conversation.

• 44:38Speaker 01

  Oh, it's a pleasure.

• 44:42Shane Parrish?

  Hey guys, this is Shane again. Just a few more things before we wrap up. You can find show notes at farnamstreetblog.com/podcast. That's F-A-R-N-A-M-S-T-R-E-E-T-B-L-O-G.com/podcast. You can also find information there on how to get a transcript. And if you'd like to receive a weekly email from me filled with all sorts of brain food, go to farnamstreetblog.com/newsletter. This is all the good stuff I've found on the web that week that I've read and shared with close friends, books I'm reading, and so much more. Thank you for listening.