The Superforecasters: With Guests Leon Panetta, Peter Bergen & Barbara Mellers (Rebroadcast)

Are there proven ways to improve our predictions?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Katy Milkman?

  Right now, prediction markets, pollsters, and data analysts are working overtime. But predicting the future, especially in high-stakes environments, is notoriously hard. Most of us ignore critical data and place too much confidence in the accuracy of our guesses. The good news: While no forecast is ever foolproof, predicting with greater accuracy is a skill you can learn. That's why we're bringing back a favorite episode. You'll hear how US officials made one of the hardest predictions in modern history: guessing the location of Osama bin Laden. Former CIA Director Leon Panetta and analyst Peter Bergen

• 0:42Katy Milkman?

  walk us through how leadership weighed thin evidence and navigated massive uncertainty. Then, psychologist and Wharton Professor Barbara Mellers breaks down her research on super forecasters and the tools anyone can use to sharpen their predictions and make better choices. Whether it's your first time listening or just a refresher, here is the super forecasters.

• 1:19Speaker 01

  It was in the middle of the night in a mud hut in Afghanistan, which was then controlled by the Taliban. It was March of 97, so it was cold.

• 1:28Katy Milkman?

  This is Peter.

• 1:29Speaker 01

  I'm Peter Bergen.

• 1:30Katy Milkman?

  Peter is an author and journalist, best known as a national security analyst on CNN. In March of 1997, Peter sat down for an interview with a man who would forever alter the course of American history.

• 1:43Speaker 01

  He appeared out of the darkness and sat down and did the interview and basically declared war on the United States.

• 1:51Katy Milkman?

  Peter had his suspicions that Osama bin Laden had planned the 1993 bombing of the World Trade Center in New York. By September 11 of 2001, bin Laden's involvement was no longer in question.

• 2:03Speaker 02

  Oh my god, another plane has just hit. It hit another building.

• 2:08Speaker 03

  You are looking at live pictures of what appears to be an attack on the Pentagon.

• 2:12Speaker 04

  Osama bin Laden, according to this newspaper editor, warned three weeks ago that he would attack American interests.

• 2:19Speaker 05

  All the indicators do seem to point to Osama bin Laden being responsible for this attack.

• 2:25Speaker 06

  Osama bin Laden and other terrorists are still in hiding. Our message to them is clear: No matter how long it takes, America will find you, and we will bring you to justice.

• 2:37Katy Milkman?

  After 9/11, a date that now haunts every American, bin Laden became the most-wanted man in the world, and for good reason. But he proved incredibly elusive. The full force of US military and intelligence was deployed in the hunt for this terrorist leader. And he was tracked to the mountain caves of Tora Bora in Afghanistan. But somehow, he managed to escape, and then the trail went cold.

• 3:03Katy Milkman?

  In this episode, we'll share the story of an incredibly difficult prediction made during the hunt for bin Laden and some key lessons from behavioral science about forecasting that you can apply to your own decisions.

• 3:22Katy Milkman?

  I'm Dr. Katie Milkman, and this is Choiceology, an original podcast from Charles Schwab. It's a show about the psychology and economics behind our decisions. We bring you true stories of high-stakes decisions. And then we examine how these stories connect to the latest research in behavioral science. We do it all to help you make better judgments and avoid costly mistakes.

• 3:59Speaker 01

  The CIA was the lead on finding bin Laden. They realized by 2005. There was no single person who was going to lead them to bin Laden who was in American detention or in detention in a foreign country. They got a lot of false leads. Bin Laden is in Rio de Janeiro, or bin Laden is here. And, you know, all of these were nonsense, as it turns out. But all of them had to be chased down.

• 4:22Katy Milkman?

  After four years of searching, the top US intelligence agency was no closer to apprehending the mastermind behind the September 11 attacks than when they'd started. The CIA would have to change its approach.

• 4:35Speaker 01

  They realized they're not going to find bin Laden directly. They're going to have to find him with a bank shot, to use a pool term.

• 4:44Katy Milkman?

  That bank shot was to focus on the courier network that relayed messages to and from bin Laden.

• 4:50Speaker 01

  The intelligence world is, you know, dealing with imperfect information, fragmentary information. You're trying to put things together.

• 4:56Katy Milkman?

  One fragment of information led to an Al Qaeda operative who had recently rejoined the network as a driver in the Pakistani city of Peshawar. Intelligence agents picked up that he was driving a white Jeep.

• 5:08Speaker 01

  They followed the Jeep to the city of Abbottabad, which is a very sleepy city in Pakistan. And he drives into a compound with one large house, one small house, and set on about an acre of property. That compound was surrounded by 20-foot walls, and the inhabitants are burning trash. They're not connected to the internet. They don't have a phone system. Kind of a strange thing for somebody that clearly has some money. At that point, the people who are looking for bin Laden brief the director of the CIA, who's Leon Panetta.

• 5:40Speaker 07

  I'm Leon Panetta, former director of the CIA. I had established a task force at the CIA to basically focus on whatever evidence might tell us where bin Laden was located.

• 5:55Katy Milkman?

  The evidence that bin Laden might be in Abbottabad was thin. But there were several clues that something was quite suspicious about this particular compound.

• 6:05Speaker 07

  We identified a family that was living on the third floor that never came out, always stayed within the compound. And we soon found out that in order to make a phone call, the couriers would go 90 miles away from this compound. So they were exercising very high security. So that's what gave us the best evidence that perhaps bin Laden might be located there.

• 6:30Katy Milkman?

  Director Panetta briefed the president on this lead in August of 2010.

• 6:35Speaker 07

  When I reported what we had found to the president, he said, obviously, what we needed to do was to conduct surveillance on the compound to try to determine if bin Laden was actually there.

• 6:48Katy Milkman?

  But surveillance was difficult for a number of reasons, including the high walls around the compound. The only views they had were from satellites, and those images were largely inconclusive. But there was one tantalizing piece of visual evidence.

• 7:03Speaker 07

  There was one moment when we were able to see somebody who was a little older come out into the yard of that compound and walk in circles. This happened usually each day. We'd come out, walk in circles, and go back into the compound, like a prisoner in a prison yard.

• 7:23Speaker 01

  The pacer would pace around this garden. But he was wearing a cowboy hat, and he was pacing underneath trellises. There was no overhead imagery that showed that this was bin Laden.

• 7:34Speaker 07

  So at that point, I said to the CIA team, "That could be bin Laden. Can you get me a telescope? Or get me a camera? Or something close to see if we could get a facial ID on that individual?" And they said it was very difficult to do because of the walls on all sides. I remember saying at the time, "I've seen movies where the CIA could do this." And we laughed about that. But we were still unable to really get a facial ID.

• 8:02Speaker 01

  At one point, Leon Panetta said, "Well, let's measure his shadow, since we know he's 6'4". And the analysts from the National Geospatial Agency came back and said, "Well, he's between 5'4" and 6'8". Well, it's not a particularly useful piece of information.

• 8:18Katy Milkman?

  So there was no photo confirming their suspicions. And there were plausible alternative explanations for the behavior of the people in the compound.

• 8:26Speaker 01

  This was a drug trafficker. This was somebody in Al Qaeda who had retired. This was somebody who just wanted to keep a low profile. So there were other explanations. Nobody could say for sure that he was that. It was an entirely circumstantial case.

• 8:41Katy Milkman?

  Despite the lack of concrete evidence of bin Laden's location, the compound in Pakistan was the best lead they had. By this point, it had been nearly 10 years since 9/11. The pressure was on to take action. But the CIA was still chastened by intelligence failures in Iraq.

• 8:59Speaker 01

  The CIA had taken a huge hit off the weapons-of-mass-destruction fiasco in Iraq, which was built on a circumstantial case that Saddam had weapons of mass destruction. Well, it turned out to be 100% wrong.

• 9:13Katy Milkman?

  Intelligence and military leaders faced a high-stakes decision in the face of inconclusive evidence.

• 9:20Speaker 01

  At that point, there's no more information to provide. It becomes a political question about what to do with the information. There was, first of all, elaborate discussions of what was the evidence that Bin Laden was there.

• 9:31Speaker 07

  At one point, I asked the key people at the CIA. They come to my office and I had my chief of staff there as well as my deputy. And I just basically asked all of them, what do you think? You know what the evidence is? You know what we've been looking at? What is your best advice here as to whether or not we should conduct a mission?

• 9:53Speaker 01

  But then it became more of a, well, okay, we don't know if he's there, but we should maybe have a plan to do something about it. And there were a bunch of ideas on the table.

• 10:03Speaker 07

  I called Admiral Bill McRaven, who was head of special forces. By that time, we had done a model of the compound, showed him that model. I asked him to come up with several options. He did.

• 10:14Speaker 01

  B-52 bombing raid, launch an experimental drone, gather more intelligence, do a joint operation with the Pakistanis.

• 10:23Katy Milkman?

  All of these proposals were fraught. Bombs might cause collateral damage and destroy evidence of success. The experimental drone had never been used in this type of operation and might fail. A joint operation with Pakistan might risk tipping off the target.

• 10:39Speaker 01

  A bunch of things that could go wrong. There could be dangers, the SEALs did a ground operation, people could be killed, captured. Bin Laden might not be there. We completely blow up our relations with the Pakistanis.

• 10:50Katy Milkman?

  The discussion was becoming one around whether or not to proceed at all. The decision hinged on everyone's level of confidence that Bin Laden was actually at the compound.

• 11:01Speaker 07

  Interestingly enough, it varied. Some thought 60% that we should go ahead. Some were below 50% thought it was too risky an operation. And there were a couple of people who felt very strongly that Bin Laden was there. I remember one analyst saying, I think it's 90% that Bin Laden is there. So it varied a great deal, but they all gave their opinions. And based on their input, I ultimately had to make up my own mind as to whether or not I would recommend to the president whether we should proceed with the mission.

• 11:36Katy Milkman?

  In addition to considering the varied opinions, there was also a concerted effort to interrogate important intelligence to rule out alternative explanations.

• 11:46Speaker 07

  We did some red team stuff where we looked at other possibilities.

• 11:51Speaker 01

  After the Iraqi weapons of mass destruction fiasco that the CIA was deeply involved in in 2003, they set up these red teams to kick the tires of any circumstantial case or any big kind of intelligence question. They would always red team it and say, what are the alternative explanations for the same set of facts that we have? So that we're not just all drinking our own bathwater and all agreeing that we're right.

• 12:16Katy Milkman?

  With potential alternative explanations explored and the limited evidence thoroughly analyzed, it came time to make a decision.

• 12:23Speaker 07

  Ultimately, it came down to a meeting of the National Security Council to determine whether or not we would go ahead with that operation.

• 12:32Speaker 01

  The last meeting was on April 28, 2011.

• 12:34Speaker 07

  A number of key people at the National Security Council meeting, the Vice President, Secretary of State, Secretary of Defense, head of the CIA, Director of National Intelligence, Chairman of the Joint Chiefs of Staff, and a number of others.

• 12:49Speaker 01

  These meetings were conducted in a very high level of secrecy. It's a big, relatively big, corporate-style conference room. There are cogs on the walls, so Kabul, Afghanistan, Baghdad, Iraq, Washington, DC.

• 13:03Speaker 07

  And the president went around the room.

• 13:07Katy Milkman?

  Like the CIA analysts, members of the National Security Council had very different levels of confidence around what should be done.

• 13:15Speaker 07

  The Vice President, he thought we needed to gather more intelligence. He thought we needed to have a little more time to be able to verify if it was Bin Laden. The Secretary of State recommended that we should go. Bob Gates, who was Secretary of Defense, was concerned. And there were others that had just shared the same concern. They thought back to when the helicopters went down during the Carter administration, going after people who'd been locked up from the embassy there. Late yesterday, I canceled a carefully planned operation. Equipment failure in the rescue helicopters made it necessary to end the mission. Two of our American aircraft collided on the ground.

• 13:55Katy Milkman?

  Operation Eagle Claw was a failed attempt to rescue 52 people held captive at the US Embassy in Tehran on April 24, 1980. Eight US servicemen were killed. A helicopter and a transport aircraft were destroyed. It was the type of scenario that the National Security Council desperately wanted to avoid. Despite that history, the president was increasingly leaning towards a ground operation.

• 14:22Speaker 01

  There are some important reasons for that. One, you could prove to yourself that you'd actually captured or killed Bin Laden. If you do drop a big bomb, you can't. If you do the drone strike, you can't. Two, you might gather a lot of intelligence from the site.

• 14:35Speaker 07

  The Chairman of the Joint Chiefs, the Director of National Intelligence, myself, obviously all recommended that the president proceed with the mission. The National Security Advisor made the same recommendation.

• 14:49Speaker 01

  President Obama went around the table. He just wanted to hear what everybody had to say. It wasn't a vote. He was asking for advice.

• 14:56Speaker 07

  I said, Mr. President, when I was in Congress and I faced a tough decision, I would pretend that I was talking to an average citizen in my district and saying, if you knew what I knew, what would you do? And I said, if I told the average citizen in my district that we had the best evidence on the location of Bin Laden since Tora Bora, I think that citizen would say, we have to go. And that's what I'm recommending to you.

• 15:28Speaker 07

  So in the end, you know, to the president's credit, he let everybody say their piece. And then, as always, it comes down to the decision by the president. The president took it all in and spent that evening thinking what his decision would be.

• 15:43Speaker 01

  President Obama went to his private office in his residence at the White House and sat up pretty late and thought about the victims of 9/11, thought about what the United States knew about the situation. He came down the following morning at 8:30 AM, met with his national security team.

• 16:04Speaker 07

  I got a call from the National Security Advisor, who told me that the president had decided to go with the mission. It was a gutsy decision because there's no question it was risky. And there was no question that we did not have 100% confirmation that Bin Laden was there. If something happened, if the helicopters went down, if Bin Laden was not there, we would have to accept responsibility. And there would be repercussions as a result of that.

• 16:34Katy Milkman?

  A decision was made. A team of Navy SEALs, SEAL Team Six, would fly by Black Hawk helicopter under cover of darkness about 120 miles into Pakistan from Afghanistan to raid the compound in Abbottabad.

• 16:49Speaker 07

  The directions I gave to Bill McRaven, I said, I want you to go in, get Bin Laden and get out of there. And if you go in and you can't find Bin Laden, get out of it. That was the direction. So a lot of riffs, a lot of risks. But at the same time, probably the most important covert mission we had ever undertaken.

• 17:13Speaker 07

  I went to mass on Sunday morning and prayed a lot. When I got to CIA headquarters and we were running the operation out of the CIA, I really felt pretty confident that we were doing the right thing.

• 17:28Katy Milkman?

  On May 2 of 2011, SEAL Team Six took off from their base in Afghanistan at approximately 10:30 PM local time. Operation Neptune Spear had begun.

• 17:42Speaker 07

  When you're going through the operation and we were following the SEALs into Pakistan, you're very consumed by the moment and wanting to make sure that the mission is going right. But I have to tell you, when one of the helicopters got above the compound, because it was hot that day, the air came up and stalled the engine. And the warrant officer, who was in charge of that helicopter, had to settle it down with a tail up on one wall. And it's one of those moments where your stomach is in your mouth because, you know, some of the worst concerns about what could happen. And I remember asking Bill McRaven, what's going on? I think I said, what the hell's going on? And he didn't miss a beat. He said, helicopter's down, but everybody's okay.

• 18:26Speaker 07

  They're going to continue with the mission. They're going to breach through the walls and they're going to continue to go after Bin Laden. And I remember saying to him, God bless you, let's do it. And he did. Again, another very important decision that had to be made in order to complete the mission. We heard gunfire, which meant that it incurred some resistance. They then tried to go into the compound itself and go after Bin Laden. And it was about 20 minutes of silence, the longest 20 minutes of my life, where you really didn't know what was happening. And it was at the end of that 20 minutes that Bill McRaven came back on and said, I think we have Geronimo, which was the code word for finding Bin Laden.

• 19:13Speaker 07

  And it was at that moment where all of us in headquarters embraced one another, knowing that all of the information, all of the intelligence, all of the work on the mission had proven right. The trip back was another hour and a half, but they finally, finally landed. And that was the moment when we really were relieved that the mission had been successful.

• 19:37Speaker 04

  Tonight, I can report to the American people and to the world that the United States has conducted an operation that killed Osama bin Laden, the leader of Al Qaeda, and a terrorist whose response

• 19:50Speaker 07

  For the murder of thousands of innocent men, women, and children.

• 19:57Speaker 07

  And it was a real source of pride that we had accomplished something special and that the world was safer as a result of what we had done. I also thought about the victims of 9/11, those who had died and their families, and thought we were finally delivering justice to those families who lost loved ones.

• 20:17Speaker 07

  I often say that leadership requires that you have to take risks. If leaders were always facing decisions that were 100%, it would be an easy job. But there are many decisions that involve questions as to whether it's the right or the wrong step. And you've got to decide whether you think there's enough evidence to justify that kind of operation and weigh the risks that are involved. That's why we elect presidents, is to ultimately make that kind of decision. And President, to his credit, not only made a gutsy decision, he made the right decision.

• 21:02Katy Milkman?

  Leon Panetta is the former director of the CIA. He has also been the Secretary of Defense, the Chief of Staff to the President, and a member of Congress. More recently, he's the author of Worthy Fights: A Memoir of Leadership in War and Peace. Peter Bergen is a journalist and producer, who is CNN's National Security Analyst. He's the author of several books, including Manhunt: The Ten-Year Search for Bin Laden from 9/11 to Abbottabad. His most recent book is called The Rise and Fall of Osama Bin Laden. You can find links in the show notes and at schwab.com/podcast.

• 21:44Katy Milkman?

  We all grapple with a degree of uncertainty when we make predictions, like how long it will take to complete a project, or what will happen with the stock market this year, or what will the weather be like on Thursday? In the case of intelligence analysts' predictions about whether or not Bin Laden was present at the compound in Pakistan, you could argue that they just got lucky. And it's very easy to see how the raid could have turned out quite differently. But the officials involved in the decision to proceed employed a number of strategies and techniques that improved their chances of success. They worked in groups comprised of highly experienced, enormously qualified analysts. They used red teaming, essentially playing devil's advocate, to interrogate ideas and find errors in their thinking.

• 22:29Katy Milkman?

  Conflicting opinions were encouraged and considered. And the range of probabilities from different intelligence analysts were articulated and combined, increasing the likelihood of an accurate prediction. My next guest has done incredibly important research in the field of forecasting, and she co-designed an elaborate multi-year study involving tournaments, where participants were randomly assigned to face different forecasting conditions and then had their forecasting abilities put to the test. The study was done to determine what conditions lead to the best forecasts. Barbara Mellers is my colleague at the University of Pennsylvania, and she is the I. George Hayman University Professor of both Marketing at the Wharton School and of Psychology at

• 23:13Katy Milkman?

  the School of Arts and Sciences. Hi, Barb. Thank you so much for taking the time to talk to me today.

• 23:20Speaker 03

  Hi, Katie. It's great to be here.

• 23:22Katy Milkman?

  I am really excited to get into some of your research on psychological strategies for becoming a better forecaster. And I wanted to start by asking you to define a few strategies that your work has proven can add huge value, before talking about the research where you tested the impact of each of these techniques. So first, could I ask you to talk about probability training? What is that exactly?

• 23:46Speaker 03

  Probability training. We created an interactive module for people that lasted about 45 minutes, and it essentially gave them a lot of tips. We didn't teach them much in the way of mathematics, but we told them about how to look for professional forecasts on the internet. If they find multiple forecasts, average them. There are a handful of cognitive biases that are important when it comes to forecasting, and those include overconfidence, and base rate neglect, and probably the confirmation bias.

• 24:21Katy Milkman?

  We've done previous shows about two of those topics, so...

• 24:24Speaker 03

  Okay, all right.

• 24:25Katy Milkman?

  Some of our listeners will be familiar with them.

• 24:27Speaker 03

  Great. Forecasters who were assigned to the condition with probability training had to do that training module prior to making any forecasts.

• 24:37Katy Milkman?

  Okay, so could you also tell us a bit about teaming? What does that look like?

• 24:43Speaker 03

  Teaming was an intervention in which we placed people together in virtual teams that consisted of, say, 15 people or fewer, and they worked together on a website, but they weren't in the same room.

• 24:58Katy Milkman?

  Great. And then what is tracking? And what does it mean in the context of trying to achieve great forecasting results?

• 25:06Speaker 03

  Well, tracking is well-known in the educational context, where we talk about putting kids with similar abilities in the same classrooms. And that's all good if the kids have high abilities. It's controversial when they don't. We decided to put forecasters together, those who had the top 2% in terms of forecasting accuracy. We placed them into teams of 10 or 15 people and allowed them to work together, because we had already learned that teaming works. But tracking means being elitist about who works together and allowing a very enriched environment for those who do extremely well.

• 25:48Katy Milkman?

  That's really interesting. Okay, thank you for defining those different techniques. And I would love it if you could talk a bit now about your research evaluating the effects of these different techniques on people's forecasting ability. Could you describe how you put together this really incredible tournament and how you assess the benefits of probability training, teaming, and tracking?

• 26:10Speaker 03

  We were one of five research groups that were funded by IARPA, which is the Intelligence Advanced Research Project's activity. It's sort of the little sister to DARPA, the research funding area. And we had a variety of people who worked with us: computer scientists, economists, political scientists, psychologists, and you name it, statisticians. And when we read the literature about how to improve human forecasting, we had no idea what the best method would be. So our group decided to run experiments, which is the thing we do that comes naturally. And we recruited a lot of people and then randomly assigned them to conditions in which

• 26:53Speaker 03

  they either got a training module on probabilistic reasoning or did not get one. And we did that with teaming as well. We had people who worked independently. We had people who worked in groups. And then eventually, after the first year of the tournament, we decided to take this elitist approach and put the best forecasters together. And that was not randomly assigned. We took the top 2% from each of our conditions and put them together in teams and allowed them to work with each other.

• 27:28Katy Milkman?

  Could you talk a little bit about what the forecasting activities were that you had people doing over this sort of two-year experiment?

• 27:35Speaker 03

  Actually, it was four years. This paper just talks about the first two years. So we replicated things four times, believe it or not. Some of the questions, I was just looking back at them. One of them was about whether Berlusconi would be elected by a particular date. Would he be the leader of Italy? Would Putin be elected in Russia in 2012? Believe it or not, there was actually uncertainty about that at the time among a number of people. Would there be an international incident in the South China Sea?

• 28:07Katy Milkman?

  So there were basically all...

• 28:09Speaker 03

  Geopolitical.

• 28:09Katy Milkman?

  You know, geopolitical events.

• 28:11Speaker 03

  Yeah.

• 28:12Katy Milkman?

  Were people doing this weekly or monthly? And were they doing dozens or hundreds of forecasts?

• 28:17Speaker 03

  Well, each year, there were roughly 100-plus questions that we launched. And we launched a few of them each week. They were questions of interest to the intelligence community, economic questions, what will the price of Brent crude be? Will a particular UN decision go through? So they were all over the map. And we gave people feedback after each question was resolved. People jumped on the website, made forecasts about as many questions as they wanted to throughout the course of nine months. And these nine-month tournaments occurred during four years, so we could fine-tune what we'd learned the previous year and test it further and build on it, so forth.

• 29:03Katy Milkman?

  I love this experiment, and I really appreciate you describing it. It's so creative and important. I would love it if you could talk a little bit about the key things that you found worked and why you think they added so much value to forecasting quality.

• 29:19Speaker 03

  It is effective to give people simple training modules that don't require too much in the way of mathematical knowledge or statistical knowledge. They benefit from tips. It's also useful in this kind of situation to allow people to work in teams where they can share knowledge, point each other to different articles, motivate each other when someone's not talking, and correct each other's errors. And that was more beneficial than working alone. Now, the rationale for working alone is statistical.

• 30:01Speaker 03

  You can obtain independent forecasts and average them. And then the idea is that the errors will average out, and you'll get a better estimate of the true score or the best forecast. And allowing people to talk and work together proved to be far more beneficial than allowing them to have independent errors.

• 30:25Katy Milkman?

  It's really interesting, and it does contradict a lot of things we thought we knew about the importance of independent evaluation. And of course, you have to worry about things like group think, people becoming echo chambers.

• 30:37Speaker 03

  Hurting.

• 30:38Katy Milkman?

  Yeah.

• 30:38Speaker 03

  Yes, social loafing.

• 30:40Katy Milkman?

  But it's really interesting that here you find this benefit from teaming that's so clear. And what about tracking?

• 30:48Speaker 03

  Tracking was, most surprisingly, it was the best intervention by far. It was sort of like putting forecasters on steroids. Once they worked together, they were really interested in helping each other and providing articles and knowledge and tracking down esoteric kinds of information on the internet. And then they debated. And we monitored what their interactions, and they were much more likely to ask each other questions than regular teams. They were much more likely to answer questions that people had asked. And they were much more likely to

• 31:31Speaker 03

  express gratitude for the interaction and sharing and saying, thank you, and so forth. So they actually became really good friends. And there are actually, believe it or not, 10 years later, there are still groups of super forecasters getting together each month to talk about political affairs and so forth in San Francisco, and New York, and Washington DC, and Los Angeles, and so forth.

• 31:54Katy Milkman?

  That is so interesting. And so, so the teaming intervention is great, but when you team up these top performers, that's when absolutely incredible things started to happen.

• 32:05Speaker 03

  Yeah, right.

• 32:06Katy Milkman?

  How do you apply what you've learned about super forecasting in your life?

• 32:10Speaker 03

  Ah, yes. I am very aware of how poorly I make forecasts. I am bad when it comes to planning, how much time will something take? Planning fallacy, of course. I'm bad at predicting how much I'm going to like a movie, or a book, or whatever. And I have learned to change my mind. I've learned to say, have a three episode rule for a TV series. Look at it three times and then reject it. I think just very simple lessons have pervaded my life about the importance of base rates and taking the outside view and not being overwhelmed with all of the details of, say, the

• 32:54Speaker 03

  Ukrainian war, or Chinese interventions in Taiwan, or whatever it is.

• 33:00Katy Milkman?

  Are there important mistakes that you think people ever make when they're trying to apply the key findings from your research to make better forecasts in their own lives or in their organizations? What do people get wrong?

• 33:15Speaker 03

  Well, I think that there's two things that people don't do that they should be doing much more often than what occurs. And one is to actually make forecasts. We don't really put ourselves out on the line and make a numerical prediction very often. We say, oh, it's likely, or it's unlikely. And that vague verbiage lets us get out of lots of tricky situations that may not end the way we want them to. Second, we need to keep records so that we can start to learn how good we are at forecasting. And where we're bad, where we could learn something, where we're pretty solid.

• 33:56Katy Milkman?

  If you could leave our listeners with a key suggestion about what they might want to do differently in their own lives, especially now that they know about your findings on probability training and teaming and tracking, what suggestion would you offer them?

• 34:13Speaker 03

  I'd say the best way to become a better forecaster is to practice. And there's no better way to do that than to write down forecasts. But force yourself to predictions of events that are resolvable, that, you know, you don't need a lawyer to determine what happened. And then start seeing where your skills are, where you might need help. In businesses, there's a company spun out from this particular intelligence forecasting tournament. It's called the Good Judgment Incorporated, and it does consulting and helps financial institutions with forecasts and so forth. So there are places that help people

• 34:57Speaker 03

  become better forecasters as well. But just testing yourself, making forecasts, keeping track, getting records, and seeing how well you do.

• 35:07Katy Milkman?

  So you can learn from the good and the bad.

• 35:09Speaker 03

  And the ugly, yes, right.

• 35:11Katy Milkman?

  Yes. Barb, thank you so much for taking the time to do this. I really appreciate it, and I learned so much.

• 35:18Speaker 03

  Thank you, Katie. It was fun.

• 35:24Katy Milkman?

  Barbara Mellors is the I. George Hayman University Professor at the University of Pennsylvania, where I also work. Barb is jointly appointed as a professor of marketing at the Wharton School and as a professor of psychology at the School of Arts and Sciences. You can find links to her research on forecasting in the show notes and at schwab.com/podcast.

• 35:51Katy Milkman?

  Forecasting is an unusual activity because the uncertainty inherent in it means that even the world's best forecasters are often wrong. Barb's work with her partner and Penn faculty colleague Phil Tetlock has revolutionized our understanding of what it takes to generate what they call super forecasters. Simple probability training can add tremendous value. A key ingredient in that training is to encourage people to look at relevant reference points when making estimates, like looking at the price of other houses that have recently sold in your neighborhood if you're forecasting the sale price of your own home. Another key ingredient is to teach people to first search for and then average independent forecasts made by

• 36:34Katy Milkman?

  others, like the price estimates offered by several different realtors who have all expressed an interest in listing your house. These are simple steps we can all take to make better forecasts. When you're forecasting how long it will take to refinish your basement, your forecast shouldn't be made in a vacuum. First, collect relevant data points. How long did it take other people to do similar projects? Maybe your project is quite unique, but those reference points will still be useful. You can collect multiple independent forecasts from people with decent knowledge of the building business, too, and average them together. Do you need to do this kind of research to make every forecast in your life? Like whether it will rain tomorrow, or what TV show you'll like most in the year ahead?

• 37:18Katy Milkman?

  That probably wouldn't be a great use of time, but for big decisions, it's a great idea. You might also pick up a copy of Super Forecasting: The Art and Science of Prediction, a terrific book on this topic that was co-authored by Barb's partner and collaborator Phil Tetlock and the journalist Dan Gartner.

• 37:52Katy Milkman?

  You've been listening to Choiceology, an original podcast from Charles Schwab. This episode is part of our back catalog, where it joins over 100 other episodes designed to help you make better decisions. You can always browse to find a topic that might help you today. And if you're all caught up, we'll have a brand new episode in two weeks exploring the toxic trap of social comparison. For that, we go inside one of the most explosive seasons in Formula One history, where two teammates got so caught up in their own rivalry that they accidentally handed the world championship to another competitor. You won't want to miss it. You can also follow some of Schwab's other great podcasts, including

• 38:37Katy Milkman?

  Financial Decoder and Washington Wise. I'm Dr. Katie Milkman. Thanks for listening.

• 38:57Speaker 06

  For important disclosures, see the show notes or visit schwab.com/podcast.