The Folly of Prediction (Rebroadcast)

Human beings love to predict the future, but we're quite terrible at it. So how about punishing all those bad predictions?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Stephen J. Dubner?

  Freakonomics Radio is sponsored by PNC Bank. Boring. Great word, bad rep. Yet, time and time again, here at Freakonomics, we find that when something culture-changing happens, boring things like calculations and careful planning are usually behind it. Which begs the question, why not apply it to other aspects of life? Like banking, for example? Well, PNC Bank did, since its inception, proving time and again that the best banking strategy isn't the exciting type, but the steady, reliable, definitely-not-yolo type. PNC Bank: brilliantly boring since 1865. Brilliantly boring since 1865 is a registered mark of the PNC Financial Services Group, Inc. PNC Bank, National Association, Member FDIC.

• 0:48Stephen J. Dubner?

  Freakonomics Radio is sponsored by Amica. You know what they say, if you want to go fast, go alone. If you want to go far, go together. So, go with Amica and get coverage from a mutual insurer that's built for their customers, one that looks after what's important to you together. Auto, home, life, and more, coverage that fits your unique needs. Amica helps you protect what matters most. Visit amica.com and get a quote today.

• 1:19Stephen J. Dubner?

  Freakonomics Radio is sponsored by Mint Mobile. With Mint Mobile's summer sale, all of Mint's plans are $15 per month, even unlimited. Free yourself from big wireless and switch to Mint for high-speed data and coverage on the T-Mobile 5G network. To get your new wireless plan for just $15 a month, go to mintmobile.com/freak. That's mintmobile.com/freak. Upfront payment, $45 for three months, $90 for six months, or $180 for 12 months. Plan required, $15 per month equivalent. Taxes and fees extra, initial plan term only. Greater than 50GB may slow when network is busy. Includes up to 20GB hotspot. Capable device required. Availability, speed, and coverage varies. See mintmobile.com.

• 2:09Stephen J. Dubner?

  Freakonomics Radio is sponsored by ChatGPT. ChatGPT Work is a new way to work in ChatGPT. ChatGPT Work can pull info from across your apps and connect to your tools to help you create finished work. You can even automate work in ChatGPT with the Scheduled Tasks feature, and the whole time, you stay in control. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting Work Mode, available on Plus and Pro plans.

• 2:46Speaker 01

  We are finding the witch, might we burn her?

• 2:50Stephen J. Dubner?

  What does it mean to be a witch exactly in Romania? Are these people that we know here as psychics or fortune tellers? Or are they different somehow?

• 2:58Speaker 02

  I don't know how is a fortune teller in the United States, but here, generally, they are women of different ages. They can, they say they can cure some diseases, they can bring back your husband or your wife, or they can predict your, your future.

• 3:20Stephen J. Dubner?

  Who is a typical client for a witch?

• 3:22Speaker 02

  There are quite a lot of politicians who are going to witches. You know that France President Nicolas Sarkozy, he went to witches last years ago. And our president in Romania and very important politicians from different parties, they are going to witches. Some of them, they were obliged to recognize they went to witches. Some of them, it's an off the record information, but me being a journalist, I know that information.

• 3:53Stephen J. Dubner?

  Vlad Mixic is a reporter in Bucharest, the capital of Romania. He knows a good bit about the witches there.

• 4:00Speaker 02

  Quite a lot of them, they are quite rich. They have very big houses with golden rooftops. And a lot of the Romanians, they are living in small apartments in blocks. So just going in such a building will give you a sense of majesty and respect.

• 4:26Stephen J. Dubner?

  But the Romanian witch industry has been under attack. First came a proposed law to regulate and tax the witches. It passed in one chamber of parliament before stalling out. But then came another proposal, arguing that witches should be penalized if the predictions they make don't turn out to be true.

• 4:45Speaker 03

  Tell me, what do you do with witches?

• 4:48Speaker 02

  So if you are one of my clients and if I'm a fortune teller, if I fail to predict your future, I would pay quite substantial fine to the state. Or if this happens many times, I will even go to jail. The punishment is between six months and three years in jail.

• 5:13Stephen J. Dubner?

  What's being proposed in Romania is revolutionary, it strikes me, because we typically don't hold anybody accountable for bad predictions. So I'm wondering, in Romania, let's say if a politician makes a bad prediction, do they get fined or penalized in any way?

• 5:34Speaker 02

  No, not at all. In fact, this is one of the hobbies of our president. He's doing a lot of predictions, which are not coming true, of course. And after that, he is re-elected. Or his popularity is rising like the sun in the morning, you know? No, anyone can do a lot of predictions here in Eastern Europe, and not a single hair will move from his or her head.

• 6:06Stephen J. Dubner?

  Come on, people, that doesn't seem fair, does it? I don't care if you're anti-witch or pro-witch, or witch-agnostic. Why should witches be the only people held accountable for bad predictions? What about politicians and money managers, sports pundits? And what about you?

• 6:35Speaker 04

  From WNYC, this is Freakonomics Radio. Today, the folly of prediction. Here's your host, Stephen Dubner.

• 6:50Stephen J. Dubner?

  All of us are constantly predicting the future, whether we think about it or not. Right now, some small part of your brain is trying to predict what this show is going to be about. Now, how do you do that? You factor in what you've heard so far, what you know about Freakonomics, maybe know a lot, maybe you've never heard of it, you might think it's some kind of communicable disease. When you predict the future, you look for cognitive cues, for data, for guidance. Here's where I go for guidance.

• 7:21Speaker 05

  I think to an economist, the best explanation for why there are so many predictions is that the incentives are set up in order to encourage predictions.

• 7:31Stephen J. Dubner?

  That's Steve Levitt. He's my Freakonomics friend and co-author, an economist at the University of Chicago.

• 7:37Speaker 05

  So, most predictions we remember are ones which were fabulously, wildly unexpected, and then came true. Now, the person who makes that prediction has a strong incentive to remind everyone that they made that crazy prediction which came true. You look at all the people, the economists who talked about the financial crisis ahead of time, those guys harp on it constantly. "I was right, I was right, I was right." But if you're wrong, there's no person on the other side of the transaction who draws any real benefit from embarrassing you by bringing up the bad prediction over and over. So, there's nobody who has a strong incentive, usually, to go back and say, "Here's the list of the 118 predictions that were false."

• 8:22Speaker 05

  And I remember, growing up, my mother, who's somewhat of a psychic, would predict a stock-

• 8:27Stephen J. Dubner?

  Wait, somewhat of a psychic?

• 8:28Speaker 05

  She's a self-proclaimed psychic. And she would predict a stock market crash every single year.

• 8:35Stephen J. Dubner?

  Right. And she's been right a couple times.

• 8:37Speaker 05

  And she has been. She's been right twice in the last 15 years. And she would talk a lot about the times she was right. I'd have to remind her about the 13 times that she was wrong. And without any sort of market mechanism or incentive for keeping prediction makers honest, there's lots of incentive to go out and to make these wild predictions. And those are the ones that are remembered and talked about. Think about one of the predictions that you hear echoed more often than almost anyone is Joe Namath's famous pronouncement about how the Jets were going to win the Super Bowl. And it was unexpected and it happened. Now, if the Jets had lost the Super Bowl, nobody would remember that Joe Namath made that pronouncement.

• 9:18Stephen J. Dubner?

  And conversely, you can probably find at least one player on every team that's lost the Super Bowl in the last 40 years who did predict that his team would win.

• 9:24Speaker 05

  That's probably right. Exactly right. Now, the flip side, which is perhaps surprising, is that in many cases, the goal of prediction is to be completely within the pack. And so I see this a lot with pension fund managers or endowment managers, which is if something goes wrong, then as long as everybody else made the same prediction, you can't be faulted very much.

• 9:49Stephen J. Dubner?

  Pension managers, football players, psychic moms, Romanian witches, who doesn't try to predict the future these days?

• 9:58Speaker 06

  Go ahead and make your predictions. Start with the masters.

• 10:01Speaker 07

  I'm going to lean towards Phil. I like his new putting stroke.

• 10:03Speaker 03

  The old farmer's almanac predicts a colder-than-usual winter.

• 10:09Speaker 06

  Bear Stearns is fine. Do not take your money. This is really, if there's one tech going, Bear Stearns is not in trouble.

• 10:17Speaker 08

  Randy and Freddie are fundamentally sound that they are not in danger of going under.

• 10:26Speaker 06

  Paul the octopus is making yet another World Cup prediction, betting on Spain.

• 10:31Speaker 07

  We will in fact find evidence of weapons programs. Gaddafi will ultimately step down.

• 10:41Stephen J. Dubner?

  And you know the worst thing? There's almost nobody keeping track of all those predictions. Nobody. Except for this guy.

• 10:50Speaker 09

  Well, I'm a research psychologist who-

• 10:52Stephen J. Dubner?

  But don't forget your name, though.

• 10:53Speaker 09

  I'm Phil Tetlock, and I'm a research psychologist. I've spent most of my career at the University of California, Berkeley, and I recently moved to the University of Pennsylvania, where I'm cross-appointed in the Wharton School in the Psychology Department.

• 11:04Stephen J. Dubner?

  Philip Tetlock has done a lot of research on cognition and decision-making and bias, pretty standard stuff for an Ivy League psych PhD. But what really fascinates him is prediction.

• 11:16Speaker 09

  There are a lot of psychologists who believe that there is a hard-wired human need to believe that we live in a fundamentally predictable and controllable universe. There's also a widespread belief among psychologists that people try hard to impose causal order on the world around them, even when those phenomena are random.

• 11:36Stephen J. Dubner?

  This "hard-wired human need," as Tetlock puts it, has created what he calls "a prediction industry." Now, don't sneer. You're part of it, too.

• 11:47Speaker 09

  I think there are many players in what you might call the prediction industry. In some sense, we're all players in it. Whenever we go to a cocktail party, or a colloquium, or whatever, where our opinions are being shared, we frequently make likelihood judgments about possible futures and the truth or falsity of particular claims about futures. The prediction business is a big business on Wall Street, and we have futures markets and so forth designed to regulate speculation in those areas. Obviously, governments have great interest in prediction. They create large intelligence agency bureaucracies and systems to help them achieve some degree of predictability in a seemingly chaotic world.

• 12:24Stephen J. Dubner?

  Let me read you something that you have said or written in the past, that "this determination to ferret out order from chaos has served our species well. We are all beneficiaries of our great collective successes in the pursuit of deterministic regularities in messy phenomena: agriculture, antibiotics, and countless other inventions." So, so talk to me for a moment about the value of prediction. Obviously, there's, there's much has been gained, much to be gained. Do we overvalue prediction, though, perhaps?

• 12:54Speaker 09

  I think there's an asymmetry of supply and demand. I think there is an enormous demand for accurate predictions in many spheres of life in which we don't have the requisite expertise to deliver. And when you have that kind of a gap between demand and real supply, you get the infusion of fake supply.

• 13:14Stephen J. Dubner?

  Fake supply. I like this guy, this Philip Tetlock. He's not an economist, but he knows the laws of supply and demand can't just be revoked. So if there's big demand for prediction in all realms of life and not enough real supply to satisfy it, what does this fake supply sound like? One could say it could.

• 13:36Speaker 03

  There could be terrorist threats.

• 13:38Stephen J. Dubner?

  The economy could remain the driving factor. Could, could.

• 13:41Speaker 06

  Could. Some of that radioactivity could carry in the atmosphere to the west coast of the United States.

• 13:46Speaker 03

  There could be an interesting battle on the cards when the Oscar nominations are announced.

• 13:49Speaker 06

  Gold could be peaking at about 1,600. It could certainly get to 2,000. Gold could go to $3,500 an ounce.

• 13:57Stephen J. Dubner?

  There's a punditocracy out there, a class of people who predict ad nauseam, often on television. They can be pretty good at making their predictions tough to audit.

• 14:07Speaker 09

  It's the art of appearing to go out on a limb without actually going out on a limb. So for example, the word could, something could happen. So the room you happen to be sitting in could be struck by a meteor in the next 23 seconds. That makes perfect sense, but the probability is of course point 0000, et cetera, 1. It's not 0, but it's extremely low. In fact, the word could, the possible meanings people attach to it range from about .01 to about .6, which covers more than half of the probability scale, right there.

• 14:38Stephen J. Dubner?

  Look, nobody likes a weasel. So more than 20 years ago, Tetlock set out to conduct one of the largest empirical studies ever of predictions. He chose to focus on predictions about political developments around the world. He enlisted some of the world's foremost experts, the kind of very smart people who have written definitive books, who show up on CNN or on the Times' op-ed page.

• 15:07Speaker 09

  In the end, we had close to 300 participants, and they were very sophisticated political observers. Virtually all of them had some postgraduate education, roughly two-thirds of them had PhDs. They were largely political scientists, but there were some economists and a variety of other professionals as well.

• 15:23Stephen J. Dubner?

  And they all participated in your study anonymously, correct?

• 15:27Speaker 09

  That was a very important condition for obtaining cooperation.

• 15:30Stephen J. Dubner?

  Now, if they were not anonymous, presumably, we would recognize some of their names. These are prominent people at political science departments and economic departments at, I'm guessing, some of the better universities around the world. Is that right?

• 15:41Speaker 09

  Well, I don't want to say too much more, but I think you would recognize some of them, yes. I think some of them had substantial Google accounts.

• 15:51Stephen J. Dubner?

  The study became the basis of a book Tetlock published a few years ago called Expert Political Judgment. There were two major rounds of data collection, the first beginning in 1988, the other in 1992. These nearly 300 experts were asked to make predictions about dozens of countries around the world. The questions were multiple choice. For instance, in democracy X, let's say it's England, should we expect that after the next election, the current majority party will retain, lose, or strengthen its status? Or for undemocratic country Y, Egypt, maybe, should we expect the basic character of the political regime to change in the next five years, in the next 10 years? And if so, in what direction? And to what effect?

• 16:35Stephen J. Dubner?

  The experts made predictions within their areas of expertise and outside, and they were asked to rate their confidence level for their predictions. So, after tracking the accuracy of about 80,000 predictions by some 300 experts over the course of 20 years, Philip Tetlock found-

• 16:54Speaker 09

  That experts thought they knew more than they knew, that there was a systematic gap between the subjective probabilities that experts were assigning to possible futures and the object of likelihoods of those futures materializing.

• 17:05Stephen J. Dubner?

  Let me translate that for you. The experts were pretty awful. Now, you may think, awful compared to what? Did they beat a monkey with a dartboard?

• 17:16Speaker 09

  Oh, the monkey with the dartboard comparison that comes back to haunt me all the time. But with respect to how they did relative to, say, a baseline group of Berkeley undergraduates making predictions, they did somewhat better than that. Did they do better than an extrapolation algorithm? No, they did not. They did, for the most part, a little bit worse than that. How did they do relative to purely random guessing strategy? Well, they did a little bit better than that, but not as much as you might hope.

• 17:45Stephen J. Dubner?

  That extrapolation algorithm that Tetlock mentioned? That's simply a computer programmed to predict no change in current situation. So it turned out that these smart, experienced, confident experts predicted the political future about as well, if not slightly worse, than the average daily reader of The New York Times.

• 18:08Speaker 09

  I think the most important takeaway would be that the experts think they know more than they do. They were systematically overconfident. Some experts were really massively overconfident. And we were able to identify those experts based on some of their characteristics of their belief system and their cognitive style, their thinking style.

• 18:28Stephen J. Dubner?

  Okay, so now we're getting into the nitty gritty of what makes people predict well or predict poorly. What are the characteristics, then, of a poor predictor?

• 18:39Speaker 09

  Dogmatism.

• 18:44Stephen J. Dubner?

  It can be summed up that easily.

• 18:46Speaker 09

  I think so. I think an unwillingness to change one's mind in a reasonably timely way in response to new evidence. A tendency, when asked to explain one's predictions, to generate only reasons that favor your preferred prediction and not to generate reasons opposed to it.

• 19:04Stephen J. Dubner?

  And I guess what's striking to me, and I'd love to hear what you have to say about this, is that it's easy to apply one word, prediction, to many, many, many different realms in life. But those realms all operate very differently. So politics is different from economics, and predicting a sports outcome is different than predicting an agricultural outcome. But it seems that we don't distinguish so much necessarily, and that there's this modern sense almost that anything can be and should be able to be predicted. Am I kind of right on that, or no?

• 19:34Speaker 09

  I think there's a great deal of truth to that. I think it is very useful in talking about the predictability of the modern world to distinguish those aspects of the world that show a great deal of linear regularity and those parts of the world that seem to be driven by complex systems that are decidedly nonlinear and decidedly difficult, if not impossible, to predict.

• 19:57Stephen J. Dubner?

  Talk to me about a few realms that generally are very, very hard to predict and a few realms that generally are much easier.

• 20:04Speaker 09

  Predicting Scandinavian politics is a lot easier than predicting Middle Eastern politics.

• 20:10Stephen J. Dubner?

  Yes, that was the first one that came to my mind, too. All right, but keep going.

• 20:16Speaker 09

  The thing about the radically unpredictable environments is that they often appear for very long periods of time to be predictable. So, for example, if you had been a political forecaster predicting regime longevity in the Middle East, you would have done extremely well predicting, you know, in Egypt, that Mubarak would continue to be president of Egypt year after year after year, in much the same way that if you'd been a Sovietologist, you'd have done very well in the Brezhnev era predicting continuity. There's an aphorism I quote in the Expert Political Judgment book from Karl Marx. I'm obviously not a Marxist, but I thought it's a beautiful aphorism that he had, which was that when the train of history hits a curve, the intellectuals fall off.

• 20:57Stephen J. Dubner?

  Coming up, who do you predict will hear from next? A bunch of people who are awesomely good at predicting the future? Yeah, right. Maybe later. First, we'll hear some more duds from Wall Street, the NFL, and the cornfield.

• 21:49Stephen J. Dubner?

  Freakonomics Radio, sponsored by Granola. You know that moment when a meeting ends and someone says, let's circle back on this next week, and everyone nods, but no one writes it down? Then next week rolls around and it's radio silence? That's why Granola exists. Granola is an AI-powered notepad built for the way real people actually meet. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup, no awkward bots. It's just your normal meeting with superpowers. If meetings are eating up your day, Granola is a no-brainer. You can try it totally free.

• 22:34Stephen J. Dubner?

  Just head to granola.ai/freakonomics. That's granola.ai/freakonomics to get your time back. Try it for free at granola.ai/freakonomics.

• 22:52Stephen J. Dubner?

  Freakonomics Radio is sponsored by Quince. You know those pieces in your closet that you find yourself wearing on repeat? The sweater that goes with everything? The perfect pair of pants? Or the tee you somehow always reach for first? Those are the pieces that make getting dressed easy, and that's what Quince does best. Quince focuses on high-quality wardrobe staples made with premium materials like 100% Mongolian cashmere, organic cotton, and merino wool. The styles are timeless, versatile, and designed to become the dependable pieces you reach for day after day. And it's not just clothing. Quince brings that same approach to everything they make, from hotel-quality bedding and bath to kitchen essentials and thoughtfully designed furniture. Find your next fall favorites at Quince.

• 23:35Stephen J. Dubner?

  Download the Quince app for app-exclusive offers or go to quince.com/freak. Get free shipping on your order and 365-day returns. Now available in Canada and the UK, too. That's Q-U-I-N-C-E.com/freak.

• 23:56Stephen J. Dubner?

  Freakonomics Radio is sponsored by hotels.com. Hotels.com helps you find and book the right stay for work faster. With business preferences, save your travel must haves like gym access or breakfast just once, then get tailored results whenever you search for work travel. Plus, with a free membership, you can save up to 20% on hotels and earn rewards on every stay for future travel. Book now at hotels.com. Visit hotels.com for details.

• 24:30Speaker 10

  Sometimes I can see the future.

• 24:32Speaker 06

  Our own future. The future of healthcare. Corn futures. A choice between two economic futures.

• 24:38Speaker 04

  From WNYC, this is Freakonomics Radio.

• 24:43Speaker 06

  Welcome to the future.

• 24:44Speaker 04

  Here's your host, Stephen Dubner.

• 24:51Stephen J. Dubner?

  So Philip Tetlock has sized up the people who predict the future, geopolitical change, for instance, and determined that they're not very good at predicting the future. He also tells us that their greatest flaw is dogmatism, sticking to their ideologies even when presented with evidence that they're wrong. You buy that? I buy it. Politics is full of ideology. Why shouldn't the people who study politics be at least a little bit ideological? So let's try a different set of people. People who make predictions that theoretically at least have nothing to do with ideology. Let's go to Wall Street.

• 25:29Speaker 06

  The experts on Wall Street are falling all over themselves to predict the year ahead. Rising inflation and rates will send the US in a downward spiral. Higher interest rates are going to hurt the housing market. They're going to make money over time. I would bet on it. Amazon. Cisco.

• 25:43Speaker 03

  Pfizer.

• 25:46Speaker 03

  Healthcare mutual funds.

• 25:47Speaker 06

  General Motors. Even good old IBM. Don't hit the sell button just yet.

• 25:53Speaker 11

  I'm Christina Fang, a professor of management at New York University's Business School.

• 25:59Stephen J. Dubner?

  Christina Fang, like Philip Tetlock, is fascinated with prediction.

• 26:04Speaker 11

  Well, I guess generally forecasting about anything, about technology, about product, whether it will be successful, about whether an idea, a venture idea, could take off. A lot of things, not just economic, but also business in general.

• 26:17Stephen J. Dubner?

  Fang wasn't interested in just your street-level predictions, though. She wanted to know about the big dogs, the people who make bold economic predictions that carry price tags in the many millions or even billions of dollars. Along with a fellow researcher, Yurker Denrel, Fang gathered data from the Wall Street Journal's Survey of Economic Forecasts. Every six months, the paper asked about 50 top economists to predict a set of macroeconomic numbers: unemployment, inflation, gross national product, things like that. Fang audited seven consecutive surveys with an eye toward a particular question. When someone correctly predicts an extreme event, a market crash maybe, or a sudden spike

• 26:59Stephen J. Dubner?

  in inflation, what does that say about his overall forecasting ability?

• 27:05Speaker 11

  In the Wall Street Journal survey, if you look at the extreme outcomes, either extremely bad outcomes and extremely good outcomes, you see that those people who correctly predicted either extremely good or extremely bad outcomes, they're likely to have overall lower level of accuracy. In other words, they're doing poorer in general.

• 27:25Stephen J. Dubner?

  Uh-oh. You catching this?

• 27:27Speaker 11

  Those people who happen to predict accurately the extreme events, they happen to also have a lower overall level of accuracy.

• 27:34Stephen J. Dubner?

  So I can be right on the big one?

• 27:36Speaker 11

  Yes.

• 27:37Stephen J. Dubner?

  If I'm right on the big one, I generally will tend to be more often wrong than the average person?

• 27:43Speaker 11

  On average, across...

• 27:45Stephen J. Dubner?

  On average.

• 27:45Speaker 11

  Across everyday predictions as well. And our research suggests that for someone who has successfully predicted those events, we are going to predict that they are not likely to repeat their success very often. In other words, their overall capability is likely to be not as impressive as their apparent success seems to be.

• 28:06Stephen J. Dubner?

  So the people who make big, bold, correct predictions are, in general, worse than average at predicting the economic future. Now, why is this a problem? Maybe they're just like home run hitters. You know, a lot of strikeouts, but a lot of power, too. All right, I'll tell you why it's a problem. Actually, I'll have Steve Levitt tell you.

• 28:28Speaker 05

  The incentives for prediction makers are to make either cataclysmic or utopian predictions, right? Because you don't get attention. If I say that what's going to happen tomorrow is exactly the same as what happened today.

• 28:38Stephen J. Dubner?

  You don't get on TV.

• 28:39Speaker 05

  I don't get on TV. If it happens to come true, who cares? I don't get any credit for it coming true either.

• 28:45Stephen J. Dubner?

  There's a strong incentive to make extreme predictions. Because, seriously, who tunes in to hear some guy say that next year will be pretty much like last year? And then, once you have been right on an extreme forecast, let's say you predicted the 2008 market crash and the Great Recession, even if you're predicting it every year like Steve Levitt's mother, you'll still be known as the guy who called the big one. And even if all your follow-up predictions are wrong, you still got the big one right. Like Joe Namath.

• 29:19Speaker 01

  The third annual Super Bowl game between the Baltimore Colts and the New York Jets.

• 29:24Speaker 06

  Baltimore has far too many weapons. Well, I like the Baltimore Colts. I've been so impressed by the Colts.

• 29:29Speaker 12

  Guys, well, wait a minute. We're going to win the game. I guarantee it.

• 29:34Speaker 01

  Well, the Jets have done it, beating the Baltimore Colts 16 to 7.

• 29:38Stephen J. Dubner?

  All right, look, predicting the economy, predicting the political future? Those are hard. Those are big, complex systems with lots of moving parts. So how about football?

• 29:51Stephen J. Dubner?

  If you're an NFL expert, how hard can it be to forecast, say, who the best football teams will be in a given year? We asked Freakonomics researcher Hayes Davenport to run the numbers for us.

• 30:03Speaker 13

  Well, I looked at the past three years of expert picking from the major NFL prediction outlets, which are USA Today, Sports Illustrated.com, and ESPN.com. We looked at 105 sets of picks total. They're picking the division winner for each year, as well as the wild card for that year. So they're basically picking the whole playoff picture for that year.

• 30:25Stephen J. Dubner?

  So talk about just kind of generally the degree of difficulty of making this kind of a pick.

• 30:29Speaker 13

  Well, if you're sort of an untrained animal making NFL picks, you're going to have about a 25% chance of picking each division correctly, because there are only four teams.

• 30:39Stephen J. Dubner?

  All right. So Hayes, you're saying that an untrained animal would be about 25% accurate if you just pick one out of four. But what about a trained animal, like a me, a casual fan? How do I do compared to the experts?

• 30:53Speaker 13

  Right. So if you're cutting off the worst team in each division, if you're not picking from among those, you'd be right about 33% of the time, one in three. And the experts are right about 36% of the time. So just a little better than that.

• 31:05Stephen J. Dubner?

  Okay. So if you're saying they'll pick at about 36% accuracy, and I, or someone by chance, would pick at about 33% accuracy. So that's a three percentage point improvement, or about 10% better. Maybe we should say, well, you know, that's not bad. If you beat the stock market by 10% every year, you'd be doing great. So should we think of these NFL pundits as picking 36% right being really wonderful, or?

• 31:31Speaker 13

  I wouldn't say that, because there's a specific fallacy that these guys are operating from, which is they tend to rely much too heavily on the previous year's standings in making their picks for the following year. They play it very conservatively, but there's a very high level of parity in the NFL right now. So that's not exactly how it works.

• 31:52Stephen J. Dubner?

  Tell me some of the pundits who, whether by luck or brilliance and hard work, turn out to be really, really good.

• 32:00Speaker 13

  Sure. There are two guys from ESPN who are sort of far ahead of the field. One is Pat Yacinkas, and the other is John Clayton, who is actually pretty well-known. He makes a lot of appearances on SportsCenter. He's kind of a nebbishy, professorial type. And they perform much better than everyone else because they're excellent wildcard pickers. They are the only people who have correctly predicted both wildcard teams in a conference and a season. But they're especially good because they actually play it much safer than everyone else.

• 32:31Stephen J. Dubner?

  Now, you say that they're very good. Persuade me that they're good and not lucky.

• 32:36Speaker 13

  I can't do that.

• 32:39Speaker 13

  There's a luck factor involved in all of these predictions. For example, if you pick the Patriots in 2008 and Tom Brady gets injured and they drop out of the playoffs, there's very little you can do to predict that. So injuries will mess with predictions all the time and other turnover rates in football that are unpredictable. So there's a luck factor to all of this.

• 33:06Stephen J. Dubner?

  So whether it's football experts calling Sunday's game, or economists forecasting the economy, or political pundits looking for the next revolution, we're talking about accuracy rates that barely beat a coin toss. But maybe all these guys deserve a break. Maybe it's just inherently hard to predict the future of other human beings. They're so malleable, so unpredictable. So how about a prediction where human beings are incidental to the main action?

• 33:36Speaker 14

  I'm Joe Prusaki, and I am Director of Statistics Division with USDA's National Agricultural Statistics Service, or NAS for short.

• 33:46Stephen J. Dubner?

  You grew up on a farm, yeah?

• 33:48Speaker 14

  Yep, I grew up in, I always call it, deep southern Illinois.

• 33:53Speaker 14

  I'm sitting here in Washington, DC, and where I grew up in Illinois is further south than where I'm sitting today. We raised, we had corn, soybeans, and raised hogs.

• 34:04Stephen J. Dubner?

  You've heard of Anna Wintour, right? The fabled editor of Vogue magazine? Joe Prisaki is kind of like Anna Wintour for farmers. He puts out publications that are read by everyone who's anyone in the industry. Titles like "Akkorage" and "Prospective Plantings" and "Crop Production." Prisaki's reports carry running forecasts of crop yields for cotton, soybeans, wheat, and corn.

• 34:31Speaker 14

  Most of the time, our monthly forecasts are probably within, I can guarantee you, within 5%. And most of the time, I can say within 2% to 3% of the final. And someone would say, "Well, that seems very good." But in the agricultural world, they, the day users, expect us to be much more precise in our forecast.

• 34:55Stephen J. Dubner?

  So how does this work? How does the USDA forecast something as vast as the agricultural output of American farmers?

• 35:02Speaker 14

  Like, at the beginning of March, we will conduct a large survey of farmers and ranchers across the United States. And the sample size this time, this year, was about 85,000.

• 35:14Stephen J. Dubner?

  The farmers are asked how many acres they plan to devote to each crop. Corn, let's say. Then, in late July, the USDA sends out a small army of enumerators into roughly 1,900 cornfields in 10 states. These guys mark off plots of corn, 20 feet long by two rows across.

• 35:32Speaker 14

  They're randomly placed. We have randomly selected fields in random location within the field. So you may get a sample that's maybe 20 paces into the field and 40 rows over. You may get one that's 250 paces in the field and 100 rows over.

• 35:48Stephen J. Dubner?

  The enumerators look at every plant in that plot.

• 35:51Speaker 14

  And then they'll count what they see or anticipate to be ears, based on looking at the plant.

• 36:00Stephen J. Dubner?

  A month later, they go back out again and check the corn stalks, check the ears.

• 36:04Speaker 14

  Well, you could have, you know, animal loss. You know, animal might cheer the plant off. You may, the plant may die. So all along, we're updating the number of plants. All along, we're updating the number of ears. The other thing we need, you need an estimate of ear weight or fruit weight.

• 36:25Stephen J. Dubner?

  So they go out again, cut off a bunch of ears and weigh them. But weight's still not done.

• 36:32Stephen J. Dubner?

  After the harvest, there's one more round of measurement.

• 36:35Speaker 14

  Once the field is harvested, the machine has gone through the field, the new man will go back out to the field, they'll lay out another plot just beyond our harvest area where we were, and they will go through and pick up off the ground any kernels that are left on the ground, pieces of ears of corn and such on the ground, so we get a measure of harvest loss.

• 36:59Stephen J. Dubner?

  So this sounds pretty straightforward, right? Compared to predicting something like the political or economic future, estimating corn yield based on constant physical measurements of corn plants is pretty simple, except for one thing. It's called the weather.

• 37:15Speaker 06

  Officials declare a drought watch for the entire state.

• 37:19Stephen J. Dubner?

  Weather remains so hard to predict in the long term that the USDA doesn't even use forecasts, uses historic averages instead. So Joe, talk to me about what happened last year with the USDA corn forecast. You must have known this was coming from me. So the Wall Street Journal's headline was USDA flubs in predicting corn crops. Explain what happened.

• 37:44Speaker 14

  Well, this is the weather factor that came into play. It turned off pretty hot and pretty dry. And I had asked a few folks that are out and about in Iowa, what happened? They said, this is just a really strange year. We just don't know. Now, if someone says, did we flub it? Well, I don't know. I mean, it was the forecast based on the information I had as of August 1. September 1, I had a different set of information. October 1, I had a different set of information. Could we have did a better job?

• 38:14Stephen J. Dubner?

  A lot of people thought they could have. Last June, the USDA lowered its estimate of corn stockpiles, and in October, it cut its estimate of corn yield. After the first report, the price of corn spiked 9%. The second report, another 6%. Joe Prisaki got quite a few emails.

• 38:35Speaker 14

  Okay, the first one is, this was, "Thanks a lot for collapsing the grain market today with your stupid..." And the word is three letters, begins with an A, and then it has two dollar signs.

• 38:45Stephen J. Dubner?

  Gotcha, I know that word.

• 38:45Speaker 14

  USD report.

• 38:48Speaker 14

  As bad as the stench of dead bodies in Haiti must be, it can't even compare to the foul stench of corruption emanating from our federal government in Washington DC.

• 38:58Stephen J. Dubner?

  It strikes me that there's room for trouble in that your forecasts are used by a lot of different people who engage in a lot of different markets and your research can move markets. I'm wondering what kind of bribes maybe come your way.

• 39:14Speaker 14

  I have, it's really interesting, I have people that, that call, I call, we call them fishers. You know, they call maybe a day or two days before and it's like, I tell them, I says, "Why do you do this? We've had this discussion before. This could do neither one of us good." Because I have to sign, there's a couple of things. One, I sign a confidentiality statement every year that says, "I shall not release any information before its due time or bad things happen." There's like a hundred, it's a hundred thousand dollar fine or time in prison. And it's like, you know, the dollar fine, okay. It's the prison part that bothers me.

• 39:57Stephen J. Dubner?

  But there's got to be a certain price at which, so, so let's say I offered you, I came to you and I said, "Joe, $10 million for, for a 24-hour head start on the corn forecast."

• 40:09Speaker 14

  I'm not gonna do it. Trust me, somebody would track me down.

• 40:13Stephen J. Dubner?

  I hear you.

• 40:14Speaker 14

  And I, and again, the prison time, it bothers me.

• 40:17Stephen J. Dubner?

  All right, so Joe Prusaki probably can't be bought and the USDA is generally considered to do a pretty good job with crop forecasts. But, look how hard the agency has to work, measuring cornfields row by row, going back to look for animal loss and harvest loss, and still, its projection, which is only looking a few months into the future, can get thrown totally out of whack by a little stretch of hot, dry weather.

• 40:50Stephen J. Dubner?

  That dry spell was essentially a random event. Kind of like Tom Brady's knee getting smashed. I hate to tell you this, but the future, it's full of random events. That's why it's so hard to predict. That's why it can be scary. Now, do we know this? Of course, we know it. Do we believe it? Hmm.

• 41:18Stephen J. Dubner?

  Some scholars say that our need for prediction is getting worse. Or more accurately, that we get more upset now when the future surprises us. After all, as the world becomes more rational and routineized, we often know what to expect. I can get a Big Mac not only in New York, but in Beijing, too. And they'll taste pretty much the same. So when you're used to that, and when things don't go as expected, watch out. Our species has been trying to foretell the future forever. Oracles and goat entrails and roosters pecking the dirt. The oldest religious texts are filled with prediction. I mean, look at the afterlife.

• 42:03Stephen J. Dubner?

  What is that if not a prediction of the future? A prediction that, as far as I can tell, can never be categorically refuted or confirmed. A prediction so compelling that it remains all these years later, a concept around which billions of people organize their lives. So what do you see when you gaze into the future? A yawning chasm of random events? Or do you look for a neat pattern, even if no such pattern exists?

• 42:34Speaker 15

  It's much more costly for someone to not detect a pattern.

• 42:40Stephen J. Dubner?

  That's Nassim Taleb, the author of Fooled by Randomness and the Black Swan.

• 42:45Speaker 15

  It's much costlier for us as a race to make the mistake of not seeing a leopard than have the illusion of pattern and imagining a leopard when there is none. And that error, in other words, mistaken the non-random for random, which is what I call the one-way, it's a bias. Now, that bias works extremely well because what's the big deal of getting out of trouble? Okay, it's not costing you anything. But in the modern world, it is not quite harmless. It makes this illusion of certainties makes you think that things that haven't exhibited risk, for example, the stock market, are riskless. We have the turkey problem.

• 43:28Speaker 15

  The butcher feeds the turkey for a certain number of days and then the turkey imagined that this is permanent.

• 43:38Stephen J. Dubner?

  The butcher feeds the turkey and the turkey imagines this is permanent. So you've got to ask yourself, who am I? The butcher? The turkey?

• 43:53Stephen J. Dubner?

  Coming up, hedgehogs and foxes and a prediction that does work. Here's a hint. If you like this song,

• 44:06Stephen J. Dubner?

  you'll probably like this one, too.

• 44:42Stephen J. Dubner?

  Freakonomics Radio is sponsored by Granola. You know that moment when a meeting ends and someone says, let's circle back on this next week and everyone nods, but no one writes it down? Then next week rolls around and it's radio silence? That's why Granola exists. Granola is an AI-powered notepad built for the way real people actually meet. You take rough notes like you normally would and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the Meeting ends. And the best part, Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup, no awkward bots. It's just your normal meeting with superpowers. If meetings are eating up your day, Granola is a no-brainer. You can try it totally

• 45:26Stephen J. Dubner?

  free. Just head to granola.ai/freakonomics. That's granola.ai/freakonomics to get your time back. Try it for free at granola.ai/freakonomics.

• 45:45Stephen J. Dubner?

  Freakonomics Radio is sponsored by Southern New Hampshire University. There's no hiding how convenient Southern New Hampshire University makes college. They offer over 80 career-focused graduate programs online. Classes have no set meeting times, so you can balance life and learning, and online tuition rates are some of the lowest in the country, making it all a little more manageable. A school this willing to work with you is worth the investment. Visit snhu.edu/freakonomics to learn more.

• 46:16Stephen J. Dubner?

  Freakonomics Radio is sponsored by PNC Bank. Boring. Great word, bad rep. Yet, time and time again, here at Freakonomics, we find that when something culture-changing happens, boring things like calculations and careful planning are usually behind it. Which begs the question, why not apply it to other aspects of life? Like banking, for example? Well, PNC Bank did, since its inception, proving time and again that the best banking strategy isn't the exciting type, but the steady, reliable, definitely-not-YOLO type. PNC Bank. Brilliantly boring since 1865. Brilliantly boring since 1865 is a registered mark of the PNC Financial Services Group, Inc. PNC Bank, National Association, Member, FDIC.

• 47:09Speaker 04

  From WNYC, this is Freakonomics Radio.

• 47:17Speaker 16

  The sunshine state will have plenty of sunshine for Al Gore.

• 47:21Speaker 17

  NBC News projects that he wins the 25 electoral votes in the state of Florida.

• 47:26Speaker 16

  You can pretty much take it to the bank.

• 47:28Stephen J. Dubner?

  Hey, guess what, sunshine? Al Gore didn't win Florida. Didn't become president either. Try walking that one back.

• 47:36Speaker 17

  We don't just have egg in our face, we've got omelet all over our suits.

• 47:41Stephen J. Dubner?

  So we are congenital predictors. But our predictions are often wrong. What then? How do you defend your bad predictions? I asked Philip Tetlock what all those political experts said when he showed them their results. He had already stashed their excuses in a neat taxonomy.

• 48:01Speaker 09

  So if you thought that Gorbachev, for example, was a fluke, you might argue, well, my understanding of the Soviet political system is fundamentally right, and the Soviet Politburo, but for some quirky statistical aberrations, the Soviet Politburo would have gone for a more conservative candidate. And another argument might be, well, I predicted that Canada would disintegrate, that Quebec would secede from Canada, and it didn't secede, but the secession almost succeeded because there was a 50.1 percentage vote against secession, and that's well within the margin of sampling error.

• 48:35Stephen J. Dubner?

  Are there others you want to name?

• 48:36Speaker 09

  Well, another popular prediction is off on timing. That comes up quite frequently in the financial world as well, and many, many very sophisticated students of finance have commented on how hard it is, saying the market can stay irrational longer than you can stay liquid, I think is a George Soros expression. So off on timing is a fairly popular belief system defense as well. And I predicted that Canada would be gone, and you know what? It's not gone yet. But just hold on.

• 49:05Stephen J. Dubner?

  You answered very economically when I asked you, what are the characteristics of a bad predictor? You used one word: dogmatism. What are the characteristics then of a good one?

• 49:13Speaker 09

  Capacity for constructive self-criticism.

• 49:16Stephen J. Dubner?

  How does that self-criticism come into play and actually change the course of the prediction?

• 49:20Speaker 09

  Well, one sign that you're capable of constructive self-criticism is that you're not dumbfounded by the question, what would it take to convince you you're wrong? If you can't answer that question, you could take that as a warning sign.

• 49:35Stephen J. Dubner?

  In his study, Tetlock found that one factor was more important than any other in someone's predictive ability: cognitive style. You know the story about the fox and the hedgehog?

• 49:48Speaker 09

  Isaiah Berlin tells us that the quotation comes from the Greek warrior poet Archilochus 2,500 years ago. And the rough translation was, "The fox knows many things, but the hedgehog knows one big thing."

• 50:02Stephen J. Dubner?

  So talk to me about what the foxes do as predictors and what the hedgehogs do as predictors.

• 50:06Speaker 09

  Sure. The foxes tend to have a rather eclectic, opportunistic approach to forecasting. They're very pragmatic. A famous aphorism by Deng Xiaoping was, "He didn't care if the cat was white or black, as long as it caught mice." And I think the attitude of many foxes, while they really didn't care whether ideas came from the left or the right, they tended to deploy them rather flexibly in deriving predictions. So they often borrowed ideas across schools of thought that hedgehogs viewed as more sacrosanct. There are many subspecies of hedgehog, but what they have in common is a tendency to approach forecasting as a deductive, top-down exercise. They start off with some abstract principles, and they apply those abstract principles

• 50:50Speaker 09

  to messy, real-world situations, and the fit is often decidedly imperfect.

• 50:55Stephen J. Dubner?

  So foxes tend to be less dogmatic than hedgehogs, which makes them better predictors. But if you had to guess, who do you think is more likely to show up on TV or in an op-ed column? The pragmatic, nuanced fox? Or the know-it-all hedgehog?

• 51:11Speaker 06

  You're going to be paying $7 or $8 a gallon for your oil very soon. Doomsday scenario. The policies are terrible. The beginning of the end. It's bullcrap.

• 51:20Stephen J. Dubner?

  You got it.

• 51:22Speaker 09

  Hedgehogs, I think, are more attractive to the media. Hedgehogs are more likely to offer quotable soundbites, whereas foxes are more likely to offer rather complex, caveat-laden soundbite. They're not soundbites anymore, after they're complex and caveat-laden.

• 51:39Stephen J. Dubner?

  So if you were to gain control of, let's say, a really big media outlet, New York Times or... NBC TV, and you said, you know, I want to dispense a different kind of news and analysis to the public. What would you do? How would you suggest building a mechanism to do a better job of keeping all this kind of poor expert prediction off the airwaves?

• 52:06Speaker 09

  I'm so glad you asked that question. I have some specific ideas about that, and I don't think they would be all that difficult to implement. I think they should try to keep score more. I think there's remarkably little effort in tracking accuracy. I mean, if you happen to be someone like Tom Friedman, or Paul Krugman, or someone who's at the top of the pundit pecking order, there's very little incentive for you to want to have your accuracy tested, because your followers are quite convinced that you're extremely accurate, and it's pretty much a game you can only lose.

• 52:37Stephen J. Dubner?

  Can you imagine, every time a pundit appeared on TV, the network would list his batting average right after his name and affiliation? You think that might cut down on blowhard predictions just a little bit?

• 52:50Stephen J. Dubner?

  Looking back at what we've learned so far, it makes me wonder. Maybe the first step toward predicting the future should be to acknowledge our limitations. Or, at the very least, let's start small.

• 53:06Stephen J. Dubner?

  For instance, if I could tell you what kind of music I like, and then you could predict for me some other music I'd want to hear that actually already exists, it's called Pandora Radio. Here's co-founder Tim Westergren.

• 53:21Speaker 18

  So what we've done is, we've broken down recordings into their basic components, for every dimension of melody, and harmony, and rhythm, and form, and instrumentation, down into kind of the musical equivalent of primary colors.

• 53:36Stephen J. Dubner?

  The Pandora database includes more than a million songs across every genre that you or I could name. Each song is broken down into as many as 480 musical attributes, almost like genetic code. Pandora's organizing system is in fact called the Music Genome Project. You tell the Pandora website a song you like, and it rummages through that massive genetic database to make an educated guess about what you want to hear next. If you like that song, you press the thumbs up button, and Pandora takes note.

• 54:09Speaker 18

  I wouldn't make the claim that Pandora can map your emotional persona. And I also don't think frankly that Pandora can predict a hit, because I think it is very hard. It's a bit of a magic. That's what makes music so fantastic. So I think that, you know, we know our limitations. But within those limitations, I think that we make it much, much more likely that you're going to find that song that just really touches you.

• 54:33Stephen J. Dubner?

  So, Tim, you were good enough to set up a station for me here. It's called Train in Vain Radio. So the song we gave you was Train in Vain by The Clash. So let me open up my radio station here, and I'll hit play and see what you got for me.

• 54:55Stephen J. Dubner?

  Oh, yeah.

• 54:58Stephen J. Dubner?

  Yeah, I like them. That's the jam, so I'm going to give it a thumbs up. All right, so I like Town Called Malice. I think there are a couple more songs in my station here.

• 55:10Stephen J. Dubner?

  Television, Tom Verlaine. He was always too cool for me. I can see why you would think that I would like him. And I appreciate your effort, Mr. Pandora. How about you? Are you-- were you a television fan?

• 55:21Speaker 18

  Yeah, yeah. And you know, one thing, of course, is that these songs are all rooted in guitar riffs.

• 55:26Stephen J. Dubner?

  Yeah.

• 55:27Speaker 18

  Those are repetitive motifs played on the guitar. And they're similar sounding. They got a little twang. And they're played kind of rambly, sort of a little bit rough, which is that sort of punk element in there.

• 55:42Stephen J. Dubner?

  I gotta tell you, even though when this song came up, and I've heard the song a few times, and I told you I didn't like television very much of this song, I'm kind of digging it now.

• 55:52Speaker 18

  See, there you go. That's exactly what we're trying to do.

• 55:57Stephen J. Dubner?

  It's a really great thing to do, but it's not really predicting the future the way most people think of it as predicting the future, is it?

• 56:04Speaker 18

  Well, I certainly wouldn't put our mission in the same category as predicting the economy or geopolitical futures. But, you know, the average American listens to 17 hours of music a week. So they spend a lot of time doing it. And I think that if we can make that a more enjoyable experience and more personalized, I think maybe we'll make some kind of meaningful contribution to culture.

• 56:32Stephen J. Dubner?

  So Pandora does a pretty good job of predicting the music you might want to hear based on what you already know you like. But again, look how much effort that takes. 480 musical attributes. And it's not really predicting the future, is it? All Pandora does is breaks down the confirmed musical preferences of one person today and comes up with some more music that'll fulfill that same person's preferences tomorrow. If we really want to know the future, we probably need to get much more ambitious. We probably need a whole new model. Like, how about prediction markets?

• 57:11Speaker 10

  A prediction market mechanically is basically like a betting market or a speculative market like orange juice futures or stock markets, things like that. The mechanics is that there's an asset of some sort that pays off if something's true, like whether a person wins the presidency or a team wins a sporting contest. And people trade that asset and the price of that asset becomes in a forecast of whether that claim is likely to be true.

• 57:39Stephen J. Dubner?

  That's Robin Hanson. He's an economics professor at George Mason University and an admitted advocate of prediction markets. As Hanson sees it, a prediction market is far more reliable than other forecasting methods because it addresses the pesky incentive problems of the old time prediction industry.

• 57:57Speaker 10

  So a prediction market gives people an incentive, a clear, personal incentive to be right and not wrong.

• 58:04Speaker 10

  Equally important, gives people an incentive to shut up when they don't know, which is often a problem with many of our other institutions. So if you, as a reporter, call up almost any academic and ask them various vaguely related questions, they'll typically try to answer them just because they want to be heard. But in a prediction market, most people don't speak up. Every one of your listeners today had the right to go speak up on orange juice futures yesterday. Every one of you could have gone and said, "Orange juice futures forecasts are too low or too high," and almost no one did. Why? Because most of you don't think you know, and that's just the way we want it. So in most of these prediction markets, what we want is the few people who know the best to speak up and everybody else to shut up.

• 58:48Stephen J. Dubner?

  Prediction markets are flourishing. Some of them are private. A multinational firm might set up an internal market to try to forecast when a big project will be done. And there are for-profit prediction markets, like Intrade, based in Dublin, where you can place a bet on, say, whether any country that currently uses the euro will drop the euro by the end of the year. As I speak, that bet has a 15% chance on Intrade. Here's another Intrade bet: whether there will be a successful WMD terrorist attack anywhere in the world by the end of 2013. That's got a 28% chance. Now, that's starting to sound a little edgy, no? Betting on terrorism?

• 59:30Stephen J. Dubner?

  Robin Hanson himself has a little experience in this area, on a US government project he worked on.

• 59:37Speaker 10

  All right, so back in 2000, DARPA, the Defense Advanced Research Projects Agency, had heard about prediction markets, and they decided to fund a research project. And they basically said, "Listen, we've heard this is useful for other things. We'd like you to show us that this can be useful for the kind of topics we are interested in." Our project was going to be forecasting geopolitical trends in the Middle East. We were going to show that prediction markets could tell you about economic growth, about riots, about perhaps wars, about whether changes of heads of state, and how these things would interact with each other.

• 1:00:15Stephen J. Dubner?

  In 2003, just as the project was about to go live, the press heard about it.

• 1:00:21Speaker 10

  On Monday morning, two senators had a press conference where they declared that DARPA and the military were going to have a betting market on terrorism. And so there was a sudden burst of media coverage. And by the very next morning, the head of the military basically declared before the Senate that this project was dead. And there was nothing more to worry about.

• 1:00:45Stephen J. Dubner?

  What do you think we collectively, you in particular, would know now about that part of the world, let's say, if this market had been allowed to take root?

• 1:00:56Speaker 10

  Well, I think we would have gotten much earlier warning about the revolutions we just had. And if we would have had participants from the Middle East forecasting those markets, not only we would get advance warning about which things might happen, but then how our actions could affect those. So, for example, the United States just came in on the side of the Libya rebels to support the Libya rebels against the Gaddafi regime. What's the chances that will actually help the situation as opposed to make it worse?

• 1:01:21Stephen J. Dubner?

  But give me an example of what you consider among the hardest problems that a prediction market could potentially help solve.

• 1:01:28Speaker 10

  Who should, not only who should we elect for president, but whether we should go to war here, or whether we should begin this initiative, or should we approve this reform bill for medicine, etc.

• 1:01:38Stephen J. Dubner?

  So that sounds very logical, very appealing. How realistic is it?

• 1:01:45Speaker 10

  Well, it depends on there being a set of customers who want this product. So, if prediction markets have an Achilles heel, it's certainly the possibility that people don't really want accurate forecasts.

• 1:02:02Stephen J. Dubner?

  Prediction markets put a price on accountability. If you're wrong, you pay. Simple as that.

• 1:02:09Stephen J. Dubner?

  Just like the proposed law against the witches in Romania. Maybe that's what we need more of. Here's Steve Levitt again.

• 1:02:16Speaker 05

  When there are big rewards to people who make predictions and get them right, and there's zero punishment for people who make bad predictions because they're immediately forgotten, then economists who predict, that's a recipe for getting people to make predictions all the time.

• 1:02:31Stephen J. Dubner?

  Because the incentives are all encouraging you to make predictions. If you get it right, there's an upside, and if you get it wrong, there's almost no downside.

• 1:02:38Speaker 05

  Right. If the flip side were that if I make a false prediction, I'm immediately sent to prison for one-year term, there'd be almost no prediction.

• 1:02:49Stephen J. Dubner?

  And all those football pundits and political pundits and financial pundits wouldn't be able to wriggle out of their bad calls, saying, "My idea was right, but my timing was wrong." I mean, that's how everybody does it. That big storm the weatherman called but never showed up? "Oh, it happened all right," he says, "but two states over." Or how about all those predictions for the end of the world, the apocalypse, the rapture, all that? "Well," they say, "we prayed so hard that God decided to spare us."

• 1:03:22Stephen J. Dubner?

  You remember back in May when an 89-year-old preacher named Harold Camping declared that the earth would be destroyed at 5:59 PM on a Saturday, and only the true believers would survive? I remember it very well because my 10-year-old son was petrified. I tried telling him that camping was a kook, that anybody can pretty much say anything they want about the future. Didn't help. He couldn't get to sleep at night. And then, the 21st came and went, and he was psyched. "I knew it all along, Dad," he said. And then I asked him what he thought should happen to Harold Camping, the false doomsday prophet.

• 1:04:06Stephen J. Dubner?

  "Oh, that's easy," he said. "Off with his head!" My son, he's not a bloodthirsty type. But he's not a turkey either.

• 1:04:27Speaker 04

  Freakonomics Radio is produced by WNYC, APM, American Public Media, and Dubner Productions. Our producers include Elizabeth Giddens, Colin Campbell, Susie Lechtenberg, Chris Neary, and Diana Nguyen. We had help from Ellen Horn and Peter Clowney. This episode was mixed by John Delore. If you want more Freakonomics Radio, you can subscribe to our podcast on iTunes or go to Freakonomics.com, where you'll find lots of radio, a blog, the books, and more.

• 1:05:04Stephen J. Dubner?

  Freakonomics Radio is sponsored by Ozempic. Innovation happens through rethinking what's possible. And when it comes to GLP-1s, Ozempic pill does just that. Learn more about Ozempic semaglutide tablets, 4 in 9mg, by calling 1-833-OZEMPIC or visit ozempic.com to view the medication guide and ask your doctor what's possible with FDA-approved Ozempic pill.

• 1:05:34Stephen J. Dubner?

  Freakonomics Radio is sponsored by Southern Company. The world and its energy needs are always changing. Southern Company's commitment to meeting this demand stays the same. That's because Southern Company believes energy is more than a utility, it's what powers possibilities. So they are looking ahead and investing $80 billion in infrastructure upgrades and are committed to fueling growth in ways that benefit all customers so that reliable and affordable energy is accessible for generations to come. Go to southerncompany.com to learn more. Southern Company, building the future of energy.

• 1:06:13Speaker 19

  This podcast is supported by Nurtac ODT. Remedy Pant. We know you didn't ask for an interruption, but migraine doesn't wait for the right moment to interrupt either. It just barges in, takes your time, and throws everything off. So when migraine takes your time, take Nurtac. It's for the acute treatment of migraine with or without aura in adults. Nurtac can provide pain relief in two hours, which can last up to two days. Ask your healthcare provider if Nurtac is right for you, because your time is for you, not migraine. Don't take if allergic to Nurtac ODT. Allergic reactions can occur even days after use. Get help right away for trouble breathing, rash, swelling of face, mouth, tongue, or throat. High blood pressure and Raynaud's syndrome can occur. Get help for high blood pressure, numbness,

• 1:06:57Speaker 19

  coolness, pain, or color changes in fingers and toes. Common side effect is nausea. For full prescribing information, call 1-833-4-NURTC or visit nurtac.com. All right, back to your podcast, playlist, or whatever's keeping you going today.

• 1:07:13Speaker 20

  This episode is brought to you by Charles Schwab. Is there a right time to sell a stock? Are you taking the right risks with your portfolio? Financial decisions can be tricky, and often your own cognitive and emotional biases can lead you astray. Financial Decoder, an original podcast from Charles Schwab, can help. Join host Mark Reapy as he offers practical solutions to help overcome the cognitive and emotional biases that may affect your investing decisions. Listen at schwab.com/financialdecoder.