Predicting elections is getting harder. What did the midterms teach us?

Famously, polls got the 2016 election wrong. And they’d been off for multiple elections since. So, Monday night, political scientist Jack Rametta nervously refreshed his Senate forecasting model https://jackrametta.com/senate-forecast/ for the final time. Would it be right this time? Or would it yet again cast doubt on whether modeling, statistics and forecasting can still help us understand what motivates people to vote? Now that the results are in, so is Jack – to tell us how he did, and what his and other researchers’ predictions tell us about the evolving science of polling and forecasting. Interested in more science behind the news? Email us your question at [email protected] mailto:[email protected]. Support public media with NPR+ and enjoy perks for over 25 podcasts like this one. This show’s perks include sponsor-free listening. Learn more at plus.npr.org http://plus.npr.org. See pcm.adswizz.com https://pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences. NPR Privacy Policy https://www.npr.org/about-npr/179878450/privacy-policy

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This message comes from Open Door. With cash now, more later. Get cash upfront for your home, plus all the profit later. That's no chaos now, no cash left behind later. Skip the showings now and move on your own timeline. Cash now, more later. Now available nationwide. Start your offer at opendoor.com/NPR. Profits calculated after fees and costs. Eligibility and offer price may vary.

• 0:26Regina Barber?

  You're listening to Short Wave

• 0:29Regina Barber?

  from NPR.

• 0:31Regina Barber?

  At the beginning of this week, there were a lot of guesses about how the midterm elections would go.

• 0:36Speaker 02

  It's 11:18 PM Eastern Time, the night before the midterm elections.

• 0:44Regina Barber?

  'Twas this past Monday, the night before, when all through Jack Remetta's eighth-floor apartment, not a creature was stirring, not even Stella, his German shepherd.

• 0:54Speaker 02

  In front of me, I have the dashboard for my Senate forecasting model.

• 0:59Regina Barber?

  Jack, a political scientist at George Mason University, was uploading his election model on his computer.

• 1:05Speaker 02

  It's a little stressful.

• 1:07Regina Barber?

  He was about to run their final forecast as he sat on the couch.

• 1:11Speaker 02

  And I'm sitting here, I'm going to press the button now. Very exciting, so here we go.

• 1:18Regina Barber?

  He and his colleague at the University of California Davis hoped these predictions would match the real results.

• 1:24Speaker 02

  My co-author and I went through the sort of painstaking process of putting this all together. And this is the very final update, the update that we will be stuck with and committed to for all time.

• 1:35Regina Barber?

  No bets or money at stake, no political sides.

• 1:39Speaker 02

  It's updating. It's grabbed some unemployment data from the Fred database. It's grabbed the generic congressional ballot from the Silver Bulletin from Nate Silver. It's gotten the latest Senate polls from the New York Times.

• 1:53Regina Barber?

  Just their credibility on the line and the relevance of this profession.

• 2:00Speaker 02

  Oh, this is good. It's noting in the little message you can see on screen that it's excluded any polls from these fraudulent pollsters that were discovered a few weeks ago.

• 2:11Regina Barber?

  More minutes pass.

• 2:14Speaker 02

  OK, it's pushing it live. Sort of nervously refresh the page here a little bit. There we go.

• 2:22Regina Barber?

  And suddenly, maps and graphs light up the screen.

• 2:29Regina Barber?

  Today on the show, now that the election results are in, how did Jack do? And what did his and other researchers' predictions tell us about the evolving science of polling and forecasting? I'm Regina Barber, and you're listening to Short Wave, the science podcast from NPR.

• 2:50Speaker 00

  Stay tuned for the following announcements and messages from our sponsors.

• 2:54Speaker 03

  This message comes from Whole Foods Market. Host Thanksgiving with industry-leading quality standards from Whole Foods Market. There are no antibiotics in their meat department, ever. So carve up turkey and spiral-cut ham with peace of mind. Shop organic sides like stuffing mix and cranberry sauce from 365 Brand. And all their delectable pies are made with no high-fructose corn syrup. Gourmet Thanksgiving flavors with the quality you want, only at Whole Foods Market. This message comes from Schwab. At Schwab, you can get everything from self-directed investing to full-service wealth management, all in one place. No matter your investing goal, life stage, amount to invest, or know-how, you can invest your way with Schwab. This message comes from Subaru.

• 3:39Speaker 03

  Discover the 2027 Subaru Crosstrek Hybrid. With up to 630 miles per tank, this Subaru Crosstrek Hybrid delivers up to 50% better city MPG along with increased horsepower, with standard symmetrical all-wheel drive and 8.7 inches of ground clearance. Equipped with a standard 11.6-inch multimedia navigation system touchscreen, it keeps your trip on track. Their most fuel-efficient, lowest-emission Subaru Crosstrek. Adventure on the edge.

• 4:08Regina Barber?

  All right, Jack, let's get right into it. You and your colleague Christopher Hare at the University of California Davis made an election forecast. So this is a prediction of how likely various candidates are to win. And it focused on the Senate, like whether it would be in favor of the Democrats or the Republicans. And we had you record a voice memo of your forecast.

• 4:29Speaker 02

  Our forecast, at this point, has an expected number of Democratic seats of 52 at the median. The probability that Democrats will retake the Senate, according to our model, is 68%.

• 4:41Regina Barber?

  I mean, I was watching last night, you were watching last night. You know, this is 9 o'clock the next morning. I'm going to let you say it. Like, how did you do? How were your predictions?

• 4:52Speaker 02

  We won't declare total victory, but I think we did pretty well. We called most of the competitive races correctly. So for instance, we correctly called Texas for Tellerico. We correctly called Ohio for Sherrod Brown, Maine for Troy Jackson, and so on. So we did pretty well. I was happy that the results across these different states landed, almost without exception, within the sort of range of outcomes that we said were possible in the model. And that made me feel good. Of course, I wasn't happy that we missed Mississippi. We didn't think that the Democrats would quite take Iowa. We did actually think they would take Kansas. So that didn't make me happy.

• 5:34Speaker 02

  So on one level, I'm happy that we got so many of the calls right. But I do think that, in the end, our model and a lot of other models did miss the margins. So Democrats really did overperform.

• 5:45Regina Barber?

  Wow. Such scientists were like, you know, we were right, kind of.

• 5:49Speaker 02

  That's right.

• 5:49Regina Barber?

  I still got to go back, and I still got to tinker with it.

• 5:53Speaker 02

  There will be more tinkering, yes.

• 5:54Regina Barber?

  So what are the challenges, then, of doing this kind of election forecasting, of making that range?

• 6:00Speaker 02

  There's a lot of challenges. So the one that we focus on most, and there's other things that you could talk about, is the modelers, people like me, academics, or non-academics like Nate Silver, G. Elliot Morris, these sort of other forecasters. You know, when you pick the model, you have to make assumptions. So you have to say, you know, historically, we thought there would be an incumbency advantage in the Senate of a couple of points, right?

• 6:26Regina Barber?

  Right.

• 6:26Speaker 02

  But that changes over time. What will it be this year?

• 6:30Regina Barber?

  Like, who is the incumbent associated with?

• 6:32Speaker 02

  That's right. You know, how much does it matter if it's an open contest versus an incumbent defending the seat? The big thing that we were unsure about is the national environment is quite hostile towards Republicans. But does that actually pass through to state races? Or is a race in Alaska, you know, more about what's going on in Alaska? Or is it more about what's happening nationally?

• 6:54Regina Barber?

  So how do you get all that information? Like, is polling involved in these forecasts? How do you get the information that goes into these models?

• 7:02Speaker 02

  To oversimplify, there's two different ways to do election forecasts. You can pay attention to primarily the fundamentals of a race. So what's the partisan lean in a given state? How, you know, how did they do in the last election? Those sorts of basic things.

• 7:16Regina Barber?

  You're like, from data from before.

• 7:17Speaker 02

  From data from before, things we know.

• 7:20Regina Barber?

  Yeah.

• 7:20Speaker 02

  Or you can look at the polls, which, you know, the advantage of the polls are it's information that's more recent. So it's something that we might have known from yesterday, right, the day before the election. The problem with polls, of course, is they can be wrong or biased in all sorts of different ways. The most important way the polls have been wrong recently is this thing that we call non-response bias, right? Where one particular group out there in the public, who will ultimately vote, we sort of systematically miss them when we try to do our polling in a way that makes the polls look wrong. So what we do in our model is we blend these two things. So we say, OK, we want to pay attention to the fundamentals, to things that we know. But we also want to incorporate polling information.

• 8:03Speaker 02

  And, you know, the extent to which you decide to blend those things and you lean more towards the polls or more towards the fundamentals, more of an art than a science, I would say, right? This is sort of a subjective choice the analyst has to make.

• 8:16Regina Barber?

  Like you said, polls have been really off before. Like famously in 2016, the presidential election, the poll said Hillary Clinton had an advantage over Donald Trump. But obviously, Trump did win. And it became pretty obvious early in the night. And that is different from this year, the midterm elections. A lot of the polls were way more accurate. So given the two together, do you think polling does work? Did we fix it?

• 8:45Speaker 02

  I think polling worked this cycle would be my claim. That doesn't necessarily mean that polling will work going into the future. But for this cycle, it seems as though the outfits that are out there trying very hard to make adjustments for these problems that we've run into in the past, many of them seem to have done quite well.

• 9:06Regina Barber?

  So what are they doing well at?

• 9:08Speaker 02

  So for instance, a big problem is response rates. Right? So like back in the 1970s and 80s, you might have expected, you know, you do these nice landline polls and you expect a very high response rate, like double-digit response rate, 20, 30, 40, 50%.

• 9:24Regina Barber?

  So somebody just calls up and asks you questions and then you answer them?

• 9:28Speaker 02

  Right. Because at a very basic level, for polling to work, the notion is we're going to grab a random sample, right, out in the public. And you actually don't need a particularly large sample to get a sense for what's going on if it's truly random. When the response rates fall and there's different estimates, but now it could be the response rates less than 1% in these different surveys.

• 9:50Regina Barber?

  Oh, wow.

• 9:50Speaker 02

  Right? So it's really low. And the sorts of people who respond to surveys, especially online, like opt-in surveys, these are people who are really different from like the sort of average American at the grocery store. Right? So different outfits are taking different approaches to try and reach more representative samples. Increasingly, polls themselves are starting to look like models.

• 10:14Regina Barber?

  Let's get back to your expertise of forecasting, of predicting. I was looking at some other forecasts, like this one from Gary King at Harvard, who forecasts 78% likelihood that enough Democrats would be voted to control the Senate. A little bit more than yours, yours was a little bit more conservative.

• 10:32Speaker 02

  Yes.

• 10:32Regina Barber?

  Why can they vary so much?

• 10:33Speaker 02

  So I had a chance to look at their forecast this morning. I think it's a good model. Right? And this goes to show you reasonable people can write down different models that get you to different answers. And like, this should be, I think, a key takeaway, is there isn't like one correct forecasting model. And this is like a complaint a lot of people have about this whole industry. Right? You can't come to like a deterministic answer about what the right forecasting model is.

• 10:59Regina Barber?

  Okay, so there's this huge influence that's been gaining momentum ahead of this election, and that's prediction markets. So like Kalshi, where people are putting money on who they think is going to win any given race. And I recently saw a survey from the nonprofit Brennan Center for Justice that found that prediction markets could shift voters' views on which candidates are electable. What do you make of all that?

• 11:23Speaker 02

  So there's two things to say. One is, in terms of forecasting outcomes, so in terms of being a useful way to aggregate information, prediction markets are pretty good. And there's a lot of evidence from economics, there are some big papers showing these sorts of things. So I think we should be trepidatious in our skepticism towards prediction markets now.

• 11:44Regina Barber?

  Oh, wow.

• 11:45Speaker 02

  I will say, now despite the fact, I think prediction markets do work very well at aggregating information. So they're not betting just out of thin air. They're doing their own kind of modeling, right? In order to bet on these contracts in the market.

• 11:58Regina Barber?

  They're collecting data.

• 11:59Speaker 02

  That's right. That's right. But the other thing is, I think there is a real concern that we should have for the prediction markets, which is, in this case, we have some instances in the primaries where it does seem as though candidates were responding to prediction markets in their choices to enter or exit races. And that's something that worries me insofar as, especially in the early parts of the cycle, you know, we had fraudulent pollsters operating and contributing to these averages, which then the prediction markets were sort of queuing off of.

• 12:31Regina Barber?

  Oh, wow. Okay.

• 12:32Speaker 02

  So I think this isn't necessarily a criticism of just prediction markets, but we should be wary about how models and markets are influencing electoral behavior.

• 12:43Regina Barber?

  It strikes me that like polling and forecasting, it seems like this pursuit to understand people, what drives them. So I'm curious, what's your big takeaway when it comes to like really understanding the views of people across our country and what drives their election decisions?

• 13:00Speaker 02

  So this year, and I don't want to overstate it, but my takeaway from this year is, things are looking very nationalized. So like our model has all of these sort of interesting nuanced considerations in there, right? So do certain candidates overperform their polls? You know, are certain states idiosyncratic? And especially in the Senate, there's like a history of thinking that Senate races will be sort of special and not necessarily subject to national tides. This year, it just looks very national. And in terms of what that means for American politics, right? There's a little bit less room to maneuver, I think, for candidates who want to separate themselves from the national environment.

• 13:42Speaker 02

  And that has a lot of implications, whether or not that travels into the future. I think it probably will, at least for the immediate future, but it's hard to say.

• 13:52Regina Barber?

  Jack, congratulations on having your prediction be mostly right.

• 13:57Speaker 02

  Thanks.

• 13:58Regina Barber?

  And thank you so much for coming on our show.

• 14:00Speaker 02

  Happy to be here.

• 14:02Regina Barber?

  This episode was produced by Alana Gordon. It was edited by showrunner Rebecca Ramirez and Tyler Jones checked the facts. Jimmy Keely was the audio engineer. I'm Regina Barber. Thank you for listening to Short Wave, the science podcast from NPR.

• 14:25Speaker 03

  Stay tuned for the following announcements and messages from our sponsors. This message comes from Midi Health. Most women never get real answers about their hormonal health. Midi is changing that with virtual care covered by most major insurances. Understand your hormones. Visit joinmidi.com to book your first visit today. This message comes from EasyCater, making it easy for organizations to order food for meetings and events from favorite restaurants, set up meal programs for their employees, and manage food spend all in one place at easycater.com.