This election season has not been kind to the oracles. Nate Silver’s team projected that Sanders would lose to Clinton in Michigan by a 22 percent margin. Nate Cohn had him losing by 12 percent. The Nates were not the only ones who were outrageously off the mark. PredictWise placed the odds of a Sanders victory at 8 percent; FiveThirtyEight at less than 1 percent. And Michigan was not the first time political forecasters had to eat their words in this cycle. Experts and betting markets wrongly expected Trump to fade away for months and months. The Party Decides framework blinded the most sophisticated analysts to the insurgency underfoot in both parties; especially in the GOP.
We shouldn’t be surprised when predictions turn out to be off the mark now and then. Getting it wrong is an occupational hazard in the forecasting business. The real problem lies not in the point estimates themselves but in the confidence attached to them. To put it bluntly, political forecasters are wildly overconfident in their predictions.
Figure 1: FiveThirtyEight’s Confidence Intervals for the Michigan Democratic Primary
Figure 1 shows FiveThiryEight’s estimates of the vote share for each candidate right before the Michigan Primary.[†] Clearly, if their distribution was anywhere close to being accurate, Sanders had virtually no chance of winning. Of course, that can’t possibly be right. What’s going on? Why might they—and indeed everyone else—be systematically underestimating their margins of error?? The answer can be found in two words: Model uncertainty.
Think of the big assumption that went into the Michigan estimates. The Mitchell polls, regarded as the most reliable of the lot, were based on an automated landline survey. Slightly more than 80 percent of respondents were 50 or older. According to Mitchell,
Because likely Primary voters are older, 54% are 60 or older and 86% are older than 50, we believe there are sufficient land line voters to get an accurate sample. We do not have to make any assumptions of likely voter turnout.
CNN’s exit poll revealed that only an estimated 47 percent of actual voters were 50 or older. So much for not having to “make any assumptions of likely voter turnout.”
My point is not that Mitchell’s assumption—that the age distribution of the respondents was more or less the same as the age distribution of primary voters—was wrong. It may well have been right. My point is that it was not certain; and crucially, this uncertainty was not reflected in the reported margin of error. Indeed, the reported plus/minus 4.5 percent margin of error merely reflected sampling error conditional on the assumption that the sample was representative.[‡]
More generally, estimated margins of error are model-dependent. Failure of model assumptions can lead to dramatically higher forecast errors than expected. Because model uncertainty is not captured by standard errors, they are no more than lower bounds for the actual uncertainty of predicted vote shares. And because both the predicted vote shares and their standard errors are required to calculate the odds of who will win any given election, such probabilities are almost always biased in favor of the front-runner.
Figure 2 illustrates this issue. The red and blue bell curves have the same means (expected vote shares) but the dispersion of the second (blue) is twice as large. The probability of victory of the trailing candidate (denoted by shaded regions) is considerably greater with greater uncertainty in vote shares.
Figure 2: Vote shares and probability of victory
As of writing, the latest poll in Illinois has Sanders leading by 2 percent, but the FiveThirtyEight model places the odds of a Clinton victory at 95 percent. And although the weighted polling averages show Sanders closing in on Clinton in Ohio, the probability assigned to a Clinton win is 98 percent. There is no way that these odds are anywhere close to being accurate. Until forecasters start paying real attention to model uncertainly, there are bound to more Michigans.
[†] I’m not trying to pick on FiveThirtyEight. The reason why I am using their estimates to illustrate is because they are admirably transparent about their model.
[‡] It does not matter whether or not one uses their reported margin of error in the final model. As long as one uses the poll numbers, one remains exposed to the model uncertainty. And because most polls share the same underlying assumptions (say about voter turnout), uncertainty arising from this source is not diversified away by averaging polls.