Are the polls broken? A steady stream of articles over the past nine months has examined the challenges of the polling industry, pointing to highly visible failures, the rise of cell phones and the decline of response rates. And indeed, in Michigan and Indiana the polls were wrong for the Democratic primary, although not for the Republican primary.
These articles raise some good points. Reporting does tend to seize on the most surprising poll which is, in fact, more likely an outlier than harbinger of a dramatic new trend. And there are technical and economic issues affecting the polling industry that are cause for serious concern.
Yet for all the claims of broken polls, little direct evidence has been provided. Yes, response rates are down but has accuracy suffered as a result? Yes, polls differ from one another, but are “wild fluctuations” more common?
I’ve examined all the presidential general election polling since 2000 — which is 2,056 polls in all. These data tell a very different story, however. The polls aren’t “broken” after all.
The graph below plots the Democratic candidate’s share minus the Republican candidate’s share. Each gray point is a poll. The blue line is a trend — estimates via what is called “local regression”— that tracks the polling average over the election.
The most important point is that the gray dots, the individual polls, are scattered around the trend line. This scatter is the measure of how much variation we get from the polls beyond what is due to the changes in trend over time. On any given day, some polls are above the trend and some below. This means one can always find “disagreement” among the polls. And when the trend is close to zero — that is, close to a tied race — then we find some polls with one party ahead and some with the other ahead.
But how variable are the polls around the trend, and has this variation become larger, as the claim that polling is broken or wildly variable would suggest? To answer that question, subtract the trend estimate on the day the poll was completed from the poll result. This is called the “residual” and is a measure of the deviation of each poll from the trend.
If polls have become increasingly unreliable over the past 16 years, we should see more variation around the trend, and the residuals should become more spread out in recent elections compared to earlier ones. If 2016 is especially bad for polls, this year should be clearly more variable. The next graph plots the residuals against time for each election.
There is no evidence that the variation in residuals has changed substantially since 2000. The variation tends to decrease closer to elections but the overall spread of the residuals has been relatively consistent across the five elections, including 2016 so far.
Are current polls producing more outliers — that is, results further from the trend than we would expect due to normal sampling variation? If the residuals were perfectly normally distributed —that is, they precisely followed a “bell curve”— we would expect 5 percent of the polls to be more than 2 standard deviations away from the trend.
Over the five elections the actual percentage of polls this far away ranges from 4.7 to 5.9 percent, with an average of 5.3 percent. In 2016 so far, what percent of the polls are outliers, by this standard: 5.3 percent. There is no evidence that 2016 is producing more outliers than previous campaign polls.
Moreover, the variation in the polls isn’t increasing over time. The standard deviation of the residuals — which captures this variation — hasn’t become larger year over year. And 2016 is not out of line with previous years. If polling were seriously broken, or exhibiting wild fluctuations, we would expect to see more variability this year than previously. We do not.
Another way to look at the variation is to plot the distribution of deviations from the trend for each year. If some years are markedly different from others we should see that in the figure. Specifically we should see a wider, or “fatter,” distribution for 2016 compared to previous years if polls have developed more “wild fluctuations.”
There is not much difference among the five election years. The standard deviation varies a bit, but there is no evidence that “one of these is not like the others.” There is no statistically significant difference in the variance of 2016 polling compared to that of previous years.
If we take all five years of polling together, a few rules of thumb emerge for how often we see polls far from the overall trend. Rounding a bit and making the range symmetric for simplicity, a pretty good summary of the polls is this:
- Half of the polls are within +/- 2 points of the trend.
- 75% of the polls are within +/- 3.4 points of the trend.
- 90% of the polls are within +/- 5.2 points of the trend
- 95% of the polls are within +/- 6.4 points of the trend
So if you see a poll that is 5 points or more away from the trend, it may well turn out to be an outlier. But variation within 3 points of the trend is pretty common. So when the trend is, say, +2 for a candidate, expect to see most polls fall between -1 and +5. The -1 isn’t an outlier and neither is the +5. Both are consistent with a +2 race. But a -4 or a +8 should arouse some doubt.
The thing about trends is they can change. What looks like an outlier today could be the first poll picking up a shift in support. But until it is joined by other polls, together moving the trend estimate, be skeptical. Telling an outlier from a harbinger is nearly impossible based on a single poll. Some patience is required while more data accumulate, certainly a challenge in a world of instant news and analysis. But when two, then three then four polls are all below the previous trend, the evidence becomes more convincing.
To be clear, there are real challenges facing the polling industry. But there has been more rhetoric than evidence for the thesis that “polling is broken.” So far, the national polls in 2016 look a great deal like those from the past four presidential elections.
Charles Franklin is Director of the Marquette Law School Poll and a Professor of Law and Public Policy at Marquette Law School.