https://twitter.com/SamWangPhD/status/529864519955537921
As election results continue to roll in, the big puzzle of the morning is not why the Republicans won so many Senate seats – as was just noted in the previous Monkey Cage post, The Monkey Cage’s Election Lab did a remarkably good job of predicting this outcome — but rather the question is — as the tweet from Princeton’s Sam Wang above illustrates (and see extended discussion on his blog here) — why, when polls were biased, they seem to have so consistently been biased in the Democrats’ favor (i.e., Republicans did better than polls predicted they would). After all, we are supposed to be living in a world where polls are more likely to miss Democrats’ supporters — especially young, cellphone only types — so why the big Republican surprise?
I’m sure there will be a great deal of money and effort spent trying to figure out the answer to this question, but I just want to suggest one possible avenue for exploration. We are living in an era where poll response rates are dropping precipitously, at least for traditional phone-based surveys. This point was dramatically illustrated in a recent Pew Report showing that response rates had fallen from 36 percent in 1997 to 9 percent in 2012:
Source: Pew Research (http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/)
This means that pollsters now have to contact more people in order to get the required number of respondents for their surveys. This in turn means that there is a greater possibility that the actual sample of people contacted is “biased” in the sense that it doesn’t accurately represent the population of interest to those doing the surveys.
Now of course, pollsters know this, which is why they “weight” survey respondents in an attempt to make the sample more representative of the population in question. This has always been an important part of polling, especially in terms of elections when pollsters have also had to try to estimate the likely voting population. Nevertheless, as response rates drop further, the importance of weighting surveys correctly can only increase.
So here’s what I am thinking about Wednesday morning: Is it possible that pollsters overestimate the previous election when constructing their weights for the current election? So yes, there are good reasons to think it is harder to reach young people today using telephone surveys. But of course pollsters know this, and so adjust the weights of their surveys accordingly. But with fewer young people in their surveys — combined with the possibility that the young people you can reach by phone are not representative of young people generally — the work that has to be done by these weights grows. Now, not wanting to get a mistaken estimate because of this bias, I wonder if the polling overcompensated in terms of weights in this regard because of the voting patterns observed in the 2012 presidential elections.
To be clear, I am not stating that I think pollsters applied presidential turnout models to an off-term election, but instead whether the subtle concerns about underestimating portions of the 2012 electorate led pollsters to overcompensate in setting their weights for the 2014 elections, which in turn had a larger effect due to declining response rates to surveys. Put another way, in a context of declining response rates, then the flux of simultaneously estimating a changing population, likely turnout models, and a changing set of people who respond to surveys could have further exacerbated the desire not to miss populations that may have been challenging to estimate in 2012.
There will undoubtedly be lots of other explanations offered up for the disparity between poll averages and the actual split of the votes in many of last night’s races. And again to be very clear, in terms of picking winners, poll averaging once again did remarkably well, especially in the Senate races. But if my hunch is correct, then it would suggest that heading into 2016, Republicans should be a little wary of inflated poll numbers as pollsters attempt to incorporate the lessons of 2014, especially if response rates to surveys continue to drop in the coming two years.