Engineers and developers work at President Obama’s campaign headquarters in Chicago in December 2011. (Joel Kowsky/Bloomberg News)
After my cranky tweets earlier this week criticizing the news media for focusing too much on early campaign polls, I saw this sharp rejoinder:
Ouch! To make things worse, David Shor is a senior data scientist at Civis Analytics — a leading purveyor of campaign data and models — so he knows a few things about polling. I decided to reach out. Below is a condensed and edited version of our correspondence.
John Sides: David, I saw your tweet. I’m happy to hear a contrary point of view.
David Shor: I think that tweet came off as unintentionally mean. It’s fun to throw the occasional friendly punch on Twitter, but I couldn’t be happier that political scientists are taking on a more prominent role in how the media talks and thinks about polling. (Not to mention that I love your blog and your work generally.)
Still I’m going to try to make a heterodox case for why we should care about early polling despite the empirical fact that they’re not directly predictive of election outcomes.
(JS: Let the record show that at this point that David cited some academic research and walked through a hypothetical statistical model of campaign polls involving a “simple non-mean-reverting random walk.” Reader, it was cool. But I will spare you.)
Here’s the upshot of this model: 480 days out from an election means you’d expect public opinion to move on average by 4.5 percent and as as much as 9 percent by election day. That 9 percent is indeed a huge margin of error, especially when you consider the standard deviation across presidential election results over the past 30 years has been about 3 percent. So this model is consistent with the finding that polling this far out is not predictive of election results.
John Sides: Hey, that was my point too. We agree after all!
David Shor: Not so fast. Even if early polls are mostly not predictive of election results, they are weakly predictive. If Democrats were currently polling at 35 percent, that would suggest that they’re unlikely to win in November—even with a margin of error of 9 percent. Knowing that the Democratic Party is not in the same position as François Hollande is useful information.
John Sides: I agree that current polls probably do provide some meaningful bounds — in essence, telling us that support for the Democrat or Republican candidate is unlikely to be very low or very high.
Alas, I think — and we agree here — that a lot of political reporting and commentary wants to do much more with the early polls than this. The general orientation is to use current polls to get some purchase on a candidate’s standing relative to 50 percent — that is, whether they’re likely to win or lose in November 2016. As you point out, the margin of error is likely too wide to say much.
David Shor: True, but although early polls are not a great indicator of public opinion on election day, they are a good indicator of public opinion right now. This is useful for two reasons.
First, there are elections happening around the country this year across government, and the overall national mood is the most important indicator for how those will turn out.
John Sides: I agree with that. Using current polls to assess the national mood is valuable.
David Shor: And interest groups and campaigns look at the direction of public opinion in order to estimate the effectiveness of messaging and tactics.
I would argue that Obama’s approval ratings not falling after he endorsed gay marriage in the spring of 2012 played a role in the GOP not attacking the President on gay marriage, even though it was an issue that did not yet have majority support.
Similarly, Republicans called off the government shutdown in fall of 2013 due to a large decline in public polling and adopted a more conciliatory approach toward the President moving forward as a result.
Even if the tactical fights right now will wash away over the next year, the observed dynamics right now are relevant to what campaigns will do right before the election.
John Sides: That all makes sense too. But to be clear, I was talking about how early polls of candidate match-ups are used—not so much polls about issues.
David Shor: I think the candidate match-ups get used too. For better or for worse, current polling numbers are guiding decisions by candidates about whether or not to enter a race, particularly for senate and gubernatorial races.
John Sides: I think that’s true. The question is whether it’s mostly “better” or mostly “worse.” There are certainly cases where early polls didn’t give a good sense of a candidate’s ultimate strength or weakness. Mitch McConnell looked vulnerable early on, but won very handily.
David Shor: Back to the presidential primary now. Probably the most important and relevant reason is that horse race polls right now give us a window into the relative performance of candidates that are currently competing against each other in the primary.
While current polling probably somewhat exaggerates the impact of individual candidates, it suggests that candidates like Ted Cruz and Donald Trump are much less electable than Jeb Bush or Scott Walker. They also suggest that Jeb Bush and Scott Walker are roughly equally electable, which is important given that Scott Walker is arguably more conservative.
So while there’s more empirical work to be done here, I think that the relative performance of candidates in early polling provides information on their merits as candidates that can’t be easily captured with other measures that political scientists often look at.
John Sides: That’s an interesting question: Are polls still valuable because they accurately separate the candidates into rough tiers? I think that could be right, but I’d like to see some systematic attempt to nail this down. Obviously, there have been many times when lower-tier candidates have surged in the polls, like Trump is now, so early polls may not even be able to get the tiers entirely correct.
David Shor: To be clear, I completely agree that a lot of election reporting regarding polls is awful, so I don’t think we disagree too much. But I think that with the proper perspective and background, current polling provides useful information. It’s just important to know how to read them.
John Sides: I can’t argue with that!
