Eric Aspengren pointed me to this thoughtful discussion by Harry Enten of the question, “Are Bad Pollsters Copying Good Pollsters?” After several pretty graphs and one ugly table, Enten concludes, “None of this proves guilt, but it does raise the possibility some pollsters may be peeking at their neighbors’ papers.”
The basic idea is simple enough: If you’re a low-quality pollster, you can just do whatever crap you want and then copy your results from some more reputable poll that was done in the previous few days. Or you could average the results from a few recent polls. And you can fiddle with the numbers too, while you’re at it. This won’t work if someone wants your raw data, but as long as you’re just releasing toplines, you can pretty much do whatever you want, as long as you find a news organization that will publish your numbers on trust. (Here’s an example from a few years ago of a Wall Street Journal reporter who ran with unsubstantiated claims regarding the political opinions of the super-rich.)
But let’s consider a thought experiment. Suppose I start a polling company — let’s call it 539.com — and I make my methods public. I state very clearly that I’ll interview N people and then I’ll use Bayesian inference to combine the information from my survey and a forecast of the election based on economic conditions and recent polls. The only thing I don’t reveal is N, but we all have trade secrets, right? It turns out that N=0.
Here’s my point. Am I “cheating”? No. I’m doing the right thing. Indeed, if good polls are already out there, it would be a waste of people’s time for me to survey them just to get one more damn poll. If there is a problem, it’s not with “539.com” which might well be providing a useful service, but with any news organization that’s treating this poll as an independent data point.
News organizations are supposed to check their sources, right? If someone calls you and says they saw XYZ, you don’t just run the story, you check it, right? Same goes with polls. If the polling organization won’t show you their raw data, don’t treat them like they’re actually reporting a survey.
