(Source: http://www.funnyjunk.com/funny_pictures/4613599/Statistics+about+statistics/; Credit: http://www.funnyjunk.com/user/lordlolland)
The following is a guest post from NYU political scientist Nathaniel Beck.
*****
On Jan. 31, the New York Times (in print and on-line) had a fascinating Page One story about opinions of Americans on global warming, primarily as a function of party identification. The continuation page of the article had a long (12 paragraph — p. 12 used 12 column inches for this material; the online description is here) discussion of the methodology of the poll, and on first reading I was ecstatic to see such details. I do not know when newspaper polls started telling people that sampling error is hardly the only source of uncertainty, but I was pleased to see it. In particular, the last two paragraphs read:
In theory, in 19 cases out of 20, overall results based on such samples will differ by no more than four percentage points in either direction from what would have been obtained by seeking to interview all American adults. This value takes into consideration the fact that weighting procedures increase the sampling error slightly. For smaller subgroups, the margin of sampling error is larger. Shifts in results between polls over time also have a larger sampling error.
In addition to sampling error, the practical difficulties of conducting any survey of public opinion may introduce other sources of error into the poll. Variation in the wording, order and translation of questions, for example, may lead to somewhat different results.
While I would quibble with the “somewhat” in the last line, this is great progress. But, alas, ecstasy is often short-lived. So on re-reading, I was surprised to see no mention of the response rate for the poll. I know response rates are low, but I was curious. I wrote to Michael Kagay, who was listing as assisting the Times on the poll. Shockingly, he replied (on a Saturday) in under an hour. He sent more details as supplied to him by Jon Krosnick of Stanford University (who used tallies — needed for AAPOR formulas — originally supplied by Social Science Research Solutions of Media, Pennsylvania, who handled the fieldwork). Some of the details provided were astoundingly good; up to 14 follow-up attempts were made, and cell phone respondents were offered $10 compensation (for a half-hour interview). Alas, the response rate for the entire poll, with all this herculean effort, was about 12% (using the American Association for Public Opinion Research standard formula). While this is typical of polls, surely readers who care would worry a bit more about the results had they seen this response rate (especially given what we know about differential response rates by group and the issues related to weighting such groups).
So the Times is to be congratulated on providing so much detail, and Michael Kagay doubly so for providing the relevant information (I think he was shocked that anyone cared about the “Methods Box”) and Jon Krosnick’s report provided great detail. But why was not the most important statistic of the operation (after the sample size) not even mentioned? How many readers would write to Michael Kagay to get this critical information?
So the bottom line is that reporting on technical issues is getting better (at least in the better newspapers). No longer do we just get a sample size and a statement that the poll is accurate plus or minus some percentage points. But clearly newspapers (and any other organization reporting polls) must do better; response rates are critical information that needs to be shown.
What can people who read The Monkey Cage do? When you see a poll reported without key information, contact the appropriate editor of the publication. Would polling organizations like everyone to see how low response rates are? Of course not. Do we need to see such response rates? Of course.