It is: trumpet the significance of small changes in responses to vague questions in a single poll. To wit:
bq. Americans’ views of the proper government role in promoting traditional values had moved in a more liberal direction since 2005, to the point that last year, as many said the government should not promote traditional values as said it should. If that trend had continued, 2009 would have marked the first time Gallup found more Americans preferring that the government refrain from actively promoting traditional values. Instead, Americans’ attitudes reverted to a more conservative point of view on the matter. Now, Americans favor the government’s promoting traditional values by an 11-point margin, similar to the double-digit margins favoring that view through much of the prior two decades.
This “finding,” combined with their previous “findings” on abortion attitudes and on attention to the news, illustrate the tendency.
This focus — on the results of a single poll that is different from some previous poll or polls — ignores the possibility that this one poll could prove an outlier. This is what happened with the abortion result, even though Gallup actually conducted two polls to ensure that their “pro-life majority” finding was robust. The next Gallup poll showed that the this majority proved ephemeral.
Gallup often relies on survey questions that are hopelessly and unhelpfully vague. With the abortion finding, it was “…would you consider yourself to be pro-life or pro-choice?” I discussed that previously.
With the news finding, it was “how closely do you follow the national news (very, somewhat, etc.)?” Can people accurately report on their attention to news? No. Even if they could, does 36% vs. 30% in 2007 signal any meaningful difference? It strikes me as weak tea. (See also Nate Silver.)
And then there is the question on traditional values. What are “traditional values”? How do people interpret that term? I don’t know. Gallup doesn’t say. Democrats and particularly independents are the ones shifting, not Republicans. Maybe it doesn’t reflect a conservative shift at all. Maybe it just reflects the fact that Democrats and some independents are more willing to say that the government should promote traditional values when a party they favor is in power. This is also evident in the small, though not statistically significant, shift in the percentage of conservatives who said “not favor any set of values.”
I don’t know if that interpretation is true. The problem is that the Gallup question’s ambiguity admits of too many interpretations. No single survey question, or any combination of questions, is going to provide a bulletproof depiction of public opinion, but surely we can do better. Gallup is up to the task at other times — e.g., in this comparison of multiple surveys on health care, each of which uses somewhat different questions. They should do more of this, which is a much more satisfying meal than bowls of mush about “traditional values.”
