AAPOR has released its report evaluating polling during this year’s presidential primaries, including the inaccuracies in pre-primary polls in New Hampshire (pdf report, pdf press release). I previously posted about New Hampshire here and here.
The report comes with a significant caveat:
bq. The fact that many pollsters did not provide us with detailed methodological information about their work on a timely basis is one reason we will never know for certain exactly what caused the problems in the primary polling that we studied.
Here are the organizations who polled in New Hampshire but would not provide their original individual-level data: ARG, LA Times/CNN/Politico, Marist College, Rasmussen, Research 2000/Concord Monitor, RKM/Franklin Pierce/WBZ, Strategic Vision, Suffolk Univ./WHDH, and Zogby/Reuters/C-SPAN.
I have nothing nice to say about them.
Here is the summary of relevant factors that may have created error in the pre-election polls:
bq. 1. Given the compressed caucus and primary calendar, polling before the New Hampshire primary may have ended too early to capture late shifts in the electorate there, measuring momentum citizens responded to the Obama victory in the Iowa caucus but not to later events in New Hampshire.
bq. 2. Patterns of nonresponse, derived from comparing the characteristics of the pre-election samples with the exit poll samples, suggest that some groups that supported Senator Hillary Clinton were underrepresented in the pre-election polls.
bq. 3. Variations in likely voter models could explain some of the estimation problems in individual polls. Application of the Gallup likely voter model, for example, produced a larger error than their unadjusted data. While the “time of decision” data do not look very different in 2008 compared to recent presidential primaries, about one-fifth of the voters in the 2008 New Hampshire primary said they were voting for the first time. This influx of first-time voters may have had an adverse effect on likely voter models.
bq. 4. Variations in weighting procedures could explain some of the estimation problems in individual polls. And for some polls, the weighting and likely voter modeling were commingled in a way that makes it impossible to distinguish their separate effects.
bq. 5. Although no significant social desirability effects were found that systematically produced an overestimate of support for Senator Obama among white respondents or for Senator Clinton among male respondents, an interaction effect between the race of the interviewer and the race of the respondent did seem to produce higher support for Senator Obama in the case of a black interviewer. However, Obama was also preferred over Clinton by those who were interviewed by a white interviewer.
The committee members rule out some other factors, such as the exclusion of cell-phone-only households, whether the polls was done with a live interviewer or with a recorded voice, and the possibility that some Independents decided late to vote in the Republican primary. They lack evidence to investigate the role of still other factors. The fact that Clinton’s name occurred before Obama’s on the ballot is one such factor (see this piece by Jon Krosnick). It could have increased her vote share, but obviously there is no way to tell since there was only one ballot in NH.
I’ve only skimmed this report but it looks to be as good as could be expected, given the limited evidence the committee members had to work with.
Generally, the inaccuracies in primary polls shouldn’t obscure the overall accuracy of polls in many other primaries and in the general election.