A new poll by the Institute for Middle East Understanding Policy Project got some attention last week. The press release said, “New Poll Shows Gaza Was A Top Issue For Biden 2020 Voters Who Cast A Ballot For Someone Besides Harris.”
That headline is an accurate summary of the poll. But the poll’s design doesn’t provide good support for the claim. It’s useful to explain why, because the issues in this poll occur in other polls as well.
A preliminary question about sampling
The poll, conducted by YouGov, is described as interviewing 2020 Biden voters who did not vote for Harris. I assume this means that respondents were screened based on how they reported voting in 2020 and 2024. (There is discussion of the survey’s methodology – see the last page of this pdf – but this point isn’t clear.)
It is possible that YouGov’s respondents, many of whom are regularly surveyed, may have been surveyed in 2020, meaning that their report of how they voted in 2020 comes from the month after that election. If so, then there are no concerns that, as of 2024, respondents wouldn’t remember correctly how they voted 4 years ago. In any case, let’s assume that the survey can, with reasonable accuracy, identify a sample of people who voted for Biden in 2020 but not Harris in 2024.
However, there is no way to know whether this poll’s sample is representative of that group. There is no U.S. Census of Biden-but-not-Harris voters and thus we don’t know how exactly to weight the sample. There is a statement at the end of the survey about how it is weighted but it’s not clear what population it’s being weighted to resemble (maybe all registered voters?). I don’t know how using different weighting schemes would or would not affect the results. That alone introduces some uncertainty here.
A problematic survey question
But the bigger challenge lies with the key survey questions, not with the sample. The main point of the survey is to ask people why they didn’t vote for Harris even though they voted for Biden. So the survey asks, “Which one of the following issues was MOST important in deciding your [vote for presidential candidate/decision to not to vote for president]?” Then respondents see a list of 6 issues to choose from. This is the result:
The first issue with this question is this: people are not good at reporting the reasons for their choices. If you were a regular reader of The Monkey Cage, the predecessor to Good Authority, you’d know this because we have been writing about it for over 16 years.
In this 2008 post, my late colleague Lee Sigelman summed up the issue:
The problem…is that when pollsters ask voters which issues were most important to them, what the voters tell them can’t be taken at anything approaching face value. People just aren’t reliable reporters of their own mental processes. (The classic statement of this point is Nisbett and Wilson’s “Telling More Than We Can Know: Verbal Reports on Mental Processes,” Psychological Review 84 (May 1977), pp. 231-259.) In the context under consideration here, this unreliability typically takes the form of ex post facto rationalizations.
Lee also cited Wendy Rahn, Jon Krosnick, and Marijke Breuning’s “Rationalization and Derivation Processes in Survey Studies of Political Candidate Evaluation,” which concluded that “Voters’ reports of the reasons for their preferences were principally rationalizations.”
Simply put, the fact that 29% of Biden-but-not-Harris voters said that “Ending Israel’s violence in Gaza” was the most important reason for not supporting Harris doesn’t tell us very much about their actual reasons.
The second challenge with the survey question concerns the list of issues. This challenge has two parts. For one, the Gaza issue is phrased as a position: “Ending Israel’s violence in Gaza.” But everything else is just a topic: “The economy,” “Medicare/Social Security,” “Immigration/border security,” and so on. So we have an apples-and-oranges problem. What if the other options had also been positions? “Bringing prices down,” or “Protecting Social Security,” or “Ensuring the U.S.-Mexico border is secure.” The results could have been different. For more on this, see Charlotte at Medium Data.
For another, asking people to choose the “most important” thing from a defined list is inherently fraught because everything depends on the list. This poll question included six reasonable issues, but there are many more not on the list: education, climate change, Biden’s age, and on and on.
So that 29% figure is highly dependent on the list! This is a well-known challenge in survey research: how to select the alternatives that people choose from. If the survey had listed other issues, the 29% figure could easily have been lower and perhaps Israel/Gaza would not even have been the “number one” issue.
Another problematic survey question
Another type of question on the survey doesn’t help much, either. This one asks, “If Kamala Harris had pledged to break from President Biden’s policy toward Gaza by promising to withhold additional weapons to Israel for committing human rights abuses against Palestinian civilians, would it have made you more or less likely to vote for Harris, or would it not make any difference?” The IMEPP makes a lot of the fact that 36% said “more likely” but only 10% said “less likely.”
But, as Alex Coppock and Matthew Graham have shown, people do not answer this question by accurately reporting whether something actually would have changed their mind or their choice. They appear to be simply reporting an attitude – in this case about whether they support withholding weapons from Israel. You can’t interpret this question as telling us anything about what would have happened in 2024 if Harris had broken with Biden on Gaza.
The takeaway
Did some voters vote for Biden but not Harris, either because they supported Trump or sat out the election? Yes. Did some of these voters disagree with Biden’s policy on Gaza? Yes. But you cannot use this type of survey to diagnose why Harris lost these voters or to put the blame on Biden’s policy toward Israel and Gaza.
Stay up to date on all things politics and political science. Bookmark our landing page and sign up for Good Authority’s weekly newsletter by entering your email address in the box below.