Every week, I write a Friday newsletter exclusively for our subscribers. To give all readers a little sample, below is what Good Authority subscribers received last week. If you’d like to subscribe, you can do so here.
***
In December 2023, The Economist published the findings of a poll conducted with the firm YouGov. The poll asked two questions about the Holocaust and found a striking amount of Holocaust denial among respondents age 18-29:
As you would expect, the finding went viral, in part because it was released amid criticism of the anti-Israel or pro-Palestinian protests on college campuses. Now, it seemed, a substantial chunk of young people actually agreed with antisemitic sentiments.
Pew found a different trend
But in early March 2024, the Pew Research Center released the results of a survey that used a different methodology: a probability-based sample rather than the opt-in panel that YouGov builds its samples from. This is what Pew found about young people and Holocaust denial:
In this sample, the American Trends Panel, very few young people denied the Holocaust. Moreover, young people expressed less conservative views of abortion — a result at odds with the YouGov/Economist poll but in line with many other polls.
As the Pew authors note, their findings confirm a pattern they have seen elsewhere: In opt-in samples, a significant fraction of younger and Hispanic respondents appear to be “bogus” and answer questions insincerely. For example, Pew asked a question designed to see if respondents would carelessly or deliberately misreport something: whether they had the credentials to operate a nuclear submarine. The percentage of young people who said yes (12%) was higher than in any other age group.
I tweeted out the Pew graph on Twitter/X and Bluesky and it got more attention than someone with my modest social media presence usually gets. And in the many replies over the past few days, this was a common sentiment:
With apologies to @RadicalCandyBar, this is the wrong lesson. All surveys, regardless of their methodology, face important challenges. There is no single gold-standard survey methodology that is always guaranteed to work. Indeed, in the 2020 presidential election, polls based on opt-in samples were more accurate than polls that used probability-based samples.
Moreover, the firm behind The Economist poll, YouGov, is actually one of the better pollsters using non-probability techniques. Their methodology produces higher-quality samples than other non-probability vendors — see here and then here. They are used widely in the academic community, including by me. In FiveThirtyEight’s pollster rankings, YouGov is one of the highest-rated pollsters, regardless of methodology.
Here’s the big takeaway
The real lesson is more mundane. Different survey methodologies may indeed produce higher-quality samples and more accurate responses — but the impact of methodology varies across subgroups and topics. From this one example about Holocaust denial, we cannot conclude that all non-probability polls are less accurate than all probability polls for all groups and all topics.
Instead, the smarter take is to be cautious about any one poll finding, especially when that finding hinges on accurately sampling a smaller subgroup, like young people, and getting respondents to report their views correctly on a controversial topic. (For more on that point, see the interstitial discussion in the Pew report on why they wouldn’t have asked the Holocaust denial question that way.)
Ultimately, I don’t think that methodology alone enables us to divide public opinion research into the “good polls” and the “bad polls.” We should be focusing more on the specific findings that polls generate. Some findings merit confidence and others require more investigation.
In other words, we’ll learn more about the babies by keeping all the bathwater.