Polls suggest that Donald Trump has weathered his impeachment remarkably well. The talk of the past couple of days among election prognosticators has been a Gallup poll that pegs Trump’s job approval at 49 percent, the highest Trump has ever registered in that poll.
Of course, you should always look skeptically at outlier polls. But it is also the case that FiveThirtyEight’s approval tracker shows Trump near his highest level of approval since he first entered office, albeit a lower estimate of about 43.5 percent.
But is Trump really getting more popular? His recent polling gains have the hallmarks of a specific kind of polling error called differential nonresponse bias.
What is “differential nonresponse bias”?
As occasional Monkey Cage contributor Andrew Gelman has explained, differential nonresponse bias refers to situations in which changes in polling results are caused by shifts in who responds to the polls rather than actual changes in public opinion. It may be that Trump’s approval is going up because Democrats feel demoralized by the apparently hopeless impeachment trial and so don’t feel like talking to pollsters. Or it could be that Republicans feel so moved to support Trump at when he’s under attack that they are more likely to talk to pollsters than usual.
One example of this occurred during the 2012 election. Gelman and his collaborators Sharad Goel, Doug Rivers, and David Rothschild showed that after Barack Obama’s poor first debate performance against Mitt Romney, the polls showed Romney’s chances of beating Obama surging. But when looking at the survey responses from a group of people who had been asked about their voting intentions repeatedly throughout the campaign, Gelman and colleagues found that survey respondents’ minds weren’t changing after the debate. Rather, Obama supporters were less likely to respond to the surveys during that negative news cycle.
As Trump was heading toward a widely-expected acquittal in the impeachment trial, were Democrats similarly just feeling unenthused about talking to pollsters?
Trump wouldn’t shake hands. Pelosi ripped up his speech. Do Americans care about civility?
Here’s how I checked on Trump’s rising approval.
If differential nonresponse is driving these changes in Trump’s job approval, there should be a clear relationship between the ratio of Democrats to Republicans in survey samples and Trump’s approval ratings. At the most basic level, samples in recent polls do indeed skew a little more than 1 percentage point more Republican.
Other things affect results in ways specific to each pollster, like whether they call respondents or contact them via email or how they statistically adjust to make their samples more representative. To account for that, I focused on changes over time within each pollster’s results. In other words, if Gallup published two polls, I’m looking at whether the one with more Democrats in the sample also has more disapproval of Trump’s job performance.
I gathered polls dating back to early August from 16 prominent pollsters who conducted multiple polls in that time period and who share enough data to determine how many Democrats and Republicans responded to each survey. Overall, that gave me nearly 200 polls to work with.
Let’s start by looking at Gallup, which triggered this most recent round of hand-wringing. In the plot below, I show the relationship between the number of Democrats relative to Republicans in their samples and Trump’s net job approval — the percentage of those who disapprove subtracted from the percentage of those who approve — in those polls.
As you can see, as the relative number of Democrats increases, Trump’s approval decreases. With Gallup, it looks like almost all the variation in Trump’s job approval from one poll to the next can be explained just by looking at how many of each party’s supporters are in their sample.
Gallup is just one pollster, of course, so we need to look further than that. Below, I have constructed similar plots for several other pollsters with enough polls to be worth looking at.
Gallup’s shift is the clearest and most dramatic among the pollsters, but there remains a general pattern that is largely consistent with probability that results are partly determined by differential nonresponse.
Using a statistical method called multilevel modeling, I can estimate how big an impact differential nonresponse has on the polling results after accounting for several other things known to affect the outcome (like so-called “house effects”). The model suggests that for every 1 percentage point increase in the difference between Democrats and Republicans in the sample, you should expect Trump’s net approval to drop by about 0.7 percentage points.
The model also suggests that even after accounting for differential nonresponse, Trump may have gained somewhat in net approval since August, perhaps 2 percentage points (or roughly half the increase detected by FiveThirtyEight’s tracker). But that upward trend goes away when I restrict the analysis to polls that measure partisanship best, by including “leaners” who call themselves independent but later disclose a clear preference for one party over the other.
This means there is reason to believe Trump’s historically stable job approval hasn’t changed much since before the impeachment process began.
What will Trump’s acquittal mean for U.S. democracy? Here are 4 big takeaways.
But there are other possible explanations
A crucial assumption of this method is that people are not disproportionately becoming Republicans as a reaction to these news events. If there was a sudden growth in the number of Republicans, then the polls would need to reflect that to be accurate. Decades of research on partisanship, however, suggests that it is very unlikely for such a meaningful change to happen so quickly.
Another possible explanation for Trump’s improving poll numbers would be that some Americans simply oppose impeachment and as a result have reevaluated Trump. After all, Bill Clinton’s job approval peaked after the House impeached him.
Or maybe with an election coming, Trump is bringing in more support, as his challengers become better known. Incumbent presidents typically become more popular during election years. Barack Obama’s approval went from a similar level to Trump’s to over 50 percent as he ramped up his reelection campaign in 2012.
These and other explanations, including the possibility that Americans are responding to strong economic numbers, are worth serious consideration. But the data available suggest interpreting the recent poll results cautiously for the moment and looking closely at how the partisan composition of polls are changing over time.
Don’t miss anything! Sign up to get TMC’s smart analysis in your inbox, three days a week.
Jacob Long (@jacobandrewlong) is a PhD candidate at Ohio State University’s School of Communication.