Last year’s British general election left egg on the faces of a great many experts who had predicted a tight race, including pollsters, none of whom came close to predicting the magnitude of the Conservative Party victory. As Britain prepares to vote June 23 on whether to leave or stay in the European Union, pollsters are showing far more caution about predicting the outcome, but once again it looks likely that at least some pollsters will end up red-faced again.
[interstitial_link url=”https://www.washingtonpost.com/blogs/monkey-cage/wp/2015/07/21/how-did-the-pollsters-get-the-british-elections-so-very-wrong/”]How did the pollsters get the British elections so very wrong?[/interstitial_link]
One of the major uncertainties with polling for the E.U. referendum has been the big and persistent gap between phone and online polls. Phone polls have tended to show much higher support for the “Remain” camp (those who support staying in the E.U.), with online polls finding more enthusiasm for “Leave.” There are many explanations for this gap, but no consensus.
The pollsters’ job is even harder than usual because of the one-off nature of the “Brexit” referendum, which means traditional demographic and political methods are not particularly helpful in predicting voting intentions and adjusting polling samples. In short, the usual baseline of previous election results is not available for pollsters this time — the last referendum on British E.U. membership was in 1975. There has been some recent convergence between phone/online modes, reminiscent of “herding” (whereby pollsters make design and reporting decisions that cause published estimates to vary less than expected). The official inquiry into failure of the polls at the 2015 general election cited herding as one of the reasons the polls were so wrong. However, despite this convergence on the Brexit vote, a large gap remains, and either one or both of the methods must be wrong.
So what’s our prediction?
Here at the Polling Observatory, we estimate current support for leaving or staying in the European Union. We pool all the available polling data while taking into account the estimated biases of the individual pollsters relative to the average of all polls. This helps reduce the impact of the random variation that each individual survey inevitably produces — and accounts for systematic differences between polling houses (a consequence of the various methodological choices pollsters must make).
This approach does not, however, address the possibility that the polling industry as a whole could be wrong. Our estimates at least overcome dangers of using the polling average — the “poll of polls” — without adjusting for the mix of polls observed in a given week. The large gaps between phone and Internet polls mean much of the apparent movement in the “poll of polls” is actually the result of shifts in the mix of polling rather than genuine shifts in public sentiment.
Our latest estimates cover the past year up to the end of May, with just three weeks to go until referendum day (see graph). The underlying trend over this period has been of a slow but steady rise in support for leaving the E.U., which stood at around 35 percent in May 2015 and now stands at 42.3 percent.
[interstitial_link url=”https://www.washingtonpost.com/news/monkey-cage/wp/2015/11/30/will-britain-vote-to-leave-the-e-u-we-looked-at-the-polls-for-you/”]Will Britain vote to leave the E.U.? We looked at the polls for you.[/interstitial_link]
Yet support for the Remain side has been gaining ground since February as well and continues to run slightly ahead at 46.5 percent. At no point over the past year has the Leave side had an overall lead. The gains of both sides since the spring reflect undecided voters coming off the fence in both directions as voting day approaches and the campaign intensifies. The end result is a much lower share of undecided voters, and a much narrower gap between the sides, than we reported in the Monkey Cage last fall.
Our method allows us to estimate how each pollster’s methods affect their results compared with the average of other pollsters. This does not reveal who is the most accurate, since there is no way to compare their results with that of a past referendum, and we do not know whether the polling industry as a whole is wrong (as it was last year). It does provide important information, however, about whether we should expect a given poll to show a lead for one side or the other. For instance, a recent ICM poll put Leave in a three-point lead using both telephone and online methods, causing much surprise, since phone polls have tended to favor Remain.
In Table 1 below, we report all polling companies’ support for Remain or Leave (and undecideds) relative to the average pollster. Here we see that the “bias” of ICM’s telephone polling does not cleanly fit the mode difference identified in much commentary (on average it tends to put Remain 1.4 points lower and Leave 0.9 points higher). If any telephone pollster was going to show a lead for Britain leaving the E.U., it was ICM, since its net bias tends to favor Leave. A lead for Leave from IPSOS-MORI, which tends to give a net 13-point advantage to Remain compared with the average pollster, would be much more surprising.
Given the wide divergence of the polls, where else can we look for possible clues about the outcome June 23? We’ve explored a range of expert predictions, summarized in Table 2 below.
The British love a good bet, and betting markets are often cited as a source of predictions about election outcomes. The implied probability of Britain leaving the E.U. is just above 25 percent with traders on the online betting exchange Betfair and with the traditional bookmakers Ladbrokes (both of which have been taking large volumes of bets). This prediction is very similar to the Fisher-Renwick-Shorrocks forecasting model, based on historical patterns of referendum polls and outcomes in Britain and on the E.U., which points to a 28 percent probability for Leave.
Predictions of “super-forecasters” (individuals whose personality traits rather than specialist knowledge make them extremely skilled forecasters) on Philip Tetlock’s Good Judgment Project have been much heralded. At present they put the probability of Brexit at a much lower 11 percent than either the betting markets or the forecasters. If you trust the super-forecasters, a bet on Britain staying in the E.U. looks like good value right now.
[interstitial_link url=”https://www.washingtonpost.com/news/monkey-cage/wp/2016/02/15/will-britain-vote-to-leave-the-eu-these-six-factors-will-make-the-difference/”]Will Britain vote to leave the E.U.? These six factors will make the difference.[/interstitial_link]
This week, the U.K. Political Studies Association released findings from its own expert survey of more than 500 political scientists, pollsters and journalists. This indicated rather higher odds of Britain leaving the E.U., with political scientists most hawkish of all about the prospects for the Leave campaign — standing at just under 40 percent.
What to make of these wildly different but highly informed and often methodologically sophisticated predictions is anyone’s guess. All the predictions agree on the favorite — Remain — but not on much else. A victory for Leave would thus upend the consensus expectations of pollsters, punters, pundits and political scientists. Just like the Conservatives’ victory did last year.
Robert Ford is professor in politics at the University of Manchester, United Kingdom. He has various research interests within the broad umbrellas of racial attitudes and intergroup relations, public opinion research and methodology and partisan and electoral politics.
Will Jennings is professor of political science and public policy at the University of Southampton, United Kingdom. He specializes in public opinion, voting and public policy.
Mark Pickup is associate professor in the department of political science at Simon Fraser University, Canada. He is a specialist in comparative politics and political methodology.