The following is a guest post by Jay Ulfelder.
****
Do recent revolutions in the Arab world show the value or reveal the futility of statistical modeling for forecasting?
In the parts of the blogosphere I watch, it feels like a lot of people have already answered this question in the negative. For example, a February 11 post on Wired.com’s Danger Room blog claims that “America’s military and intelligence agencies have spent more than $125 million” in the last three years on computer models to forecast political unrest, “but if any of these algorithms saw the upheaval in Egypt coming, the spooks and the generals are keeping the predictions very quiet.” On a February 8 post to the The Arabist, Issandr El Amrani brings it more forcefully:
Quantitative analysis and the behaviouralist approach of most American PoliSci academics is a big steaming turd of horseshit when applied in the Middle East. Statistics are useful, yes, when you are in a country that has relevant statistics or where polling is allowed. But things like electoral statistics tell you very little about the political reality of dictatorships, because the data sets are inherently flawed, since they’re either unavailable, fraudulent, or irrelevant.
Frankly, I don’t know how many statistical models have been designed to forecast the kinds of popular unrest and regime change we’ve seen spread across North Africa and the Middle East so far this year. Of the forecasting models I do know about, I don’t know what most of them had forecast for 2011 in the countries where upheavals have occurred. At the same time, I can’t say who qualifies as an expert on politics in those countries, or what those experts have said in the past year or two about the prospects for revolutionary change in the countries they study. Of course, no one else knows the answers to those questions either, so it’s odd to see some people already offering summary judgments about how statistical forecasting have fared in these cases, and how they are faring in comparison to country or area experts. Heck, in most of these cases, we don’t even know the outcomes yet!
Since we can’t systematically compare expert and statistical forecasts of revolutions in the Arab world, I’d like to recast the moment as an opportunity to revisit the pros and cons of statistical modeling for forecasting in a more general way.
The following is a guest post by Jay Ulfelder.
****
Do recent revolutions in the Arab world show the value or reveal the futility of statistical modeling for forecasting?
In the parts of the blogosphere I watch, it feels like a lot of people have already answered this question in the negative. For example, a February 11 <a href=”http://www.wired.com/dangerroom/2011/02/pentagon-predict-egypt-unrest”>post</a> on Wired.com’s Danger Room blog claims that “America’s military and intelligence agencies have spent more than $125 million” in the last three years on computer models to forecast political unrest, “but if any of these algorithms saw the upheaval in Egypt coming, the spooks and the generals are keeping the predictions very quiet.” On a February 8 <a href=”http://www.arabist.net/blog/2011/2/8/egypt-and-poli-sci-us-academia.html”>post</a> to the The Arabist, Issandr El Amrani brings it more forcefully:
<blockquote>Quantitative analysis and the behaviouralist approach of most American PoliSci academics is a big steaming turd of horseshit when applied in the Middle East. Statistics are useful, yes, when you are in a country that has relevant statistics or where polling is allowed. But things like electoral statistics tell you very little about the political reality of dictatorships, because the data sets are inherently flawed, since they’re either unavailable, fraudulent, or irrelevant.</blockquote>
Frankly, I don’t know how many statistical models have been designed to forecast the kinds of popular unrest and regime change we’ve seen spread across North Africa and the Middle East so far this year. Of the forecasting models I do know about, I don’t know what most of them had forecast for 2011 in the countries where upheavals have occurred. At the same time, I can’t say who qualifies as an expert on politics in those countries, or what those experts have said in the past year or two about the prospects for revolutionary change in the countries they study. Of course, no one else knows the answers to those questions either, so it’s odd to see some people already offering summary judgments about how statistical forecasting have fared in these cases, and how they are faring in comparison to country or area experts. Heck, in most of these cases, we don’t even know the outcomes yet!
Since we can’t systematically compare expert and statistical forecasts of revolutions in the Arab world, I’d like to recast the moment as an opportunity to revisit the pros and cons of statistical modeling for forecasting in a more general way.
Let me make clear at the start that I’m not a disinterested party. From 2001 until the end of 2010, I worked for a for-profit firm, <a href=”http://www.saic.com”>SAIC</a>, as research director for the <a href=”http://globalpolicy.gmu.edu/pitf/”>Political Instability Task Force</a>, a U.S. Government-funded program that aims to forecast and explain various forms of political change in countries worldwide. I left that job at the start of 2011, but I continue to do similar work as a consultant. In short, I have a vested interest in peoples’ beliefs on this topic. Still, I think that the case for statistics-based forecasting is strong enough to stand on its own. With all of that on the table, here are a few of what I see as the general advantages of using statistical models instead of expert judgment to forecast political change.
<ul>
<li>Clarity. Statistical modeling forces us to organize our information and ideas about what predicts what in a transparent and concise way, and it can quickly give us sharp feedback on how useful those constructs are likely to be in the form of statistical measures of forecast accuracy. To do statistical analysis (well), we have to define clearly and then quantify our measures; scrutinize the resulting data; and choose a model that matches our best ideas about how those data are generated and hang together. Those ideas may turn out to be badly wrong, but statistics can quickly tell us if that’s so, and then we can tinker with the various elements one at a time to discover where we went off track. It is virtually impossible for experts to subject the mental models on which their forecasts are based to the same kind of scrutiny, and my impression is that most experts don’t even try.</li>
<li>Comparability. This virtue is really an extension of the first one. Statistical modeling produces forecasts that are directly comparable across cases and over time. Some decision-makers have to consider a lot of cases simultaneously, and they often get their information about those cases from a gaggle of analysts who specialize in specific cases or issues. Other decision-makers may be following a small number of cases or even a single case and want to know whether risks are increasing or declining over time. It’s very difficult to calibrate subjectively generated forecasts from numerous people so they are directly comparable to one another. Analysts may define the same event in different ways, or mean different things by terms like “unlikely,” or exhibit a host of other subjective ticks that make it difficult to compare their assessments. By contrast, that kind of comparability is built into forecasts based on statistical models, which can be used to generate both a point estimate of an event’s likelihood and a mathematical estimate of our uncertainty about that point estimate.</li>
<li>Efficiency. Good data does not come cheap, but neither do structured expert judgments. Once we have developed a statistical model for forecasting and have access to updated inputs, the process of generating new forecasts is usually simple and quick. By contrast, the process of eliciting and assembling fresh and comparable judgments from a large number of experts is usually neither.</li>
<li>Flexibility. Statistical modeling gives us a structured way to consider alternative futures. Changes in values of predictor variables suggest specific changes in the likelihood of an event’s occurrence. Forecasting models can provide a structure for considering the likely implications of those changes, including events and trends that might not directly affect the risk of the event of interest but could shape the value of those predictor variables.</li>
</ul>
In the past couple of weeks, I’ve been intrigued to see people who probably would not describe themselves as statistically inclined trying to draw lessons from Tunisia and Egypt by spotting patterns in <a href=”http://www.nytimes.com/2011/02/05/opinion/05blow.html”>tables</a> or <a href=”http://blogs.ft.com/beyond-brics/2011/02/08/chart-of-the-week-understanding-the-arab-moment/”>charts</a> summarizing selected conditions in those countries before their revolutions. These exercises go something like this: “Here’s a list of factors that might have driven those events; there seems to be a pattern in those factors; and, based on that pattern, here’s where unrest is likely to happen next.” Whenever I see that, I think, “Hey, we’ve got a technology that will efficiently find the patterns for you!” Statistical models are usually better at finding those patterns than we are on our own, and they can generate very explicit forecasts when they do. Those forecasts won’t always be correct, but they will find patterns and give formal expression to your ideas in ways that table-scanning or chart-eyeballing rarely will.
I don’t want to sound like the Ginsu-knife pitchman here, though. Even well-designed statistical models have significant limitations, and it’s important to be honest about where those limits lie.
<ul>
<li>Inertia. Statistical models are not good at predicting novel events, or familiar events in novel circumstances. If the conditions under which future events occur don’t resemble the circumstances under which some portion of past events on which the model was trained occurred–in other words, if the future bears little resemblance to the past–then a statistical model is going to be hard pressed to see that future coming. This is the classic problem of induction, and forecasts based on statistical models are vulnerable to this problem by design. Modelers can hedge against this problem by carefully choosing their frames of reference and frequently updating their estimates, but they can never make it go away.</li>
<li>Simplicity. Simplicity is often a virtue, but it can also be a vice. One important weakness of statistical models of political change is their inability to reflect and predict the dynamic processes by which that change arises. Models that produce annual forecasts of revolution might prove quite accurate in their predictions and still have next to nothing to say about the causal mechanisms at work and actors involved. Those mechanisms and actors are often of great interest while those changes are occurring, especially to policy-makers contemplating how to respond to unfolding events. By necessity, the crises that statistical models usually forecast well–civil-war onsets, coups, regime changes, and so on–are observable from high altitude, while the day-to-day and even week-to-week or month-to-month events that carry societies to those crises (or not) are things that we still cannot forecast well (and probably with good reason, because the process generating them is, I believe, a lot less deterministic).</li>
</ul>
I’ll wrap up this guest post by talking about one aspect on which I haven’t tried to compare expert judgments to statistical forecasts, namely, accuracy. Advocates of statistical forecasting often support their position by claiming that, on average, statistical models produce more accurate forecasts <a href=”http://www.pharmaphorum.com/2011/01/19/the-trouble-with-expert-judgment-in-forecasts-2/”>than experts do</a>. To be honest, though, it’s not clear to me whether or not that’s true–or, more specifically, under what conditions it’s true. Even if it were true, it’s a generalization, and we can’t know in advance how it will apply to specific problems, like forecasting revolutions (or, for that matter, <a href=”doi:10.1016/j.ijforecast.2007.05.003″>the outcomes of football games</a>. Just as there is no “average individual,” there is no average forecasting problem, so it’s impossible to talk confidently about relative accuracy without question-specific evidence.
One thing we do know, though, is that the news media pay disproportionate attention to <a href=”http://www.project-syndicate.org/commentary/tetlock1/English”>less accurate expert forecasts</a>, and this bias makes it hard for the would-be consumers of these prognostications to figure out to whom to listen. Before naysayers speak too loudly about the flaws they see in statistical forecasting, they might want to think carefully about how to overcome this bias and help decision-makers find useful information in the thick but muddy flow of available expert judgments. Hey, maybe a statistical model could help with that…