Skip down a few posts and you’ll find two by John Sides (here and here), documenting (celebrating?) a change that he perceives in political science research over the last several decades, as indicated by the frequency with which the term “counterintuitive finding” or “counterintuitive result” has been used in the political science journals that are archived in JSTOR. To judge from the trend line in John’s chart, which points sharply upwards, this change has been pretty darned dramatic.
Well, maybe. Or maybe not.
John’s chart shows the number of items in which “counterintuitive finding” or “counterintuitive result” is mentioned (with or without the hyphen) as increasing from 0 during the 1950s and 1960s to 73 during the 1990s. Those figures are exactly correct. But I don’t think that the tale they tell is one of substantial change.
The omitted part of the story is the denominator, which John hasn’t taken into account – the number of items in which the terms in question appear as a percentage of the total number of items. There are many more political science journals now than there used to be. So to some extent the trend that John documents can be attributed simply to the fact that more political science journals are publishing more articles. I don’t want to overstate this effect, though, because even when the total number of items is taken into account the trend line points upwards. To see this, look at the following graph, where I’ve plotted the number of political science articles mentioning “counterintuitive finding” or “counterintuitive result” in a given decade as a proportion of all the political science articles published during that decade. The line trends upwards.(1)
But now let’s be real. There is upwards and there is upwards. What we see in the graph is that the trend line moves upwards from 0 during the 1950s to one-third of one percent (.0036) during the 1990s; that is, during the 1990s the term “counterintuitive finding” or “counterintuitive result” appeared in a grand total of three out of every thousand political science articles.
I acknowledge that tastes may vary vis-à-vis how much change must occur to be important, significant, or noteworthy. At the very least, though, the graph above conveys an altogether different message than John’s does. And naturally I think that the message that my chart conveys is more appropriate than the one that John’s conveys. Namely: Once upon a time, political scientists didn’t talk about “counterintuitive findings” or “counterintuitive results” at all; now they almost never do.
As a follow-up, I also counted the number of articles in which the term “surprising finding” or “surprising result” appeared, which turned out to be more frequent than the “counterintuitive” counterparts of these terms. Even then, though, the proportions were tiny.
I confess that I’m uncertain whether I should be pleased or displeased about the infrequency with which we political scientists describe our findings and results as counterintuitive or surprising. Counterintuitive or surprising findings could be evidence of the vitality of our research programs, indications that we’re operating out on the frontiers and confronting an ever-widening array of previously unexplored new phenomena. That’s a happy reading, isn’t it? Or they could be evidence that our theories are so weak that when, after the 1950s and 1960s, we’ve finally gotten around to devising reasonable empirical tests of them, we find that they simply don’t hold up. That’s a less happy reading, isn’t it?
(1)Rather than taking into account all the items published in these journals, as John did, I confined my analysis to all articles published in these journals. I did this because it seemed appropriate to disregard the miscellany that appears in these journals, and because, as it happened, this restriction actually stacked the deck in John’s favor.