I posted a couple days ago on a controversy over methods of counting war deaths. This is not an area I know much about, and fortunately some actual experts (in addition to Mike Spagat, who got the ball rolling) wrote in to comment.
Their comments are actually better than my original discussion, and so I’m reposting them here:
Bethany Lacina writes:
I didn’t work on the Spagat et al. piece, but I’m behind the original battle deaths data. Your readers might be interested in the “Documentation of Coding Decisions” available at the Battle Deaths Database website. The complete definition of “battle deaths”–admittedly a tricky concept–starts on page 5. The discussion of Guatemala starts on page 219.
The goal of the Documentation is to preserve all the sources we used and the logic of how we interpreted them. If you or any of your readers know of sources we haven’t consulted, for any conflict, it would be terrific to hear about them: battledeaths@prio.no
Ameilia Hoover writes:
This debate is missing a key part — namely, any sort of awareness that there are estimation methods out there that improve on both surveys (usual stalking horse of Spagat, et al.) and convenience data such as press reports (usual stalking horse of many other people).
Spagat et al. are more or less correct about all of the many, many problems with survey data. They’re right to criticize OMG (OMG!). But this isn’t, or at any rate shouldn’t be, a debate between survey and convenience methods.
The authors dismiss (at page 936; again in footnote 2) estimation techniques other than retrospective mortality surveys and “collation of other reports”. But while it’s true that demographers often (usually? Help me out here, demographers) use retrospective survey data in their analyses, there’s also a long-standing literature that uses census data instead, matching across sources in order to model (a) patterns of inclusion in convenience sources and (b) the number of uncounted cases. This method accurately counts deer, rabbits, residents of the United States, children with various genetic disorders, and HIV patients in Rome (to name a few examples I can think of) — and, yes, also conflict-related deaths.
Bethany Lacina’s link to the PRIO documentation is really interesting on this point. For El Salvador, the case with which I’m most familiar, PRIO’s best estimate is 75,000 total deaths — 55,000 battle deaths and 20,000 “one sided” deaths. I think this is reasonable-ish (maybe the total is between 50,000 and 100,000?), but there’s no actual evidence to support such a number. The sources PRIO cites are expert guesses, rather than statistical analyses of any sort.
PRIO’s El Salvador estimates are based on *neither* documented/documentable convenience data (e.g., press reports, NGO reports) *nor* survey data. The United Nations-sponsored Truth Commission for El Salvador’s list of documented (and partially documented) deaths includes about 14,000 total deaths, many of which are duplicates. Two other NGO databases include about 6,000 and about 1,500 deaths, respectively. Again, there’s significant overlap and many duplicates. Yet no one imagines that the total deaths in this conflict were 21,500. In the Salvadoran case as in many others, inclusion in the data is incredibly biased toward urban, educated, and politically active victims. (They’re also biased in any number of other ways, of course.)
Prof. Gelman is right to point out the discrepancy between the Guatemala survey numbers, the Guatemala convenience (PRIO) numbers, and the number that most people cite as the best approximation for Guatemala (200,000). Importantly, that “200,000” is based in large part on census numbers. (See http://shr.aaas.org/mtc/chap11.html and http://shr.aaas.org/guatemala/ceh/mds/spanish/toc.html, statistical analyses from the Commission for Historical Clarification, Guatemala’s Truth Commission.) So why ignore census correction methods?
Given that discrepancies between survey and convenience data are very often dwarfed by discrepancies between those numbers and the numbers we believe to be correct, I worry that the surveys-versus-convenience-data fight isn’t more about protecting academic projects and prerogatives than about actually finding the correct answer.
Romesh Silva writes:
The claim that demographers often/usually use retrospective mortality surveys in their analyses is a bit off the mark. Looks like it is borne out of some confusion in some parts of the academy between the methods of demographers and epidemiologists…
Broadly speaking, demographers use a wide array of sources including population censuses, vital registration systems, demographic surveillance systems, and surveys (of all flavors: longitudinal, panel, and retrospective).
In the field of conflict-related mortality, demographers have actually relied almost exclusively on sources other than surveys. For example, Patrick Heuveline and Beth Daponte have used population censuses (and voter registration lists) in Cambodia and Iraq, respectively, and demographers at the ICTY (Helge Brunborg and Ewa Tabeau) have used various types of “found data” which equate to (incomplete) registration lists along side census correction methods. Distinguished demographers Charles Hirschman and Sam Preston were in the minority amongst demographers, when they used a household survey to estimate Vietnamese military and civilian casualties between 1965 and 1975.
The folks who routinely use surveys in the field of conflict-related mortality are epidemiologists, not demographers. The folks at Johns Hopkins, Columbia’s Mailman School of Public Health, Harvard Humanitarian Initiative, Physicians for Human Rights, MSF, Epicentre, etc use variants of the SMART methodology with a 2-stage cluster design are epidemiologists. This design and methodology has been coarsely adapted from a back-of-the-envelope method used to evaluate vaccination coverage in least developed countries. However, epidemiologists at the London School of Hygiene and Tropical Medicine have recently noted that this method “tends to be followed without considering alternatives” and “there is a need for expert advice to guide health workers measuring mortality in the field” (See http://www.ete-online.com/content/4/1/9).
I just thought it might help to put this all in one place.