In honor of the “Supreme Court’s opening session”:http://www.washingtonpost.com/wp-srv/package/supremecourt/index.html, I thought I would report in on a really interesting seminar that I attended last week where one of my colleagues, “Anna Harvey”:http://as.nyu.edu/object/AnnaLHarvey.html presented a paper she had co-written with one of her graduate students, “Michael Woodruff”:http://politics.as.nyu.edu/object/politics.meetPhD. The paper, which was motivated by an interest in how much the partisan composition of Congress affects Supreme Court decisions, also wrestled with a much more broadly applicable problem of research design: the fear of biased coding. In this particular case, Harvey and Woodruff were concerned that cases decided by liberal courts might be more likely to be coded as liberal rulings than cases decided by conservative courts.
What made the question so interesting, however, is that there was an intermediary step in the coding decision. Namely, cases had to be classified in terms of issues areas; once the issue area was assigned, the case could be classified as either a liberal or conservative ruling. However, the same case could conceivably be assigned more than one issue area, and those issue areas might lead to _the opposite_ classification as either liberal or conservative. For example, a case involving a joint operating agreement between two newspapers (e.g. _Citizen Publishing Co. v. U.S._ (394 U.S. 131 (1969))) can either be classified as a first amendment case or as an anti-trust case. A ruling in favor of the newspapers would be considered _liberal_ if the case is coded as a first amendment case, but _conservative_ if the case is coded as an anti-trust case. So the opportunity for bias is not so much in conversion from the ruling to the liberal/conservative coding _given_ a particular issue area, but rather in the assigning of the issue area in the first place. Here’s the abstract from the paper:
bq. This paper investigates the possibility of confirmation bias in the United States Supreme Court Judicial Database (USSCJD) issue and judgment codes. We ask whether an opinion issued by a liberal Court is more likely to be assigned a USSCJD issue code that leads to a liberal judgment code, relative to an otherwise similar opinion issued by a conservative Court (and vice versa). Using a sample of cases from the USSCJD that pose comparable issue coding choices, we find that cases are disproportionately assigned issue codes that tend to lead to judgment codes confirmatory of expectations about the ideological character of the judgments typically issued by the deciding Court. We also find considerable evidence that variation in the Court’s decision making as a function of congressional preferences has been coded out” of the USSCJD as a result of confirmation bias in the issue codes. Finally, we recode a subset of the USSCJD judgment codes to eliminate confirmation bias. We find that this bias may have led many researchers using the original USSCJD judgment codes to reject the hypothesis of congressional constraint on the Court, despite compelling evidence for the existence of such constraint using the recoded judgment codes.
I’m blogging about this paper for two reasons. First and foremost, I would recommend that people who use the USSCJD data take a look at the paper. But second, I was wondering if other scholars had come across this type of confirmation bias problem in their own research, where the bias is essentially one step removed from the final coding decision, and, if so, how they had dealt with this issue and/or if they had used methods similar to the ones in this paper.
As a final bonus, the take home substantive point from the paper is pretty provocative: despite prevailing views to the contrary, it does indeed appear that the Supreme Court is sensitive to the partisan preferences of Congress. Something to think about as Supreme Court goes about its business this session.