This graph that Brendan Nyhan posted the other day got some attention from my coblogger John Sides and others.
For example, Kevin Drum describes the chart as “pretty cool” and writes, “I think I’m more interested in the placement of senators themselves. Democrats are almost all bunched into a single grouping, with only four outliers. Republicans, by contrast, are spread through considerably more space on both the economic and social dimensions.”
Matthew Yglesias also labels the chart as “cool” and answers Drum by describing the pattern as “an illustration of the importance of setting the agenda. The Democratic leadership has only brought to a vote bills that unite the overwhelming majority of Democrats. . . .”
Yglesias may well be right on this point, but before going further I’d like to stand athwart history and yell Stop” for a moment.
My first reaction when seeing the above graph was, Huh? it doesn’t look right to me. The graph seems to imply that Dems and Reps have a huge huge overlap on social issues, with the median positions of the two parties being virtually identical (and a Democratic senator in Vermont being quite a bit more socially conservative than Republican senators in Indiana, Tennessee, and two senators in Arizona). Can this really make sense?
I asked Brendan, who responded:
The graph is an auto-generated plot of the Lewis-Poole optimal classification scores for the 111th congress generated by Royce Carroll, one of Poole’s students. So the important thing to keep in mind is that it’s only being run on part of one Congress (rather than say, DW-NOMINATE, which Poole runs on all the Congresses as a batch), so the estimates may be screwy depending on the set of available votes. In this case, there apparently haven’t been a lot of votes dividing the Senate Dems internally so their estimated ideal points are tightly clustered, whereas GOP divisions on the votes to date have caused their ideal point estimates to spread in two dimensions (this is not true for the House, where the Dems have had more internal division). Also, the second dimension that’s being recovered for the last ten months in the Senate may or may not be the “social issues” dimension that Carroll labels it. The second dimension is always an interpretive mess in the post-civil rights period, and it’s even worse for <.5 of one Congress. . . . This is my (Brendan's) best guess at what's going on and I don't know exactly what Carroll and/or Poole are doing behind the scenes.
OK, this makes sense. My take-home message here is that we should ignore the second dimension of the above graph, at least until someone can come up with some interpretation of it. The problem isn’t simply an artifact of sparse data or agenda-setting; more fundamentally, we have to know the meaning of a variable before we start talking about it! A key step in any statistical analysis is to connect the inferences back to what is already known about the underlying system (in this case, the positions of senators on social issues).
Or, to put it another way, don’t believe everything you see on a graph.
P.S. I don’t mean this to be intended as some sort of devastating critique of Carroll’s work. I’ve presented enough mistaken graphs on my website that I certainly can’t blame others for posting things without making a sanity check first. Actually, posting stuff quickly on the web is a great way to get others to find your mistakes! And I hope that this post and others will be helpful to Carroll as he continues his research (and also helpful to me once I receive the inevitable corrections of whatever mistakes I’m making here).