Austria’s Daniela Iraschko-Stolz soars during a women’s ski jumping training session at the 2014 Winter Olympics. (Matthias Schrader/Associated Press)
This is a guest post by Dartmouth economist Eric Zitzewitz. When not writing about such topics as benchmark indices and equity compensation, he is the author of two papers on bias in figure skating judging.
Maybe it’s something in the water, but journalists seem especially interested in corruption this Olympics. And the Olympics are cooperating: halfway through the first week, we already have criticism of the slopestyle judging and far more serious allegations of a figure skating vote trading scheme reminiscent of the scandals in 1998 and 2002.
Some amount of bias is probably inevitable in judged sports. The question is, can a sport do anything to control it? As an otherwise serious academic economist, I got interested in Olympic judging because I was interested in how groups of people make decisions. A group of judges picking a gold medalist is at least a little analogous to a committee deciding whom to hire. Individual members each have their own opinions about the decision, and these opinions reflect both valuable information and the members’ biases. The question is how to combine these opinions, and how to create incentives for members to stay as unbiased as possible.
Both ski jumping and figure skating have nationalistic judging biases, where judges give higher scores to athletes from their countries. But the sports take very different approaches to dealing with this. Ski jumping has its international federation select the judges for competitions like the Olympics, and I find that they select the least biased judges. Figure skating lets its national federations select the judges, and my research showed that they select the most biased judges.
This creates different incentives for judges. Ski jumping judges display less nationalism in lower-level competitions — it appears they keep their nationalism under wraps in less important contests to avoid missing their chance at judging the Olympics. Figure skating judges are actually more biased in the lesser contests; they may actually be more biased than they would like to be due to pressure from their federations.
Despite being nationalistically biased, ski jumping judges appear to care about fairness. Having a compatriot judge is actually not an advantage in ski jumping. I find that while you get a higher score from your compatriot, the other four judges lower your score ever so slightly, leaving you no better (or worse) off.
The reverse happens in figure skating. Not only does a figure skater with a compatriot judge get a higher score from that judge, but they also get higher scores on average from the other judges, too (compared with events when they are not represented on the panel). This is evidence of vote trading, of the kind that occurred at the Olympics in 1998, 2002, and (allegedly) is occurring in 2014. Most of the benefit of having a compatriot judge actually comes through the vote trading. Skaters even benefit from having compatriot judges on the panels of other events, which is consistent with the fact that the vote trading we know about is often across events.
The dysfunctionality of the sport is also revealed by how it reacted to the 2002 judging scandal. The International Skating Union made a couple of sensible reforms, such as increasing the size of the judging panel (at least temporarily) and making the scoring system more objective (although some think they went too far). But most of their response consisted of hiding the evidence of bias. The ISU stopped revealing which judge gave which score, making it much harder for competitors and fans to see whether the judging was fair. The ISU even went back and altered online score sheets from earlier competitions, obfuscating which judge gave which score and even which country each judge represented. They also began randomly dropping scores from three out of 12 judges. As any statistician can tell you, an average of nine out of 12 scores is essentially the average of the 12 scores plus a random number. When Yale statistics professor Jay Emerson noticed that in one case this randomness had altered who won a medal, it appears that the ISU responded by scrambling the order that scores were reported on score sheets. The only plausible purpose of this change was to make it harder to identify cases where randomness had affected results. All this suggests that the focus has been on hiding problems rather than fixing them.
While the scrubbing of past score sheets and randomization are very hard to defend, the ISU argues that judge anonymity should actually help prevent vote trading deals by making it easier for judges to secretly renege on them. The argument is basically the same for the secret ballot in elections or voice voting in legislatures, where the idea is to protect voters or lawmakers from external influences. Anonymity, though, also protects judges from monitoring by outsiders. I find that judging biases got about 20 percent larger after anonymity was introduced, suggesting that in this case, transparency is the better approach.
Of all these results, I am most intrigued by the contrast between the ski jumping judges undoing each other’s biases and the figure skating judges reinforcing them. When we make decisions in a group at work, we often encounter individuals with strong biases — say to hire a particular type of job candidate. When we do, we have a choice. We can act like a ski jumping judge, and resist the bias, in an effort to keep things fair. Or we can act like a figure skating judge and say “hiring this guy really seems important to Joe, I wonder what he’ll give me in return if I go along.” We have probably all seen examples of both in our lives.
What determines whether we get “ski jumping” or “figure skating” behavior out of our colleagues? Incentives are part of the story, and an organizational culture helps create those incentives. Whether a skilled vote trader is viewed as a savvy operator, or as someone who undermines the organization’s mission, is partly a question of values. But culture is also a product of the incentives people face. Whether the “skilled operators” move up in the world affects the extent to which that skill is respected.
As an economist, I know how to fix figure skating judges’ incentives. Fixing culture is harder, but my hope would be that if you can get the incentives right, the culture will eventually follow.
