During our discussion of estimates of teacher performance, Steve Sailer wrote:
I suspect we’re going to take years to work the kinks out of overall rating systems.
By way of analogy, Bill James kicked off the modern era of baseball statistics analysis around 1975. But he stuck to doing smaller scale analyses and avoided trying to build one giant overall model for rating players. In contrast, other analysts such as Pete Palmer rushed into building overall ranking systems, such as his 1984 book, but they tended to generate curious results such as the greatness of Roy Smalley Jr.. James held off until 1999 before unveiling his win share model for overall rankings.
I remember looking at Pete Palmer’s book many years ago and being disappointed that he did everything through his Linear Weights formula. A hit is worth X, a walk is worth Y, etc. Some of this is good–it’s presumably an improvement on counting walks as 0 or 1 hits, also an improvement on counting doubles and triples as equal to 2 and 3 hits, and so forth. The problem–besides the inherent inflexibility of a linear model with no interactions–is that Palmer seemed chained to it. When the model gave silly results, Palmer just kept with it. I don’t do that with my statistical models. When I get a surprising result, I look more carefully. And if it really is a mistake of some sort, I go and change the model (see, for example, the discussion here). Now this is a bit unfair: after all, Palmer was doing this in his spare time and I’m a professional statistician–it’s my job to check my models.
Still and all, my impression is that Palmer was locked into his regression models and that it hurt his analysis. Bill James had a comment once about some analysis of Palmer that gave players negative values in the declining years of their careers. As James wrote, your first assumption is that when a team keeps a player on their roster, they have a good reason. (I’m excepting Jim Rice from this analysis. Whenever he came up to bat with men on base, it was always a relief to see him strike out, as that meant that he’d avoided hitting into a double play.)
Bill James did not limit himself to linear models. He often used expressions of the form (A+B)/(C+D) or sqrt(A^2+B^2). This gave him more flexibility to fit data and also allowed him more entries into the modeling process: more ways to include prior information than simply to throw in variables.
What about my own work? I use linear regression a lot, to the extent that a couple of my colleagues once characterized my work on toxicology as being linear modeling. True, these were two of my stupider colleagues (and that’s saying a lot), but the fact that a couple of Ph.D.’s could confuse a nonlinear differential equation with a linear regression does give some sense of statisticians’ insensitivity to functional forms. we tend to focus on what variables go into the model without much concern for how they fit together. True, sometimes we use nonparametric methods–lowess and the like–but it’s not so common that we do a Bill James and carefully construct a reasonable model out of its input variables.
But maybe I should be emulating Bill James in this way. Right now, I get around the constraints of linearity and additivity by adding interaction after interaction after interaction. That’s fine, but perhaps a bit of thoughtful model construction would be a useful supplement to my usual brute-force approach.
P.S. Actually, I think that James himself could’ve benefited from the discipline of quantitative models. I don’t know about Roy Smalley,Jr., but, near the end of the Baseball Abstract period, my impression was that James started to mix in more and more unsupported opinions, for example in 1988 characterizing Phil Bradley as possibly the best player in baseball. That’s fine–I’m no baseball expert, and maybe Phil Bradley really was one of the top players of 1987, or maybe he’s a really nice guy and Bill James wanted to help him out, or maybe James was just kidding on that one.. My guess (based on a lot of things in the last couple of Baseball Abstracts, not just that Phil Bradley article) is simply that James had been right on so many things where others had been wrong that he started to trust his hunches without backing them up with statistical analysis. Whatever. In any case, Win Shares was probably a good idea for Bill James as it kept him close to the numbers.
P.P.S. See also the comments here.