From Brendan Nyhan:
* Pre-accept articles.
* Choose a random sample of articles and “audit” them by replicating them.
* Give “frequent flyer” points to scholars who are particularly diligent in helping journals, e.g., via peer review.
I think these ideas are interesting. At this point, given the many dysfunctions in scholarly publishing, let’s be honest: it’s time to experiment. I like the idea of pre-accepting articles based on their hypotheses and proposed research design. It helps eliminate the “file drawer problem” — where null results never get published — and it gives authors a real incentive to pre-commit to a set of hypotheses rather than data-dredging until they find something that is “statistically significant.”
Some colleagues and I had also discussed the problem of replication. Some journals are already requiring replication datasets. But I agree with Brendan: we need to go further. I would favor scholarly journals actually paying scholars a fee to do the replication. This is something many graduate students could do, for example. They would benefit from the learning process of retracing someone else’s steps (and possibly correcting them), and getting paid never hurts. There also needs to be a cultural shift: we need to value replication as much as (or almost as much as) original scholarship. At this point, we might learn as much from the former as the latter.
As for frequent flyer points, I cannot agree more. In political science, we are producing better trained scholars, often armed with better-quality data. This leads to an increasing number of manuscripts and often slow review processes at top journals, who struggle to get enough people to peer review and to do so promptly. (I just turned in two reviews late myself.) For junior scholars on the tenure clock, the resulting delays are a big problem. Nor do these delays help the discipline accumulate knowledge effectively. I’d also be in favor of publishing the names of scholars who do not review, not just the ones who do. There needs to be some accountability.
Maybe Brendan’s ideas or mine would have some unforeseen consequences. Maybe they’d even prove (somehow) worse than the status quo. But why not try them and see? There are lots of journals. Maybe a few would be willing to experiment.