Peer Review is Not Scientific

How a process designed to ensure scientific rigor is tainted by randomness, bias, and arbitrary delays.

A mess of papers. Image by Christa Dodoo, edited by author in Pixlr.

In my last essay, I discussed the ways in which the academic publishing process is unnecessarily slow, labor-intensive, and exploitative of researchers who are not paid for the research and writing they devote years of their lives to. Academic journal articles are mightily expensive to get access to ($35–55 for a single article, thousands of dollars for an annual subscription), and yet the people who collected the data, analyzed it, wrote it up, submitted it for publication, and revised it receive no compensation no matter how much money is spent accessing their work.

Today, I’d like to set my sights on another troublingly biased and sloppy aspect of the academic publishing process: Peer review. The peer-reviewed process, which is intended to boost the rigor and objectivity of scientific work, is not itself done in an objective, systematic, or scientific way. In fact, it’s one of the most scattershot, inconsistent messes possible.

The process of selecting and assigning reviewers is unsystematic and filled with room for bias and error; review processes themselves are not standardized in any way; the content of reviews is often arbitrary and influenced by personal agendas; it takes a miserably long time to conduct a review; and worst of all, no one involved in the process earns a cent.

…

The first thing I want all lovers of science to know is this: peer-reviewers are not paid. When you are contacted by a journal editor and asked to conduct a review, there is no discussion of payment, because no payment is available. Ever. Furthermore, peer reviewing is not associated in any direct way with the actual job of being a professor or researcher. The person asking you to conduct a peer review is not your supervisor or the chair of your department, in nearly any circumstance. Your employer does not keep track of how many peer reviews you conduct and reward you appropriately.

Instead, you’re asked by journal editors, via email, on a voluntary basis. And it’s up to you, as a busy faculty member, graduate student, post-doc, or adjunct, to decide whether to say yes or not.