SUMMARY: Richard Poynder's essay on peer review is thoughtful and
stimulating but quite wrong! Peer review is like water-quality
control: Everyone shouldn't have to risk doing it all for himself.
(And it has nothing to do with OA, which is about making the filtered,
quality-controlled water free for all.)
http://poynder.blogspot.com/2006/10/open-access-death-knell-for-peer.html
Richard Poynder, in "Open Access: death knell for peer review?" has
written yet another thoughtful, stimulating essay. But I think he (and
many of the scholars and scientists he cites) are quite baldly wrong
on this one!
What is peer review? Nothing more nor less than qualified experts
vetting the work of their fellow specialists to make sure it meets
certain established standards of reliability, quality and usability --
standards that correspond to the quality level of the journal whose
name and track-record certifies the outcome.
Peer review is dynamic and answerable: Dynamic, because it is not just
an "admit/eject" decision by a gate-keeper or an "A/B/C/D" mark
assigned by a schoolmarm, but an interactive process of analysis,
criticism and revision that may take several rounds of successive
revisions and re-refereeing. And answerable, because the outcome must
meet the requirements set out by the referees as determined by the
editor, sometimes resulting in an accepted final draft that is very
different from the originally submitted preprint -- and sometimes in
no accepted draft at all.
Oh, and like all exercises in human judgment, even expert judgment,
peer review is fallible, and sometimes makes errors of both omission
and commission (but neither machinery nor anarchy can do better). It
is also approximate rather than exact; and, as noted,
quality-standards differ from journal to journal, but are generally
known from the journal's public track record. (The only thing that
does resemble an A/B/C/D marking system is the journal-quality
hierarchy itself: Meeting the quality-standards of the best journals
is rather like receiving an A+, and the bottom rung is not much better
than a vanity press.)
But here are some other home truths about peer review (from an editor
of 25 years' standing who alas knows all too well whereof he speaks):
Qualified referees are a scarce, over-harvested resource. It is hard
to get them to agree to review, and hard to get them to do it within a
reasonable amount of time. And it is not easy to find the right
referees; ill-chosen referees (inexpert or biassed) can suppress a
good paper or admit a bad one; they can miss detectable errors, or
introduce gratuitous distortions.
Those who think spontaneous, self-appointed vetting can replace the
systematic selectivity and answerability of peer review should first
take on board the ineluctable fact of referee scarcity, reluctance and
tardiness, even when importuned by a reputable editor, with at least
the prospect that their efforts, though unpaid, will be heeded. (Now
ask yourself the likelihood that that the right umpires will do their
duty on their own, and not even sure of being heeded for their pains.)
Friends of self-policed vetting should also sample for a while the raw
sludge that first makes its way to the editor's desk, and ask
themselves whether they would rather everyone had to contend directly
with that commodity for themselves, instead of having it filtered for
them by peer review, as now. (Think of it more as a food-taster for
the emperor at risk of being poisoned -- rather than as an elitist
"gate-keeper" keeping the hoi-poloi out of the club -- for that is
closer to what a busy researcher faces in trying to decide what work
to risk some of his scarce reading time on, or (worse) his even
scarcer and more precious research time in trying to build upon.)
And peer-review reformers or replacers should also reflect on whether
they think that those who have nothing better to do with their time
than to wade through this raw, unfiltered sludge on their own
recognisance -- posting their take-it-or-leave-it "reviews" publicly,
for authors and users to heed as they may or may not see fit -- are
the ones they would like to trust to filter their daily sludge for
them, instead of answerable editors' selected, answerable experts.
Or whether they would like to see the scholarly milestones, consisting
of the official, certified, published, answerable versions, vanish in
a sea of moving targets, consisting of successive versions of unknown
quality, crisscrossed by a tangle of reviews, commentaries and
opinions of equally unknown quality.
Not that all the extras cannot be had too, alongside the peer-reviewed
milestones: In our online age, no gate-keeper is blocking the public
posting of unrefereed preprints, self-appointed commentaries, revised
drafts, and even updates and upgrades of the published milestones --
alongside the milestones themselves. What is at issue here is whether
we can do without the filtered, certified milestones themselves (until
we once again reinvent peer review).
The question has to be asked seriously; and if one hasn't the
imagination to pose it from the standpoint of a researcher trying to
make tractable use of the literature, let us pose it more luridly,
from the standpoint of how to treat a family member who is seriously
ill: navigate the sludge directly, to see for oneself what's on the
market? ask one's intrepid physician to try to sort out the reliable
cure from the surrounding ocean of quackery? And if you think this is
not a fair question, do you really think science and scholarship are
that much less important than curing sick kin?
Eppur, eppur... what tugs at me on odd days of the week is the
undeniable fact that most research is not cited, nor worth citing,
anyway, so why bother with peer review [or, horribile dictu,
OA!] for all of that? And on the other end, the few authors of the
very, very best work are virtually peerless, and can exchange their
masterworks amongst themselves, as in Newton's day. So is all this
peer review just to keep the drones busy? I can't say. (I used to
mumble things in reply to the effect that "we need to countenance
the milk in order to be ensured of the cream rising to the top" or
"we need to admit the chaff in order to sift out its quota of wheat"
or "we need to screen the gaussian noise if we want to ensure our
ration of signal")
Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock,
A difficult balance: Peer review in biomedical publication.) Nature
322: 24 - 5.
But I *can* say that it has nothing to do with Open Access (except that
it can be obtruded, along with so many other irrelevant things, to
slow OA's progress). If self-archiving mandates were adopted
universally, all of this would be mooted. The current peer-reviewed
literature, such as it is, would at long-last be OA -- which is the
sole goal of the OA movement, and the end of the road for OA advocacy,
the rest being about scholars and scientists making use of this
newfound bonanza, whilst other processes (digital preservation,
journal reform, copyright reform, peer review reform) proceed apace.
As it is, however, second-guessing the future course of peer review is
still one of the at-least 34 pieces of irrelevance distracting us from
getting round to doing the optimal and inevitable at long, long
last...
Harnad, S. (1990) Scholarly Skywriting and the Prepublication
Continuum of Scientific Inquiry. Psychological Science 1 342 - 343.
http://cogprints.org/1581/
Harnad, S. (1998) The invisible hand of peer review. Nature
[online] (c. 5 Nov. 1998) Exploit Interactive version
http://helix.nature.com/webmatters/invisible/invisible.html
http://www.exploit-lib.org/issue5/peer-review/
Peer Review Reform Hypothesis-Testing (started 1999)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#480
A Note of Caution About "Reforming the System" (2001)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#1170
Self-Selected Vetting vs. Peer Review: Supplement or
Substitute? (2002)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#2341
Peer Review: Streamline It Or Sideline It?
http://www.ecs.soton.ac.uk/~harnad/Temp/peerev.pdf
Hyperlinked version of this posting:
http://openaccess.eprints.org/index.php?/archives/148-guid.html
Stevan Harnad
Received on Sat Oct 21 2006 - 12:54:26 BST