On Wed, 27 Nov 2002, Jan Velterop wrote:
> I meant to give an example of a complement to quantification.
Signed open secondary reviews are certainly a complement to both
scientometric measures and primary (peer) reviews. All direct human
judgments are. But they are also countable, content-analyzable, comparable
against other data, including the track-record of the reviewer's name,
hence amenable to scientometrics.
By the way, primary peer reviews are not usually signed by the referees'
names, but they are always signed by the journal-name. Hence the journal
and its editor are openly accountable for the quality of the papers it
accepts (and, indirectly, for those it rejects too!). That is why the
journal-name and track-record are such important indicators, both for
scientometric assessment and for navigation by the would-be user trying
to decide what is worth reading and safe to try to build upon.
> Much of the trouble is not quantification per se, but the lack of
> information to enable weighting the votes.
To a great extent scientometrics is about finding the proper weightings
for those votes!
> The journals (well, at least some of them) lend a certain weight to
> their peer-review, but this peer-review is almost always anonymous.
Journal quality varies, both within journals (owing to human
fallibility) and between journals (owing to systematic differences in
peer-review standards and hence quality). The journal, however, is never
anonymous. Its reputation is answerable to the degree to which it
improves article quality through peer review, and the quality
selectivity it exercises.
I will not rehearse here the long, old list of arguments for and against
referee anonymity. The primary argument against referee anonymity is
answerability (to ensure qualifications, minimize bias, etc.). The
primary argument for anonymity is freedom (to exercise judgment without
risk of counter-bias, e.g., when a junior researcher is reviewing the
work of a senior researcher). Referee anonymity is normally offered as
an option which some referees choose to exercise and some do not,
depending on the referee and the circumstances. But the real protection
against bias is supposed to be the editor (to whom the referee certainly
is not anonymous) and the reputation of the journal. A biassed choice of
referees will generate biassed referee reports and biassed journal
contents. That is a matter of public record. The remedy is either to
replace the editor or to switch to a rival journal.
But this is all on the topic of peer review reform, which is not the
focus of this Forum. This Forum is concerned with freeing the current
peer-reviewed research literature (20,000 peer-reviewed journals) from
access-tolls, not about freeing it from, or modifying, peer review. That
second agenda will first require some empirical testing and comparison,
which has not yet been done, to my knowledge. To put it another way:
the alternative to toll-access, namely, open-access, has been tried,
tested, shown to work, and shown to be far more beneficial to research
and researchers. The alternatives to peer-review have not (yet) been.
The present RAE assessment/impact thread is about ways to accelerate the
transition to open access by SUPPLEMENTING classical peer review with
rich new scientometric measures of impact that are co-evolving with an
open-access database. It is not about substitutes or reforms for classical
peer review. Those are another (worthy) matter, for another forum.
"Peer Review Reform Hypothesis-Testing"
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/0479.html
> Reviewers may not even be proper 'peers' in some cases.
Yes, occasionally some conscientious editors err in their choice of
referees, or in their evaluation of their reports. Some human error is
inevitable (even by the most peerless of peers), but one hopes that when
the error is systematic (i.e., bias or incompetence) the open,
answerable dimension of the system -- namely, the journal's and editor's
names and reputations -- will help expose, control and correct such errors.
> Stevan speculates that "Perhaps reviewer-names could accrue some
> objective scientometric weight...". I would perhaps remove the 'perhaps'.
Note that I was speaking of secondary, open reviewers, in review journals
or in open peer commentary or in ratings, all appearing after the article
has been published. Those are all valuable supplements to the current
system. But I was certainly not recommending abandoning the option
of referee anonymity in primary peer review (until the logic and
empirical consequences of such a change are analyzed and tested
thoroughly) -- although untested recommendations along those lines have
been made by others (including some bearing the same surname as myself!
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/0303.html ).
> Maybe it has its own set of problems, but disclosing the peers' identity may
> be a great help in assessing the weight or significance of the review.
And perhaps a great hindrance in getting some peers to review at all,
under a variety of conditions.
(There have been similar -- unresolved -- back-and-forths about
author-anonymization. Characteristically, decisions were taken without
prior empirical testing, on a-priori ideological or conceptual grounds. I
have not followed the outcomes, but to my knowledge classical peer
review has tended to be reverted to after these sorties, and pretty much
proceeds apace, with optional referee anonymity to the author and author
non-anonymity to the referee remaining the norm.)
> Besides, it may disclose possible conflicts of interest. All BMC's medical
> journals have open peer review which works most satisfactorily.
That is interesting to know, and will need to be evaluated and compared
with suitable control-alternatives after a few years (and once any
"hawthorn effect" has dissipated). It is not ready to be recommended
for wider adoption yet: BMC is very much a conscious experiment by the
self-selected sample of authors and referees who have collaborated so
far. Any generalizations will require more time, and a larger sample.
> All journals also have a comments section enabling a public, open
> discussion.
This supplement (as opposed to substitute) already has a long
tried-and-true history (including an open-peer-commentary journal
I myself edited for 25 years,
http://www.bbsonline.org/ , and
for over a decade now, an open-access online-only journal
too
http://psycprints.ecs.soton.ac.uk/ both modeled on a
still longer-standing journal, going for over four decades now:
http://www.journals.uchicago.edu/CA/home.html )
> The point of Faculty of 1000 is that an open, secondary review of published
> literature by acknowledged leaders in the field, signed by the reviewer, is
> seen by increasing numbers of researchers (measured by the fast-growing
> usage figures of F1000) as a very meaningful addition to quantitative data
> and a way to sort and rank articles in order of importance.
I agree completely. Open peer commentary is an extremely valuable
supplement to classical peer review:
Harnad, S. (1979) Creative disagreement. The Sciences 19: 18 - 20.
Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study
in scientific quality control, New York: Cambridge University Press.
Harnad, Stevan (1985) Rational disagreement in peer
review. Science, Technology and Human Values, 10 p.55-62.
http://cogprints.soton.ac.uk/documents/disk0/00/00/21/28/
Harnad, S. (1997) Learned Inquiry and the Net: The Role
of Peer Review, Peer Commentary and Copyright. Learned
Publishing 11(4) 283-292.
http://cogprints.soton.ac.uk/documents/disk0/00/00/16/94/
Harnad, S. (1998/2000) The invisible hand of peer review. Nature
[online] (5 Nov. 1998) & Exploit Interactive 5 (2000):
http://cogprints.soton.ac.uk/documents/disk0/00/00/16/46/
> Of course one can subsequently quantify such qualitative information. But
> what a known and acknowledged authority thinks of an article is to many
> more interesting than what anonymous peer-reviewers think. Any research
> assessment exercise should seriously look at resources such as offered
> by Faculty of 1000.
Let 1000 flowers bloom. But it's rather mis-stating the options to
describe them as open-review vs. anonymous-review! Classical peer review
is one thing. Then there is post-hoc open-review thereafter. And then
there is the scientometric quantification of all of this through usage,
citation, and content analysis -- including the status, weightings,
and predictiveness of "authorities", among both authors and journals.
Let 1000 flowers bloom -- but let it be in an open-access field, as soon
as possible!
Stevan Harnad
NOTE: A complete archive of the ongoing discussion of providing open
access to the peer-reviewed research literature online is available at
the American Scientist September Forum (98 & 99 & 00 & 01 & 02):
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html
or
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/index.html
Discussion can be posted to: american-scientist-open-access-forum_at_amsci.org
See also the Budapest Open Access Initiative:
http://www.soros.org/openaccess
the Free Online Scholarship Movement:
http://www.earlham.edu/~peters/fos/timeline.htm
the OAI site:
http://www.openarchives.org
and the free OAI institutional archiving software site:
http://www.eprints.org/
Received on Wed Nov 27 2002 - 13:06:40 GMT