Re: UK Research Assessment Exercise (RAE) review

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Fri, 29 Nov 2002 12:48:42 +0000

On Thu, 28 Nov 2002, Barry Mahon wrote:

> This whole argument (OA is better/cheaper/more efficient.....etc and
> misunderstood) runs the risk of becoming like politics and religion
> as subjects for argument, ideology replace reality. Despite all the
> hype, and noise scientists still seem to prefer the well known and well
> understood paths to publishing - at the moment.

I am not quite sure why you describe the promotion of open access as
hype and noise. The Budapest Open Access Initiative
http://www.soros.org/openaccess/ and other active proponents of open
access are trying to help research and researchers; they are not trying
to sell a product.
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2212.html

Perhaps you are thinking about the promotion of open-access journals?
But surely journals have the right to promote themselves, regardless
of their cost-recovery model?

> I would wager that they understand quite well the concepts and
> advantages/disadvantages of OA but so far they consider the tried and
> tested to be as good if not better.

My own experience over a number of years of rather active proselytizing
for open access http://www.ecs.soton.ac.uk/~harnad/talks.htm suggests
that you would lose that wager. Researchers are remarkably,
breath-takingly under-informed and confused on this issue. That is why
the BOAI's main mission is not to invent the means of attaining open
access (the means exist already, have been adundantly tested, and work)
but to inform the research community about the causal connection between
research access and research impact, and then how to go about maximizing
it (through open access). I don't think you will find many researchers
who will say: "I know and understand the causal connection between
maximizing access and maximizing impact, and I am not interested in
(or opposed to) maximizing the impact of my work." Even less will you
find university employers of researchers or government funders of
research who would endorse such conclusions.

So it's a far better bet that the problem is under-informedness and
ill-informedness, rather than rational judgment. Indeed, why the
research community is so slow to realize and and act upon what is,
upon just a little reflection, optimal, reachable, and indeed inevitable
for research and researchers might better be dubbed a "koan" than an
instance of clear understanding!

    Re: The "big koan"
    http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2053.html

    Harnad, S. (1999) Free at Last: The Future of Peer-Reviewed Journals.
    D-Lib Magazine 5(12) December 1999
    http://cogprints.soton.ac.uk/documents/disk0/00/00/16/85/

Unfortunately, Barry, your own further comments indicate how widespread
the misunderstanding still is:

> One of the possible problems of OA is the lack of simple (i.e. easy
> to access/available off a shelf) sources with well known titles and
> an inherent quality perception.

Assuming that by OA you mean open access, it is rather hard to construe
what, exactly, you have in mind above. Open Access, as clearly stated,
for example, in the BOAI statement, means free online access to the
peer-reviewed research literature. Now what on earth can your statement
mean relative to that definition? There are currently 20,000
peer-reviewed journal titles, each with its own inherent quality, and
perceptions of it: 2,000,000 articles annually. Open access is about
making access to those 2,000,000 annual articles free online.

You speak of

    "the lack of simple (i.e. easy to access/available off a shelf)
    sources with well known titles"

How are we even to begin to construe this, so many are the internal
consistencies betraying a rather transparent nonunderstanding of what open
access is about -- the very same kinds of nonunderstanding that prevail
in the rest of the research community:

Lack of easy-to-access titles? But that's precisely the problem to which
open access is the proposed solution. Most titles are not easy to access:
They require toll-access.

"Off the shelf"? Apart from rather biassing the question toward paper
access, what can this possibly mean? A researcher has neither on-paper
nor on-line access to journals for which his institution cannot afford
the access-tolls, and that is what this is all about!

So unless you were suggesting that inaccessibility is evidence of the
absence of a need for accessibility here, I suspect that what you might
have had in mind is also the most common error in the research community's
construal of open access today: Open access is not an alternative to
peer-reviewed journals, it is an alternative means of accessing them.

Two such means have been proposed: BOAI-1 is the authors of those
2,000,000 articles providing open access to them by self-archiving
them, and BOAI-2 is the pubishers of those 20,000 journals providing
open-access to them by conversion to open-access or the creation of
new open-access journals.

If is this very last subset of the several parallel strategies --
the creation of new, open-access titles -- that you were referring to
when you said "lack of well-known titles," then you are right: So far less
than 200 of the existing 20,000 peer-reviewed journals (i.e., < 1%)
are open-access. But that is part of what the open access initiative is
trying to remedy, and BOAI-2 is certainly not the only way. Nothing is
stopping the authors of the remaining 1,980,000 articles that are not yet
in open-access journals from self-archiving them in their institutional
Eprints Archives right now! Why they are not yet doing it is the "big
koan." The BOAI and other open access efforts are intended to resolve
the koan by informing the research community about the access/impact
connection and the ways to maximize it.

Is this the activity you are calling hype? Are you in favor instead of
leaving the research community in its hazy state of uninformedness, on
the assumption that they already know what is best for them?

> The same is true of RAE, in a way, it is perhaps crude but it is simple
> and it fits the understanding of present publication patterns by those
> who advise the governement on such matters as RAE (we must not forget
> that these decisions are taken with the agreement of at least some of
> those who are so assessed).

I couldn't follow this. The point was not to do away with the RAE but to
recommend a way of doing the RAE far more cheaply, efficiently, and, most
important, accurately, by extending its scientometric predictive power.
The proposed method was online submission of standardized RAE-CVs
for each researcher linked to the full-text open-access draft of every
peer-reviewed research publication in the researcher's institution's
Eprint Archive. What has your comment to do with that? The old
predictivity (the journal-impact factors) is preserved, but also greatly
enriched by further scientometric predictors, all at a far smaller cost
in time, effort and resources for all concerned.

> The newer ways of publishing have, like most new ideas, to overcome some
> 'not invented here' like reaction, some competitive jealousy from those
> economically affected and inertia.

Is this just a restatement of the challenges facing BOAI-2? But what
about BOAI-1?

> In addition OA has to prove that the
> writing will be seen by those who matter, including those performing RAE,
> and be easly to find when you are looking for citable material.

A nonsequitur. Peer-reviewed research that is openly accessible will
be seen by potential user with access to the web -- incomparably more
users than those with toll-access today (but including them too, as a
subset). The RAE has from the outset said all peer-reviewed research --
regardless of the medium of the journal -- is eligible for assessment
(though, of course, weighted by the quality, track-record, and impact
factor of the journal).

So, again, what is the point?

> OA will become an accepted part of the research results dissemination
> process, it will be incorporated in whatever sorts of RAEs we will have
> and OA originated material will be identified and quoted like everything
> else.

It already is, and always has been. So this is completely irrelevant.
The point was not about new journals, online-only journals, or
open-access journals. Medium and economic model are already completely
irrelevant to the RAE. Only quality, track-record, and citation impact
count. And they are counted mainly via the journal impact factor.

Open access is both a means of maximizing impact and a means of
generating far richer and more powerful measures for assessing impact.
You seem to have missed this point completely, and to be merely referring
to the non-problem of open-access journals vs. older toll-access journals.
New journals, without track-records, are always at a disadvantage, as
they should be, until their track records are known. So what?

> Do we have to agree that it will replace all the other methods?? In
> my opinion, no, we can discuss that as one scenario, if we wish, but
> let it not become the sine qua non of the discussions.

What is the "it" and what are the "other methods" it may or may not
replace? Are we talking here about open access (via either BOAI-1,
self-archiving, or BOAI-2, open-access journals) replacing toll-access?

(Answer: yes, one hopes it will, open access being both optimal and
inevitable for research and researchers, because it maximizes research
usage and impact by minimizing -- in fact eliminating -- usage/impact
blockage by access-tolls.)

Or are we talking about replacing RAE's univariate impact-factor
measure, and the expensive and inefficient ways in which the data
for it are gathered every 4 years through great effort and time-loss
by researchers, their institutions and their assessors?

(Answer: Online RAE-CV submission coupled with Eprint Archives of
institutional refereed research output certainly sounds like a promising
hypothesis for cutting costs and increasing efficiency and accuracy. It
is also a way to hasten open access worldwide. Is it the only method or
the best one? That remains to be seen.)

Stevan Harnad
Received on Fri Nov 29 2002 - 12:48:42 GMT

This archive was generated by hypermail 2.3.0 : Fri Dec 10 2010 - 19:46:44 GMT