Dear Rachel, I am afraid we are going around in circles:
It is uncontested that RAE formally states:
(1) You can submit anything
(2) We will evaluate everything
(3) We don't count citations
But the fact is that (1) is untrue, because your department will not
(sensibly, mercifully) let you submit just anything. It won't even
submit you as an active researcher if you can't come up with 4 items that
meet their criteria (which are, in many fields, sensibly: articles in
high-impact refereed journals and conference proceedings).
And the fact is that (2) is not only untrue, but absurd, because RAE
has neither the time nor the expertise to (re)-evaluate everything;
hence the only sensible thing to do (and they do it) is for the RAE
evaluators to rely on the track-record of the journal that peer-reviewed
the submitted article.
And the fact is that although it is true that RAE do not count citations
explicitly (3), both the (sensible and realistic) departmental stress
on high-impact peer-reviewed journal submissions (1) and the necessary
(and sensible) taking-into-account by the RAE evaluators of the track
record of the peer-reviewed journal in which an article appeared (2)
(rather than any absurd and quixotic attempt to re-do peer review in-house
for all of the UK's research output!) result in the many times reported
outcome that the RAE rankings are highly correlated with (hence highly
predictable from) citation rankings.
http://www.ariadne.ac.uk/issue35/harnad/
I have several times promised to bow out of this discussion, but I do find
it mind-boggling how it manages to keep getting bogged down in matters of
superficial form while completely missing matters of substantive content!
I've gone on to re-annotate the water that has already gone under the
bridge in this meandering discussion (below) to try to spare some of
us from just repeating history:
On Mon, 20 Mar 2006, Rachel Bruce wrote:
> Going back to the discussion on RAE rules re publication I have spoken to a
> colleague at HEFCE to get their view:
>
> RAE Guidance on submissions (RAE 03.2005) allows any form of publicly
> available assessable output to be submitted as well as confidential reports
> which may be submitted if the HEI has prior permission that the outputs may
> be made available for assessment.
May be made available. But will they get you a good rank? And will your
department allow you to submit just anything (at the risk of its rank)?
> A proportion of outputs submitted to the
> RAE are not text-based, and hence the RAE refers to outputs rather than
> publications.
Irrelevant to the matter under discussion.
> In the generic statement on criteria and working methods (at the beginning
> of each set of subject based panel criteria), it is clearly stated that "all
> forms of research output will be assessed on a fair and equal basis.
A pious platitude, with no substance, one way or the other.
> Sub-panels will neither rank outputs, nor regard any particular form of
> output as of greater of lesser quality than another per se.
Utter, abject nonsense, immediately contradicted by the next sentence.
> Some panels may
> specify in their criteria that where they do not examine an output in detail
> , they may use, as one measure of quality, evidence that the output has
> already been reviewed or refereed by experts (who may include users of
> research) and has been judged to embody research of high quality.
An otiose and long-winded way of saying "please ignore what I just
said in the previous sentence."
> no panel
> will use journal impact factors as a proxy measure for assessing quality."
> (paragraph 32 of the Generic statement (RAE 01.2006)).
"And just in case that has not confused you sufficiently, I shall now
contradict it again."
(In taking the journal's name and quality track-record into account,
you are implicitly influenced by its impact factor, whether or not you
do an explicit calculation -- which the RAE does not do. [Btw, if it had
any sense, the RAE *would* count submitted research/researcher citations
explicitly -- not journal impact factors, which are blunt instruments,
being merely the average for the journal: That way RAE would realise how
much it is already relying on citation impact implicitly, and it might
at also scrap a lot of useless make-work and time-wasting on the part of
both the submitters and the evaluators!)
> From: Morag Greig, 16 March 2006 12:43
>
> At Glasgow we found that particularly in the initial stages of setting up
> our repository some academics were under the impression that we were asking
> them to stop publishing in peer reviewed journals and to simply make their
> research available in the repository.
Solution: Correct the impression: They are being asked to make their
published, peer-reviewed research available in the repository.
> This lead to concerns that such a move
> would have an adverse impact on performance in the RAE, as there seemed to
> be a fairly widespread belief that the RAE panels would consider such
> 'publications' of low value.
The widespread belief is correct, but since no one is proposing
to submit unpublished work to RAE, but simply to deposit published work
in the IR (for submission to the RAE), nothing is at issue.
> The issue of making publications being submitted to the RAE available in a
> repository is different, as here I assume you would be talking about
> material that had been published in a peer-reviewed journal, conference
> proceeding etc..
Both published and unpublished work can be deposited in an IR, but
published, peer-reviewed journal articles are the primary, high-priority
target.
Submitting to the RAE via IRs is a separate matter, but there too,
published, peer-reviewed journal articles are the primary target.
> From: John Murtagh, 16 March 2006 10:32
>
> Interesting debate on the Research Assessment Exercise and IRs.
> From the responses so far I tend to agree with John Smith at Kent that
> unpublished material is (or will in the future) be as much worth to
> research output for an institution as anything else, as long as it is
> used and cited. That this shouldn't necessarily be a so-called vanity
> project.
This is an unrealistic fallacy. Vide supra.
> "...once we move away from the concept of 'impact factors' and use
> individual article citation analysis instead,
So far, so good, since individual citation counts are a sharper
instrument than journal averages (though the journal deserves a
separate component in the regression equation too).
> peer review will merely
> be the first stage (and not necessarily the most important) in assessing
> the real value of someone's work."
Nonsense in the case of most work -- though no doubt some values (or
errors) will come out in the post-peer-review wash too, in some cases,
individual peer judgement, be it ever so expert, being fallible too...
> I was interested to know from Morag Greig's response about how the RAE
> factored into the "confusion" of academics who were unwilling to
> deposit into the IR because of it not being a substitute for
> peer-review.
Please, let us now dispel this confusion, rather than compounding it.
> Could Morag describe this a little more? I would like to use the RAE
> exercise as an inducement, but it sounds like a deterrent from
> depositing (in the gathering of materials at least).
RAE is an inducement to depositing in an IR: It's not only a more
efficient way to submit to RAE, but self-archiving (early!) raises
research impact, hence RAE rating.
http://opcit.eprints.org/oacitation-biblio.html
> From: Morag Greig, Wed, 15 Mar 2006
> Subject: Re: Generic Rationale and Model for University Open Access
> Mandate
>
> It's worth pointing out that many repositories have gone to some
> lengths to explain to authors that repositories are (in most cases) NOT a
> substitute for publication in a peer reviewed journal (whether that be an OA
> journal or a subscription journal). Confusion over this issue has lead to many
> academics being unwilling to deposit in their institutional repositories.
Hear, hear!
> From: Leslie Carr
>
> Whether articles are disseminated by an institutional repository is
> irrelevant to RAE assessment.
>
> It is whether they have been published by a journal (or conference or
> whatever the assessment panel uses as a criterion) that matters.
>
> In other words, appearing in an IR does not make an article
> "published" in the eyes of the RAE.
Spot-on (for those with eyes to see and ears to hear!).
> On 15 Mar 2006, at 09:50, Wolfgang Greller wrote:
>
> Can anyone tell me whether articles published in an institutional e-
> print repository count as publication in UK RAE terms?
No it doesn't.
> From: "C.Oppenheim"
>
> I don't agree with Les.
>
> Panels are instructed to consider *any* sort of published output, and
> this would consider something that had only ever appeared on an IR, or
> on a web page; indeed, statistics from the last RAE show that a small
> proportion of the items examined by the panels were in web page form
> only. However, if you want to impress the RAE panel, the item should
> have been through a proper refereeing process.
Charles later conceded that he was speaking about RAE's formal criteria,
not actual submission practice, let alone successful/optimal practice
(in terms of RAE outcome), by departments.
> From: Stevan Harnad
>
> All this means is that RAE also considers unpublished papers. It has
> nothing whatsoever to do with whether or not those papers were self-
> archived on the web. (Plus, as we know, and as many studies by
> Charles himself has shown, the RAE outcome is highly correlated with
> the citation counts of the journal articles submitted; and
> Departments place high -- too high -- emphasis on the "citation
> impact factor" of the journal in which the 4 articles selected for
> submission were published.)
>
> So although the RAE nominally considers all papers, and it definitely
>
> does not count citations directly, there is considerable emphasis on
> journal articles on both sides. Moreover, I profoundly doubt that
> many RAE submissions actually get read (let alone read by someone as
> competent to evaluate them as the referees of the journal in which
> the originally appeared). So the fate of an unpublished manuscript
> amounts to the fate of a book judged by its cover (without even the
> help of its publisher and his prior track record for quality)...
Amen
Stevan Harnad
Received on Mon Mar 20 2006 - 12:22:37 GMT