The UK RAE is planning to scrap the time-consuming and costly panel
re-review of already-peer-reviewed articles in favour of metrics because
metrics have been shown to correlate highly with the RAE panel rankings
anyway (although it is not yet decided what combination of metrics will be
appropriate to each discipline).
Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated
online RAE CVs Linked to University Eprint Archives: Improving
the UK Research Assessment Exercise whilst making it cheaper and
easier. Ariadne 35 (April 2003).
http://www.ariadne.ac.uk/issue35/harnad/
Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open
Research Web: A Preview of the Optimal and the Inevitable, in Jacobs,
N., Eds. Open Access: Key Strategic, Technical and Economic Aspects,
chapter 20. Chandos.
http://eprints.ecs.soton.ac.uk/12453/
I trust that Australia's RQF is not going to mechanically recapitulate
the many years that the RAE wasted of its researchers' time submitting for
and performing panel re-review. RQF plans are probably just a bit out of
phase right now, and Australia will catch up in time for its first RQF
exercise or soon thereafter. By then the arbitrary constraint of submitting
only 4 papers will also be mooted by Open Access submission of all research
output self-archived in each instution's Institutional Repository.
See remarks below.
On Fri, 17 Nov 2006, Linda Butler wrote:
> Many of Arthur Sale's points about
> the Australian RQF, particularly in relation to IRs and the way in which
panels
> will access submitted publications, are accurate. However, his "definition" of
> quality and impact in the RQF context is seriously misleading. Yes, the terms
> are used in an unusual way, but his attempt to paraphrase the meaning is way
> off. The definitions contained in the official document are:
>
> the quality of original research including its intrinsic merit and academic
> impact. Academic impact relates to the recognition of the originality of
> research by peers and its impact on the development of the same or related
> discipline areas within the community of peers;
That, presumably, is what journal peer review has already done for a
researcher's published papers. Journals differ in their peer-review
quality standards, but that too can be triangulated via metrics. The
best of journals will have refereed their content by consulting the
top experts in each subspecialty, worldwide, not an assembled panel of
national representatives to the discipline from the UK or AUSTRALIA,
re-reviewing all content in their discipline.
The fact that the RAE panels (having wasted the researchers' time
and their own in re-reviewing already peer-reviewed publications)
nevertheless come up with rankings that agree substantially with metrics --
with prior funding counts, regrettably, because those probably explicitly
influenced their rankings, but also with citation counts, which they
are explicitly forbidden to consult, hence showing that human judgment
in skimming and ranking the 4 papers per researcher averages out to the
same outcome as the human judgment involved in deciding what to cite --
is another indication that the panel review is superfluous in most of
the disciplines tested so far. For some disciplines new combinations
of metrics will no doubt have to be tested and validated, and that is
partly the reason the next RAE will be a parallel panel/metric exercise,
to cross-validate the metric rankings with the panel rankings.
> the impact or use of original research outside the peer community that will
> typically not be reported in traditional peer reviewed literature (that is,
> the extent to which research is successfully applied during the assessment
> period for the RQF).
Sounds like another candidate metric...
> Broader impact relates to the recognition by qualified end users that
> methodologically sound and rigorous research has been successfully applied
> to achieve social, economic, environmental and/or cultural outcomes.
Sounds again like peer review, in the case of peer-reviewed publication. For
unpublished research, other metrics (patents, downloads) are possible. There
may be a few specialties in which human evaluation is the only option, but
that tail should certainly not be allowed to wag the RQF dog: Specific
exceptions can simply be made for those specialities, until and unless a
valid combination of metrics is found.
> Quality is NOT a solely metrics-based exercise.It is the peer assessment of 4
> outputs per active researcher (as in the RAE), informed by quantitative
> indicators supplied to the panel (citations, competitive grants, ranked
outputs
> - details of proposed measures are on the DEST website in the background
> papers).
Quality has already been peer-reviewed for peer-reviewed publications, hence
panel re-review is not recourse to metrics instead of human judgment, it is
an exercise in (blunt) redundancy (in most cases).
> Impact, the most difficult to assess, is judged from an "evidence-based
> statement of claims". Obviously, there is a lot of detail behind that
> statement - again, background papers are available on the DEST website. It
> will definitely not be judged in the way outlined below.
"Evidence-based statement of claims": Sounds like a tall order for a
small panel of national peers hand-re-reviewing a set of mostly already
peer-reviewed papers, and a time-consuming one. Let's hope the RQF will
learn from the RAE's long, costly and wasteful history, rather than just
repeating it. The growing body of Open Access scientometrics that will
become available in coming years will make it possible for enterprising
data-miners (possibly PGs of the same researchers that are wasting
their research time submitting to and performing the panel reviews) to
demonstrate prominently just how redundant the panel rankings really are.
Pertinent Prior American Scientist Open Access Forum Topic Threads:
UK "RAE" Evaluations (began Nov 2000)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#1018
Scientometric OAI Search Engines (began Aug 2002)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#2238
Australia stirs on metrics (June 2006)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/5417.html
Big Brother and Digitometrics (began May 2001)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#1298
UK Research Assessment Exercise (RAE) review (began Oct 2002)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#2326
Need for systematic scientometric analyses of open-access
data (began Dec 2002)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#2522
Potential Metric Abuses (and their Potential Metric
Antidotes) (began Jan 2003)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#2643
Future UK RAEs to be Metrics-Based (began Mar 2006)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#5251
Let 1000 RAE Metric Flowers Bloom: Avoid Matthew Effect as
Self-Fulfilling Prophecy (Jun 2006)
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/5418.html
Stevan Harnad
> Linda Butler
> Research Evaluation and Policy Project
> The Australian National University
>
> At 03:34 PM 17/11/2006, you wrote:
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> ---------- Forwarded message ----------
> Date: Fri, 17 Nov 2006 14:44:39 +1100
> From: Arthur Sale <ahjs_at_ozemail.com.au>
> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>
> The Australian Government has released a definitive, if incomplete,
> description of Australia's Research Quality Framework (RQF) which
> is our
> equivalent of the UK's RAE. If familiar with the RAE, you will
> recognize the
> family resemblance. I extract the essentials of the RQF for an
> international
> readership, and analyze some of the consequences likely to flow
> from it. To
> see the documentation, see
>
http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_i
> ssues/research_quality_framework/rqf_development_2006.htm.
>
> ESSENTIAL POINTS
>
> 1. The first RQF assessment will be based on submissions by
> the 38
> Australian universities by 30 April 2008. Funding based on the
> assessment
> will flow in calendar year 2009. Six years will elapse before the
> next
> assessment (ie 2014), but there is provision to shorten this.
>
> 2. The Unit of Assessment is the Research Group. Research
> Groups will
> be defined by up to three RFCD four-digit codes (to allow for
> multi-disciplinary groups). The RFCD classification is uniquely
> Australian,
> and for example there are six four-digit codes in the field of ICT.
> Engineering has more but for example Civil Engineering is one. If
> you are
> interested in the codes see
> http://www.research.utas.edu.au/publications/docs/14_rfcd.doc, the
> four
> digit codes are the sub-headings.
>
> 3. Each Research Group will be allocated to and assessed by
> one of 13
> Panels. The Panel is determined by the primary RFCD code. Thus
> Mathematics,
> Computing and Information Technology is Panel 4.
>
> 4. Each University will submit an Evidence Portfolio (EP) for
> each
> identified Research Group. There is provision for cross-university
> Research
> Groups.
>
> 5. The ratings will be based on Quality and Impact separately.
> These
> words have peculiar (ie not common-usage) meanings. Approximately,
> Quality
> is a bag of quantifiable metrics, and Impact is all the soft things
> like
> Fellowships of Academies, Honors, journal associate editorships,
> etc. The
> relative importance of Quality and Impact will vary by Panel and is
> similarly not yet resolved. Quality is based on the best four
> publications
> (Research Output) of each researcher in the group over the six
> years
> 2002-2007, on a full list of all Research Output from the group
> including
> honorary and emeritus professors, and on competitive grants
> received over
> the period. Impact is covered in the Context Statement of the EP
>
> 6. Impact for each Research Group will be assessed on a scale
> of 1 (not
> important) to 5 (prestigious)..
>
> 7. Impact is rated A (outstanding) to E (poor).
>
> 8. Research Groups which rate below 2 for Quality, or below D
> for
> Impact, will attract no funding to their university, though the two
> factors
> are separately aggregated for the University. The weighting of
> funding is
> stated to be linear with rating, but the gradient will be
> determined during
> 2007.
>
> 9. The Panels require access to the electronic versions of any
> of the
> Research Output within four working days. The Panels will (a) rank
> the
> outputs by things like journal impact factors, journal standing,
> etc, (b)
> assess citation counts, both in aggregate and by the percentage
> that fall in
> the top decile for the discipline, and (c) competitive grant
> income.
>
> 10. The RQF is based on a semi-centralized IT model (or
> semi-decentralized). In other words, the full-texts of the research
> outputs
> (publications) will be held in IRs in each university, while the
> RQF
> secretariat will run a repository with all the EPs and develop the
> citation
> counts independent of the universities (in conjunction with Thomson
> Scientific and possibly EndNote Web). The Australian Government
> will be
> approached for funds to universities to establish these IRs.
>
> ANALYSIS FOR OPEN ACCESS
>
> * The RQF will actually use citation metrics in the
> assessment, not
> just test them as a "shadow exercise" as in the next RAE. This will
> mean
> that the OA citation advantage will suddenly look very attractive
> to
> Australian universities, though it is a bit late to do anything
> about it
> five years into a six-year window. However, with 2014 in mind,
> there will be
> pressure to increase citations.
>
> * Every university will have to have an IR to hold the
> full-text of
> Research Outputs. About half already do, with EPrints and DSpace
> being the
> most popular software with a few Fedora-based repositories and
> outsourced
> ProQuest hosts. There will be funding to establish repositories.
>
> * I expect a mad scramble in the smaller universities, with
> outsourcing and hosting solutions being very attractive. Money
> fixes
> everything. The ones that have been dithering will regret it.
>
> * All Research Output generated by all Research Groups will
> have to
> be in the IRs for the RQF. This may amount to 50% of the university
> research
> production over six years, or more or less depending on how
> research
> intensive it is. There are two corollaries: (a) this is Mandate by
> Money,
> and (b) there will be frantic activity over 2007 to put in the
> backlog of
> 2002-2006 publications.
>
> * Since one does not know what Research Output will be
> needed in
> 2014, and only a general clue in 2007, 100% institutional mandates
> are
> likely to spring up all over the place, in the form of Mandate by
> Administration. What I mean by this is that the deposition of the
> paper will
> be integrated with the already present administrative annual
> requirement to
> report the publication to the Australian Government.
>
> * Although it is nowhere stated explicitly that I can see, I
> read
> between the lines that the RQF may be expecting to get access to
> the
> publisher's pdf. This means that it will have to be in the
> repository as
> "restricted access" in most cases or as a link to an OA source.
> There is no
> reason why the OA postprint cannot be there as "open access" as
> well, of
> course, and if a citation advantage is to be got, it will need to
> be.
>
> Please feel free to blog this or forward this to anyone you think
> may be
> interested. My apologies for cross-posting.
>
> Arthur Sale
> Professor of Computing (Research)
> University of Tasmania
>
> Linda Butler
> Research Evaluation and Policy Project
> Research School of Social Sciences
> The Australian National University
> ACT 0200 Australia
> Tel: 61 2 61252154 Fax: 61 2 61259767
> http://repp.anu.edu.au
Received on Fri Nov 17 2006 - 17:14:14 GMT