Re: Australia's RQF

From: Linda Butler <linda.butler_at_ANU.EDU.AU>
Date: Fri, 17 Nov 2006 17:35:09 +1100

about the Australian RQF, particularly in relation to IRs and the way in
which panels will access submitted publications, are accurate. However,
his "definition" of quality and impact in the RQF context is seriously
misleading. Yes, the terms are used in an unusual way, but his attempt
to paraphrase the meaning is way off. The definitions contained in the
official document are:

^ the quality of original research including its intrinsic merit and
academic impact. Academic
impact relates to the recognition of the originality of research by peers
and its impact on the
development of the same or related discipline areas within the community
of peers; and
^ the impact or use of original research outside the peer community that
will typically not be
reported in traditional peer reviewed literature (that is, the extent to
which research is
successfully applied during the assessment period for the RQF). Broader
impact relates to the
recognition by qualified end users that methodologically sound and
rigorous research has
been successfully applied to achieve social, economic, environmental
and/or cultural
outcomes.

Quality is NOT a solely metrics-based exercise. It is the peer
assessment of 4 outputs per active researcher (as in the RAE), informed
by quantitative indicators supplied to the panel (citations, competitive
grants, ranked outputs - details of proposed measures are on the DEST
website in the background papers).

Impact, the most difficult to assess, is judged from an "evidence-based
statement of claims". Obviously, there is a lot of detail behind that
statement - again, background papers are available on the DEST website.
It will definitely not be judged in the way outlined below.

Linda Butler
Research Evaluation and Policy Project
The Australian National University

At 03:34 PM 17/11/2006, you wrote:
      Adminstrative info for SIGMETRICS (for example unsubscribe):
      http://web.utk.edu/~gwhitney/sigmetrics.html

      ---------- Forwarded message ----------
      Date: Fri, 17 Nov 2006 14:44:39 +1100
      From: Arthur Sale <ahjs_at_ozemail.com.au>
      To:
      AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG

      The Australian Government has released a definitive, if
      incomplete,
      description of Australia's Research Quality Framework (RQF)
      which is our
      equivalent of the UK's RAE. If familiar with the RAE, you
      will recognize the
      family resemblance. I extract the essentials of the RQF for
      an international
      readership, and analyze some of the consequences likely to
      flow from it. To
      see the documentation, see
      http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_i
      ssues/research_quality_framework/rqf_development_2006.htm.

      ESSENTIAL POINTS

      1. The first RQF assessment will be based on submissions
      by the 38
      Australian universities by 30 April 2008. Funding based on
      the assessment
      will flow in calendar year 2009. Six years will elapse before
      the next
      assessment (ie 2014), but there is provision to shorten this.

      2. The Unit of Assessment is the Research Group.
      Research Groups will
      be defined by up to three RFCD four-digit codes (to allow for
      multi-disciplinary groups). The RFCD classification is
      uniquely Australian,
      and for example there are six four-digit codes in the field
      of ICT.
      Engineering has more but for example Civil Engineering is
      one. If you are
      interested in the codes see
      http://www.research.utas.edu.au/publications/docs/14_rfcd.doc,
      the four
      digit codes are the sub-headings.

      3. Each Research Group will be allocated to and assessed
      by one of 13
      Panels. The Panel is determined by the primary RFCD code.
      Thus Mathematics,
      Computing and Information Technology is Panel 4.

      4. Each University will submit an Evidence Portfolio
      (EP) for each
      identified Research Group. There is provision for
      cross-university Research
      Groups.

      5. The ratings will be based on Quality and Impact
      separately. These
      words have peculiar (ie not common-usage) meanings.
      Approximately, Quality
      is a bag of quantifiable metrics, and Impact is all the soft
      things like
      Fellowships of Academies, Honors, journal associate
      editorships, etc. The
      relative importance of Quality and Impact will vary by Panel
      and is
      similarly not yet resolved. Quality is based on the best four
      publications
      (Research Output) of each researcher in the group over the
      six years
      2002-2007, on a full list of all Research Output from the
      group including
      honorary and emeritus professors, and on competitive grants
      received over
      the period. Impact is covered in the Context Statement of the
      EP

      6. Impact for each Research Group will be assessed on a
      scale of 1 (not
      important) to 5 (prestigious)..

      7. Impact is rated A (outstanding) to E (poor).

      8. Research Groups which rate below 2 for Quality, or
      below D for
      Impact, will attract no funding to their university, though
      the two factors
      are separately aggregated for the University. The weighting
      of funding is
      stated to be linear with rating, but the gradient will be
      determined during
      2007.

      9. The Panels require access to the electronic versions
      of any of the
      Research Output within four working days. The Panels will (a)
      rank the
      outputs by things like journal impact factors, journal
      standing, etc, (b)
      assess citation counts, both in aggregate and by the
      percentage that fall in
      the top decile for the discipline, and (c) competitive grant
      income.

      10. The RQF is based on a semi-centralized IT model (or
      semi-decentralized). In other words, the full-texts of the
      research outputs
      (publications) will be held in IRs in each university, while
      the RQF
      secretariat will run a repository with all the EPs and
      develop the citation
      counts independent of the universities (in conjunction with
      Thomson
      Scientific and possibly EndNote Web). The Australian
      Government will be
      approached for funds to universities to establish these IRs.

      ANALYSIS FOR OPEN ACCESS

      * The RQF will actually use citation metrics in the
      assessment, not
      just test them as a "shadow exercise" as in the next RAE.
      This will mean
      that the OA citation advantage will suddenly look very
      attractive to
      Australian universities, though it is a bit late to do
      anything about it
      five years into a six-year window. However, with 2014 in
      mind, there will be
      pressure to increase citations.

      * Every university will have to have an IR to hold the
      full-text of
      Research Outputs. About half already do, with EPrints and
      DSpace being the
      most popular software with a few Fedora-based repositories
      and outsourced
      ProQuest hosts. There will be funding to establish
      repositories.

      * I expect a mad scramble in the smaller universities,
      with
      outsourcing and hosting solutions being very attractive.
      Money fixes
      everything. The ones that have been dithering will regret it.

      * All Research Output generated by all Research Groups
      will have to
      be in the IRs for the RQF. This may amount to 50% of the
      university research
      production over six years, or more or less depending on how
      research
      intensive it is. There are two corollaries: (a) this is
      Mandate by Money,
      and (b) there will be frantic activity over 2007 to put in
      the backlog of
      2002-2006 publications.

      * Since one does not know what Research Output will be
      needed in
      2014, and only a general clue in 2007, 100% institutional
      mandates are
      likely to spring up all over the place, in the form of
      Mandate by
      Administration. What I mean by this is that the deposition of
      the paper will
      be integrated with the already present administrative annual
      requirement to
      report the publication to the Australian Government.

      * Although it is nowhere stated explicitly that I can
      see, I read
      between the lines that the RQF may be expecting to get access
      to the
      publisher's pdf. This means that it will have to be in the
      repository as
      "restricted access" in most cases or as a link to an OA
      source. There is no
      reason why the OA postprint cannot be there as "open access"
      as well, of
      course, and if a citation advantage is to be got, it will
      need to be.

      Please feel free to blog this or forward this to anyone you
      think may be
      interested. My apologies for cross-posting.

      Arthur Sale
      Professor of Computing (Research)
      University of Tasmania

Linda Butler
Research Evaluation and Policy Project
Research School of Social Sciences
The Australian National University
ACT 0200 Australia
Tel: 61 2 61252154 Fax: 61 2 61259767
http://repp.anu.edu.au
Received on Fri Nov 17 2006 - 11:45:28 GMT

This archive was generated by hypermail 2.3.0 : Fri Dec 10 2010 - 19:48:35 GMT