Enhance UK research impact and assessment 

by making the RAE webmetric

Stevan Harnad

Unlike journalists or book-authors, researchers do not receive royalties or fees for their writings. They write for "research impact," which is the sum of all the effects of their own research on other research and researchers, and on the society that funds it all: How much is my research read, used, cited, and built-upon, in further research and applications?

It is for the sake of its impact that research is supported by tax-payers, that researchers are salaried by universities and funded to do research, that prizes and honours are awarded to scholars and scientists, and that universities receive overhead funding and prestige. So how is research impact measured? One natural way resembles the way that Google measures the relevance of a website: Google rank-orders search results by how many other websites link to them: more links, higher rank. This works amazingly well, but it is far too crude for measuring research impact, which is about how much a research paper is being used by other researchers. But there is a cousin of weblinks that researchers have been using for decades as a measure of impact that works much the same way: citations.

It may occasionally happen that one paper cites another paper just to say it's wrong. But mostly citations are referencing the building blocks a paper is using to make its own contribution to knowledge. The more often a paper is used as a building block, the higher its research impact. Citation counts are such powerful measures of impact that in Psychology they predict 80% of the outcome of the UK Research Assessment Exercise (RAE). The RAE is unique to the UK and involves ranking all the departments in all the universities by their research impact and then funding them according to their rank. Yet the RAE does not actually count citations. It requires universities to spend vast amounts of time and energy to compile a massive paper dossier of all sorts of performance indicators. Then still more time and effort is expended by teams of assessors assessing and ranking all the dossiers: http://www.hefce.ac.uk/news/hefce/2003/rareview.htm

In many cases, citation counts alone would have saved at least 80% of all that time and effort. But the Google-like idea also suggests ways to do even better, enriching citation counts by another measure of impact: How often is the paper read? Web "hits" (downloads) predict citations that will come later. To be used and cited, a paper first has to be accessed and read. And downloads are also usage (hence impact) measures in their own right. Google also uses "hubs" and "authorities" to weight link-counts. Not all links are equal. It means more to be linked to by a high-link site than a low-link site. This is exactly equivalent to co-citation analysis, in which it matters more if you are cited by a Nobel laureate than by a fresh PhD. There is also the vast world of co-text analysis, in which combinations of concepts in papers may turn out to be be predictive of impact.

But what this rich new world of webmetrics requires in order to be mined and used to encourage and reward research is not a 4-year exercise in paperwork like the present RAE. All university research output should be continuously accessible and hence assessable: not only the references cited but the full text. Then computer programs can be used to extract a whole spectrum of impact indicators, adjustable for any differences between disciplines.

Nor is time-saving, efficiency, and the power and richness of these webmetric impact indicators their only benefit or even their principal one, for the citation counts of papers whose full texts are freely accessible on the web are over 300% higher. So all of UK research stands to increase its impact dramatically by putting it all online. Every UK researcher should have a standardised online CV, continuously updated with all the RAE performance indicators listed and every journal paper linked to its full-text in that university's online eprint. archive (the bibliography only, for books and data too, for data-based disciplines). Webmetric assessment engines can do all the rest. At Southampton we have designed (free) software for creating the RAE CVs and eprint archives, along with citebase, a webmetric engine that analyses citations and downloads. The only thing still needed is a UK national policy of self-archiving all research output to enhance and assess its impact. http://www.ariadne.ac.uk/issue35/harnad