Measuring and maximizing UK research impact


Stevan Harnad

Unlike journalists or book-authors, researchers do not receive royalties or fees for their writings. They write for "research impact", which is the sum of all the effects of their own work on the work of others and on the society that funds it all. Measuring how a piece of research is read, used, cited and built upon in further research and applications is clearly of great importance, not least for the tax-payer.

One natural way to measure research impact would be to adopt the approach of the web search engine Google. Google measures the importance of a website. It does this by rank-ordering search results according to how many other websites link to them: the more links, the higher the rank. This works amazingly well, but is really far too crude for measuring research impact, which is about how much a paper is being used by other researchers. However, there is a cousin of weblinks that researchers have been using for decades as a measure of impact: citations.

Occasionally, one paper cites another just to say it is wrong. But mostly citations are referencing the building blocks that a piece of research is using to make its own contribution to knowledge. The more often a paper is used as a building block, the higher its research impact. Citation counts are powerful measures of impact. In fact, one recent study has shown that in the field of psychology, citation counts predict the outcome of the Research Assessment Exercise (RAE) with an accuracy of more than 80 per cent.

The RAE involves ranking all departments in all universities by their research impact and then funding them accordingly. Yet it does not actually count citations. Instead, it requires universities to spend vast amounts of time and energy to compile massive paper dossiers of all sorts of performance indicators. Then still more time and effort is expended by teams of assessors assessing and ranking all the dossiers.http://news.bbc.co.uk/2/hi/uk_news/education/2944316.stm

In many cases, citation counts alone would save at least 80% of all that time and effort. But the Google-like idea also suggests ways to do even better, enriching citation counts by another measure of impact: how often a paper is read. Web "hits" (downloads) predict citations that will come later. To be used and cited, a paper first has to be accessed and read. And downloads are also usage (and hence impact) measures in their own right.

Google also uses "hubs" and "authorities" to weight link-counts. Not all links are equal. It means more to be linked to by a high-link site than a low-link site. This is the exact equivalent to co-citation analysis, in which it matters more if you are cited by a Nobel laureate than by a fresh post doc.

What this rich new world of webmetrics requires in order to be mined and used to encourage and reward research is not a four-year exercise in paperwork like the present RAE. All university research output should be continuously accessible -- and hence assessable -- online: not only the references cited but the full texts. Then computer programs can be used to extract a whole spectrum of impact indicators, adjustable for any differences between disciplines.

Nor are the time-saving, efficiency, power and richness of these webmetric impact indicators their only or even principal benefits. For the citation counts of papers whose full texts are already freely accessible on the web are over 300% higher than those that are not. So all of UK research stands to increase its impact dramatically by putting it online. Every researcher should have a standardised electronic CV, continuously updated with all the RAE performance indicators listed and every journal paper linked to its full-text in that university's online "eprint" archive. Webmetric assessment engines can do all the rest.

At Southampton we have designed (free) software for creating the RAE CVs and eprint archives, along with citebase, a webmetric engine that analyses citations and downloads. The only thing still needed is a national policy of self-archiving all research output to enhance and assess its impact. http://www.ariadne.ac.uk/issue35/harnad/