Comparing the impact of Open Access (OA) vs. non-OA articles in the same journals

Stevan Harnad

Abstract

The way to test the impact advantage of Open Access (OA) is not to compare the citation impact factors of OA and non-OA journals but to compare the citation counts of individual OA and non-OA articles appearing in the same (non-OA) journals. Such ongoing comparisons are revealing dramatic citation advantages for OA.

A recent Institute for Scientific Information  (ISI) study has reported that traditional journals and Open Access (OA) journals have similar citation impact factors (Pringle 2004). The ISI's press release announced:

"Of the 8,700 selected journals currently covered in Web of Science, 191 are OA journals... [A study on] whether OA journals perform differently from other journals in their respective fields [found] no discernible difference in terms of citation impact or frequency with which the journal is cited." http://www.isinet.com/oaj

It is certainly welcome news that there are no impact differences between the 191 OA journals and the 8509 non-OA journals indexed by ISI, equating for comparable journals as closely as possible: http://www.isinet.com/oaj.  This proves that the skeptics who thought OA journals would be of lower quality or impact and would not be indexed by ISI were wrong (at least for those OA journals): OA journals are indexed by ISI, and they do have comparable citation impacts. But obviously this methodology has a certain element of circularity!

To get a realistic estimate of the effect of OA on impact, it is not enough to compare only the 2% of ISI journals that are OA journals with the 98% that are not, to find that they are equal in impact (for this may well be comparing apples with oranges, even if you equate for subject matter). 

What further needs to be compared is:

(1) the citation impact of the much higher percentage (perhaps as high as 20-40% according to Swan & Brown's (2004) sample) of articles from the 98% non-OA journals that have been made OA by their authors (by self-archiving them)

with

(2) the citation impact of articles from those very same journals and issues that have not been made OA by their authors.

As the founder of citation analysis as well as the ISI, Gene Garfield (1998) has often stressed, the article (and author) citation counts need to be analyzed, not just the average citation counts of the journals in which the article appears.

What this kind of analysis is beginning to reveal in the OA era is that there is indeed a discernible difference in terms of the frequency with which the article is cited: there is a dramatic advantage in favor of the articles that their authors have made OA (Lawrence 2001; Kurtz 2004; Brody et al. 2004). Results are only available for computer science, astronomy, and physics so far, but all other disciplines are currently being analyzed.

The earlier Lawrence (2001) study on the impact enhancing effects of OA in computer science needed to be replicated in other fields to check whether it was merely an artifact of the fact that computer science is conference- rather than journal-based, and whether the advantage really reflected OA vs. non-OA rather than just online access vs. paper access. But, thanks to the ISI database licensed to the Observatoire des Sciences et des Technologies (OST) and a special contract generously provided by ISI to conduct the study, we are in the process of testing the Lawrence effect across all disciplines in a 10-year ISI sample of 14 million articles. The physics analyses up to 2001 are already completed (Brody et al. 2004), and they reveal even larger effects than those reported by Lawrence, with OA/non-OA citation ratios of 2.5 - 5.8. All indications are that the 2002 data will raise them even further, as the biggest effects occur within the first 3 years of publication in scientific disciplines (and both OA and the awareness and visibility of OA articles are also increasing yearly).

Access is not a sufficient condition for citation, but it is a necessary one. Because Open Access (OA) dramatically increases the number of potential users of any given article by adding those who would otherwise have been unable to use it because their institution could not afford the access-tolls of the journal in which it appeared, it stands to reason that OA can only increase both usage and impact. The ratio of "reads" to "cites" will no doubt vary by field. Kurtz (2004) and co-workers report it as 17:1 and even 12:1 in astrophysics. Odlyzko (2002) predicts analogous trends in mathematics. Tim Brody's remarkable correlator/predictor gives the size of the download/citation correlation by field, and can be used to predict citation 6-24 months later from downloads today (with an adjustable time-window): http://citebase.eprints.org/analysis/correlation.php

The proportion of articles for which their authors provide OA is likely to increase dramatically now, in part because of the mounting evidence for the impact advantage OA confers. OA will also increase because of the growing number of journals that have already given their official "green light" to author self-archiving, partly because journal impact factors too benefit from increased article impact, and partly because journals are eager to demonstrate that they have no wish to stand in the way of OA and its benefits to research and researchers. (The proportion of  "green" journals rose from 55% to 83% from 2003-2004: http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0033.gif, http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0037.gif.)

A JISC survey (Swan & Brown 2004):

"asked authors to say how they would feel if their employer or funding body required them to deposit copies of their published articles in one or more… repositories. The vast majority... said they would do so willingly.”

It is hence clear that the ball is now in the the universities' court: The sooner they extend their existing publish-or-perish policies to require also providing OA for all those published articles, the sooner the entire research community will enjoy the benefits of maximizing its research impact by maximizing access to its research: http://www.eprints.org/signup/sign.php

REFERENCES

Brody, T., Stamerjohanns, H., Harnad, S. Gingras, Y. Vallieres, F. & Oppenheim, C. (2004) The effect of Open Access on Citation Impact. Presented at: National Policies on Open Access (OA) Provision for University Research Output: an International meeting. Southampton University, Southampton UK. 19 February 2004.
http://opcit.eprints.org/feb19prog.html
http://www.ecs.soton.ac.uk/~harnad/Temp/OATAnew.pdf

Garfield, E. (1998) The use of journal impact factors and citation analysis in the evaluation of science. Presented At
the 41st Annual Meeting of the Council of Biology Editors, Salt Lake City, UT, May 4, 1998 - April 17, 1998
http://www.garfield.library.upenn.edu/papers/eval_of_science_oslo.html

Kurtz, M.J. (2004) Restrictive access policies cut readership of electronic research journal articles by a factor of two, Harvard-Smithsonian Centre for Astrophysics, Cambridge, MA
http://opcit.eprints.org/feb19oa/kurtz.pdf

Lawrence, S. (2001) Online or Invisible? Nature 411 (6837): 521.
http://www.neci.nec.com/~lawrence/papers/online-nature01/

Odlyzko, A.M. (2002) The rapid evolution of scholarly communication." Learned Publishing 15: 7-19 http://www.catchword.com/alpsp/09531513/v15n1/contp1-1.htm

Pringle, J. (2004) Do open access journals have impact? Nature (Web Focus).
http://www.nature.com/nature/focus/accessdebate/19.html

Swan, A. & Brown, S.N. (2004) JISC/OSI Journal Authors Survey Report.
http://www.jisc.ac.uk/uploaded_documents/JISCOAreport1.pdf