Compare impact of open vs. non-open access articles in the same journals

Stevan Harnad

In "Open-access journals rank well" (The Scientist, April 27, 2004), Allison McCook reports that an Institute for Scientific Information  (ISI) study found traditional journals and open-access journals to have similar impact factors. The ISI's press release said:

"Today, Thomson ISI... announced that journals published in the new Open Access (OA) model are beginning to register impact in the world of scholarly research... Of the 8,700 selected journals currently covered in Web of Science, 191 are OA journals... [A study on] whether OA journals perform differently from other journals in their respective fields [found] that there was no discernible difference in terms of citation impact or frequency with which the journal is cited."

But if you want to get a better idea of the effect of OA on impact, don't just compare the 2% of ISI journals that are OA journals with the 98% that are not, to find that they are equal in impact (for this may well be comparing apples with oranges even if you equate the subject matter). Compare the much higher percentage of articles from the 98% non-OA journals that have been made OA by their authors -- by self-archiving them -- with articles (from the very same journals and volumes) that have not been made OA by their authors: You will find that there is indeed a discernible difference in terms of frequency with which the article is cited, and that that difference is from 250%-550% in favor of the articles that their authors have made OA! That is what an ongoing series of comparisons based on a 10-year sample of the same ISI database across all disciplines is revealing (in computer science and physics so far: Lawrence 2001; Brody et al. 2004; Kurtz 2004).

On Wed, 14 Apr 2004, Eugene Garfield wrote (in the American Scientist Open Access Forum):

EG: The results obtained for computer science by analysis of  CiteSeer are distorted for a variety of reasons. They cannot  be compared with the literature of e.g. life sciences. Computer science is heavily dependent upon conference literature. I cannot comment upon the physics literature, but there are other studies which seem to indicate that readership increases will not necessarily be followed by increased citation impact.

Gene Garfield, the father of citation analysis as well as ISI, is quite right that the Lawrence (2001) study on the impact enhancing effects of open access in computer science needed to be replicated in other fields to check whether it was merely an artifact of the fact that computer science is conference- rather than journal-based.

But, thanks to the ISI database licensed to the Observatoire des Sciences et des Technologies (OST) and a special contract generously provided by ISI to conduct the study, we are in the process of testing the Lawrence effect across all disciplines in a 10-year ISI sample of 14 million articles. The physics analyses up to 2001 are already done (Brody et al. 2004), and they reveal even larger effects than those reported by Lawrence, with OA/non-OA citation ratios of 2.5 - 5.8. All indications are that the 2002 data will raise them even further, as the biggest effects occur within the first 3 years of publication in scientific disciplines (and both OA and the awareness and visibility of OA articles are also increasing yearly).

EG: In one study of a single chemical journal that I refereed there were about 100 readerships for each citation of that
journal, but there did not seem to be any perceptible increase of citation by the research literature.

The ratio of "reads" to "cites" will no doubt vary by field. Kurtz (2004) and co-workers report it as 17:1 and even 12:1 in astrophysics. Odlyzko (2002) reports similar experiences in mathematics.

Tim Brody's remarkable download/citation correlator/predictor gives the size of the correlation by field, and can be used to predict citation 6-24 months later from downloads today (with an adjustable time-window):

EG: Undoubtedly the web will increase apparent readership of literature, but that will not necessarily change the population of relevant researchers who are in a position to cite particular studies.

Not necessarily, but very probably! And also actually, in the fields tested so far. After all, access is a necessary if not a sufficient precondition for citation. And since Open Access (OA) dramatically increases the number of would-be users who would otherwise have been denied access to the article (maximises it, in fact, for all who have access to the web) it stands to reason that OA can only increase both usage and impact.

The way to test this, however, is not just to compare apples and oranges (i.e., OA and non-OA journals). The right way is to compare OA and non-OA articles in the same journals (and years). That is what our study with the ISI data is doing.

(As Gene himself has often stressed, it is the article [and author] citation counts that should be weighed, and not just the average citation counts of the journals in which the article appears!)

EG: I do not think the ISI study is definitive but it is not irrelevant.

It is certainly not irrelevant to have shown "that there was no discernible difference in terms of citation impact or frequency" between the 191 OA journals and the 8509 non-OA journals indexed by ISI, equating for comparable journals as closely as possible: But obviously there is a certain risk of circularity in this! It does show that the skeptics are wrong (for these OA journals): OA journals are indexed by ISI, and they do have comparable citation impacts.

But the real test of the effect of OA on citation (and download) impact is on an article basis, where the journals can be equated exactly (by being the same journal and year!). And there the impact-maximising effects of OA are proving to be very dramatic indeed.


Brody, T., Stamerjohanns, H., Harnad, S. Gingras, Y. Vallieres, F. & Oppenheim, C. (2004) The effect of Open Access on Citation Impact. Presented at: National Policies on Open Access (OA) Provision for University Research Output: an International meeting, Southampton, 19 February 2004.

Kurtz, M.J. (2004) Restrictive access policies cut readership of electronic research journal articles by a factor of two, Harvard-Smithsonian Centre for Astrophysics, Cambridge, MA

Lawrence, S. (2001) Online or Invisible? Nature 411 (6837): 521.

Odlyzko, A.M. (2002) The rapid evolution of scholarly communication." Learned Publishing 15: 7-19