How to Review a Paper:

A guide for newcomers and a refresher for the experienced.

V2.0 16th Jan 2007

Hugh Davis www.ecs.soton.ac.uk/~hcd

(Please feel free to reproduce this document with acknowledgement, and send suggestions for improvement to the author)

Introduction

Reviewing a paper is a difficult skill that takes practice to acquire. The purpose is:

Answering the question of whether work is suitable for publication in a particular venue requires you to understand something about the standards and aims of the journal, conference or workshop concerned. Clearly the expectations for significance of results, degree of originality of results, and completion of evaluation will be very different for a workshop or small conference, compared to a major international journal.  However no venue would expect to publish work that is flawed or plagiarised.

Please also note the requirement for *constructive* criticism. Let us start with the assumption that most work has been conducted by other hard working and honest people like ourselves who are proud of the work they have done. If you don't think the work is ready for publication, then explain why, and tell them how useful it will be when suitably improved; don't 'diss' them! The main reason why papers are turned down is not because the work is flawed, but because it's value has been badly communicated or it has been submitted to an inappropriate venue.

“Desirable traits in a referee include objectivity, fairness, speed, professionalism, confidentiality, honesty, and courtesy” (Parberry, 1989)

What goes into a review

To start with, you should assume that everything you say in a review will be shown to the authors - and it is good practice to assume that the authors will find out who did the review - so make sure you would be happy to defend your review to the authors! There may be some things that should not go to the authors, and we will cover this later, under comments to the committee/editors

1. Summarise the paper in your own words.

In affect you should write the abstract you would have written if you had written the paper. Clearly this abstract should say what the authors have done, what their contribution is, and what  the significance of that contribution is (*however small you think the contribution is you should still do this*). The point here is for you to show you have read and understood the point of the paper.

There are many sorts of papers including technical papers, review papers, thought experiments, experiences papers and evaluations. It is important that you understand what sort of paper this claims to be, and that you show that you have reviewed it with that understanding. Note that many review forms tend to assume that all papers are straight technical papers, which may well not be the case inter-disciplinary areas such as Technology Enhanced Learning. You may have to fight against the constraints of the review form, and comment on this to the committee/editors.

2. "Technical Content"

Remember to keep it constructive! If you think they have not explained something fully, then ask the question, or explain that you could not understand. Don't just tell them they have done it wrong.  Here you might examine:

3. Writing and Presentation

This section is concerned with the quality of the writing.

4. Typos

It is good manners to produce a table of any grammatical errors, typos and other minor textual problems that you have spotted. But it's not the reviewer's job to copy edit the paper, so don't go out of your way to look for typos. As a rule of thumb I would not expect to list more than ten such errors. If there are many more I would say something like "There are multiple typos that need serious attention before publication". If the English throughout the paper is poor, then you must say so, but do be charitable. Presumably the problem is that English is not the authors' native language; they need to be advised to get help before submitting the final version.

5. Conclusions

In the conclusions you should:

6. Overall Recommendation

Most review forms will have some tick list or Lickert Scale for you to use e.g.

OR

  1. Excellent - This paper is amongst the best papers I have ever read (short-list for best paper award)
  2. Very good paper (Consider short listing for best paper award)
  3. Sound paper - I recommend acceptance
  4. Borderline - This paper could be accepted if there is room
  5. Poor - This paper has limited contribution, or the work is not yet ready for publication. I do not believe it should be accepted, but if other reviewers differ, I would not oppose strongly
  6. Unacceptable - The work makes no contribution or, worse, it is flawed or scurrilous. I believe that publication of this paper would reflect badly on our community. I would strongly oppose any other outcome.

These are both good sets of words to consider if there is no particular response mode prescribed.

To the Editors/Committee

you can assume that the authors will see all the responses above. Most review forms also provide a space for "Comments to the committee". If they do not, then these comments will typically go in an email to the editors/committee.

This is NOT the place for "sneak attacks"! If you have anything to say about the content of the paper, you should be happy to say it to the authors.

This place is where you can tell the committee/editors about your own reflection on the quality of your review.

This is also the place where you might explain any difficulty you had in conforming to the review form template they have presented. For example you may on occasions decide to recommend acceptance of a paper that is generally quite poor because you believe it would spark important debate in the community; or you may think that it is an excellent review of the state of the art that you would like to see published in spite of the fact that it has no original technical content.

Finally, this is the place to present any evidence you have of unprofessionalism on the part of the authors. If you have evidence that they have plagiarised, fabricated results or otherwise mis-represented their work - or maliciously mis-represented the work of others, then you should say so here. You will need to prove your assertions.

Some Useful References

Dale J. Benos, Kevin L. Kirk and John E. Hall. How to Review a Paper. Advan. Physiol. Edu. 27: 47-52, 2003; http://advan.physiology.org/cgi/content/full/27/2/47 (Last accessed 16th Jan 2007)

Ralph E. Johnson, Kent Beck, Grady Booch, William Cook, Richard Gabriel, Rebecca Wirfs-Brock. 1993 How to Get a Paper Accepted at OOPSLA. http://www.acm.org/sigplan/oopsla/oopsla96/how93.html (last accessed 16th Jan 2007)

Ian Parberry A Guide for New Referees in Theoretical Computer Science. ACM SIGACT News (Nov 1989)  Available at http://portal.acm.org/citation.cfm?id=74090

Barak A. Pearlmutter. How to Review a Scientific Paper. http://www.cs.unm.edu/~bap/how-to-review.html (last accessed 16th Jan 2007)