How to Review

Michael Bieber IS Department New Jersey Institute of Technology

Outline
Introduction Process Considerations What to Consider in a Submission Whether to Correct Grammar and Spelling Editors: What You Can Do in Return for Referees Sample Referee Forms Your Comments, Please

Introduction
This guide concerns refereeing for conference and journal articles. It follows the philosophy that the reviewing process benefits the author as much as the editor. Authors deserve detailed feedback in a non-hostile tone, regardless of whether you recommend rejecting, revising or accepting their work. Do the quality of review that you wish others would do for you. By agreeing to review, you take on the responsibility of doing a thorough job. If you cannot commit to this, you should notify the editor at once and return both the paper and reviewing package. Editors understand that you may not have the time to do a thorough review, but are unforgiving if you commit to a review and then do a poor job. Also, good reviewing does not go unnoticed by editors, and can establish a useful reputation for you over time. Authors also have a responsibility to reviewers. Reviewers volunteer time from busy schedules to conduct thorough reviews, and are happy to do so for well-thought-through pieces. If you submit premature work, not only will you annoy your colleagues who take the time to referee, but you will gain an unsavory reputation over time. Researchers work in rather small communities and reputations spread.

Process Considerations
Plan to read the submission three times: the first to get a feel for the it, the second reading the paper in depth, and the third to actually mark it up. Fill out the refereeing form right after the third reading, while things are still fresh in memory.

You can split your review into three sections: (a) the actual refereeing form; (b) general comments on the paper; and (c) specific comments on individual passages in the paper. You could mark up section (c) directly on the paper. Unless disallowed, you can return the editor a

copy of the paper with your markup, to be returned the author. Some reviewers make a photocopy of the paper and transcribe their comments clearly on the copy. Otherwise you can type your comments separately, clearly referencing the page, section, paragraph and/or line number.

You can write the editor a separate, confidential, note if you wish. Be sure to mark it "confidential" so the editor doesn't forward it by mistake. Many refereeing forms have a specific area for confidential comments for the editor. Do not feel obligated to write things to the editor that you do not share with the authors. In general authors benefit from as much as you can tell them.

Do not be afraid to reject a bad paper; i.e., do not feel obligated to accept a paper which you believe needs more work. Readers don't want to read poor pieces, and authors will benefit from revising and publishing quality work. Papers can be poor for many reasons. Authors may not explain themselves clearly, so the concepts, motivation, background, actual results and contributions do not come across well. The paper may require drastic proofreading. The research may be of low quality or not a significant contribution. The research may be premature to publish. The paper may have been "thrown together" (unsuccessfully) at the last minute to meet a deadline. It may not fit the characteristics of the intended journal or conference. It may too similar to other publications the author has made (though some journals will publish slight modifications of papers that have appeared in conference proceedings---check with the editor if you have any questions). For poor papers you still owe the editor and the author a review (or in the worst cases, a thorough explanation). If the paper is too premature to complete all parts of the review, state this in your comments.

Be professional and non-hostile in your review. If you get angry with an author (e.g., for doing a poor job or not for proofreading), you can say so -- either directly to the author in your comments or just to the editor.

What to Consider in a Submission

This is a partial list of what to look for when reviewing a submission. The items should help both to guide your evaluation, and form helpful comments and suggestions for the author. Not all are necessary or even desirable for all papers. Be sure to read the refereeing form, as this will give you additional criteria for the particular journal or conference.

Does the author cover everything promised in the introduction?

Does the author provide adequate motivation?

Does the author provide adequate background information?

Are all descriptions clear? Are the tables and figures clear? Do they make sense on their own or only if one has read the text carefully? Are there too many? Would an additional table or figure help? Would an example help?

Are the research contributions clear? Are the contributions significant?

Is the approach clearly explained and well laid out? Does the author justify each of the points made?

Are the equations, algorithms, methods, experiments, and conclusions: correct, robust comprehensive and sensible?

Is the research properly ground in the literature?

Has the author expressed the limitations of the research and the author's approach?

Has the author performed a complete analysis and drawn insightful conclusions?

Has the author described his or her future research plans? Is it clear where the research described in this paper will lead and what the next step will be?

Is the conclusion significant? Is it just a rehash of the paper? Does it provide new synthesis or insights? Does it leave the reader excited about the research, the research domain or the future?

Does the author use important, current and adequate citations? Are there too many citations? Are any irrelevant or insignificant? Are they at an adequate level for the publication (e.g., research oriented vs. trade journals; technical reports vs. journal articles)? Are there too many citations to the author's own work? Can you suggest any missing citations the author may have overlooked?

How is the author's writing style? Is it too "dense" to make sense? Does it keep the reader's interest? Is it too informal? Note that an informal style in itself sometimes is very effective in getting a paper's ideas across. Similarly, many authors use humor very effectively in research papers. Only if the informality or humor gets in the way, should it be discouraged. (On the other hand, there are certain fields which enforce very formal writing styles, in which an informal style is deemed inappropriate.)

Whether to Correct Grammar and Spelling

Proofreading includes checking for correct grammar, correct spelling and overall, that a paper "reads well." As you know, spelling checkers check neither grammar or comprehension. Authors should have enough respect for the reviewers and the editors to submit a paper which has been thoroughly proofread. Authors who are not native English speakers (or whatever language the forum allows) are responsible for ensuring that their submission is of the quality a native speaker would submit, even if they must pay someone to help in the editing process. Nevertheless, as a reviewer you will often find small spelling or grammatical mistakes the author has overlooked (e.g., a typo within a correction made after employing a spell-checker). And of course you may be able to suggest better ways to phrase certain passages in the paper. In all these cases, it is up to you to decide the extent to which you edit the paper. You may decide to correct the first couple of pages, or the first couple of cases of a recurring problem. If the paper requires major corrections and you know a later draft will be reviewed again, you may suggest the author undertakes such proofreading as part of the revision process.

Editors: What You Can Do in Return for Referees

A note for editors. Many referees appreciate feedback on their reviews. Like many other aspects of academia and research, reviewing is a learning process. You may consider sending each reviewer the same package you send the author, i.e., a copy of your correspondence with the paper's author, as well as a copy of each of the paper's reviews (including his or her own, in case the referee didn't make a copy). This makes the reviewer feel more a part of the process and gives valuable feedback. In addition, many referees are building tenure and promotion files. A written acknowledgment (not email) of the referee's help looks good in these files and is much appreciated.

Sample Referee Forms

Here are some example referee forms, for your information. HICSS'95 Minitrack on Hypermedia in Information Systems and Organizations HICSS'96 Minitrack on Hypermedia Research

Your Comments, Please

I would welcome your comments on these guidelines.

How to review a scientific conference paper quickly and correctly

Tags: checklist, paper, writing

The following checklist will help you to write a review a conference paper.

What is the contribution of this study?

The value of a study can not be measured without its contribution. So we should start by determining what it has been done. In a well-written paper this information can be found in the abstract or in the introduction (check also conclusion). What is new? In order to answer this question we should be aware of the literature on that area, if not check some of the references given in the paper.

Check the notations, abbreviations and definitions

Before any deep reading of the paper, it is a good practice to list all the important notations, abbreviations and definitions on a paper. This can be very helpful while checking the proofs.

Look at the some of the relevant references Are the results are correct?
This is the hardest longest part of a paper review process. Check the proofs.

Do spell checking
Even though there exist built-in spellcheckers in most of the word processors (Latex , MS Word), authors does spell checking manually for a second time. Dont trust computers nor humans, do it again!

P.S. This site is a great resource of all sort of advices (writing thesis, presentation, timet management etc.) for people in academia.

Comment
You must be logged in to post a comment.

Reviewing a CS conference paper Reviewing a conference paper is a non-trivial task. Often, reviewers have to read and review more than one paper, and usually under a tight time deadline. Regardless, there is a right way to review a paper, and many wrong things. This document contains my thought on the "right way", and points out several mistakes that many (or even most) reviewers make. The following is a list of principles to use; I'll elaborate on each of these below: 1. 2. 3. 4. 5. 6. Review the paper Review to accept papers Don't demand too much Write a review Note little things, but don't make them your review Things to avoid

Much of this is subjective and is based on my reviewing experiences (from both sides) in Computer Graphics and Geometric Modeling. Things may differ in your area, and you may have a different opinion. Feel free to send me email about this, although please apply the criteria below when composing your message. Stephen Mann April, 2009 =========== 1. Review the paper This may sound obvious, but it's harder to do than it sounds. The grossest violation of this principle are the reviews that say "the authors should have done this instead." But your review should not be about what should have been done; rather it should be a critique of what they authors actually did. If you feel the authors should have done something else, accept the paper and discuss it with them at the conference. But reviewing the paper is hard for an important reason: usually, the authors are too close to their work, and thus have difficulties stating precisely what they did, why it's of interest, and why it's important. "Reviewing the paper" means reading to a level that you understand what the authors did, why it's interesting, and why it's important. As part of your review, you should note these things. And you should accept or reject the paper based on whether you think the *contribution* is significant enough. If you think the paper is poorly written, or the contribution is poorly described, state that, but do not make it your basis for rejecting the paper. This rule is usually violated because reviewers are overloaded and under time pressure. A poorly stated result may be hard to tease out of the paper, but if you're not going to take the time to do so, then you shouldn't be reviewing the paper, and if you don't have the time to do so, you need to reduce your reviewing load. [NOTE TO AUTHORS: to help ensure that reviewers can determine what you did, etc., spell it out. Mention it in the abstract; state it in the introduction; and restate it in the conclusions, where you should link back to the body of text to support your statements.]

2. Review to accept papers. When you read a paper, try to find reasons to accept the paper. If nothing else, if you're following the first principle (Read the paper), you should spot what is good about the paper and highlight that in your review. If you don't like the approach, fine, but try to decide what about the authors paper makes it acceptable for publication. Yes, not all papers are worth publishing, but almost all papers have an idea that the author is promoting and you should review to accept that idea. Sometimes the idea is bad/wrong/already-been-done. And that's fine - the paper can't be accepted. But read the paper looking for a reason to accept it, and don't reject it unless that reason doesn't exist. And sometimes an idea is clearly half-done. The temptation is to reject the paper with the recommendation that it be resubmitted when the research is complete. But often it's the idea itself that is the research contribution. And if it's a good idea, then consider accepting the paper on that basis. This becomes particularly important when you realize that a lot of research is done by graduate students, and papers submitted on their work may be all that ever gets done on it. By rejecting a great idea because it wasn't perfectly polished, the idea may never get published despite being worthy of publication, since that student's work is done. Related to this is when you write your review, write with the mind set "how to improve this paper" rather than "here's a list of things that are wrong with this paper." 3. Don't demand too much The paper is a conference submission, and there are page limits. Don't write a review saying "the authors should include the following", where "the following" would push the paper well past the page limits. If there is something so critical that it MUST be included, suggest something to remove/reduce so that the authors can kept to the page limits. Likewise, don't demand any additional work that can't be done in the time between acceptance notification and the final submission deadline. While analysis can sometimes be redone, it's unlikely that another experiment can be run or significant code can be written. 4. Write a review Review forms have check boxes, and there is a temptation is rely on the check boxes with minimal comments. But your written comments are really the important part of your review, and you should write comments that help both the program committee and the authors. In particular, be sure to cover the following in your written comments; some of this material will be in response to questions on the review form, but regardless it should all be covered somewhere in your review:

1. Outline the paper. 2. Highlight the contribution of the paper, both what the authors perceive it to be and what you perceive it to be, as well as how significant it is. 3. State your recommendation and why. 4. State ways to improve the paper, but don't ask too much (see both the previous and next sections). The first forces you to reread the paper, which helps in writing your review. The second is the hard part: you need to figure out what the authors think is their contribution. The 3rd is one that the program committee will focus on. Of these four, the 2nd and 3rd are the important ones. 5. Note little things, but don't make them your review "The authors should include the following references." "The grammar needs to be improved." "The figures are poor quality." No paper is perfect. There will be details that are wrong, often of the above variety, but sometimes of a bit more substance ("the authors give the wrong formula for X"). These are not reasons to reject a paper (although if you can NOT read a paper because the grammar is terrible, you have no choice but to reject it for that reason). Again, focus on the contribution and base your recommendation on the contribution and not the writing details. You should note the small things, but ideally place them at the end of your review in a separate section as "details to improve". 6. Things to avoid Here's a list of miscellaneous things to watch out for in your reviews. A. Do not say "the authors should add additional references on X" without actually listing those references. If you're enough of an expert to make the judgement, then you should be enough of an expert to explicitly list those references and state why they should be added. In particular, since there is a page limit, the bibliography should be focused on the most relevant work, and not be a complete survey of the topic. So if you think a paper should be cited, give a strong reason as to why, since potentially the authors know of the reference and decided not to list it for their own reasons - your argument to include the reference should be strong enough to convince someone who has already decide to not include it. And NEVER, NEVER, NEVER reject a paper because it omitted references! Don't even HINT at rejecting a paper for this reason. This may sound obvious, but if you decide that a paper should not be accepted and in summarizing your reasons you mention the missing references, you have just

hinted that you rejected the paper for missing references. (As a more general rule, never reject a paper for something that can be fixed in 5 minutes.) B. Usually you get to rank the paper on a scale like 1 to 5 as to whether or not the paper should be accepted. Around 2/3 your rankings should be 1 or 5, around 1/3 should be 2 or 4, and you should rarely, rarely, rarely give a rating of 3, which should be considered a reject anyway. If you can't give a strong recommendation, then you likely didn't understand the paper well enough to review it. I have heard the statement "I never give a 1 because I don't want to hurt the authors feelings." That just makes "2" the new "1", and you won't have spared anyones feelings. If you don't want to hurt the authors feelings, then do a good job of understanding the paper and base your decision on what the authors did, and write your review as "how to improve the paper" rather than "bash the paper". C. Some people will try to tell you that for some conferences, the conference papers are the same quality as a journal paper. This is wrong for several reasons: there is usually an explicit page limit, and there is no chance for resubmission (resubmitting the paper to a future conference is different than resubmitting a journal paper). The result is a lower quality paper than a journal paper. This doesn't mean conference papers are terrible, nor does it mean they are worthless. Making a distinction between the two is important, however, since you review a journal submission with different standards/criteria/etc than you review a conference submission. In particular, a journal paper needs to be more complete than a conference paper: there needs to be a better literature review; there needs to be a more complete result; there needs to be a more in-depth analysis of the result. Understanding the difference will help improve your converence paper reviews. D. Don't be insulting, be positive. Other review guidelines usually state the former; I've never seen an insulting review, but I guess it happens. More of a problem is being positive: the authors put a lot of effort into writing the paper and will be sensitive to (and even insulted by) criticism. So phrase things positively. In general, write your entire review in a tone of having accepted a paper, even when you're not recommending acceptance. This will help change what you subconciously write as condemning criticism to helpful comments.