Mistakes Reviewers Make

The scientific community is built on peer review, where a jury of your colleagues help judge the quality of your submitted work. As the pressure to publish (or perish) grows, scientific journals and conferences in all fields are receiving an ever-increasing number of submissions to be reviewed for publication. Considering that each scientific submission typically requires a minimum of three reviews, this in turn means that the reviewer pool must also be increased to handle the demand.

Unfortunately, training new reviewers is time-consuming because writing fair, balanced, and professional reviews in a timely manner is difficult. In a perfect world, new reviewers would have the opportunity to work closely with mentors in learning the craft of peer review, or would be given the chance to extensively calibrate themselves with trusted colleagues in forming their opinions without real repercussions. This would allow reviewers to mature and cultivate their ability to distinguish between major and minor issues in paper submissions before being asked to review real submissions. Unfortunately, this is rarely the case; there is little in the form of training programs for new reviewers. Instead, they are often immediately thrust into live situations where they are asked to make important decisions on paper submissions, decisions that have real outcomes that sometimes make or break careers.

Having people essentially learn by doing is problematic because new reviewers often make mistakes due to their lack of experience, maturity, and calibration with the rest of the field. This causes papers that should have been accepted to be rejected, and vice versa. The end result is that the quality of the review process suffers as a whole, and, in response, conferences and journals lose in quality as well.

Fortunately, there is often a common pattern to the mistakes reviewers make. In the more than 10 years I have been active in reviewing papers — as an external reviewer as well as an associate editor and program committee member — I have seen my fair share of such mistakes (and made some myself!). My primary goal with writing this document is to characterize these mistakes in the hope that I can reduce the learning curve for new reviewers and help them improve their reviewing. Naturally, my ulterior motive is to improve the quality of peer review itself! As all scientists know, few things are so aggravating as having your work be rejected due to nonsensical or uninformed reviews. My ultimate goal is to help reduce the frequency of such low-quality reviews.

This is not a general guide on how to review papers. Rather, it is a guide on how to avoid the many mistakes that new and inexperienced reviewers tend to make. Actually, as it turns out, some of these mistakes are still common among more senior reviewers! I would encourage everyone involved in the scientific review process to at least skim through what I have to say.

Know Thyself

Professional analysts know that objectivity is impossible and that everyone has a bias of some form. Instead of vainly trying to be unbiased, they recognize and embrace their bias, correcting their analysis in response. Similarly, inexperienced reviewers often exhibit several common traits that are useful to be aware of when preparing your review:

• Competitive: Most new reviewers are graduate students that are just starting research, which means that they are often young, hungry, and eager to prove themselves. This may also mean that they view other people’s submissions as potential competitors.
• Detail-oriented: New researchers are often immersed in the minutiae of research, such as building software, collecting data, and running experiments. This means that they tend to focus on details (which may or may not be significant) rather than the bigger picture.
• Specialized expertise: Many new reviewers rank themselves as having high expertise in a topic, but the reality is often that their expertise is very narrow and specialized.
• Star-struck: Young researchers may easily find themselves idolizing senior, established researchers. For single-blind reviewing, this may cause them to view papers written by such senior researchers more favorably than otherwise (or, sometimes, the opposite).

Common Sentiments

Peer review is ultimately a matter of subjective opinion. You are being asked to give your own personal opinion on a scientific work for the purpose of deciding whether the work should be published or not. However, even if the review represents your own sentiments, you are expected to provide support for them in your review. Furthermore, many sentiments are not necessarily appropriate. Below is a list of common sentiments that you should think twice before using:

• “This work does not fit topic X.” — This sentiment gets thrown around a lot, even by senior people. When reviewers come across a paper that does not fit the typical paper mold for a particular venue, one common response is to dismiss it for this very reason. If you find yourselves writing this in your review, stop and think whether you are being needlessly conservative. Is it possible that this new approach will actually move the field forward?
• “This work is not new.” — Lack of novelty is a very common reason for rejecting a paper. However, when you use this line of argument in your review, make sure you also give concrete evidence. You cannot base your criticism on a hunch, a feeling, or some perception of what “they” (as in, “others in the field”) have done. If you find yourself dismissing the work because it is not novel, yet cannot find an existing paper, product, or website to back up your statement, you may be wrong. Even simple ideas can be novel.
• “This work is not enough.” — Even when a contribution is novel, another sticking point is whether it is novel enough or significant enough compared to the state of the art. This is notoriously difficult to determine, and depends on the stature and expectations of the conference or journal. Again, be sure to think twice before writing this in your review. Are you sure that you are being fair and factual in making this assessment? Have you lined up the evidence to back up your statement? If the significance is a borderline case, you may want to hold off on your most brutal takedown since it is possible you are wrong.
• “This work is too simple.” — A special variant of the “not enough” sentiment, the “too simple” sentiment gets used when reviewers (even senior ones) come across a seemingly simple idea yet are unable to find prior work that has already attempted the idea. Such simple ideas can be provocative to reviewers because they suggest that they could be conceived by anyone, even the reviewer herself or himself. Furthermore, there is sometimes snobbish resistance against accepting work that seems like low-hanging fruit and requires little effort. However, before giving in to these feelings, first remember that simplicity is difficult, and the very best ideas are those that only seem straightforward in retrospect, but which require something close to a stroke of genius to discover in the first place.
• “This work does not matter.” — We’re now at a point when you can agree that the work fits the topic, is sufficiently novel, and is sufficiently developed to warrant publication. However, you may still feel that the paper ultimately does not benefit the field, perhaps because you don’t agree with the problem, the solution, or the validation. Maybe the authors investigated a topic that you think is not important. Maybe they conducted a user study with no significant findings. Maybe you are a practitioner and you feel that the proposed solution will never matter in “the real world”, or maybe you are a researcher and you feel that the work focuses too much on engineering and has no scientific contribution. While these are all valid reasons, be sure to again provide plenty of evidence to back up your sentiment. Also take a moment to reflect on whether your own background may be affecting your assessment, and remember that there should be space for both applied and basic research in science.

In addition to the above sentiments that deal with the contribution of a paper, there are a few additional and more specialized sentiments that commonly arise:

• “There is no user study.” — Work in human-computer interaction often have user studies (controlled or otherwise), yet fringe HCI work in the information visualization field does not always do. Resist the temptation to reflexively reject a paper (regardless of venue) just because it does not have a user study. First ask yourself whether a user study is needed for the type of contribution the authors are putting forth. On a more general level, consider using Munzner’s nested evaluation model to determine an appropriate evaluation method. There are many ways to validate a contribution.
• “They did not cite (my) important work.” — Yes, if there exists prior work that is relevant to the current paper, the authors should have cited it, and it can be aggravating to review work that seems oblivious to prior contributions. This is exacerbated if it is your own work that the authors neglected to cite. It is easy to demonize authors as willfully choosing not to cite work for some insidious reason, but it is much more likely that they are simply not aware of the prior work (or forgot about it). Remember that adding a new citation or two is often doable in a minor revision, and is typically not grounds for rejection (except when the omission is critical to the contribution of the paper). Instead use your review to suggest that the authors improve their literature survey.
• “Their response made me angry.” — Journals and some conferences (such as ACM CHI or ACM UIST) allow authors to provide a response to prior reviews together with the new revision of a paper. Sometimes the authors may respond in a way that appears petty, impolite, or even arrogant. It is easy for a reviewer to feel that they should reply in kind, most often by summarily rejecting the paper. Before you give in to this reflex, first take a moment to consider things from the viewpoint of the authors. Perhaps they felt that you misunderstood their work, that you blew minor details out of proportion, or that you used negative or even rude phrases in your original review. Perhaps they did not perceive that their language would cause affront, possibly because English may not be their mother tongue. Try to look past the angry response and stay objective about the submission; did they address your comments and provide an improved revision, or are they just choosing to ignore the feedback from you and other reviewers? If they did improve the paper, consider being the bigger person and ignore the angry or nasty tone they used in their response.

Common Mistakes

The previous section dealt with simple sentiments that are not appropriate to use as grounds for rejecting a paper. In this section we produce a list of some more complex mistakes that reviewers commonly make:

• Reviewing your version of the paper, rather than the submitted version. It is easy to start reading between the lines of a submitted paper under review, particularly if the paper deals with an area that you have thought about or even worked in yourself. You may find yourself filling in the blanks, interpreting results in a particular way, or agreeing/disagreeing with specific design choices based on your own background. This can both have negative or positive outcomes. Remember that the author’s version is what was actually submitted, and while it can be improved based on reviewer feedback, it is likely never going to conform to your idealized version of the paper. Review the paper at hand, not the dream.
• Over-calibration. All reviewers — new ones in particular — should calibrate their opinions by reading reviews and ratings written by other reviewers for the same paper. This serves a good educational purpose for the reviewer pool. Many conferences allow reviewers to read each other’s reviews before the decisions are made and the authors are notified. This allows calibration to happen already for the current paper, but may also have a detrimental impact on the exchange of ideas in the review process. More specifically, sometimes your review and rating may be inconsistent with other reviews. A common reaction is to want to conform by hastily revising your own rating closer to the general consensus. This phenomenon generally causes ratings to converge to the neutral: highly negative ratings are raised, whereas highly positive ones are lowered. However, avoid mindlessly changing your rating (particularly lowering it) until you have read the corresponding review in detail. If the other reviewer has found good or bad points that you missed entirely, definitely go ahead and revise your score (and update your review text). On the other hand, if they just have a different opinion of the severity of issues you were already aware of, consider keeping your score as it is. Especially non-traditional papers need more champions willing to stand up for them in the review process.
• Rudeness or nastiness. There is never cause for being rude or downright nasty in a review (even for blatant plagiarism; there are better ways to deal with such situations). Remember that there is another person on the other end of the review process that will read your writing. Often at least one of the authors is a student, perhaps just starting out in research just like yourself, to whom rude and nasty reviews can have a serious effect. Others are hoping that acceptance will let them land a new job, new promotion, a new grant, or simply a reason to attend that particular conference. Such vulnerable authors can easily take harsh rejections as personal attacks on their work. It is not uncommon that a particularly nasty formulation in a review causes tears and even students to be turned off research altogether. Even if you want to recommend rejecting a paper, there are many ways to write such reviews that do not resort to hurtful or personal attacks.
• Inflexibility (“stick in the mud” syndrome). Sometimes a reviewer turns out to be wrong. You may simply have misunderstood key parts of the paper, and recommended accepting or (more likely) rejecting it for the wrong reasons. Perhaps the authors themselves pointed this out to you in a response to your review, or perhaps other reviewers or the editor/program committee member told you. If this happens, keep an open mind and be ready to change your opinion. There is a tendency for people to feel that they have to stick to their guns because their professional reputation is at stake, but this is part of the reason that the review process is blind. You can easily change your mind without losing face. No one will know, and you are a bigger person for doing it.
• Expecting perfection. No paper is perfect. Tamara Munzner talks about science as a “conversation carried out through paper-sized units“. Using this analogy, each paper-sized unit should be seen as worthwhile (and be accepted for publication) if it somehow propels the conversation forward. Nothing in this premise requires each paper to be perfect and do everything. Since the process is a conversation (with many participants), it is enough that each spoken line contributes something that others can build on, correct, or use. Remember this notion of a conversation every time you reject papers due to minute or minor details. If the paper has something meaningful to say, maybe you should accept despite its minor flaws?
• Reviewing the authors, not the paper. Avoid using “the authors” as the pronoun, and instead use “the paper”; for example, say “the paper fails to correct for pairwise comparisons…” rather than “the authors fail to correct for pairwise comparisons…” The latter puts the blame on the authors themselves, whereas the former makes it clear the criticism is directed at the paper itself. This makes your review much less personal in tone and will make it easier for the authors to separate themselves from faults in their work.
• Reviewing the authors’ repertoire, not the paper. Knowing the names of the authors can make it difficult to separate your opinion about the present submission and the authors’ combined body of work. Even if you hold one or several of the authors in very high regard, try to ignore that context when reviewing their new submission. Even established and well-known researchers conduct bad research or write bad papers at times. Again, review the paper at hand, not the dream (or the person).

Questions to Ask

Finally, after finishing your review, make sure to answer the below questions for yourself:

• Do you believe in the contribution? Too many times, inexperienced reviewers reject work because of minor problems that do not really affect the overall work. The real question is if you believe in the contribution. If so, the work should likely be accepted. For example, a paper describing a cool new system with a flawed user study may still be worthwhile to publish. As long as the user study is not an embarrassment or sets a dangerous precedent, perhaps cool systems and novel ideas should prevail?
• Are you making a mountain out of a molehill? Sometimes the main criticism about a paper is really a minor detail in the grand scope of things. New reviewers are particularly prone to making this mistake since they often lack the ability to see the bigger picture. Ask yourself how important a non-optimal color scale, missing user study detail, or poor user interface design really is to the overall message of the paper. A pointed comment in your review may be sufficient; even better, make the fix a condition for acceptance!
• Would it be embarrassing to have this paper be published? Above all, do no harm (to the conference or journal you are asked to review for). Papers that you would be embarrassed to see show up in a conference session or in a journal in your field should absolutely be rejected. Even if the peer review process is anonymous, it is not nice to hear other people speak angrily about the bad paper that you were part of accepting to the conference or journal. (Note that when I say “embarrassing” here, I don’t so much mean the topic of the paper — any topic should go as long as it fits within the general research area — but whether the work is of sufficiently high quality and has no fatal flaws. As a rule, you should not reject work just because you are embarrassed by its content, i.e. because it goes against your principles or is uncomfortable to you; you are supposed to be an objective scientist, remember?)
• Could you be wrong? Are you absolutely sure that your review is factual and that you have a correct view of the problem and solution? Are you sure that you have sufficient expertise in this topic to make strong recommendations either way? Again, be careful about using your strongest attacks if you are not 100% sure that you are in the right. Otherwise, it will be embarrassing (anonymity or not) if you turn out to be wrong.
• Would you be angry if you received this review yourself? This is truly the litmus test of a good review. Would you be upset if another reviewer gave you the feedback you are providing? And by this, I don’t so much mean the outcome of the review — everyone is upset when their paper gets rejected — but rather the way you wrote it. It is possible to recommend rejection while still being polite, friendly, and encouraging.

Summary

After finishing reading my list of sentiments, mistakes, and questions above, you may feel that I have taken away virtually every weapon in your arsenal. How can I reject a paper if I am not allowed to comment on the fit, novelty, size, scope, and relevance of a contribution? The answer is that of course you can reject a paper based on any or all of these criteria. For example, some papers that are clearly off-topic for a research area should be summarily rejected. Some papers absolutely need a user study to validate their contribution. Not citing prior work is certainly grounds for rejection if that prior work is central to the topic of the paper. Arrogant authors that willfully insult their reviewers and refuse to listen to feedback deserve to be rejected. The list goes on.

Rather, my goal with this guide is to force you think a little deeper and introspectively on your sentiments, and point out common themes in mistakes that reviewers tend to make. At its core, peer review is an imperfect mechanism to make objective decisions based on a highly subjective and personal process. After all, as stated earlier in this document, when you are asked to write a review, you are being trusted with the task of passing judgment on a scientific submission based on your own expertise, experience, and opinions. I simply want you to be wielding this trust at the best of your ability. Remember Uncle Ben from Spider-Man: with great power comes great responsibility.