Dealing with Rejection

As scientists, we communicate our results by submitting papers to academic venues, where they undergo scrutiny under a system known as peer-review, i.e. they are reviewed by our colleagues in the research field (see my other guides on how to review papers and avoiding common mistakes for more information on this). Because academic venues by definition are selective, our papers will more often than not be rejected for publication at the specific venue. This means that the reviewers do not think the paper in its current form is fit for publication in that particular venue.

Having a paper rejected can be tough, sobering, and depressing, all at the same time. Rejections can be particularly difficult for students who are just starting in research and lack the confidence to take bad news in stride. This document will try to outline the general process of dealing with rejection and how to best turn this to your advantage.

Understand that Rejections Happen

All scientists, even the very best ones, have papers rejected. Even the very best papers get rejected. Part of the reason for this is that in many cases, the peer review process is non-deterministic. You might be unlucky and get assigned reviewers whose expertise does not fit for your paper, reviewers who do not have time to do a good job, or reviewers who simply are bad at reviewing (in that they are petty, insulting, stubborn, or resistant to change and new ideas). Having a paper rejected is not the end of the world, and it does not mean that your idea or your paper is bad. It just means that the people who reviewed your paper for that specific academic venue did not like it.

As I said above, even the best scientists get their papers rejected. Ben Shneiderman, who is a well-known researcher from my own university (UMD) and one of the founders of information visualization, describes a particularly brutal rejection that he received early in his career (this was a case of a bad reviewer, as it happens). He went on to submit the paper to another, equally prestigious venue, where it was accepted and now has been cited thousands of times ever since. As for myself, more than once I have had a paper rejected at a relatively low-tier conference, only to rework the paper and then have it accepted at ACM CHI (a top HCI conference).

So, in other words, understand that rejections happen. It is depressing and sometimes even sad, but it is a part of the life of a scientist. Just by looking at the average acceptance rate of a normal conference — around 25–30% — it is clear that for every accepted paper you have, expect to have 2–3 rejected ones.

You also need to understand that even if rejections happen, acceptances happen too. Now that your paper has been rejected, you are free to improve it by using the reviews to your own advantage, and then resubmit the paper somewhere else. My rule of thumb is that every paper you write will sooner or later (sometimes later rather than sooner) be accepted somewhere. Perhaps it will not be the best of venue, and not the venue you had originally planned, but every paper you write should be seen as a self-contained unit of work that eventually will see the light of day. Don’t worry about this work being wasted. It will eventually be published.

What Not to Do

Let’s start with the obvious: what not to do when you receive a paper rejection. Here is a non-exhaustive list:

• (Don’t) Complain to the program chairs or editor: I get it. You received an awful review. The reviewers are wrong! Worse, they are actively out to get you, and the review process is a conspiracy against you! Your paper should have been accepted! The chairs need to know about this great injustice! Before you act on this (sometimes understandable) first impulse: don’t. First of all, once final decisions have been made, there is very little chance that your complaint will have an impact (unless there truly was a clerical error, i.e. where your paper was rejected even if the review says “accept”). Second, it may also cost you a great deal of goodwill with the editor or papers chair, who did not make their decision lightly, are not (believe it or not) out to get you, and who may take personal offense against the suggestion that they are.
• (Don’t) Post the reviews online: Another common impulse is to publicize the unfair treatment you are receiving. Surely if everyone sees the sad state of your reviews, there will be a massive groundswell in your support? If nothing else, posting the reviews online and responding to them publicly may give you a modicum of satisfaction in that you at least get the final word. Again: don’t. No good will come of this, and your name and reputation (and possibly that of your co-authors) will be tarnished as a result. In fact, there is an ongoing argument in the HCI field that publishing reviews violates the copyright of the reviewers since they did not consent to their words being made publicly available. Some conferences may even have explicit rules against doing this. Instead of raising a fuss, accept the rejection quietly. If you really feel the need to respond to your reviewers, use the old trick of writing a detailed (and possibly angry) response and then discarding it without ever showing it to anyone else.
• (Don’t) Write an angry rebuttal: Some conferences (and all journals) have a built-in rebuttal process where authors get to respond to reviews. Resist the temptation to write an angry or snarky response to reviews you perceive as unfair or incorrect. Even if you happen to win the argument by virtue of superior logic, you will most likely lose because you used uncivil language. The little satisfaction you will get from this is not worth the rejection when the same superior logic or argumentation without the snarkiness may have been enough to win reviewers over.
• (Don’t) Discard the reviews: Peer review is kind of amazing (when it works) in that you get to have several world-leading experts in the field pore over your work and give feedback. It would be foolish of you to just discard these reviews when they go against you, i.e. when they do not recommend acceptance. Instead, make sure to address the comments raised by the reviewers, even if they hurt. The worst that can happen is that you resubmit the paper without changes and one of the original reviewers gets it again. They will not be pleased to see you ignored their hard work.
• (Don’t) Give up: The thesis of this article is that all good work will eventually be published. Rejection is part of life for an academic, and even the best papers get rejected. While rejection can hurt, the correct way to deal with rejection is to pick up the pieces and try again. The rest of this document explains how.

Some people have noted that sometimes you receive genuinely awful and incorrect reviews, and the only way to provide feedback on such matters is to email the editors or papers co-chairs. I understand this viewpoint. My only piece of advice for this situation is to (1) take a few days before you respond so that you are not emailing in the heat of the moment, (2) send a polite email to only the affected papers chairs or editors (i.e. none of those “open letters to the editors” or involving a steering committee or some other higher authority, both strategies which are likely to do more harm than good), and (3) not expect that anything will change for your present submission, only that change may be possible in the future.

Read the Reviews

Reading the reviews for a rejected paper can be tough; in fact, it may even be tough for a paper that has been accepted! Therefore, it is often a good idea to put your reviews aside for a while (a few days or a week) when you receive them so that you do not get too depressed. Reading critical reviews can be very hard, and may push you over the edge and sink your confidence considerably. It is a lot better to do this once you have had some distance and some perspective on the rejection.

When you do read the reviews, do so with a critical eye. Do not accept everything that the reviewers say. Remember that reviewers are not infallible. Sometimes they misunderstand things. Sometimes they want to toot their own horn and shoot down anyone who is telling another story than their own. Sometimes they are plain nasty or bitter. When you read their reviews, try to mentally look past the words and determine for each comment whether it is a plausible criticism, or whether it is irrational.

The very first times one of my student receives a review for a paper we wrote together, I usually sit down with the student and go through the reviews line by line. My goal as advisor is to help the student to cope with the comments and see which are indications of real problems, and which are just instances of reviewers being misinformed or plain wrong.

Summarize the Reviews

After having read the reviews once, you should have an understanding of the main complaints that the reviewers had. Now is the time to use these reviews to your own advantage and improve your paper so that it will have a better chance of success the next time around. Go through them once more and make sure to find the important points. Different reviewers often have the same feedback but express it in different ways; try to summarize these comments and order them by importance. Some of the comments and feedback you receive will be critical for getting the paper accepted the next time around. Some comments may not be. Some comments may require a lot of work. Some are easy to fix (even though they may look daunting on the surface).

It is important not to ignore this step. Your reviewers have spent hours of their own time to read your paper and give you thoughtful feedback. This is a great boon! Make sure to take their comments to heart and revise your work accordingly. Even review comments that you do not agree with likely have nuggets of truth about your work that you would be remiss to ignore. In particular, if the reviewer misunderstood your argument, maybe the problem is in the presentation of your argument and not with the reviewer. Before you direct all your anger and frustration at a faceless reviewer, first make sure that the problem does not lie with you. In my experience, it is easy to ignore problems with your own work in favor of being upset with an anonymous reviewer.

The end result of your summarization process should be a plan of action that outlines the main problems that reviewers found with your paper, and a list of things that you are planning to do in order to fix these concerns. This is good, this is productive. The next step is to…

Look Around for Alternate Venues

In a way, rejection grants a certain degree of freedom — your paper is now out of the peer-review process, and you are once more free to submit it to anywhere you would like (half-full rather than half-empty!). This is the time for thinking about the potential of the paper, and where it could go. You need to look for alternate venues that are conveniently located in time so that you will be able to revise the paper and then resubmit it in time for the deadline.

At this point, you basically have two options:

• Find another conference: For instance, you could look at my list of top HCI and visualization venues to find a suitable conference or workshop where the paper would fit. Also look at the submission deadlines. You want it to be sufficiently close in time so that you do not have to wait too long for acceptance, yet sufficiently far away that you will have time to actually enact your changes. Submitting the same old paper to a new conference is just asking for trouble: you might get the same reviewers for this new venue, and they will be upset to receive the same paper again where none of their comments have been taken into account. Don’t do it!
• Find a suitable journal: Some papers are simply not fit for the format of a conference, and may benefit from being published at a journal. This is certainly true for many theoretical model papers or literature surveys, which do not necessarily need (or lend themselves to) a public talk at a scientific event. Some papers are too complex or too long to be conveniently discussed at a conference, and instead require the back-and-forth of the journal review cycle, as well as the additional page allowances of a journal.

At the end of all this, all that remains is to do to the work and resubmit the work. Then the waiting game begins anew… If the outcome is another rejection, remember to start over from the beginning of this guide. If not, then ???PROFIT!!! Congratulations!

Once More into the Breach!

“If at first you don’t succeed, try and try again!“ That is also the motto that scientists need to adopt. Peer review is unpredictable and imperfect. The only way we can really work with the system is to never give up and keep improving our work, despite temporary setbacks (such as rejections) and any misgivings that arise from these setbacks. Remember the rule of thumb that sooner or later, all papers will be accepted somewhere. Just don’t give up, and don’t lose hope.

Originally published at sites.umiacs.umd.edu on October 25, 2016.