The Sublime Art of Recruiting External Reviewers

My thoughts on recruiting external reviewers for peer review.

Here’s a straightforward post on one of the mundanities of academic life: recruiting external reviewers. Ever since becoming a faculty member in summer 2008, amazingly now 10 years ago (yikes!), I have found myself being invited to an ever-increasing number of program committees and editorial boards where my main task is to recruit external reviewers for submissions. While this is not exactly rocket science, I found I have some experience to share on this topic. Here is my method.

There are basically three parts to this task:

1. The finding. Selecting the right people for the job.
2. The inviting. Inviting them to review; rinse and repeat until a sufficient number has accepted.
3. The follow-up. Chasing reviews until they are all turned in.

The Finding

Selecting external reviewers for a submission you are managing is perhaps one of the most influential decisions you have over the paper. Recall that peer review is a somewhat democratic process, where each of the reviewers has at least some say in the ultimate fate of the submission under review. The relative power of each reviewer is often some nebulous function of the expertise, experience, and eloquence as perceived by the primary reviewer or editor who makes the final decision. However, once you have invited your reviewer panel, it wouldn’t look too good if you just disregarded their opinions. For this reason, finding suitable reviewers should not be done lightly.

My first step when finding external reviewers is to get a firm grasp of the paper first. While you may not need to read the whole thing first, you should at least read the title and abstract so that you can form an opinion of the type of expertise you need in a reviewer.

After that, it becomes a matter of coming up with names of people with the required expertise. If you’re experienced and have a significant social network within this topic (as well as good memory), this could very well be the last step in this process. You just collect your names and move on to inviting them. There are a few caveats that I cover in the next section.

However, many times, you may not know enough about the experts in a particular field to be able to assemble your reviewers off the top of your head. This either happens because you are new, or because you are being asked to manage submissions that are outside your area of expertise (not at all uncommon). In this situation, Google and the professional society digital libraries become your best friend when trying to find appropriate reviewers who have published on the specific topic of the submission under review. In other words, it’s time to exercise your search engine muscles!

However, there is an even easier answer right in front of you: the submission itself. I often find that reading the references in the submission will quickly surface a couple of promising names of authors who have published on this topic, and who should be in a good situation to review it. The only wrinkle here is that the submission is not guaranteed to have a complete literature review (after all, it is still only a submission and has not yet been accepted by a panel of peers), and that there may be additional people who would be in a good position to review the paper that you may miss if you rely entirely on this tactic. Mixing in some of your own independent information seeking is probably a good idea.

Balancing Your Panel

There are a few things to keep in mind when settling on a review panel. First of all, for every potential reviewer, you should do your due diligence on whether there is a conflict of interest between the reviewer and any of the authors of the submission under review. This may be hard to spot for the reviewers themselves if the paper is anonymized, whereas you presumably know the names of the authors. Second, depending on how many reviewers you get to invite (see below), take at least a moment to consider the balance of your panel: what is the mix of seniority, gender, nationality and ethnicity, industry vs. academia, etc. I know that these aspects can be difficult to balance in a panel of only three or four external reviewers, let alone when you are selecting only a single one (as for CHI and VIS), but we are all responsible for promoting inclusivity and diversity in our academic communities.

A note about seniority: in a perfect world, you would only want the most experienced and knowledgeable reviewers on every submitted paper; basically the top three or four authorities on the subject. However, in the real world, senior people get a lot of requests just because they are so knowledgeable, so they are more likely to suffer from reviewer fatigue (see below) and thus they may often either (a) decline your request, or (b) not have time to write more than a quick review after a cursory read of the submission. This is where involving more junior (but dependable) people is a useful strategy: these people often have more time on their hands, and they will thus be able to do a more careful job. Furthermore, involving them will contribute to their development into senior authorities of their own.

Also keep in mind your past experiences with your potential reviewers. Some people are a better fit for reviewing certain types of contributions, and a worse fit for other types. Also, some people are persistently late, flaky, or too harsh in their judgments. Don’t invite people who have not done a good job reviewing for you in the past.

Overall, I find that a diverse set of reviewers is the best strategy. However, this can be challenging for those venues, such as CHI and VIS, where each PC member only invites one external reviewer: how can I balance my panel if I only have one slot to fill? While the reasoning for this is sound at its core-diminishing the power of individual PC members-I feel that it contributes to reviewer fatigue because with only one slot to fill, PC members will tend to go only for the most experienced and thus “safe” reviewers, which is only a small subset of the entire available reviewer pool. It is a problem with no easy solution.

The Inviting

This step should be easy enough, right? And it mostly is. However, there are a few tricks to help out here as well. For one thing, just coming up with the right people is not enough; you need to get them to accept as well! Surprisingly often you will find that people turn down your review requests. Reviewer fatigue is a thing, and the more senior a person is, the more review requests they tend to get. This mean that you may have to go back to the previous step to whistle up some more reviewers to try to recruit.

For some journals and venues, you may need to ask quite a few would-be reviewers before you have a full complement. For conference reviewing, this often happens because an entire program committee is simultaneously asking the same limited pool of reviewers to review their assigned papers. For journals, this is more likely to happen for less prestigious journals, or for journals owned by controversial publishers (such as Elsevier). My record is something like six or seven invitations until I was able to get three acceptances.

To minimize this, it makes sense to put a little effort into your invitations. I find that the optimal strategy is to write a personal note for each invitation email. Instead of merely accepting the standard invitation email that the review system sends out, I add this personalized invitation at the top. It takes a little more time, but typically pays off in more accepted reviews. Remember that your would-be reviewers are busy people with full schedules, so treat them with respect! Hopefully you even know them, and can write something a little more interesting and specific than just the boring boilerplate that the review system provides.

The Follow-Up

Once the invitations are out and enough reviewers have accepted, all you have to do is to await the reviews to come in. Simple as this sounds, there are a few nuances here as well. Reviewing is rarely anyone’s top priority, so in the interest of receiving your reviews on time, it makes sense to send one, and possibly even two, reminders before the review deadline. I know a few people who will only do a task if they are reminded in a timely manner about it. Again, personalized emails from you rather than just from the review system, editor-in-chief, or papers chairs generally work better. After all, you were the one who asked your reviewers to do this work, so they are more likely to respond to your emails rather than a mass or automated mailing.

Striking the right tone in your reminders is important. Remember that reviewing is voluntary work with no pay, so avoid snarky, shaming, or accusatory language. While you could argue that accepting a task means completing it well and on time, keep in mind that the reviewers are actually doing a favor, not the other way around. Before the review deadline, in particular, make sure that you are keeping a light tone and that you respect people’s individual work habits. For example, there is nothing wrong with a reviewer leaving a review to the very last minute as long as the reviewer does a good job. Some people are merely just-in-time reviewers; I am often that way myself. In other words, don’t suggest that just because they are close to the deadline, their reviews are going to automatically suffer. I suggested this myself in a mass email one time, and received a few angry responses for my mistake.

Above all, don’t try to shame your reviewers, and don’t include a papers chair or editor to lend weight to your email (the latter may work, but will leave your reviewer bitter). Instead, appeal to their better nature: after all, your reviewers are likely authors as well, and every author appreciates getting timely reviews. It never hurts to remind them of this and their overall role in the review process. Also, once the review deadline has passed and your reviewers are officially late, you are forgiven for turning up the heat on your entreaties. At this point, the reviewers are slowing down the whole process. I typically ask whether there is something preventing my reviewers from completing their work, and offering to reallocate the review if that is the case. The latter approach often does the trick; no one likes to make themselves look bad by handing over a professional task to someone else.

One more issue remains, perhaps one of the more frustrating among the interactions between you and your external reviewers: when your would-be reviewer never responds to your review invitation in the first place. The problem is that this leaves you in limbo, as the non-responding reviewer is holding up the entire process. In this situation, I will often wait a few days, and then send a personal email asking whether they had seen the invitation. After all, sometimes these automated invitation emails from review systems end up in spam folders, whereas a personalized email has a greater chance of reaching them. If they still don’t respond after a short few days, you should probably disinvite them and find an alternate reviewer. Perhaps they saw your invitation and, depending on the review system, have already started reading, but non-communicative reviewers are just not helpful. It is better to cut your losses and find someone else that is more reliable as soon as possible.

Final Words

Whew, that was a lot more detail than I had originally envisioned when I started this post! Most of it is, as always, common sense. Regardless, hopefully this is useful to someone out there, perhaps to newcomers who have just accepted their first program committee or editorial board invitation. Furthermore, your mileage may vary, so please only take my ruminations here as guidance rather than some definite word or policy. After all, this is just one person’s viewpoint.

Originally published in July 2018 at https://sites.umiacs.umd.edu/elm/.