What Happens in the InfoVis Reviewing Process?

Reflections on IEEE InfoVis 2016 from the Papers Co-Chairs.

Note: This is a new import of an old blog post from 2016.

• Niklas Elmqvist, University of Maryland, College Park
• Bongshin Lee, Microsoft Research
• Kwan-Liu Ma, University of California, Davis

When talking to people in the InfoVis community, we have noticed that many are confused or curious about the InfoVis reviewing process. While most of the community is engaged in writing external reviews-which is something everyone should do, not merely as a favor to the community but also as quid pro quo for submitting their own work for review-only more senior people are actually part of the InfoVis program committee (PC), and fewer still ever become papers co-chairs. Our goal with this article is to make the process a little more transparent by describing what actually goes on in the program committee and in meetings between the papers co-chairs and the TVCG leadership.

Note that this narrative is based on what happened during the InfoVis 2016 process, and that this may or may not be consistent with other years. Furthermore, the process we describe here is our personal perception of what happened. In other words, this is not an official policy document of IEEE InfoVis, and it may be inaccurate or incomplete. Your mileage may vary.

InfoVis does not have a physical program committee meeting. This means that essentially all of the reviewing happens electronically (we say essentially, because papers co-chairs have been known to meet physically for their deliberations) supported by web-based review systems such as PCS. This practice is quite unlike some older (and larger) conferences such as ACM CHI and ACM SIGGRAPH, which have physical meetings where the entire program committee comes together to make their final deliberations. It is significant because it means that InfoVis makes its final accept and reject decisions based almost solely on discussions conducted in reviews and online forums. While physical program committee meetings have their own set of problems, they do make for a more open and transparent process that involves more people. In contrast, final InfoVis decisions are made solely by the papers co-chairs, although they are obviously guided by the PC and external reviewers as well as overseen by the TVCG leadership.

The main roles in the reviewing process are the external reviewers, the program committee, and the papers co-chairs. After talking about each of these roles, we describe the timeline and sequence of the process itself.

The majority of the work in the InfoVis reviewing process is performed by external (or tertiary) reviewers who are invited by program committee members to review a specific submission. The external reviewers are the unsung heroes who closely read the submissions they are assigned and write detailed, constructive reviews of each one. Their efforts are vital to the entire process; more than 160 papers were submitted to InfoVis 2016, which means that some 670 external reviews from more than 300 reviewers had to be solicited from the community to make the process work.

Volunteering as an external reviewer is the easiest (and first) step you can take to become involved in the InfoVis reviewing process. Just go to the PCS website for IEEE VGTC ( http://precisionconference.com/~vgtc ), create an account if you don’t have one yet, and volunteer as a reviewer. You will be asked to agree to ‘Reviewing Agreement’ statements, which includes the VGTC ethics guidelines . You will be also asked to provide your past reviewing experience, your technical expertise, and your willingness to review a specific number of submissions for a particular venue. While it may take a while for you to get your first review invitations, you will eventually be asked to help out in reviewing a paper, and the frequency of invitations will increase as you start to build a name for yourself as a reliable and fair reviewer.

The InfoVis program committee (PC) consists of people with a proven track record of working and publishing in InfoVis and related conferences. The absolute majority of PC members already have their Ph.D., but it is not unheard of to have accomplished practitioners (or even senior Ph.D. students) serve on the committee. Each PC member is invited to serve on the committee for a maximum of three year, after which they are required to rotate off the PC for at least one year. New PC members are selected by the InfoVis papers chairs so that the committee as a whole is diverse in terms of gender, seniority, expertise, industry vs. academia, geographic region, etc. The new PC members are then approved by the InfoVis steering committee (SC). Most new PC members are selected from the pool of external reviewers who also have a publication record at InfoVis, so good and consistent performance as an external reviewer as well as regular research contributions to the conference is the best way to eventually be invited into the program committee.

What does the program committee do? The main tasks of each PC member include (1) inviting external reviewers for their assigned papers, (2) writing their own reviews, (3) encouraging and moderating discussion between reviewers for each submission, (4) writing a summary review for each paper they serve as a primary reviewer, and (5) confirming that the second round version of a paper conforms to the required changes specified by the reviewers for conditional acceptance.

Instead of being summarily assigned to review specific papers, PC members get to bid for which papers they want, are qualified, and are not conflicted to review. This bidding happens during the period between abstract and full submission deadlines. It is also the reason why it is important that abstract submissions include full author information so that PC members can avoid conflicts of interest during bidding. After the full submission deadline, PC members get assigned approximately 6–8 papers, usually half as a primary reviewer and the other half as a secondary reviewer. The only difference between these two roles is that only the primary reviewer is responsible for also writing a summary review that reflects the viewpoint of all of the reviewers. The primary reviewer is also in charge of launching the discussion among the reviewers as well as checking that the second round version of a paper fulfills all of the criteria required for final acceptance.

The papers chairs are ultimately responsible for the overall reviewing process. While there are hundreds of external reviewers for InfoVis, and 59 program committee members in 2016, there were only three papers co-chairs. Their main responsibilities include (1) forming the program committee, (2) assigning submissions to PC members, and (3) making the final acceptance and rejection decisions based on reviews, discussions between reviewers, and summaries provided by the primary reviewer. An overarching responsibility for the papers chairs is to maintain the overall quality of the InfoVis technical program and to communicate with the VIS organizing committee, the InfoVis steering committee, and the TVCG leadership.

Describing the selection process for papers chairs is outside the scope of this document, except to say that each chair is appointed by the InfoVis steering committee and generally serves two years in the position: first as junior co-chair and then as senior co-chair. This ensures continuity and preserves institutional memory in the position, which is important since the papers chair position has many complex and important responsibilities. Currently, the InfoVis system is that most papers chairs first serve two years as poster chair before becoming papers chair.

Of the three main tasks that papers chairs perform-forming the committee, assigning papers to PC members, and making final decisions-the final deliberations for accept and reject decisions is by far the most opaque and also the most important from the perspective of outside observers. However, this task has strong dependencies to the earlier tasks: the composition of the committee governs which people can be assigned to specific submissions, which in turn impacts the external reviewers that are invited, as well as the expertise that the PC members bring to bear on each submission. Papers chairs actually make very few uniquely own decisions. Ultimately, it is the external reviewers and the PC members who have the most influence on paper decisions through their combined ratings and review contents. It is only for truly borderline submissions that the papers chairs need to make judgment calls on which paper to accept or reject. Even then, borderline cases are often resolved by inviting additional (and often senior and expert) reviewers to write “crash reviews” in a very short amount of time.

Papers chairs work under many conflicting constraints. On the one hand, the papers chairs are the final stewards of the quality of the InfoVis conference for that particular year and must ensure that only valid and high-quality submissions are accepted. This is made more challenging by the fact that their work is overseen by the IEEE TVCG editor-in-chief (EiC) and associate EiCs, who are charged with maintaining the quality and the acceptance rate of the journal as a whole. However, on the other hand, the papers chairs also endeavor to accept as many as possible valid submissions to build an interesting, exciting, and potentially provocative technical program while fully considering the feedback provided by the external reviewers and PC members in their reviews. In addition to this, every papers chair naturally has their own expertise, preferences, and opinions that they bring to bear on paper decisions. The checks and balances built into the system are designed to prevent any of these factors from becoming too significant.

The figure at the top of this article shows an overall timeline of the InfoVis review process from start to end. Here are the main milestones:

• Abstract deadline: Authors submit their abstracts, including full title, author, affiliation, and keyword information. It is important that this information is complete to allow the papers chairs and the program committee to start bidding and allocating papers.
• Conflicts of interest: Each PC member provides information about authors and affiliations they are conflicted with based on the guidance provided in the IEEE VGTC ethics guidelines .
• Bidding: Here, PC members express their reviewing preference for each submitted abstract based on the title, abstract text, authors, and keywords. The understanding is that PC members should primarily be allocated papers they actively want or at least would be willing to review.
• Full paper deadline: The major submission deadline is when the full papers are submitted, including all of the (optional) supplemental material. Right after the paper submission deadline is passed, the papers co-chairs eliminate submissions that are incomplete (e.g., have no authors).
• Desk rejections: Before any papers are allocated to PC members and sent out for review, the papers co-chairs will go through all of the submissions, eliminate the incomplete ones without a submission document (this happens often), and identify papers for desk-rejection. Desk-rejected papers are those that are out of scope for the conference, clearly incomplete, or of such low quality that the papers co-chairs deem them not ready for peer review. Furthermore, papers that do not conform to formatting or submission requirements can be desk-rejected at this stage. The corresponding author will receive a notification email at this stage if their paper is desk-rejected.
• Paper allocations: Satisfied that all submissions are of sufficient quality, the papers co-chairs then allocate the papers to the program committee based on reviewing preferences. The allocation is done so that PC members get as many of their preferred papers as possible (which is not always 100% possible since some submissions are popular and others are less so). Each paper gets assigned one primary and one secondary reviewer from the program committee, and the allocation is then propagated to the committee.
• External review invitations: Now the ball is in the court of the program committee, whose members have to recruit at least one external reviewer per submission (in 2016). This means that each submission has at least four reviewers in total: one primary, one secondary, and at least two external (one each invited by the primary and secondary). Recruiting reviewers has several constraints and guidelines, including avoiding using several reviewers from the same affiliation for the same paper, finding people with the right expertise, and avoiding conflicts of interest.
• External reviews: The most labor-intensive part of the process is where all of the reviewers-primaries, secondaries, and externals alike-read their assigned papers, view the supplemental material, and write their reviews. All reviews are due by a specified deadline established by the papers co-chairs to fit the reviewing timeline.
• Discussions: Once all reviews are completed (and the PC members have hounded their external reviewers and other PC memners to finish them), the reviewing moves to the discussion phase. Prior to this phase, external reviewers are unable to see each other’s reviews, presumably to avoid premature calibration. Now, the primary reviewer is charged with informally summarizing the main points and encouraging the reviewers to discuss any inconsistencies or missing parts in their collective reviews. Enterprising primaries often post a draft of their summary reviews (see below) for discussion by the review panel.
• Summary reviews: The only difference between a primary and a secondary reviewer (beyond starting the discussion) is that the primary reviewer is responsible for writing the summary review for each submission. The summary review condenses the main points of each of the individual reviews and highlights the important ones. In particular, for papers that are borderline or likely to be recommended for acceptance, the primary is asked to provide a clear list of required changes that the authors must make as a condition for acceptance. The understanding is that if authors address all of the required changes to the satisfaction of the primary reviewer (which can be difficult!), the paper should be accepted.
• Accept/reject decisions: Once the deadline for summary reviews rolls around, the ball comes back into the court of the papers co-chairs. This is also the most difficult and labor-intensive part of the process of the papers co-chairs, because they have to look at all of the submissions and their reviews and make individual acceptance or rejection decisions for each of them. The specific process will vary widely, but typically involves focusing on a few difficult borderline cases, whereas the clear accept or clear reject (based on the reviews) are relatively easy decisions. In addition, since InfoVis proceedings are a special issue of IEEE TVCG, the TVCG EiC is involved in approving all papers that the papers co-chairs recommend for acceptance (in 2016, the EiC approved all of the papers the co-chairs recommended for acceptance). Once this process is complete, the conditional accept and rejection notification go out to authors.
• Second-round submission: All InfoVis acceptances are conditional, which means that authors are required to submit a second round version where they have addressed all of the required changes imposed by the primary reviewer.
• Final accept/reject decisions: Once the authors submit their second-round version, the primary reviewer is asked to look over the new revision and ensure that the authors have made all of the required changes. The primary reviewer has the opportunity to contact the authors directly to communicate specific instructions or requests; they may also choose not to contact the authors. At the end of this process, the primary is asked to give a final accept or rejection recommendation. The papers co-chairs takes this recommendation into account when making their final decision (again with the approval from TVCG). While it is uncommon, it is very possible that conditionally accepted papers are rejected at this stage if the authors are unable (or unwilling) to make the required changes imposed by the reviewers (led by the primary). Accepted papers, on the other hand, have no more hoops to jump through and receive the green light for publication and presentation at the conference.
• Camera-ready submission: The absolute last stage of the InfoVis review process is when authors submit their camera-ready versions, including all of their supplemental material. At this stage, only the papers co-chairs will check the submissions, and only for those that received special instructions on things to change. This is also when authors submit copyright release forms and other material required by the IEEE.

Hopefully, this document will have cleared up some of the mysteries of the InfoVis review process. The very best way to learn more about the process (and influence it!) is to get involved in it yourself! As stated earlier in this document, the starting point for most people is to volunteer as an external reviewer. If you work hard to write useful and constructive reviews and also publish papers in the conference, you will eventually be identified as a reliable and knowledgeable person and given additional responsibilities. Good luck!

Originally published at https://sites.umiacs.umd.edu.