Detailed instructions for Area Chairs

Primary vs Secondary area chair

All papers for which you are either Primary or Secondary AC are listed on CMT under the Meta-reviewer role, by selecting "Manage Assignments, Bids and Conflicts".

In the column “Assigned Meta-Reviewer”, the first name listed is the Primary AC, the second name listed is the Secondary AC.

Responsibilities

• Both ACs must manually assign 1 reviewer for the paper.
• The Primary AC must ensure that all reviewers are submitted on time, or if necessary request additional reviews.
• The Primary AC leads the discussion with PC members.
• The Primary AC must write a meta-review.
• The Secondary AC writes a meta-review for each paper not decided on unanimously by PC members.

Load

Most ACs have been assigned 16–20 papers as Primary AC, and a similar number as Secondary AC. Numbers vary based on the number of submissions per subject area, and conflict constraints.

Selecting reviewers

This is done on CMT, under the Meta-reviewer role, by selecting "Manage Assignments, Bids and Conflicts".

You should see a list of candidate PC members, with information about their bidding preference, their ranking according to the Toronto paper matching system, and their relevance score (overlap of keyword between the paper and reviewer).

You can sort either by suggested rank or relevance score (we recommend the former —just click on the column heading). You should also be able to see what PC members have already been assigned. Do not remove these assignments.

You should try to select PC members that offer complementary expertise and background (e.g., avoid reviewers from the same lab). You should also allow for a mix of seniority levels (e.g., grad students / faculty).

Note that each PC member's load has been capped at 7 papers (or less in some cases). If the ideal reviewer for a paper has reached his/her quota, you can email us to find a solution. You can also email us to add a new reviewer into the system (you should contact them first to confirm that they are willing, but please avoid forwarding submissions by email.)

Watch out for conflicts. Send us email if you are in conflict with a paper, or notice that a PC member with a conflict has been assigned. CMT prunes the reviewer set based on declared conflicts (e.g., domain-based), but many conflicts go undetected. A conflict between an author and a reviewer occurs when (1) they have had an active collaboration (project, paper, grant, co-supervision) in the last 3 years; (2) they worked at the same institution/department in the last 3 years; (3) one is, or was at any previous time, the thesis supervisor of the other; (4) they have a close personal relationship.

Collecting reviews

You must ensure that you have 3 good reviews for each paper for which you are Primary AC. In the case of very weak papers, it is ok for some of these reviews to be provided by the ACs.

Program committee members have been instructed to submit their first two reviews by next week, and the full set by April 3. We will be sending reminders to PC members as the deadline approaches. However it is often helpful if the supervising AC sends a more personal reminder. If you suspect that a review will not be provided in time, you should seek an alternate reviewer.

All reviews must be in the system for the author response period, April 9–12.

Reviewer criteria are described in detail on the Detailed Instructions for Program Committee members.

Author response and discussion

The author response period is happening April 9–12. Authors have been instructed to keep their responses brief (<4000 characters), and focused on correcting factual errors in the reviews, and answering questions raised in the reviewer comments.

We are still seeking a few outstanding reviews (<1%) as the author response period is starting. We are also adding extra reviews for those papers where the original 3 reviews have low confidence.

This year, authors are given the option of uploading a new version of their paper during the author response period. This has been used in a few other conferences (e.g., AIStats), and can be useful to clearly fix errors (e.g., mathematical notation) and help authors prepare a more polished draft. We encourage the program committee to consider all material available when making a decision, but it is optional similar to supplementary material.

Discussion on some papers started before the end of the author response period. This can be useful to clarify the questions to authors. But in general we favor having the discussion once all relevant information is available, including author response.

The discussion period is happening April 13–20. The role of the primary area chair here is to prompt reviewers to clarify the key contributions and rule out any misunderstanding. When the program committee has a unanimous opinion on a paper, area chairs should accept their decision. When reviewers disagree after discussion, both area chairs for a paper should make an accept or reject decision (ideally independently). When area chairs disagree, the program chairs will become directly involved.

It's not necessary to reach a consensus among either reviewers or area chairs, but it is important to send a clear message to the authors about why their paper was accepted or rejected.

If you find any part of the above process plausibly generating a mistake, bring it to the attention of the program chairs.

Final recommendation and meta-reviews

Area chairs need to write a meta-review for each paper for which they are the primary AC, due April 23rd. The meta-review must make a clear recommendation for accept/reject, and outline the reasons for this recommendation.

In the case of borderline papers (where reviewers disagree), a meta-review by the secondary AC is also needed.