Meta-Reviewer FAQ
(Please see also the Reviewer FAQ!)
Adjusting Review Assignments:
-- How do I add a new reviewer to the system?
-- How do I adjust the reviewer assignments for my papers?
-- What are expert reviewers and how should I use them?
Discussion and Author Response:
-- Do authors know who I am or who the reviewers are?
-- Do reviewers know who other reviewers are?
Making Decision and Meta-Reviews:
-- What is the timeline for decision-making?
Adjusting Review Assignments
-- How do I add a new reviewer to the system?
From your meta-reviewer console, click on “Reviewer” next to “Submissions” on the upper left corner, followed by Actions on the upper right corner, and then “Add Reviewer”. You can now add a reviewer by their email, and assign them the paper.
If you add a new reviewer to the system, please verify that they are willing to review the paper you assign them. Please also send us their email and desired quota, so that we can set it appropriately.
-- How do I adjust the reviewer assignments for my papers?
For each paper, click on “More -> Edit Assignments” to see the list of all possible reviewers for the paper. Note that the great majority of reviewers have a maximum quota of 7 papers, although some will have stipulated fewer papers. The sooner you make your assignments, the larger the pool you will be able to draw on. Once a reviewer is at quota, you will not be able to assign additional papers to them.
Adding a new reviewer not already in the ICML pool is allowed, but before adding a new reviewer, you should first confirm that your proposed reviewer is willing to review the paper. To add a reviewer, click on the “Reviewers” tab in the upper left corner, and then the “Actions” button on the upper right, followed by “Add Reviewer”. When you add a reviewer, please email us the name, affiliation and email of any reviewer you add to the pool so that we can set their review quota appropriately.
-- What are expert reviewers and how should I use them?
We have recruited around 100 "expert reviewers" who can help you in cases where there is some specific reason you need a short turn-around answer on a difficult question (in general, we do not expect expert reviewers to write full reviews). Because there are ~300 meta-reviewers, please use expert reviewers judiciously, perhaps only on one paper in your batch. You may continue to work with the expert reviewer until the decision/meta-review deadlines.
To add an expert reviewer for a paper: (1) click "Edit -> Assignments"; (2) under "User Type" select "Expert Reviewer"; (3) find an expert who is not yet at quota and email them to get their agreement; (4) assign them through CMT; (5) inform them by email specifically what question(s) you would like them to focus on in the paper.
Discussion and Author Response
-- Do authors know who I am or who the reviewers are?
No, the review process is double-blind and your identity, as well as the identity of reviewers, is hidden from authors.
-- Do reviewers know who other reviewers are?
No, only you (and the program chairs) know who the reviewers are. As such, please do not refer to reviewers by name in your discussion or in your meta-review.
Making Decision and Meta-Reviews
-- What is the timeline for decision-making?
- May 5: reviewers will have finalized their reviews (you will need to prod them to do this) and the discussion period will be finished.
- May 8: meta-reviewers will need to provide decisions and meta-reviews for "clear" cases (both clear accepts and clear rejects); we estimate this will be about 60-75% of the papers in your batch.
- May 15: meta-reviewers will need to provide decisions and meta-reviews for all papers, including a ranking of those that are not "clear" cases.
- May 25: the program chairs will finalize all decisions and relay those decision back to the meta-reviewers
- May 29: meta-reviewers will be able to update meta-reviews until this date (though hopefully they're already finalized on May 15)
- June 1: authors will be notified of the decisions on their papers, together with updated reviews and meta-reviews