Detailed instructions for Program Committee members
CMT
The reviewing process will be done electronically using CMT:
Confidentiality
All ICML submissions must be considered confidential. This means that you cannot forward them, or distribute in any way. You also cannot use the ideas presented in your own work.
We are using a double-blind reviewing process. This means that reviewers are not aware of the identity of the authors, and similarly authors are not aware of the identity of the reviewers for their paper. In this day and age, it is often easy to discover the identity of an author. You should avoid this as much as possible, and evaluate the paper as submitted. If you accidentally discover the authors’ identity, you should go ahead with reviewing, unless you think this information significantly affects your evaluation, or you notice a conflict of interest.
Note that program committee members can see each others’ identity during the discussion process, including viewing the other reviews.
Reviewing instructions
Your review has two intended audiences: the authors, and the rest of the program committee. Try to provide helpful points for both. For the authors, you should give constructive feedback on the content of the paper. For the area and program chairs, you should summarize the major strengths and weaknesses of the paper. Here are the main evaluation criteria.
Every accepted paper should be correct. It is your responsibility to verify this, or comment on why this is difficult to assess. Minor errors should not necessarily disqualify a paper —if you can see how to fix minor errors, include them in your review and discount them when considering the accept/reject decision. Authors had the opportunity to upload supplementary material. You are not required to consider this, but you are encouraged to consult it, if it helps evaluate the main paper.
Every accepted paper should be about new research. This can include new theoretical, algorithmic, and empirical insights. If you have evidence that the paper has been published in the past, you should raise your concerns in the review. Note that in a few cases, this is allowed according to our double submission guide; you will receive an email if you are reviewing a paper that falls under one of those guidelines.
Every accepted paper should be interesting to the right audience. A good paper should present results that can be used by others, whether to solve existing problems, or to develop new ideas. The intended audience need not traditionally attend ICML, but you should clearly discuss when this is so.
Every accepted paper should be written clearly enough that the result is understandable, and ideally reproducible. Adequate references should be provided. For correctable grammar errors or complementary related work references please provide feedback and discount these in your consideration.
Watch out for Application papers. ICML is mostly a machine learning technology conference. Since technology developed in a vacuum often goes astray, we want to keep ICML connected to real applications of the technology. Good application papers may not have new ML technology. Look for novelty in the application itself, how well the problem is framed, and the degree of success.
All reviews are due April 3. We have a tight reviewing schedule, please respect this deadline! If you are unable to provide a review, please contact the primary area chair as soon as possible.
Author response and discussion
The author response period is happening April 9–12. Authors have been instructed to keep their responses brief (<4000 characters), and focused on correcting factual errors in the reviews, and answering questions raised in the reviewer comments.
This year, authors are given the option of uploading a new version of their paper during the author response period. This has been used in a few other conferences (e.g., AIStats), and can be useful to clearly fix errors (e.g., mathematical notation) and help authors prepare a more polished draft. We encourage the program committee to consider all material available when making a decision, but it is optional similar to supplementary material.
Discussion on some papers started before the end of the author response period. This can be useful to clarify the questions to authors. But in general we favor having the discussion once all relevant information is available, including author response.
The discussion period is happening April 13–20. The role of the program committee here is to identify the key contributions, and make a recommendation about acceptance. It's not necessary to reach a consensus among the reviewers, though it is usually preferable, both to send a clear message to the authors, and because the process of reaching consensus can shed important light on the key issues.
When the reviewers are able to reach a consensus, the primary area chair should make a recommendation consistent with the PC's decision. When the reviewers cannot reach a consensus, the secondary area chair will assist in making the final recommendation.
It is always possible to revise reviews throughout the discussion process. You should use this to reflect any important points raised in the discussion. The authors will receive the final version of reviews, along with the decision, on April 30th.
If any part of this decision process appears to be making a mistake, point it out to the program chairs.