(Adapted in part from the NIPS 2014 and ICML 2015 reviewer instructions)
Executive Summary
- Your load is a maximum of 8 papers.
- Please write detailed and thoughtful reviews and answer all questions thoroughly.
- The deadline for reviews is March 18th. Borderline papers may get an extra review during the discussion phase.
- Please let us know if you identify dual-submissions that violate our policy. The policy is detailed in the author instructions.
- An excellent summary of “best practices” for reviewing can be found here: https://sites.umiacs.umd.edu/elm/2016/02/01/mistakes-reviewers-make/
Reviewing Timeline
- February 19: The reviewing process begins.
- March 18: Reviewing deadline. Reviews must be completed and entered into CMT.
- March 18 – March 25: Initial discussion period and possible additional reviews.
- March 25 – March 29: Author feedback period.
- March 29 – April 6: Reviewers respond to author feedback, and final discussion period between Reviewers and Area Chairs.
Introduction
Thank you for reviewing for ICML! Your help is vital to our community: the technical content of the program is largely determined by the efforts and comments of the reviewers. Below are instructions and guidelines that will help you write reviews efficiently and effectively.
Conference system access
All reviews must be entered electronically into the CMT system.
Reviewers may visit this site multiple times and revise their reviews as often as necessary before the reviewing deadline. When you were invited to become a reviewer, you should have been sent an email with instructions on how to login to the system. Use your email address as the login id; you can change your password using the forgotten password feature.
During the review period, you will probably get many emails sent from CMT (e.g., those telling you about paper assignments). Please make sure emails from CMT are not snagged by your spam filter!
Confidentiality
By viewing the papers, you agree that the ICML review process is confidential. Specifically, you agree not to use ideas and results from submitted papers in your work, research or grant proposals, unless and until that material appears in other publicly available formats, such as a technical report or as a published work. You also agree not to distribute submitted papers or the ideas in them to anyone unless approved by the program chairs.
Double blind reviewing
This year, we will continue to use double blind reviewing. The authors do not know the identity of the reviewers; this also holds for authors who are on the program committee. In addition, the reviewers do not know the identity of the authors. Note, however, that the area chairs do know the authors’ identities. This helps avoid accidental conflicts of interest.
Of course, double blind reviewing is not perfect: by searching the Internet, a reviewer may discover (or think he/she may have discovered) the identity of an author. We encourage you not to actively attempt to discover the identities of the authors. If you have good reason to suspect that a paper has been published in the past, you can go and search on the Internet, but we ask that you first completely read the paper. Also, based on the experience of other double-blind conferences, we caution reviewers that the assumed authors may not be the actual authors; multiple independent invention is common and different groups build on each others’ work.
If you believe that you have discovered the identity of the author, please let us know in the “Confidential comments to PC members” in your review (see below).
Supplementary material
Some papers include supplementary material. Submission of additional material is allowed: authors can submit up to 10 MB of material containing proofs, audio, images, video, or even data or source code.
Your responsibility as a reviewer is to read and judge the main paper. Reading the supplementary material is optional. However, keeping in mind the space constraints of an ICML paper, you may want to consider looking at the supplementary material before complaining that the authors did not provide a fully rigorous proof of their theorems, or only demonstrated qualitative results on a small number of examples.
Formatting issues
We ask that you double-check that your papers have followed the submission guidelines with respect to length (i.e. they are at most 10 pages, where the ninth and tenth page must contain only references), format, and anonymity, and that they are not joke papers! Please notify the area chair responsible for the paper if you find serious formatting issues or anonymity problems with a submitted paper.
Previously published work and dual-submissions
Where possible, reviewers should identify submissions that are very similar (or identical) to versions that have been previously published, or that have been submitted in parallel to other conferences. A clarification of this policy is given in the Author and Submission Instructions. If you detect violations of the dual submission policy please note this in the “Confidential comments to PC members” section of the review form.
Writing your reviews: Review Content
We ask that reviewers pay particular attention to the question: does the paper add value to the ICML community? We would prefer to see solid, technical papers that explore new territories or point out new directions for research, as opposed to ones that may advance the state of the art, but only incrementally. A solid paper that heads in a new and interesting direction is taking a risk that we want to reward.
The high quality of ICML depends on having a complete set of reviews for each paper. Reviewer scores and comments provide the primary input used by the program committee to judge the quality of submitted papers. Far more than any other factor, reviewers determine the scientific content of the conference. However, we also stress that short superficial reviews that venture uninformed opinions about a paper are damaging. They may result in the rejection of a high quality paper that the reviewer simply failed to understand. Please take the time to fully assess the paper.
Reviewer comments have two purposes: to provide feedback to authors, and to provide input to the program committee. Reviewer comments to authors whose papers are rejected will help them understand how ICML papers are rated and how they might improve their submissions in the future. Reviewer comments to authors whose papers are accepted will help them improve the paper for the final conference proceedings. Reviewer comments to the program committee are the basis on which accept/reject decisions are made. Your comments are seen by the area chair and the other reviewers. Please do express your honest opinions. However please also make sure that your comments are considerate; this will help maximize the effectiveness of the overall process, in particular, the author rebuttal phase.
Content of the Review
The goal of the review is to give the community a better sense of the quality of the paper as a whole. You should discuss the strengths and weaknesses of each paper and address all the explicit criteria provided in the review form. Please read the review criteria and use those to guide your decisions. It is tempting to include only weaknesses in your review. However, it is important to also mention and take into account the strengths, as an informed decision needs to take both into account. It is particularly useful to include a list of arguments for and against acceptance.
All ICML papers should be good scientific papers, regardless of their specific area. We judge whether a paper is good using specific criteria; a reviewer should comment on all of these. We give specific instructions for how to fill out each section in the review form. This year, the review form contains the following sections:
(1) Summary of the Paper
As a whole, your written review should begin by summarizing the main ideas of each paper and relating these ideas to previous work at ICML and elsewhere. While this part of the review may not provide much new information to authors, it is invaluable to members of the program committee, and it demonstrates to the authors that you understand their paper. Please write your summary in your own words and avoid using phrases from the abstract or the paper itself.
(2) Paper Clarity
In this section you should assess the clarity of the presentation and reproducibility of the results. Feel free to make suggestions to improve the manuscript.
Is the paper clearly written? Is it well-organized? Is the result section or are the proofs clear enough so that expert readers will understand them and can reproduce the results (if applicable)?
Excellent (Easy to follow): An exceptionally clear paper. Much of this paper would be accessible even to someone outside the field of machine learning.
Above Average: A good paper, clearly written. Accessible to most machine learning researchers.
Below Average: A decent paper, but could be written more clearly. Likely to be accessible only to those in similar areas.
Poor (Hard to follow): The paper is unclear. Even people working in similar areas of machine learning will have a difficult time understanding / reproducing this papers’ contributions.
(3) Paper Significance
In this section you should assess the importance (potential impact) of this submission and its relation to existing work. Consider checking the proceedings of recent machine learning conferences to make sure that each paper is significantly different from papers in previous proceedings.
Does the paper contribute a major breakthrough or an incremental advance?
Are other people (practitioners or researchers) likely to use these ideas or build on them? Does the paper address a difficult problem in a better way than previous research?
Is it clear how this work differs from previous contributions and is related work adequately referenced?
Excellent (substantial, novel contribution): This paper makes a contribution that could become widely known or adopted within the community.
Above Average: This paper makes a strong contribution that supports its publication.
Below Average: This paper makes an incremental contribution, and may be useful to only a small number of people.
Poor (minimal or no contribution): This paper makes little or no contribution.
(4) Detailed comments.
In this section you should address the soundness of the paper.
Are claims well-supported by theoretical analysis or experimental results? Is this a complete piece of work, or merely a position paper? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work?
Please also provide any additional comments to the authors and explain the basis for your ratings while providing constructive feedback.
Finally, please provide a short 1-2 sentence summary of your review.
(5) Overall Rating
Reviewers choose an overall rating between four classes for each paper. The program committee will interpret these classes in the following way:
Strong accept: An excellent paper, well above the acceptance threshold. I vote and argue for acceptance.
Weak accept: A good paper, above the acceptance threshold. I vote for acceptance, although would not be upset if it were rejected.
Weak reject: A decent paper, but just below the acceptance threshold. I vote for rejecting it, although would not be upset if it were accepted.
Strong reject: A paper below the acceptance threshold. I vote and argue for rejection.
(6) Confidence Score
Reviewers also choose a confidence rating between three classes for each paper. The program committee will interpret these classes in the following way:
Reviewer is an expert: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature.
Reviewer is knowledgeable: The reviewer is fairly confident that the evaluation is correct. It is possible that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature. Mathematics and other details were not carefully checked.
Reviewer’s evaluation is an educated guess: Either the paper is not in the reviewer’s area, or it was extremely difficult to understand.
(7) Confidential Comments
This section should include information that you wish only the program committee to see. The confidential comments to the program committee have many uses. Reviewers can use this section to make recommendations for oral versus poster presentations, to make explicit comparisons of the paper under review to other submitted papers, and to disclose conflicts of interest that may have emerged in the days before the reviewing deadline. You can also use this section to provide criticisms that are more bluntly stated.
Best reviewing practices
Importantly, reviewer comments should be detailed, specific and polite, avoiding vague complaints and providing appropriate citations if authors are unaware of relevant work. As you write a review, think of the types of reviews that you like to get for your papers. Even negative reviews can be polite and constructive! Remember that you are assessing the paper’s quality as a scientific contribution to the field.
Reviewers should NOT assume that they have received an unbiased sample of papers, nor should they adjust their scores to achieve an artificial balance of high and low scores. Scores should reflect absolute judgments of the contributions made by each paper.
An excellent summary of “best practices” for reviewing can be found here: https://sites.umiacs.umd.edu/elm/2016/02/01/mistakes-reviewers-make/
Author feedback and reviewer consensus
Between March 25 and March 29, authors will have a chance to submit feedback on their reviews. This is an opportunity to correct possible misunderstandings about the contents of the paper, or about previous work. Authors may point out aspects of the paper that you missed, or disagree with your review.
It is important to convey to the authors that their comments were read, even if they do not change the final evaluation of the paper. Therefore, please read each rebuttal carefully and keep an open mind. Do the authors’ comments make you change your mind about your review? Have you overlooked something? There is a separate box for reviewers to write responses to the author feedback. Please use this to let the authors know that you read and absorbed their rebuttal, and whether it swayed you one way or the other.
From March 18 to April 6, the area chairs will, where necessary, lead a discussion via the website and try to come to a consensus amongst the reviewers. The discussion will involve both marginal papers, trying to reach a decision on which side of the bar they should fall, and controversial papers, where the reviewers disagree. Many papers fall into these categories, and therefore this phase is a very important one. While engaging in the discussion, recall that different people have somewhat different points of view, and may come to different conclusions about a paper. It may be helpful to ask yourself “do the other reviewers’ comments make sense?”, and “should I change my mind given what the others are saying?” Reviewer consensus is valuable, but is not mandatory. If the reviewers do come to a consensus, the program committee takes it very seriously; only rarely is a unanimous recommendation overruled. However, we do not require conformity: if you think the other reviewers are not correct, you are not required to change your mind.
Conflicts of interest
In reviewing, you may accidentally discover the identity of the authors. If this happens, please contact your area chair immediately.
Again, we thank you for your help so far and in the future. Your carefully considered input is crucial to the success of the conference.
Nina Balcan and Kilian Weinberger, ICML 2016 Program Co-chairs
icml2016pc@gmail.com