Skip to yearly menu bar Skip to main content


Review Form

"Review the papers of others as you would wish your own to be reviewed"

Background: We share here the review form to be used with ICML 2022 with some extra explanations in order to help both authors and the PC members. More information about "how to be a good reviewer" can be found here. All reviewers are expected to adhere to the principles mentioned there.

The review is written for the Meta-Reviewer (MR) and the authors. The MR wants to see that the reviewer understood the paper (or how much they understood), and they also want to get the opinion of the reviewer about relevance, soundness, novelty, completeness and the quality of writing/presentation. The authors want to know whether there are any ways to improve the paper. Eventually, the paper needs to serve the readers: they want new, interesting, correct results, or new insights that are well supported and written up in an easy-to-read paper. The reviewing process should be a collaborative process where all participants work together towards ICML publishing papers that readers will find useful, interesting and a pleasure to read. 

Each aspect listed below should be evaluated on its own merit. However, if a paper has a major flaw, it is unnecessary to list all the issues (e.g., issues with writing). However, use this carefully: It is unfair to bring up major issues after the reviews have been released to the authors. In other words, the review needs to have complete information about the possible concerns with the paper before it is released to the authors. Also, it is essential to explain which issues are considered major and which are considered minor.

Summarize the contributions made in the paper with your own words. Aim for precision and conciseness. This part of the review serves the purpose of showing to the MR and the authors how much you understood of the paper and what you think the paper is about. This is not the point to evaluate the contributions for their strengths or correctness. Merely provide a summary of what the contributions are.

Novelty, relevance, significance. Assuming the contributions of the paper stand, are they relevant for our community? Are they new? A precise justification is needed if the answer is no (or partially no, e.g., citations of precise results in earlier papers), so that the authors know how to fix the paper if it is fixable. Are the results sufficiently interesting to make this a sufficiently “complete” (or, one may say, significant) paper? Note that significance does not necessarily mean solving a major open problem. Small, interesting results can also be significant. Science is incremental. But there has to be a detectable, positive increment of sufficient interest. Regarding relevance, keep in mind that the machine learning community has traditionally been quite open-minded to new ideas from different areas. So, think whether this result could benefit some sub-area of ML, or a part of the community, down the line.

Soundness. A paper ideally makes claims, which should be well supported, either by theoretical arguments, or by experimental results. Either say, the paper is sound, or list the problems. Only list major problems. Any problem listed needs a justification: do not just say that a result is incorrect, include an explanation of why you think it is incorrect. For example, a proof may have some gaps, an experiment may fail to support a claim because of its design or outcome (or the lack of its outcome). For experimental papers, the paper may fail to use a sound experimental design (e.g., the data collection may have problems).

Quality of writing/presentation. Is the paper well organized and clearly written? Does it do a good job at explaining the novelty and the results? Does the paper include enough information needed to support the claims it makes? There is an overlap here with soundness: Sometimes soundness cannot be decided if the claims in the paper are not well supported, sometimes it can still be decided (e.g. by the reviewer doing additional work; we do not expect though reviewers doing this work). In cases like this, note on the previous item that soundness cannot be decided for reasons explained under this heading. For experimental papers, it is a presentation issue if the paper did not include enough details to reproduce the experimental results with a reasonable effort. A superbly written experimental paper explains: (1) why were the experimental conditions selected the way they were selected; (2) the subsequent choices of what to measure and what to plot (including perhaps what is not shown); (3) how the results obtained substantiate the claims made. The paper should follow standard, best practices (e.g., includes error bars, better yet, uses box plots or something similar unless you suspect near Gaussian variables, etc.)

Literature. Is the paper appropriately placed into contemporary literature? If not, be specific about what is missing. Note that oftentimes it is a question of judgement of whether a result should be mentioned as papers are subject to page limits. The must-mention results are directly relevant to the topic of the paper. If you ask authors to include other papers, you will need to declare (privately, see below) whether you are an author of any of the recommended papers. It is OK to recommend relevant papers authored by yourself, however, it is not OK to recommend papers that are not relevant to be included. When in doubt, it is better to err on the side of not recommending your own papers (or ask the MR). It is not reasonable to expect a paper to refer to unpublished works that appeared within one month before the submission deadline. Concurrent works should be referred to, but cannot be held against the paper in terms of novelty. Often, these are delicate decisions and reviewers should consult their MR for guidance. 

Basis of review. Declare how much of the paper you read. E.g., “I read the full paper, including all the proofs.”. The goal is to ensure full coverage of all parts of the paper by the reviewers and the MR.

Summary

List the strong and weak points of the paper, but also provide further input to whether (and why) you think the strengths or the weaknesses are dominating. For each point, indicate the importance of the point at hand: is this a major (important, critical) strength/weakness, or a minor one?

When evaluating these points take into account that some things are easy, while others are harder to fix. Include a justification. Remember, that the goal is to publish innovative, interesting, correct, good papers. Could this paper be one of those worthy to be published at ICML?

In general, reviewers are not expected to make accept/reject recommendations, this will be the job of the MRs based on all the information they have, including this summary. [We make an exception to this in Phase 1, see below.]

Miscellaneous minor issues

List any typos, grammar, etc. issues which you view as minor but should be addressed in the final version of the paper

Declaration [visible to MR, SMR and PC, not to other reviewers or the authors]

In my review, I recommended a paper co-authored by myself to be cited in a revision of the paper. [Yes/No]

Phase 1 recommendation.

Should progress to phase 2: Yes/No

Only recommend no if you believe that the paper is not acceptable AND it cannot be fixed in simple ways. Easy to fix issues are: Few missing references, a few trivial improvements to the presentation, some fixes in proofs that are within reach, some extra experiments that are nice to have but not essential to publish the paper. On the latter point, if a paper would need substantially more experimental support, doing this is beyond the scope of the reviewing process. We do not expect authors to run extensive new experiments during the rebuttal process: Simply, there is no time for this, nor is there time to properly reevaluate the outcome of these experiments.

Note that this is a recommendation and the MR has the right to overwrite it. By default, papers with two negative recommendations are heading for rejection for Phase 1. MRs are asked to check the reviews and the papers that receive two negative recommendations and they can reverse this default decision. 

Do you have concerns regarding the ethics of the research presented in the paper? [Yes/No]

This will be used to flag the papers to go through a review by the ethics board.