Skip to yearly menu bar Skip to main content


ICML 2023 Reviewer Tutorial

 

Review the papers of others as you would wish your own to be reviewed

[With thanks to the ICML 2022 program committee for past instructions]

Background: The review is written for the Area Chair (AC), the authors and the research community. The AC wants to understand how well the reviewer understood the paper, and what the reviewer’s opinion is of the paper.  It is also a chance to give valuable feedback to the authors. Ultimately papers must serve the wider research community, providing new insights that can help advance the field. 

Policy on Large Language Models (LLMs) for ICML 2023 Reviewing: Given the above objectives, it is important that the opinion of the reviewer be expressed in the written review. It is acceptable to use large language models to copy edit or modify the language in a potential review, but the reviewer is ultimately responsible for their review submitted, its content, and its correctness. It is not acceptable for a reviewer to have a LLM generate the full review and submit it as if it is their own, just as it would not be acceptable to have someone else review a paper and submit it as one’s own review. Note, in addition, many LLM services may store and use the prompts and data submitted to them: in such cases it is not permitted to upload any part of authors’ paper submission, since these submissions are submitted assuming that their content will not be shared broadly before acceptance. [LLMs are a fast evolving technology and this policy may be revised in ICML 2024 or in future years].

For new reviewers, we encourage you to learn more about reviewing. Reviewing is an important research skill to cultivate, and, like all skills, can be learned. The following slide deck provides training and resources (please note that there is no separate Phase 1 and Phase 2 on the reviewing process for ICML 2023)

How to be a good reviewer-tutorials for ICML reviewers.pptx

Below we give some additional details and examples for a few aspects of the reviewer form: most other aspects will be largely self explanatory. Don’t hesitate to reach out to your AC or others if you have questions. 

 

Reviewer Form Details

Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary. Aim for precision and conciseness. This part of the review serves the purpose of showing to the MR and the authors how much you understood of the paper and what you think the paper is about. A few sentences suffices.

Examples:

“In this paper the authors extend the Carrot features framework, which has recently had great success in domain adaptation. The results show improvement over prior methods.”
– extend in what way? What problem does the paper address? What is the research question?


“This paper discusses that existing methods on including classifiers in a cGAN biases the generator in generating easy to classify images. Therefore, they propose a way to include classifiers in a cGAN to improve its performance in a principled manner. To do so, they decompose the joint probability distribution by the Bayes rule that results in linking classifiers to conditional discriminators. The proposed formulation shows that a joint generator model can be trained from two directions: a conditional discriminator and an un-conditional discriminator with a classifier. They combine the formulation of these two routes and propose a new method called Energy-based cGAN (ECGAN). ECGAN shows how to use a classifier for cGANs, and it explains other variants of cGANs such as ContraGAN, ACGAN, and ProjGAN as variations of its framework. They empirically show that ECGAN outperforms existing cGANs by achieving higher FID (and similar ones) score on two sets of datasets (CIFAR10, Tiny ImageNet).”
 

 

Please provide a thorough assessment of the strengths and weaknesses of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage people to be broad in their definitions of originality and significance. For example, originality may arise from creative combinations of existing ideas, application to a new domain, or removing restrictive assumptions from prior theoretical results. You can incorporate Markdown and Latex into your review. See https://openreview.net/faq. Note that significance does not necessarily mean solving a major open problem. Small, interesting results can also be significant. Science is incremental. But there has to be a detectable, positive increment of sufficient interest. 

In addition, a paper ideally makes claims, which should be well supported, either by theoretical arguments, or by experimental results. If there is a claim that is incorrect or not well supported, please include that in your review. For example: a proof may have some gaps, an experiment may fail to support a claim because of its design or outcome (or the lack of its outcome), or the work may not use a sound experimental design (e.g., the data collection, or hyperparameter selection may have problems).

Examples:

Novelty:
“This paper proposes a new model for doing X, and it beats the state of the art.” 

– not enough detail: what exactly is novel? What has not been done before?

“Representation learning of molecules has become an important problem in computational chemistry and pharmacy. Yet, a known problem is that existing methods cannot well represent structural motifs. Existing methods do …. but they suffer from high complexity. This paper is the first to connect the theory of spectral semi-snippets with graph representation learning. This theory allows to compactly represent motifs, and the paper demonstrates how to exploit this idea in a parametrized form, which, with some more re-writing, leads to a model that runs in linear instead of cubic time in the size of the graph. This idea is to my knowledge completely novel in the graph representation literature and may inspire further such models.” 

– this has more details and specific justification

“This result is well known, see reference [1]” 

– say what exactly is known, e.g., which part of the theorem and under which conditions, and where in the reference one can find it. It is very important to point out the exact references, too.
 

Soundness:

Review says: “Lemma 3 is wrong.” 

– where exactly is the mistake? Is it possibly fixable (the reviewer does not need to do long alternative derivations, but they may see an easy alternative route)? How does it affect the rest of the paper?
Instead try:

“The proof of Lemma 3 has a mistake. Equation (25) only holds for nonnegative numbers, but it is needed for all reals. This makes Lemma 3 only valid for x in the range [..]. Theorem 2 relies on Lemma 3, and then only holds in a limited range of [...]. Since Theorem 2 is the main result of the paper, this is a major point weakening the contribution of the paper.” 

Example on a to-the-point phrasing from a reviewer raising serious formulation issues (i.e., what to avoid from the authors’ perspective): 


I have to admit that I had some troubles in properly evaluating the algorithm because the problem it addresses is not formalized and thus we do not know what the paper is solving here.

Several points to clarify are :

  • what is the optimization problem the authors want to solve?
  • does the algorithm provide a solution to the exact problem or to an approximate one?
  • what are the hypotheses needed for convergence?
  • does the iterative procedures converge? why do we use an iterative procedure?
  • if it converges, does it converge towards a solution of the problem when T goes to infinity?
     

Ultimately, all good reviews substantiate their claims: it is important that you flesh out, as clearly as possible, the reasons motivating your assessment (whether positive or negative) of the submission.

Soundness. Please assign the paper a numerical rating on the following scale to indicate the soundness of the technical claims, experimental and research methodology and on whether the central claims of the paper are adequately supported with evidence.