ICML 2025 Reviewer Instructions

Thank you for serving on the ICML 2025 program committee, and welcome to the party!

Immediate TODO items:

Please make sure you are available in the key reviewing periods, and are able to meet the reviewing deadlines (see key dates below).
Please read the entirety of these reviewer instructions (this document), as well as these fun and informative Peer Reviewing memes.
Please carefully review the ethical conduct and use of generative AI policies (below).
The reviewer form has changed this year. Please go through the reviewer form (below) before beginning to review.

Responsibilities of Reviewers

The responsibilities of a reviewer for ICML are as follows:

Indicate your areas of research expertise, and “bid” on submissions to review.
Check your reviewing assignments and notify the overseeing area chair of any problems (e.g., conflicts of interest).
Carefully review the correctness and merits of the submissions you have been assigned. The review form this year is new and has specific items that you will be required to complete.
Read and acknowledge the Authors’ Responses.
Actively participate in discussions.

Key Dates

Bidding period: January 27–February 3, 2025
Full paper submission deadline: January 30, 2025
Submission assignment period: February 4–12, 2025
Reviewing period: February 13–March 13, 2025
Deadline for reviews: March 13, 2025
Authors’ response and discussion period: March 25–April 8, 2025
Deadline to acknowledge authors’ response: April 4, 2025
AC-reviewer discussion period: April 1–April 13
Author notification: May 1, 2025

Ethical Conduct for Peer Review

Members of the program committee, including reviewers, are expected to follow standard ethical conduct for peer review. In particular, ICML prohibits:

use of privileged information (e.g., information and discussions about submissions) for any purpose other than reviewing;
use of Generative AI tools in reviewing (as described below);
all forms of collusion, whether explicit or tacit (e.g., an arrangement between authors and reviewers, ACs, or SACs to obtain favorable reviews).

Use of Generative AI tools (such as LLMs) for reviewing is strictly prohibited. In particular, reviewers cannot use Generative AI tools to write their reviews, and reviewers cannot input any content from any submission or review into a Generative AI tool.

Please also see the ethics guidelines of ICML 2025.

If you believe someone may be engaging in unethical conduct, please notify ICML via the Ethics Violation Reporting form.

All suspected unethical conduct will be investigated by ICML’s oversight committee. Individuals found violating the rules may face sanctions, have their own submissions rejected, etc.

Track Specific Instructions

Reviewer Instruction this year differ between the main track and the Position Paper Track. We thus provide below an expanded description as it applies to each track.

Main Track Reviewer Form Instructions

Summary

Briefly summarize the paper (including the main findings, main results, main algorithmic/conceptual ideas, etc. that the paper claims to contribute). This summary should not be used to critique the paper. A well-written summary should not be disputed by the authors of the paper or other readers.

Claims and Evidence

Are the claims made in the submission supported by clear and convincing evidence? If not, which claims are problematic and why?
Do proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense for the problem or application at hand?
Did you check the correctness of any proofs for theoretical claims? Please specify which ones, and discuss any issues.
Did you check the soundness/validity of any experimental designs or analyses? Please specify which ones, and discuss any issues.
Did you review the supplementary material? Which parts?

Relation to Prior Works

How are the key contributions of the paper related to the broader scientific literature? Be specific in terms of prior related findings/results/ideas/etc.
Are there related works that are essential to understanding the (context for) key contributions of the paper, but are not currently cited/discussed in the paper? Be specific in terms of prior related findings/results/ideas/etc. (Example: “The key contribution is a linear-time algorithm, and only cites a quadratic-time algorithm for the same problem as prior work, but there was also an O(n log n) time algorithm for this problem discovered last year, namely Algorithm 3 from [ABC’24] published in ICML 2024.”)
How well-versed are you with the literature related to this paper? (Examples: “I keep up with the literature in this area.”; “I am only familiar with a few key papers in this area, namely [ABC’02], [DEF’04], and [GHI’05].”) Note: Your response to this item will not be visible to authors. Please also see instructions regarding concurrent work.

Other Aspects

Enter any comments on other strengths and weaknesses of the paper, such as those concerning originality, significance, and clarity. We encourage you to be open-minded in terms of potential strengths. For example, originality may arise from creative combinations of existing ideas, removing restrictive assumptions from prior theoretical results, or application to a real-world use case (particularly for application-driven ML papers, indicated in the flag above and described in the Reviewer Instructions).
If you have any other comments or suggestions (e.g., a list of typos), please write them here.

Questions for Authors

If you have any important questions for the authors, please carefully formulate them here. Please reserve your questions for cases where the response would likely change your evaluation of the paper, clarify a point in the paper that you found confusing, or address a critical limitation you identified. Please number your questions so authors can easily refer to them in the response, and explain how possible responses would change your evaluation of the paper.

Ethical Issues

If you believe there are ethical issues with this paper, please flag the paper for an ethics review. For guidance on when this is appropriate, please review the ethics guidelines.
If you flagged this paper for ethics review, what area of expertise would it be most useful for the ethics reviewer to have? Please click all that apply:
1. Discrimination / Bias / Fairness Concerns
2. Inappropriate Potential Applications & Impact (e.g., human rights concerns)
3. Privacy and Security
4. Legal Compliance (e.g., GDPR, copyright, terms of use)
5. Research Integrity Issues (e.g., plagiarism)
6. Responsible Research Practice (e.g., IRB, documentation, research ethics)
7. Other
If you flagged this paper for an ethics review, please explain your concerns in detail.

Code of Conduct Acknowledgement

Please affirm the following: While performing my duties as a reviewer (including writing reviews and participating in discussions), I have and will continue to abide by the ICML code of conduct.

Overall Recommendation

Indicate an overall recommendation:
1. Strong accept
2. Accept
3. Weak accept (i.e., leaning towards accept, but could also be rejected)
4. Weak reject (i.e., leaning towards reject, but could also be accepted)
5. Reject

Position Paper Reviewer Form Instructions

Position

Does the paper clearly state a position on a machine learning topic (policy, implementation, deployment, monitoring, etc.)? Examples include (but are not limited to) an argument in favor or against a particular research priority (not a particular algorithm or solution), a call to action, a value statement, a policy proposal, or a recommendation for changes to how we conduct and evaluate research.

If the paper describes new research without advocating a position, select No.

Position In Title

Does the paper’s title state the position? Can a reader understand the paper’s position from the title alone? If the title consists of a topic or concept without a position about it (e.g., "Position: Psychic Quantum Atelic Learning", "Position: Directions in Hypertrophic Learning"), select No. For an example like "Position: Psychic Quantum Atelic Learning is a Waste of GPUs", select Yes.

Paper Summary

Briefly summarize the paper, its contributions, and the position it advocates (if present). This summary should not be used to critique the paper. A well-written summary should not be disputed by the authors of the paper or other readers.

Strengths and Weaknesses

Please provide a thorough assessment of the strengths and weaknesses of the paper, focusing on the stated position. Is it well supported with reasoning and/or evidence? Is the topic of relevance and importance to the ICML community? Is it likely to inspire discussion? Is it clearly argued? Does it cite related work and events appropriately? Suggestions for improvement are welcome.
(Do not comment on whether you agree with the paper’s position.)
If the paper does not state a position, please note that here along with suggestions for improvement, rather than leaving this field blank.
You can incorporate Markdown and Latex into your review. See https://openreview.net/faq

Support

Please assign the paper a numerical rating on the following scale to indicate how well the paper supports its position with clear reasoning and evidence where appropriate.

4: excellent
3: good
2: fair
1: poor

Significance

Please assign the paper a numerical rating on the following scale to indicate how well the paper demonstrates that the position is important, in terms of scope, impact, timeliness, risks, benefits, etc.

4: excellent
3: good
2: fair
1: poor

Discussion Potential

Please assign the paper a numerical rating on the following scale to indicate the paper’s potential to inspire constructive, useful discussion within the ICML community. BThe reviewer need not agree with the stated position.B

4: excellent
3: good
2: fair
1: poor

Argument Clarity

Please assign the paper a numerical rating on the following scale to indicate the paper’s clarity of the presentation (including writing style).

4: excellent
3: good
2: fair
1: poor

Related Work

Please assign the paper a numerical rating on the following scale to indicate the degree to which the paper includes a discussion of (and citations to) literature and events relevant to the stated position.

4: excellent
3: good
2: fair
1: poor

Questions

Please articulate any important questions for the authors. Reserve your questions for cases where the response would likely change your opinion, clarify a confusion, or address a critical limitation you identified. This can be very important for a productive rebuttal and discussion phase with the authors. Please number your questions for ease of response.

Ethics Flag

If you believe there are ethical issues with this paper, please flag the paper for an ethics review. For guidance on when this is appropriate, please review the ethics guidelines (https://icml.cc/Conferences/2025/PublicationEthics).

Ethics Review Area

If you flagged this paper for ethics review, what area of expertise would it be most useful for the ethics reviewer to have? Please click all that apply.

Discrimination / Bias / Fairness Concerns
Inappropriate Potential Applications & Impact (e.g., human rights concerns)
Privacy and Security (e.g., personally identifiable information)
Legal Compliance (e.g., EU AI Act, GDPR, copyright, terms of use)
Research Integrity Issues (e.g., plagiarism, collusion rings, etc.)
Responsible Research Practice (e.g., IRB, documentation, research ethics)
Other Expertise

Ethical Review Concerns

If you flagged this paper for an ethics review, please explain your concerns in detail. (And if you selected "Other expertise" above, please specify.)

Rating

Choose an overall recommendation for this paper.

5: Strong accept
4: Accept
3: Weak accept (i.e., leaning towards accept, but could also be rejected)
2: Weak reject (i.e., leaning towards reject, but could also be accepted)
1: Reject

Confidence

Please indicate how confident you are in your evaluation.

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the other details carefully.
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Other details were not carefully checked.
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Other details were not carefully checked.
1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Other details were not carefully checked.

Code of Conduct Acknowledgment

Please affirm the following: While performing my duties as a reviewer (including writing reviews and participating in discussions), I have and will continue to abide by the ICML code of conduct (https://icml.cc/public/CodeOfConduct).

Details of the Reviewing Process

Bidding

During the bidding period, reviewers will indicate “bids” on (abstracts of) submissions they are interested in (and qualified for) reviewing. This process is done within OpenReview. These bids will be used, in part, to help with the assignment of submissions to reviewers.

Reviewing

Once the assignments are finalized, reviewers should read the assigned submissions carefully, critically, and with empathy. Reviewers are encouraged (but not required) to read any Supplementary Material provided alongside their assigned submissions. Reviews should be provided in OpenReview by completing the items in the review form. All reviews must be submitted by the stated deadline.

Authors’ Responses

After the reviewing deadline, authors are given the opportunity to respond to the initial reviews. They are encouraged to briefly clarify points, explain misunderstandings, etc. They are not expected to respond to every individual point in the reviews.

After authors’ responses are posted, reviewers may engage in additional discussion with the authors, to follow-up on any remaining questions or concerns.

Reviewers are expected to read and acknowledge the authors’ responses (including appropriately updating their reviews and checking an appropriate box) by the stated deadline.

AC-Reviewer Discussions

Reviewers are encouraged to discuss the paper with each other by posting and responding to notes on OpenReview (visible only to Reviewers/ACs). This is especially important if there are important contradictions between reviews, or if any important aspects of the paper remain unclear to a reviewer. There is no need to wait for the ACs to initiate these discussions.

Discussions will not be visible to authors, so reviewers are encouraged to appropriately update their reviews after discussions for the benefit of the authors.

Visibility

Reviews and the discussion between reviewers and authors for all accepted papers will be made public on OpenReview after the reviewing period. Authors of rejected papers may also opt-in to this public release (and instructions for doing so will be made available later).

The identities of the reviewers will be kept hidden from each other. They will be visible only to ACs, SACs, and program chairs.

Concurrent Works

Authors cannot expect to discuss other papers that have only been made publicly available within four months of the submission deadline. (This cut-off is adopted from the AISTATS and ICLR reviewing instructions.) Such recent papers should be considered as concurrent and simultaneous. Good judgement is necessary to decide whether a paper that has not yet been peer-reviewed should be discussed. The guideline is to follow the best practices of the specific subfield; the Area Chair can help in each case with these.

Reviewer Guidelines

(Adapted from the ICML 2022 Reviewer Tutorial.)

"Review the papers of others as you would wish your own to be reviewed"
– Mihir Bellare, IACR Distinguished Lecture: Caught in Between Theory and Practice

Guiding Principles and Professionalism

The guiding principle for reviewing is that reviewing should create value for:

the authors, by giving them actionable feedback to potentially improve their papers;
the community, by helping authors improve their papers and helping with the decisions to publish papers that advance our field.

A critical aspect of this is the professional conduct of all ICML reviewers. Reviewers are expected to be polite, respectful, and overall professional in their conduct during the whole process. Unprofessional reviews can harm the community in multiple ways: frustration for authors (particularly students) who may slow down in their research, drop out of the field, and/or end up reviewing unprofessionally themselves as a result; loss of promising ideas that could advance the community; resubmission of very similar versions of the paper due to lack of constructive feedback; and so on.

Tips for Reviewing

Before starting to review a paper, (re-)read the Review Form, and think about the aspects of the paper that need to be evaluated.

Read the paper carefully, critically, and with empathy. As you read, keep in mind that you will need to provide an evaluation of the paper via the Review Form, so it will likely help to take notes for yourself (e.g., highlight the main contributions, mark sections you will need to re-read or check more carefully in a subsequent pass).

After reading the paper, think carefully about whether the paper has properly substantiated the claimed contributions. This may involve verifying proofs, checking whether hypotheses are actually tested by the experiments, checking whether empirical claims do indeed follow from empirical results, etc. This is often the most time-consuming part of the reviewing process. Good judgement is needed to determine the severity of any issues that you identify. It is helpful to point out minor issues that are easily fixed, but it is more important to focus on major issues that are critical to the main contributions.

Consider whether the paper places the research presented into the context of current research. Assessments about a paper’s “originality” and “significance” often crucially depend on how the paper compares to prior works, and thus such prior works should be cited and discussed in the paper.

Note that in many cases, it is difficult and often unnecessary to cite all related prior works. If some relevant prior works are missed, then think about whether or not including them would change the conclusions of the paper. Some omissions may be considered minor issues that are easily fixed.

Please give constructive comments and suggestions to the authors to help them potentially improve their paper. In particular, any comments about strengths and weaknesses must be substantiated.

Other Resources

Please see the ICML 2022 Reviewer Tutorial and Peer Reviewer Guidelines, Memefied for more tips, suggestions, and resources.

Supplementary Guidelines for Reviewing Application-Driven ML Submissions

Application-driven ML papers refer to those which introduce novel methods, datasets, tasks, and/or metrics according to the needs of a real-world use case. During submission, authors may select whether their paper should be assessed as an application-driven paper. Such papers should follow the same standards of quality and relevance to the ICML community as other submissions; however, the form of the contributions may differ from that of other submissions. Reviewers and ACs should bear in mind the following points in considering application-driven papers:

Claims and Evidence

Do the methods fit the problem? In application-driven settings, the ML task should be determined by the needs of the user and the specifics of the problem. For example, tasks may need to accommodate the structure of the data, incorporate physical constraints, or be evaluated using metrics such as interpretability or robustness, depending on the use case.

Expect non-standard datasets. Application-driven methods are frequently evaluated on datasets that fall outside common ML benchmarks. Use-informed datasets should be encouraged if they are appropriately documented and validated.

Relation to Prior Works

Prior work outside the ML literature should be considered in addition to ML work. If the paper presents novel methods, for example, these should be compared against any commonly used non-ML approaches as well as baseline ML methods.

Other Aspects

Originality need not mean wholly novel methods. It may mean a novel combination of existing methods to solve the task at hand, a novel dataset, or a new way of framing tasks or evaluating performance so as to match the needs of the user.

Clarity should be measured in relation to an ML audience. While a paper's application may provide motivation for the problem to be solved, the authors should not require the reader to have expertise in the domain of application and should explain any relevant jargon.

Significance can include the potential for impact on an important application in addition to impact for the ML community. For this reason, novel ideas that are simple to apply may be especially valuable.

Updates

January 22, 2025: Added Position Paper Reviewer Form Instructions
January 8, 2025: Expanded Review Form (Claims and Evidence, Other Aspects); added Supplementary Guidelines for Reviewing Application-Driven ML Submissions.