Skip to yearly menu bar Skip to main content


ICML 2024 Reviewer Instructions

Thank you for serving on the program committee as a reviewer, AC, or SAC.  The commitment and time investment of the program committee is essential to the success of ICML, and we are deeply grateful for your effort.

We have published a blog post that details some of our considerations in the review process this year, which you can read if you’re interested: https://medium.com/@icml2024pc/reviewing-at-icml-2024-a7aa81169d8c.

We are no longer accepting nominations to become a reviewer or AC.

Key Dates:

  • Papers Due: February 1
  • Reviewer/AC Bidding: February 2-7
  • Review Period: February 16 - March 14
  • Author Rebuttal Phase: March 21 - March 28
  • Author/Reviewer Discussions: March 28 - April 4
  • Reviewer/AC Discussions: April 4 - April 11
  • AC Metareviews Due: April 16
  • Author Notification: May 2

During the author/reviewer discussion pharse, please participate early, and acknowledge/respond to the information in author responses.

Reviewer Instruction this year differ between the main track and the Position Paper Track.  We thus provide below an expanded description as it applies to each track.

Review Form for Main Track

You will be asked on the review form for each paper and some guidelines on what to consider when answering these questions.  Remember that answering “no” to some questions is typically not grounds for rejection. When writing your review, please keep in mind that after decisions have been made, reviews and meta-reviews of accepted papers and opted-in rejected papers will be made public. 

  1. Summary: Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.
  2. Strengths and Weaknesses: Please provide a thorough assessment of the strengths and weaknesses of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage people to be broad in their definitions of originality and significance. For example, originality may arise from creative combinations of existing ideas, application to a new domain, or removing restrictive assumptions from prior theoretical results. You can incorporate Markdown and Latex into your review. See https://openreview.net/faq.
  3. Questions: Please list up and carefully describe any questions and suggestions for the authors. Think of the things where a response from the author can change your opinion, clarify a confusion or address a limitation. This can be very important for a productive rebuttal and discussion phase with the authors.
  4. Limitations: Have the authors adequately addressed the limitations and potential negative societal impact of their work? If not, please include constructive suggestions for improvement. Authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact.
  5. Ethical concerns: If you believe there are ethical issues with this paper, please flag the paper for an ethics review. For guidance on when this is appropriate, please review the ethics guidelines (https://icml.cc/Conferences/2024/PublicationEthics).
  6. Ethics Review Area: If you flagged this paper for ethics review, what area of expertise would it be most useful for the ethics reviewer to have? Please click all that apply:
    • Discrimination / Bias / Fairness Concerns
    • Inappropriate Potential Applications & Impact  (e.g., human rights concerns)
    • Privacy and Security
    • Legal Compliance (e.g., GDPR, copyright, terms of use)
    • Research Integrity Issues (e.g., plagiarism)
    • Responsible Research Practice (e.g., IRB, documentation, research ethics)
    • Other
  7. Details of Ethics Concerns: If you flagged paper for ethics review, please provide details of your concerns.
  8. Soundness: Please assign the paper a numerical rating on the following scale to indicate the soundness of the technical claims, experimental and research methodology and on whether the central claims of the paper are adequately supported with evidence.
    • 4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  9. Presentation: Please assign the paper a numerical rating on the following scale to indicate the quality of the presentation. This should take into account the writing style and clarity, as well as contextualization relative to prior work.
    • 4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  10. Contribution: Please assign the paper a numerical rating on the following scale to indicate the quality of the overall contribution this paper makes to the research area being studied. Are the questions being asked important? Does the paper bring a significant originality of ideas and/or execution? Are the results valuable to share with the broader ICML community?
    • 4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  11. Rating: Please provide an "overall score" for this submission. Choices:
    • 10: Award quality: Technically flawless paper with groundbreaking impact on one or more areas of AI, with exceptionally strong evaluation, reproducibility, and resources, and no unaddressed ethical considerations.
    • 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI and excellent impact on multiple areas of AI, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
    • 8: Strong Accept: Technically strong paper with, with novel ideas, excellent impact on at least one area of AI or high-to-excellent impact on multiple areas of AI, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
    • 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate-to-high impact on more than one area of AI, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
    • 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
    • 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
    • 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
    • 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
    • 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
    • 1: Very Strong Reject: For instance, a paper with trivial results or unaddressed ethical considerations
  12. Confidence:  Please provide a "confidence score" for your assessment of this submission to indicate how confident you are in your evaluation.  Choices
    • 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
    • 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
    • 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
    • 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
    • 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
  13. Code of conduct acknowledgement. While performing my duties as a reviewer (including writing reviews and participating in discussions), I have and will continue to abide by the ICML code of conduct (https://icml.cc/public/CodeOfConduct).

 

Review Form for Position Paper Track

Below is a description of the questions you will be asked on the review form for each position paper, along with some guidelines on what to consider when answering these questions.  When writing your review, please keep in mind that after decisions have been made, reviews and meta-reviews of accepted papers and opted-in rejected papers will be made public.

  1. Position: Does the paper clearly state a position on a machine learning topic (research, implementation, deployment, monitoring, etc.)? Examples include (but are not limited to) an argument in favor or against a particular research priority, a call to action, a value statement, a statement of concern about ICML community procedures, or a recommendation for changes to how we conduct and evaluate research. If the paper describes new research without advocating a position, select No.
  2. Summary: Briefly summarize the paper, its contributions, and the position it advocates (if present). This is not the place to critique the paper; the authors should generally agree with a well-written summary.
  3. Strengths and Weaknesses: Please provide a thorough assessment of the strengths and weaknesses of the paper, focusing on the stated position.  Is it clearly argued? Is it well supported with reasoning and/or evidence? Is the topic of relevance and importance to the ICML community? Is it likely to inspire discussion? Suggestions for improvement are welcome. (Do not comment on whether you agree with the paper’s position.) If the paper does not state a position, please note that here along with any suggestions, rather than leaving this field blank. You can incorporate Markdown and Latex into your review. See https://openreview.net/faq 
  4. Questions: Please articulate any questions for the authors. Think of the things where a response from the author could change your opinion, clarify a confusion, or address a limitation. This can be very important for a productive rebuttal and discussion phase with the authors.
  5. Ethical Concerns: If there are ethical issues with this paper, please flag the paper for an ethics review. For guidance on when this is appropriate, please review the ethics guidelines (https://icml.cc/Conferences/2024/PublicationEthics).
  6. Ethics Review Area: If you flagged this paper for ethics review, what area of expertise would it be most useful for the ethics reviewer to have? Please click all that apply:
    • Discrimination / Bias / Fairness Concerns
    • Inappropriate Potential Applications & Impact  (e.g., human rights concerns)
    • Privacy and Security
    • Legal Compliance (e.g., GDPR, copyright, terms of use)
    • Research Integrity Issues (e.g., plagiarism)
    • Responsible Research Practice (e.g., IRB, documentation, research ethics)
    • Other
  7. Details of Ethics Concerns: If you flagged paper for ethics review, please provide details of your concerns.
  8. Support: Please assign the paper a numerical rating on the following scale to indicate whether the paper supports its position with clear reasoning and evidence where appropriate.
    • 4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  9. Significance: Please assign the paper a numerical rating on the following scale to indicate whether the paper demonstrates that the topic is important, in terms of scope, impact, timeliness, risks, benefits, etc.
    • 4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  10. Discussion: Please assign the paper a numerical rating on the following scale to indicate whether the topic is likely to inspire constructive, useful discussion within the ICML community. The reviewer need not agree with the stated position.
    • ​​​​​4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  11. Presentation: Please assign the paper a numerical rating on the following scale to indicate the quality of the presentation (writing style and clarity).
    • ​​​​​​​4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  12. ​​​​​​​​​​​​​​Context: Please assign the paper a numerical rating on the following scale to indicate whether the paper includes a discussion of (and citations to) literature and events relevant to the stated position.
    • ​​​​​​​4 excellent
    • 3 good
    • 2 fair
    • 1 poor
  13. ​​​​​​​​​​​​​​Rating: Please provide an "overall score" for this submission. Choices:
    • ​​​​​​​​​​​​​​10: Award quality: The paper makes an outstanding argument of exceptional clarity about a timely, urgent issue that impacts not only the ICML community but broader society or the world.
    • 9: Very Strong Accept: The paper makes a very compelling argument about an important issue that has not yet been discussed widely and impacts the entire ICML community.
    • 8: Strong Accept: The paper presents a strong argument about an important issue that has not yet been discussed widely and is of importance to a sub-area within the ICML community.
    • 7: Accept: The paper presents a solid argument about an important issue that has received attention but remains unresolved and is of importance to at least one sub-area of the ICML community.
    • 6: Weak Accept: The paper presents a solid argument about an issue of moderate importance with relevance to at least one sub-area of the ICML community.
    • 5: Borderline accept: The paper presents a position, and the reasons to accept outweigh reasons to reject, e.g., unclear reasoning or limited support for the claims. Please use sparingly.
    • 4: Borderline reject: The paper presents a position, but the reasons to reject, e.g., unclear reasoning or limited support for the claims, outweigh reasons to accept. Please use sparingly.
    • 3: Reject: For instance, a position paper with a weakly stated position, technical or logical flaws, inadequate support for claims, or addresses an issue of limited scope.
    • 2: Strong Reject: For instance, a position paper that is poorly written, makes several flawed arguments, or addresses a trivial issue.
    • 1: Very Strong Reject: The paper does not advocate for a position. It may have been mistakenly submitted to this track.
  14. ​​​​​​​Confidence: Please provide a "confidence score" for your assessment of this submission to indicate how confident you are in your evaluation.
    • 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the other details carefully.
    • 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
    • 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Other details were not carefully checked.
    • 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Other details were not carefully checked.
    • 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Other details were not carefully checked.​​​​​​​​​​​
  15. ​​​​​​​​​​​​​​Code of conduct acknowledgement. While performing my duties as a reviewer (including writing reviews and participating in discussions), I have and will continue to abide by the ICML code of conduct (https://icml.cc/public/CodeOfConduct).

 

Ethics Review

You will be asked to flag papers for ethical concerns. Categories of ethical concerns, including an illustrative example of expectations for each category,  can be found at https://icml.cc/Conferences/2024/PublicationEthics. 

If you flag a paper, if possible, please provide constructive suggestions for improvement. 

Flagged papers will be triaged by your Area Chair. If the AC determines they need additional information to assess or mitigate the ethical concern, the paper will undergo ethics review by researchers who specialize in the area of concern. In cases where mitigation is not possible, final decisions will be made by the Program Chairs with input from the AC and Ethics Chairs.