Reviewer FAQ

General questions

-- I discovered the identities of authors of some papers I review, what to do now?

-- Some of my papers violate ICML style/anonymity guidelines, shall I review them?

-- Do other reviewers and the meta-reviewer know who I am?

-- Do authors know who I am?

-- Am I required to review supplementary materials?

-- What if I don’t have enough expertise to write a strong review for some paper?

-- I know that the paper I review is under submission to a different conference, what to do?

-- Do you provide any accessibility support?

-- What if something is unclear in the paper and I have questions for the authors?

-- What happens after my review is submitted?

 

Author Response and Discussion

-- What should I do now that author response is in and discussion has begun?

-- When does the discussion period end?

-- What can the author response include? Links? Author identity? Should authors have responded to every issue brought up in reviews?

-- Does ICML allow "revise and resubmit" decisions on papers?

-- What is the Reproducibility Checklist?

-- Do other reviewers know my identity? Does the meta-reviewer?

-- There was a paper I reviewed but it is not there anymore! What happened?

-- I am asked if I read the author response, but there is no author response. What should I do?

-- Oh no, my name shows up in the discussion when I posted a comment! I thought I was supposed to be anonymous!

 

Code Submission

-- What is the usage policy for code?

-- Is there anything I need to be aware of before I access the code?

-- What do we expect from the authors by way of code submission?

-- How do I use the code in the decision process?

-- What if the submitted code is not runnable? What if it is just pseudo-code?

-- What if the submitted code is not anonymous?

-- What if I find that the submitted code is empty or the link does not work?

 

Specific questions

-- A very recent paper is not cited, is it a problem?

-- Not all baselines are considered due to limited computational resources, is it a problem?

-- How should I review papers for which a new dataset is a significant contribution?

-- I cannot open pdf of the submission in Adobe Acrobat, what shall I do?


 

General Questions

-- I discovered identities of the authors of some papers I review, what to do now?

Under no circumstances should you attempt to find out the identities of the authors for any of your assigned submissions (e.g., by searching on Google or arXiv). If you accidentally find out, please do not divulge the identities to anyone, but do tell the meta-reviewer assigned to that paper that this has happened. You should not let the authors’ identities influence your decision in any way.

 

 -- Some of my papers violate ICML style/anonymity guidelines, shall I review them?

If you are assigned any submissions that violate the ICML style (e.g., by decreasing margins or font size), page limits or reveals author identities, please contact the meta-reviewer in charge of that paper for guidance.

 

-- Do other reviewers and the Meta-reviewer know who I am?

Your identity is NOT visible to other reviewers but is visible to the meta-reviewer in charge of a submission, as well as the program chairs.

 

-- Do authors know who I am?

No, the review process is double-blind and your identity, as well as the identity of meta-reviewers, is hidden from authors.

 

-- Am I required to review supplementary materials?

Your responsibility as a reviewer is to read and review the submission itself; looking at supplementary material is at your discretion. That said, you may wish to look at supplementary material before criticizing a submission for insufficient details, proofs, or experimental results.

 

-- What if I don’t have enough expertise to write a thoughtful review for some paper?

Please inform the meta-reviewer for that paper immediately (ideally by 27 Feb 2020).

 

-- I know that the paper I review is under submission to a different conference or journal; what to do?

Please bring it to the attention of the meta-reviewer for that paper immediately.

 

-- Do you provide any accessibility support?

Yes, this year we asked authors to provide Latex source of their submissions, because there is better accessibility support (in the form of screen-readers) for Latex than pdf. The latex source will be made available upon request for accessibility reasons only. Please let us know by 27 Feb 2020 if this applies to you.

 

-- What if something is unclear in the paper and I have questions for the authors?

If you have specific questions for the authors, please list them clearly in the "Detailed Comments for Authors" section of your review. Authors will have a one-time opportunity to provide a response to the reviews by 08 April 2020, at which time they may answer your specific question. You will then be able to update your review and score.

 

-- What happens after my review is submitted?

After you have submitted your review, you will be able to see review from all the other reviewers (anonymously). Additionally, if needed, you will have the opportunity to participate in discussion amongst the reviewers and meta-reviewer. Authors also have an opportunity to respond to questions and concerns raised in reviews during author feedback period (see previous question). The discussion period ends on Tuesday 28 April 2020, at which point meta-reviewers will make acceptance recommendations and provide meta-reviews for their papers.

 

Author Response and Discussion

 

-- What should I do now that author response is in and discussion has begun?

1. Please read the author feedback, and let the authors know that you have done so (by at least checking the "I read the response" button on the review form).

2. Please update your scores and your review in light of the feedback, if applicable. Please take author feedback into account. Authors put a lot of time into providing feedback, and it is disheartening when it feels like it was ignored. If the authors have satisfactorily addressed your criticism(s), then please update your score accordingly. If not, then explain what remains problematic.

3. If your scores disagree substantially from the other reviewers, then please begin discussion with the other reviewers and the Area Chair in charge of the paper, to ensure that all relevant aspects of the paper have been discussed (if consensus can be reached, that's great, but it may not always happen).

 

-- When does the discussion period end?

Discussion must conclude by May 5 at the absolute latest, and all reviews must be updated by that date. In order to get notifications out in time for (possible) resubmissions to NeurIPS, we cannot allow any extensions beyond that date.

 

-- What can the author response include? Links? Author identity? Should authors have responded to every issue brought up in reviews?

In responding to author feedback, please remember that at ICML, authors are not allowed to significantly modify their submissions during author responses, nor are they allowed to supply additional figures through, for example, external websites. If an author response includes a link, please do not follow it.

As the length of the author response is limited, and as authors cannot reveal their identities in the response, please keep in mind that not all issues raised by a reviewer can be addressed; a paper need not be flawless to be accepted, so please evaluate the degree to which the feedback addresses the *important* points in a review, rather than whether it addresses *all* points. If you observe any author feedback that violates the rules (e.g., include external links, or de-anonymize authorship), please bring them to our attention.

 

-- Does ICML allow "revise and resubmit" decisions on papers?

ICML does not allow "revise and resubmit," so if the authors cannot satisfactorily deal with the reviewer criticisms without a major revision (which would need to be checked in a second review), then the paper should be rejected.

 

-- What is the Reproducibility Checklist?

Authors have been asked to fill out a "Reproducibility Checklist" as part of their submission, in which they declare what aspects of their paper are reproducible and how. You may wish to check the author's responses to the Reproducibility Checklist when doing a final evaluation of their paper, in particular when code is available (but even if not). You can access this through the CMT interface if you have not already.

 

-- Do other reviewers know my identity? Does the meta-reviewer?

No, other reviewers do not know your identity (so please do not reveal it in the discussion). Meta-reviewers, however, do know your identity. They've been told not to reveal it during the discussion to other reviewers. When you post in the discussion you will see your own name, but other reviewers will not see it (they'll see "Reviewer #3" for instance).

 

-- There was a paper I reviewed but it is not there anymore! What happened?

Several authors have withdrawn their papers following the review process, in which case you will no longer be able to see their paper on CMT. You needn't worry about these cases, but please remember to delete the paper and code locally.

 

-- I am asked if I read the author response, but there is no author response. What should I do?

In this case, please just select "yes."

 

-- Oh no, my name shows up in the discussion when I posted a comment! I thought I was supposed to be anonymous!

We realize this is confusing. You will see your own name in the discussion thread, but other reviewers will not. (The meta-reviewer, however, will.) But rest assured, other reviewers cannot see your name (nor can you see theirs).

 

Code Submission

-- What is the usage policy for code?

Any submitted code is confidential, just like any paper submission. You are only allowed to use it for the purpose of ICML reviewing simply for the duration of the review period. Please remember that code is proprietary. Just like the paper submission, you are required to keep the code confidential and to delete it at the end of the review period to comply with the need for confidentiality.

 

-- Is there anything I need to be aware of before I access the code?

Please remember that just like any other untrusted code, any submitted code may contain security vulnerabilities. To guard against these, we recommend that either you look at the code, or you run any downloaded code on a virtual machine or a docker that is not connected to the internet. See https://docs.docker.com/get-started/ for an example.

Some authors may also submit an anonymous link to a code repository (like github). In such cases, please make sure to either manually download the code or to clone (and NOT fork) the repository as the authors will be able to see who forks their repository and thus infer your identity. Please make sure you log out of your own github account before accessing such links, as an extra precaution.

You may wish to also be cautious about accessing other web links provided from the paper, as these may contain vulnerabilities or may log visitor IP addresses.

 

-- What do we expect from the authors by way of code submission?

Authors who choose to submit code are supposed to submit it in anonymized form, either as a zip file or as part of an anonymous github link. The code submitted may or may not be standalone or runnable by a third party; and submission of detailed pseudocode that can help you verify the credibility of the results is also allowed. We are not asking authors to submit any private data; however, they can choose to submit toy or simulated data for testing code. Also, if the authors are introducing a new dataset, then they are encouraged to make it available in an appropriate data repository.  

 

-- How do I use the code in the decision process?

Just like supplementary material, submitted code provides additional evidence to add credibility to the results in a paper. High quality papers whose results appear credible should be accepted, even if code is not submitted. However, if something is unclear in the paper, then please use code, if submitted, to clarify the details of the results.

 

-- What if the submitted code is not runnable? What if it is just pseudo-code?

Authors are permitted to submit code that is not runnable as well as detailed pseudocode. Our advice in this case is to look over the code to validate the results as much as you can. If the validity is still unclear, please mention this in your review.  

 

-- What if the submitted code is not anonymous?

If the code reveals the authors or the organization name, please bring it to the attention of the meta-reviewer for this paper immediately.

 

-- What if I find that the submitted code is empty or the link does not work?

Please put this in your review.

 

Specific questions

-- A very recent paper is not cited, is it a problem?

Submissions should not be penalized for not comparing with or citing papers which show up on arxiv up to a month before the abstract submission deadline (i.e., any paper appearing in calendar year 2020 should not be grounds for penalty).

 

-- Not all baselines are considered due to limited computational resources, is it a problem?

Submissions should not be penalized for not providing comparisons to baselines that require extensive computing resources (e.g. tens or hundreds of GPUs), as these resources may not be available to some authors.

 

-- How should I review papers for which a new dataset is a significant contribution?

Some papers may release a dataset. We have encouraged authors to follow several norms for dataset release (e.g., providing links, DOIs, licenses, etc.). As part of the review, you are asked which of these norms is followed. Such information is not required and should not impact the score you give to a paper.

 

-- I cannot open pdf of the submission in Adobe Acrobat, what shall I do?

This year, we have a surprisingly large number of submissions with corrupted PDFs that cannot be opened in Adobe Acrobat. However, most of these submissions can be opened in browsers (i.e., Chrome, Edge, Safari) and using other software (e.g., Apple preview), so please consider using these tools. If you strictly prefer Adobe Acrobat, then we suggest you open the file in browser/preview and save it as PDF into a separate file; this solution seems to fix most of the issues.