Reviewer FAQ:

 

General questions:

-- I discovered the identities of authors of some papers I review, what to do now?

-- Some of my papers violate ICML style/anonymity guidelines, shall I review them?

-- Do other reviewers and the meta-reviewer know who I am?

-- Do authors know who I am?

-- Am I required to review supplementary materials?

-- What if I don’t have enough expertise to write a strong review for some paper?

-- I know that the paper I review is under submission to a different conference, what to do?

-- Do you provide any accessibility support?

-- What if something is unclear in the paper and I have questions for the authors?

-- What happens after my review is submitted?

 

Code Submission

-- What is the usage policy for code?

-- Is there anything I need to be aware of before I access the code?

-- What do we expect from the authors by way of code submission?

-- How do I use the code in the decision process?

-- What if the submitted code is not runnable? What if it is just pseudo-code?

-- What if the submitted code is not anonymous?

-- What if I find that the submitted code is empty or the link does not work?

 

Specific questions

-- A very recent paper is not cited, is it a problem?

-- Not all baselines are considered due to limited computational resources, is it a problem?

-- How should I review papers for which a new dataset is a significant contribution?

-- I cannot open pdf of the submission in Adobe Acrobat, what shall I do?


 

General Questions

 

-- I discovered identities of the authors of some papers I review, what to do now?

 

Under no circumstances should you attempt to find out the identities of the authors for any of your assigned submissions (e.g., by searching on Google or arXiv). If you accidentally find out, please do not divulge the identities to anyone, but do tell the meta-reviewer assigned to that paper that this has happened. You should not let the authors’ identities influence your decision in any way.

 

 -- Some of my papers violate ICML style/anonymity guidelines, shall I review them?

 

If you are assigned any submissions that violate the ICML style (e.g., by decreasing margins or font size), page limits or reveals author identities, please contact the meta-reviewer in charge of that paper for guidance.

 

 

-- Do other reviewers and the Meta-reviewer know who I am?

Yes, your identity is NOT visible to other reviewers but is visible to the meta-reviewer in charge of a submission, as well as the program chairs.

 

 

-- Do authors know who I am?

No, the review process is double-blind and your identity, as well as the identity of meta-reviewers, is hidden from authors.

 

 

-- Am I required to review supplementary materials?

Your responsibility as a reviewer is to read and review the submission itself; looking at supplementary material is at your discretion. That said, you may wish to look at supplementary material before criticizing a submission for insufficient details, proofs, or experimental results.

 

 

-- What if I don’t have enough expertise to write a thoughtful review for some paper?

Please inform the meta-reviewer for that paper immediately (ideally by 27 Feb 2020).

 

 

-- I know that the paper I review is under submission to a different conference or journal; what to do?

Please bring it to the attention of the meta-reviewer for that paper immediately.

 

 

-- Do you provide any accessibility support?

Yes, this year we asked authors to provide Latex source of their submissions, because there is better accessibility support (in the form of screen-readers) for Latex than pdf. The latex source will be made available upon request for accessibility reasons only. Please let us know by 27 Feb 2020 if this applies to you.

 

 

-- What if something is unclear in the paper and I have questions for the authors?

If you have specific questions for the authors, please list them clearly in the "Detailed Comments for Authors" section of your review. Authors will have a one-time opportunity to provide a response to the reviews by 08 April 2020, at which time they may answer your specific question. You will then be able to update your review and score.

 

 

-- What happens after my review is submitted?

After you have submitted your review, you will be able to see review from all the other reviewers (anonymously). Additionally, if needed, you will have the opportunity to participate in discussion amongst the reviewers and meta-reviewer. Authors also have an opportunity to respond to questions and concerns raised in reviews during author feedback period (see previous question). The discussion period ends on Tuesday 28 April 2020, at which point meta-reviewers will make acceptance recommendations and provide meta-reviews for their papers.

 

Code Submission

 

 

-- What is the usage policy for code?

 

Any submitted code is confidential, just like any paper submission. You are only allowed to use it for the purpose of ICML reviewing simply for the duration of the review period. Please remember that code is proprietary. Just like the paper submission, you are required to keep the code confidential and to delete it at the end of the review period to comply with the need for confidentiality.

 

 

-- Is there anything I need to be aware of before I access the code?

 

Please remember that just like any other untrusted code, any submitted code may contain security vulnerabilities. To guard against these, we recommend that either you look at the code, or you run any downloaded code on a virtual machine or a docker that is not connected to the internet. See https://docs.docker.com/get-started/ for an example.

 

Some authors may also submit an anonymous link to a code repository (like github). In such cases, please make sure to either manually download the code or to clone (and NOT fork) the repository as the authors will be able to see who forks their repository and thus infer your identity. Please make sure you log out of your own github account before accessing such links, as an extra precaution.

 

You may wish to also be cautious about accessing other web links provided from the paper, as these may contain vulnerabilities or may log visitor IP addresses.

 

 

 

-- What do we expect from the authors by way of code submission?

 

Authors who choose to submit code are supposed to submit it in anonymized form, either as a zip file or as part of an anonymous github link. The code submitted may or may not be standalone or runnable by a third party; and submission of detailed pseudocode that can help you verify the credibility of the results is also allowed. We are not asking authors to submit any private data; however, they can choose to submit toy or simulated data for testing code. Also, if the authors are introducing a new dataset, then they are encouraged to make it available in an appropriate data repository.  

 

 

-- How do I use the code in the decision process?

 

Just like supplementary material, submitted code provides additional evidence to add credibility to the results in a paper. High quality papers whose results appear credible should be accepted, even if code is not submitted. However, if something is unclear in the paper, then please use code, if submitted, to clarify the details of the results.

 

 

-- What if the submitted code is not runnable? What if it is just pseudo-code?

 

Authors are permitted to submit code that is not runnable as well as detailed pseudocode. Our advice in this case is to look over the code to validate the results as much as you can. If the validity is still unclear, please mention this in your review.  

 

 

-- What if the submitted code is not anonymous?

 

If the code reveals the authors or the organization name, please bring it to the attention of the meta-reviewer for this paper immediately.

 

 

-- What if I find that the submitted code is empty or the link does not work?

 

Please put this in your review.

 

Specific questions

 

-- A very recent paper is not cited, is it a problem?


Submissions should not be penalized for not comparing with or citing papers which show up on arxiv up to a month before the abstract submission deadline (i.e., any paper appearing in calendar year 2020 should not be grounds for penalty).

 

 

-- Not all baselines are considered due to limited computational resources, is it a problem?


Submissions should not be penalized for not providing comparisons to baselines that require extensive computing resources (e.g. tens or hundreds of GPUs), as these resources may not be available to some authors.

 

 

-- How should I review papers for which a new dataset is a significant contribution?

 

Some papers may release a dataset. We have encouraged authors to follow several norms for dataset release (e.g., providing links, DOIs, licenses, etc.). As part of the review, you are asked which of these norms is followed. Such information is not required and should not impact the score you give to a paper.

 

-- I cannot open pdf of the submission in Adobe Acrobat, what shall I do?

 

This year, we have a surprisingly large number of submissions with corrupted PDFs that cannot be opened in Adobe Acrobat. However, most of these submissions can be opened in browsers (i.e., Chrome, Edge, Safari) and using other software (e.g., Apple preview), so please consider using these tools. If you strictly prefer Adobe Acrobat, then we suggest you open the file in browser/preview and save it as PDF into a separate file; this solution seems to fix most of the issues.