How were decisions made and how should I interpret the reviews and review scores my paper received?
The review form (see https://icml.cc/Conferences/2020/ReviewForm) provided reviewers and opportunity to comment on the merits and aspects of the paper that led to their evaluation. Their evaluation, which then was encoded as a number from one (1) to six (6), was one of:
6. Outstanding paper, I would fight for it to be accepted.
5. Very good paper, I would like to see it accepted.
4. Borderline paper, but has merits that outweigh flaws.
3. Borderline paper, but the flaws may outweigh the merits.
2. Below the acceptance threshold, I would rather not see it at the conference.
1. Wrong or known results, I would fight to have it rejected.
However, it is important to keep in mind that scores are only one part -- and a relatively minor part -- of how decisions for a conference like ICML get made. (After all, they provide <3 bits of information, in comparison to the full text of a review!) In general, in most cases we do not expect that meta-reviewers (area chairs) will simply sort by average score and make decisions. They (and we) much more information with which to make a decision and so as a result, just as there were some papers with relatively low average scores that were accepted, there were also some papers with relatively high average scores that were not.
Final decisions for acceptance of papers at ICML was made in close collaboration with meta-reviewers, as a result of reviews, author feedback, and discussions. For instance, on papers where there was disagreement between reviewers (~1000 papers), there was substantially discussion amongst reviewers and meta-reviewers: a mean discussion length of 4.0 turns (median=3), with some discussions going as long as 38 turns!
Of course, even with everyone's best efforts, mistakes do happen, and when they do we know they can be frustrating and demoralizing, and we are sorry when this happens. Please keep in mind that reviews are there to help papers improve, too: A common theme we see is that a paper has a really good idea that simply needs to be worked out a bit more, resubmitted, and it will be great.
We request that you do not email the program chairs to raise disagreements with your reviews, meta-reviews, or decisions. Everyone involved is trying their best to make decisions on a very large number of papers during a very challenging time, and we need to focus our time and energy on making decisions in the best way we can, in the most equitable way we can. We're also keeping in mind that statistically there are often disparities in who feels comfortable asking for special exemptions, and wish to avoid a decision making process that compounds inequities.
The exception to this is if you believe that one of your reviews or your meta-review violated the ICML Code of Conduct (https://icml.cc/public/CodeOfConduct), for instance text that constitutes harassment, bullying, discrimination, and/or retaliation. If this is the case, please contact the program chairs *immediately* at program-committee@icml.cc with a subject line that makes clear that this is a Code of Conduct violation.