ICML 2012 Survey Results
Conditioned on questionee being an author of an accepted paper
Background
1) Did you serve as an Area chair for ICML 2012?
17/230 = 7% Yes
213/230 = 93% No
2) Did you serve as a Reviewer / PC member for ICML 2012?
66/230 = 29% Yes
164/230 = 71% No
3) Are you an author of a paper submitted to ICML?
230/230 = 100% Yes
0/230 = 0% No
4) You are:
230/230 = 100% An author of an accepted paper
34/230 = 15% An author of a rejected paper
5) Are you going to attend ICML 2012 in Edinburgh?
176/230 = 77% Yes
54/230 = 23% No
Paper Assignment
6) Which information do you support using for assignment of papers to PC members? (choose all that apply)
43/ 66 = 65% Keywords
57/ 66 = 86% Toronto matching service
61/ 66 = 92% Bids
52/ 66 = 79% Choices made by the Area Chairs
7) How did you like your assignments?
5/ 66 = 8% Very interesting
59/ 66 = 89% Good fit for my expertise
1/ 66 = 2% Mostly not my area
1/ 66 = 2% Terrible
Paper Reviewing
9) Did you consult supplementary material for any of the papers you reviewed?
45/ 65 = 69% Yes
20/ 65 = 31% No
10) Did the supplementary material help you form a decision for any paper?
25/ 63 = 40% Yes
38/ 63 = 60% No
11) Did you have sufficient time to prepare your reviews?
58/ 65 = 89% Yes
7/ 65 = 11% No
12) Would you prefer if the paper submission deadline was earlier, to allow more time for the initial review period?
4/ 65 = 6% Yes, the review period should be longer.
58/ 65 = 89% The review period was fine.
3/ 65 = 5% No, in fact the review period could be shorter.
Author Response
13) Did you read the author responses for any of the papers you reviewed?
65/ 66 = 98% Yes
1/ 66 = 2% No
13_5) Was the author response informative for you?
58/ 65 = 89% Yes
7/ 65 = 11% No
14) Did you consult a new version of a paper uploaded during author response?
27/ 66 = 41% Yes
39/ 66 = 59% No
15) Did you change any of your (meta-)reviews due to the author response or new version?
20/ 66 = 30% It changed my mind about a paper.
40/ 66 = 61% It helped clarify a few questions, but did not change my mind.
6/ 66 = 9% It was not useful.
0/ 66 = 0% I did not see it.
16) Do you think the author feedback has an influence on the decision of acceptance/rejection?
53/ 66 = 80% Yes
13/ 66 = 20% No
Discussion
17) Did you participate in the discussion for any of the papers you reviewed?
64/ 66 = 97% Yes
2/ 66 = 3% No
18) Did you change any of your reviews due to the discussion?
42/ 65 = 65% Yes
23/ 65 = 35% No
Assessment of Reviewing Process
19) Do you think the ICML 2012 reviews were different in quality from the reviews at previous ICMLs?
4/ 66 = 6% ICML 2012 substantially better
21/ 66 = 32% Somewhat better
25/ 66 = 38% The same
2/ 66 = 3% Somewhat worse
0/ 66 = 0% Substantially worse
14/ 66 = 21% Don't know
20) Do you think the ICML 2012 reviews were different in quality from the reviews at other similar conferences?
5/ 66 = 8% ICML 2012 substantially better
25/ 66 = 38% Somewhat better
27/ 66 = 41% The same
3/ 66 = 5% Somewhat worse
0/ 66 = 0% Substantially worse
6/ 66 = 9% Don't know
21) Compared to other conferences, how much effort was it to participate in the program committee for ICML 2012?
2/ 66 = 3% ICML 2012 substantially less
1/ 66 = 2% Somewhat less
32/ 66 = 48% The same
19/ 66 = 29% Somewhat more
7/ 66 = 11% Substantially more
5/ 66 = 8% Don't know
22) Throughout the reviewing process, was it always clear to you what needed to be done?
62/ 66 = 94% Yes
4/ 66 = 6% No
23) How many hours did you spend on the review process?
Averaged over 51 entries: 30
24) If you submitted a rebuttal during the author response period, do you believe it had some influence on the final decision?
9/224 = 4% Very strong influence
79/224 = 35% Substantial influence
115/224 = 51% Marginal influence
21/224 = 9% No influence
25) Do you prefer having the author response option?
212/228 = 93% Yes
16/228 = 7% No
26) If you uploaded a new file during the author response period, do you believe it had some influence on the final decision?
10/179 = 6% Very strong influence
35/179 = 20% Substantial influence
93/179 = 52% Marginal influence
41/179 = 23% No influence
27) The majority of the reviews of your paper were:
38/229 = 17% Right to the point
166/229 = 72% High quality
23/229 = 10% Low quality
2/229 = 1% The reviewers did not understand my paper.
28) Do you prefer the option to revise the paper during author response?
139/224 = 62% Yes
85/224 = 38% No
29) Do you think that the meta-reviews adequately summarized the reviewers' opinions and made the right decision?
125/227 = 55% Yes
73/227 = 32% Mostly
25/227 = 11% Somewhat
4/227 = 2% No, there was no justification of the decision.
Conference Format
30) With 242 accepted papers, ICML can no longer offer a full talk to all accepted papers (assuming 5 parallel tracks over 3 days). Which of the following do you support (choose all that apply):
65/227 = 29% Accept fewer papers.
93/227 = 41% Add a day.
83/227 = 37% Add an additional track.
75/227 = 33% Present some papers only as posters.
90/227 = 40% Present some papers as short talk + poster.
6/227 = 3% Other (briefly comment below)
30-c) Other
31) The workshop program has been extended to 2-days. What is your opinion of this new format?
100/229 = 44% Good idea, I plan on attending.
58/229 = 25% Good idea, but I won’t attend.
61/229 = 27% I don’t care.
8/229 = 3% Bad idea, ICML is too long already.
2/229 = 1% Bad idea, I don’t like workshops.
32) Which conferences would like to see in co-location with ICML in the future? (choose all that apply):
114/187 = 61% COLT
103/187 = 55% UAI
24/187 = 13% ECML-PKDD
47/187 = 25% SIGKDD
45/187 = 24% IJCAI
45/187 = 24% AAAI
11/187 = 6% ECAI
25/187 = 13% EMNLP-CoNLL
22/187 = 12% SIGIR
25/187 = 13% ACL
13/187 = 7% RSS
50/187 = 27% CVPR
9/187 = 5% RSS
6/187 = 3% Other (comment below)