ICML 2012 Survey Results
Conditioned on questionee being a reviewer
Background
1) Did you serve as an Area chair for ICML 2012?
25/198 = 13% Yes
173/198 = 87% No
2) Did you serve as a Reviewer / PC member for ICML 2012?
198/198 = 100% Yes
0/198 = 0% No
3) Are you an author of a paper submitted to ICML?
89/198 = 45% Yes
109/198 = 55% No
4) You are:
66/ 88 = 75% An author of an accepted paper
42/ 88 = 48% An author of a rejected paper
5) Are you going to attend ICML 2012 in Edinburgh?
78/198 = 39% Yes
120/198 = 61% No
Paper Assignment
6) Which information do you support using for assignment of papers to PC members? (choose all that apply)
127/195 = 65% Keywords
158/195 = 81% Toronto matching service
172/195 = 88% Bids
150/195 = 77% Choices made by the Area Chairs
7) How did you like your assignments?
16/197 = 8% Very interesting
168/197 = 85% Good fit for my expertise
12/197 = 6% Mostly not my area
1/197 = 1% Terrible
Paper Reviewing
9) Did you consult supplementary material for any of the papers you reviewed?
128/197 = 65% Yes
69/197 = 35% No
10) Did the supplementary material help you form a decision for any paper?
75/189 = 40% Yes
114/189 = 60% No
11) Did you have sufficient time to prepare your reviews?
160/194 = 82% Yes
34/194 = 18% No
12) Would you prefer if the paper submission deadline was earlier, to allow more time for the initial review period?
31/196 = 16% Yes, the review period should be longer.
151/196 = 77% The review period was fine.
14/196 = 7% No, in fact the review period could be shorter.
Author Response
13) Did you read the author responses for any of the papers you reviewed?
196/198 = 99% Yes
2/198 = 1% No
13_5) Was the author response informative for you?
166/196 = 85% Yes
30/196 = 15% No
14) Did you consult a new version of a paper uploaded during author response?
105/197 = 53% Yes
92/197 = 47% No
15) Did you change any of your (meta-)reviews due to the author response or new version?
47/196 = 24% It changed my mind about a paper.
133/196 = 68% It helped clarify a few questions, but did not change my mind.
16/196 = 8% It was not useful.
0/196 = 0% I did not see it.
16) Do you think the author feedback has an influence on the decision of acceptance/rejection?
144/196 = 73% Yes
52/196 = 27% No
Discussion
17) Did you participate in the discussion for any of the papers you reviewed?
188/196 = 96% Yes
8/196 = 4% No
18) Did you change any of your reviews due to the discussion?
122/195 = 63% Yes
73/195 = 37% No
Assessment of Reviewing Process
19) Do you think the ICML 2012 reviews were different in quality from the reviews at previous ICMLs?
7/197 = 4% ICML 2012 substantially better
43/197 = 22% Somewhat better
67/197 = 34% The same
6/197 = 3% Somewhat worse
1/197 = 1% Substantially worse
73/197 = 37% Don't know
20) Do you think the ICML 2012 reviews were different in quality from the reviews at other similar conferences?
14/197 = 7% ICML 2012 substantially better
68/197 = 35% Somewhat better
75/197 = 38% The same
10/197 = 5% Somewhat worse
2/197 = 1% Substantially worse
28/197 = 14% Don't know
21) Compared to other conferences, how much effort was it to participate in the program committee for ICML 2012?
3/196 = 2% ICML 2012 substantially less
15/196 = 8% Somewhat less
89/196 = 45% The same
48/196 = 24% Somewhat more
20/196 = 10% Substantially more
21/196 = 11% Don't know
22) Throughout the reviewing process, was it always clear to you what needed to be done?
183/195 = 94% Yes
12/195 = 6% No
23) How many hours did you spend on the review process?
Averaged over 161 entries: 28
24) If you submitted a rebuttal during the author response period, do you believe it had some influence on the final decision?
3/ 85 = 4% Very strong influence
10/ 85 = 12% Substantial influence
53/ 85 = 62% Marginal influence
19/ 85 = 22% No influence
25) Do you prefer having the author response option?
82/ 88 = 93% Yes
6/ 88 = 7% No
26) If you uploaded a new file during the author response period, do you believe it had some influence on the final decision?
2/ 66 = 3% Very strong influence
3/ 66 = 5% Substantial influence
27/ 66 = 41% Marginal influence
34/ 66 = 52% No influence
27) The majority of the reviews of your paper were:
13/ 87 = 15% Right to the point
49/ 87 = 56% High quality
22/ 87 = 25% Low quality
3/ 87 = 3% The reviewers did not understand my paper.
28) Do you prefer the option to revise the paper during author response?
41/ 87 = 47% Yes
46/ 87 = 53% No
29) Do you think that the meta-reviews adequately summarized the reviewers' opinions and made the right decision?
32/ 87 = 37% Yes
31/ 87 = 36% Mostly
15/ 87 = 17% Somewhat
9/ 87 = 10% No, there was no justification of the decision.
Conference Format
30) With 242 accepted papers, ICML can no longer offer a full talk to all accepted papers (assuming 5 parallel tracks over 3 days). Which of the following do you support (choose all that apply):
69/193 = 36% Accept fewer papers.
64/193 = 33% Add a day.
72/193 = 37% Add an additional track.
73/193 = 38% Present some papers only as posters.
84/193 = 44% Present some papers as short talk + poster.
8/193 = 4% Other (briefly comment below)
30-c) Other
31) The workshop program has been extended to 2-days. What is your opinion of this new format?
50/195 = 26% Good idea, I plan on attending.
64/195 = 33% Good idea, but I won’t attend.
68/195 = 35% I don’t care.
9/195 = 5% Bad idea, ICML is too long already.
4/195 = 2% Bad idea, I don’t like workshops.
32) Which conferences would like to see in co-location with ICML in the future? (choose all that apply):
92/157 = 59% COLT
92/157 = 59% UAI
15/157 = 10% ECML-PKDD
53/157 = 34% SIGKDD
37/157 = 24% IJCAI
40/157 = 25% AAAI
8/157 = 5% ECAI
19/157 = 12% EMNLP-CoNLL
24/157 = 15% SIGIR
28/157 = 18% ACL
6/157 = 4% RSS
39/157 = 25% CVPR
3/157 = 2% RSS
5/157 = 3% Other (comment below)