ICML 2012 Survey Results
Conditioned on questionee being an author of an rejected paper
Background
1) Did you serve as an Area chair for ICML 2012?
10/267 = 4% Yes
257/267 = 96% No
2) Did you serve as a Reviewer / PC member for ICML 2012?
42/267 = 16% Yes
225/267 = 84% No
3) Are you an author of a paper submitted to ICML?
267/267 = 100% Yes
0/267 = 0% No
4) You are:
34/267 = 13% An author of an accepted paper
267/267 = 100% An author of a rejected paper
5) Are you going to attend ICML 2012 in Edinburgh?
54/267 = 20% Yes
213/267 = 80% No
Paper Assignment
6) Which information do you support using for assignment of papers to PC members? (choose all that apply)
26/ 41 = 63% Keywords
34/ 41 = 83% Toronto matching service
34/ 41 = 83% Bids
27/ 41 = 66% Choices made by the Area Chairs
7) How did you like your assignments?
4/ 42 = 10% Very interesting
37/ 42 = 88% Good fit for my expertise
1/ 42 = 2% Mostly not my area
0/ 42 = 0% Terrible
Paper Reviewing
9) Did you consult supplementary material for any of the papers you reviewed?
25/ 41 = 61% Yes
16/ 41 = 39% No
10) Did the supplementary material help you form a decision for any paper?
14/ 39 = 36% Yes
25/ 39 = 64% No
11) Did you have sufficient time to prepare your reviews?
33/ 40 = 83% Yes
7/ 40 = 18% No
12) Would you prefer if the paper submission deadline was earlier, to allow more time for the initial review period?
8/ 41 = 20% Yes, the review period should be longer.
32/ 41 = 78% The review period was fine.
1/ 41 = 2% No, in fact the review period could be shorter.
Author Response
13) Did you read the author responses for any of the papers you reviewed?
41/ 42 = 98% Yes
1/ 42 = 2% No
13_5) Was the author response informative for you?
34/ 42 = 81% Yes
8/ 42 = 19% No
14) Did you consult a new version of a paper uploaded during author response?
21/ 42 = 50% Yes
21/ 42 = 50% No
15) Did you change any of your (meta-)reviews due to the author response or new version?
10/ 42 = 24% It changed my mind about a paper.
26/ 42 = 62% It helped clarify a few questions, but did not change my mind.
6/ 42 = 14% It was not useful.
0/ 42 = 0% I did not see it.
16) Do you think the author feedback has an influence on the decision of acceptance/rejection?
26/ 42 = 62% Yes
16/ 42 = 38% No
Discussion
17) Did you participate in the discussion for any of the papers you reviewed?
40/ 42 = 95% Yes
2/ 42 = 5% No
18) Did you change any of your reviews due to the discussion?
26/ 42 = 62% Yes
16/ 42 = 38% No
Assessment of Reviewing Process
19) Do you think the ICML 2012 reviews were different in quality from the reviews at previous ICMLs?
1/ 42 = 2% ICML 2012 substantially better
11/ 42 = 26% Somewhat better
16/ 42 = 38% The same
4/ 42 = 10% Somewhat worse
1/ 42 = 2% Substantially worse
9/ 42 = 21% Don't know
20) Do you think the ICML 2012 reviews were different in quality from the reviews at other similar conferences?
3/ 42 = 7% ICML 2012 substantially better
9/ 42 = 21% Somewhat better
20/ 42 = 48% The same
6/ 42 = 14% Somewhat worse
1/ 42 = 2% Substantially worse
3/ 42 = 7% Don't know
21) Compared to other conferences, how much effort was it to participate in the program committee for ICML 2012?
1/ 42 = 2% ICML 2012 substantially less
1/ 42 = 2% Somewhat less
19/ 42 = 45% The same
7/ 42 = 17% Somewhat more
8/ 42 = 19% Substantially more
6/ 42 = 14% Don't know
22) Throughout the reviewing process, was it always clear to you what needed to be done?
37/ 42 = 88% Yes
5/ 42 = 12% No
23) How many hours did you spend on the review process?
Averaged over 31 entries: 27
24) If you submitted a rebuttal during the author response period, do you believe it had some influence on the final decision?
3/262 = 1% Very strong influence
15/262 = 6% Substantial influence
108/262 = 41% Marginal influence
136/262 = 52% No influence
25) Do you prefer having the author response option?
223/264 = 84% Yes
41/264 = 16% No
26) If you uploaded a new file during the author response period, do you believe it had some influence on the final decision?
5/211 = 2% Very strong influence
15/211 = 7% Substantial influence
68/211 = 32% Marginal influence
123/211 = 58% No influence
27) The majority of the reviews of your paper were:
20/261 = 8% Right to the point
127/261 = 49% High quality
66/261 = 25% Low quality
48/261 = 18% The reviewers did not understand my paper.
28) Do you prefer the option to revise the paper during author response?
172/265 = 65% Yes
93/265 = 35% No
29) Do you think that the meta-reviews adequately summarized the reviewers' opinions and made the right decision?
45/261 = 17% Yes
81/261 = 31% Mostly
85/261 = 33% Somewhat
50/261 = 19% No, there was no justification of the decision.
Conference Format
30) With 242 accepted papers, ICML can no longer offer a full talk to all accepted papers (assuming 5 parallel tracks over 3 days). Which of the following do you support (choose all that apply):
37/255 = 15% Accept fewer papers.
83/255 = 33% Add a day.
91/255 = 36% Add an additional track.
110/255 = 43% Present some papers only as posters.
107/255 = 42% Present some papers as short talk + poster.
9/255 = 4% Other (briefly comment below)
30-c) Other
31) The workshop program has been extended to 2-days. What is your opinion of this new format?
46/256 = 18% Good idea, I plan on attending.
100/256 = 39% Good idea, but I won’t attend.
89/256 = 35% I don’t care.
13/256 = 5% Bad idea, ICML is too long already.
8/256 = 3% Bad idea, I don’t like workshops.
32) Which conferences would like to see in co-location with ICML in the future? (choose all that apply):
90/184 = 49% COLT
76/184 = 41% UAI
48/184 = 26% ECML-PKDD
43/184 = 23% SIGKDD
51/184 = 28% IJCAI
62/184 = 34% AAAI
11/184 = 6% ECAI
14/184 = 8% EMNLP-CoNLL
26/184 = 14% SIGIR
23/184 = 13% ACL
13/184 = 7% RSS
51/184 = 28% CVPR
12/184 = 7% RSS
6/184 = 3% Other (comment below)