ICML 2010 Feedback
1. Are you an author of an *accepted* ICML 2010 paper?
 answered question170
 
skipped question0
 Response
Percent		Response
Count
Yes
100,0%170
No 0,0%0
2. Are you going to attend ICML 2010 in Haifa?
 answered question170
 
skipped question0
 Response
Percent		Response
Count
Yes
65,3%111
No
34,7%59
viewMy main reason for not attending is:56
3. What model do you prefer for assigning area chairs to papers? (multiple selections possible)
 answered question143
 
skipped question27
 Response
Percent		Response
Count
Authors supply area keywords, and area chairs bid on papers primarily in the areas they agreed to cover. The assignment of area chairs to areas is public to the authors. (this year's model)
86,0%123
Authors supply area keywords, and the system matches area chairs to papers by keyword overlap.
11,9%17
Authors bid on area chairs. (last year's model)
10,5%15
viewI have the following alternative proposal:
4,2%6
4. What model do you prefer for assigning reviewers to papers? (multiple selections possible)
 answered question145
 
skipped question25
 Response
Percent		Response
Count
Two-phase model mixing bidding and manual assignment. In Phase I, reviewer bidding is used to assign two reviewers to each paper. In Phase II, area chairs manually assign additional reviewers based on the Phase I reviews and the author rebuttal. (this year's model)
75,2%109
Two-phase model like above, but area chairs manually assign reviewers in both phases.
13,8%20
Single-phase model, where reviewers are assigned to papers based on their bids.
9,0%13
Single-phase model, where reviewers are manually assigned by the area chairs.
9,0%13
viewI have the following alternative proposal:
3,4%5
5. What reviewing model do you prefer?
 answered question144
 
skipped question26
 Response
Percent		Response
Count
double-blind (reviewers do not know the authors, and vice versa) (as this year)
84,7%122
single-blind (reviewers know the authors, but authors don't know the reviewers)
9,7%14
open (reviewers and authors know each other's identities)
5,6%8
6. Do you think the Phase I reviews were different in quality from the Phase II reviews?
 answered question138
 
skipped question32
 Response
Percent		Response
Count
Phase I substantially better
4,3%6
Phase I slightly better
16,7%23
Both were the same
43,5%60
Phase II slightly better
23,2%32
Phase II substantially better
12,3%17
7. Do you think the ICML2010 reviews were different in quality from the reviews at previous ICMLs?
 answered question119
 
skipped question51
 Response
Percent		Response
Count
ICML2010 substantially better
8,4%10
ICML2010 slightly better
20,2%24
About the same
65,5%78
Previous ICMLs slightly better
4,2%5
Previous ICMLs substantially better
1,7%2
viewAdditional comments:13
8. Do you think the ICML2010 reviews were different in quality from those at related conferences?
 answered question131
 
skipped question39
 Response
Percent		Response
Count
ICML2010 substantially better
15,3%20
ICML2010 slightly better
35,1%46
About the same
40,5%53
Other conferences slightly better
8,4%11
Other conferences substantially better
0,8%1
viewWhat conference are you comparing to and what is the subject area?24
9. What do you think of early rejects by area chairs (i.e., rejecting a paper based on two reviews and author rebuttal)?
 answered question143
 
skipped question27
 Response
Percent		Response
Count
I object to early rejects. Rejected papers should get at least three reviews.
12,6%18
I support early rejects, but only in very clear cases (approximately 13% of submissions this year).
76,9%110
Area chairs should make more liberal use of early rejects.
10,5%15
10. What do you think of early accepts (i.e., accepting a paper after only two reviews and author rebuttal)?
 answered question142
 
skipped question28
 Response
Percent		Response
Count
We should not allow this (as it was this year).
67,6%96
We should allow this for papers with exceptionally good reviews.
32,4%46
11. Do you think the author feedback has an influence on the decision of acceptance/rejection?
 answered question141
 
skipped question29
 Response
Percent		Response
Count
Very strong influence.
4,3%6
Substantial influence.
27,7%39
Marginal influence.
60,3%85
No influence.
7,8%11
12. When do you think should authors be asked to provide their feedback?
 answered question145
 
skipped question25
 Response
Percent		Response
Count
After Phase I of the reviewing process, so it can inform the selection of Phase II reviewers and is immediately available during the discussion. (this year's model)
65,5%95
After Phase II of the reviewing process, so authors can respond to all reviews, not just the Phase I reviews.
14,5%21
Ask for author feedback both after Phase I and Phase II, even though the paper submission deadline would need to be moved to January 15.
11,0%16
Do not ask for author feedback at all.
2,8%4
viewI have the following alternative proposal:
6,2%9
13. Do you think the majority of the reviews of your paper were
 answered question137
 
skipped question33
 Response
Percent		Response
Count
right to the point
19,7%27
high quality
65,0%89
low quality
13,9%19
the reviewers did not understand my paper
1,5%2
14. How do you rate the length of the majority of the reviews?
 answered question137
 
skipped question33
 Response
Percent		Response
Count
too long
1,5%2
just right
81,0%111
too short
17,5%24
15. Do you think the reviewers read your feedback?
 answered question139
 
skipped question31
 Response
Percent		Response
Count
yes, and they took it into consideration
25,9%36
yes, but they ignored it
12,2%17
I'm not sure
61,2%85
certainly not
0,7%1
16. Do you think that the meta reviewer (area chair) adequately summarized the reviewers' opinions and made a right decision?
 answered question135
 
skipped question35
 Response
Percent		Response
Count
yes
69,6%94
somewhat, but mostly OK
27,4%37
no
3,0%4
there was no justification of the decision 0,0%0
17. How do you rate the length of the meta review(s)?
 answered question132
 
skipped question38
 Response
Percent		Response
Count
too long 0,0%0
just right
84,8%112
too short
15,2%20
18. Will you submit again to a future ICML conference?
 answered question137
 
skipped question33
 Response
Percent		Response
Count
yes
95,6%131
no
0,7%1
don't know yet
3,6%5
19. Which conference tool do you like best?
 answered question97
 
skipped question73
 Response
Percent		Response
Count
CMT (this year)
88,7%86
CyberChair
8,2%8
START
3,1%3
viewOther (please specify)9
20. Did you find CMT
 answered question130
 
skipped question40
 Response
Percent		Response
Count
great
11,5%15
comfortable
42,3%55
o.k.
39,2%51
clumsy
6,2%8
awful
0,8%1
21. Do you have any suggestions for improving CMT that we should pass on to its developers?
 answered question15
 
skipped question155
 Response
Count
view15
22. With ca. 160 accepted and invited papers, the program currently fills 5 parallel tracks on three days to the limit. Should we allow the conference to grow further?
 answered question128
 
skipped question42
 Response
Percent		Response
Count
No, we already accept too many papers.
17,2%22
No, we should not accept more papers even if we get substantially more submissions.
32,0%41
Yes, but include accepted papers that will only be presented as posters.
28,1%36
Yes, add another parallel track.
9,4%12
Yes, add another day of sessions to the program.
13,3%17
viewOther suggestions?5
23. This year, we invited 7 application papers, which got a free entry to the program and a slot in the proceedings. Do you think this is a good idea?
 answered question136
 
skipped question34
 Response
Percent		Response
Count
Yes, we should keep that.
11,0%15
Sounds good in principle, but I have not yet seen the papers.
61,0%83
No, application papers are fine, but we do not need additional incentives for attracting them
16,9%23
No, we should find another way for attracting application papers.
11,0%15
viewOther comments?4
24. At this year´s conference, proceedings on paper, USB or CD will *not* be available, but papers can be downloaded from the Web page. Does this work for you?
 answered question136
 
skipped question34
 Response
Percent		Response
Count
Yes.
91,9%125
Yes, but I think it's a bad idea.
5,9%8
No.
2,2%3
25. As in previous years, printed proceedings will be available from Omnipress (this year only after the conference). This service is not free (about 6$ per registration). Do you think we should still produce printed proceedings for archival purposes?
 answered question136
 
skipped question34
 Response
Percent		Response
Count
Yes.
30,1%41
Yes, but try to find a cheaper way.
8,8%12
No.
61,0%83
26. What other conferences do you regularly attend?
 answered question112
 
skipped question58
 Response
Percent		Response
Count
NIPS
82,1%92
AIStats
23,2%26
UAI
25,0%28
COLT
13,4%15
ECML/PKDD
18,8%21
SIGKDD
16,1%18
ICDM
8,0%9
IJCAI
20,5%23
AAAI
25,9%29
ECAI
2,7%3
EMNLP
9,8%11
CoNLL
2,7%3
SIGIR
8,0%9
viewOther (please specify)19