Result Summary for Survey:ICML 2010 Feedback

ICML 2010 Feedback

1. Are you an author of a rejected ICML 2010 paper?
	answered question		288
	skipped question		0
		Response Percent	Response Count
Yes		100,0%	288
No		0,0%	0

2. Are you going to attend ICML 2010 in Haifa?
	answered question		288
	skipped question		0
		Response Percent	Response Count
Yes		13,5%	39
No		86,5%	249
viewMy main reason for not attending is:			173

3. What model do you prefer for assigning area chairs to papers? (multiple selections possible)
	answered question		230
	skipped question		58
		Response Percent	Response Count
Authors supply area keywords, and area chairs bid on papers primarily in the areas they agreed to cover. The assignment of area chairs to areas is public to the authors. (this year's model)		65,2%	150
Authors supply area keywords, and the system matches area chairs to papers by keyword overlap.		24,3%	56
Authors bid on area chairs. (last year's model)		22,2%	51
viewI have the following alternative proposal:		2,6%	6

4. What model do you prefer for assigning reviewers to papers? (multiple selections possible)
	answered question		235
	skipped question		53
		Response Percent	Response Count
Two-phase model mixing bidding and manual assignment. In Phase I, reviewer bidding is used to assign two reviewers to each paper. In Phase II, area chairs manually assign additional reviewers based on the Phase I reviews and the author rebuttal. (this year's model)		50,6%	119
Two-phase model like above, but area chairs manually assign reviewers in both phases.		21,3%	50
Single-phase model, where reviewers are assigned to papers based on their bids.		19,1%	45
Single-phase model, where reviewers are manually assigned by the area chairs.		15,3%	36
viewI have the following alternative proposal:		7,2%	17

5. What reviewing model do you prefer?
	answered question		238
	skipped question		50
		Response Percent	Response Count
double-blind (reviewers do not know the authors, and vice versa) (as this year)		80,7%	192
single-blind (reviewers know the authors, but authors don't know the reviewers)		10,9%	26
open (reviewers and authors know each other's identities)		8,4%	20

6. Do you think the Phase I reviews were different in quality from the Phase II reviews?
	answered question		224
	skipped question		64
		Response Percent	Response Count
Phase I substantially better		16,5%	37
Phase I slightly better		13,4%	30
Both were the same		41,5%	93
Phase II slightly better		17,4%	39
Phase II substantially better		11,2%	25

7. Do you think the ICML2010 reviews were different in quality from the reviews at previous ICMLs?
	answered question		177
	skipped question		111
		Response Percent	Response Count
ICML2010 substantially better		2,3%	4
ICML2010 slightly better		9,0%	16
About the same		63,3%	112
Previous ICMLs slightly better		15,8%	28
Previous ICMLs substantially better		9,6%	17
viewAdditional comments:			37

8. Do you think the ICML2010 reviews were different in quality from those at related conferences?
	answered question		218
	skipped question		70
		Response Percent	Response Count
ICML2010 substantially better		9,6%	21
ICML2010 slightly better		18,3%	40
About the same		45,4%	99
Other conferences slightly better		17,9%	39
Other conferences substantially better		8,7%	19
viewWhat conference are you comparing to and what is the subject area?			55

9. What do you think of early rejects by area chairs (i.e., rejecting a paper based on two reviews and author rebuttal)?
	answered question		233
	skipped question		55
		Response Percent	Response Count
I object to early rejects. Rejected papers should get at least three reviews.		23,6%	55
I support early rejects, but only in very clear cases (approximately 13% of submissions this year).		68,2%	159
Area chairs should make more liberal use of early rejects.		8,2%	19

10. What do you think of early accepts (i.e., accepting a paper after only two reviews and author rebuttal)?
	answered question		233
	skipped question		55
		Response Percent	Response Count
We should not allow this (as it was this year).		56,2%	131
We should allow this for papers with exceptionally good reviews.		43,8%	102

11. Do you think the author feedback has an influence on the decision of acceptance/rejection?
	answered question		236
	skipped question		52
		Response Percent	Response Count
Very strong influence.		2,5%	6
Substantial influence.		9,7%	23
Marginal influence.		53,0%	125
No influence.		34,7%	82

12. When do you think should authors be asked to provide their feedback?
	answered question		233
	skipped question		55
		Response Percent	Response Count
After Phase I of the reviewing process, so it can inform the selection of Phase II reviewers and is immediately available during the discussion. (this year's model)		42,1%	98
After Phase II of the reviewing process, so authors can respond to all reviews, not just the Phase I reviews.		24,9%	58
Ask for author feedback both after Phase I and Phase II, even though the paper submission deadline would need to be moved to January 15.		17,2%	40
Do not ask for author feedback at all.		9,4%	22
viewI have the following alternative proposal:		6,4%	15

13. Do you think the majority of the reviews of your paper were
	answered question		222
	skipped question		66
		Response Percent	Response Count
right to the point		9,9%	22
high quality		45,0%	100
low quality		31,1%	69
the reviewers did not understand my paper		14,0%	31

14. How do you rate the length of the majority of the reviews?
	answered question		223
	skipped question		65
		Response Percent	Response Count
too long		2,7%	6
just right		76,2%	170
too short		21,1%	47

15. Do you think the reviewers read your feedback?
	answered question		227
	skipped question		61
		Response Percent	Response Count
yes, and they took it into consideration		8,4%	19
yes, but they ignored it		21,6%	49
I'm not sure		58,6%	133
certainly not		11,5%	26

16. Do you think that the meta reviewer (area chair) adequately summarized the reviewers' opinions and made a right decision?
	answered question		218
	skipped question		70
		Response Percent	Response Count
yes		19,7%	43
somewhat, but mostly OK		47,2%	103
no		25,2%	55
there was no justification of the decision		7,8%	17

17. How do you rate the length of the meta review(s)?
	answered question		217
	skipped question		71
		Response Percent	Response Count
too long		1,4%	3
just right		65,9%	143
too short		32,7%	71

18. Will you submit again to a future ICML conference?
	answered question		228
	skipped question		60
		Response Percent	Response Count
yes		67,1%	153
no		3,1%	7
don't know yet		29,8%	68

19. Which conference tool do you like best?
	answered question		141
	skipped question		147
		Response Percent	Response Count
CMT (this year)		85,8%	121
CyberChair		9,2%	13
START		5,0%	7
viewOther (please specify)			20

20. Did you find CMT
	answered question		198
	skipped question		90
		Response Percent	Response Count
great		10,6%	21
comfortable		37,9%	75
o.k.		41,9%	83
clumsy		6,1%	12
awful		3,5%	7

21. Do you have any suggestions for improving CMT that we should pass on to its developers?
	answered question	20
	skipped question	268
		Response Count
view		20

22. With ca. 160 accepted and invited papers, the program currently fills 5 parallel tracks on three days to the limit. Should we allow the conference to grow further?
	answered question		194
	skipped question		94
		Response Percent	Response Count
No, we already accept too many papers.		10,3%	20
No, we should not accept more papers even if we get substantially more submissions.		22,2%	43
Yes, but include accepted papers that will only be presented as posters.		39,7%	77
Yes, add another parallel track.		10,8%	21
Yes, add another day of sessions to the program.		17,0%	33
viewOther suggestions?			11

23. This year, we invited 7 application papers, which got a free entry to the program and a slot in the proceedings. Do you think this is a good idea?
	answered question		205
	skipped question		83
		Response Percent	Response Count
Yes, we should keep that.		21,0%	43
Sounds good in principle, but I have not yet seen the papers.		49,3%	101
No, application papers are fine, but we do not need additional incentives for attracting them		17,6%	36
No, we should find another way for attracting application papers.		12,2%	25
viewOther comments?			9

24. At this year´s conference, proceedings on paper, USB or CD will not be available, but papers can be downloaded from the Web page. Does this work for you?
	answered question		213
	skipped question		75
		Response Percent	Response Count
Yes.		88,3%	188
Yes, but I think it's a bad idea.		8,0%	17
No.		3,8%	8

25. As in previous years, printed proceedings will be available from Omnipress (this year only after the conference). This service is not free (about 6$ per registration). Do you think we should still produce printed proceedings for archival purposes?
	answered question		211
	skipped question		77
		Response Percent	Response Count
Yes.		36,5%	77
Yes, but try to find a cheaper way.		13,7%	29
No.		49,8%	105

26. What other conferences do you regularly attend?
	answered question		164
	skipped question		124
		Response Percent	Response Count
NIPS		70,1%	115
AIStats		18,9%	31
UAI		18,3%	30
COLT		11,0%	18
ECML/PKDD		31,1%	51
SIGKDD		22,0%	36
ICDM		16,5%	27
IJCAI		22,0%	36
AAAI		22,6%	37
ECAI		4,3%	7
EMNLP		3,7%	6
CoNLL		0,6%	1
SIGIR		8,5%	14
viewOther (please specify)			31