General comments$We summarize our main contributions that addresses some of the reviewers’ comments. Our paper provides upper and lower bounds for the mean square error (MSE) for the maximum likelihood parameter estimator, and analogous results for the Borda score ranking method. The upper bound for the MSE is derived using an approach based on the analysis of the log-likelihood function. This approach follows the same main steps as in Hajek et al 2014, but we deploy several new technical steps like the matrix Chernoff’s inequalities, etc. Our results for the Borda score ranking method are derived using a new approach. The results of the paper shed light on how the parameter estimation accuracy is affected (Q1) by the structure of comparison sets, and (Q2) by the cardinality of comparison sets. For Q1, the Fiedler eigenvalue of a specific pair-weight matrix identified in our paper is found to play a key role. For Q2, it is found that for any unbiased comparison schedule, the MSE is largely insensitive to the cardinality of comparison sets, for a large class of generalized Thursone choice models, which includes all models used by popular rating systems. These results are of interest in practice as they provide a quantitative method to assess the accuracy of the parameter estimator for given input data (e.g. in an online platform where the goal is to rank workers with respect to their abilities), and provide guidelines to the designers of rating systems (e.g. they suggest that it is much better to structure a competition schedule such that a user participates in a large number of contests each with a few opponents).   Reviewer 1 We thank the reviewer for a well-rounded assessment of our main contributions. -We will comment on different terminology used to refer to generalized Thurstone models. -We will emphasise that Hajek et al 2014 considered lower bounds for full-ranking outcomes according to a generalized Thustone model. -We will check and consider to refer to the suggested references for the concept of a Fiedler eigenvalue.   Reviewer 2 -The constants A_{F,b}, B_{F,b}, C_{F,b} and D_{F,b} in Theorem 2 depend only on distribution F and parameter b, and are independent of the structure of comparison sets, and in particular, on the parameter k when each comparison set is of cardinality k. -It is non-trivial to express conditions A1-A3 in terms of conditions on distribution F and parameter k because p_k(x) depends on F in a non-trivial way as given in Equation (2). Note that A1-A3 hold for any generalized Thurstone choice model in the limit of small b, in which case all the constants in A1-A3 tend to 1. We will add intuitive explanations for conditions A1-A3. -For the maximum-likelihood parameter estimator of a given generalized Thurstone choice model, the weights w(k) should be chosen as in Equation (9), which is an interesting finding of the paper. -According to intuition, the larger k guarantees the more accurate estimation. We will explicitly state that the smaller \gamma_{F,k} indicates the better performance in both Section 3 and Section 4.   Reviewer 3 -We will put in a significant effort to account for the reviewer’s comments regarding the presentation. -We will add intuitive explanations for conditions A1-A3. -The smallest eigenvalue of the Hessian of \ell(\theta) is bounded bellow by \lambda_2(\Lambda_M). Under G2, thus, \ell is strictly concave since \lambda_2(\Lambda_M)>0. We will provide an intuitive explanation. -The log(n) gap between upper and lower bound allows us to say “with probability 1-2/n”. One can remove the log(n) gap by using a constant probability (e.g., “with probability 3/4”). -It is one of our main goals to characterize how the MSE depends on the structure of a comparison schedule, which is found to be captured by the Fiedler eigenvalue of a pair-weight matrix; this is why we present Fiedler values for different schedules in the experimental results section.   Reviewer 4 -We will clarify that generalized Thurstone choice model is a well-known model and point to relevant references. We will check that the goals of the paper are clearly stated along the lines as in general comments of this feedback. –Our goal is not to argue in favour of using the maximum likelihood or some other estimator, but to characterize the accuracy of a canonical maximum likelihood parameter estimator and shed light on questions Q1 and Q2 (please refer to general comments section). –We do consider situations in which the input data consists of top-1 lists that is common in many practical situations, e.g., those pointed out in the paper’s introduction. The generalized Thurstone choice model is a standard model for top-1 lists. –We think that our paper provides valuable contributions to the theoretical understanding of the given maximum likelihood parameter estimation problem, and provides insights for practitioners who design rating systems. Please see general comments section.