An Efficient Joint Learning Approach for Item Response Theory

Item response theory (IRT) is widely used in areas such as recommender systems, education, psychology, and other fields. A popular model for IRT is the Rasch model. Under this model, if a user with ability $\theta$ performs a task with difficulty $\beta$ then its label $X \sim \text{Bernoulli} (1 / (1 + \exp(-(\theta - \beta)))$. Existing joint maximum likelihood estimation approaches for this problem do not perform well on small datasets and also lack theoretical guarantees. Recently, Nguyen and Zhang proposed a two step approach: (1) spectral method for estimation of task parameters, (2) likelihood optimization for estimation of user parameters. While this approach is theoretically sound, it is not computationally efficient. In this work, we propose an EM-based algorithm for joint estimation of item and user parameters by introducing Pólya-Gamma latent variables, which simplify the logistic log-likelihood. We show that our algorithm is both theoretically sound and consistently outperforms existing methods on synthetic and real-world datasets.

Item Response Theory (IRT) is a framework using which we can estimate a student's ability from their answers to test questions, while also estimating how difficult the questions are. It has historically been used in psychometric tests and educational settings (including standardized exams such as GRE and GMAT), has recently gained traction in crowdsourcing of data and in evaluation of large language models. In this work, we develop a new algorithm for estimating these hidden ("latent") abilities and difficulties from observed responses, that is, whether the student answered the question correctly or not. Existing methods generally perform poorly when these responses are sparse, which is common in real-world datasets. Some methods are prone to making mistakes in their estimates, while some are very slow. We make use of an interesting mathematical technique called Polya-Gamma data augmentation to develop our algorithm. We show theoretically as well as empirically that our method is faster and more accurate than existing methods, especially when the data is sparse. These improvements make IRT more practical for large-scale applications. Moreover, the techniques and results could be used to develop better algorithms for other such "latent parameter estimation" problems.