When to retrain a machine learning model

A significant challenge in maintaining real-world machine learning models is responding to the continuous and unpredictable evolution of data. Most practitioners are faced with the difficult question: when should I retrain or update my machine learning model? This seemingly straightforward problem is particularly challenging for three reasons: 1) decisions must be made based on very limited information - we usually have access to only a few examples, 2) the nature, extent, and impact of the distribution shift are unknown, and 3) it involves specifying a cost ratio between retraining and poor performance, which can be hard to characterize. Existing works address certain aspects of this problem, but none offer a comprehensive solution. Distribution shift detection falls short as it cannot account for the cost trade-off; the scarcity of the data, paired with its unusual structure, makes it a poor fit for existing offline reinforcement learning methods, and the online learning formulation overlooks key practical considerations. To address this, we present a principled formulation of the retraining problem and propose an uncertainty-based method that makes decisions by continually forecasting the evolution of model performance evaluated with a bounded metric. Our experiments, addressing classification tasks, show that the method consistently outperforms existing baselines on 7 datasets. We thoroughly assess its robustness to varying cost trade-off values and mis-specified cost trade-offs.

One of the biggest challenges in keeping machine learning models working well in the real world is handling changes in the data over time. A common and difficult question that practitioners face is: When should I retrain or update my model? This question sounds simple but is actually quite complex, for a few key reasons: 1) We usually have very limited data to make decisions with, 2)we don’t know how the data has changed or how much that change will affect the model’s performance, and 3) we have to weigh the cost of retraining the model against the cost of keeping a model that might perform poorly which can be hard to define.While there are existing methods that address parts of this problem, none provide a complete solution. For example, detecting data shifts doesn’t tell us whether retraining is worthwhile; online methods usually focus only on maintaining performance, and other approaches often require large amounts of data to be effective.To tackle this, we introduce a new, well-defined approach to the retraining problem. Our method uses uncertainty estimates to decide when to retrain, based on predictions about how the model’s performance will change over time. In our experiments on classification tasks across 7 datasets, our method consistently beats existing techniques. We also test how well it handles different and even incorrect assumptions about the cost of retraining versus performance loss.