Machine Learning API Shift Assessments: Change is Coming!

A growing number of applications rely on machine learning (ML) prediction APIs. Model updates or retraining can change an ML API silently. This leads to a key challenge to API users, who are unaware of if and how the ML model has been changed. We take the first step towards the study of ML API shifts. We first evaluate the performance shifts from 2020 to 2021 of popular ML APIs from Amazon, Baidu, and Google on a variety of datasets. Interestingly, some API’s predictions became notably worse for a certain class and better for another. Thus, we formulate the API shift assessment problem as estimating how the API model’s confusion matrix changes over time when the data distribution is constant. Next, we propose MASA, a principled adaptive sampling algorithm to efficiently estimate confusion matrix shifts. Empirically, MASA can accurately estimate the confusion matrix shifts in commercial ML APIs with up to 77% fewer samples than random sampling. This paves the way for understanding and monitoring ML API shifts efficiently.