Can I read Did the Model Change? Efficiently Assessing Machine Learning API Shifts on EtoBox?

Did the Model Change? Efficiently Assessing Machine Learning API Shifts by Chen, Lingjiao; Cai, Tracy; Zaharia, Matei; Zou, James is a scholarly article available to read on EtoBox.

What is Did the Model Change? Efficiently Assessing Machine Learning API Shifts about?

Machine learning (ML) prediction APIs are increasingly widely used. An ML API can change over time due to model updates or retraining. This presents a key challenge in the usage of the API because it is often not clear to the user if and how the ML model has changed. Model shifts can affect downstream application performance and also create oversight issues (e.g. if consistency is desired). In this paper, we initiate a systematic investigation of ML API shifts. We first quantify the performance shifts from 2020 to 2021 of popular ML APIs from Google, Microsoft, Amazon, and others on a variety of datasets. We identified significant model shifts in 12 out of 36 cases we investigated. Interestingly, we found several datasets where the API's predictions became significantly worse over time. This motivated us to formulate the API shift assessment problem at a more fine-grained level as estimating how the API model's confusion matrix changes over time when the data distribution is constant. Monitoring confusion matrix shifts using standard random sampling can require a large number of samples, which is expensive as each API call costs a fee. We propose a principled adaptive sampling algo

Author

Chen, Lingjiao; Cai, Tracy; Zaharia, Matei; Zou, James

Published

2021

Language

EN