Can I read Characterizing and Measuring Linguistic Dataset Drift on EtoBox?

Characterizing and Measuring Linguistic Dataset Drift by Chang, Tyler A.; Halder, Kishaloy; John, Neha Anna; Vyas, Yogarshi; Benajiba, Yassine; Ballesteros, Miguel; Roth, Dan is a scholarly article available to read on EtoBox.

What is Characterizing and Measuring Linguistic Dataset Drift about?

NLP models often degrade in performance when real world data distributions differ markedly from training data. However, existing dataset drift metrics in NLP have generally not considered specific dimensions of linguistic drift that affect model performance, and they have not been validated in their ability to predict model performance at the individual example level, where such metrics are often used in practice. In this paper, we propose three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift. These dimensions correspond to content word frequency divergences, syntactic divergences, and meaning changes not captured by word frequencies (e.g. lexical semantic change). We propose interpretable metrics for all three drift dimensions, and we modify past performance prediction methods to predict model performance at both the example and dataset level for English sentiment classification and natural language inference. We find that our drift metrics are more effective than previous metrics at predicting out-of-domain model accuracies (mean 16.8% root mean square error decrease), particularly when compared to popular fine-tuned embedding distances (mean 47

Author

Chang, Tyler A.; Halder, Kishaloy; John, Neha Anna; Vyas, Yogarshi; Benajiba, Yassine; Ballesteros, Miguel; Roth, Dan

Published

2023

Language

EN