Can I read Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Tasks on EtoBox?

Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Tasks by Kar, Sudipta; Castellucci, Giuseppe; Filice, Simone; Malmasi, Shervin; Rokhlenko, Oleg is a scholarly article available to read on EtoBox.

What is Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Tasks about?

Multi-Task Learning (MTL) is widely-accepted in Natural Language Processing as a standard technique for learning multiple related tasks in one model. Training an MTL model requires having the training data for all tasks available at the same time. As systems usually evolve over time, (e.g., to support new functionalities), adding a new task to an existing MTL model usually requires retraining the model from scratch on all the tasks and this can be time-consuming and computationally expensive. Moreover, in some scenarios, the data used to train the original training may be no longer available, for example, due to storage or privacy concerns. In this paper, we approach the problem of incrementally expanding MTL models' capability to solve new tasks over time by distilling the knowledge of an already trained model on n tasks into a new one for solving n+1 tasks. To avoid catastrophic forgetting, we propose to exploit unlabeled data from the same distributions of the old tasks. Our experiments on publicly available benchmarks show that such a technique dramatically benefits the distillation by preserving the already acquired knowledge (i.e., preventing up to 20% performance drops on ol

Author

Kar, Sudipta; Castellucci, Giuseppe; Filice, Simone; Malmasi, Shervin; Rokhlenko, Oleg

Published

2023

Language

EN