Can I read Meta-Learning Fast Weight Language Models on EtoBox?

Meta-Learning Fast Weight Language Models by Clark, Kevin; Guu, Kelvin; Chang, Ming-Wei; Pasupat, Panupong; Hinton, Geoffrey; Norouzi, Mohammad is a scholarly article available to read on EtoBox.

What is Meta-Learning Fast Weight Language Models about?

Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens and substantially improves LM performance. However, it requires over 3x more compute than standard inference. We present Fast Weight Layers (FWLs), a neural component that provides the benefits of dynamic evaluation much more efficiently by expressing gradient updates as linear attention. A key improvement over dynamic evaluation is that FWLs can also be applied at training time so the model learns to make good use of gradient updates. FWLs can easily be added on top of existing transformer models, require relatively little extra compute or memory to run, and significantly improve language modeling perplexity.

Author

Clark, Kevin; Guu, Kelvin; Chang, Ming-Wei; Pasupat, Panupong; Hinton, Geoffrey; Norouzi, Mohammad

Published

2022

Language

EN