Can I read Transformer on a Diet on EtoBox?

Transformer on a Diet by Wang, Chenguang; Ye, Zihao; Zhang, Aston; Zhang, Zheng; Smola, Alexander J. is a scholarly article available to read on EtoBox.

What is Transformer on a Diet about?

Transformer has been widely used thanks to its ability to capture sequence information in an efficient way. However, recent developments, such as BERT and GPT-2, deliver only heavy architectures with a focus on effectiveness. In this paper, we explore three carefully-designed light Transformer architectures to figure out whether the Transformer with less computations could produce competitive results. Experimental results on language model benchmark datasets hint that such trade-off is promising, and the light Transformer reduces 70% parameters at best, while obtains competitive perplexity compared to standard Transformer. The source code is publicly available.

Author

Wang, Chenguang; Ye, Zihao; Zhang, Aston; Zhang, Zheng; Smola, Alexander J.

Published

2020

Language

EN