Can I read Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules on EtoBox?

Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules by Gong, Zhuocheng; Lv, Ang; Guan, Jian; Yan, Junxi; Wu, Wei; Zhang, Huishuai; Huang, Minlie; Zhao, Dongyan; Yan, Rui is a scholarly article available to read on EtoBox.

What is Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules about?

Is it always necessary to compute tokens from shallow to deep layers in Transformers? The continued success of vanilla Transformers and their variants suggests an undoubted "yes". In this work, however, we attempt to break the depth-ordered convention by proposing a novel architecture dubbed mixture-of-modules (MoM), which is motivated by an intuition that any layer, regardless of its position, can be used to compute a token as long as it possesses the needed processing capabilities. The construction of MoM starts from a finite set of modules defined by multi-head attention and feed-forward networks, each distinguished by its unique parameterization. Two routers then iteratively select attention modules and feed-forward modules from the set to process a token. The selection dynamically expands the computation graph in the forward pass of the token, culminating in an assembly of modules. We show that MoM provides not only a unified framework for Transformers and their numerous variants but also a flexible and learnable approach for reducing redundancy in Transformer parameterization. We pre-train various MoMs using OpenWebText. Empirical results demonstrate that MoMs, of different p

Author

Gong, Zhuocheng; Lv, Ang; Guan, Jian; Yan, Junxi; Wu, Wei; Zhang, Huishuai; Huang, Minlie; Zhao, Dongyan; Yan, Rui

Published

2024

Language

EN