Can I read Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs on EtoBox?

Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs by Zhu, Jinguo; Zhu, Xizhou; Wang, Wenhai; Wang, Xiaohua; Li, Hongsheng; Wang, Xiaogang; Dai, Jifeng is a scholarly article available to read on EtoBox.

What is Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs about?

To build an artificial neural network like the biological intelligence system, recent works have unified numerous tasks into a generalist model, which can process various tasks with shared parameters and do not have any task-specific modules. While generalist models achieve promising results on various benchmarks, they have performance degradation on some tasks compared with task-specialized models. In this work, we find that interference among different tasks and modalities is the main factor to this phenomenon. To mitigate such interference, we introduce the Conditional Mixture-of-Experts (Conditional MoEs) to generalist models. Routing strategies under different levels of conditions are proposed to take both the training/inference cost and generalization ability into account. By incorporating the proposed Conditional MoEs, the recently proposed generalist model Uni-Perceiver can effectively mitigate the interference across tasks and modalities, and achieves state-of-the-art results on a series of downstream tasks via prompt tuning on 1% of downstream data. Moreover, the introduction of Conditional MoEs still holds the generalization ability of generalist models to conduct zero-s

Author

Zhu, Jinguo; Zhu, Xizhou; Wang, Wenhai; Wang, Xiaohua; Li, Hongsheng; Wang, Xiaogang; Dai, Jifeng

Published

2022

Language

EN