Can I read Revisiting Vision Transformer from the View of Path Ensemble on EtoBox?

Revisiting Vision Transformer from the View of Path Ensemble by Chang, Shuning; Wang, Pichao; Luo, Hao; Wang, Fan; Shou, Mike Zheng is a scholarly article available to read on EtoBox.

What is Revisiting Vision Transformer from the View of Path Ensemble about?

Vision Transformers (ViTs) are normally regarded as a stack of transformer layers. In this work, we propose a novel view of ViTs showing that they can be seen as ensemble networks containing multiple parallel paths with different lengths. Specifically, we equivalently transform the traditional cascade of multi-head self-attention (MSA) and feed-forward network (FFN) into three parallel paths in each transformer layer. Then, we utilize the identity connection in our new transformer form and further transform the ViT into an explicit multi-path ensemble network. From the new perspective, these paths perform two functions: the first is to provide the feature for the classifier directly, and the second is to provide the lower-level feature representation for subsequent longer paths. We investigate the influence of each path for the final prediction and discover that some paths even pull down the performance. Therefore, we propose the path pruning and EnsembleScale skills for improvement, which cut out the underperforming paths and re-weight the ensemble components, respectively, to optimize the path combination and make the short paths focus on providing high-quality representation for

Author

Chang, Shuning; Wang, Pichao; Luo, Hao; Wang, Fan; Shou, Mike Zheng

Published

2023

Language

EN