Can I read LocalViT: Analyzing Locality in Vision Transformers on EtoBox?

LocalViT: Analyzing Locality in Vision Transformers by Li, Yawei; Zhang, Kai; Cao, Jiezhang; Timofte, Radu; Magno, Michele; Benini, Luca; Van Gool, Luc is a scholarly article available to read on EtoBox.

What is LocalViT: Analyzing Locality in Vision Transformers about?

The aim of this paper is to study the influence of locality mechanisms in vision transformers. Transformers originated from machine translation and are particularly good at modelling long-range dependencies within a long sequence. Although the global interaction between the token embeddings could be well modelled by the self-attention mechanism of transformers, what is lacking is a locality mechanism for information exchange within a local region. In this paper, locality mechanism is systematically investigated by carefully designed controlled experiments. We add locality to vision transformers into the feed-forward network. This seemingly simple solution is inspired by the comparison between feed-forward networks and inverted residual blocks. The importance of locality mechanisms is validated in two ways: 1) A wide range of design choices (activation function, layer placement, expansion ratio) are available for incorporating locality mechanisms and proper choices can lead to a performance gain over the baseline, and 2) The same locality mechanism is successfully applied to vision transformers with different architecture designs, which shows the generalization of the locality conce

Author

Li, Yawei; Zhang, Kai; Cao, Jiezhang; Timofte, Radu; Magno, Michele; Benini, Luca; Van Gool, Luc

Published

2021

Language

EN