Can I read DCT-Based Decorrelated Attention for Vision Transformers on EtoBox?

DCT-Based Decorrelated Attention for Vision Transformers by Pan, Hongyi; Hamdan, Emadeldeen; Zhu, Xin; Biswas, Koushik; Cetin, Ahmet Enis; Bagci, Ulas is a scholarly article available to read on EtoBox.

What is DCT-Based Decorrelated Attention for Vision Transformers about?

Central to the Transformer architectures' effectiveness is the self-attention mechanism, a function that maps queries, keys, and values into a high-dimensional vector space. However, training the attention weights of queries, keys, and values is non-trivial from a state of random initialization. In this paper, we propose two methods. (i) We first address the initialization problem of Vision Transformers by introducing a simple, yet highly innovative, initialization approach utilizing Discrete Cosine Transform (DCT) coefficients. Our proposed DCT-based attention initialization marks a significant gain compared to traditional initialization strategies; offering a robust foundation for the attention mechanism. Our experiments reveal that the DCT-based initialization enhances the accuracy of Vision Transformers in classification tasks. (ii) We also recognize that since DCT effectively decorrelates image information in the frequency domain, this decorrelation is useful for compression because it allows the quantization step to discard many of the higher-frequency components. Based on this observation, we propose a novel DCT-based compression technique for the attention function of Visio

Author

Pan, Hongyi; Hamdan, Emadeldeen; Zhu, Xin; Biswas, Koushik; Cetin, Ahmet Enis; Bagci, Ulas

Published

2024

Language

EN