Can I read Towards Understanding How Attention Mechanism Works in Deep Learning on EtoBox?

Towards Understanding How Attention Mechanism Works in Deep Learning by Ruan, Tianyu; Zhang, Shihua is a scholarly article available to read on EtoBox.

What is Towards Understanding How Attention Mechanism Works in Deep Learning about?

Attention mechanism has been extensively integrated within mainstream neural network architectures, such as Transformers and graph attention networks. Yet, its underlying working principles remain somewhat elusive. What is its essence? Are there any connections between it and traditional machine learning algorithms? In this study, we inspect the process of computing similarity using classic metrics and vector space properties in manifold learning, clustering, and supervised learning. We identify the key characteristics of similarity computation and information propagation in these methods and demonstrate that the self-attention mechanism in deep learning adheres to the same principles but operates more flexibly and adaptively. We decompose the self-attention mechanism into a learnable pseudo-metric function and an information propagation process based on similarity computation. We prove that the self-attention mechanism converges to a drift-diffusion process through continuous modeling provided the pseudo-metric is a transformation of a metric and certain reasonable assumptions hold. This equation could be transformed into a heat equation under a new metric. In addition, we give a

Author

Ruan, Tianyu; Zhang, Shihua

Published

2024

Language

EN